Logstash濾波

Question

我具有與以下格式寫入JSON對象（逐行）到/var/log/myLog.json一個Python腳本：

{"timestamp":"2016-07-21T01:20:04.392799-0400","in_iface":"docker0","event_type":"alert","src_ip":"172.17.0.2","dest_ip":"172.17.0.3","proto":"ICMP","icmp_type":0,"icmp_code":0,"alert":{"action":"allowed","**gid**":2,"signature_id":2,"rev":0,"signature":"ICMP msg","category":"","severity":3},"payload":"hFuQVwAA","payload_printable":"kk"} 

我想使用Logstash，以便：

1. 閱讀JSON對象，逐行從/var/log/myLog.json

2. 解析GID和轉發到另一臺機器作爲UDP MSG（賦予了特定的IP地址+端口） - 例如：if gid == 2 th恩此JSON對象轉發到172.123.10.3:10001

此外，我希望能夠動態更新Logstash配置文件過濾器（又名，才能夠添加另一個規則，如：「如果GID == x然後將這個json對象轉換爲另一個IP）。

我該怎麼做？

Logstash配置文件應該如何顯示？ 以及如何插入/刪除動態過濾器的命令看起來像？

謝謝，夥計們。

Answer 1

您可以按照以下配置運行logstash。 和我已經測試了兩個樣本json數據。

{"timestamp":"2016-07-21T01:20:04.392799-0400","in_iface":"docker0","event_type":"alert","src_ip":"172.17.0.2","dest_ip":"172.17.0.3","proto":"ICMP","icmp_type":0,"icmp_code":0,"alert":{"action":"allowed","gid":2,"signature_id":2,"rev":0,"signature":"ICMP msg","category":"","severity":3},"payload":"hFuQVwAA","payload_printable":"kk"} 
{"timestamp":"2016-07-21T01:20:04.392799-0400","in_iface":"docker0","event_type":"alert","src_ip":"172.17.0.2","dest_ip":"172.17.0.3","proto":"ICMP","icmp_type":0,"icmp_code":0,"alert":{"action":"allowed","gid":3,"signature_id":2,"rev":0,"signature":"ICMP msg","category":"","severity":3},"payload":"hFuQVwAA","payload_printable":"kk"} 



input { 
    file { 
     path => "/etc/logstash/jsonSample.log" 
     start_position => "beginning" 
     sincedb_path => "/dev/null" 
    } 
} 

filter { 
       json { 
         source => "message" 
         target => "doc" 
         add_field => {"alert.gid" => "%{[doc][alert][gid]}"} 
         add_tag => ["tagName_%{[doc][alert][gid]}"] 
       } 


} 


output { 
if "tagName_2" in [tags] { 
stdout {codec => rubydebug} 
}else if "tagName_3" in [tags] { 
} 

} 

然後你就可以看到結果

{ 
     "message" => "{\"timestamp\":\"2016-07-21T01:20:04.392799-0400\",\"in_iface\":\"docker0\",\"event_type\":\"alert\",\"src_ip\":\"172.17.0.2\",\"dest_ip\":\"172.17.0.3\",\"proto\":\"ICMP\",\"icmp_type\":0,\"icmp_code\":0,\"alert\":{\"action\":\"allowed\",\"gid\":2,\"signature_id\":2,\"rev\":0,\"signature\":\"ICMP msg\",\"category\":\"\",\"severity\":3},\"payload\":\"hFuQVwAA\",\"payload_printable\":\"kk\"}", 
     "@version" => "1", 
    "@timestamp" => "2016-07-25T04:41:11.980Z", 
      "path" => "/etc/logstash/jsonSample.log", 
      "host" => "baklava", 
      "doc" => { 
       "timestamp" => "2016-07-21T01:20:04.392799-0400", 
       "in_iface" => "docker0", 
       "event_type" => "alert", 
        "src_ip" => "172.17.0.2", 
        "dest_ip" => "172.17.0.3", 
        "proto" => "ICMP", 
       "icmp_type" => 0, 
       "icmp_code" => 0, 
        "alert" => { 
        "action" => "allowed", 
        "gid" => 2, 
      "signature_id" => 2, 
        "rev" => 0, 
       "signature" => "ICMP msg", 
       "category" => "", 
       "severity" => 3 
     }, 
        "payload" => "hFuQVwAA", 
     "payload_printable" => "kk" 
    }, 
    "alert.gid" => 2, 
      "tags" => [ 
     [0] "tagName_2" 
    ] 
} 

你也可以改變施加在它上面的配置。

問候。

你可以參考事件和JSON過濾 https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html