geoip查找失敗彈性堆棧logstash

使用filebeat從Windows系統發送apache日誌到linux EC2中的logstash服務器，然後發送到彈性搜索和Kibana。geoip查找失敗彈性堆棧logstash

彈性搜索和Kibana - 5.3 Logstash和filebeat - 5.3

filebeat.yml：

filebeat.prospectors: 

- input_type: log 

    # Paths that should be crawled and fetched. Glob based paths. 
    paths: 
    #- /var/log/*.log 
    #- c:\programdata\elasticsearch\logs\* 
    - C:\Users\Sagar\Desktop\elastic_test4\data\log\* 

output.logstash: 
    # The Logstash hosts 
    hosts: ["10.101.00.11:5044"] 
    template.name: "filebeat-poc" 
    template.path: "filebeat.template.json" 
    template.overwrite: false

logstash.conf在Ubuntu Linux操作系統的EC2實例

input { 
    beats { 
    port => 5044 
    } 
} 
filter { 
    grok { 
     match => { 
     "message" => "%{COMBINEDAPACHELOG}" 
     } 
    } 
    geoip { 
     source => "clientip" 
     target => "geoip" 
     add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] 
     add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] 
    } 
    mutate { 
     convert => [ "[geoip][coordinates]", "float"] 
    } 
} 
output { 
    elasticsearch { 
    hosts => ["elastic-instance-1.es.amazonaws.com:80"] 
    index => "apache-%{+YYYY.MM.dd}" 
    document_type => "apache_logs" 
} 
    stdout { codec => rubydebug } 
}

我的虛擬日誌文件。

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846 
64.242.88.10 - - [07/Mar/2004:16:06:51 -0800] "GET /twiki/bin/rdiff/TWiki/NewUserTemplate?rev1=1.3&rev2=1.2 HTTP/1.1" 200 4523 
64.242.88.10 - - [07/Mar/2004:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291 
64.242.88.10 - - [07/Mar/2004:16:11:58 -0800] "GET /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 200 7352 
64.242.88.10 - - [07/Mar/2004:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253 
64.242.88.10 - - [07/Mar/2004:16:23:12 -0800] "GET /twiki/bin/oops/TWiki/AppendixFileSystem?template=oopsmore¶m1=1.12¶m2=1.12 HTTP/1.1" 200 11382 
64.242.88.10 - - [07/Mar/2004:16:24:16 -0800] "GET /twiki/bin/view/Main/PeterThoeny HTTP/1.1" 200 4924 
64.242.88.10 - - [07/Mar/2004:16:29:16 -0800] "GET /twiki/bin/edit/Main/Header_checks?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12851 
64.242.88.10 - - [07/Mar/2004:16:30:29 -0800] "GET /twiki/bin/attach/Main/OfficeLocations HTTP/1.1" 401 12851 
64.242.88.10 - - [07/Mar/2004:16:31:48 -0800] "GET /twiki/bin/view/TWiki/WebTopicEditTemplate HTTP/1.1" 200 3732 
64.242.88.10 - - [07/Mar/2004:16:32:50 -0800] "GET /twiki/bin/view/Main/WebChanges HTTP/1.1" 200 40520 
64.242.88.10 - - [07/Mar/2004:16:33:53 -0800] "GET /twiki/bin/edit/Main/Smtpd_etrn_restrictions?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12851

我可以將這些日誌發送到彈性和kibana儀表板。管道設置和它的工作，但geoip不起作用。

這是我的搜索kibana輸出。

{ 
     "_index": "apache-2017.06.15", 
     "_type": "apache_logs", 
     "_id": "AVyqJhi6ItD-cRj2_AW6", 
     "_score": 1, 
     "_source": { 
      "@timestamp": "2017-06-15T05:06:48.038Z", 
      "offset": 154, 
      "@version": "1", 
      "input_type": "log", 
      "beat": { 
      "hostname": "sagar-machine", 
      "name": "sagar-machine", 
      "version": "5.3.2" 
      }, 
      "host": "by-df164", 
      "source": """C:\Users\Sagar\Desktop\elastic_test4\data\log\apache-log.log""", 
      "message": """64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846""", 
      "type": "log", 
      "tags": [ 
      "beats_input_codec_plain_applied", 
      "_grokparsefailure", 
      "_geoip_lookup_failure" 
      ] 
     } 
     }

任何想法爲什麼我面臨這個問題。

來源

2017-06-15 blackOcean

您有_grokparsefailure，因此clientip字段不存在。這會導致_geoip_lookup_failure，因爲geoip篩選器正在採購不存在的clientip字段。

您的日誌匹配%{COMMONAPACHELOG}模式，而不是您正在使用的模式。所以，你的配置看起來像：

filter { 
    grok { 
     match => { 
     "message" => "%{COMMONAPACHELOG}" 
     } 
    } 
    ... 
}

使用正確的模式後，你應該注意到了clientip領域存在，在此之後，希望在geoip篩選器工作。 :)

來源

2017-06-15 20:27:51 fylie

我不知道你的日誌格式是否正確或不適用於Apache。因爲你的日誌正在尋找這樣

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846

和標準Apache日誌看起來是這樣

149.148.126.144 - - [10/Sep/2017:06:30:44 -0700] "GET /apps/cart.jsp?appID=6944 HTTP/1.0" 200 4981 "http://hernandez.net/app/main/search/homepage.php" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/5322 (KHTML, like Gecko) Chrome/13.0.896.0 Safari/5322"

我建議請您標準化未來Apache日誌格式。否則默認grok配置將無法爲您工作。然後你必須爲自定義日誌編寫自己的grok模式。這將解析您的到來日誌行

除此之外，還有很多原因你得到這樣的錯誤

在filebeat 配置你沒有評論「filebeat模板」

。 filebeat模板，我們將在您從filebeat直接發送日誌到彈性模式時使用。

更改您的filebeat的配置。

filebeat.prospectors: 
- input_type: log 
    paths: C:\Users\Sagar\Desktop\elastic_test4\data\log\*.log 

output.logstash: 
    hosts: ["10.101.00.11:5043"]

您必須安裝 '攝取-geoip的' 過濾器插件安裝到彈性搜索。如果你沒有使用任何外部數據庫或服務。

您可以使用下面的命令

elasticsearch-plugin install ingest-geoip

安裝彈性插件，我不知道你的彈性實例，因爲它是由默認監聽9200端口，而不是80端口。

你必須改變logstash的配置腳本。下面是這樣的東西。

input { 
    beats { 
     host => "10.101.00.11" 
     port => "5044" 
    } 
} 

filter { 
    grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } 
    geoip { source => "clientip" } 
} 

output { 
    elasticsearch { 
     #hosts => ["elastic-instance-1.es.amazonaws.com:80"] 
     hosts => ["elastic-instance-1.es.amazonaws.com:9200"] 
     index => "apache-%{+YYYY.MM.dd}" 
    } 
    stdout { codec => rubydebug } 
}

應用這些配置後，您的輸出將如下所示。

{ 
    "_index": "apache-2017.09.21", 
    "_type": "log", 
    "_id": "AV6kqsr3A-YOTHfOm2US", 
    "_version": 1, 
    "_score": null, 
    "_source": { 
    "request": "/apps/cart.jsp?appID=9421", 
    "agent": "\"Mozilla/5.0 (Windows 95; sl-SI; rv:1.9.2.20) Gecko/2017-08-19 13:55:15 Firefox/12.0\"", 
    "geoip": { 
     "city_name": "Beijing", 
     "timezone": "Asia/Shanghai", 
     "ip": "106.121.102.198", 
     "latitude": 39.9289, 
     "country_name": "China", 
     "country_code2": "CN", 
     "continent_code": "AS", 
     "country_code3": "CN", 
     "region_name": "Beijing", 
     "location": { 
     "lon": 116.3883, 
     "lat": 39.9289 
     }, 
     "region_code": "11", 
     "longitude": 116.3883 
    }, 
    "offset": 11050275, 
    "auth": "-", 
    "ident": "-", 
    "input_type": "log", 
    "verb": "POST", 
    "source": "C:\\Users\\admin\\Desktop\\experiment\\Elastic\\access_log_20170915-005134.log", 
    "message": "106.121.102.198 - - [19/Dec/2017:05:54:29 -0700] \"POST /apps/cart.jsp?appID=9421 HTTP/1.0\" 200 4984 \"http://cross.com/login/\" \"Mozilla/5.0 (Windows 95; sl-SI; rv:1.9.2.20) Gecko/2017-08-19 13:55:15 Firefox/12.0\"", 
    "type": "log", 
    "tags": [ 
     "beats_input_codec_plain_applied" 
    ], 
    "referrer": "\"http://cross.com/login/\"", 
    "@timestamp": "2017-09-21T13:39:55.047Z", 
    "response": "200", 
    "bytes": "4984", 
    "clientip": "106.121.102.198", 
    "@version": "1", 
    "beat": { 
     "hostname": "DESKTOP-16QDF02", 
     "name": "DESKTOP-16QDF02", 
     "version": "5.5.2" 
    }, 
    "host": "DESKTOP-16QDF02", 
    "httpversion": "1.0", 
    "timestamp": "19/Dec/2017:05:54:29 -0700" 
    }, 
    "fields": { 
    "@timestamp": [ 
     1506001195047 
    ] 
    }, 
    "sort": [ 
    1506001195047 
    ] 
}

我希望這是你正在尋找的解決方案..

來源

2017-09-25 11:03:24

您可能需要確保Apache日誌是在正確的模式：

SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY})?%{SYSLOGHOST:logsource} %{SYSLOGPROG}: 
COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) 
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

有關的圖案grok比賽，你可以查看https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns的詳細資料。

除此之外，你可能對https://www.ip2location.com/tutorials/how-to-use-ip2location-filter-plugin-with-elk看看了。

來源

2017-11-07 01:28:11 Gary

雖然鏈接是有幫助的，這將是更好的格式化代碼塊在這裏複製的相關部分。 [不是每個人都可以訪問外部網站，並鏈接可能會隨時間分解（https://stackoverflow.com/help/how-to-ask） –

雖然此鏈接可以回答這個問題，最好是包括主要部分的答案，並提供參考鏈接。如果鏈接頁面更改，則僅鏈接答案可能會失效。 - [來自評論]（/ review/low-quality-posts/17860045） – G5W

geoip查找失敗彈性堆棧logstash

回答

相關問題