Logstash - > Elasticsearch未正確映射

因此，我最近創建了一個使用此站點作爲模板的ELK集羣。 ELK Cluster Setup Logstash - > Elasticsearch未正確映射

我遇到了Logstash處理節點上的json模板未在實際Elasticsearch數據節點上使用的問題。我可以看到映射已經在HQ中創建，但另一個是使用一些動態創建的映射創建的。正確完成的映射在數據節點上稱爲「Sourcefire」，但它也創建了一個名爲「sourcfire」的不正確的映射。

我無法弄清楚這一點，我正在學習這東西，所以任何幫助表示讚賞。請參閱下面的代碼片段。

Logstash.conf

input { 
    tcp { 
     port => 5170 
     type => "sourcefire" 
    } 
} 

filter { 

    mutate{ 
     split => ["message", "|"] 
     add_field => { 
      "event" => "%{message[5]}" 
      "eventSource" => "%{message[1]}" 
     } 
    } 

    kv { 
     include_keys => ["dhost", "dst", "dpt", "shost", "src", "spt", "rt"] 
    } 

    mutate { 
     rename => [ "dhost", "destinationHost" ] 
     rename => [ "dst", "destinationAddress" ] 
     rename => [ "dpt", "destinationPort" ] 
     rename => [ "shost", "sourceHost" ] 
     rename => [ "src", "sourceAddress" ] 
     rename => [ "spt", "sourcePort" ] 
    } 

    date { 
     match => ["rt","UNIX_MS"] 
     target => "eventDate" 
    } 

    geoip { 
     add_tag => [ "sourceGeo" ] 
     source => "src" 
     database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" 
    } 

    geoip { 
     add_tag => [ "destinationGeo" ] 
     source => "src" 
     database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat" 
    } 
} 

output { 
    if [type] == "sourcefire" { 
     elasticsearch { 
      cluster => "XXX-cluster" 
      flush_size => 1 
      manage_template => true 
      template => "/opt/logstash/lib/logstash/outputs/elasticsearch/elasticsearch-sourcefire.json" 
     } 
    } 
}

Elasticsearch JSON模板

{ 
    "template": "logstash-*", 
    "settings": { 
     "index.refresh_interval": "5s" 
    }, 
    "mappings": { 
     "Sourcefire": { 
      "_all": { 
       "enabled": true 
      }, 
      "properties": { 
       "@timestamp": { 
        "type": "date", 
        "format": "basicDateTimeNoMillis" 
       }, 
       "@version": { 
        "type": "string", 
        "index": "not_analyzed" 
       }, 
       "geoip": { 
        "type": "object", 
        "dynamic": true, 
        "path": "full", 
        "properties": { 
         "location": { 
          "type": "geo_point" 
         } 
        } 
       }, 
       "event": { 
        "type": "string", 
        "index": "not_analyzed" 
       }, 
       "eventDate": { 
        "type": "date", 
        "format": "basicDateTimeNoMillis" 
       }, 
       "destinationAddress": { 
        "type": "ip" 
       }, 
       "destinationHost": { 
        "type": "string", 
        "index": "not_analyzed" 
       }, 
       "destinationPort": { 
        "type": "integer", 
        "index": "not_analyzed" 
       }, 
       "sourceAddress": { 
        "type": "ip" 
       }, 
       "sourceHost": { 
        "type": "string", 
        "index": "not_analyzed" 
       }, 
       "sourcePort": { 
        "type": "integer", 
        "index": "not_analyzed" 
       } 
      } 
     } 
    } 
}

來源

2014-09-30 Elorion

可以使用elasticsearch輸出的template_overwrite財產。但是，不能保證始終正常工作，特別是如果您有多個同時運行的logstash實例。另外，根據您的elasticsearch映射配置設置，特別是動態映射和默認設置（http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-dynamic-mapping.html），您可能會得到與預期不同的結果。

根據我的經驗，我發現最好手動控制elasticsearch中的索引映射（使用fiddler或elasticsearch-head管理站點等工具）。這是因爲當多個logstash實例一起覆蓋映射時，我經歷了各種意想不到的結果，禁用了我設置的特定彈性搜索字段（如_ttl）。

來源

2014-10-03 08:03:37 Roman

只需刪除存儲的模板，它應該重新回來：例如，如果你的模板名稱是logstash：

curl -XDELETE localhost:9200/_template/logstash

此外，如果你正在寫的同一個索引，你不能改變的映射。您需要重新創建索引（確保首先停止logstash以防止任何機上）。

來源

2015-12-29 03:22:14 kuronue

Logstash - > Elasticsearch未正確映射

回答

相關問題