fluent-plugin-elasticsearch：「無法推送日誌到Elasticsearch」錯誤與「錯誤」=> {「type」=>「mapper_parsing_exception」}

Question

當我使用流暢插件彈性搜索將Fluentd收集的數據注入Elasticsearch ，一些數據引起以下錯誤：

2017-04-09 23:47:37 +0900 [error]: Could not push log to Elasticsearch: {"took"=>3, "errors"=>true, "items"=>[{"index"=>{"_index"=>"logstash-201704", "_type"=>"ruby", "_id"=>"AVtTLz_cUzkwT9CQCxrH", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [message]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:27"}}}}, .....]}

似乎elasticsearch禁止數據錯誤failed to parse [message]和Can't get text on a START_OBJECT at 1:27。但是我看不到發送給Elasticsearch的數據和錯誤。

任何想法？

Answer 1

fluent-plugin-elasticsearch使用_bulk API發送數據。我把請求傾銷碼上/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-5.0.4/lib/elasticsearch/api/actions/bulk.rb如下：

def bulk(arguments={}) 
    ... 
     payload = body 
    end 
    $log.info([method, path, params, payload].inspect) # <=== here ($log is global logger of fluentd) 
    perform_request(method, path, params, payload).body 

而且我發現發送到Elasticsearch請求是如下：

POST /_bulk 
{"index":{"_index":"logstash-201704","_type":"ruby"}} 
{"level":"INFO","message":{"status":200,"time":{"total":46.26,"db":33.88,"view":12.38},"method":"PUT","path":"filtered","params":{"time":3815.904,"chapter_index":0},"response":[{}]},"node":"main","time":"2017-04-09T14:39:06UTC","tag":"filtered.console","@timestamp":"2017-04-09T23:39:06+09:00"} 

問題是message字段包含JSON對象，雖然這一領域被映射爲Elasticsearch上的分析字符串。