1. 首頁
  2. 問答
  3. 用於websphere/java日誌的Logstash Multiline過濾器

用於websphere/java日誌的Logstash Multiline過濾器

2015-11-04 83 views
1

你好，我的logstash多行配置有問題。我正在解析websphere/java日誌，並且多行不適用於某些日誌情況。用於websphere/java日誌的Logstash Multiline過濾器

我的多行配置看起來像這樣。我嘗試了幾種類型的正則表達式，但沒有人工作。

codec => multiline { 
    pattern => "^\A%{SYSLOG5424SD}" 
    negate => true 
    what => previous 
}

這是日誌的例子是不正確的方式解析：

[1.6.2015 15:02:46:635 CEST] 00000109 BusinessExcep E CNTR0020E: EJB threw an unexpected (non-declared) exception during invocation of method  "processCommand" on bean  "BeanId(Issz_Produkcia_2.1.63#Ssz_Server_EJB.jar#CommandDispatcherImpl, null)". Exception data: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back; nested exception is: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back; nested exception is: javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout 
javax.ejb.EJBTransactionRolledbackException: Transaction rolled back; nested exception is: javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout 
javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout 
at com.ibm.tx.jta.impl.EmbeddableTranManagerImpl.completeTxTimeout(EmbeddableTranManagerImpl.java:62) 
at com.ibm.tx.jta.impl.EmbeddableTranManagerSet.completeTxTimeout(EmbeddableTranManagerSet.java:85) 
at com.ibm.ejs.csi.TransactionControlImpl.completeTxTimeout(TransactionControlImpl.java:1347) 
at com.ibm.ejs.csi.TranStrategy.postInvoke(TranStrategy.java:273) 
at com.ibm.ejs.csi.TransactionControlImpl.postInvoke(TransactionControlImpl.java:579) 
at com.ibm.ejs.container.EJSContainer.postInvoke(EJSContainer.java:4874) 
at sk.sits.upsvar.server.ejb.entitymanagers.EJSLocal0SLDokumentManagerImpl_18dd4eb4.findAllDokumentPripadByCriteriaMap(EJSLocal0SLDokumentManagerImpl_18dd4eb4.java) 
at sk.sits.upsvar.server.ejb.DataAccessServiceImpl.executeDokumentCmd(DataAccessServiceImpl.java:621) 
at sk.sits.upsvar.server.ejb.DataAccessServiceImpl.executeCmd(DataAccessServiceImpl.java:220) 
at sk.sits.upsvar.server.ejb.EJSLocal0SLDataAccessServiceImpl_6e5b0656.executeCmd(EJSLocal0SLDataAccessServiceImpl_6e5b0656.java) 
at sk.sits.upsvar.server.ejb.CommandDispatcherImpl.processSoloCommand(CommandDispatcherImpl.java:222) 
at sk.sits.upsvar.server.ejb.CommandDispatcherImpl._processCommand(CommandDispatcherImpl.java:151) 
at sk.sits.upsvar.server.ejb.CommandDispatcherImpl.processCommand(CommandDispatcherImpl.java:100) 
at sk.sits.upsvar.server.ejb.EJSLocal0SLCommandDispatcherImpl_b974dd5c.processCommand(EJSLocal0SLCommandDispatcherImpl_b974dd5c.java) 
at sk.sits.upsvar.server.ejb.SszServiceImpl.process(SszServiceImpl.java:146) 
at sk.sits.upsvar.server.ejb.EJSRemote0SLSszService_8e2ee81c.process(EJSRemote0SLSszService_8e2ee81c.java) 
at sk.sits.upsvar.server.ejb._EJSRemote0SLSszService_8e2ee81c_Tie.process(_EJSRemote0SLSszService_8e2ee81c_Tie.java) 
at sk.sits.upsvar.server.ejb._EJSRemote0SLSszService_8e2ee81c_Tie._invoke(_EJSRemote0SLSszService_8e2ee81c_Tie.java) 
at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:678) 
at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:525) 
at com.ibm.rmi.iiop.ORB.process(ORB.java:576) 
at com.ibm.CORBA.iiop.ORB.process(ORB.java:1578) 
at com.ibm.rmi.iiop.Connection.doRequestWork(Connection.java:3076) 
at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2946) 
at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:64) 
at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118) 
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1700) 
javax.ejb.EJBTransactionRolledbackException: Transaction rolled back; nested exception is: javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout 
Caused by: javax.transaction.TransactionRolledbackException: Transaction is ended due to timeout 
at com.ibm.tx.jta.impl.EmbeddableTranManagerImpl.completeTxTimeout(EmbeddableTranManagerImpl.java:62) 
at com.ibm.tx.jta.impl.EmbeddableTranManagerSet.completeTxTimeout(EmbeddableTranManagerSet.java:85) 
at com.ibm.ejs.csi.TransactionControlImpl.completeTxTimeout(TransactionControlImpl.java:1347) 
at com.ibm.ejs.csi.TranStrategy.postInvoke(TranStrategy.java:273) 
at com.ibm.ejs.csi.TransactionControlImpl.postInvoke(TransactionControlImpl.java:579) 
at com.ibm.ejs.container.EJSContainer.postInvoke(EJSContainer.java:4874) 
at sk.sits.upsvar.server.ejb.entitymanagers.EJSLocal0SLDokumentManagerImpl_18dd4eb4.findAllDokumentPripadByCriteriaMap(EJSLocal0SLDokumentManagerImpl_18dd4eb4.java) 
at sk.sits.upsvar.server.ejb.DataAccessServiceImpl.executeDokumentCmd(DataAccessServiceImpl.java:621) 
at sk.sits.upsvar.server.ejb.DataAccessServiceImpl.executeCmd(DataAccessServiceImpl.java:220) 
at sk.sits.upsvar.server.ejb.EJSLocal0SLDataAccessServiceImpl_6e5b0656.executeCmd(EJSLocal0SLDataAccessServiceImpl_6e5b0656.java) 
at sk.sits.upsvar.server.ejb.CommandDispatcherImpl.processSoloCommand(CommandDispatcherImpl.java:222) 
at sk.sits.upsvar.server.ejb.CommandDispatcherImpl._processCommand(CommandDispatcherImpl.java:151) 
at sk.sits.upsvar.server.ejb.CommandDispatcherImpl.processCommand(CommandDispatcherImpl.java:100) 
at sk.sits.upsvar.server.ejb.EJSLocal0SLCommandDispatcherImpl_b974dd5c.processCommand(EJSLocal0SLCommandDispatcherImpl_b974dd5c.java) 
at sk.sits.upsvar.server.ejb.SszServiceImpl.process(SszServiceImpl.java:146) 
at sk.sits.upsvar.server.ejb.EJSRemote0SLSszService_8e2ee81c.process(EJSRemote0SLSszService_8e2ee81c.java) 
at sk.sits.upsvar.server.ejb._EJSRemote0SLSszService_8e2ee81c_Tie.process(_EJSRemote0SLSszService_8e2ee81c_Tie.java) 
at sk.sits.upsvar.server.ejb._EJSRemote0SLSszService_8e2ee81c_Tie._invoke(_EJSRemote0SLSszService_8e2ee81c_Tie.java) 
at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:678) 
at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:525) 
at com.ibm.rmi.iiop.ORB.process(ORB.java:576) 
at com.ibm.CORBA.iiop.ORB.process(ORB.java:1578) 
at com.ibm.rmi.iiop.Connection.doRequestWork(Connection.java:3076) 
at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2946) 
at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:64) 
at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:118) 
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1700)

它是由線分析，我需要它解析在一起。我不知道是否有一些人物將他們分開。

我想這些模式：

pattern => "%{DATESTAMP} %{WORD:zone}]" 
pattern => "^\[" 
pattern => "\A"

還有很多很多我不記得他們。面對這個問題的人能幫助我嗎？

非常感謝您。

這是我的完整配置。

input { 
    file { 
     path => "D:\Log\Logstash\testlog.log" 
     type => "LOG" 
     start_position => "beginning" 
     codec => plain { 
      charset => "ISO-8859-1" 
     } 
     codec => multiline { 
      pattern => "^\A%{SYSLOG5424SD}" 
      negate => true 
      what => previous 
     } 
    } 
} 
filter { 
    grok{ 
     match => [ "message",".*exception.*"] 
     add_tag => "exception" 
    } 
    mutate{ 
     remove_tag => "_grokparsefailure" 
    } 
    grok { 
     match => [ "message","%{DATESTAMP} %{WORD:}] %{WORD:} %{WORD:}\s* W"] 
     add_tag => "Warning" 
     remove_tag => "_grokparsefailure" 
    } 
    grok { 
     match => [ "message","%{DATESTAMP} %{WORD:}] %{WORD:} %{WORD:}\s* F"] 
     add_tag => "Fatal" 
     remove_tag => "_grokparsefailure" 
    } 
    grok { 
     match => [ "message","%{DATESTAMP} %{WORD:}] %{WORD:} %{WORD:}\s* O"] 
     add_tag => "Message" 
     remove_tag => "_grokparsefailure" 
    } 
    grok { 
     match => [ "message","%{DATESTAMP} %{WORD:}] %{WORD:} %{WORD:}\s* C"] 
     add_tag => "Config" 
     remove_tag => "_grokparsefailure" 
    } 
    #if ("Warning" not in [tags]) { 
     grok { 
      match => [ "message","%{DATESTAMP} %{WORD:}] %{WORD:} %{WORD:}\s* E"] 
      add_tag => "Error" 
      remove_tag => "_grokparsefailure" 
     } 
    #}else { 
     grok { 
      match => [ "message","%{DATESTAMP} %{WORD:}] %{WORD:} %{WORD: }\s* I"] 
      add_tag => "Info" 
     } 
    #} 
    grok { 
     match => [ "message", "%{DATESTAMP} %{WORD:zone}] %{WORD:ID} %{WORD:CLASS}\s* . (.*\s){0,}%{GREEDYDATA:OBSAH}" ] 
    remove_tag => "_grokparsefailure" 
    } 
    grok { 
     match => [ "message", "%{DATESTAMP} %{WORD:zone}] %{WORD:ID} %{WORD:CLASS}\s* . (.*\s){0,}%{WORD:WAS_CODE}:%{GREEDYDATA:OBSAH}" ] 
              #"message","%{DATESTAMP} %{WORD:zone}] %{WORD:ID} %{WORD:CLASS}\s* W \s*\[SID:%{WORD:ISSZSID}]%{GREEDYDATA:OBSAH}"] 
     remove_tag => "_grokparsefailure" 
     add_tag => "was_error" 
    } 
    if ("was_error" not in [tags]) { 
     grok { 
      match => [ "message","%{DATESTAMP} %{WORD:zone}] %{WORD:ID} %{WORD:CLASS}\s* . \s*\[SID:%{WORD:ISSZSID}]%{GREEDYDATA:OBSAH}" ] 
      remove_tag => "_grokparsefailure" 
     } 
     if "_grokparsefailure" not in [tags] { 
      if [ISSZSID] != "null" { 
       mutate{ 
        add_tag => "ISSZwithID" 
        remove_tag => "_grokparsefailure" 
       } 
      } else { 
       mutate{ 
        add_tag => "ISSZnull" 
        remove_tag => "_grokparsefailure" 
       } 
      } 
     } 
    } 
} 

output { 
    if "_grokparsefailure" not in [tags] { 
     elasticsearch { 
      hosts => ["127.0.0.1:9200"] 
      #protocol => "http" 
     } 
    } 
    stdout {} 
}

來源

2015-11-04 Kristián Klostermann

+0

'codec => multiline {'看起來非常錯誤......'multiline'是一個過濾器，你能分享你的完整配置嗎？ – pagid

+0

我編輯了問題並添加了完整的配置。 –

回答

2

爲使用multiline與其它編解碼器以及編解碼器是相當不在於它的目的是爲承擔。我寧願將它用作單個編解碼器或作爲過濾器。

將您的配置到這一點，你會得到您要查找的結果：

input { 
    file {  
    path => "D:\Log\Logstash\testlog.log" 
    type => "LOG" 
    start_position => "beginning" 
    codec => plain { charset => "ISO-8859-1" } 
    } 
} 
filter { 
    multiline { 
     pattern => "^\A%{SYSLOG5424SD}" 
     negate => true 
     what => previous 
    } 
    # ... all other filters 
} 
output { 
# your output definitions 
}

一個著名的多解析例子是從約旦Sissle關於MySQL日誌解析之一：https://gist.github.com/jordansissel/3753353

乾杯

來源

2015-11-18 12:36:13 pagid

+0

現在我遇到了工人的問題。由於多行輸入中的過濾器，我可以只有一個。有沒有解決方案？而我不知道爲什麼1個工人不解析我的文件。當我將它改回到我的配置中，更多的工人再次工作。 –

+0

右 - 多行有一些顯着的缺點。放棄其他編解碼器可能是一種選擇，但可能會導致字符被破壞。 – pagid

相關問題

 相關問題