如何讀取shell腳本.sh文件中的json日誌文件，並通過計算日誌文件中出現的次數來獲取分析數據

Question

需要從json日誌文件讀取數據並從使用shell獲取分析數據腳本。

日誌文件包含如下JSON：

{ 
info: 'label1', 
description: 'some desc', 
timestamp: '2016-07-27T06:24:50.335Z' 
} 
{ 
info: 'label2', 
description: 'some desc', 
timestamp: '2016-07-27T06:24:50.335Z' 
} 
{ 
info: 'label2', 
description: 'some desc', 
timestamp: '2016-07-27T06:24:50.335Z' 
} 
{ 
info: 'label2', 
description: 'some desc', 
timestamp: '2016-07-29T06:24:50.335Z' 
} 
{ 
info: 'label3', 
description: 'some desc', 
timestamp: '2016-07-29T06:24:50.335Z' 
} 

我需要的結果如下（使用shell腳本）：

Labels Date     Count 

label1 2016-07-27    1 
label2 2016-07-27    2    
label2 2016-07-29    1 
label3 2016-07-29    1 

這是到目前爲止，我可以去，需要一些關於如何處理的建議。\

#!/bin/bash 
my_dir=`dirname $0` 
file="out.log" 
#keysFile="$my_dir/keys.txt" 
for log in $(cat $file | jq '{id: .info,time: .timestamp}'); do 
#This is as far as I could get. I was able to read the data in the form of {id: 'label1', time: '2016-07-27T06:24:50.335Z' } 
#Now I need to somehow create a key value thing in shell and store timestamp/label as key and increment the count 
echo $log 
done 

Answer 1

鑑於你的輸入，你可以創建爲CSV數據使用此：

$ jq -rs ' 
def to_csv($headers): 
    def _object_to_csv: 
     ($headers | @csv), 
     (.[] | [.[$headers[]]] | @csv); 
    def _array_to_csv: 
     ($headers | @csv), 
     (.[][:$headers|length] | @csv); 
    if .[0]|type == "object" then 
     _object_to_csv 
    else 
     _array_to_csv 
    end; 
map({ label: .info, date: .timestamp[:10] }) 
    | group_by(.) 
    | map(.[0] + { count: length }) 
    | to_csv(["label", "date", "count"]) 
' input.json 

這產生了：

"label","date","count" 
"label1","2016-07-27",1 
"label2","2016-07-27",2 
"label2","2016-07-29",1 
"label3","2016-07-29",1 

Answer 2

下面是使用減少，而不是GROUP_BY的方法。

假設你的數據在out.log和filter.jq

["Labels", "Date", "Count"], 
["", "", ""], 
(
    reduce .[] as $r (
     {} 
    ; [$r.info, $r.timestamp[0:10]] as $p 
    | setpath($p;getpath($p)+1) 
) 
    | tostream 
    | select(length==2) 
    | flatten 
) 
| @tsv 

以下過濾器可以運行

jq -M -s -r -f filter.jq out.log 

產生製表符分隔輸出

Labels Date Count 

label1 2016-07-27 1 
label2 2016-07-27 2 
label2 2016-07-29 1 
label3 2016-07-29 1