如何讀取Pig中的值列表作爲包並將其與特定值進行比較？

Question

輸入：

IDS：

1111,2222,3333,4444 

員工：

{"name":"abc","id":"1111"} {"name":"xyz","id":"10"} 
{"name":"z","id":"100"} {"name":"m","id":"99"} 
{"name":"pqr","id":"3333"} 

我想篩選員工的ID存在於給定的列表。

預期輸出：

{"name":"xyz","id":"10"} {"name":"z","id":"100"} 
{"name":"m","id":"99"} 

現有代碼：

idList = LOAD 'pathToFile' USING PigStorage(',') AS (id:chararray); 
empl = LOAD 'pathToFile' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad') AS (data:map[]); 
output = FILTER empl BY data#'id' in (idList); 
-- not working, states: A column needs to be projected from a relation for it to be used as a scalar 
output = FILTER empl BY data#'id' in (idList#id); 
-- not working, states: mismatched input 'id' expecting set null 

Answer 1

JsonLoad()是pig > 0.10本地人，你可以指定模式：

empl = LOAD 'pathToFile' USING JsonLoader('name:chararray, id:chararray'); 
DUMP empl; 

(abc,1111) 
(xyz,10) 
(z,100) 
(m,99) 
(pqr,3333) 

您正在加載idList作爲chararray類型的一列表，但您需要一個列表。

加載它作爲一個列的表（意味着你修改文件，所以每行只有一條記錄）：

idList = LOAD 'pathToFile' USING PigStorage(',') AS (id:chararray); 
DUMP idList; 

(1111) 
(2222) 
(3333) 
(4444) 

或作爲一個在線文件，所以沒有關係，我們將更改分隔符「T拆分成多列（否則會導致裝載只有第一列）：

idList = LOAD 'pathToFile' USING PigStorage(' ') AS (id:chararray); 
idList = FOREACH idList GENERATE FLATTEN(TOKENIZE(id, '[,]')) AS id; 
DUMP idList; 

(1111) 
(2222) 
(3333) 
(4444) 

現在我們可以做一個LEFT JOIN，看看哪些id中不存在的idList然後FILTER只有那些保持。 output是保留字，請不要使用：

res = JOIN empl BY id LEFT, idList BY id; 
res = FILTER res BY idList::id IS NULL; 
DUMP res; 

(xyz,10,) 
(m,99,) 
(z,100,)