基於標題記錄的Hive查詢結果

Question

我有一個像下面這樣的標記文件，它包含從源系統將4個不同文件合併爲一個文件的數據。

NEWFILE =是數據的分隔符。例如NEWFILE = STUDENT行後面的所有數據，直到NEWFILE = SUBJECT行都屬於STUDENT文件。 問題是我們沒有任何模式來分隔每個文件的記錄。 另外源系統不能將文件分成4個文件。

我需要加載這個單一的輸入文件，並根據記錄的標題分開記錄。

我所做的是將數據加載到Hive表中，並嘗試了ROW_NUMBER &隨機函數。

我想過使用ROW_NUMBER函數來標識每個頭的行，然後過濾標題行之間的記錄，但是ROW_NUMBER函數輸出與文件的實際行順序不同。由於這一行屬於學生可能被分配到SUBJECT。

我不能使用隨機函數，因爲它也並沒有給實際的行號

文件內容數據如下

NEWFILE=STUDENT 
100 XYZ 
101 ABC 
102 DEF 
NEWFILE=SUBJECT 
1 ENGLISH 
2 MATHS 
NEWFILE=TEACHERS 
110 AAAAAAAA 
111 BBBBBBB 
222 CCCCCCC 
333 DDDDDD 
NEWFILE=CLASSES 
1 CLASS-1 
2 CLASS-2 

給請告知我如何能實現我想要的輸出。

Answer 1

create external table myfile (rec string) 
row format delimited 
fields terminated by ',' 
tblproperties ('serialization.last.column.takes.rest'='true')  
; 

---

select  rec 
      ,ifn 
      ,ifn_newfile_seq 

      ,row_number() over 
      (
       partition by ifn_newfile_seq 
       order by  boif 
      ) as ifn_newfile_rec_seq 


from  (select rec 
        ,input__file__name    as ifn 
        ,block__offset__inside__file  as boif 

        ,count(case when rec like 'NEWFILE=%' then 1 end) over 
        (
         partition by input__file__name 
         order by  block__offset__inside__file 
        ) as ifn_newfile_seq 


      from myfile 
      ) l 
; 

---

+------------------+----------------------------------------------+-----------------+---------------------+ 
| rec    | ifn           | ifn_newfile_seq | ifn_newfile_rec_seq | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| NEWFILE=STUDENT | file:/home/cloudera/local_db/myfile/file.txt | 1    | 1     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| 100 XYZ   | file:/home/cloudera/local_db/myfile/file.txt | 1    | 2     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| 101 ABC   | file:/home/cloudera/local_db/myfile/file.txt | 1    | 3     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| 102 DEF   | file:/home/cloudera/local_db/myfile/file.txt | 1    | 4     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| NEWFILE=SUBJECT | file:/home/cloudera/local_db/myfile/file.txt | 2    | 1     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| 1 ENGLISH  | file:/home/cloudera/local_db/myfile/file.txt | 2    | 2     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| 2 MATHS   | file:/home/cloudera/local_db/myfile/file.txt | 2    | 3     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| NEWFILE=TEACHERS | file:/home/cloudera/local_db/myfile/file.txt | 3    | 1     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| 110 AAAAAAAA  | file:/home/cloudera/local_db/myfile/file.txt | 3    | 2     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| 111 BBBBBBB  | file:/home/cloudera/local_db/myfile/file.txt | 3    | 3     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| 222 CCCCCCC  | file:/home/cloudera/local_db/myfile/file.txt | 3    | 4     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| 333 DDDDDD  | file:/home/cloudera/local_db/myfile/file.txt | 3    | 5     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| NEWFILE=CLASSES | file:/home/cloudera/local_db/myfile/file.txt | 4    | 1     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| 1 CLASS-1  | file:/home/cloudera/local_db/myfile/file.txt | 4    | 2     | 
+------------------+----------------------------------------------+-----------------+---------------------+ 
| 2 CLASS-2  | file:/home/cloudera/local_db/myfile/file.txt | 4    | 3     | 
+------------------+----------------------------------------------+-----------------+---------------------+