如何在兩個CSV文件中找到相似的行？

Question

這裏是我的代碼，但它永遠花費巨大的文件：

require 'rubygems' 
require "faster_csv" 

fname1 =ARGV[0] 
fname2 =ARGV[1] 
if ARGV.size!=2 
    puts "Display common lines in the two files \n Usage : ruby user_in_both_files.rb <file1> <file2> " 
    exit 0 
end 

puts "loading the CSV files ..." 
file1=FasterCSV.read(fname1, :headers => :first_row) 
file2=FasterCSV.read(fname2, :headers => :first_row) 
puts "CSV files loaded" 

#puts file2[219808].to_s.strip.gsub(/\s+/,'') 

lineN1=0 
lineN2=0 
# count how many common lines 
similarLines=0 
file1.each do |line1| 
    lineN1=lineN1+1 
    #compare line 1 to all line from file 2 
    lineN2=0 
    file2.each do |line2| 
     puts "file1:l#{lineN1}|file2:l#{lineN2}" 
     lineN2=lineN2+1 
     if (line1.to_s.strip.gsub(/\s+/,'') == line2.to_s.strip.gsub(/\s+/,'')) 
      puts "file1:l#{line1}|file2:l#{line2}->#{line1}\n" 
      similarLines=similarLines+1 
     end 
    end 
end 
puts "#{similarLines} similar lines." 

Answer 1

紅寶石已設置的可用操作使用數組：

a_ary = [1,2,3] 
b_ary = [3,4,5] 
a_ary & b_ary # => 3 

所以，從你應該嘗試：

puts "loading the CSV files ..." 
file1 = FasterCSV.read(fname1, :headers => :first_row) 
file2 = FasterCSV.read(fname2, :headers => :first_row) 
puts "CSV files loaded" 

common_lines = file1 & file2 
puts common_lines.size 

如果需要預處理的陣列，你加載它們做到這一點：

file1 = FasterCSV.read(fname1, :headers => :first_row).map{ |l| l.to_s.strip.gsub(/\s+/, '') } 
file2 = FasterCSV.read(fname2, :headers => :first_row).map{ |l| l.to_s.strip.gsub(/\s+/, '') } 

Answer 2

你gsub每次遍歷文件1荷蘭國際集團文件2。我會先做，然後比較結果。

編輯像這樣（未經）

file1lines = [] 
file1.each do |line1| 
    file1lines = line1.strip.gsub(/\s+/, '') 
end 

# Do the same for `file2lines` 

file1lines.each do |line1|  
    lineN1=lineN1+1  
    #compare line 1 to all line from file 2  
    lineN2=0  
    file2lines.each do |line2|  
    puts "file1:l#{lineN1}|file2:l#{lineN2}"  
    lineN2=lineN2+1  
    if (line1 == line2)  
     puts "file1:l#{line1}|file2:l#{line2}->#{line1}\n"  
     similarLines=similarLines+1  
    end  
    end  
end  

我也得到了循環擺脫所有的puts ES的，除非你真的需要它們。如果這些文件很大，那可能會使其顯着減慢。