杜克重複數據刪除引擎：完全相同的記錄不匹配

Question

我正在嘗試使用杜克匹配記錄從一個CSV到另一個。第一個CSV和第二個都有ID,Model,Price,CompanyName,Review,Url列。我正在嘗試與另一個csv匹配來查找重複記錄。

package no.priv.garshol.duke; 

import no.priv.garshol.duke.matchers.PrintMatchListener; 

public class RunDuke { 

    public static void main(String[] argv) throws Exception { 
    Configuration config = 
     ConfigLoader 
      .load("/home/kishore/Duke-master/doc/example-data/presonalCare.xml"); 
    Processor proc = new Processor(config); 
    proc.addMatchListener(new PrintMatchListener(true, true, true, false, config.getProperties(), 
     true)); 
    proc.link(); 
    proc.close(); 
    } 

} 

這裏是personalCare.xml一個例子：

<!-- language: xml --> 
<!-- For more information, see https://github.com/larsga/Duke/wiki/ Improvements 
    needed: - some area numbers have spaces in them - not stripping accents from 
    names --> 
<duke> 
    <schema> 
     <threshold>0.7</threshold> 
     <property type="id"> 
      <name>ID</name> 
     </property> 
     <property> 
      <name>Model</name> 
      <comparator>no.priv.garshol.duke.comparators.Levenshtein</comparator> 
      <low>0.4</low> 
      <high>0.8</high> 
     </property> 
     <property> 
      <name>Price</name> 
      <comparator>no.priv.garshol.duke.comparators.ExactComparator</comparator> 
      <low>0.04</low> 
      <high>0.73</high> 
     </property> 
     <property> 
      <name>CompanyName</name> 
      <comparator>no.priv.garshol.duke.comparators.Levenshtein</comparator> 
      <low>0.4</low> 
      <high>0.8</high> 
     </property> 
     <property> 
      <name>Review</name> 
      <comparator>no.priv.garshol.duke.comparators.Levenshtein</comparator> 
      <low>0.12</low> 
      <high>0.93</high> 
     </property> 
     <property> 
      <name>Url</name> 
      <comparator>no.priv.garshol.duke.comparators.Levenshtein</comparator> 
      <low>0.12</low> 
      <high>0.93</high> 
     </property> 
    </schema> 

    <database class="no.priv.garshol.duke.databases.InMemoryDatabase"> 
    </database> 

    <group> 
     <csv> 
      <param name="input-file" value="personal_care_11.csv" /> 
      <param name="header-line" value="false" /> 
      <column name="1" property="ID" /> 
      <column name="2" property="Model" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/> 
      <column name="3" property="Price" /> 
      <column name="4" property="CompanyName" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/> 
      <column name="5" property="Review" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/> 
      <column name="6" property="Url" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/> 
     </csv> 
    </group> 

    <group> 
     <csv> 
      <param name="input-file" value="personal_care_11.csv" /> 
      <param name="header-line" value="false" /> 
      <column name="1" property="ID" /> 
      <column name="2" property="Model" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/> 
      <column name="3" property="Price" /> 
      <column name="4" property="CompanyName" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/> 
      <column name="5" property="Review" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/> 
      <column name="6" property="Url" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/> 
     </csv> 
    </group> 


</duke> 

上面的代碼工作正常，但它不匹配的確切記錄例如

STHDRNFKAQ4AFYE8,Littmann 3M Classic II S.E Acoustic 
Stethoscope,6297,Littmann,,http://dl.flipkart.com/dl/littmann-3m-classic-ii-s-e-acoustic-stethoscope/p/itme3uhzbqxhzfda?pid=STHDRNFKAQFAFYE8&affid=3ba0de4902524e2b90e43b84b89ea0ef 

這兩個CSV文件。我也想知道.xml文件中給出的低和高屬性值的工作，如何確定列值的低和高值。

Answer 1

您正在執行記錄鏈接（兩個數據集）而不是重複數據刪除（單個數據集），因此請取出.deduplicate（）調用。

此外，請不要使用'no.priv.garshol.duke'軟件包名稱。你不應該使用你自己不擁有的域名。

無論如何，你找不到任何匹配的原因是這兩個記錄具有相同的ID。 Duke證實它不報告記錄是否與自己匹配，所以匹配被過濾掉。如果您製作了csv文件的副本並將其用於組2，請更改ID，然後杜克發現重複。

這裏發生了什麼，當我嘗試：

[lars.garshol@laptop tmp]$ java -cp ~/cvs-co/duke/duke-core/target/duke-core-1.3-SNAPSHOT.jar:. no.priv.garshol.duke.Duke --showmatches presonalCare.xml 

MATCH 0.9982630751840313 
ID: 'SHDRNFKAQ4AFYE8', Model: 'littmann 3m classic ii s.e acoustic stethoscope', Price: '6297', CompanyName: 'littmann', Url: 'http://dl.flipkart.com/dl/littmann-3m-classic-ii-s-e-acoustic-stethoscope/p/itme3uhzbqxhzfda?pid=sthdrnfkaqfafye8&affid=3ba0de4902524e2b90e43b84b89ea0ef', 
ID: 'STHDRNFKAQ4AFYE8', Model: 'littmann 3m classic ii s.e acoustic stethoscope', Price: '6297', CompanyName: 'littmann', Url: 'http://dl.flipkart.com/dl/littmann-3m-classic-ii-s-e-acoustic-stethoscope/p/itme3uhzbqxhzfda?pid=sthdrnfkaqfafye8&affid=3ba0de4902524e2b90e43b84b89ea0ef',