org.apache.solr.common.SolrException：TransactionLog不知道如何序列化org.bson.types.ObjectId類;嘗試實施ObjectResolver？

Question

當執行從MongoDB的數據導入，Solr中引發以下錯誤：

org.apache.solr.common.SolrException: TransactionLog doesn't know how to serialize class org.bson.types.ObjectId; try implementing ObjectResolver? 
at org.apache.solr.update.TransactionLog$1.resolve(TransactionLog.java:100) 
at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:234) 
at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:589) 
at org.apache.solr.update.TransactionLog.write(TransactionLog.java:395) 
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532) 
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:516) 
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:320) 
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239) 
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:194) 
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67) 
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) 
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:979) 
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1192) 
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748) 
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103) 
at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:80) 
at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:254) 
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:526) 
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414) 
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329) 
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232) 
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:415) 
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:474) 
at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:457) 
at java.lang.Thread.run(Thread.java:748) 

我的Solr的版本是6.6.0。錯誤的原因是什麼？如何解決？

Answer 1

根據錯誤信息，

您需要實現JavaBinCodec.ObjectResolver爲org.bson.types.ObjectId類型，所以Solr的會知道如何序列化該類的實例。

JavaBinCodec.ObjectResolver Documentation

public static interface JavaBinCodec.ObjectResolver Allows extension of JavaBinCodec to support serialization of arbitrary data types. Implementors of this interface write a method to serialize a given object using an existing JavaBinCodec

一旦你寫你的JavaBinCodec.ObjectResolver實現你應該使用JavaBinCodec

JavaBinCodec Documentation

public class JavaBinCodec extends Object Defines a space-efficient serialization/deserialization format for transferring data. JavaBinCodec has built in support many commonly used types. This includes primitive types (boolean, byte, short, double, int, long, float), common Java containers/utilities (Date, Map, Collection, Iterator, String, Object[], byte[]), and frequently used Solr types (NamedList, SolrDocument, SolrDocumentList). Each of the above types has a pair of associated methods which read and write that type to a stream.

Classes that aren't supported natively can still be serialized/deserialized by providing an JavaBinCodec.ObjectResolver object that knows how to work with the unsupported class. This allows JavaBinCodec to be used to marshall/unmarshall arbitrary content.

NOTE -- JavaBinCodec instances cannot be reused for more than one marshall or unmarshall operation.

Answer 2

註冊它，我碰到這個問題，而試圖祁門功夫t來自mongoDB中多個集合的數據。

假設您未使用mongo連接器，我使用以下導入數據。

• Solr的-6.6.0
• Solr的-dataimporthandler-6.6.0
• mongo-java-driver-3.5.0
• Solr Mongo importer

由於返回 '_id' 是類型的ObjectId的，我的工作圍繞解決方案在將索引到solr之前將'_id'轉換爲字符串，並在查詢'_id'的同時，在運行查詢之前將其轉換爲ObjectId類型。

下載solr mongo導入程序並進行以下更改。

MongoMapperTransformer.java

public class MongoMapperTransformer extends Transformer { 

@Override 
public Object transformRow(Map<String, Object> row, Context context) { 

    for (Map<String, String> map : context.getAllEntityFields()) { 
     String mongoFieldName = map.get(MONGO_FIELD); 
     String mongoId = map.get(MONGO_ID); 
     if (mongoFieldName == null) 
      continue; 

     String columnFieldName = map.get(DataImporter.COLUMN); 

     //If the field is ObjectId convert it into String 
     if (mongoId != null && Boolean.parseBoolean(mongoId)) { 
      Object srcId = row.get(columnFieldName); 
      row.put(columnFieldName, srcId.toString()); 
     } 
     else{ 
      row.put(columnFieldName, row.get(mongoFieldName)); 
     } 
    } 

    return row; 
} 


public static final String MONGO_FIELD = "mongoField"; 

//To identify the _id field 
public static final String MONGO_ID = "objectIdToString"; 

} 

下，更換功能

public Iterator <Map<String, Object>> getData(String query){...} 

在MongoDataSource.java下列要求：

@Override 
public Iterator<Map<String, Object>> getData(String query) { 

    DBObject queryObject = new BasicDBObject(); 

    /* If querying by _id, since the id is a string now, 
    * it has to be converted back to type ObjectId() using the 
    * constructor 
    */ 
    if(query.contains("_id")){ 
     @SuppressWarnings("unchecked") 
     Map<String, String> queryWithId = (Map<String, String>) JSON.parse(query); 
     String id = queryWithId.get("_id"); 
     queryObject = new BasicDBObject("_id", new ObjectId(id)); 
    } 
    else{ 
     queryObject = (DBObject) JSON.parse(query); 
    } 

    LOG.debug("Executing MongoQuery: " + query.toString()); 

    long start = System.currentTimeMillis(); 
    mongoCursor = this.mongoCollection.find(queryObject); 
    LOG.trace("Time taken for mongo :" 
      + (System.currentTimeMillis() - start)); 

    ResultSetIterator resultSet = new ResultSetIterator(mongoCursor); 
    return resultSet.getIterator(); 
} 

這些更改後，您可以使用構建JAR螞蟻。

將jars（solo mongo importer和mongo-java-driver）複製到lib目錄中。我將它們複製到$ solr-install-dir/contrib/dataimport-handler/lib中。在solr-config中添加lib指令。XML對於上述罐子：

<lib dir="${solr.install.dir:../../../..}/contrib/dataimporthandler/lib" regex=".*\.jar" /> 

最後，這裏是蒙戈集合和數據-config.xml中

User collection 
{ 
    "_id" : ObjectId("56e9c892e4b0355017b2fa0f"), 
    "name" : "User1", 
    "phone" : "123456789" 
} 

Address collection 
{ 
    "_id" : ObjectId("56e9c892e4b0355017b2fa0f"), 
    "address" : "#666, Maiden street" 
} 

數據-config.xml中的一個例子

不要忘記提及objectIdToString = _id字段的「true」，以便MongoMapperTransformer可以對id進行字符串化。

<dataConfig> 
    <dataSource name="MyMongo" 
      type="MongoDataSource" 
      database="test" 
      /> 
    <document name="UserDetails"> 
    <!-- if query="" then it imports everything --> 
     <entity name="users" 
      processor="MongoEntityProcessor" 
      query="" 
      collection="user" 
      datasource="MyMongo" 
      transformer="MongoMapperTransformer"> 
       <field column="_id" name="id" mongoField="_id" objectIdToString="true" /> 
       <field column="phone" name="phone" mongoField="phone"/> 

      <entity name="address" 
       processor="MongoEntityProcessor" 
       query="{_id:'${users._id}'}" 
       collection="address" 
       datasource="MyMongo" 
       transformer="MongoMapperTransformer"> 
       <field column="address" name="adress" mongoField="address"/> 
      </entity> 
    </entity> 
    </document> 
</dataConfig> 

託管架構將使用id字段作爲字符串。 另外，如果你在mongodb中有嵌套的對象，你將不得不使用腳本變換器來索引它們。

希望這會有所幫助， 祝你好運！