2012-09-05 45 views
1

據我所知,Kryo在每創建一個className < - >numberID地圖。這張地圖太狹窄了。因爲在你的對象模型中,實例往往屬於同一個類,下一個writeObject將會再次構建並序列化一個類似的地圖(並且再次,再次,再次)。我知道可以通過手動註冊這些類來共享地圖,但這是一種繁瑣的手動硬編碼。我希望地圖可以通過第一次寫入對象來啓動,就像通常那樣,但會話中的所有後續寫入操作都會重新使用並擴展它。這樣,註冊將在運行時自動發生,無需額外的運行時間開銷,而更頻繁使用的對象自然會收到低ID號。之後,地圖可以作爲解密密鑰分別存儲在附件中。反序列化器將首先加載這張地圖。你如何喜歡這個想法,以及如何實施?Kryo自動班級註冊

我的問題與此類似 Strategy for registering classes with kryo其中用戶可以使用List將單個writeObject下的所有寫入組合在一起。正如我所建議的那樣,這將比單獨存儲地圖更簡單。但是,他似乎不想這樣做。在我的情況下,這種組合是不可能的,因爲大的java模型,我避免通過分片序列化將它完全保存在內存中。在我的場景中,用戶打開一個項目,進行更改並刷新它們。所以,該項目可以維護一個類的映射並將其用於所有序列化。

更新!我意識到有類/對象註冊和autoReset。他們似乎被創造出來適合這項任務。但是,我不明白這些東西如何解決它。 Autoreset=false確實使第二次寫入確實小得多。但是,在這種情況下,我無法反序列化對象。正如你例子看,第二反序列化失敗:

public class A { 
    String f1; 
    A(String a) { 
     f1 = a; 
    } 
    List list = new ArrayList(); 
    public String toString() { 
     return "f1 = " + f1 + ":" + f1.getClass().getSimpleName(); 
    } 

    public static void main(String[] args) { 
     test(true); 
     test(false); 
    } 


    static void write(String time, Kryo kryo, ByteArrayOutputStream baos, Object o) { 
     Output output = new Output(baos); 
     kryo.writeClassAndObject(output, o); 
     output.close(); 
     System.err.println(baos.size() + " after " + time + " write"); 
    } 

    private static void test(boolean autoReset) { 
     Kryo kryo = new Kryo(); 
     kryo.setAutoReset(autoReset); 
     kryo.setInstantiatorStrategy(new StdInstantiatorStrategy()); 
     System.err.println("-------\ntesting autoreset = " + autoReset); 
     ByteArrayOutputStream baos = new ByteArrayOutputStream(); 
     A a = new A("a"), b = new A("b"); 
     write("first", kryo, baos, a); 
     write("second", kryo, baos, b); 
     A o1 = restore("first", baos, kryo); 
     A o2 = restore("second", baos, kryo); // this fails 
     System.err.println((o1.f1.equals(o2.f1)) ? "SUCCESS" : "FAILURE"); 

    } 

    private static A restore(String time, ByteArrayOutputStream baos, Kryo k) { 
     ByteArrayInputStream in = new ByteArrayInputStream(baos.toByteArray()); 
     Input input = new Input(in); 
     A o = (A) k.readClassAndObject(input); 
     System.err.println("reading object " + time + " time, got " + o); 
     return o; 
    } 

輸出是

------- 
testing autoreset = true 
41 after first write 
82 after second write 
reading object first time, got f1 = a:String 
reading object second time, got f1 = a:String 
SUCCESS 
------- 
testing autoreset = false 
41 after first write 
52 after second write 
reading object first time, got f1 = a:String 
reading object second time, got null 
Exception in thread "main" java.lang.NullPointerException 
    at kryo_test.AutoresetDemo.test(AutoresetDemo.java:40) 
    at kryo_test.AutoresetDemo.main(AutoresetDemo.java:18) 

UPDATE2它也可能發生自動復位=虛假記載對象除了類名的引用。確實值得自動復位。

Update3我發現很難序列化類的映射(即類 - >註冊),因爲註冊包含引用kryo對象並保持某種狀態的序列化程序。然後,很難在很多kryo對象之間共享地圖。

回答

2

好吧,這裏是KRYO 2.20

public class GlobalClassKryo extends Kryo { 

    public static class ExternalizableClassResolver implements ClassResolver { 

     //local serializers 
     final Map<Class, Registration> fromClass = new HashMap(); 
     final Map<Integer, Registration> fromId = new HashMap(); 

     public static class GlobalRegistration { int id; Class type; Class<? extends Serializer> serializer; } 

     public final Map<Integer, GlobalRegistration> globalIds; 
     public final Map<Class, GlobalRegistration> globalClasses; 

     // I synchronize because I have one reader and one writer thread and 
     // writer may break the reader when adds something into the map. 
     public ExternalizableClassResolver() {this (
       Collections.synchronizedMap(new HashMap()), 
       Collections.synchronizedMap(new HashMap()) 
      ) ;} 

     public ExternalizableClassResolver(Map<Integer, GlobalRegistration> ids, Map<Class, GlobalRegistration> classes) { 
      globalIds = ids; 
      globalClasses = classes; 
     } 

     public ExternalizableClassResolver (DataInput in) throws ClassNotFoundException, IOException { 
      this(); 
      int id; 
      while ((id = in.readInt()) != -1) { 
       GlobalRegistration e = new GlobalRegistration(); 
       globalIds.put(e.id = id, e); 
       e.type = Class.forName(in.readUTF()); 
       e.serializer = (Class<? extends Serializer>) Class.forName(in.readUTF()); 
       globalClasses.put(e.type, e); 
      } 
     } 

     public void save(DataOutput out) throws IOException { 
      for (GlobalRegistration entry : globalIds.values()) { 
        out.writeInt(entry.id); 
        out.writeUTF(entry.type.getName()); 
        out.writeUTF(entry.serializer.getName()); 
      } 
      out.writeInt(-1); 
     } 

     static final boolean TRACE = false; 
     void log(String msg) { 
      System.err.println(kryo != null ? Utils.fill(kryo.getDepth(), ' ') + msg : msg); 
     } 
     @Override 
     public Registration writeClass(Output output, Class type) { 
      if (type == null) {output.writeInt(0, true); return null;} 
      Registration registration = kryo.getRegistration(type); 
      output.writeInt(registration.getId(), true); 
      return registration; 
     } 
     @Override 
     public Registration readClass(Input input) { 
      int classID = input.readInt(true); 
      if (classID == 0) return null; 
      Registration registration = fromId.get(classID); 
      if (registration == null) { 
       registration = tryGetFromGlobal(globalIds.get(classID), classID + ""); 
      } 
      if (registration == null) throw new KryoException("Encountered unregistered class ID: " + classID); 
      return registration; 
     } 

     public Registration register(Registration registration) { 
      throw new KryoException("register(registration) is not allowed. Use register(type, serializer)"); 
     } 

     public Registration getRegistration(int classID) { 
      throw new KryoException("getRegistration(id) is not implemented"); 
     } 

     Registration tryGetFromGlobal(GlobalRegistration globalClass, String title) { 
      if (globalClass != null) { 
       Serializer serializer = kryo.newSerializer(globalClass.serializer, globalClass.type); 
       Registration registration = register(globalClass.type, serializer, globalClass.id, "local"); 
       if (TRACE) log("getRegistration(" + title + ") taken from global => " + registration); 
       return registration; 
      } else 
       if (TRACE) log("getRegistration(" + title + ") was not found"); 
      return null; 
     } 
     public Registration getRegistration(Class type) { 
      Registration registration = fromClass.get(type); 
      if (registration == null) { 
       registration = tryGetFromGlobal(globalClasses.get(type), type.getSimpleName()); 
      } else 
       if (TRACE) log("getRegistration(" + type.getSimpleName() + ") => " + registration); 

      return registration; 
     } 

     Registration register(Class type, Serializer serializer, int id, String title) { 
      Registration registration = new Registration(type, serializer, id); 
      fromClass.put(type, registration); 
      fromId.put(id, registration); 

      if (TRACE) log("new " + title + " registration, " + registration); 

      //why dont' we put into fromId? 
      if (registration.getType().isPrimitive()) fromClass.put(getWrapperClass(registration.getType()), registration); 
      return registration; 
     } 

     int primitiveCounter = 1; // 0 is reserved for NULL 
     static final int PRIMITIVE_MAX = 20; 

     //here we register anything that is missing in the global map. 
     // It must not be the case that something available is registered for the second time, particularly because we do not check this here 
     // and use registered map size as identity counter. Normally, check is done prior to callig this method, in getRegistration 
     public Registration register(Class type, Serializer serializer) { 

      if (type.isPrimitive() || type.equals(String.class)) 
       return register(type, serializer, primitiveCounter++, "primitive"); 

      GlobalRegistration global = globalClasses.get(type); 

      if (global != null) 
        throw new RuntimeException("register(type,serializer): we have " + type + " in the global map, this method must not be called"); 

      global = new GlobalRegistration(); 
      globalIds.put(global.id = globalClasses.size() + PRIMITIVE_MAX, global); 
      globalClasses.put(global.type = type, global); 
      global.serializer= serializer.getClass(); 

      return register(global.type, serializer, global.id, "global"); 
     } 

     public Registration registerImplicit(Class type) { 
      throw new RuntimeException("registerImplicit is not needed since we register missing automanically in getRegistration"); 
     } 

     @Override 
     public void reset() { 
      // super.reset(); //no need to reset the classes 
     } 

     Kryo kryo; 
     public void setKryo(Kryo kryo) { 
      this.kryo = kryo; 
     } 
    } 




    public ExternalizableClassResolver ourClassResolver() { 
     return (ExternalizableClassResolver) classResolver; 
    } 

    public GlobalClassKryo(ClassResolver resolver) { 
     super(resolver, new MapReferenceResolver()); 
     setInstantiatorStrategy(new StdInstantiatorStrategy()); 
     this.setRegistrationRequired(true); 
    } 
    public GlobalClassKryo() { 
     this(new ExternalizableClassResolver()); 
    } 

    @Override 
    public Registration getRegistration (Class type) { 
     if (type == null) throw new IllegalArgumentException("type cannot be null."); 

     if (type == memoizedClass) return memoizedClassValue; 
     Registration registration = classResolver.getRegistration(type); 
     if (registration == null) { 
      if (Proxy.isProxyClass(type)) { 
       // If a Proxy class, treat it like an InvocationHandler because the concrete class for a proxy is generated. 
       registration = getRegistration(InvocationHandler.class); 
      } else if (!type.isEnum() && Enum.class.isAssignableFrom(type)) { 
       // This handles an enum value that is an inner class. Eg: enum A {b{}}; 
       registration = getRegistration(type.getEnclosingClass()); 
      } else if (EnumSet.class.isAssignableFrom(type)) { 
       registration = classResolver.getRegistration(EnumSet.class); 
      } 
      if (registration == null) { 
       //registration = classResolver.registerImplicit(type); 
       return register(type, getDefaultSerializer(type)); 
      } 
     } 
     memoizedClass = type; 
     memoizedClassValue = registration; 
     return registration; 
    } 

    public Registration register(Class type, Serializer serializer) { 
     return ourClassResolver().register(type, serializer);} 

    public Registration register(Registration registration) { 
     throw new RuntimeException("only register(Class, Serializer) is allowed");} 

    public Registration register(Class type) { 
     throw new RuntimeException("only register(Class, Serializer) is allowed");} 

    public Registration register(Class type, int id) { 
     throw new RuntimeException("only register(Class, Serializer) is allowed");} 

    public Registration register(Class type, Serializer serializer, int id) { 
     throw new RuntimeException("only register(Class, Serializer) is allowed"); 
    } 

    static void write(String title, Kryo k, ByteArrayOutputStream baos, Object obj) { 
     Output output = new Output(baos); 
     k.writeClassAndObject(output, obj); 
     output.close(); 
     System.err.println(baos.size() + " bytes after " + title + " write"); 
    } 
    static class A { 
     String field = "abcABC"; 
     A a = this; 
     //int b = 1; // adds 1 byte to serialization 
     @Override 
     public String toString() { 
      return field 
        + " " + list.size() 
        //+ ", " + b 
        ; 
     } 

     // list adds 3 bytes to serialization, two 3-byte string items add additionally 10 bytes in total 
     ArrayList list = new ArrayList(100); // capacity is trimmed in serialization 
     { 
      list.add("LLL"); 
      list.add("TTT"); 

     } 
    } 

    private static void test() throws IOException, ClassNotFoundException { 
     GlobalClassKryo k = new GlobalClassKryo(); 

     ByteArrayOutputStream baos = new ByteArrayOutputStream(); 

     write("first", k, baos, new A()); // write takes 24 byts 

     //externalize the map 
     ByteArrayOutputStream mapOut = new ByteArrayOutputStream(); 
     DataOutputStream dataOut = new DataOutputStream(mapOut); 
     k.ourClassResolver().save(dataOut); 
     dataOut.close(); 

     //deserizalize the map 
     DataInputStream serialized = new DataInputStream(new ByteArrayInputStream(mapOut.toByteArray())); 
     ExternalizableClassResolver resolver2 = new ExternalizableClassResolver(serialized); 

     //use the map 
     k = new GlobalClassKryo(resolver2); 
     write("second", k, baos, new A()); // 24 bytes 

     Input input = new Input(new ByteArrayInputStream(baos.toByteArray())); 
     Object read = k.readClassAndObject(input); 
     System.err.println("output " + read); 
    } 

    public static void main(String[] args) throws IOException, ClassNotFoundException { 
     Kryo k = new Kryo(); 
     ByteArrayOutputStream baos = new ByteArrayOutputStream(); 
     write("first", k, baos, new A()); // write takes 78 bytes 
     write("second", k, baos, new A()); // +78 bytes 
     System.err.println("----------------"); 

     test(); 
    } 
} 

溶液中生成的物流是清潔類名。不幸的是,與默認的Java序列化(2x或更多)相比,Kryo的速度太慢了,儘管它產生的流更加密集。 Kryo獨自使我的樣本序列化幾乎縮小了10倍。你會發現在這個答案中提出的解決方案增加了3倍的額外因素。但是,在序列化兆字節的字段中,與Java Kryo一起存儲到磁盤時,對於java序列化和2倍減速,我只獲得2倍壓縮。

+2

很高興Kryo靈活地做你想做的事。雖然速度聽起來不對,你有沒有看到放緩的地方? – NateS