I have a quite complex Java object to serialize (which worked fine a couple of weeks ago). After having implemented a lot in the meantime, the serialization now fails throwing the following exception:
java.io.NotSerializableException: sun.java2d.SunGraphics2D
I have inspected all classes which seemed relevant, but could not find a field in any of them holding a Graphics2D type (or a similar type such as BufferedImage or the like). As I see it, the problem could also be that I'm using a class (from a library or so) which itself is serializable, but has a field of the unserializable type Graphics2D.
So, my question is: is there a "good" way to find the place where the serialization fails? Inspecting all classes and all changes made seems to be hardly feasible.
Thank you in advance
Lukas
PS. Here is the full stack trace of the exception:
java.io.NotSerializableException: sun.java2d.SunGraphics2D
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1707)
at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:482)
at java.awt.Container.writeObject(Container.java:3697)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1707)
at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:482)
at java.awt.Container.writeObject(Container.java:3697)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1707)
at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:482)
at java.awt.Container.writeObject(Container.java:3697)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1707)
at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:482)
at java.awt.Container.writeObject(Container.java:3697)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.awt.Window.writeObject(Window.java:2943)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.awt.AWTEventMulticaster.save(AWTEventMulticaster.java:946)
at java.awt.Window.writeObject(Window.java:2931)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.util.LinkedList.writeObject(LinkedList.java:1131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at eas.simulation.SerializableSimulationState.save(SerializableSimulationState.java:121)
at eas.plugins.standard.visualization.ControlPanel.actionPerformed(ControlPanel.java:202)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2346)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
at java.awt.Component.processMouseEvent(Component.java:6527)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3321)
at java.awt.Component.processEvent(Component.java:6292)
at java.awt.Container.processEvent(Container.java:2234)
at java.awt.Component.dispatchEventImpl(Component.java:4883)
at java.awt.Container.dispatchEventImpl(Container.java:2292)
at java.awt.Component.dispatchEvent(Component.java:4705)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4898)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4533)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4462)
at java.awt.Container.dispatchEventImpl(Container.java:2278)
at java.awt.Component.dispatchEvent(Component.java:4705)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:746)
at java.awt.EventQueue.access$400(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:697)
at java.awt.EventQueue$3.run(EventQueue.java:691)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:75)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:86)
at java.awt.EventQueue$4.run(EventQueue.java:719)
at java.awt.EventQueue$4.run(EventQueue.java:717)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:75)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:716)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Okay, inspired by the idea of Ben Lawry who, however, did not present an actual implementation, I have also programmed a class crawler which finds "unserializable fields", i.e., non-static, non-transient fields that are either not implementing the Serializable interface themselves or contain fields that are non-static, non-transient and do not implement the Serializable interface.
You can add several classes to be tested in the list in the main method (see comment). The output is a list of fields from these classes or any referenced classes that are not serializable by the above definition (which, however, is not exactly equal to the actually serializable fields, but captures a superset of the latter); for each field which itself does implement the Serializable interface, a list of fields referenced in the type of that field is given which cause the field being not serializable (if the list is empty the field itself does not implement the Serializable interface).
Here is the code, hope it helps others, too:
import java.io.Serializable;
import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.LinkedList;
import javax.swing.JComponent;
public class Crawler {
public static boolean crawlRecursively(Field field, HashSet<Class<?>> alreadyCrawled, HashMap<Field, HashSet<String>> badFields) {
if (alreadyCrawled.contains(field.getType())) {
return !badFields.keySet().contains(field);
}
alreadyCrawled.add(field.getType());
if (Modifier.isStatic(field.getModifiers())
|| Modifier.isTransient(field.getModifiers())
|| field.getType().isPrimitive()) {
return true;
} else if (Serializable.class.isAssignableFrom(field.getType())) {
boolean allGood = true;
for (Field f : field.getType().getDeclaredFields()) {
boolean isGood = crawlRecursively(f, alreadyCrawled, badFields);
if (!isGood) {
if (!badFields.containsKey(field)) {
badFields.put(field, new HashSet<>());
}
badFields.get(field).add(f.getType().getSimpleName() + " " + f.getName());
allGood = false;
}
}
return allGood;
} else {
if (!badFields.containsKey(field)) {
badFields.put(field, new HashSet<>());
}
return false;
}
}
public static HashMap<Field, HashSet<String>> initiateCrawling(Collection<Class<?>> roots) {
HashMap<Field, HashSet<String>> badFields = new HashMap<>();
for (Class<?> root : roots) {
for (Field f : root.getDeclaredFields()) {
crawlRecursively(f, new HashSet<>(), badFields);
}
}
return badFields;
}
public static void main(String[] args) {
LinkedList<Class<?>> roots = new LinkedList<>();
roots.add(JComponent.class); // ADD YOUR CLASSES HERE.
HashMap<Field, HashSet<String>> badFields = initiateCrawling(roots);
if (badFields.keySet().size() == 0) {
System.out.println("All fields are serializable (not having checked the given class(es) themselves).");
} else {
System.out.println("The following fields are not serializable in the class tree(s) given by " + roots + ":");
}
for (Field field : badFields.keySet()) {
System.out.println("<UnSer> "
+ field.getType().getSimpleName() + " "
+ field.getName() + " ("
+ field.getDeclaringClass().getName() + ") => " + badFields.get(field));
}
}
}
Example output for class JComponent:
The following fields are not serializable in the class tree(s) given by [class javax.swing.JComponent]:
<UnSer> Border border (javax.swing.JComponent) => []
<UnSer> ComponentInputMap windowInputMap (javax.swing.JComponent) => [JComponent component]
<UnSer> VetoableChangeSupport vetoableChangeSupport (javax.swing.JComponent) => [VetoableChangeListenerMap map, Object source]
<UnSer> SingleSelectionModel selectionModel (javax.swing.JPopupMenu) => []
<UnSer> JPopupMenu popupMenu (javax.swing.JComponent) => [SingleSelectionModel selectionModel]
<UnSer> Object source (java.beans.VetoableChangeSupport) => []
<UnSer> VetoableChangeListenerMap map (java.beans.VetoableChangeSupport) => []
<UnSer> JComponent component (javax.swing.ComponentInputMap) => [VetoableChangeSupport vetoableChangeSupport, JPopupMenu popupMenu, Border border, InputVerifier inputVerifier]
<UnSer> InputVerifier inputVerifier (javax.swing.JComponent) => []
The library class can be assumed to be correctly serializable, i.e. to declare all non-serializable fields as transient, and therefore shouldn't be the source of the error.
If your code is using ContextedRuntimeExceptions, they are an exception to the above rule because the objects they reference may not be serializable, and this is noted in the Javadoc at the methods addContextValue() and setContextValue(). If context values are the problem, it would explain why the suspect classes got a reference to a Graphics2D object without having such a field.
I have had limited success replicating NotSerializableException for SunGraphics2D without leaving an obvious field with "Graphics" in its name laying around, which could be due to testing on a simpler class structure. I found that in addition to contexted exceptions, collections were a good suspect but only if they used raw types. Collections could also cause intermittent failures if they were only sometimes populated with Graphics.
Depending on your application, Ben Lawry's automated solution to a similar issue may be easier to implement than manually hunting for the above. If that path works, it will likely be more thorough as well. Can you get a stack trace?
Related
I'm trying to send objects using Java RMI in a JavaFX 2 project, but when the following code runs it returns a NotSerializableException.
My Admin class is Serializable and so is the super class. However it seems the exception is pointing towards the JavaFX SimpleIntegerProperty fields inside the Admin class.
I don't know what to do from here as the class being sent via RMI is serializable. Any help much appreciated.
ObservableList<Admin> data = null;
try
{
data = FXCollections.observableArrayList(Main.docServer.getAllAdmins());
}
catch (RemoteException e)
{
e.printStackTrace();
}
The error I receive:
java.rmi.UnmarshalException: error unmarshalling return; nested exception is:
java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: javafx.beans.property.SimpleIntegerProperty
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:191)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)
at $Proxy0.getAllAdmins(Unknown Source)
at com.reece.doc.views.ViewAdmin.getContent(ViewAdmin.java:34)
at com.reece.doc.ApplicationWindow.start(ApplicationWindow.java:32)
at com.reece.doc.Main.start(Main.java:57)
at com.sun.javafx.application.LauncherImpl$5.run(LauncherImpl.java:319)
at com.sun.javafx.application.PlatformImpl$5.run(PlatformImpl.java:206)
at com.sun.javafx.application.PlatformImpl$4.run(PlatformImpl.java:173)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:76)
Caused by: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException: javafx.beans.property.SimpleIntegerProperty
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at java.util.ArrayList.readObject(ArrayList.java:733)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:324)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:173)
... 10 more
Caused by: java.io.NotSerializableException: javafx.beans.property.SimpleIntegerProperty
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1180)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
at java.util.ArrayList.writeObject(ArrayList.java:710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:975)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1480)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:292)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:332)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
the class being sent via RMI is serializable
It's only serializable if it extends Serializable or Externalizable and all of its non-static non-transient member variables do so as well, and so on recursively until closure. In this case you clearly have one of type javafx.beans.property.SimpleIntegerProperty, which as the exception tells you isn't serializable:
java.io.NotSerializableException: javafx.beans.property.SimpleIntegerProperty
So any class that refers to it, directly or indirectly, isn't serializable either.
You may mark a member variable not to be serialized. And add to class new int field.
private int variableInt = 0;
private transient IntegerProperty variable;
public IntegerProperty variableProperty()
{
if(variable == null)
{
variable = new SimpleIntegerProperty();
variable.set(variableInt);
}
return variable;
}
public void setVariable(int variable)
{
if(this.variable != null)
this.variable.set(variable);
variableInt = variable;
}
public int getVariable()
{
if(variable == null)
return variableInt;
else
return variable.get();
}
i'm working with jcs with more clients of one single remote cache server.
Here the server configuration:
registry.host=xx.xx.xx.xx
registry.port=1104
remote.cache.service.port=1104
remote.cache.rmiSocketFactoryTimeoutMillis=5000
remote.cluster.LocalClusterConsistency=true
remote.cluster.AllowClusterGet=true
jcs.default=DC
jcs.default.cacheattributes=org.apache.commons.jcs.engine.CompositeCacheAttributes
jcs.default.cacheattributes.MaxObjects=100
jcs.default.cacheattributes.MaxMemoryIdleTimeSeconds=1800
jcs.default.cacheattributes.UseMemoryShrinker=true
jcs.default.elementattributes.IsEternal=false
jcs.default.elementattributes=org.apache.commons.jcs.engine.ElementAttributes
jcs.auxiliary.DC=org.apache.commons.jcs.auxiliary.disk.indexed.IndexedDiskCacheFactory jcs.auxiliary.DC.attributes=org.apache.commons.jcs.auxiliary.disk.indexed.IndexedDiskCacheAttributes
jcs.auxiliary.DC.attributes.DiskPath=/home/java/conf/data
jcs.auxiliary.DC.attributes.MaxPurgatorySize=1000
jcs.auxiliary.DC.attributes.MaxKeySize=1000
jcs.auxiliary.DC.attributes.OptimizeAtRemoveCount=300
jcs.auxiliary.DC.attributes.OptimizeOnShutdown=true
jcs.auxiliary.DC.attributes.ClearDiskOnStartup=true
jcs.auxiliary.DC.attributes.DiskLimitType=SIZE
And the client configuration:
jcs.auxiliary.RC=org.apache.commons.jcs.auxiliary.remote.RemoteCacheFactory
jcs.auxiliary.RC.attributes=org.apache.commons.jcs.auxiliary.remote.RemoteCacheAttributes
jcs.auxiliary.RC.attributes.FailoverServers=xx.xx.xx.xx:1104
jcs.auxiliary.RC.attributes.RemoveUponRemotePut=false
jcs.auxiliary.RC.attributes.Receive=false
jcs.auxiliary.RC.elementattributes.UseMemoryShrinker=true
Here the client code put:
public class JcsSharedCache<K, V> implements ICache<K, V> {
private CompositeCache<K, V> internal;
public void put(K key, V value) throws BeCacheException {
try {
synchronized(this){
internal.update(new CacheElement<>(internal.getCacheName(), key, value, internal.getElementAttributes()));
}
} catch (Exception ex) {
throw new BeCacheException(ex);
}
}
}
Sometimes a client get a ConcurrentModificationException and jcs disable the remote put. Every client had a synchronyzed version of put, but this happens(in my opinion) when two client try to put something in the same cache concurrently.
Is there a manner to avoid this behavior?
Below the stack trace:
2018-07-10 18:02:52 ERROR RemoteCache:141 - Disabling remote cache due
to error: Failed to put [da4a81e6-85df-426a-b6f6-a5b6777b0891] to
nodes java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
at java.util.HashMap$KeyIterator.next(HashMap.java:1466)
at java.util.HashSet.writeObject(HashSet.java:287)
at sun.reflect.GeneratedMethodAccessor135.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.util.HashMap.internalWriteEntries(HashMap.java:1790)
at java.util.HashMap.writeObject(HashMap.java:1363)
at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.util.ArrayList.writeObject(ArrayList.java:766)
at sun.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.commons.jcs.utils.serialization.StandardSerializer.serialize(StandardSerializer.java:55)
at org.apache.commons.jcs.utils.serialization.SerializationConversionUtil.getSerializedCacheElement(SerializationConversionUtil.java:74)
at org.apache.commons.jcs.auxiliary.remote.AbstractRemoteAuxiliaryCache.processUpdate(AbstractRemoteAuxiliaryCache.java:421)
at org.apache.commons.jcs.auxiliary.AbstractAuxiliaryCacheEventLogging.updateWithEventLogging(AbstractAuxiliaryCacheEventLogging.java:65)
at org.apache.commons.jcs.auxiliary.AbstractAuxiliaryCacheEventLogging.update(AbstractAuxiliaryCacheEventLogging.java:50)
at org.apache.commons.jcs.engine.CacheAdaptor.handlePut(CacheAdaptor.java:96)
at org.apache.commons.jcs.engine.AbstractCacheEventQueue$PutEvent.doRun(AbstractCacheEventQueue.java:362)
at org.apache.commons.jcs.engine.AbstractCacheEventQueue$AbstractCacheEvent.run(AbstractCacheEventQueue.java:281)
at org.apache.commons.jcs.engine.CacheEventQueue$QProcessor.run(CacheEventQueue.java:228)
I've got the serialization error with DefaultListableBeanFactory:
Caused by: java.io.NotSerializableException: DefaultListableBeanFactory has no serialization id
at org.springframework.beans.factory.support.DefaultListableBeanFactory.writeReplace(DefaultListableBeanFactory.java:1523)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1118)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.catalina.session.StandardSession.writeObject(StandardSession.java:1673)
at org.apache.catalina.session.StandardSession.writeObjectData(StandardSession.java:1079)
at org.apache.catalina.session.StandardManager.doUnload(StandardManager.java:432)
at org.apache.catalina.session.StandardManager.unload(StandardManager.java:353)
at org.apache.catalina.session.StandardManager.stopInternal(StandardManager.java:518)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5622)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1575)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1564)
... 4 more
After inspecting the code I've found out, that the class code requires some arbirtrary serialization id to be set somewhere:
protected Object writeReplace() throws ObjectStreamException {
if (this.serializationId != null) {
return new SerializedBeanFactoryReference(this.serializationId);
}
else {
throw new NotSerializableException("DefaultListableBeanFactory has no serialization id");
}
}
I'm not setting that serializationId, and I haven't found out, how can I set it. I suppose, something is missing in my spring xml :
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
">
<context:component-scan base-package="com.example.beans"/>
</beans>
The singleton beans are made serializable using aop scoped proxy:
#Component
#Scope(proxyMode = ScopedProxyMode.INTERFACES)
public class DatabaseAPIImpl implements DatabaseAPI {
but obviously, something is wrong with that proxy and some crucial informations are not set. What I am missing here? What should I set so that the DefaultListableBeanFactory is properly initialized?
I had a similar problem and solved it doing this:
#Component
public class SessionFixBeanFactoryPostProcessor implements BeanFactoryPostProcessor {
private static final String SERIALIZATION_ID = "4086d293-966c-4d89-8485-f1c1f5c09218";
#Override
public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
if ((beanFactory instanceof DefaultListableBeanFactory)) {
DefaultListableBeanFactory dlbf = (DefaultListableBeanFactory) beanFactory;
dlbf.setSerializationId(SERIALIZATION_ID);
}
}
}
You may want to read this question: https://github.com/spring-projects/spring-session/issues/799
Below is the code for my attempt to run aggregates on hazelcast using Hazelcast Client. The first aggregate works out just fine, but the 2nd one throws an java.io.NotSerializableException: co.near.hazelcast.AggregateExperiment
My code:
public class AggregateExperiment {
public void runTest(){
ClientConfig clientConfig = new ClientConfig();
clientConfig.addAddress("127.0.0.1:5701");
clientConfig.setClassLoader(this.getClass().getClassLoader());
HazelcastInstance client = HazelcastClient.newHazelcastClient(clientConfig);
IMap<String, Integer> map = client.getMap("customers");
Supplier<String, Integer, Integer> supplier = Supplier.all();
// Choose the average aggregation
Aggregation<String, Integer, Integer> aggregation = Aggregations.integerAvg();
int average = map.aggregate(supplier, aggregation);
System.out.println("Average of inputs = "+average);
supplier = Supplier.fromKeyPredicate(new MyKeyPredicate());
// Choose the sum aggregation
aggregation = Aggregations.integerSum();
average = map.aggregate(supplier, aggregation );
System.out.println("Average of inputs = "+average);
}
public class MyKeyPredicate implements KeyPredicate<String> {
public boolean evaluate(String key) {
return Integer.parseInt(key) % 4 == 0;
}
}
}
Error messages :
Average of inputs = 1
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.hazelcast.core.HazelcastException: java.util.concurrent.ExecutionException: com.hazelcast.nio.serialization.HazelcastSerializationException: java.io.NotSerializableException: co.near.hazelcast.AggregateExperiment
at com.hazelcast.client.proxy.ClientMapProxy.aggregate(ClientMapProxy.java:988)
at com.hazelcast.client.proxy.ClientMapProxy.aggregate(ClientMapProxy.java:960)
at co.near.hazelcast.AggregateExperiment.runTest(AggregateExperiment.java:31)
at co.near.hazelcast.MainExperiment.main(MainExperiment.java:131)
... 6 more
Caused by: java.util.concurrent.ExecutionException: com.hazelcast.nio.serialization.HazelcastSerializationException: java.io.NotSerializableException: co.near.hazelcast.AggregateExperiment
at com.hazelcast.client.spi.impl.ClientInvocationFuture.resolveException(ClientInvocationFuture.java:188)
at com.hazelcast.client.spi.impl.ClientInvocationFuture.resolveResponse(ClientInvocationFuture.java:160)
at com.hazelcast.client.spi.impl.ClientInvocationFuture.access$000(ClientInvocationFuture.java:41)
at com.hazelcast.client.spi.impl.ClientInvocationFuture$1.run(ClientInvocationFuture.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: java.io.NotSerializableException: co.near.hazelcast.AggregateExperiment
at com.hazelcast.nio.serialization.SerializationServiceImpl.handleException(SerializationServiceImpl.java:380)
at com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:307)
at com.hazelcast.nio.serialization.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:315)
at com.hazelcast.mapreduce.aggregation.impl.KeyPredicateSupplier.writeData(KeyPredicateSupplier.java:79)
at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:140)
at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:39)
at com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:37)
at com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:305)
at com.hazelcast.nio.serialization.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:315)
at com.hazelcast.mapreduce.aggregation.impl.SupplierConsumingMapper.writeData(SupplierConsumingMapper.java:75)
at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:140)
at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:39)
at com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:37)
at com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:305)
at com.hazelcast.nio.serialization.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:315)
at com.hazelcast.mapreduce.impl.client.ClientMapReduceRequest.writeData(ClientMapReduceRequest.java:204)
at com.hazelcast.mapreduce.impl.client.ClientMapReduceRequest.write(ClientMapReduceRequest.java:173)
at com.hazelcast.client.impl.client.ClientRequest.writePortable(ClientRequest.java:86)
at com.hazelcast.nio.serialization.PortableSerializer.writeInternal(PortableSerializer.java:62)
at com.hazelcast.nio.serialization.PortableSerializer.write(PortableSerializer.java:53)
at com.hazelcast.nio.serialization.PortableSerializer.write(PortableSerializer.java:29)
at com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:37)
at com.hazelcast.nio.serialization.SerializationServiceImpl.toData(SerializationServiceImpl.java:227)
at com.hazelcast.nio.serialization.SerializationServiceImpl.toData(SerializationServiceImpl.java:207)
at com.hazelcast.client.spi.impl.ClientInvocationServiceSupport.send(ClientInvocationServiceSupport.java:104)
at com.hazelcast.client.spi.impl.ClientSmartInvocationServiceImpl.invokeOnRandomTarget(ClientSmartInvocationServiceImpl.java:60)
at com.hazelcast.client.spi.impl.ClientInvocation.invokeOnSelection(ClientInvocation.java:163)
at com.hazelcast.client.spi.impl.ClientInvocation.invoke(ClientInvocation.java:147)
at com.hazelcast.client.proxy.ClientMapReduceProxy$ClientJob.invoke(ClientMapReduceProxy.java:124)
at com.hazelcast.mapreduce.impl.AbstractJob.submit(AbstractJob.java:119)
at com.hazelcast.mapreduce.impl.AbstractJob$ReducingSubmittableJobImpl.submit(AbstractJob.java:348)
at com.hazelcast.client.proxy.ClientMapProxy.aggregate(ClientMapProxy.java:985)
at com.hazelcast.client.proxy.ClientMapProxy.aggregate(ClientMapProxy.java:960)
at co.near.hazelcast.AggregateExperiment.runTest(AggregateExperiment.java:31)
at co.near.hazelcast.MainExperiment.main(MainExperiment.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
at ------ End remote and begin local stack-trace ------.(Unknown Source)
at com.hazelcast.client.spi.impl.ClientInvocationFuture.resolveException(ClientInvocationFuture.java:175)
... 8 more
Caused by: java.io.NotSerializableException: co.near.hazelcast.AggregateExperiment
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at com.hazelcast.nio.serialization.DefaultSerializers$ObjectSerializer.write(DefaultSerializers.java:223)
at com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:37)
at com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:305)
at com.hazelcast.nio.serialization.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:315)
at com.hazelcast.mapreduce.aggregation.impl.KeyPredicateSupplier.writeData(KeyPredicateSupplier.java:79)
at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:140)
at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:39)
at com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:37)
at com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:305)
at com.hazelcast.nio.serialization.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:315)
at com.hazelcast.mapreduce.aggregation.impl.SupplierConsumingMapper.writeData(SupplierConsumingMapper.java:75)
at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:140)
at com.hazelcast.nio.serialization.DataSerializer.write(DataSerializer.java:39)
at com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:37)
at com.hazelcast.nio.serialization.SerializationServiceImpl.writeObject(SerializationServiceImpl.java:305)
at com.hazelcast.nio.serialization.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:315)
at com.hazelcast.mapreduce.impl.client.ClientMapReduceRequest.writeData(ClientMapReduceRequest.java:204)
at com.hazelcast.mapreduce.impl.client.ClientMapReduceRequest.write(ClientMapReduceRequest.java:173)
at com.hazelcast.client.impl.client.ClientRequest.writePortable(ClientRequest.java:86)
at com.hazelcast.nio.serialization.PortableSerializer.writeInternal(PortableSerializer.java:62)
at com.hazelcast.nio.serialization.PortableSerializer.write(PortableSerializer.java:53)
at com.hazelcast.nio.serialization.PortableSerializer.write(PortableSerializer.java:29)
at com.hazelcast.nio.serialization.StreamSerializerAdapter.write(StreamSerializerAdapter.java:37)
at com.hazelcast.nio.serialization.SerializationServiceImpl.toData(SerializationServiceImpl.java:227)
at com.hazelcast.nio.serialization.SerializationServiceImpl.toData(SerializationServiceImpl.java:207)
at com.hazelcast.client.spi.impl.ClientInvocationServiceSupport.send(ClientInvocationServiceSupport.java:104)
at com.hazelcast.client.spi.impl.ClientSmartInvocationServiceImpl.invokeOnRandomTarget(ClientSmartInvocationServiceImpl.java:60)
at com.hazelcast.client.spi.impl.ClientInvocation.invokeOnSelection(ClientInvocation.java:163)
at com.hazelcast.client.spi.impl.ClientInvocation.invoke(ClientInvocation.java:147)
at com.hazelcast.client.proxy.ClientMapReduceProxy$ClientJob.invoke(ClientMapReduceProxy.java:124)
at com.hazelcast.mapreduce.impl.AbstractJob.submit(AbstractJob.java:119)
at com.hazelcast.mapreduce.impl.AbstractJob$ReducingSubmittableJobImpl.submit(AbstractJob.java:348)
at com.hazelcast.client.proxy.ClientMapProxy.aggregate(ClientMapProxy.java:985)
at com.hazelcast.client.proxy.ClientMapProxy.aggregate(ClientMapProxy.java:960)
at co.near.hazelcast.AggregateExperiment.runTest(AggregateExperiment.java:31)
at co.near.hazelcast.MainExperiment.main(MainExperiment.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Any help is much appreciated. Thanks
Instances of the inner class MyKeyPredicate contain an implicit reference to parent AggregateExperiment instance. This makes MyKeyPredicate not serializable. You should make MyKeyPredicate static:
public static class MyKeyPredicate implements KeyPredicate<String> {
...
Hazelcast as of now does not support distributed classloading which would be needed for the code above to work properly.
An issue is github is already opened for the same https://github.com/hazelcast/hazelcast/issues/7394
For now, those who really want to use Hazelcast in this manner can explore https://github.com/serkan-ozal/hermgen as well
I'm using Apache Spark Streaming 1.2.0 and trying to define a file filter for file names when creating an InputDStream by invoking the fileStream method. My code is working perfectly fine when I don't use a file filter, e.g. by invoking the other fileStream method (described here).
According to the documentation of fileStream method, I can pass it
scala.Function1<org.apache.hadoop.fs.Path,Object> filter
But so far, I could not create a fileFilter. My initial attempts have been
1- Tried to implement it as:
Function1<Path, Object> fileFilter = new Function1<Path, Object>() {
#Override
public Object apply(Path v1) {
return true;
}
#Override
public <A> Function1<A, Object> compose(Function1<A, Path> g) {
return Function1$class.compose(this, g);
}
#Override
public <A> Function1<Path, A> andThen(Function1<Object, A> g) {
return Function1$class.andThen(this, g);
}
};
But apparently my implementation of andThen is wrong, and I couldn't understand how I should implement it. It complains that the anonymous function
is not abstract and does not override abstract method <A>andThen$mcVJ$sp(scala.Function1<scala.runtime.BoxedUnit,A>) in scala.Function1
2- Tried to implement it as:
Function1<Path, Object> fileFilter = new AbstractFunction1<Path, Object>() {
#Override
public Object apply(Path v1) {
return true;
}
};
This one compiles but then when I run it I get an exception:
2015-02-02 13:42:50 ERROR OneForOneStrategy:66 - myModule$1
java.io.NotSerializableException: myModule$1
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
at org.apache.spark.streaming.DStreamGraph$$anonfun$writeObject$1.apply$mcV$sp(DStreamGraph.scala:169)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985)
at org.apache.spark.streaming.DStreamGraph.writeObject(DStreamGraph.scala:164)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.spark.streaming.CheckpointWriter.write(Checkpoint.scala:184)
at org.apache.spark.streaming.scheduler.JobGenerator.doCheckpoint(JobGenerator.scala:263)
at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:167)
at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$start$1$$anon$1$$anonfun$receive$1.applyOrElse(JobGenerator.scala:76)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$start$1$$anon$1.aroundReceive(JobGenerator.scala:74)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Any ideas how I can implement a fileFilter so that I can pass it fileStream method, so that I can make Spark Streaming process only the file name patterns I want?
I had to create another file named FileFilter.java:
import org.apache.hadoop.fs.Path;
import scala.runtime.AbstractFunction1;
import java.io.Serializable;
public class FileFilter extends AbstractFunction1<Path, Object> implements Serializable {
#Override
public Object apply(Path v1) {
if ( v1.toString().endsWith((".json")) ) {
return Boolean.TRUE;
} else {
return Boolean.FALSE;
}
}
}
And then pass it to the fileStream method as in:
fileStream(inDirectory, new FileFilter(), false, ...)
And it worked without any problems.