Develop Reference

java.lang.StackOverflowError when doing Spark Streaming

I'm doing Spark Streaming to parse some kafka messages in real time. Before I parse the message, I read some file from local and construct two variables GridMatrix GM and LinkMatcher LM which are useful to parsing. Here is the code that gives me java.lang.StackOverflowError when I submit it using spark-submit xxx.jar:
public class Stream implements Serializable {
GridMatrix GM = GridMatrixConstructor.init_Grid_Matrix(0.001);
LinkMatcher LM = new LinkMatcher();
public void parse_rdd_record(String[] fields) {
try {
System.out.println(InetAddress.getLocalHost().getHostName() + "---->" + Thread.currentThread());
}
catch (Exception e) {
e.printStackTrace();
}
System.out.println(LM.GF.toString());
System.out.println(GM.topleft_x);
}
public void Streaming_process() throws Exception {
SparkConf conf = new SparkConf()
.setAppName("SparkStreaming")
.setMaster("local[*]");
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
conf.registerKryoClasses(new Class<?>[]{
Class.forName("Streaming.Stream")
});
JavaSparkContext sc = new JavaSparkContext(conf);
sc.setLogLevel("WARN");
JavaStreamingContext ssc = new JavaStreamingContext(sc, new Duration(2000));
Map<String, Object> kafkaParams = new HashMap<>();
kafkaParams.put("bootstrap.servers", "xxx.xx.xx.xx:20103,xxx.xx.xx.xx:20104,xxx.xx.xx.xx:20105");
kafkaParams.put("key.deserializer", StringDeserializer.class);
kafkaParams.put("value.deserializer", StringDeserializer.class);
kafkaParams.put("group.id", "use_a_separate_group_id_for_each_stream");
kafkaParams.put("auto.offset.reset", "latest");
kafkaParams.put("enable.auto.commit", false);
Collection<String> topics = Arrays.asList("nc_topic_gis_test");
JavaInputDStream<ConsumerRecord<String, String>> GPS_DStream =
KafkaUtils.createDirectStream(
ssc,
LocationStrategies.PreferConsistent(),
ConsumerStrategies.<String, String>Subscribe(topics, kafkaParams)
);
JavaPairDStream<String, String> GPS_DStream_Pair = GPS_DStream.mapToPair(
(PairFunction<ConsumerRecord<String, String>, String, String>) record ->
new Tuple2<>("GPSValue", record.value()));
GPS_DStream_Pair.foreachRDD(PairRDD -> PairRDD.foreach(rdd -> {
String[] fields = rdd._2.split(",");
this.parse_rdd_record(fields);
}));
ssc.start();
ssc.awaitTermination();
}
public static void main(String[] args) throws Exception {
new Stream().Streaming_process();
}
}
It gives me the following error:
Exception in thread "streaming-job-executor-0" java.lang.StackOverflowError
at java.io.Bits.putDouble(Bits.java:121)
at java.io.ObjectStreamClass$FieldReflector.getPrimFieldValues(ObjectStreamClass.java:2168)
at java.io.ObjectStreamClass.getPrimFieldValues(ObjectStreamClass.java:1389)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1533)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.util.HashMap.internalWriteEntries(HashMap.java:1790)
at java.util.HashMap.writeObject(HashMap.java:1363)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1140)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.util.HashMap.internalWriteEntries(HashMap.java:1790)
at java.util.HashMap.writeObject(HashMap.java:1363)
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
However, if I change the GM and LM to static variable, it runs well. Change line 2 and line 3 to:
private static final GridMatrix GM = GridMatrixConstructor.init_Grid_Matrix(0.001);
private static final LinkMatcher LM = new LinkMatcher();
Could someone tell me why it won't work with none static variables?

The difference between static and non static version is that when non static it sends them to all workers as the Stream closure, when static not by default, except it using by one of streaming lambda, which is not a case.
While sending to the workers it trying to serialize an object. And it fails according to the provided stack trace. The reason most probably because those structures has cyclic declaration on themselfs inside and can not be properly serializable.

JCS Disable auxiliary cache due ConcurrentModification

i'm working with jcs with more clients of one single remote cache server.
Here the server configuration:
registry.host=xx.xx.xx.xx
registry.port=1104
remote.cache.service.port=1104
remote.cache.rmiSocketFactoryTimeoutMillis=5000
remote.cluster.LocalClusterConsistency=true
remote.cluster.AllowClusterGet=true
jcs.default=DC
jcs.default.cacheattributes=org.apache.commons.jcs.engine.CompositeCacheAttributes
jcs.default.cacheattributes.MaxObjects=100
jcs.default.cacheattributes.MaxMemoryIdleTimeSeconds=1800
jcs.default.cacheattributes.UseMemoryShrinker=true
jcs.default.elementattributes.IsEternal=false
jcs.default.elementattributes=org.apache.commons.jcs.engine.ElementAttributes
jcs.auxiliary.DC=org.apache.commons.jcs.auxiliary.disk.indexed.IndexedDiskCacheFactory jcs.auxiliary.DC.attributes=org.apache.commons.jcs.auxiliary.disk.indexed.IndexedDiskCacheAttributes
jcs.auxiliary.DC.attributes.DiskPath=/home/java/conf/data
jcs.auxiliary.DC.attributes.MaxPurgatorySize=1000
jcs.auxiliary.DC.attributes.MaxKeySize=1000
jcs.auxiliary.DC.attributes.OptimizeAtRemoveCount=300
jcs.auxiliary.DC.attributes.OptimizeOnShutdown=true
jcs.auxiliary.DC.attributes.ClearDiskOnStartup=true
jcs.auxiliary.DC.attributes.DiskLimitType=SIZE
And the client configuration:
jcs.auxiliary.RC=org.apache.commons.jcs.auxiliary.remote.RemoteCacheFactory
jcs.auxiliary.RC.attributes=org.apache.commons.jcs.auxiliary.remote.RemoteCacheAttributes
jcs.auxiliary.RC.attributes.FailoverServers=xx.xx.xx.xx:1104
jcs.auxiliary.RC.attributes.RemoveUponRemotePut=false
jcs.auxiliary.RC.attributes.Receive=false
jcs.auxiliary.RC.elementattributes.UseMemoryShrinker=true
Here the client code put:
public class JcsSharedCache<K, V> implements ICache<K, V> {
private CompositeCache<K, V> internal;
public void put(K key, V value) throws BeCacheException {
try {
synchronized(this){
internal.update(new CacheElement<>(internal.getCacheName(), key, value, internal.getElementAttributes()));
}
} catch (Exception ex) {
throw new BeCacheException(ex);
}
}
}
Sometimes a client get a ConcurrentModificationException and jcs disable the remote put. Every client had a synchronyzed version of put, but this happens(in my opinion) when two client try to put something in the same cache concurrently.
Is there a manner to avoid this behavior?
Below the stack trace:
2018-07-10 18:02:52 ERROR RemoteCache:141 - Disabling remote cache due
to error: Failed to put [da4a81e6-85df-426a-b6f6-a5b6777b0891] to
nodes java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
at java.util.HashMap$KeyIterator.next(HashMap.java:1466)
at java.util.HashSet.writeObject(HashSet.java:287)
at sun.reflect.GeneratedMethodAccessor135.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.util.HashMap.internalWriteEntries(HashMap.java:1790)
at java.util.HashMap.writeObject(HashMap.java:1363)
at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.util.ArrayList.writeObject(ArrayList.java:766)
at sun.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.commons.jcs.utils.serialization.StandardSerializer.serialize(StandardSerializer.java:55)
at org.apache.commons.jcs.utils.serialization.SerializationConversionUtil.getSerializedCacheElement(SerializationConversionUtil.java:74)
at org.apache.commons.jcs.auxiliary.remote.AbstractRemoteAuxiliaryCache.processUpdate(AbstractRemoteAuxiliaryCache.java:421)
at org.apache.commons.jcs.auxiliary.AbstractAuxiliaryCacheEventLogging.updateWithEventLogging(AbstractAuxiliaryCacheEventLogging.java:65)
at org.apache.commons.jcs.auxiliary.AbstractAuxiliaryCacheEventLogging.update(AbstractAuxiliaryCacheEventLogging.java:50)
at org.apache.commons.jcs.engine.CacheAdaptor.handlePut(CacheAdaptor.java:96)
at org.apache.commons.jcs.engine.AbstractCacheEventQueue$PutEvent.doRun(AbstractCacheEventQueue.java:362)
at org.apache.commons.jcs.engine.AbstractCacheEventQueue$AbstractCacheEvent.run(AbstractCacheEventQueue.java:281)
at org.apache.commons.jcs.engine.CacheEventQueue$QProcessor.run(CacheEventQueue.java:228)

How to define the serialization id of the DefaultListableBeanFactory?

I've got the serialization error with DefaultListableBeanFactory:
Caused by: java.io.NotSerializableException: DefaultListableBeanFactory has no serialization id
at org.springframework.beans.factory.support.DefaultListableBeanFactory.writeReplace(DefaultListableBeanFactory.java:1523)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1118)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.catalina.session.StandardSession.writeObject(StandardSession.java:1673)
at org.apache.catalina.session.StandardSession.writeObjectData(StandardSession.java:1079)
at org.apache.catalina.session.StandardManager.doUnload(StandardManager.java:432)
at org.apache.catalina.session.StandardManager.unload(StandardManager.java:353)
at org.apache.catalina.session.StandardManager.stopInternal(StandardManager.java:518)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5622)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1575)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1564)
... 4 more
After inspecting the code I've found out, that the class code requires some arbirtrary serialization id to be set somewhere:
protected Object writeReplace() throws ObjectStreamException {
if (this.serializationId != null) {
return new SerializedBeanFactoryReference(this.serializationId);
}
else {
throw new NotSerializableException("DefaultListableBeanFactory has no serialization id");
}
}
I'm not setting that serializationId, and I haven't found out, how can I set it. I suppose, something is missing in my spring xml :
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
">
<context:component-scan base-package="com.example.beans"/>
</beans>
The singleton beans are made serializable using aop scoped proxy:
#Component
#Scope(proxyMode = ScopedProxyMode.INTERFACES)
public class DatabaseAPIImpl implements DatabaseAPI {
but obviously, something is wrong with that proxy and some crucial informations are not set. What I am missing here? What should I set so that the DefaultListableBeanFactory is properly initialized?

I had a similar problem and solved it doing this:
#Component
public class SessionFixBeanFactoryPostProcessor implements BeanFactoryPostProcessor {
private static final String SERIALIZATION_ID = "4086d293-966c-4d89-8485-f1c1f5c09218";
#Override
public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
if ((beanFactory instanceof DefaultListableBeanFactory)) {
DefaultListableBeanFactory dlbf = (DefaultListableBeanFactory) beanFactory;
dlbf.setSerializationId(SERIALIZATION_ID);
}
}
}
You may want to read this question: https://github.com/spring-projects/spring-session/issues/799

How to define a file filter for file name patterns in Apache Spark Streaming in Java?

I'm using Apache Spark Streaming 1.2.0 and trying to define a file filter for file names when creating an InputDStream by invoking the fileStream method. My code is working perfectly fine when I don't use a file filter, e.g. by invoking the other fileStream method (described here).
According to the documentation of fileStream method, I can pass it
scala.Function1<org.apache.hadoop.fs.Path,Object> filter
But so far, I could not create a fileFilter. My initial attempts have been
1- Tried to implement it as:
Function1<Path, Object> fileFilter = new Function1<Path, Object>() {
#Override
public Object apply(Path v1) {
return true;
}
#Override
public <A> Function1<A, Object> compose(Function1<A, Path> g) {
return Function1$class.compose(this, g);
}
#Override
public <A> Function1<Path, A> andThen(Function1<Object, A> g) {
return Function1$class.andThen(this, g);
}
};
But apparently my implementation of andThen is wrong, and I couldn't understand how I should implement it. It complains that the anonymous function
is not abstract and does not override abstract method <A>andThen$mcVJ$sp(scala.Function1<scala.runtime.BoxedUnit,A>) in scala.Function1
2- Tried to implement it as:
Function1<Path, Object> fileFilter = new AbstractFunction1<Path, Object>() {
#Override
public Object apply(Path v1) {
return true;
}
};
This one compiles but then when I run it I get an exception:
2015-02-02 13:42:50 ERROR OneForOneStrategy:66 - myModule$1
java.io.NotSerializableException: myModule$1
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
at org.apache.spark.streaming.DStreamGraph$$anonfun$writeObject$1.apply$mcV$sp(DStreamGraph.scala:169)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985)
at org.apache.spark.streaming.DStreamGraph.writeObject(DStreamGraph.scala:164)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.spark.streaming.CheckpointWriter.write(Checkpoint.scala:184)
at org.apache.spark.streaming.scheduler.JobGenerator.doCheckpoint(JobGenerator.scala:263)
at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:167)
at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$start$1$$anon$1$$anonfun$receive$1.applyOrElse(JobGenerator.scala:76)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$start$1$$anon$1.aroundReceive(JobGenerator.scala:74)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Any ideas how I can implement a fileFilter so that I can pass it fileStream method, so that I can make Spark Streaming process only the file name patterns I want?

I had to create another file named FileFilter.java:
import org.apache.hadoop.fs.Path;
import scala.runtime.AbstractFunction1;
import java.io.Serializable;
public class FileFilter extends AbstractFunction1<Path, Object> implements Serializable {
#Override
public Object apply(Path v1) {
if ( v1.toString().endsWith((".json")) ) {
return Boolean.TRUE;
} else {
return Boolean.FALSE;
}
}
}
And then pass it to the fileStream method as in:
fileStream(inDirectory, new FileFilter(), false, ...)
And it worked without any problems.

A good way to find unserializable fields in Java

I have a quite complex Java object to serialize (which worked fine a couple of weeks ago). After having implemented a lot in the meantime, the serialization now fails throwing the following exception:
java.io.NotSerializableException: sun.java2d.SunGraphics2D
I have inspected all classes which seemed relevant, but could not find a field in any of them holding a Graphics2D type (or a similar type such as BufferedImage or the like). As I see it, the problem could also be that I'm using a class (from a library or so) which itself is serializable, but has a field of the unserializable type Graphics2D.
So, my question is: is there a "good" way to find the place where the serialization fails? Inspecting all classes and all changes made seems to be hardly feasible.
Thank you in advance
Lukas
PS. Here is the full stack trace of the exception:
java.io.NotSerializableException: sun.java2d.SunGraphics2D
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1707)
at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:482)
at java.awt.Container.writeObject(Container.java:3697)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1707)
at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:482)
at java.awt.Container.writeObject(Container.java:3697)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1707)
at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:482)
at java.awt.Container.writeObject(Container.java:3697)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1707)
at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:482)
at java.awt.Container.writeObject(Container.java:3697)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.awt.Window.writeObject(Window.java:2943)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.awt.AWTEventMulticaster.save(AWTEventMulticaster.java:946)
at java.awt.Window.writeObject(Window.java:2931)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at java.util.LinkedList.writeObject(LinkedList.java:1131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at eas.simulation.SerializableSimulationState.save(SerializableSimulationState.java:121)
at eas.plugins.standard.visualization.ControlPanel.actionPerformed(ControlPanel.java:202)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2346)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
at java.awt.Component.processMouseEvent(Component.java:6527)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3321)
at java.awt.Component.processEvent(Component.java:6292)
at java.awt.Container.processEvent(Container.java:2234)
at java.awt.Component.dispatchEventImpl(Component.java:4883)
at java.awt.Container.dispatchEventImpl(Container.java:2292)
at java.awt.Component.dispatchEvent(Component.java:4705)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4898)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4533)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4462)
at java.awt.Container.dispatchEventImpl(Container.java:2278)
at java.awt.Component.dispatchEvent(Component.java:4705)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:746)
at java.awt.EventQueue.access$400(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:697)
at java.awt.EventQueue$3.run(EventQueue.java:691)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:75)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:86)
at java.awt.EventQueue$4.run(EventQueue.java:719)
at java.awt.EventQueue$4.run(EventQueue.java:717)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:75)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:716)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

Okay, inspired by the idea of Ben Lawry who, however, did not present an actual implementation, I have also programmed a class crawler which finds "unserializable fields", i.e., non-static, non-transient fields that are either not implementing the Serializable interface themselves or contain fields that are non-static, non-transient and do not implement the Serializable interface.
You can add several classes to be tested in the list in the main method (see comment). The output is a list of fields from these classes or any referenced classes that are not serializable by the above definition (which, however, is not exactly equal to the actually serializable fields, but captures a superset of the latter); for each field which itself does implement the Serializable interface, a list of fields referenced in the type of that field is given which cause the field being not serializable (if the list is empty the field itself does not implement the Serializable interface).
Here is the code, hope it helps others, too:
import java.io.Serializable;
import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.LinkedList;
import javax.swing.JComponent;
public class Crawler {
public static boolean crawlRecursively(Field field, HashSet<Class<?>> alreadyCrawled, HashMap<Field, HashSet<String>> badFields) {
if (alreadyCrawled.contains(field.getType())) {
return !badFields.keySet().contains(field);
}
alreadyCrawled.add(field.getType());
if (Modifier.isStatic(field.getModifiers())
|| Modifier.isTransient(field.getModifiers())
|| field.getType().isPrimitive()) {
return true;
} else if (Serializable.class.isAssignableFrom(field.getType())) {
boolean allGood = true;
for (Field f : field.getType().getDeclaredFields()) {
boolean isGood = crawlRecursively(f, alreadyCrawled, badFields);
if (!isGood) {
if (!badFields.containsKey(field)) {
badFields.put(field, new HashSet<>());
}
badFields.get(field).add(f.getType().getSimpleName() + " " + f.getName());
allGood = false;
}
}
return allGood;
} else {
if (!badFields.containsKey(field)) {
badFields.put(field, new HashSet<>());
}
return false;
}
}
public static HashMap<Field, HashSet<String>> initiateCrawling(Collection<Class<?>> roots) {
HashMap<Field, HashSet<String>> badFields = new HashMap<>();
for (Class<?> root : roots) {
for (Field f : root.getDeclaredFields()) {
crawlRecursively(f, new HashSet<>(), badFields);
}
}
return badFields;
}
public static void main(String[] args) {
LinkedList<Class<?>> roots = new LinkedList<>();
roots.add(JComponent.class); // ADD YOUR CLASSES HERE.
HashMap<Field, HashSet<String>> badFields = initiateCrawling(roots);
if (badFields.keySet().size() == 0) {
System.out.println("All fields are serializable (not having checked the given class(es) themselves).");
} else {
System.out.println("The following fields are not serializable in the class tree(s) given by " + roots + ":");
}
for (Field field : badFields.keySet()) {
System.out.println("<UnSer> "
+ field.getType().getSimpleName() + " "
+ field.getName() + " ("
+ field.getDeclaringClass().getName() + ") => " + badFields.get(field));
}
}
}
Example output for class JComponent:
The following fields are not serializable in the class tree(s) given by [class javax.swing.JComponent]:
<UnSer> Border border (javax.swing.JComponent) => []
<UnSer> ComponentInputMap windowInputMap (javax.swing.JComponent) => [JComponent component]
<UnSer> VetoableChangeSupport vetoableChangeSupport (javax.swing.JComponent) => [VetoableChangeListenerMap map, Object source]
<UnSer> SingleSelectionModel selectionModel (javax.swing.JPopupMenu) => []
<UnSer> JPopupMenu popupMenu (javax.swing.JComponent) => [SingleSelectionModel selectionModel]
<UnSer> Object source (java.beans.VetoableChangeSupport) => []
<UnSer> VetoableChangeListenerMap map (java.beans.VetoableChangeSupport) => []
<UnSer> JComponent component (javax.swing.ComponentInputMap) => [VetoableChangeSupport vetoableChangeSupport, JPopupMenu popupMenu, Border border, InputVerifier inputVerifier]
<UnSer> InputVerifier inputVerifier (javax.swing.JComponent) => []

The library class can be assumed to be correctly serializable, i.e. to declare all non-serializable fields as transient, and therefore shouldn't be the source of the error.
If your code is using ContextedRuntimeExceptions, they are an exception to the above rule because the objects they reference may not be serializable, and this is noted in the Javadoc at the methods addContextValue() and setContextValue(). If context values are the problem, it would explain why the suspect classes got a reference to a Graphics2D object without having such a field.
I have had limited success replicating NotSerializableException for SunGraphics2D without leaving an obvious field with "Graphics" in its name laying around, which could be due to testing on a simpler class structure. I found that in addition to contexted exceptions, collections were a good suspect but only if they used raw types. Collections could also cause intermittent failures if they were only sometimes populated with Graphics.
Depending on your application, Ben Lawry's automated solution to a similar issue may be easier to implement than manually hunting for the above. If that path works, it will likely be more thorough as well. Can you get a stack trace?