I have experienced "task not serializable" issue when running spark streaming. The reason can be found in this thread.
After I have tried couple methods and fixed the issue, I don't understand why it works.
public class StreamingNotWorking implements Serializable {
private SparkConf sparkConf;
private JavaStreamingContext jssc;
public StreamingNotWorking(parameter) {
sparkConf = new SparkConf();
this.jssc = createContext(parameter);
JavaDStream<String> messages = functionCreateStream(parameter);
messages.print();
}
public void run() {
this.jssc.start();
this.jssc.awaitTermination();
}
public class streamingNotWorkingDriver {
public static void main(String[] args) {
Streaming bieventsStreaming = new StreamingNotWorking(parameter);
bieventsStreaming.run();
}
Will give the same "Task not serializable" error.
However, if I modify the code to:
public class StreamingWorking implements Serializable {
private static SparkConf sparkConf;
private static JavaStreamingContext jssc;
public void createStream(parameter) {
sparkConf = new SparkConf();
this.jssc = createContext(parameter);
JavaDStream<String> messages = functionCreateStream(parameter);
messages.print();
run();
}
public void run() {
this.jssc.start();
this.jssc.awaitTermination();
}
public class streamingWorkingDriver {
public static void main(String[] args) {
Streaming bieventsStreaming = new StreamingWorking();
bieventsStreaming.createStream(parameter);
}
works perfectly fine.
I know one of the reasons is that sparkConf and jssc need to be static. But I don't understand why.
Could anyone explain the difference?
neither JavaStreamingContext nor SparkConf implements Serializable.
You can't serialize instances of classes without this interface.
Static members wont be serialized.
More information can be found here:
http://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html
Related
This problem doesn't have a solution to me but I hope you will. I have only experienced forge modding in 1.16.5 and in 1.18.1, there is so much Confusion, My Brain is Literally Broken and I have Dizziness just looking at it. I followed a code from 1.17 from TurtyWurty and this is it:
public static final List<ConfiguredFeature<?, ?>> OVERWORLD_ORES = new ArrayList<>();
public static final List<ConfiguredFeature<?, ?>> END_ORES = new ArrayList<>();
public static final List<ConfiguredFeature<?, ?>> NETHER_ORES = new ArrayList<>();
public static final RuleTest END_TEST = new BlockMatchTest(Blocks.END_STONE);
public static void registerOres() {
ConfiguredFeature<?, ?> bronzeOre = Feature.ORE
.configured(new OreConfiguration(List.of(
OreConfiguration.target(OreFeatures.STONE_ORE_REPLACEABLES,
Blocks.GLOWSTONE.defaultBlockState())),
11));
bronzeOre.placed(commonOrePlacement(8, HeightRangePlacement.triangle(VerticalAnchor.bottom(), VerticalAnchor.aboveBottom(20))));
OVERWORLD_ORES.add(register("bronze_ore", bronzeOre));
}
private static <Config extends FeatureConfiguration> ConfiguredFeature<Config, ?> register(String name,
ConfiguredFeature<Config, ?> configuredFeature) {
return Registry.register(BuiltinRegistries.CONFIGURED_FEATURE, new ResourceLocation(FendersMetalMod.MOD_ID, name),
configuredFeature);
}
#Mod.EventBusSubscriber(modid = FendersMetalMod.MOD_ID, bus = Mod.EventBusSubscriber.Bus.FORGE)
public static class ForgeBusSubscriber {
#SubscribeEvent
public static void biomeLoading(BiomeLoadingEvent event) {
List<Supplier<ConfiguredFeature<?, ?>>> features = event.getGeneration()
.getFeatures(GenerationStep.Decoration.UNDERGROUND_ORES);
switch(event.getCategory()) {
default -> MetalOreGen.OVERWORLD_ORES.forEach(ore -> features.add(() -> ore));
}
}
}
private static List<PlacementModifier> orePlacement(PlacementModifier p_195347_, PlacementModifier p_195348_) {
return List.of(p_195347_, InSquarePlacement.spread(), p_195348_, BiomeFilter.biome());
}
private static List<PlacementModifier> commonOrePlacement(int p_195344_, PlacementModifier p_195345_) {
return orePlacement(CountPlacement.of(p_195344_), p_195345_);
}
But the code has a problem, see in this part of the code:
List<Supplier<ConfiguredFeature<?, ?>>> features = event.getGeneration()
.getFeatures(GenerationStep.Decoration.UNDERGROUND_ORES);
It has an error in which I don't know but I think that:
"event.getGeneration().getFeatures(GenerationStep.Decoration.UNDERGROUND_ORES);"
is not correct and should provide a "List<Supplier<ConfiguredFeature<?, ?>>>"
because "event.getGeneration().getFeatures(GenerationStep.Decoration.UNDERGROUND_ORES);"
is a "List<Supplier<PlacedFeature>>", not a "List<Supplier<ConfiguredFeature<?, ?>>>".
I tried to change it but there are more errors that would appear if I would Change it.
Instrumentation using ByteBuddy is not working when I tried to instrument 3-rd party classes
I manage to create code which instrument my own code and everything worked as expected.
When I tried to use the same code for class which are part of 3-rd party dependencies the instrumentation didn't work.
This code is working for me:
public class A {
public void print(){
System.out.println("in class A method 'print'");
}
}
public class AgentLoad {
public static void agentmain(String agentArgs, Instrumentation inst) throws Exception {
final ElementMatcher.Junction<NamedElement> matcher = ElementMatchers.named("com.instrumentation.A");
new AgentBuilder.Default()
.type(matcher)
.transform( new AgentBuilder.Transformer.ForAdvice()
.include(AgentLoad.class.getClassLoader())
.advice(named("print"), AAdvice.class.getName()))
.installOn(inst);
}
public static void main(String[] args) throws Exception{
AgentLoader.loadAgentClass(AgentLoad.class.getName(), null);
}
public static class AAdvice {
#Advice.OnMethodEnter
public static void enter() {
System.out.println("Enter Yes!!!!");
}
#Advice.OnMethodExit
public static void exit() {
System.out.println("Exist Yes!!!!");
}
}
}
This code is not working for me:
public static void agentmain(String agentArgs, Instrumentation inst) {
AgentBuilder builder = new AgentBuilder.Default();
builder.type(ElementMatchers.named("com.amazonaws.http.AmazonHttpClient"))
.transform( new AgentBuilder.Transformer.ForAdvice()
.include(AgentLoad.class.getClassLoader())
.advice(ElementMatchers.named("execute").and(isAnnotatedWith(SdkInternalApi.class)),
AmazonHttpClientAdvice.class.getName()));
builder.installOn(inst);
}
public static class AmazonHttpClientAdvice {
#Advice.OnMethodEnter
public static void executeEnter(#Advice.Argument(0) Request<?> request) {
System.out.println("Eenter !!!" + request);
}
#Advice.OnMethodExit
public static void exit(#Advice.Return(readOnly = false, typing = DYNAMIC) Object returned) {
System.out.println("Exist !!!! " + returned);
}
}
}
The expected result is a print of enter and exit when calling the 'execute' in class AmazonHttpClient.
Note: The agent is dynamically attached using the next lib: https://github.com/electronicarts/ea-agent-loader
Byte Buddy's API is immutable, all method calls are without side-effects but only return a new builder instance. This means that:
AgentBuilder builder = new AgentBuilder.Default();
builder.type(ElementMatchers.named("com.amazonaws.http.AmazonHttpClient"))
.transform(new AgentBuilder.Transformer.ForAdvice()
.include(AgentLoad.class.getClassLoader())
.advice(ElementMatchers.named("execute").and(isAnnotatedWith(SdkInternalApi.class)),
AmazonHttpClientAdvice.class.getName()));
builder.installOn(inst);
does not do anything. You need to reassign the result of the builder chain to the builder variable again or call installOn within the chain.
I'm getting the Task not serializable error in Spark. I've searched and tried to use a static function as suggested in some posts, but it still gives the same error.
Code is as below:
public class Rating implements Serializable {
private SparkSession spark;
private SparkConf sparkConf;
private JavaSparkContext jsc;
private static Function<String, Rating> mapFunc;
public Rating() {
mapFunc = new Function<String, Rating>() {
public Rating call(String str) {
return Rating.parseRating(str);
}
};
}
public void runProcedure() {
sparkConf = new SparkConf().setAppName("Filter Example").setMaster("local");
jsc = new JavaSparkContext(sparkConf);
SparkSession spark = SparkSession.builder().master("local").appName("Word Count")
.config("spark.some.config.option", "some-value").getOrCreate();
JavaRDD<Rating> ratingsRDD = spark.read().textFile("sample_movielens_ratings.txt")
.javaRDD()
.map(mapFunc);
}
public static void main(String[] args) {
Rating newRating = new Rating();
newRating.runProcedure();
}
}
The error gives:
How do I solve this error?
Thanks in advance.
Clearly Rating cannot be Serializable, because it contains references to Spark structures (i.e. SparkSession, SparkConf, etc.) as attributes.
The problem here is in
JavaRDD<Rating> ratingsRD = spark.read().textFile("sample_movielens_ratings.txt")
.javaRDD()
.map(mapFunc);
If you look at the definition of mapFunc, you're returning a Rating object.
mapFunc = new Function<String, Rating>() {
public Rating call(String str) {
return Rating.parseRating(str);
}
};
This function is used inside a map (a transformation in Spark terms). Because the transformations are executed directly into the worker nodes and not in the driver node, their code must be serializable. This forces Spark to try serialize the Rating class, but it is not possible.
Try to extract the features you need from Rating, and placing them in a different class that does not own any Spark structure. Finally, use this new class as return type of your mapFunc function.
In addition you have to be sure to not include non-serializable variables in your class like JavaSparkContext and SparkSession. if you need to include them you should define like this:
private transient JavaSparkContext sparkCtx;
private transient SparkSession spark;
Good luck.
I am currently having some trouble with this code:
public class ChatPlugin extends JavaPlugin implements Listener{
private static ChatPlugin instance;
private final static String CHANNEL = "chat";
private JedisPool jedisPool;
private ChatChannel chatChannel;
#Override
public void onEnable()
{
instance = this;
saveDefaultConfig();
this.jedisPool = new JedisPool(new JedisPoolConfig(), getConfig().getString("redis-host"), 6379, 0,getConfig().getString("redis-password"));
this.chatChannel = new ChatChannel();
this.jedisPool.getResource().subscribe(this.chatChannel, new String[]{"chat"});
Bukkit.getPluginManager().registerEvents(this, this);
getCommand("chat").setExecutor(this);
}
#Override
public void onDisable()
{
instance = null;
this.chatChannel.unsubscribe();
this.jedisPool.destroy();
}
#Override
public boolean onCommand(CommandSender sender, Command command, String label, String[] args) {
if (command.getName().equalsIgnoreCase("chat"))
{
getJedis().publish(CHANNEL, args[0]);
sender.sendMessage(ChatColor.GREEN + "Sent!");
}
return true;
}
public Jedis getJedis(){
return jedisPool.getResource();
}
public static ChatPlugin getInstance() {
return instance;
}
public class ChatChannel extends JedisPubSub {
#Override
public void onMessage(String channel, String message) {
new BukkitRunnable(){
#Override
public void run() {
Bukkit.broadcastMessage(message);
}
}.runTask(ChatPlugin.getInstance());
}
}
}
This is my first time working with redis' PUB/SUB feature, and I don't really know what is happening. There are no stacktraces, it's just, when the plugin enables, it freezes. Yes, I know this is because I am doing it on the main thread, but even if I run it async (Which I have) it still does not work.
Hope someone can help!
Thanks
You didn't sent us your saveDefaultConfig method so first of all check out if the plugin config is created correctly. After checking that you don't made any mistakes regarding Bukkit or any logical mistake, try to google for any example or try to find examples on programcreek.com (I've found some for jedis there).
If you still get no clue: try doing what you should have done before you've asked here. Read the wiki of Jedis!
https://github.com/xetorthio/jedis/wiki
I'm working my way through the example code of some Storm topologies and bolts, but I'm running into something weird. My goal is to set up Kafka with Storm, so that Storm can process the messages available on the Kafka bus. I have the following bolt defined:
public class ReportBolt extends BaseRichBolt {
private static final long serialVersionUID = 6102304822420418016L;
private Map<String, Long> counts;
private OutputCollector collector;
#Override #SuppressWarnings("rawtypes")
public void prepare(Map stormConf, TopologyContext context, OutputCollector outCollector) {
collector = outCollector;
counts = new HashMap<String, Long>();
}
#Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
// terminal bolt = does not emit anything
}
#Override
public void execute(Tuple tuple) {
System.out.println("HELLO " + tuple);
}
#Override
public void cleanup() {
System.out.println("HELLO FINAL");
}
}
In essence, it should just output each Kafka message; and when the cleanup function is called, a different message should appear.
I have looked at the worker logs, and I find the final message (i.e. "HELLO FINAL"), but the Kafka messages with "HELLO" are nowhere to be found. As far as I can tell this should be a simple printer bolt, but I can't see where I'm going wrong. The workers logs indicate I am connected to the Kafka bus (it fetches the offset etc.).
In short, why are my println's not showing up in the worker logs?
EDIT
public class AckedTopology {
private static final String SPOUT_ID = "monitoring_test_spout";
private static final String REPORT_BOLT_ID = "acking-report-bolt";
private static final String TOPOLOGY_NAME = "monitoring-topology";
public static void main(String[] args) throws Exception {
int numSpoutExecutors = 1;
KafkaSpout kspout = buildKafkaSpout();
ReportBolt reportBolt = new ReportBolt();
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout(SPOUT_ID, kspout, numSpoutExecutors);
builder.setBolt(REPORT_BOLT_ID, reportBolt);
Config cfg = new Config();
StormSubmitter.submitTopology(TOPOLOGY_NAME, cfg, builder.createTopology());
}
private static KafkaSpout buildKafkaSpout() {
String zkHostPort = "URL";
String topic = "TOPIC";
String zkRoot = "/brokers";
String zkSpoutId = "monitoring_test_spout_id";
ZkHosts zkHosts = new ZkHosts(zkHostPort);
SpoutConfig spoutCfg = new SpoutConfig(zkHosts, topic, zkRoot, zkSpoutId);
KafkaSpout kafkaSpout = new KafkaSpout(spoutCfg);
return kafkaSpout;
}
}
Your bolt is not chained with the spout. You need to use storm's grouping in order to do that .. Use something like this
builder.setBolt(REPORT_BOLT_ID, reportBolt).shuffleGrouping(SPOUT_ID);
The setBolt typically returns a InputDeclarer object. In your case by specifying shuffleGrouping(SPOUT_ID) you are telling storm that you are interested in consuming all the tuples emitted by component having id REPORT_BOLT_ID.
Read more on stream groupings and choose the one based on your need.