Develop Reference

Deserialize Avro from kafka as SpecificRecord Failing. Expecting type to be a PojoTypeInfo

I am using Flink v1.11.2 and Avro v1.10.1.
I am trying to deserialize an Avro record as a Specific record from a Kafka topic, but for some reason keep getting this error:
I was able to output it as a generic record using this:
FlinkKafkaConsumer<GenericRecord> eventsConsumer = new FlinkKafkaConsumer(
fsiProcessorProps.getKafkaEventsInput(),
AvroDeserializationSchema.forGeneric(Sde.getClassSchema()),
properties);
And am now trying:
FlinkKafkaConsumer<Sde> eventsConsumer = new FlinkKafkaConsumer(
fsiProcessorProps.getKafkaEventsInput(),
AvroDeserializationSchema.forSpecific(Sde.class),
properties);
but get the following error when trying to start the job:
"org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute application.
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:103)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.CompletionException: org.apache.flink.util.FlinkRuntimeException: Could not execute application.
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
... 7 more
Caused by: org.apache.flink.util.FlinkRuntimeException: Could not execute application.
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:81)
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:67)
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:100)
... 7 more
Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Expecting type to be a PojoTypeInfo
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:78)
... 9 more
Caused by: java.lang.IllegalStateException: Expecting type to be a PojoTypeInfo
at org.apache.flink.formats.avro.typeutils.AvroTypeInfo.generateFieldsFromAvroSchema(AvroTypeInfo.java:74)
at org.apache.flink.formats.avro.typeutils.AvroTypeInfo.<init>(AvroTypeInfo.java:56)
at org.apache.flink.formats.avro.AvroDeserializationSchema.getProducedType(AvroDeserializationSchema.java:168)
at org.apache.flink.streaming.connectors.kafka.internals.KafkaDeserializationSchemaWrapper.getProducedType(KafkaDeserializationSchemaWrapper.java:66)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.getProducedType(FlinkKafkaConsumerBase.java:1066)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getTypeInfo(StreamExecutionEnvironment.java:2172)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1608)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1569)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.java:1551)
at FsiProcessor.main(FsiProcessor.java:46)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)
... 12 more
"
The Sde class is a typical generated avro class so I am not sure what I am missing. Also the data does not have the schema embedded (tried with schema embedded as well and got same issue).

So I wasn't able to find the answer to that because things switched to using a schema registry, but wanted to post how it was done with a schema registry at least.
So instead of doing:
FlinkKafkaConsumer<Sde> eventsConsumer = new FlinkKafkaConsumer(
fsiProcessorProps.getKafkaEventsInput(),
AvroDeserializationSchema.forSpecific(Sde.class),
properties);
It was changed to:
private static <T extends SpecificRecordBase> FlinkKafkaConsumerBase<T> getFlinkKafkaConsumer(
Properties properties, String topic, Class<? extends SpecificRecordBase> type, String schemaRegistryUrl, Map<String, String> sslConfiguration) {
ConfluentRegistryAvroDeserializationSchema<?> schema =
ConfluentRegistryAvroDeserializationSchema.forSpecific(
type, schemaRegistryUrl, sslConfiguration);
return new FlinkKafkaConsumer<T>(topic, (DeserializationSchema<T>) schema, properties);
}
I won't mark this as an answer, but wanted to give it as at least another way involving schema registry which is a good thing to use with Avro anyway.

Make it sure Avro generated POJO has both get/set method implemented by adding this:
<createSetters>true</createSetters>

java.lang.NoClassDefFoundError when using IfcJava3DViewer of APSTEX IFC Framework

I'm using APSTEX IFC Framework to creat a IFC 3D Viewer in my Java program.
My code is as follow.
import com.apstex.ifcjava3dviewer.IfcJava3DViewer;
public class JavaViewer2 {
public static void main(String[] args) {
IfcJava3DViewer ifcViewer = new IfcJava3DViewer();
}
}
The error is as followed:
java.lang.NoClassDefFoundError: com/jogamp/opengl/GLCapabilitiesImmutable
at com.apstex.javax.media.j3d.Pipeline$PipelineCreator.run(SourceFile:74)
at com.apstex.javax.media.j3d.Pipeline$PipelineCreator.run(SourceFile:1)
at java.security.AccessController.doPrivileged(Native Method)
at com.apstex.javax.media.j3d.Pipeline.b(SourceFile:91)
at com.apstex.javax.media.j3d.MasterControl.c(SourceFile:858)
at com.apstex.javax.media.j3d.VirtualUniverse.<clinit>(SourceFile:267)
at com.apstex.gui.core.controller.ApplicationController.<init>(SourceFile:1097)
at com.apstex.gui.core.kernel.Kernel.getApplicationController(SourceFile:59)
at com.apstex.ifcjava3dviewer.IfcJava3DPanel.<init>(SourceFile:71)
at com.apstex.ifcjava3dviewer.IfcJava3DViewer.<init>(SourceFile:38)
at test.main.JavaViewer2.main(JavaViewer2.java:7)
Caused by: java.lang.ClassNotFoundException: com.jogamp.opengl.GLCapabilitiesImmutable
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 11 more
Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class com.apstex.javax.media.j3d.VirtualUniverse
at com.apstex.javax.media.j3d.Canvas3D.<clinit>(SourceFile:3821)
at com.apstex.gui.ifc.views.view3d.j3d.ModelViewer.<init>(SourceFile:93)
at com.apstex.gui.ifc.views.view3d.j3d.ModelViewer.<init>(SourceFile:79)
at com.apstex.ifcjava3dviewer.IfcJava3DPanel.<init>(SourceFile:1079)
at com.apstex.ifcjava3dviewer.IfcJava3DViewer.<init>(SourceFile:38)
at test.main.JavaViewer2.main(JavaViewer2.java:7)
How can I correct the error?

Thanks for the two commends above to help me solve the problem. By installing a library of JOGL the program works.

Simplest horrorss code doesn't run

I am using the Java RSS parser horrorss
I added horrorss-2.2.0.jar to my Build path in Eclipse.
I can't even get the Simplest bit of Code to run.
import org.horrabin.horrorss.*;
public class Test {
public static void main(String[] args) throws Exception {
RssParser rss = new RssParser();
try{
RssFeed feed = rss.load("http://rss.slashdot.org/Slashdot/slashdot");
// Gets the channel information of the feed and
// display its title
RssChannelBean channel = feed.getChannel();
System.out.println("Feed Title: " + channel.getTitle());
}catch(Exception e){
System.err.println(e);
}
}
}
The Console Output is always the following
Exception in thread "main" java.lang.NoClassDefFoundError: com/hp/hpl/sparta/Parser
at org.horrabin.horrorss.RssParser.parseFromReader(RssParser.java:431)
at org.horrabin.horrorss.RssParser.parseFromURL(RssParser.java:455)
at org.horrabin.horrorss.RssParser.load(RssParser.java:115)
at org.horrabin.horrorss.RssParser.load(RssParser.java:147)
at Test.main(Test.java:10)
Caused by: java.lang.ClassNotFoundException: com.hp.hpl.sparta.Parser
at java.net.URLClassLoader$1.run(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 5 more
I hope you guys are able to help me out (or show me another nice method for Handling RSS Feeds in Java)

Did you read this on the homepage?
It requires Sparta-XML, a light-weight XML parser based on XPath.
You need to get the Sparta-XML jar and add that to the eclipse build path too.

HTTP Post with Matlab

I followed the example given in this post: How to send data from Matlab to Rails, but am receiving an error message that I cannot find any information on. My script looks like this:
javaaddpath('./httpcomponents/httpclient-4.2.2.jar')
javaaddpath('./httpcomponents/httpcore-4.2.2.jar')
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
httpclient = DefaultHttpClient();
httppost = HttpPost('http://localhost:3000/signin');
httppost.addHeader('Content-Type','application/json');
httppost.addHeader('Accept','application/json');
tokenRequest = StringEntity('{"session", "{email_address:""email#aol.com,""password:""password""}"}');
httppost.setEntity(tokenRequest);
response = httpclient.execute(httppost);
On the last line, I get the error:
Java exception occurred: java.lang.VerifyError: Cannot inherit from final class
From searches online, I gather that this is a software version issue. I tried using the 4.2 builds of those files (the same ones used in the other post), but I received the same error. Does anyone have an idea what could be wrong? Or know a way to do what I am trying to do without using these external libraries?
EDIT:
originally I tried using this code:
tokenRequest = {'session', '{''email_address'':''email#aol.com'',''password'':''password''}'};
token = urlread('http://localhost:3000/signin','POST',tokenRequest);
but I received the NoMethodError that led me to that other post:
NoMethodError (undefined method `each' for "{'email_address':'email#aol.com','password':'password'}":String):
app/models/session.rb:14:in `initialize'
I think the reason it throws this error is because the server thinks it is receiving a String object, which doesn't have an each method. I assume I would fix this by using the 'Content-Type' argument to specify that its json. Is there a way to do this using urlread?
EDIT: full stack trace for java libs issue
Java exception occurred:
java.lang.VerifyError: Cannot inherit from final class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(Unknown Source)
at java.security.SecureClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.access$000(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at com.mathworks.jmi.CustomURLClassLoader.findClass(ClassLoaderManager.java:760)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClassInternal(Unknown Source)
at org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:157)
at org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:448)
at org.apache.http.impl.client.AbstractHttpClient.createClientConnectionManager(AbstractHttpClient.java:309)
at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:466)
at org.apache.http.impl.client.AbstractHttpClient.createHttpContext(AbstractHttpClient.java:286)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:851)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)

Looks like the data being sent is not properly quoted. End result should be
{'email_address':'email#aol.com','password':'password'}
Try changing the code to
tokenRequest = StringEntity('{"session", {"email_address":"email#aol.com","password":"password"}}');

urlread2 turned out to be a much better solution. This code does the job:
tokenRequest = '{ "session" : { "email_address": "email#aol.com", "password": "password" } }';
header = http_createHeader('Content-Type','application/json');
token = urlread2('http://localhost:3000/signin.json','POST',tokenRequest,header);

Skype error? Or Eclipse error?

I tried making a simple project for my skype-bot, and it looks like this:
import com.skype.Skype;
import com.skype.SkypeException;
public class SkypeDemo {
public static void main(String[] args) throws SkypeException {
System.out.println(Skype.getVersion());
}
}
But when i started it i get this error:
Exception in thread "main" com.skype.SkypeException: Loading libskype.jnilib failed.
at com.skype.Utils.convertToSkypeException(Unknown Source)
at com.skype.Utils.getProperty(Unknown Source)
at com.skype.Skype.getVersion(Unknown Source)
at SkypeDemo.main(SkypeDemo.java:6)
Caused by: com.skype.connector.LoadLibraryException: Loading libskype.jnilib failed.
at com.skype.connector.ConnectorUtils.loadLibrary(Unknown Source)
at com.skype.connector.osx.SkypeFramework.init(Unknown Source)
at com.skype.connector.osx.OSXConnector.initializeImpl(Unknown Source)
at com.skype.connector.Connector.initialize(Unknown Source)
at com.skype.connector.Connector.connect(Unknown Source)
at com.skype.connector.Connector.assureAttached(Unknown Source)
at com.skype.connector.Connector.execute(Unknown Source)
at com.skype.connector.Connector.execute(Unknown Source)
at com.skype.connector.Connector.execute(Unknown Source)
at com.skype.connector.Connector.execute(Unknown Source)
... 3 more
So how do I fix this error? Keep in mind I have both Skype jars in the build path.

The error is telling you that it cannot load a native library ("libskype.jnilib"). This is (probably) neither Eclipse or Skype's fault.
It is most likely your fault because either the native library is not where it needs to be (or you haven't told Eclipse where to look), or because you have the wrong flavour of native code (DLL, .so or whatever)
(If there was another chained "cause" in the stack trace, it might tell you more ...)
Reference:
This blog post explains how to add a native library to an Eclipse project - http://www.eclipsezone.com/eclipse/forums/t49342.html (In fact, it describes two ways ...)