RuntimeException: Could not extract key occurs only on runtime environment - java

I am running flink locally on my machine , I am getting the exception below when reading from kafka topic. when running from the ide (intellij) it is running perfectly. however when I deploy my jar to flink runtime environment (locally) using
/bin/flink run ~MyApp-1.0-SNAPSHOT.jar
my class looks like this
case class Foo(id: String, value: String, timestamp: Long, counter: Int)
I am getting this exception
java.lang.RuntimeException: Could not extract key from Foo(some-uuid,some-text,1540348398,1)
at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)
at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)
at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:689)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:667)
at org.apache.flink.streaming.api.operators.StreamFilter.processElement(StreamFilter.java:40)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:689)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:667)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:111)
at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:398)
at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:89)
at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:154)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:738)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:56)
at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:99)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Could not extract key from Foo("some-uuid","text",1540348398,1)
at org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner.selectChannels(KeyGroupStreamPartitioner.java:61)
at org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner.selectChannels(KeyGroupStreamPartitioner.java:32)
at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:106)
at org.apache.flink.streaming.runtime.io.StreamRecordWriter.emit(StreamRecordWriter.java:81)
at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:107)
... 22 more
Caused by: java.lang.NullPointerException
at com.bluevoyant.StreamingJob$$anonfun$3.apply(StreamingJob.scala:41)
at com.bluevoyant.StreamingJob$$anonfun$3.apply(StreamingJob.scala:40)
at org.apache.flink.streaming.api.scala.DataStream$$anon$2.getKey(DataStream.scala:411)
at org.apache.flink.streaming.runtime.partitioner.KeyGroupStreamPartitioner.selectChannels(KeyGroupStreamPartitioner.java:59)
... 26 more
my key partition is simple (partitionFactor = some number)
env.addSource(kafkaConsumer)
.filter(_.id != null)
.keyBy{ r =>
val h = fastHash(r.id) % partitionFactor
math.abs(h)
}
.map(...)
again, this happens only on runtime not when I run it from intellij
this so frustrating, any advice ?

Related

Java call C library using native-image from GraalVM

I have a simple project: https://github.com/MarcoLunar/native-pid-test
All what it is doing is calling getpid from C library. Project is as simple as:
public static void main(String[] args) throws Exception {
System.out.println("start");
C_lib cLib = Native.loadLibrary("c", C_lib.class);
int getpid = cLib.getpid();
System.out.println("pid = " + getpid);
System.out.println("end");
}
When launching from IDE everything works fine:
start
pid = 155080
end
When trying to build using native-image from GraalVM I got this error:
[simpletest:155323] compile: 2,597.81 ms, 2.05 GB
Fatal error:org.graalvm.compiler.graph.GraalGraphError: java.lang.NullPointerException
at node: 43|&
at method: Object com.oracle.svm.reflect.JNIGeneratedMethodSupport_getFieldOffsetFromId_5041c78d77a7b3d62103393b72fc35d80d2cc709.invoke(Object, Object[])
at org.graalvm.compiler.phases.common.CanonicalizerPhase$Instance.tryCanonicalize(CanonicalizerPhase.java:397)
at org.graalvm.compiler.phases.common.CanonicalizerPhase$Instance.processNode(CanonicalizerPhase.java:325)
at org.graalvm.compiler.phases.common.CanonicalizerPhase$Instance.processWorkSet(CanonicalizerPhase.java:302)
at org.graalvm.compiler.phases.common.CanonicalizerPhase$Instance.run(CanonicalizerPhase.java:264)
at org.graalvm.compiler.phases.common.CanonicalizerPhase.run(CanonicalizerPhase.java:177)
at org.graalvm.compiler.phases.common.CanonicalizerPhase.run(CanonicalizerPhase.java:73)
at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:214)
at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:147)
at com.oracle.svm.hosted.code.CompileQueue.doInlineTrivial(CompileQueue.java:587)
at com.oracle.svm.hosted.code.CompileQueue.access$000(CompileQueue.java:156)
at com.oracle.svm.hosted.code.CompileQueue$TrivialInlineTask.run(CompileQueue.java:284)
at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.NullPointerException
at org.graalvm.compiler.nodes.calc.BinaryArithmeticNode.tryConstantFold(BinaryArithmeticNode.java:120)
I have tried many times, many different solutions ... but for now I have no more to check.
I am asking for help in repairing the project. I think this is possible, because with some configurations I have the same error as in https: //github.com/oracle/graal/issues/2261 ... where someone fixed it ... but didn't write the solution: (
I think currently JNA doesn't work in native image. You can use JNI if that's possible. Or you can use another interface to native code that will work specifically in the native images. Some info can be found in the javadoc of org.graalvm.nativeimage.c and its subpackages: https://www.graalvm.org/sdk/javadoc/index.html?org/graalvm/nativeimage/c/package-summary.html
Here's an example of using it: https://www.praj.in/posts/2020/opengl-demo-using-graalvm/

(java string concatenation)How come this is giving this exception?

this is my simple code.
class atmPin{
public static void main(String[] args) {
int x = 0;
System.out.println("hi there"+x);
}
}
When i run it i get this exception errors.
java atmPin
Exception in thread "main" java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: java/lang/invoke/StringConcatFactory
at atmPin.main(atmPin.java:6)
Caused by: java.lang.NoClassDefFoundError: java/lang/invoke/StringConcatFactory
... 1 more
Caused by: java.lang.ClassNotFoundException: java.lang.invoke.StringConcatFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
is this something to do with java version? i have no idea why is it happening. this is such a simple thing. :(
getting the same kind of exception even with strings.
eg.
String x = "";
x += "a";
java.lang.invoke.StringConcatFactory is a class which appeared in Java 9 in order to implement JEP 280 (Indify String Concatenation) which changes behaviour of javac compiler so that it uses invokedynamic instructon together with the given class for string concatentaion. So the reason for this error most probably is compiling using javac from JDK 9 or later while running this code under Java 8 or less (or, with much less chance, using custom java runtime).
In order to solve it another version of compiler might be used or another version may be specified to the compiler or another JRE should be used for running the compiled class(es).

Drools 6.4.0 fails with error uncomparable values

I have java class Fact as below
class AssertionFact {
String name;
Object value;
}
When i was using Drools 5.2 & rule as below it was working perfectly fine.
AssertionFact(
object > 100,
name == "testName"
)
But now when i have upgraded my drools version to 6.4.0 & JDK 1.8 I am getting below error for same rules.
Caused by: java.lang.RuntimeException: uncomparable values <<TESTVALUE>> and <<100.0>>
at org.mvel2.math.MathProcessor.doOperationNonNumeric(MathProcessor.java:274)
at org.mvel2.math.MathProcessor._doOperations(MathProcessor.java:234)
at org.mvel2.math.MathProcessor.doOperations(MathProcessor.java:79)
at org.mvel2.ast.BinaryOperation.getReducedValueAccelerated(BinaryOperation.java:117)
at org.mvel2.MVELRuntime.execute(MVELRuntime.java:85)
at org.mvel2.compiler.CompiledExpression.getDirectValue(CompiledExpression.java:123)
at org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:119)
at org.mvel2.MVEL.executeExpression(MVEL.java:929)
at org.drools.core.util.MVELSafeHelper$RawMVELEvaluator.executeExpression(MVELSafeHelper.java:496)

Groovy project won't build in GGTS

I have some Groovy code which will from fine from the command line, but when I try to use Springsource's GGTS fails:
Caught: java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at empyrean.Empyrean.run(Empyrean.groovy:20)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.ParseException
... 1 more
I know this is because I have not got paths or something similar set correctly in GGTS but I cannot work out how to fix this (I used to use STS without a problem, this is the first time I have tried GGTS)
The non-compiling code is this (first line here is line 20 in the file):
def empyreanCli = new CliBuilder(usage:'empyrean [-d] <empyrean script>')
empyreanCli.d(longOpt:'debug',
'output debug data as we go')
empyreanCli.f(longOpt:'file',
'Empyrean script to run')
empyreanCli.u(longOpt:'usage',
'show this usage message')
def empyreanParse = empyreanCli.parse(args)
if (empyreanParse.u || args.size() == 0)
empyreanCli.usage()
else {
def engine = new EmpyreanEngine()
if (empyreanParse.d)
engine.debug = true
if (empyreanParse.f)
engine.process(binsicParse.f)
else
engine.process(args[args.size() - 1])
}
Which, as I say, runs fine from the command line...
To get the project to build I have to explictly add the necessary .jar files to the build path:
project -> properties -> java build path -> libraries -> add JARs

JavaClassFormat exception [duplicate]

I am getting this strange error while executing the following code.
EncoderRequest encoderRequest = new EncoderRequest(sid,appTxnId,pfid,transactionType,"",isUpdatetype9,true);
I have checked all the parameter values are valid. I am using java7 plateform.
can any one have come across this situation, please help.
following is the part of stacktrace i am getting.
Caused by: java.lang.ClassFormatError: Illegal local variable table length 48 in method com.cmc.facts.encoder.EncoderRequest.<init>(JLjava/lang/String;Ljava/lang/Long;Lcom/cmc/facts/enums/TransactionType;Ljava/lang/String;ZZ)V at com.cmc.facts.nist.NistReaderModel.preprossingOfNistFile(NistReaderModel.java:180) at com.cmc.facts.action.interstate.InterStateAction.uploadFIIF(InterStateAction.java:645) ... 115 more
There have been previous reports of the same error, on Junit tests and similar..
For them, adding the JVM arg -XX:-UseSplitVerifier seemed to work
Have a look at this article
You can also do this config :
Add -noverify in your jvg args
For ant config you can do : <jvmarg value="-noverify"/>
You can follow the link for more details on why we need to do this.

Categories

Resources