I'm creating a cluter in EMR aws and when spark runs my application I'm getting error below:
Exception in thread "main" java.lang.UnsupportedClassVersionError:
com/example/demodriver/MyClassFromJAR has been compiled by a more recent version of the Java Runtime (class file version 55.0),
this version of the Java Runtime only recognizes class file versions up to 52.0
I'm using releaseLabel emr-6.5.0 on clusters and My driver jar is built in java11
How run java11 application in EMR? Or this error is about anything else?
In the most recent versions of EMR, java 11 is installed. To enable it, you can provide the following configuration.
[
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"Properties": {
"JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
}
}
]
},
{
"Classification": "spark-defaults",
"Properties": {
"spark.executorEnv.JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
}
}
]
This does not seem to be documented.
The defaultJavaOptions and extraJavaOptions might contain incompatible options for java 11 which you still might need to adapt/update.
Here is the full configuration including the necessary JVM options:
[
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"Properties": {
"JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
}
}
]
},
{
"Classification": "spark-defaults",
"Properties": {
"spark.executorEnv.JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64",
"spark.driver.defaultJavaOptions": "-XX:OnOutOfMemoryError='kill -9 %p' -XX:MaxHeapFreeRatio=70",
"spark.executor.defaultJavaOptions": "-verbose:gc -Xlog:gc*::time -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p' -XX:MaxHeapFreeRatio=70 -XX:+IgnoreUnrecognizedVMOptions"
}
}
]
For running Spark (3.3.0) on EMR 6 (6.9.0) I had to provide the following (note the additional "hadoop-env" and the empty "Properties" elements):
[
{
"Classification": "hadoop-env",
"Configurations": [
{
"Classification": "export",
"Configurations": [],
"Properties": {
"JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
}
}
],
"Properties": {}
},
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"Configurations": [],
"Properties": {
"JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
}
}
],
"Properties": {}
},
{
"Classification": "spark-defaults",
"Properties": {
"spark.executorEnv.JAVA_HOME": "/usr/lib/jvm/java-11-amazon-corretto.x86_64"
}
}
]
Related
I am using:
Java 17
Quarkus 2.12.0.Final
Jolt 0.1.7
graalvm-ce-22.0.0
Native image
mvn clean package -DskipTests -Pnative -Dquarkus.native.container-build=true
Using this transformer:
{
"operation": "default",
"spec": {
"user": {
"enabled": true
}
}
}
When a run the application using non-native image, it works!!
When a run the application the application with native image, occurs this error:
at com.bazaarvoice.jolt.common.DeepCopy.simpleDeepCopy(DeepCopy.java:53)
at com.bazaarvoice.jolt.defaultr.MapKey.applyLiteralKeyToContainer(MapKey.java:57)
at com.bazaarvoice.jolt.defaultr.MapKey.applyChild(MapKey.java:44)
at com.bazaarvoice.jolt.defaultr.Key.applyChildren(Key.java:162)
at com.bazaarvoice.jolt.defaultr.MapKey.applyLiteralKeyToContainer(MapKey.java:67)
at com.bazaarvoice.jolt.defaultr.MapKey.applyChild(MapKey.java:44)
at com.bazaarvoice.jolt.defaultr.Key.applyChildren(Key.java:162)
at com.bazaarvoice.jolt.Defaultr.transform(Defaultr.java:242)
at com.bazaarvoice.jolt.Chainr$ContextualTransformAdapter.transform(Chainr.java:113)
at com.bazaarvoice.jolt.Chainr.doTransform(Chainr.java:240)
at com.bazaarvoice.jolt.Chainr.transform(Chainr.java:180)
at com.mypackage.myclass.Play.run(Play.java:20)
at io.quarkus.amazon.lambda.runtime.AmazonLambdaRecorder$1.processRequest(AmazonLambdaRecorder.java:170)
at io.quarkus.amazon.lambda.runtime.AbstractLambdaPollLoop$1.run(AbstractLambdaPollLoop.java:130)
at java.lang.Thread.run(Thread.java:833)
at com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:704)
at com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:202)
caused by: java.io.InvalidClassException: java.lang.Boolean; no valid constructor
at java.io.ObjectStreamClass$ExceptionInfo.newInvalidClassException(ObjectStreamClass.java:174)
at java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:921)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2236)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1744)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:514)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:472)
at com.bazaarvoice.jolt.common.DeepCopy.simpleDeepCopy(DeepCopy.java:50)
... 26 more
As java.lang.Boolean has no default constructor, I tried registerForReflection, but not works too.
EDIT
I have found I workaround:
{
"operation": "default",
"spec": {
"user": {
"enabled": "true"
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"user": {
"enabled": "=toBoolean"
}
}
}
EDIT 2
This also happen with other values like Integer, Long, because they do not have default constructor.
Those errors happen because how the copy is made:
Look here, but I am Newbie in native image, so I don't know why this kind of copy does not work.
I've tried to create a VS Code launch configuration which passes command line arguments to a Spring Boot app.
#SpringBootApplication
public class DemoApplication {
public static void main(String[] args) {
for (String arg : args) {
System.out.println(arg);
}
SpringApplication.run(DemoApplication.class, args);
}
}
"configurations": [
{
"type": "java",
"name": "CodeLens (Launch) - DemoApplication",
"request": "launch",
"mainClass": "com.jackdry.processors.kafkalz4_compress.DemoApplication",
"projectName": "kafka-lz4_compress",
"vmArgs": "-Dspring-boot.run.arguments=-color=red"
}
]
But when I press "Debug" in VSCode, -color=red isn't being printed. Can anybody help?
If useful, the command which "Debug" actually invokes is:
cd /home/jack/jackdry/processors/kafka-lz4_compress ; /usr/lib/jvm/java-11-openjdk-amd64/bin/java -agentlib:jdwp=transport=dt_socket,server=n,suspend=y,address=localhost:38675 -Dspring-boot.run.arguments=-color=red -Dfile.encoding=UTF-8 #/tmp/cp_3noz6bed0u43t5tl3jj9cqoel.argfile com.jackdry.processors.kafkalz4_compress.DemoApplication
When running a Java program from the command line, assertions can be enabled with the -enableassertions option for the java command. Running this program would then (and only then) fail with an AssertionError:
public class App {
public static void main(String[] args) throws Exception {
foo(2);
}
private static void foo(int x) {
assert x > 5;
System.out.println(x);
}
}
How can this be done when running a Java program in Visual Studio Code with the Java Extension Pack?
Visual Studio Code manages launch configurations in the launch.json file in the project folder root.
The -enableassertions option can be added there with the vmArgs key like this:
{
"configurations": [
{
"type": "java",
"name": "My App",
"request": "launch",
"mainClass": "App",
"projectName": "my-app",
"vmArgs": "-enableassertions"
}
]
}
I need help with setting up ChromeDriver in Selenium WebDriver.
I have downloaded the latest version of ChromeDriver and added the path to the exe to my PATH environment variable.
However, when I try to run any scripts they fail on the line
driver = new ChromeDriver();
An instance of Chrome opens with a black screen and the address set as "data;," and I get a popup stating:
"Chrome Automation Extension has crashed".
Continuing from that point, I get the following exception details:
unknown error: unable to discover open pages
(Driver info: chromedriver=2.21.371459 (36d3d07f660ff2bc1bf28a75d1cdabed0983e7c4),platform=Windows NT 6.1 SP1 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 61.19 seconds
Build info: version: '2.52.0', revision: '4c2593c', time: '2016-02-11 19:03:33'
As I say, I've got the latest version of ChromeDriver and I have checked that Chrome is currently running the latest version so I know it isn't an issue with outdated versions of either.
I'm sure it must be something simple which I'm overlooking, but I just can't think what. Does anyone have any ideas?
Edit
Here's the log for the case, where Capability is not passed as a variable:
[0.885][INFO]: COMMAND InitSession {
"desiredCapabilities": {
"browserName": "chrome",
"chromeOptions": {
"args": [ ],
"extensions": [ ]
},
"platform": "ANY",
"version": ""
}
}
[0.886][INFO]: Populating Preferences file: {
"alternate_error_pages": {
"enabled": false
},
"autofill": {
"enabled": false
},
"browser": {
"check_default_browser": false
},
"distribution": {
"import_bookmarks": false,
"import_history": false,
"import_search_engine": false,
"make_chrome_default_for_user": false,
"show_welcome_page": false,
"skip_first_run_ui": true
},
"dns_prefetching": {
"enabled": false
},
"profile": {
"content_settings": {
"pattern_pairs": {
"https://*,*": {
"media-stream": {
"audio": "Default",
"video": "Default"
}
}
}
},
"default_content_setting_values": {
"geolocation": 1
},
"default_content_settings": {
"geolocation": 1,
"mouselock": 1,
"notifications": 1,
"popups": 1,
"ppapi-broker": 1
},
"password_manager_enabled": false
},
"safebrowsing": {
"enabled": false
},
"search": {
"suggest_enabled": false
},
"translate": {
"enabled": false
}
}
[0.889][INFO]: Populating Local State file: {
"background_mode": {
"enabled": false
},
"ssl": {
"rev_checking": {
"enabled": false
}
}
}
[0.936][INFO]: Launching chrome: "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --disable-background-networking --disable-client-side-phishing-detection --disable-component-update --disable-default-apps --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-sync --disable-web-resources --enable-logging --ignore-certificate-errors --load-extension="C:\Users\Rory\AppData\Local\Temp\scoped_dir10152_5411\internal" --log-level=0 --metrics-recording-only --no-first-run --password-store=basic --remote-debugging-port=12607 --safebrowsing-disable-auto-update --safebrowsing-disable-download-protection --test-type=webdriver --use-mock-keychain --user-data-dir="C:\Users\Rory\AppData\Local\Temp\scoped_dir10152_31299" data:,
[61.070][INFO]: RESPONSE InitSession unknown error: unable to discover open pages
I have the following metadata for the artist Joseph Turner:
{
"activePlaceCount":0,
"birth":{
"place":{
"name":"London, United Kingdom",
"placeName":"London",
"placeType":"inhabited_place"
},
"time":{
"startYear":1775
}
},
"birthYear":1775,
"date":"1775\u20131851",
"death":{
"place":{
"name":"Chelsea, United Kingdom",
"placeName":"Chelsea",
"placeType":"neighbourhood"
},
"time":{
"startYear":1851
}
},
"fc":"Joseph Mallord William Turner",
"gender":"Male",
"id":558,
"mda":"Turner, Joseph Mallord William",
"movements":[
{
"era":{
"id":290,
"name":"18th century"
},
"id":345,
"name":"Picturesque"
},
{
"era":{
"id":350,
"name":"19th century"
},
"id":364,
"name":"Romanticism"
},
{
"era":{
"id":290,
"name":"18th century"
},
"id":349,
"name":"Sublime"
}
],
"startLetter":"T",
"totalWorks":41861,
"url":"http://www.tate.org.uk/art/artists/joseph-mallord-william-turner-558"
}
If I wanted to map fc to firstName in my Java program, I could simply do the following:
#JsonProperty("fc")
private String fullName;
That, or private String fc.
What would I do if wanted to retrieve "death"."time"."startYear"?
Based on this SO answer, it is currently not possible to create an annotation of this sort, but it is a ticket for the future (updates of Java).