I'm new to Spark so am trying to setup a project from the book Learning Spark: Lightning-Fast Big Data Analysis. The book uses version 1.3 but I've only got 2.1.1 so am trying to work around a few differences.
All the Spark related jars that I'm importing into my Java project have a "import org.apache cannot be resolved". I know it's because the project cannot find the jar files specified.
I can manually add each by going to Build Path > Configure Build path and adding them to the Libraries section but I think I shouldn't need to do this. The project uses Maven so I believe if I have the Spark dependencies configured correctly in my pom.xml it should work. Is this correct?
I also set the following environment variables:
export SPARK_HOME=/Users/mymac/spark-2.1.1-bin-hadoop2.7/
export PATH="$SPARK_HOME/bin/:$PATH"
Are there any others I should be aware of?
Here's the contents of my pom.xml:
<project>
<groupId>com.oreilly.learningsparkexamples.mini</groupId>
<artifactId>learning-spark-mini-example</artifactId>
<modelVersion>4.0.0</modelVersion>
<name>example</name>
<packaging>jar</packaging>
<version>0.0.1</version>
<dependencies>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.1.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.8</version>
</dependency>
</dependencies>
<properties>
<java.version>1.8</java.version>
</properties>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
This should be setup as a Maven project, not a Java project. In my case to resolve deleted the project from my workspace, re-created it in the workspace as a general project, then converted it to a Maven project. I probably should have just set it up as a Maven project at the start.
I'm trying to deploy a project on Heroku but I'm getting Maven compilaton errors like this:
remote: [ERROR] /tmp/build_5d64555c50abcb9638e3ef5b331a0107/src/main/java/com/davioooh/myapp/services/TestService.java:[3,43] package com.davioooh.myapp.domain does not exist
In my project I'm using both Java and Groovy classes. All Groovy classes are in com.davioooh.myapp.domain that can't be found during compilation.
I also tried to move all Groovy classe in src/main/groovy folder but it's still not working...
Is there a way to correctly deploy my application?
I finally solved adding Groovy Eclipse Maven Plugin as compiler plugin in my project pom.xml.
<build>
...
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<!-- 2.8.0-01 and later require maven-compiler-plugin 3.1 or higher -->
<version>3.1</version>
<configuration>
<compilerId>groovy-eclipse-compiler</compilerId>
</configuration>
<dependencies>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-compiler</artifactId>
<version>2.9.1-01</version>
</dependency>
<!-- for 2.8.0-01 and later you must have an explicit dependency on groovy-eclipse-batch -->
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-batch</artifactId>
<version>2.3.7-01</version>
</dependency>
</dependencies>
</plugin>
</plugins>
...
</build>
I am using eclipse as IDE. When I right click on the project and then click maven update my java version change to 1.5. Here is what I did so far, I followed all the steps listed here
http://qussay.com/2013/09/13/solving-dynamic-web-module-3-0-requires-java-1-6-or-newer-in-maven-projects/
I changed "Java build path" to "workspace default jre 1.8.0_25"
Then changed "java compiler" to 1.8
Then changed "project facets">java>1.8
Changed pom.xml java version to 1.8
<build>
<plugins>
<plugin>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-maven-plugin</artifactId>
<version>9.1.3.v20140225</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugin</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
After all this when I click on "Maven update" my java version change to 1.5 automatically. Also in above steps, first two step's version also change to 1.5 automatically. How can I fix this?
Open your pom.xml file and add the following lines on it:
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
Where 1.8 is the Java version of your current JDK/JRE. Another way of doing this is adding a <build> with the maven-compile-plugin as:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version> <!-- or whatever current version -->
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
If you are looking for a way to make it work with Java versions 9+ please take a look at #JDelorean's answer.
Had the same issue when I installed Java 9. My project would default to J2SE-1.5 Execution Environment. Strangely, Java 9 compliance level is not referenced like previous versions, i.e. "1.8", but as "9". So I had to provide my properties and Maven compiler plugin config accordingly:
<properties>
<maven.compiler.source>9</maven.compiler.source>
<maven.compiler.target>9</maven.compiler.target>
</properties>
and
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>9</source>
<target>9</target>
</configuration>
</plugin>
This seems to have solved the problem. Works for versions 9 and above.
The root-cause of this issue is that if for any reason Eclipse's cannot resolve a valid value for the maven.compiler.source property when generating/updating the .classpath file from the pom, it will simply default to using org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/J2SE-1.5.
As expertly answered by #jorge-campos, there are multiple ways to set that property.
However, Jorge's answer didn't appear to work for me. Here were my settings:
<properties>
<javaVersion>1.8</javaVersion>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
</properties>
...
Exactly. ${java.version} is never going to resolve to the (completely different) property javaVersion and Eclipse ignored the property and used the default.
Which brings me back to the "for any reason" part I opened with; developer stupidity can be one of those reasons.
Add this lines to your pom.xml, then right click your JRE System Library -> Properties -> Set your correct execution environment to Java 1.8 or version you want to set.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version> <!-- or whatever current version -->
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
I encounter similar issue on one of my team mate machine. He was using old version of Eclipse, I believe it he was using Keppler. Project after being updated change JRE version to 1.5.
Simple updating Eclipse to latest version solve this problem.
In my case (old JBoss Developer Studio), the issue was the JRE environments did not include 1.8 (only 1.7). When I switched the maven-compiler-plugin version to 1.7 and did maven update project, it updated the Eclipse JRE system library to 1.7.
So the solution is to either get a newer IDE version that includes a built-in JRE environment that is 1.8 or later, or try to install it manually (see https://stackoverflow.com/a/35204314)
I had this problem. In my case the <properties> tag & nested tags Jorge Campos mentions above were in the wrong place. If I put them between the <hostversion> and <dependencies> tags in the pom.xml file, then this behaviour stopped.
That can be picked up in Eclipse if validation of these files is switched on.
I am using Java 11.
This is how the complete pom.xml file looks like after adding
<properties> and <plugin>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.akshay</groupId>
<artifactId>1000SpringSecurityEg</artifactId>
<packaging>war</packaging>
<version>0.0.1-SNAPSHOT</version>
<name>1000SpringSecurityEg Maven Webapp</name>
<url>http://maven.apache.org</url>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<properties>
<javaVersion>11</javaVersion>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
</properties>
<build>
<finalName>1000SpringSecurityEg</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>11</source>
<target>11</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
The above code worked for me.
Hope it works for you as well.
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
</properties>
I added these lines in my "pom.xml" file and it worked.
I allow myself to update that subject with Java 11.
I have installed OpenJDK11 on my computer, and I wanted to use it in an app.
I had trouble because Eclipse would always change my JRE to JavaSE-1.5 when I updated my project with Maven.
I had set everything as you said, but I was always directly selecting in my Java Build Path "java-11-openjdk.x86_64" as one of my Alternante JRE. I fixed my problem by selecting in "Execution environment" JavaSE-10 (but you have to double click on it and then choose as a compatible JRE your OpenJDK11 version) as shown on the picture.
Execution environment setup
The project will use Java 11 thanks to that (picture) but you have to write 10 for the java-version in the pom.xml and also set java 10 on the Project Facets.
I've resolved the issue installing the eclipse update "JAVA 12" from the market.
It makes my eclipse pass from Kepler to Luna.
After that, i have been able to set 1.8 as standard JDK, fixing the "maven update" problem.
I experienced with JRE 15.0.1 one must ONLY specify the compiler plugin like
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>15</source>
<target>15</target>
</configuration>
</plugin>
If I also provide the properties like
<properties>
<maven.compiler.source>15</maven.compiler.source>
<maven.compiler.target>15</maven.compiler.target>
</properties>
this will again reset to JRE 1.5 on Maven / Update Project
!!!
Check in pom.xml under properties if there is any tag with this maven.enforcer.plugin.version. Delete it and replace that with the below code
<javaVersion>1.8</javaVersion>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
under properties tag.
And under build, replace the plugins with the below code:
<build>
<finalName>1000SpringSecurityEg</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
This solution resolved my issue.
I changed Eclipse from kepler to neon and then updated my project by with Maven -> Update Project.
It seems Maven keep using use an old version of my code when packaging a war archive.
I build my war using a simple "mvn clean install".
I deleted the /target folder by hand and checked that there were no .class elsewhere in my project (as described here).
Each time I check the generated archive, an old version of my code has been packaged inside. Yet, the same code is properly packaged on my colleague's machine (he uses m2e).
Does anyone have the same problem ?
Here is the most relevant part of the POM:
<packaging>war</packaging>
...
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<!-- <version>2.3.2</version> -->
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.4</version>
</plugin>
</plugins>
</build>
It appears that the old code is still present somewhere; my guess would be your local repository. Try deleting any entry there that could contain the old code and then building your project again.
Check the <dependency> entry in pom.xml for that jar file. Change the <version> of that dependency to the latest one.
I need to run an annotation processor on my project's sources. The annotation processor should not become a transitive dependency of the project since it's only needed for annotation processing and nothing else.
Here is the complete (non-working) test pom I use for this:
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>test</groupId>
<artifactId>test</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>Test annotations</name>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<hibernate-jpamodelgen.version>1.2.0.Final</hibernate-jpamodelgen.version>
</properties>
<dependencies>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-api</artifactId>
<version>6.0</version>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.0</version>
<configuration>
<annotationProcessors>
<annotationProcessor>
org.hibernate.jpamodelgen.JPAMetaModelEntityProcessor</annotationProcessor>
</annotationProcessors>
<debug>true</debug>
<optimize>true</optimize>
<source>1.6</source>
<target>1.6</target>
<compilerArguments>
<AaddGeneratedAnnotation>true</AaddGeneratedAnnotation>
<Adebug>true</Adebug>
</compilerArguments>
</configuration>
<dependencies>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-jpamodelgen</artifactId>
<version>${hibernate-jpamodelgen.version}</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>
I explicitly defined org.hibernate.jpamodelgen.JPAMetaModelEntityProcessor as an annotation processor in the plugin configuration for tests and I know it shouldn't be required.
The problem I'm encountering is that the hibernate-jpamodelgen dependency is not added to the compiler classpath so the annotation processor is not found and the build fails.
As per this answer, I tried adding the dependency as a build extension (not sure I understand what those are supposed to be!) like so:
<extensions>
<extension>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-jpamodelgen</artifactId>
<version>${hibernate-jpamodelgen.version}</version>
</extension>
</extensions>
This also doesn't add hibernate-jpamodelgen to the compiler classpath.
The only thing I found which works so far is adding the dependency to the project in the <dependencies> section. This has the unfortunate side-effect of adding hibernate-jpamodelgen as a transitive dependency afterwards which I want to avoid.
My previous working setup uses the maven-processor-plugin plugin to achieve what I want. However, this plugin is not supported by eclipse m2e and the latest version of the maven-compiler-plugin now handles multiple compiler arguments properly so I'd rather use the latter.
The annotationProcessorPaths option can be used in recent versions of the Maven compiler plug-in:
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.1</version>
<configuration>
<annotationProcessorPaths>
<annotationProcessorPath>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-jpamodelgen</artifactId>
<version>5.2.6.Final</version>
</annotationProcessorPath>
</annotationProcessorPaths>
</configuration>
</plugin>
</plugins>
</pluginManagement>
That way the processor is separated from the actual project dependencies. This option is also picked up by the Eclipse M2E plug-in if annotation processing is enabled for the project.
Add the dependency as an optional dependency (<optional>true</optional>). This will add the dependency under compilation, but will prevent it for being a transitive dependency:
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-jpamodelgen</artifactId>
<version>${hibernate-jpamodelgen.version}</version>
<optional>true</optional>
</dependency>
If you're creating an artifact in this module with all your dependencies in it (like a .war), you may use the <scope>provided</scope> instead. This both prevents the dependency to be transitive and to be included in the artifact the module produces.
For JDK 10 I really had to go a bit crazy to get it to work, Hoping someone finds this useful
<jaxb.version>2.3.0</jaxb.version>
<maven.hibernate.version>5.3.2.Final</maven.hibernate.version>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven.compiler.version}</version>
<goals>
<goal>compile</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/generated-sources/annotations</outputDirectory>
<annotationProcessorPaths>
<annotationProcessorPath>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-jpamodelgen</artifactId>
<version>${maven.hibernate.version}</version>
</annotationProcessorPath>
<annotationProcessorPath>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>${jaxb.version}</version>
</annotationProcessorPath>
</annotationProcessorPaths>
<annotationProcessors>
<annotationProcessor>org.hibernate.jpamodelgen.JPAMetaModelEntityProcessor</annotationProcessor>
</annotationProcessors>
<compilerArgs>
<arg>-AaddGeneratedAnnotation=false</arg>
</compilerArgs>
<compilerArguments>
<AaddGeneratedAnnotation>false</AaddGeneratedAnnotation>
<Adebug>true</Adebug>
</compilerArguments>
<failOnError>true</failOnError>
</configuration>
<dependencies>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-jpamodelgen</artifactId>
<version>${maven.hibernate.version}</version>
<optional>true</optional>
</dependency>
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>${jaxb.version}</version>
<type>jar</type>
</dependency>
</dependencies>
</plugin>
The problem is really in 3.* version of the maven-compiler-plugin. It acts a bit different from the 2.* version. In particular, it seems that maven-compiler-plugin3.* doesn't add its dependencies and build extensions to the classpath because it uses javax.tools instruments for running compile process. To get back the old behavior for maven-compiler-plugin you should use a new configuration property forceJavacCompilerUse:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<forceJavacCompilerUse>true</forceJavacCompilerUse>
</configuration>
....
</plugin>
Please take a look jpa-metamodels-with-maven
For further visitors, I found that there are some significant changes in maven-compiler-plugin 3.x series.
This is how I do this. (I'm the one who you linked)
The point is that my solution does not work with those 3.x series of maven-compiler-plugin.
<project ...>
<build>
<extensions>
<extension>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-jpamodelgen</artifactId>
<version>1.3.0.Final</version>
</extension>
</extensions>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.5.1</version> <!-- See this? -->
</plugin>
</plugins>
</build>
</project>
Not sure what kind of build error you got, but here is my case:
I got the following compile error in Idea:
Annotation processor 'org.hibernate.jpamodelgen.JPAMetaModelEntityProcessor' not found error
But, when compiled from Maven, it was all fine.
So, the problem of mine was that somehow I got wrong configuration in Idea settings. Particularly, it appeared that Idea somehow detected the processor and put in into the settings of module processor profiles. It is discussed here.
I fixed it as the following:
Go to Idea > Settings > Annotation Processors.
For each processor profile make sure that:
Enable annotation processing is Yes;
There is no annotation processor FQ name of one you have error about (e.i. "JPAMetaModelEntityProcessor") in the list on the right side. If it is listed there, just select and click '-' minus button to remove it.
I think this is a better way to contain such dependencies in profiles to solve such problems.
<profile>
<id>XXX-profile</id>
<dependencies>
<dependency>
// XXX artifact path
</dependency>
</dependencies>
</profile>