Spark related jars cannot be resolved in Eclipse - java

I'm new to Spark so am trying to setup a project from the book Learning Spark: Lightning-Fast Big Data Analysis. The book uses version 1.3 but I've only got 2.1.1 so am trying to work around a few differences.
All the Spark related jars that I'm importing into my Java project have a "import org.apache cannot be resolved". I know it's because the project cannot find the jar files specified.
I can manually add each by going to Build Path > Configure Build path and adding them to the Libraries section but I think I shouldn't need to do this. The project uses Maven so I believe if I have the Spark dependencies configured correctly in my pom.xml it should work. Is this correct?
I also set the following environment variables:
export SPARK_HOME=/Users/mymac/spark-2.1.1-bin-hadoop2.7/
export PATH="$SPARK_HOME/bin/:$PATH"
Are there any others I should be aware of?
Here's the contents of my pom.xml:
<project>
<groupId>com.oreilly.learningsparkexamples.mini</groupId>
<artifactId>learning-spark-mini-example</artifactId>
<modelVersion>4.0.0</modelVersion>
<name>example</name>
<packaging>jar</packaging>
<version>0.0.1</version>
<dependencies>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.1.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.8</version>
</dependency>
</dependencies>
<properties>
<java.version>1.8</java.version>
</properties>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>

This should be setup as a Maven project, not a Java project. In my case to resolve deleted the project from my workspace, re-created it in the workspace as a general project, then converted it to a Maven project. I probably should have just set it up as a Maven project at the start.

Related

maven dependency not properly compile into war

I am using the Eclipse IDE and in the past, I have been developing a maven project depending on local sources (from github (sap-olingo)) as well as maven dependencies loaded as jar. I have now updated the github source and also all the dependencies which seems to break standard eclipse export functionalities (right click porject -> export -> war)
The webproject is working fine when using the maven install functionality and is loading all sources into the META-INF/lib classes, but when using the export as war from eclipse (which I did previously) it is missing to compile a proper dependency. Missing in the sense of that the jar is still present in the META-INF/lib directory of the war, but the content is empty and the size is only 1kb
When comparing the mvn clean install to the exported, there are rarely any differences. Except for the projects that are to be compiled during the runtime. Sizes of the libs in the mvn created war and the exported war roughly matches except for that one jar.
Is there a way to root cause that issue? I am using solely the maven integrated into the IDE which is why I doubt differences in the maven versions and or jdk.
Unfortunately I have no idea in where to start and what info you require
Thanks for your input
EDIT:
My projects pom:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.test</groupId>
<artifactId>odata</artifactId>
<packaging>war</packaging>
<version>4.0.0</version>
<name>odataMaven Webapp</name>
<url></url>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.sap.olingo</groupId>
<artifactId>odata-jpa-processor</artifactId>
<version>0.3.7-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>mssql-jdbc</artifactId>
<version>6.1.0.jre8</version>
</dependency>
<dependency>
<groupId>org.eclipse.persistence</groupId>
<artifactId>javax.persistence</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>2.5</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.eclipse.persistence</groupId>
<artifactId>eclipselink</artifactId>
<version>2.6.2</version>
</dependency>
</dependencies>
<build>
<finalName>odata</finalName>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>${project.build.source}</source>
<target>${project.build.source}</target>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.build.source>1.8</project.build.source>
</properties>
</project>
The odata-jpa-processor source/project is pulled from here. The problem exists solely with the olingo-jpa-metadata jar which is empty. All other maven dependencies are exported correctly.

Maven pom file, what is it doing?

I have just started using Maven, in a newbie capacity, just want to understand something around dependencies.
I am trying to build a micro web service using iText and the pdf output functionality.
So my very first steps is seeing if I can get a pdf output from a very simple Java program.
In my pom file i have the following dependencies:
<!-- iText Core -->
<dependency>
<groupId>com.itextpdf</groupId>
<artifactId>itext7-core</artifactId>
<version>${itext.version}</version>
<type>pom</type>
</dependency>
<!-- iText pdfHTML add-on -->
<dependency>
<groupId>com.itextpdf</groupId>
<artifactId>html2pdf</artifactId>
<version>2.1.6</version>
</dependency>
After reading the information on the Maven site, the pom file should do all of the heavy lifting in getting the dependencies, this is the bit i'm a little confused on.
Will the pom file physically download the files to the the app location on application start so that he app can utilize these files?
if that's the case it doesn't seem to be doing this and so am I missing something in the pom file to enable this?
The full pom file is:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany.app</groupId>
<artifactId>my-app</artifactId>
<version>1.0-SNAPSHOT</version>
<name>my-app</name>
<!-- FIXME change it to the project's website -->
<url>http://www.example.com</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<itext.version>RELEASE</itext.version>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<!-- iText Core -->
<dependency>
<groupId>com.itextpdf</groupId>
<artifactId>itext7-core</artifactId>
<version>${itext.version}</version>
<type>pom</type>
</dependency>
<!-- iText pdfHTML add-on -->
<dependency>
<groupId>com.itextpdf</groupId>
<artifactId>html2pdf</artifactId>
<version>2.1.6</version>
</dependency>
</dependencies>
<build>
<pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
<plugins>
<!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.1.0</version>
</plugin>
<!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.1</version>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
</plugin>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
</plugin>
<!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
<plugin>
<artifactId>maven-site-plugin</artifactId>
<version>3.7.1</version>
</plugin>
<plugin>
<artifactId>maven-project-info-reports-plugin</artifactId>
<version>3.0.0</version>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
Any help appreciated.
Thanks
When maven build is executed, Maven automatically downloads all the dependency jars into the local repository.
the local repository of Maven is a folder location on the developer's machine, where all the project artifacts are stored locally.
Usually this folder is named .m2.
Here's where the default path to this folder is – based on OS:
Windows: C:\Users\User_Name\ .m2
Linux: /home/User_Name/.m2
Mac: /Users/user_name/.m2
https://www.baeldung.com/maven-local-repository
Maven does download the dependencies to the local m2 repository. But this is more meant for building the application, not for running.
What you want (copy the dependencies next to the output jar) can be achieved with the goal dependency:copy-dependencies
See this blog post:
https://technology.amis.nl/2017/02/09/download-all-directly-and-indirectly-required-jar-files-using-maven-install-dependencycopy-dependencies/
Managing dependencies is one of the key features of Maven.
Dependency management: It is possible to define dependencies to other
projects. During the build, the Maven build system resolves the
dependencies and it also builds the dependent projects if needed.
Resolving dependencies does mean it downloads all the specified jars in the local system.
The Maven tooling reads the pom file and resolves the dependencies of
the project. Maven validates if required components are available in a
local repository. The local repository is found in the .m2/repository
folder of the users home directory.
Note that .m2/ is a hidden folder. If you are using Linux, would be this path /home/someuser/.m2
Read this
If however its not downloading the jars or creating the .m2 directory at all, then either you are not building the project right or you are not connected to the internet.

JDK sources in Eclipse using maven

I have some Maven projects that I develop under Eclipse. When I want to see the source of a third party class, if the sources were published, I can simply read them:
However when I try to do the same with any of the default classes, those that are part of the JDK, I see a "Source not found" screen:
One of my pom.xml contains this:
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-collections4</artifactId>
<version>4.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.3</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
I was looking for a way to put the JDK in as a dependency, but I can't find a clue, how. I'm not even sure, if that would solve the problem, because my project compiles fine. I also am also aware that I can simply attach the sources through Eclipse, but again I'm not sure, if that would be a nice solution, as Maven can download the other sources.
Do you have any idea what could cause this, and how to solve the problem?

Errors deploying Java + Groovy mixed code on Heroku

I'm trying to deploy a project on Heroku but I'm getting Maven compilaton errors like this:
remote: [ERROR] /tmp/build_5d64555c50abcb9638e3ef5b331a0107/src/main/java/com/davioooh/myapp/services/TestService.java:[3,43] package com.davioooh.myapp.domain does not exist
In my project I'm using both Java and Groovy classes. All Groovy classes are in com.davioooh.myapp.domain that can't be found during compilation.
I also tried to move all Groovy classe in src/main/groovy folder but it's still not working...
Is there a way to correctly deploy my application?
I finally solved adding Groovy Eclipse Maven Plugin as compiler plugin in my project pom.xml.
<build>
...
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<!-- 2.8.0-01 and later require maven-compiler-plugin 3.1 or higher -->
<version>3.1</version>
<configuration>
<compilerId>groovy-eclipse-compiler</compilerId>
</configuration>
<dependencies>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-compiler</artifactId>
<version>2.9.1-01</version>
</dependency>
<!-- for 2.8.0-01 and later you must have an explicit dependency on groovy-eclipse-batch -->
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-eclipse-batch</artifactId>
<version>2.3.7-01</version>
</dependency>
</dependencies>
</plugin>
</plugins>
...
</build>

maven 3 interproject depedency with war packaging

I have Eclipse Indigo and M2E plugin installed.
So essentially I have a standard maven web project (let's call it proj-service) that is built into a war file in the package phase. This all works fine. My issue comes in when I have my other project (lets call it proj1) that needs to use classes from proj-service. I know that this is possible in maven+eclipse but it does not seem to be working at the moment. I have the following in proj1's pom right now:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany.foo</groupId>
<artifactId>proj1</artifactId>
<packaging>war</packaging>
<version>1.0</version>
<name>proj1</name>
<properties>
<spring.version>3.1.0.RELEASE</spring.version>
</properties>
<dependencies>
<!-- Maven Repo Libraries -->
.........
<!-- Interproject dependencies -->
<dependency>
<groupId>com.mycompany.foo</groupId>
<artifactId>proj-service</artifactId>
<version>1.0</version>
<type>war</type>
</dependency>
</dependencies>
<build>
<finalName>lsoap</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
Unfortunately with Maven's war packaging you can't reuse classes from war project, because there is no direct build artifact you can use for the class path.
So, in order to do share classes properly you need to extract those common classes into a 3rd common project (jar packaging) and make it as dependency in both of your other projects.
First you have to change the configuration of your proj-service project in the way to change the configuration of the maven-war-plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<configuration>
<attachClasses>true</attachClasses>
<archiveClasses>true</archiveClasses>
...
</configuration>
</plugin>
This will it make possible to use the classes from the proj-service project in other projects via the following dependencies:
<dependency>
<groupId>myGroup</groupId>
<artifactId>myArtifact</artifactId>
<version>myVersion</myVersion>
<classifier>classes</classifier>
</dependency>
This will result in changing your dependency from:
<dependency>
<groupId>com.mycompany.foo</groupId>
<artifactId>proj-service</artifactId>
<version>1.0</version>
<type>war</type>
</dependency>
into:
<dependency>
<groupId>com.mycompany.foo</groupId>
<artifactId>proj-service</artifactId>
<version>1.0</version>
<classifier>classes</classifier/>
</dependency>

Categories

Resources