I am new with Maven and Spark and I would like to play a bit with both of them.
I am on OSx so I've installed both using brew.
On eclipse I created a Maven project with the quick wizard and created the following pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.and.app</groupId>
<artifactId>myapp</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.sparkjava</groupId>
<artifactId>spark-core</artifactId>
<version>2.4.3</version>
</dependency>
</dependencies>
</project>
while the main class is:
package myapp;
import static spark.Spark.*;
public class Driver {
public static void main(String[] args) {
get("/hello", (req, res) -> "Hello World"); }
}
But I get the following error:
Missing artifact com.sparkjava:spark-core:jar:2.4.3
I thought it was because of the brew installation and so I've add the line:
<systemPath>\usr\local\Cellar\apache-spark\2.4.3\libexec\jars</systemPath>
as an explicit path for spark jars just after the version tag in the pom.xml
Until now I have no success.
What am I doing wrong?
P.S.: I verified Maven and Spark installation.
Maven is getting the dependencies automatically from the network. If you want it to use locally installed jars, you have to add them to your local maven repository:
https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
But I am wondering: You could also tell maven to get the spark dependency itself. It seems to be available for maven users:
https://mvnrepository.com/artifact/org.apache.spark/spark-core
So the dependency should be:
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>2.4.3</version>
</dependency>
Check the definition of the system scope:
This scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository.
It means that you should provide the path to jar for each launch of your code. And for compilation as well (because of import). Change the scope or provide the path to the jar for compiler explicitly.
Related
I'm just starting to learn Java.
I am using IntelliJ to create a SparkJava (see: sparkjava.com) app using openjdk 17. I followed the tutorial instructions outlined in "Setting up Spark with Maven" (https://sparkjava.com/tutorials/maven-setup). I believe the instructions are very outdated because they did not work. After some googling, I finally just arrived at the following code and POM.xml. When I build the project, I get an error: java: package spark does not exist
I don't know what to do.
I added the dependency to my POM.xml.
I added the apache.spark.core_2.13 library via Project Structure.
I googled "IntelliJ package does not exist" but couldn't find a helpful answer. Most everyone said to "add the dependency to the POM" but I've already done that.
I googled "add package to Java project IntelliJ" but after clicking a number of linkis, I couldn't find a helpful answer. I tried a few of the suggestions, but none resolved this problem.
I think I am missing something fundamental here, like somehow I'm not telling IntelliJ where to find the spark code.
src/main/java/Sparky.java
import static spark.Spark.*;
public class Sparky {
public static void main(String[] args){
get("/hello", (req, res) -> "Hello World");
}
}
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>sparky</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>17</maven.compiler.source>
<maven.compiler.target>17</maven.compiler.target>
</properties>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.13</artifactId>
<version>3.2.0</version>
</dependency>
</dependencies>
I guess you didn't configure IntelliJ for Maven properly. Either your project is not a Maven project (as far as IntelliJ is concerned, even if the structure of the project is valid for Maven), or you're missing an IntelliJ plugin or something. Look at IntelliJ's doc on how to create and manage a Maven project.
The dependency package was completely wrong. It was pointing at Apache Spark, not Spark Java.
<dependency>
<groupId>com.sparkjava</groupId>
<artifactId>spark-core</artifactId>
<version>2.9.3</version>
</dependency>
https://github.com/bvanalderweireldt/concurrent-unique-queue
I have tried to set up a Maven dependency within IntelliJ, but I am not sure how the contents of this repository should be built and imported into a Java project. Could someone with more experience please advise on how this is done?
Kind regards,
L
If you want to use this project in another project, you will create a dependency to this using the dependency entry mentioned on the github readme:
<dependency>
<groupId>com.hybhub</groupId>
<artifactId>concurrent-util</artifactId>
<version>0.1</version>
</dependency>
For this, you need the artifact in your local maven repository*. For this, you need to build this project or use a reference from Maven Central (Thanks #Mark Rotteveel )
Clone the project locally, you need to build it in one of the following ways
Build it from the command line: Navigate to the project's location in your shell (bash or cmd) and run mvn install
This will build the project and add the artifact (jar) to the local .m2 repository.
Import to Intellij Idea (File -> New -> From Existing Sources). Once imported, build this project from the "Maven Projects" view.
Once you have done this, you can use this in other projects using the <dependency> entries
*For production ready apps, you may want to have a common maven repository for team your like Nexus or Artifactory and use that to maintain artifacts. You would also have a build system like Jenkins.
In the link you gave it had the dependency Maven entry for that library.
<dependency>
<groupId>com.hybhub</groupId>
<artifactId>concurrent-util</artifactId>
<version>0.1</version>
</dependency>
That entry would need to be nested into you <dependencies> tag. Like the example below.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany.app</groupId>
<artifactId>my-app</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>Maven Quick Start Archetype</name>
<url>http://maven.apache.org</url>
<dependencies>
<dependency>
<groupId>com.hybhub</groupId>
<artifactId>concurrent-util</artifactId>
<version>0.1</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.2</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
If I wanted to use just the normal git via the command line and not the one in IntelliJ, what do I need to include in the version control so when I download it, I can get the Maven libraries without manually installing them?
Edit: There is no pom.xml file when the libraries are added to an IntelliJ project, so I was wondering what I need to include so Maven inside IntelliJ can download the libraries.
what do I need to include in the version control so when I download it, I can get the Maven libraries without manually installing them?
The pom.xml file does this:
Some of the configuration that can be specified in the POM are the project dependencies, the plugins or goals that can be executed, the build profiles, and so on. Other information such as the project version, description, developers, mailing lists and such can also be specified.
Running mvn install will cause Maven to download your dependencies.
Intellij will automatically understand the changes in the pom files and update libraries of course you should have pom.xml file.
If its a maven based project, you definitely need a pom.xml file like below
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd"
<modelVersion>4.0.0</modelVersion>
<groupId>com.test</groupId>
<artifactId>testing</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>testing</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Once you have a pom.xml file, you can define dependencies like the JUnit dependency defined above with the version you need and maven will automatically take care of downloading the dependency for the project. Once you add any new dependency to the pom.xml, you can run "mvn clean install" from the directory where you have the pom.xml file so that it installs the new dependency.
Hope this helps.
This is my pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>cassandra</groupId>
<artifactId>connector</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>cassandra</name>
<description>cassandra connector</description>
<dependencies>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>2.1.5</version>
</dependency>
</dependencies>
</project>
Error is shown at line 12.
I have copied and pasted the maven dependency from here.
http://mvnrepository.com/artifact/com.datastax.cassandra/cassandra-driver-core/2.1.5
One way to get rid of this error is by downloading bundle file on your local machine from below URL & try below mvn:install command to update it on your local repository.
http://central.maven.org/maven2/com/datastax/cassandra/cassandra-driver-core/2.1.5/cassandra-driver-core-2.1.5.jar
mvn install:install-file -Dfile="cassandra-driver-core-2.1.5.jar" -DgroupId=com.datastax.cassandra -DartifactId=cassandra-driver-core -Dversion=2.1.5 -Dpackaging=jar
Or other way, if you are connecting to Maven repository like Nexus & having rights to update your repository then you can deploy your bundle to Nexus using below mvn:deploy command.
mvn deploy:deploy-file -Durl=http://<>:<>/nexus/content/groups/public -DrepositoryId=nexuspublic -Dfile="cassandra-driver-core-2.1.5.jar" -DgroupId=com.datastax.cassandra -DartifactId=cassandra-driver-core -Dversion=2.1.5 -Dpackaging=jar
In my case I figured out two issues causing this problem, firstly older versions usually throws such errors, so make sure to use the latest version of the dependency. Secondly, using nexus instead of maven repository fixed my issue and it does not say "missing artifact" anymore, I figured this out when I saw a note in https://mvnrepository.com/artifact/pentaho-kettle/kettle-engine/9.3.0.0-115
indicating that the artifact is actually in nexus repository. Lastly, try to be consistent with your versions, for instance, if your using multiple dependencies for the same bundle, stick with the same version as they are most compatible with each other I believe (preferably latest version).
I have installed maven and I created a project using this command:
mvn archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false
The result is there are 2 folder and 1 file created in my-app folder: src, target, and pom.xml.
Then I modify the pom.xml in order to get the all of required apache POI jars.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany.my-app</groupId>
<artifactId>my-app</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>my-app</name>
<url>http://maven.apache.org</url>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<!-- This is what I added -->
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>3.10-FINAL</version>
</dependency>
</dependencies>
</project>
Then I run:
mvn package
but no jars downloaded into the project folder although I got message "BUILD SUCCESS".
What you've done is correct. However, the poi jars won't download to your project folder but to your local Maven repository. This is exactly what Maven is supposed to do so that you don't have to manage many libraries/jars yourself and get all in a mess. If you do a search of your local Maven repository, you should find it there.
I also suggest you read up on how Maven uses external dependencies, this is all explained here:
http://maven.apache.org/guides/getting-started/index.html#How_do_I_use_external_dependencies
If you want to package up all of your dependent jars in to one big jar look here:
How can I create an executable JAR with dependencies using Maven?
Your project has a jar packaging type. Java not support nested jar and then maven package doesn't put any jar in your project . To do this you have to use Maven Assembly Plugin or use Spring-boot to make your uber jar