Setting for pom.xml under hadoop mapReduce project - java

I created a word count project and import with maven the GeoIP by maxmind(maven) - this Geoip project is build with maven.
After I import I have a new project(geoIP) next to word count project with a pom.xml (very long) which was created by eclipse .
But in the above link they are saying add to pom.xml
<dependency>
<groupId>com.maxmind.geoip2</groupId>
<artifactId>geoip2</artifactId>
<version>v2.3.0</version>
</dependency>
I don't get it if eclipse created pom.xml for me should I delete OR add it with the above code ?
How many pom.xml files do I need ?
Does MapReduce project needs to be built with maven as well ?

I haven't worked with GeoIP2 but it shouldn't make a difference here.
if eclipse created pom.xml for me should I delete OR add it with the above code ?
Eclipse created a pom.xml file for your geoip project by default. You should add your new dependency for your file in order to access GeoIP.
How many pom.xml files do I need ?
Really depends on your overall project structure. You should probably take a look at the answers of this SO question and take a peek at this example of multiple maven files for a project.
Does MapReduce project needs to be built with maven as well ?
Nope but I would recommend it, it makes your life easier. Here are my dependencies for Map-Reduce Jobs:
<dependencies>
<!-- junit testing framework -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<!-- hadoop hdfs components -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.1</version>
</dependency>
<!-- hadoop map-reduce components -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.7.1</version>
</dependency>
<!-- hadoop common components -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.1</version>
</dependency>
</dependencies>
With all additional dependencies located in the Maven Repository for Hadoop.

Related

Multiple Maven projects in Eclipse not seeing test folders

I am using Eclipse and Maven to build a project that is an excuse to try putting together some technologies I would like to get more experience with, and using ports and adapters (hexagonal) architecture. I am using Cucumber for Java as the testing framework, and I am trying to write the code using TDD & BDD. The project (in the general sense) consists of multiple Eclipse projects set up with the Maven nature.
Here is a list of the relevant Eclipse projects:
pricetracker.data
src/main/java - contains entity POJOs
src/test/java - empty
pricetracker.data.repo - depends on pricetracker.data
src/main/java - data access interface layer that defines the interface and factories for persistence
src/test/java - contains step definitions and step data (currently called StepData)
src/test/resources - contains Cucumber .feature files
pricetracker.data.repo.derby - depends on pricetracker.data.repo
src/main/java
src/test/java - contains a RunCucumberTest class, initializes the repo in StepData
pricetracker.data.repo.memory - depends on pricetracker.data.repo
src/main/java
src/test/java - contains a RunCucumberTest class
I believe I have all of the dependencies pointing correctly. However, the RunCucumberTest classes cannot see the StepData class (The import MyGroupId.pricetracker.data.repo.StepData cannot be resolved). And I cannot tell whether this is an Eclipse problem or a Maven problem.
Here is what the pom.xml looks like in pricetracker.data.repo.derby (replacing my actual groupId with myGroupId):
<dependencies>
<dependency>
<groupId>myGroupId</groupId>
<artifactId>pricetracker.data.repo</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>myGroupId</groupId>
<artifactId>pricetracker.data</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
<version>10.14.2.0</version>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java</artifactId>
<version>${cucumber.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit</artifactId>
<version>${cucumber.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-picocontainer</artifactId>
<version>${cucumber.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
Now if in pricetracker.data.repo I move StepData from the src/test/java folder to the src/main/java folder, it becomes visible from the pricetracker.data.repo.* subpackages RunCucumberTest files. However, this is not production code and should not be under src/main/java.
In Eclipse itself, in each project, the test folders are all marked as "Contains test sources: Yes" and are set to compile into a separate folder from the main classes.
Any pointers are appreciated!

NoClassDefFoundError with EncryptedDocumentException

I am building a java program that opens an excel document. While in the IDE everything works, when i execute the code from the built jar everything falls apart and it gives me the following exception
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/poi/EncryptedDocumentException
at com.exceltogpx.Main.main(Main.java:19)
It stops the execution at the import of the class
import org.apache.poi.EncryptedDocumentException;
and it's driving me crazy
it's a maven build, dependencies imports following
<dependencies>
<!-- https://mvnrepository.com/artifact/org.json/json -->
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20210307</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.13</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.poi/poi -->
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>5.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml -->
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
<version>5.0.0</version>
</dependency>
</dependencies>
I think something may be wrong with either the classpath or the way I create the build with maven, but it seems I can't find the issue itself
Found out the answer to the question
I am a newbie maven user and I had never worked with pom a before. Be sure to specify a goal for your project and to specify that it will be a "jar-with-dependencies"
Here's the guide I used to fix my pom file (I followed the second part of the guide, the one that uses Apache maven assembly plugin)
Other suggestions for others using maven for the first time. If you used any external libraries so you have dependencies, in the log you should see the download of those libs. Otherwise it means you probably did something wrong with the pom... The guide I linked works pretty good and explains what the main tags do.

Spring and Hibernate dependencies are not being resolved

I am having this error message for all the imported packages "The import org..... cannot be resolved". and my Spring and Hibernate dependencies are not being resolved. I have tried multiple solutions to fix this error but nothing seams to work, below are the solution I've tried.
Maven => Update Project but same error. Also did => Force Update of Snapshots/Releases
I have also tried to Right-click on the project and choose Properties, and then Maven. Uncheck the box labelled "Resolve dependencies from Workspace projects", hit Apply, and then OK
Deleted my local Maven repo, the .m2 directory and Restarted Eclipse.
https://i.stack.imgur.com/LCUta.png
'''
<modelVersion>4.0.0</modelVersion>
<groupId>com.luv2code.springdemo</groupId>
<artifactId>spring-crm-rest</artifactId>
<version>1.0.0</version>
<packaging>war</packaging>
<properties>
<springframework.version>5.0.6.RELEASE</springframework.version>
<hibernate.version>5.4.1.Final</hibernate.version>
<mysql.connector.version>5.1.45</mysql.connector.version>
<c3po.version>0.9.5.2</c3po.version>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<!-- Spring -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>${springframework.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-tx</artifactId>
<version>${springframework.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-orm</artifactId>
<version>${springframework.version}</version>
</dependency>
<!-- Add Jackson for JSON converters -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.5</version>
</dependency>
<!-- Hibernate -->
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>${hibernate.version}</version>
</dependency>
<!-- MySQL -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>${mysql.connector.version}</version>
</dependency>
<!-- C3PO -->
<dependency>
<groupId>com.mchange</groupId>
<artifactId>c3p0</artifactId>
<version>${c3po.version}</version>
</dependency>
<!-- Servlet+JSP+JSTL -->
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.1.0</version>
</dependency>
<dependency>
<groupId>javax.servlet.jsp</groupId>
<artifactId>javax.servlet.jsp-api</artifactId>
<version>2.3.1</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
<version>1.2</version>
</dependency>
<!-- to compensate for java 9 not including jaxb -->
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.3.0</version>
</dependency>
</dependencies>
<build>
<finalName>spring-crm-rest</finalName>
<plugins>
<!-- Builds a Web Application Archive (WAR) file from the project output
and its dependencies. -->
<plugin>
<!-- Add Maven coordinates (GAV) for: maven-war-plugin -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>3.2.0</version>
</plugin>
</plugins>
</build>
'''
You need as first step to run from terminal in your project root directory the mvn clean install command. In this way you will "install the package into the local repository, for use as a dependency". After that for sure you need to refresh your workspace and run a mvn update
A major problem is apparent from the picture you provided, but I'm not sure how you got it into this state.
By default, Maven projects will have a source directory at "src/main/java" and "src/test/java". Your source directory appears to be just "src", so it's finding classes in "main/java/com/..." so it wants the packages to start with "src.main.java.com...".
You appear to have overridden the settings in the .classpath file somehow. If you're using Maven in Eclipse, you should let the m2e plugin determine the classpath.
From what I can see, if you remove the "src" source directory and replace it with "src/main/java", it will more likely be able to compile your code.
Update:
To answer your question in the comment (this is too long for a comment):
Neither. I would suggest copying the project to a location outside of your workspace, if it isn't already outside of the workspace. If the project was already located outside of the workspace, now delete the project, but DO NOT delete the contents. If the project was located inside the workspace, as you've now copied it out, you can delete the project. At this point, go to the location where the project lies outside of the workspace, and delete the ".classpath" file from that location. Now, import the project back into Eclipse. If your project is conventional otherwise, this will "reset" it to a sane organization with respect to m2e.

Compile subset of dependencies into my jar file

I have a project with multiple Maven dependencies and want to minimize the size of my compiled JAR. Right now it seems that IntelliJ is compiling all of the external dependency source files into my JAR even though I only use a small subset of their functionalities.
I would like to include only the files that are directly used by my module.
You can declare a dependency as optional:
<dependencies>
<dependency>
<groupId>sample.dependency</groupId>
<artifactId>small-dependency</artifactId>
<version>1.0</version> <!-- Will be packaged in JAR -->
</dependency>
<dependency>
<groupId>sample.dependency</groupId>
<artifactId>really-big-dependency</artifactId>
<version>1.0</version>
<optional>true</optional>
</dependency>
</dependencies>
Another approach is to use the provided scope. The difference is provided is used if you know the dependency will be included in the classpath of the application that will run your JAR (e.g. a Web or Java EE container):
<dependencies>
<dependency>
<groupId>sample.dependency</groupId>
<artifactId>small-dependency</artifactId>
<version>1.0</version> <!-- Will be packaged in JAR -->
</dependency>
<dependency>
<groupId>sample.dependency</groupId>
<artifactId>really-big-dependency</artifactId>
<version>1.0</version>
<scope>provided</scope> <!-- Will not be packaged in JAR, needs to be provided in classpath at runtime -->
</dependency>
</dependencies>
Sources:
Maven - Optional Dependencies and Dependency Exclusions
Maven - Dependency Scope

idea intellij maven project can't make

I have maven project. Maven build complete SUCCESS. But i can't make this project.
[INFO] BUILD SUCCESS
Project contains one module. Idea can't see dependencies.
Error:(3, 38) java: D:\Dropbox\Programming\java\spring\springBook\src\main\java\ch14\validator\ContactTestValidator.java:3:
package org.springframework.stereotype does not exist
POM:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>springBook</groupId>
<artifactId>springBook</artifactId>
<version>1.0-SNAPSHOT</version>
<!-- Shared version number properties -->
<properties>
<org.springframework.version>3.2.3.RELEASE</org.springframework.version>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<!--для валидации бинов необходима-->
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.5.6</version>
</dependency>
<dependency>
<!--Spring Data JPA использует его - добавим и пользуемся-->
<!--API для работы с датами-->
<!--ch14 validator-->
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.0</version>
</dependency>
<dependency>
<!--ch14 validator-->
<!--API интерфейса JSR-303-->
<groupId>javax.validation</groupId>
<artifactId>validation-api</artifactId>
<version>1.0.0.GA</version>
</dependency>
<dependency>
<!--ch14 validator-->
<!--API, которая поддерживает интерфейс JSR-303 - BeanValidation-->
<groupId>org.hibernate</groupId>
<artifactId>hibernate-validator</artifactId>
<version>4.0.2.GA</version>
</dependency>
<dependency>
<!--для валидации. (#Type...)-->
<groupId>org.hibernate</groupId>
<artifactId>hibernate-annotations</artifactId>
<version>3.5.6-Final</version>
</dependency>
<dependency>
<groupId>cglib</groupId>
<artifactId>cglib</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjtools</artifactId>
<version>1.6.2</version>
</dependency>
<!--Spring framework-->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-beans</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>${org.springframework.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aop</artifactId>
<version>${org.springframework.version}</version>
</dependency>
</dependencies>
<build>
<finalName>springBook</finalName>
</build>
</project>
Could you help me? Unfortunately, I can't add screenshots..
There are screenshots:
http://screencast.com/t/iNaWO9gy
http://screencast.com/t/iBGTyMpgH
Project settings - Libraries - empty;
reimport didn't help;
Do the following steps:
Find your pom.xml in your files
Right click on it --> Maven --> Import
Wait for the project to load :D
Press ctrl+alt+shift+s (This opens the Projects window.)
Go to libraries and see if they are empty (they shouldnt)
If they are post your full pom.xml please so I can see the problem
If you still have a problem got to File --> Invalidate Caches
/Restart
Go to maven and Reimport All Maven Projects
Option Maven -> Reimport
should force Intellij to reload all dependencies. Alternatively, as mentioned in comments you can click the Reimport All Maven Projects button in Maven Projects view.
In my case, I had my own apache-maven-3.5.2 folder and a misconfiguration created my problem. Maybe it's your case too:
In rush, I set the local repository to C:/my/folder/apache-maven-3.5.2/settings.xml, but it must be a folder so Maven didn't allow me to download the dependencies. Then I changed the default (in the picture) and it worked.
Open the module dependencies (press F12 on the module), go to Modules -> Dependencies and make sure maven imports are part of the of this.
Try to import the project from sources, if this answer didn't help you.
The following worked for me.
Deleted the maven dependency from my local repository
mvn clean install downloaded the dependency again
File --> Invalidate Caches/Restart
I still saw the compilation error
on my mac, I hit option + enter, intellij asked me to add the jar to classpath
build -> make project
I had faced the same problem, and the issue was small. I had changed the location of settings.xml file of maven and Intellij was picking up the default from C:\users\.m2\settings.xml
So point your maven settings.xml to the correct file as below:
File--> Settings--> Build, Execution, Deployment--> Maven
then on the right window, override "User settings file" to the settings.xml where you have in your local disk.
If you have installed maven in a non-default location then this file you can find in:
\apache-maven-3.2.3\conf\settings.xml
Right click on pom.xml and click Add as Maven Project

Categories

Resources