Develop Reference

Setting for pom.xml under hadoop mapReduce project

I created a word count project and import with maven the GeoIP by maxmind(maven) - this Geoip project is build with maven.
After I import I have a new project(geoIP) next to word count project with a pom.xml (very long) which was created by eclipse .
But in the above link they are saying add to pom.xml
<dependency>
<groupId>com.maxmind.geoip2</groupId>
<artifactId>geoip2</artifactId>
<version>v2.3.0</version>
</dependency>
I don't get it if eclipse created pom.xml for me should I delete OR add it with the above code ?
How many pom.xml files do I need ?
Does MapReduce project needs to be built with maven as well ?

I haven't worked with GeoIP2 but it shouldn't make a difference here.
if eclipse created pom.xml for me should I delete OR add it with the above code ?
Eclipse created a pom.xml file for your geoip project by default. You should add your new dependency for your file in order to access GeoIP.
How many pom.xml files do I need ?
Really depends on your overall project structure. You should probably take a look at the answers of this SO question and take a peek at this example of multiple maven files for a project.
Does MapReduce project needs to be built with maven as well ?
Nope but I would recommend it, it makes your life easier. Here are my dependencies for Map-Reduce Jobs:
<dependencies>
<!-- junit testing framework -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<!-- hadoop hdfs components -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.1</version>
</dependency>
<!-- hadoop map-reduce components -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.7.1</version>
</dependency>
<!-- hadoop common components -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.1</version>
</dependency>
</dependencies>
With all additional dependencies located in the Maven Repository for Hadoop.

Missing plugins found while loading a transformation on Kettle

I receive this error whenever I run my extraction from the command line, not in the Spoon UI.
Missing plugins found while loading a transformation
Step : MongoDbInput
at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:1200)
at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:643)
at org.pentaho.di.job.Job.execute(Job.java:714)
at org.pentaho.di.job.Job.execute(Job.java:856)
... 4 more
Caused by: org.pentaho.di.core.exception.KettleMissingPluginsException:
Missing plugins found while loading a transformation
My maven dependencies is as follows.
<dependency>
<groupId>rhino</groupId>
<artifactId>js</artifactId>
</dependency>
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-core</artifactId>
</dependency>
<dependency>
<groupId>pentaho-kettle</groupId>
<artifactId>kettle-engine</artifactId>
</dependency>
<dependency>
<groupId>pentaho-library</groupId>
<artifactId>libbase</artifactId>
<version>5.1.0.0-752</version>
</dependency>
<dependency>
<groupId>pentaho</groupId>
<artifactId>pentaho-big-data-plugin</artifactId>
<version>5.1.0.0-751</version>
</dependency>
<dependency>
<groupId>pentaho</groupId>
<artifactId>pentaho-mongodb-plugin</artifactId>
<version>5.1.0.0-751</version>
</dependency>
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>2.11.1</version>
</dependency>
<dependency>
<groupId>pentaho</groupId>
<artifactId>metastore</artifactId>
<version>5.1.0.0-751</version>
</dependency>
<dependency>
<groupId>pentaho-library</groupId>
<artifactId>libformula</artifactId>
</dependency>
<dependency>
<groupId>simple-jndi</groupId>
<artifactId>simple-jndi</artifactId>
</dependency>
<dependency>
<groupId>org.drools</groupId>
<artifactId>drools-compiler</artifactId>
</dependency>
<dependency>
<groupId>org.drools</groupId>
<artifactId>drools-core</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
</dependency>
</dependencies>
Am I missing anything?

before init kettleEnviroment
KettleEnviroment.init();
you should add this code like
StepPluginType.getInstance().getPluginFolders().add(new PluginFolder("your plugins path", false, true));

Having the MongoDB plugin as a dependency only works for compilation. When PDI runs it looks for a plugins/ folder under the current directory, and will load all plugins from there. You can override the location by setting the KETTLE_PLUGIN_BASE_FOLDERS system property. Either way, you'll want a plugins/ folder and to unzip the MongoDB plugin package (not the JAR) into plugins/. That should put a folder called pentaho-mongodb-plugin under plugins/, and if PDI is pointing at that plugins/ folder, it should load and use the plugin successfully at runtime.

You must run Kitchen from Kettle installation directory.
As written here:
http://wiki.pentaho.com/display/EAI/Kitchen+User+Documentation
Please make sure that you are positioned in the Kettle directory
before running the samples below. If you put these scripts into a
batch file or shell script, simply do a change directory to the
installation directory

I had the same error message after an update from version 7 to 8.3. In my case, it was the step 'Abort' and 'Attend' that caused the error. In version 8 these two steps aren't part of the library "kettle-engine-8.3.0.0-371.jar" anymore, they are now part of the library "pdi-core-plugins-impl-8.3.0.0-371.jar" in a new plugin. I had to add the plugin in the maven-dependencies:
<dependency>
<groupId>org.pentaho.di.plugins</groupId>
<artifactId>pdi-core-plugins-impl</artifactId>
<version>8.3.0.0-371</version>
</dependency>
But that was still not everything, I still got the error message. The reason: "To use non-native plugins with an embedded Pentaho Server, you must configure the server to find where the plugins reside" (https://help.hitachivantara.com/Documentation/Pentaho/8.0/Developer_Center/PDI/Embed).
So above the line KettleEnvironment.init() in my java-code, I added this one:
System.setProperty("KETTLE_PLUGIN_CLASSES", "org.pentaho.di.trans.steps.abort.AbortMeta,org.pentaho.di.trans.steps.append.AppendMeta");
KettleEnvironment.init();
This way worked for me. There's also a suggestion using a plugin-folder (see the link or the other answers) and working with KETTLE_PLUGIN_BASE_FOLDER instead.
Unfortunately, the error message doesn't mention which plugin is missing. I deleted and added them (in my local Spoon application) until I found exactly the one that was missing.

Just edit spoon.sh and add
OPT="$OPT -DKETTLE_PLUGIN_BASE_FOLDERS=$BASEDIR/plugins"
right after the line where the OPT variable is set.
This is actually the solution from mattyb. Editing spoon.sh will fix the problem for good.
Another solution is to create a symbolic link in your ${KETTLE_HOME}/.kettle directory which points to the plugins folder.

Cant run feature in Cucumber

Im having issues running a feature in Cucumber, the feature is very basic as it's from a tutorial.
It is not defined and is as follows:
Feature: Proof that my concept works
Scenario: My first test
Given this is my first step
When this is my second step
Then this is my final step
My Cucumber runner class is as follows:
package cucumber;
import org.junit.runner.RunWith;
import cucumber.api.junit.Cucumber;
#RunWith(Cucumber.class)
#Cucumber.Options(
format = {"pretty", "json:target/"},
features = {"src/cucumber/"}
)
public class CucumberRunner {
}
Also the external .jar files that I have in the project are as follows:
The exception that I'm getting is:
Exception in thread "main" cucumber.runtime.CucumberException: Failed
to instantiate public
cucumber.runtime.java.JavaBackend(cucumber.runtime.io.ResourceLoader)
with [cucumber.runtime.io.MultiLoader#75d837b6]
I've tried to look around online for the solution to this problem but have not had any luck.
I've also discussed with the OP of the tutorial and I'm still awaiting feedback but it has been a while.

I ran into a similar issue and got the same error as you did.
Firstly mention the path to the feature file
features = {"src/cucumber/myfile.feature"}
Anyway, that didn't cause the error.
To just run your Cucumber runner class, all the dependencies you need are
cucmber-junit
cucumber-java and
junit.
I had an additional cucumber-guice which was creating the problem and once I removed it, the error went away and runner was executed successfully.
From the link to the image you have mentioned it looks like you are not using cucumber-guice but still I would recommend you remove other unnecessary cucumber dependencies and try again.

1, I ran into this too few days ago, its simple just remove cucumber-Spring from the dependency.
2 If that doesn't work try updating cucumber-core, cucumber-junit, and cucumber-java all version 1.2.3

I believe the issue is that many of the cucumber add-ins, such as cucumber-testng, cucumber-spring, and (in my case) cucumber-guice, expect the corresponding module they link to be included as well. But apparently the cucumber experts decided not to include this dependency in their pom.xml files, so the problem doesn't manifest itself until runtime.
So (to answer Eugene S's question under LING's answer) if you want to actually use guice with cucumber, you need to also add guice itself as a dependency.

This worked for me, I hope it will work for you as well.
Update your Cucumber dependencies in pom.xml
i.e
cucumber-java (1.2.2)
cucumber-jvm (1.2.2)
cucumber-junit (1.2.2)
And update your Junit dependency as well. (4.11).

The only reason for this error is the version of all the cucumber libraries are not same. It should be like this:
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java8</artifactId>
<version>4.2.6</version>
</dependency>
<!-- https://mvnrepository.com/artifact/io.cucumber/cucumber-picocontainer -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-picocontainer</artifactId>
<version>4.2.6</version>
</dependency>
<!-- https://mvnrepository.com/artifact/io.cucumber/cucumber-testng -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-testng</artifactId>
<version>4.2.6</version>
<exclusions>
<exclusion>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</exclusion>
</exclusions>
</dependency>

First Thing : We would request you to use Cucumber v >=4.0.0 as you are using pretty old dependency(v1.2.5) of Cucumber.
Key Point : We shall not mix direct & transitive dependencies specially their versions! Doing so can cause unpredictable outcome.
Solution: Please remove. cucumber-core, cucumber-java, cucumber-jvm-deps, gherkin and cucumber-html. They're transitive dependencies and will be provided by your direct dependencies.
You can add below set of cucumber minimal dependencies.
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit</artifactId>
<version>4.2.6</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-picocontainer</artifactId>
<version>4.2.6</version>
<scope>test</scope>
</dependency>

After spending a lot of time on this issue, most of the errors I was receiving were due to dependencies and dependencies versions mismatch. Adding these dependencies to pom.xml file worked for me:
<!-- https://mvnrepository.com/artifact/junit/junit -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-scala_2.11</artifactId>
<version>4.7.1</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/io.cucumber/cucumber-jvm -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-jvm</artifactId>
<version>4.8.1</version>
<type>pom</type>
</dependency>
<!-- https://mvnrepository.com/artifact/io.cucumber/cucumber-junit -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit</artifactId>
<version>4.8.1</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/io.cucumber/cucumber-java8 -->
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-java8</artifactId>
<version>4.8.1</version>
</dependency>

LocalTransportSender not found when using web services

I am not a big fan of web services, but sometimes you have to conform with a client interface. I have successfully generated code from the the provided WSDLs, but when I try and run the application which actually uses the generated classes, I get the following:
java.lang.ClassNotFoundException: org.apache.axis2.transport.local.LocalTransportSender
I am keeping the generated code in a separate project and have the following dependencies in my pom:
<dependencies>
<dependency>
<groupId>org.apache.axis2</groupId>
<artifactId>axis2-adb</artifactId>
<version>1.6.1</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.axis2</groupId>
<artifactId>axis2-kernel</artifactId>
<version>1.6.1</version>
<scope>compile</scope>
</dependency>
</dependencies>
As stated, the jar gets generated without any issues, but when it is includes in the application that makes use of it, I get the said exception.
Any ideas?

Adding below dependency would probably should solve this problem.
<dependency>
<groupId>org.apache.axis2</groupId>
<artifactId>axis2-transport-local</artifactId>
<version>1.6.2</version>
</dependency>

Use next dependencies , and you should not have any problem
For api axis 2
axis2-adb
For runtime
axis2-transport-local
Laxis2-transport-http

Writing Maven Dependency for javax.persistence

Can someone help me write the dependency for javax.persistence. I have googled it but nothing worked.
I bumped into this page that gives some details on how to write the dependency, but yet i am unable to write it. Can someone help me out?

This is the one for javax.persistence:
<dependency>
<groupId>javax.persistence</groupId>
<artifactId>persistence-api</artifactId>
<version>1.0.2</version>
<scope>provided</scope>
</dependency>
and this is for the whole Java EE 6 stack:
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-api</artifactId>
<version>6.0</version>
<scope>provided</scope>
</dependency>
Edit
Note that I specified a provided scope here, which means that your dependency is available at compile- and test-time, but will not be packaged into your artifacts. This is usually needed if you want to deploy your artifacts in an application server, since they provide their own implementation of the api.

And add this dependency in your pom.xml:
<dependency>
<groupId>javax.persistence</groupId>
<artifactId>persistence-api</artifactId>
<version>1.0.2</version>
</dependency>
That "Coping with Sun JARs" page might be a little outdated, this JAR is available in the Maven Central Repository

Updated link:
https://mvnrepository.com/artifact/javax.persistence/javax.persistence-api/2.2 is here.
and the maven dependency is as below:
<dependency>
<groupId>javax.persistence</groupId>
<artifactId>javax.persistence-api</artifactId>
<version>2.2</version>
</dependency>

For the latest versions javax.persistance is not working instead of that we can use jakarta.persistence to create an entity or resolve the error Cannot resolve symbol 'Entity'. For that need to add the dependency
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>6.1.6.Final</version>
<type>pom</type>
</dependency>