apache storm in cluster mode - java

i used apache storm to precessing data with kafka source, but where i run storm in cluster mode he return for me this erreur :
i user this commande line storm jar /path to my jar file args1
Exception in thread "main" java.lang.RuntimeException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar.
at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:106)
at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:126)
at backtype.storm.utils.Utils.readStormConfig(Utils.java:146)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:45)
at com.storm.Topologie.main(Topologie.java:48)
this my file dependencies in pom.xml:
<dependencies>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>0.9.5</version>
<!-- keep storm out of the jar-with-dependencies -->
<scope>provide</scope>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-kafka</artifactId>
<version>0.10.0</version>
<type>jar</type>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.9.2</artifactId>
<version>0.8.1.1</version>
<exclusions>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
and the last part of my pom.xml
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.storm.Topologie</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>

<scope>provide</scope>
That "scope" isn't valid. The name is 'provided':
<scope>provided</scope>
Is your maven job successfully running with that pom.xml? Check, it's also likely that you're trying to publish the artifact (fat jar) from previously successful builds. Try executing the clean goal too.

Related

Cant access Spring health endpoint

I can't access the actuator or swagger doc endpoints when I compile my app and run the jar from my target folder. But, I am able to access other endpoints with my app.
I can access these endpoints when I run my app directly from Intellj.
I think there must be some sort of problem with how I'm packaging my project in maven. I've included the pom.xml below.
application.properties
# Server
server.port = 9000
server.servlet.context-path = /api
# App
app.service_name = my-api
app.batch_limit = 100
# Health
management.endpoints.web.exposure.include = health,info
management.endpoint.health.show-details = ALWAYS
management.endpoints.web.base-path = /
management.endpoints.web.path-mapping.health = /_health
management.health.neo4j.enabled = false
DockerFile
FROM openjdk:17-alpine
RUN apk --no-cache add curl
WORKDIR /app
COPY ./my-api.jar /app/
COPY ./application.properties /app/config
EXPOSE 9000
CMD sleep 30 ; exec java $JAVA_OPTS -jar ./my-api.jar
pom.xml
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<version>${spring-boot.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
<version>${spring-boot.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
<version>${spring-boot.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
<version>${spring-boot.version}</version>
</dependency>
<dependency>
<groupId>io.swagger.core.v3</groupId>
<artifactId>swagger-annotations</artifactId>
<version>${swagger.version}</version>
</dependency>
<dependency>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
<version>${spring-doc.version}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven-compiler.version}</version>
<configuration>
<release>17</release>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>${maven-jar.version}</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>src</classpathPrefix>
<mainClass>com.api.myApp</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>${maven-assembly.version}</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>src</classpathPrefix>
<mainClass>com.api.myApp</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
From what you've posted, I assume that you deploy your application to a local docker container. You specify the health endpoint to be on port 9000 of the container. Which is not, by default, the same as port 9000 of your machine (a.k.a. localhost).
From the dockerfile reference documentation:
The EXPOSE instruction informs Docker that the container listens on the specified network ports at runtime. You can specify whether the port listens on TCP or UDP, and the default is TCP if the protocol is not specified.
The EXPOSE instruction does not actually publish the port. It functions as a type of documentation between the person who builds the image and the person who runs the container, about which ports are intended to be published. To actually publish the port when running the container, use the -p flag on docker run to publish and map one or more ports, or the -P flag to publish all exposed ports and map them to high-order ports.
(You could, in theory, run multiple containers on the same machine that all specify internally to listen to port 9000. On container startup, you would then map those container ports to different ports of your machine.)
When you run the application in Intellij, port 9000 there actually means port 9000 of your machine.
My project seems to build correctly when using the spring-boot-maven-plugin instead of maven-assembly-plugin. I can now access health and swagger endpoints.
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>${spring-boot-maven.version}</version>
<configuration>
<mainClass>com.api.myApp</mainClass>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>

Error java.lang.NoClassDefFound with maven

Hello everyone and thank you in advance for you help. I'm a student and learning java. For this semester we have a group project were we need to complete a code with already some functions implemented. My main task was to implement an algorithm in the class assign from the directory assignment mainly using functions from class in the directory data model. Every class though in different directorie is in the same package. When calling static fields from data model classes in assign I encounter no problem. However when calling a method here is what I get error ClassDefNotFound. I shall add that I run the program with a .jar file. I tried a LOT of things to solve the problem but unfortunately I haven't solved it yet and I'm starting to run out for time so here I am looking for your help.
I tried to change the pom.xml file from assignment directory by adding in the configuration however it's still not working. Besides I'm confused by which path should I exactly write down. I know it as to be an absolute path but I can't writ my user directory because the project is supposed to work on any computer.
Thanks again for you future help. I've been pulling my hairs out on this one thing for ever so any help solving this would be a life saver !
Think you are missing the dependencies when generating the jar. thats why it says that the class isn't defined, try this in your pom.xml
source:
Including dependencies in a jar with Maven
<build>
<plugins>
<!-- any other plugins -->
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
Personal pom.xml example (notice at the begining the use of the above plugin and also I defined a manifest, try both ways:
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>NuevaUtilidadExcel</groupId>
<artifactId>NuevaUtilidadExcel</artifactId>
<version>0.0.1-SNAPSHOT</version>
<build>
<sourceDirectory>src</sourceDirectory>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.nuevaUtilidad.vista.InterfazPrincipal</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<!-- this is used for inheritance merges -->
<phase>package</phase>
<!-- bind to the packaging phase -->
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.7.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>unknown-jars-temp-repo</id>
<name>A temporary repository created by NetBeans for libraries and jars it could not identify. Please replace the dependencies in this repository with correct ones and delete this repository.</name>
<url>file:${project.basedir}/lib</url>
</repository>
<repository>
<id>netbeans</id>
<name>Netbeans rep</name>
<url>http://bits.netbeans.org/maven2/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi</artifactId>
<version>4.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
<version>4.1.0</version>
</dependency>
<dependency>
<groupId>com.opencsv</groupId>
<artifactId>opencsv</artifactId>
<version>4.0</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.9.6</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.9</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>org.apache.xmlbeans</groupId>
<artifactId>xmlbeans</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>11.2.0</version>
</dependency>
<dependency>
<groupId>com.toedter</groupId>
<artifactId>jcalendar</artifactId>
<version>1.4</version>
</dependency>
</dependencies>
<name>NuevaUtilidadExcelSVG</name>
</project>

unable to find logger class while running a maven build jar file

I am trying to write a maven integrated Java API. I have included log4j for logging purpose. Which works well when running through eclipse, but when maven package is done and the jar is run it is unable to run from cmd line using java -jar jar_name.jar throwing an error
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Logger
Now the log4j.properties file is placed under src/main/resources folder. And the pom.xml is mentioned. Have tried searching for answers but none worked for me.
Any help available
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>Weather_Simulator</groupId>
<artifactId>weather_simulator</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>weather_simulator</name>
<url>http://maven.apache.org</url>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<mainClass>com.test.weather.simulator.MainClass</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<!-- NOTE: We don't need a groupId specification because the group is
org.apache.maven.plugins ...which is assumed by default.
-->
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.1</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-configuration2</artifactId>
<version>2.1.1</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.6.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
Your <scope> in your pom.xml appears to be wrong. Try this (notice I've changed "test" to "compile").
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.6.1</version>
<scope>compile</scope> <!-- look here -->
</dependency>
The way you currently have your pom.xml configured (with "test"), maven will only provide the log4j jar when doing "mvn test". If you need the jar at both compile time and run time (which is the scenario that appears to be causing problems for you), the scope needs to be "compile".
Note that "compile" is the default scope, so if you leave the <scope> element off, the scope will be "compile".
From the maven docs: "This [compile] is the default scope, used if none is specified. Compile dependencies are available in all classpaths of a project. Furthermore, those dependencies are propagated to dependent projects"
For more info about maven "scopes" look here.
This is a classpath problem, the jar generated by Maven contains only your classes. To fix this you can pack all the dependencies inside your project jar: How can I create an executable JAR with dependencies using Maven?
There are two things you should be aware if you like to create an executable jar which contains all dependent jars you have to use maven-assembly-plugin as you already did but you forgot to bind it to the life cycle...which looks like this:
<project>
[...]
<build>
[...]
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id> <!-- this is used for inheritance merges -->
<phase>package</phase> <!-- bind to the packaging phase -->
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
[...]
</project>
Furthermore having plugins as dependencies is simply wrong..
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.1</version>
</dependency>
This means remove this entry from your pom file.
Defining a dependency with a scope test means it will be available only during the unit tests which means also it will never being packaged into a resulting jar file. This means you have to change the following:
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.6.1</version>
<scope>test</scope>
</dependency>
into the following:
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
<version>2.6.1</version>
</dependency>
After you have fixed those issues you should be able to build your app via:
mvn clean package
and find the resulting jar file which contains the dependencies in the target directory named like weather_simulator-1.0.0-SNAPSHOT-jar-with-dependencies.jar which you should use to call your app.

Not adding JARs to staging folder on Google Cloud when using Maven

I'm running integration tests on the cloud for the Google Cloud Dataflows that I have written; checking that they read from Pub/Sub and write to BigQuery correctly, but when using Maven (mvn clean install), the staging folder is not populated with the required JARs. The only JAR that appears is a surefirebooter.jar. As a result, I get a NoClassDefFoundError for PipelineOptions (most likely because it is the first class from a dependency that's trying to be referenced) in the Stackdriver logs, and consequently the tests fail. Since they're running on the cloud I am indeed using a DataflowRunner as opposed to a DirectRunner.
When I run the integration tests from my IDE they work fine; the staging folder is populated with all the JARs and all is well. Also, when I run the tests using Maven but with a DirectRunner the tests run successfully, thus my problem only occurs when using Maven and a DataflowRunner. I assume that problem therefore lies with the pom.xml file, which I have given below:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>group</groupId>
<artifactId>artifact</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<appendAssemblyId>false</appendAssemblyId>
<archive>
<manifest>
<addClasspath>true</addClasspath>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>com.google.cloud.dataflow</groupId>
<artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
<version>2.0.0-beta3</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.25</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.25</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>5.0.0-M3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>2.0.2-beta</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Anyone know why this is happening and how I may resolve it?
When staging files, the Dataflow runner will automatically stage the the classes available to the current class loader. I believe that surefire plays some tricks with the classloader to make the tests easier to run.
One option would be to specify filesToStage on the pipeline options, which will override the normal "detect JARs to stage from the class loader". Alternatively, look at how surefire is managing the classpath, and make sure the SDK JARs are available in the classloader the test is running in.

Jasper Reports w/ Maven - How do I specify Java version to compile with?

Is there any way I can specify which version of Java to use when compiling my .jrxml files with Jasper Reports in Maven (using jasperreports-maven-plugin)? I saw this blog post saying claiming that Jasper uses the "default virtual machine set in your computer" and not "same version of the maven-compiler-plugin". If I cannot change or guarantee the JAVA_HOME environment variable, how can I get Jasper to compile with Java6?
Here is a snippet from my pom.xml:
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>jasperreports-maven-plugin</artifactId>
<version>1.0-beta-2</version>
<configuration>
<outputDirectory>${project.build.directory}/classes</outputDirectory>
</configuration>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile-reports</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>net.sf.jasperreports</groupId>
<artifactId>jasperreports</artifactId>
<version>5.0.1</version>
</dependency>
</dependencies>
</plugin>
<plugin>
</plugins>
Looking on the Codehaus docs, there is a parameter you can use, but it doesn't say how to specify which Java version.
Thanks!
According to this issue the folllowing parameters can help you:
<configuration>
...
<maven.compiler.source>1.6</maven.compiler.source>
<maven.compiler.target>1.6</maven.compiler.target>
<compiler>net.sf.jasperreports.engine.design.JRJdtCompiler</compiler>
...
</configuration>
1.0-beta-2, however, does not have these properties, so the later version is necessary. You can either use a snapshot plugin version from here, of build a plugin from source code yourself. As far as I can see, plugin code from trunk supports these parameters.
I had to make some additional configurations:
set eclipse:jdtcore as exclusion in jasperresports;
<dependency>
<groupId>net.sf.jasperreports</groupId>
<artifactId>jasperreports</artifactId>
<version>5.6.0</version>
<exclusions>
<exclusion>
<groupId>eclipse</groupId>
<artifactId>jdtcore</artifactId>
</exclusion>
</exclusions>
</dependency>
set org.eclipse.jdt.core.compiler:ecj as plugin dependency;
jasperreports-maven-plugin:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>jasperreports-maven-plugin</artifactId>
<version>1.0-beta-4-OPENNMS-20160912-1</version>
<configuration>
<outputDirectory>src/main/webapp/WEB-INF/reports</outputDirectory>
<maven.compiler.source>${compileSource}</maven.compiler.source>
<maven.compiler.target>${compileSource}</maven.compiler.target>
<compiler>net.sf.jasperreports.engine.design.JRJdtCompiler</compiler>
</configuration>
<executions>
<execution>
<phase>process-classes</phase>
<goals>
<goal>compile-reports</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.eclipse.jdt.core.compiler</groupId>
<artifactId>ecj</artifactId>
<version>4.4.1</version>
</dependency>
<dependency>
<groupId>net.sf.jasperreports</groupId>
<artifactId>jasperreports</artifactId>
<version>5.6.0</version>
</dependency>
<dependency>
<groupId>net.sf.barcode4j</groupId>
<artifactId>barcode4j</artifactId>
<version>2.0</version>
</dependency>
</dependencies>
Note: the dependencies order of plugin jasperreports-maven-plugin was relevant for me (don't ask me why).

Categories

Resources