I am new to Java and i have to execute a code snippet related to Kafka which is given below:
import java.util.*;
import org.apache.kafka.clients.producer.*;
public class Producer {
public static void main(String[] args) throws Exception
{
String topicName = "SimpleProducerTopic";
String key = "Key1";
String value = "Value-1";
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092,localhost:9093");
props.put("key.serializer","org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer <>(props);
ProducerRecord<String, String> record = new ProducerRecord<>(topicName,key,value);
producer.send(record);
producer.close();
System.out.println("SimpleProducer Completed.");
}
}
I have downloaded IntelliJ Idea editor and running the above script there but it is giving me an error
Error:(2, 1) java: package org.apache.kafka.clients.producer does not
exist
I know that i apache kafka is missing so i downloaded a jar file of apache and add it to modules but the error still persist. What should i do? How do install the pacakge?
Simply adding the jar to the corresponding module does not give you access to it. Have you tried right click on the jar Add as Library... option?
Edit: you could perhaps explore other options for external library usage like maven, or gradle.
As being new to Java, you'll want to understand what is the classpath.
Putting JARs directly into your IDE isn't changing that
Even from the command line, you'd need to explicitly specify -cp kafka-clients.jar
There's multiple ways to modify the module classpath in Intellij, but manually downloading JARs should be avoided, and that problem is solved by dependency management tools such as Maven or Gradle (or sbt, etc)
Your profile mentions other languages, so think Nuget, npm, pip, etc. Apply that knowledge to Java
One way to install the package is to use Maven. If you want to configure Maven and IntelliJ, take a look at this tutorial. Eventually, once you are done, you should add this to the auto-generated pom.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>Whatever you put during setup</groupId>
<artifactId>Whatever you put during setup</artifactId>
<version>1.0-SNAPSHOT</version>
//Add this - copy and paste
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>2.1.0</version>
</dependency>
</dependencies>
</project>
You can add any additional packages/dependencies within the <dependencies></dependencies> tags. There are plenty of tutorials online on how to handle dependencies using Maven.
Related
Beginner in implementing Apache Spark in my Java project. I'm using Spark-3.3, and the jars are downloaded from the maven repository. A simple snippet as the following throws an error, I'm very confusing:
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SaveMode;
import org.apache.spark.sql.SparkSession;
public class main{
public static void main(String[] args) {
String in_path = "./test.csv";
String out_path = "./out.csv";
SparkSession spark = SparkSession.builder()
.appName("CSV to Dataset")
.master("local")
.getOrCreate();
Dataset<Row> df = spark.read().format("csv")
.option("header", "true")
.load(in_path);
//write
Dataset<Row> outputDf = df
.filter("confidence_level = 'high'")
.repartition(1);
outputDf
.write()
.format("csv")
.option("header", true)
.mode(SaveMode.Overwrite)
.save(out_path);
}
}
One can make a .csv file to reproduce this error:
,step,value
0,0,0.48335474743993967
1,1,0.1158508331018181
2,2,0.9587111373188968
3,3,0.8701416114549719
4,4,0.1568403204008163
5,5,0.12215751676273201
6,6,0.5040615339539852
7,7,0.5291894043380058
8,8,0.40721487378992893
9,9,0.9284453533942072
10,10,0.8224097122571449
11,11,0.31928057533043286
12,12,0.9255140336657344
The error is this:
java: cannot access scala.collection.immutable.Seq class file for
scala.collection.immutable.Seq not found
Many thanks
Update1:
After including all these jars:
spark-sql_2.13-3.3.0.jar
spark-network-common_2.13-3.3.0.jar
spark-mllib_2.13-3.3.0.jar
spark-core_2.13-3.3.0.jar
spark-catalyst_2.13-3.3.0.jar
slf4j-simple-1.7.36.jar
slf4j-api-1.7.36.jar
scala-library-2.13.8.jar
log4j-core-2.18.0.jar
hadoop-mapreduce-client-jobclient-3.3.4.jar
hadoop-mapreduce-client-core-3.3.4.jar
hadoop-mapreduce-client-common-3.3.4.jar
hadoop-common-3.3.4.jar
hadoop-client-3.3.4.jar
guava-31.1-jre.jar
commons-lang3-3.12.0.jar
commons-configuration2-2.8.0.jar
It still throws:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/thirdparty/com/google/common/collect/Maps
A nightmare, now I cannot find which jar is related to.
I have slightly different version of spark, but I think steps below won't be any different.
Suppose I have spark-3.1.2 installed here: /home/qq/.sdkman/candidates/spark/3.1.2/
Suppose also I have Main.java like yours.
I compiled and run it without errors like this:
javac -cp .:/home/qq/.sdkman/candidates/spark/3.1.2/jars/* Main.java
java -cp .:/home/qq/.sdkman/candidates/spark/3.1.2/jars/* Main
I strongly suggest to use a depedency management tool like Maven. You will sooner or later have to update your Spark version (if only for bugfixes or fixed security flaws). Spark 3.3.0 needs at least 139 different jars (plus optional packages like MLlib or GraphX). Managing the correct versions of 139+ different jars is no fun!
If you cannot use Maven directly in your project, you can at least use this tool to create the list of jars that you have to include:
First create a minimal pom.xml:
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>mygroup</groupId>
<artifactId>simplesparkapp</artifactId>
<version>1</version>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.13</artifactId>
<version>3.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.13</artifactId>
<version>3.3.0</version>
</dependency>
</dependencies>
</project>
Then use Maven's dependency plugin to either
create a list of all required jars: mvn dependency:list (result is here) or
let Maven copy all required jars into a folder on your machine: mvn dependency:copy-dependencies -DoutputDirectory=out
This way you can at least automate parts of your build and deployment road.
While migrating from Oracle JDK 8 to Open JDK 11.0.1 using Eclipse 2018-12 I obviously found another JPMS-related bug making it hard to work with external non-modularized .jar's within a modular java project. I tracked my problem down to the full example below.
The example was derived from the migration process of a real project (using the still non-modularized javax.servlet.api) which caused some headaches. It consists of four maven projects M, N, Y and X making up one java module each, and another maven project that makes up a non-modular java project W. I use maven and the maven-compiler-plugin 3.8.0. My observations are:
Eclipse shows errors in M.java but running class M with default options runs without errors
if I include the artifact w as additional maven dependency in project M, the errors remain
if I rename project Y to project O along with artifact, package name and module-info, no errors are shown
if I remove the requires w in module m, no errors are shown
if I make project W modular by adding a module-info.java, no errors are shown
if I make project W modular by adding Automatic-Module-Name: w in MANIFEST.MF, the error remains
Obviously, re-declaring automated modules like module w in top-level projects seems to cause problems with the built-in Eclipse compiler and does not allow us to work properly with Eclipse (whereas running the project works well). In my opinion, this mismatch is another bug in Eclipse 2018-12 (along with the problems I described in Automatic modules not found in Eclipse 2018-12 when project is opened and Java module not found at runtime even with requires transitive).
My question is: Can someone confirm this as a bug, or is it already known? For us it is a complete show stopper since our project depends on different libraries that are neither modular nor have they the Automatic-Module-Name attribute. And as long the Eclipse bug described in this post exist we are not be able to migrate further to JDK 11.
Sidemark: We don’t want to configure our projects after checking out from SCM to get it running in Eclipse. For us, it was not necessary until now (and that's actually really great when working with Maven and Eclipse, thanks to everyone who made that possible so far!), and I hardly try to avoid to manually configure module paths of our eclipse projects or run configurations.
So, here is the complete and reproducable example:
Project M (modular)
// M.java
package m;
import com.example.n.N;
public class M {
public static void main(String[] args) {
System.out.println("M");
N.main(null);
}
}
// module-info.java
open module m {
requires n;
requires w;
}
// pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mavenexample2</groupId>
<artifactId>m</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.mavenexample2</groupId>
<artifactId>n</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.mavenexample2</groupId>
<artifactId>y</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
</dependencies>
</project>
Project N (modular)
// N.java
package com.example.n;
public class N {
public static void main(String[] args) {
System.out.println("N");
}
}
// module-info.java
open module n {
exports com.example.n;
}
// pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mavenexample2</groupId>
<artifactId>n</artifactId>
<version>0.0.1-SNAPSHOT</version>
</project>
Project Y (modular)
// Y.java
package com.example.y;
public class Y {
public static void main(String[] args) {
System.out.println("Y");
}
}
// module-info.java
open module com.example.y {
exports com.example.y;
requires com.example.x;
}
// pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mavenexample2</groupId>
<artifactId>y</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.mavenexample2</groupId>
<artifactId>x</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
</dependencies>
</project>
Project X (modular)
// X.java
package com.example.x;
public class X {
public static void main(String[] args) {
System.out.println("X");
}
}
// module-info.java
open module com.example.x {
exports com.example.x;
requires w;
}
// pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mavenexample2</groupId>
<artifactId>x</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.mavenexample2</groupId>
<artifactId>w</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
</dependencies>
</project>
Project W (non-modular)
// W.java
package external;
public class W {
public static void main(String[] args) {
System.out.println("W");
}
}
// pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mavenexample2</groupId>
<artifactId>w</artifactId>
<version>0.0.1-SNAPSHOT</version>
</project>
Please do a Maven > Update Projects … > all to get everything in sync after defining the projects or changing your module dependencies. Also, please close also the non-modular project M after doing a mvn clean install since otherwise you would get the error described here: Automatic modules not found in Eclipse 2018-12 when project is opened.
Indeed Eclipse had a bug, which surfaced only when compilation was performed in a very specific order.
Background: In the era of JPMS a package has different content
depending on which module is asking. In the example different modules
see different configurations for package com.example: from some
p.o.v. it contains a child package n in other perspectives it
doesn't. For performance sake, each result of such lookup is cached,
which caused the order dependence: which module first looked up
package com.example decided what contributions to the package were
known.
Curiously, the same JPMS that makes split packages illegal, requires a compiler to treat each parent package with multiple contributing modules as a split package, causing a significant increase in implementation complexity.
(Edited:) The bug has been addressed as Eclipse bug 543765, the fix is available since release 2019-03.
I am using "spring MVC" to build something(project “A”). I use maven to get the jar I need.
But I have a question, if I have a Java file(a separate file), a class in it, how can I use it in project “A” just like the other in pom.xml?
maybe it like this： when i use JSONObject i must edit pom.xml like:
<dependency>
<groupId>net.sf.json-lib<groupId>
.........
</dependency>
I mean i want my java file like JSONObject,i put it in pom file when it want use it in some project:
test.java:
class bc {
public String test() {
return "bc->test";
}
}
pom.xml
<dependency>
<groupId>test<groupId>
.........
</dependency>
controller file (in springMVC)
import ...
#controller
#requestMapping(value="/api")
public class apiController {
#RequestMapping.....................
public #ResponseBody String testMyMavenClass() {
return new bc().test()
}
}
Im not sure about how to use .class in maven pom file but Im sure you can define your own jar file.
First, combine your classes into jar file.
Then, Define it inside pom.xml as external library.
Reference: https://www.tutorialspoint.com/maven/maven_external_dependencies.htm
One more note, you can add below line to make sure maven knows where is external repository:
<repositories>
<repository>
<id>in-project</id>
<name>In Project Repo</name>
<url>file:/${project.basedir}/libs</url>
</repository>
</repositories>
Method 1: Install jar file to local maven repository:
Export your project to jar file, then
http://www.mkyong.com/maven/how-to-include-library-manully-into-maven-local-repository/
Method 2: Use a repository manager. This is more advanced and requires some server skills. I can give you the name & link only, and you should do your research your self: https://www.sonatype.com/download-oss-sonatype
I'm trying to implement the example here
https://github.com/bytedeco/sample-projects/tree/master/opencv-stitching to my android project.
but I get all kinds of different errors...no luck at all.
I want to first make sure I'm referencing the correct lib. this example uses maven and its pom.xml can be found here. https://github.com/bytedeco/sample-projects/blob/master/opencv-stitching/pom.xml
this is a bit more complicated that what I can translate to gradle. I keep getting errors. Can someone please tell me what I should do to reference the same lib as the example? Thanks a lot!!
Since this project is so small, you try doing gradlew --init. It will convert a maven project to a gradle project if it detects it.
Lets take the pom.xml files:
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>org.bytedeco.javacpp-presets.opencv</groupId>
<artifactId>opencv-stitching</artifactId>
<version>0.11</version>
<dependencies>
<dependency>
<groupId>org.bytedeco.javacpp-presets</groupId>
<artifactId>opencv</artifactId>
<version>2.4.11-0.11</version>
</dependency>
</dependencies>
</project>
This is the same as a basic build.gradle:
group = "org.bytedeco.javacpp-presets.opencv"
version = "0.11"
dependencies {
compile "org.bytedeco.javacpp-presets:opencv:2.4.11-0.11"
}
We have a number of different Java Maven projects, each with different versions and each using one or more of the other Maven projects. They are all imported into Eclipse as Maven java projects, with (Maven) Workspace Resolution enabled and 'everything' is dandy except for the Find References functionality which doesn't work when a Maven project version is updated.
Scenarios is as follows:
Sample setup:
Initial Project A pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.deepak.test</groupId>
<artifactId>a-project</artifactId>
<version>1.0.0-SNAPSHOT</version>
</project>
Class in Project A
public ClassA {
public static void someMethodA() {
}
}
Project B pom.xml what uses Project A as a dependency:
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.deepak.test</groupId>
<artifactId>b-project</artifactId>
<version>1.0.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.deepak.test</groupId>
<artifactId>a-project</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
</dependencies>
</project>
Class in project B that uses Class A (import omitted) from Project A
public ClassB {
public static void someMethodB() {
ClassA.someMethodA();
}
}
With this setup, in project A if I right click on someMethodA() class ClassA and select References -> Workspace, Eclipse correctly shows ClassB.someMethodB() the Search View.
However IF I update project A's version tag to say the release version, i.e. change version tag in Project A pom.xml to:
<version>1.0.0</version>
And then if we perform the Find References action again, Eclipse will NOT show ClassB.someMethodB() as a reference. Is there any way we can make Eclipse show this as a potential match (with an older version of the jar perhaps)?
I know that doing a File Search is the only other solution, but that is not as accurate as doing a reference match from class files, and results in duplicate matches with Maven sub-modules etc.
I am using Eclipse 4.5.2 (Mars), with JDK 1.8 and Maven 3.3 if that helps