I am trying to upgrade Java and Scala versions on Bitnami Spark image by updating the Dockerfile:
https://github.com/bitnami/bitnami-docker-spark/blob/master/3.3/debian-11/Dockerfile
Going through the rest of the code it looks like it uses below repository URL to download the installation tar files:
https://downloads.bitnami.com/files/stacksmith
I need to find the correct package name with Java 17 and Spark 3.3.0 with Scala 2.13.
How can I view the available packages? Above URL redirect it back to bitnami.com!
Related
I am trying to use jhipster, so far Ive seen how it works on java8, I even found an example repo with java7 that is certainly java 8 (https://github.com/jhipster/jhipster-sample-app-java7).
I was wondering if there is any way I can generate an app to run with java7? I know is deprecated from jhipster but seems like if I can run jhipster v 1.7.0 I could do it, any idea how?
So far this is what jhipster -help shows
C:\Users\user>jhipster -help
Usage: jhipster [command] [options]
Commands:
app Create a new JHipster application based on the selected options
aws Deploy the current application to Amazon Web Services
ci-cd Create pipeline scripts for popular Continuous Integration/Continuous Deployment tools
client Create a new JHipster client-side application based on the selected options
cloudfoundry Generate a `deploy/cloudfoundry` folder with a specific manifest.yml to deploy to Cloud Foundry
docker-compose Create all required Docker deployment configuration for the selected applications
entity [name] Create a new JHipster entity: JPA entity, Spring server-side components and Angular client-side components
export-jdl [jdlFile] Create a JDL file from the existing entities
heroku Deploy the current application to Heroku
import-jdl [jdlFiles...] Create entities from the JDL file passed in argument
info Display information about your current project and system
kubernetes Deploy the current application to Kubernetes
languages [languages...] Select languages from a list of available languages. The i18n files will be copied to the /webapp/i18n folder
openshift Deploy the current application to OpenShift
rancher-compose Deploy the current application to Rancher
server Create a new JHipster server-side application
service [name] Create a new Spring service bean
upgrade Upgrade the JHipster version, and upgrade the generated application
completion Print command completion script
Options:
-h, --help output usage information
-d, --debug enable debugger
-V, --version output the version number
You're referring to a 2 years old project sample. Current version of JHipster 4.x requires JDK 8, there's no way to generate a project compatible with JDK 7 with it.
The only solution would be to install an obsolete version of JHipster like 2.22.0 which is the one used to generate this JDK 7 sample project:
npm install -g generator-jhipster#2.22.0
then follow instructions of this version's documentation: http://www.jhipster.tech/documentation-archive/v2.22.0/
I created a Docker container whose purpose is to run a Docker image that implements a REST API. The REST API is created with Java (using the Eclipse IDE, Maven and Spring Boot). When creating a jar file, it (jar file) is titled: workserver-0.0.1-SNAPSHOT.jar
The code is commited to a Gitlab server. When this takes place, a Jenkins job is created. The Jenkins job pulls down the code from the Gitlab Repository, creates a .jar file and then goes through the actions to turn the .jar file into a Docker image (or rather a .zip version of the Docker image).
"scp" is used move the zip file to a target system - where - the .zip file is unpacked (revealing the Docker image) and a container is started. The thing is, the Docker image being used has a version of "latest" (ex: imagename:latest).
I would like to use versions in this scenario starting with Eclipse (i.e. a pom.xml file holding a target workserver-2.2.13.jar file would eventually lead to Docker image that would be named imagename:2.3.13)
I have seen here how one can assign a version number in Docker:
Adding tags to docker image from jenkins
I have also seen that one can use tags and version numbers in Git :
ex: git tag -a v2.5 -m 'Version 2.5' As mentioned above, the Maven
pom.xml file contains instructions to produce a .jar file called:
workserver-0.0.1-SNAPSHOT.jar
The system is working fine. I can commit a change in Eclipse and in a few minutes, a new version of the Docker container has been spun up on the delivery system - ready for use.
The issue I have now is setting up the version numbers.
Any guidance in this area would be greatly appreciated.
TIA
I would recommend using the fabric8io docker maven plugin to create the docker image using maven. This plugin allows you to build, run, and push docker images.
In particular, you can setup the name field in the plugin configuration to be:
<name>workserver:%l<name>
The %l will be resolved to the maven project version, which is the same as the jar version. You can run the plugin explicitly using:
mvn io.fabric8:docker-maven-plugin:build
Or you can set the packaging in the pom to be:
<packaging>docker-build</packaging>
This will build the image whenever you docker an mvn package and will push the image on mvn deploy
What I'm doing:
Trying to connect Spark and Cassandra to retrieve data stored at cassandra tables from spark.
What steps have I followed:
Download cassandra 2.1.12 and spark 1.4.1.
Built spark with sudo build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean packag and sbt/sbt clean assembly
Stored some data into cassandra.
Downloaded these jars into spark/lib:
cassandra-driver-core2.1.1.jar and spark-cassandra-connector_2.11-1.4.1.jar
Added the jar file paths to conf/spark-defaults.conf like
spark.driver.extraClassPath \
~/path/to/spark-cassandra-connector_2.11-1.4.1.jar:\
~/path/to/cassandra-driver-core-2.1.1.jar
How am I running the shell:
After running ./bin/cassandra, I run spark like-
sudo ./bin/pyspark
and also tried with sudo ./bin/spark-shell
What query am I making
sqlContext.read.format("org.apache.spark.sql.cassandra")\
.options(table="users", keyspace="test")\
.load()\
.show()
The problem:
java.lang.NoSuchMethodError:\
scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
But org.apache.spark.sql.cassandra is present in the spark-cassandra-connecter.jar that I downloaded.
Here is the full Log Trace
What have I tried:
I tried running with the option --packages and --driver-class-path and --jars options by adding the 2 jars.
Tried downgrading scala to 2.1 and tried with the scala shell but still the same error.
Questions I've been thinking about-
Are the versions of cassandra, spark and scala that I'm using compatible with each other?
Am I using the correct version of the jar files?
Did I compile spark in the wrong way?
Am I missing something or doing something wrong?
I'm really new to spark and cassandra so I really need some advice! Been spending hours on this and probably it's something trivial.
A few notes
One you are building spark for 2.10 and using Spark Cassandra Connector libraries for 2.11. To build spark for 2.11 you need to use the -Dscala-2.11 flag. This is most likely the main cause of your errors.
Next to actually include the connector in your project just including the core libs without the dependencies will not be enough. If you got past the first error you would most likely see other class not found errors from the missing deps.
This is why it's recommended to use the Spark Packages website and --packages flag. This will include a "fat-jar" which has all the required dependencies. See
http://spark-packages.org/package/datastax/spark-cassandra-connector
For Spark 1.4.1 and pyspark this would be
//Scala 2.10
$SPARK_HOME/bin/pyspark --packages datastax:spark-cassandra-connector:1.4.1-s_2.10
//Scala 2.11
$SPARK_HOME/bin/pyspark --packages datastax:spark-cassandra-connector:1.4.1-s_2.11
You should never have to manually download jars using the --packages method.
Do not use spark.driver.extraClassPath , it will only add the dependencies to the driver remote code will not be able to use the dependencies.
There is a working jar file on Github version 1.4 (https://github.com/ggodlewski/smux-anki-converter/downloads)
The original author has updated the code to versions: 1.5, 1.6 and 1.7 but they are not standalone working jar files but mere zip files containing:
.classpath,
.gitignore,
.project
pom.xml
and some of the folders contained in the working version 1.4.
Version 1.4 is about 7MB and the updates are only about 45kB each.
The project itself seems to be dead and I cannot get in touch with the author.
My question is: how do I make those updates work? How to update version 1.4 to 1.7 on my PC?
I have tried to figure it out using Eclipse import/export but I am not a programmer and I just don't know how to do it.
Please help.
You should be able to build from source as follows:
Make sure that you have a Java JDK installed. Java 6 or later is required, but I would advise Java 7 or Java 8.
Make sure you have Maven installed and configured. (If you need to use an HTTP proxy to access the internet, you will need to instruct Maven ...)
Use git clone <url> to create a clean sandbox.
Change directory to the sandbox.
Run mvn assembly:assembly to build. (That is what the README file says ...)
The first time you do this, Maven is probably going to download a lot of stuff from the Maven Central repo.
If it works, the built JAR(s) will be somewhere in the target subdirectory.
(Note: I haven't tried this myself. YMMV.)
I'm trying to embed Jetty 9 in my project, and the tutorial at http://www.eclipse.org/jetty/documentation/current/advanced-embedding.html#jetty-helloworld suggests I need a file called "jetty-all.jar", but I can't seem to find this in the download distribution. Where do I get it from? I'm not using maven.
I found it in maven repo with a different version:
http://repo1.maven.org/maven2/org/eclipse/jetty/aggregate/jetty-all/9.0.4.v20130625/jetty-all-9.0.4.v20130625.jar
The Search Engine for The Central Repository may helps you find jar files even you don't use maven