Running Hadoop Pi Example - java

I need some help running Hadoop Pi example in Windows. I have sucecssfully setup Hadoop on Windows and can see all NameNode, SecondaryNameNode, JobTracker and TaskTracker running. I can also hit urls:
http://localhost:50070/, http://localhost:50030 and http://localhost:50090/.
However when I am trying to run the in-built Pi example I am getting following exception.
$ hadoop jar hadoop/hadoop-examples-1.2.1.jar pi 4 100
Not a valid JAR: C:\cygwin\home\chandeln\hadoop\hadoop-examples-1.2.1.jar
I have also set both hadoop and java in my path.
$ java -version
java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)
$ hadoop version
Hadoop 1.2.1
Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152
Compiled by mattf on Mon Jul 22 15:23:09 PDT 2013
From source with checksum 6923c86528809c4e7e6f493b6b413a9a
This command was run using /C:/cygwin/usr/local/hadoop/hadoop-core-1.2.1.jar
My .bashrc looks like this:
export JAVA_HOME=/cygdrive/c/Java/jdk1.7.0_40
export HADOOP_PREFIX=/cygdrive/c/usr/local/hadoop
export PATH=$PATH:$HADOOP_PREFIX/bin
Can anybody shed some light on this ?

Even though Hadoop is written in Java, it does not play well on Windows. If you really want to get up and running with Hadoop in the shortest possible time, I would recommend using Linux. You can download the free VMWare Player application, and install it on Windows. Then, download the latest version on Ubuntu, and install it as a virtual machine using VMWare Player. Then you can use your Ubuntu virtual machine to install a single node cluster of Hadoop.
If this seems like too much work, open an Amazon AWS account, and find pre-configured Hadoop machines to work with.

That is most likely due to your Java version not matching with the Jar. Try with jdk1.6.
(Looks like you are playing with YDN Hadoop?)

try
$yarn jar hadoop/hadoop-examples-1.2.1.jar pi 4 100

Related

Use different Java versions on single Ubuntu 18.04.4 LTS server

I have Jenkins server which works on Java 8 but, he should make mvn install before deployment. But compiled applications use Java 14.
After Java 14 was installed, Jenkins starting crash with
$ sudo service jenkins start Job for jenkins.service failed because the control process exited with error code. See "systemctl status jenkins.service" and "journalctl -xe" for details.
With Java 14
$ java -version
java version "14.0.1" 2020-04-14
Java(TM) SE Runtime Environment (build 14.0.1+7)
Java HotSpot(TM) 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)
Jenkins not work.
How I can be sure what Jenkins runs with the 8th version but my app compiling with the 14th version without Docker?
Perhaps I can use JRE from 8th and Compiler from 14th somehow? Or use 8th as active and 14th only for compile but how?
Java:
$ update-java-alternatives --list
java-1.8.0-openjdk-amd64 1081 /usr/lib/jvm/java-1.8.0-openjdk-amd64
java-14-oracle 1091 /usr/lib/jvm/java-14-oracle
Ubuntu: Ubuntu 18.04.4 LTS
I am not a Java/Jenkins developer.
Generally, in these cases where multiple program versions are involved, setting the application configuration to use a specific Java version works best, if the application supports that kind of configuration of course. You should check whether you can specify either of Jenkins or the compiler to use certain Java environment.
If that is not possible, then give it a try using environment variables. First, set Java 14 as the default so the compiler gets the latest version. Second, just before starting the Jenkins server, change Java environment temporarily, so that Jenkins sees Java 8 while running,
$ export JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk-amd64" \
&& export PATH=$JAVA_HOME/bin:$PATH \
&& java -jar jenkins.war --httpPort=8080
On another note, using Docker in these scenarios is really a good idea. You should consider that.
I hope this helps.

Does not load JDBC Library on ARM by execute Java Application

We have a Java-application "app.jar" in a unix home directory with a external Sqlite Driver Library.
- myapp/app.jar
- myapp/lib/sqlite-jdbc-3.8.7.jar
Device Udoo ARM Cortex V9 simliar to Raspberry Pi.
java -version
java version "1.8.0_06
Java(TM) SE Runtime Environment (build 1.8.0_06-b23)
Java HotSpot(TM) Client VM (build 25.6-b23, mixed mode)
Try to run this application failed.
java -classpath lib/sqlite-jdbc-3.8.7.jar -jar myapp.jar
It seems that the Application cannot find the Library.
java.lang.Exception: No native library is found for os.name=Linux and os.arch=arm
at org.sqlite.SQLiteJDBCLoader.loadSQLiteNativeLibrary(SQLiteJDBCLoader.java:284)
at org.sqlite.SQLiteJDBCLoader.initialize(SQLiteJDBCLoader.java:65)
I can't find a solution until now. It is possible to copy the .jar File to the jdk/lib folder?
Edit:
Now I tried to add the sqlite-jdbc-3.7.2.jar to the JRE and it works.
jdk1.8.0_06/jre/lib/ext
The newer version sqlite-jdbc-3.8.7.jar does not work.
I compared the SQLiteJDBCLoader code in version 3.8.7 with version 3.7.2 and found some insight. 3.7.2 seems to have a pure java mode while the corresponding code is from 3.8.7 missing. From my little knowledge and a quick glance on the download page, I conclude one would need a native library for the target platform. I could not find a download for ARM.
If you need a pure java datebase, you might take another choice.
I believe your command line is wrong.
If you use -jar, then the -classpath is ignored!
Either the manifest in app.jar needs to refer to the library, or you need to start it like this: java -classpath lib/sqlite-jdbc-3.8.7.jar:myapp.jar name.of.MainClass

running neo4j in a machine with java 1.6

I am trying to run neo4j for the first time.
In my RHEL 6 machine, I have Java 1.6 installed. I do not want to change that because other programs are using that. When I tried to run neo4j, downloaded from here, I got the error: ERROR! Neo4j cannot be started using java version 1.6.0_24. Please use Oracle(R) Java(TM) 7. Clearly, I need to install JAVA 7. I was wondering if I can just download Java 7 and let neo4j know the path to java executables.
Set the JAVA_HOME and PATH before starting the neo4j server.
set the path on the terminal.
export JAVA_HOME=/home/jdk/jdk1.7.0_45
export PATH=$JAVA_HOME/bin:$PATH
then start the neo4j or you can create a bash script to do this automatically.
The different major versions are normally different packages (e.g., java-1.7.0-openjdk-devel) that go in different directories (usually in /opt), and you can set the JAVA_HOME and/or JDK_HOME environment variables in your startup script. That said, there's rarely any harm in running your programs on the version 7 JVM (just like you can upgrade the kernel without touching your installed programs), and Java 6 is EOL and won't get new security updates.
neo4j 2.0.2 uses java7 it seems and hence requires jdk7.
Go to neo4j website and download 1.8.3 version
http://dist.neo4j.org/neo4j-community-1.8.3-unix.tar.gz
and if you have java 1.6.45 in ubuntu, you are done
untar and copy to /opt and issue the below command from neo4j root directory
sudo ./bin/neo4j start

Eclipse only sees JDK with sudo

I have to start by saying that I have just switched to Ubuntu from Windows and I am new to Linux in general. Coming from Windows, I really am not familiar with how exactly the permissions work, the Linux file system, where I am able to create folders that I can use without the sudo command etc.
I use eclipse on a daily basis for android development and other simple java swing projects. I was able to download and extract the latest available version of eclipse (kepler) to this path: /opt/eclipse. Inside this directory is the eclipse executable and other eclipse related files.
I also installed the JDK and verified with the java -version command. It says:
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)
I am running on a 64 bit CPU architecture so both programs are the 64 bit version.
Now, my problem is that if I open eclipse from the Unity dash or from the file explorer, it give me this error:
However if I run sudo eclipse in the terminal, everything runs just fine as long as I do it as the super user. I have a hunch that the JDK is installed in a folder that is restricted to normal users and only works if I run the sudo command but this is just a hunch. As I said, I am new to Linux and I am not really sure what I am doing wrong here, I am learning as I go. I need a good explanation as to how to make it start from the Unity dock without the sudo command. Thanks in advance!
You'll want to use the chmod command from the terminal to change the permissions on the Java folder. See this link for more information:
http://www.linux.org/threads/file-permissions-chmod.4094/
Well after running the which java cmd, I noticed that the path to the JDK from the eclipse.ini file was not pointing to the correct location. I fixed the path and now it works. I don't know how the sudo cmd overrides the eclipse.ini path to JDK but well, it works

How to install JPype on OS X Lion to use with Neo4j?

I am trying to use Neo4j for a project, and want to interface with it through Python since I'm a newbie to programming and don't know any Java. I'm following the installation instructions, but I'm stuck on
the first step, which is to install JPype.
I'm using OS X 10.7 (lion). I think my configuration is pretty standard
with Python 2.7.2 downloaded from the Python website and Java 1.6.0 downloaded from the Apple website.
When I run
% sudo python setup.py install
On the JPype installer, I get about a 100 lines of error code about various .h files, then it
terminates with the lines:
lipo: can't figure out the architecture type of: /var/tmp//
ccwOzLi9.out
error: command 'gcc-4.2' failed with exit status 1
I found a blog post about a gcc error with JPype, but I followed the instructions there to no avail. I also emailed the author of that post, and he told me had never actually used JPype, had been working in OS X 10.6, and didn't have any insight.
I also emailed the creator of JPype, who told me that he only uses Windows, and has no idea how to make the install work on OS X. But if we can solve this, I can point him to the answer and maybe he can add the solution to the JPype documentation and help lots of other people as well!
So, anyone know what I'm doing wrong? I would like to use Neo4j, but I
don't know Java so I'm at a complete loss for how to fix a compiler
error.
Based on reading every Google result available, my two running
theories are that:
I'm somehow using a 32-bit version of Python or of Java (though I
used standard official installations and can't figure out how to
switch to 64-bit or if that's even possible)
The JPype files can only be compiled using GCC 4.0 instead of 4.2.
But I can't find anything online about how to rollback to GCC 4.0 (or
if it comes shipped with 2011 MacBooks and there is some way to force
JPype to compile with that instead).
There is another similar question, but the solution there is to use a different adapter that goes through REST instead of hooking directly into Java. I will try that if I have to, but I would really rather use the recommended Neo4j method if it's possible.
I'm not a Python guy, but tried installing JPype on my machine:
% uname -a
Darwin fatty-i7.local.tld 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug 9 20:54:00 PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
% java -version
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
On OSX Lion, the latest JDK appears to be located here:
/Developer/SDKs/MacOSX10.7.sdk/System/Library/Frameworks/JavaVM.framework/
A little googling turned up this post: http://blog.y3xz.com/post/5037243230/installing-jpype-on-mac-os-x
I followed those instructions to modify setup.py, then ran sudo python setup.py install with no problems.
Does that help?
On my Lion, the "include" directory in the JDK was gone, so JPype couldn't find jni.h.
I updated setup.py in two places, one to set where to find jar libraries:
def setupMacOSX(self):
self.javaHome = '/System/Library/Frameworks/JavaVM.framework'
self.jdkInclude = ""
self.libraries = ["dl"]
self.libraryDir = [self.javaHome+"/Home/lib"]
self.macros = [('MACOSX',1)]
And one to set where to find jni.h:
def setupInclusion(self):
self.includeDirs = [
self.javaHome+"/Headers",
<other stuff>
For me,
self.javaHome = '/System/Library/Frameworks/JavaVM.framework/Versions/Current/'
worked.
$ uname -a Darwin 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64
$ java -version java version "1.6.0_33" Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720) Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode)
Here are directions I put in my README to install JPype on OS X 10.7. Same idea as the answers here but different enough to warrant the submission.
The python interface to java (JPype) needs mods to the setup.py:
In summary, you need to make sure the JPype setup.py script can see your Java SDK "Headers" and "Home" directory
I had to install Java from Apple first since my default OS X installation did not come with Headers in the typical Java install path which I found by executing:
/usr/libexec/java_home
If you do happen to have a Headers dir, you probably do not need to reinstall Java and can set the path vars below based on your java HOME directory which is likely different from those in these directions
After I installed Java, I found the new installation in:
/Library/Java/JavaVirtualMachines/
My Home directory was:
/Library/Java/JavaVirtualMachines/1.6.0_37-b06-434.jdk/Contents/Home/
and my Header directory was:
/Library/Java/JavaVirtualMachines/1.6.0_37-b06-434.jdk/Contents/Headers/
1) The JPype script assumes that Headers is within the Home directory but it's not, so I changed the Home path var and created a new Content path var in the setup.py script - In setupMacOSX(self):
self.javaHome = '/Library/Java/JavaVirtualMachines/1.6.0_37-b06-434.jdk/Contents/Home/'
self.javaContents = '/Library/Java/JavaVirtualMachines/1.6.0_37-b06-434.jdk/Contents/'
2) In setupMacOSX(self), change self.libraryDir:
self.libraryDir = [self.javaContents + "/Libraries"]
Note that this step was mentioned as required but I did not have to do this for it to work so maybe try without it first
3) In setupInclusion, add paths to your "Home/include" dir and your "Headers" dir:
self.javaHome+"/include",
self.javaContents + "/Headers",
4) Running the installation should now work:
sudo python setup.py install
For everyone who is still trying to install Jpype but has meanwhile updated Mac OS: Andreas Kolleger's answer just works fine, but with XCode 4.3 the path changed to /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk/System/Library/Frameworks/JavaVM.framework/Versions/Current/
In ML it's MacOSX10.8.sdk. At least on my system. :)
For me, on Mountain Lion
self.javaHome = '/System/Library/Frameworks/JavaVM.framework/'
worked.
$uname -a
Darwin 12.0.0 Darwin Kernel Version 12.0.0: Sun Jun 24 23:00:16 PDT 2012; root:xnu-2050.7.9~1/RELEASE_X86_64 x86_64
$ java -version
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode)
I did the same, but choosing
self.javaHome = '/Developer/SDKs/**MacOSX10.6.sdk**/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/'
instead
self.javaHome = '/Developer/SDKs/**MacOSX10.7.sdk**/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/'
realize that I use version 6 vs 7. With the 7 version I got the same errors that the beginning.
To make Will's answer a bit more specific: I had to modify the setupInclusion(self) method by adding the directory path he suggested, ie, '/System/Library/Frameworks/JavaVM.framework/Headers' on Mountain Lion (java version 1.6). With that the installation of JPype succeeded (though gave a bunch of warnings...)
Here's what worked for me - recommend you put the first line in your profile.
export JAVA_HOME=$(/usr/libexec/java_home)
cd $JAVA_HOME
sudo ln -s include Headers
sudo cp include/darwin/* include/
cd -
Note that instead of changing setup.py, I'm changing my JDK install. This has the advantage of fixing the issue for other projects.
For those trying to install on Mountain Lion, I had to further edit the setup.py file to include the header files here:
/System/Library/Frameworks/JavaVM.framework/Versions/Current/Headers

Categories

Resources