Develop Reference

Decide GNU or MUSL build of linux in Java

I have a Java desktop application which is supposed to run in both GNU Linux distributions (Debian and Ubuntu) and MUSL Linux distributions (Alpine). My application uses a native library also and native library build is different for both type of Linux distributions.
I will deliver both with my application in different folders. So at runtime Java program needs to pick the right distribution of native library to pick as per Linux (GNU or MUSL).
I don't find any mechanism to know that in the Java program, which Linux distribution JVM is running on.
One way I was thinking to read the OS file from /etc/ folder of Linux. But I don't think it would be a good solution (as some custom build might change this details), can someone suggest some better solution for this problem? Or how this can be done?

Using Java/JNA, you can map the gnu_get_libc_version() function and attempt to execute it after loading libc. If it works, you're on glibc (GNU). If you get an UnsatisfiedLinkError that the function is not found, you're on some other libc.
Map the function:
public interface Libc extends Library {
Libc INSTANCE = Native.load("c", Libc.class);
String gnu_get_libc_version();
}
Call it:
public class GnuOrMusl {
public static void main(String[] args) {
try {
System.out.println("On GNU libc version " + Libc.INSTANCE.gnu_get_libc_version());
} catch (UnsatisfiedLinkError e) {
System.out.println("Not on glibc!");
}
}
}
There may be similar approaches with a unique function to distinguish other libc variants from MUSL, but as far as I'm aware, MUSL attempts to be so standards-compliant that it doesn't really allow identifying itself.
Another option for finding GNU distributions is the uname -o command that you can execute with a ProcessBuilder.
On non-GNU (Alpine) it is just "Linux" while on Ubuntu, Debian and OpenSUSE it is "GNU/Linux".
You may also have success determining GNU vs. MUSL by iterating /lib* directories looking for libc variants. This is similar to the approach taken when compiling the JDK, which executes the ldd command and parses libraries from that output.
For example, iterating the /lib directory in Alpine linux gives this link: libc.musl-x86_64.so.1 -> ld-musl-x86_64.so.1
In Debian /lib32 has libc.so.6 -> libc-2.28.so, and in OpenSUSE /lib64 I see something similar: libc.so.6 -> libc-2.26.so, and Ubuntu /lib/aarch64-linux-gnu has libc-2.27.so.
If you stay within Java, determining which /lib path to search may require some trial-and-error. Parsing the output of a command line such as ldd `which ls` will likely get you a string containing gnu or musl.
As far as determining which Linux Distribution to use, reading from an /etc folder is a good intuition. I manage the Java-based Operating System and Hardware Information (OSHI) project, and went through pretty much all the options to identify which distribution you are running. You can see the results of all that labor in this class.
I'll quote a comment in that file:
There are two competing options for family/version information. Newer
systems are adopting a standard /etc/os-release file:
https://www.freedesktop.org/software/systemd/man/os-release.html
Some systems are still using the lsb standard which parses a variety
of /etc/*-release files and is most easily accessed via the
commandline lsb_release -a, see here:
https://linux.die.net/man/1/lsb_release In this case, the
/etc/lsb-release file (if it exists) has optional overrides to the
information in the /etc/distrib-release files, which show:
"Distributor release x.x (Codename)"
The code's logic goes:
Attempt /etc/system-release
Attempt /etc/os-release
Run lsb_release command
Read /etc/lsb-release
Look for and read any /etc/*-release file.
Those files contain keys like NAME that help you out.
Feel free to copy and use that file or a variant, or just use that project as a dependency.

You can detect which Linux Distribution is running by following this StackOverflow answer
Unfortunately there aren't so many alternatives to detect if you are running on MUSL or GLIBC

Download (only) OpenJDK and not install it

I do not wish to change my host's notion of what Java is, only download one or more JDKs including OpenJDK. I need to test and compare JDKs as part of a build system.
Just as I am able to download and then not install Oracle JDK with no problem, I need to download and not install a copy of open JDK. The website http://openjdk.java.net/install/index.html doesn't offer this possibility.
Has anyone found a solution to this?

The easiest way to do that would be to download OpenJDK source code and build it. For 8, it basically boils down to cloning the right set of trees and then doing the regular build steps:
cd /path/to/where/you/cloned/jdk8u
mkdir build
bash ../configure # not ../configure because configure is not marked as executible
make all
Look under build/ and search for a j2sdk-image directory. That's your jdk:
cp -a build/images/j2sdk-image /location/where/i/put/my/jdks/
The distribution packages of OpenJDK often include various changes that make it not suitable for moving around (liked tweaked RPATHs). That said, if you are feeling up to the risk, you can do something like:
yum --download-only install 'java-1.8.0-openjdk*'
Now you have a bunch of RPMs. Extract them manually:
rpm2cpio ./name.of.rpm.x86_64.rpm | cpio -id
You should see usr and other fumnny looking directories in the current working directory. Go through the, find the OpenJDK dir and move it over to some other location. For example:
cp -a usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b16.fc24.x86_64/ /location/where/i/put/my/jdks/
Note that it's usr/, not /usr/. Strip non-$ORIGIN RPATHs if you see on any ELF files.
Something similar probably exists for Debian/Ubuntu, but I am not familiar with them.

What do I need to do additionally to set up libhdfs (for c++)?

I'm new to hadoop and I got hadoop installed. Datanode, Namenode etc. are coming up, everything is smooth. Do i have to download libhdfs separately or it comes in the package ? All I can see is Java stuff for it. Thanks a lot.

There is an example: http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/LibHdfs.html (though some of the information, notoriously how to make the example, is outdated)
In order to compile it, you'll need to include the $HADOOP_HOME/include directory and to link to $HADOOP_HOME/lib/native/libhdfs.so (if you are on Linux). Also, you'll need to make sure that when you run your example, the CLASSPATH includes all hadoop libraries. One of the tricks to do it:
for file in `hadoop classpath | tr ':' ' ' | sort | uniq`
do
export CLASSPATH=$CLASSPATH:$file
done
Caveats:
0) the latest linux distro for Hadoop (2.4.1) includes 32-bit shared libraries, which makes 64-bit systems complain. In order to make this example running on 64bit system, you have to provide native libhdfs.so library. That can be done, for example, by downloading Hadoop from source and build a distibution yourself (a useful intro can be found here: http://csrdu.org/nauman/2014/01/23/geting-started-with-hadoop-2-2-0-building/).
1) on Ubuntu, you'll have to instantiate jvm yourself (via JNI) prior to calling libhdfs, because the libhdfs code mistakenly identifies YamVM for the running JVM, therefore does not instantiate
2) on RedHat, I saw this example hanging if the hdfs was not used after datanode and namenode were restarted.

How to setup java on freebsd?

I have both Java JRE and Java JDK on a FreeBSD 7.2 box (running PFSense) from http://www.freebsdfoundation.org/downloads/java.shtml
find / -name gives me output like:
/usr/local/diablo-jre1.6.0/bin/java
/usr/local/diablo-jdk1.6.0/bin/java
/usr/local/diablo-jdk1.6.0/jre/bin/java
so I make a link to /usr/local/bin like so:
ln /usr/local/diablo-jre1.6.0/bin/java /usr/local/bin/java
and now I get
# rehash
# java
Error: could not find libjava.so
Error: could not find Java 2 Runtime Environment.
SOOOOOO, I'm wondering if there is some tool I can use to turn on a particular java vm similar to Ubuntus' /etc/jvm?

The /usr/local/diablo-jre1.6.0/bin/java application probably locates libjava.so relative to the location of the java application itself. By creating a hard link to java and executing it via that hard link, you've probably broken that mechanism.
Suggestions:
Put /usr/local/diablo-jre1.6.0/bin on your search path.
Create an alias for java instead of a link.
Replace the hard link with a symbolic link; i.e. use ln -s to create it.

The FreeBSD packages should have installed a wrapper-script in /usr/local/bin/java that knows about the different installed JVMs, and their JAVA_HOMEs etc. Did you by chance accidentally download the tarball instead of the package?
What does pkg_info | grep jdk yield?

If this is in a jail, you may need to move /proc temporarily so it doesn't try to use it to find out where the Java libraries are. Got bitten by this one!

How to see JIT-compiled code in JVM?

Is there some way to see the native code produces by the JIT in a JVM?

General usage
As explained by other answers, you can run with the following JVM options:
-XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly
Filter on a specific method
You can also filter on a specific method with the following syntax:
-XX:+UnlockDiagnosticVMOptions -XX:CompileCommand=print,*MyClass.myMethod
Notes:
you might need to put the second argument within quotes depending on OS etc.
if the method gets inlined, you could miss some optimisations
How to: Install the required libraries on Windows
If you are running Windows, this page has instructions on how to build and install hsdis-amd64.dll and hsdis-i386.dll which are required to make it work. We copy below and extend the content of that page* for reference:
Where to get prebuilt binaries
You can download prebuilt binaries for Windows from the fcml project
hsdis-amd64.dll
hsdis-i386.dll
How to build hsdis-amd64.dll and hsdis-i386.dll on Windows
This version of the guide was prepared on Windows 8.1 64bit using 64-bit Cygwin and producing hsdis-amd64.dll
Install Cygwin. At the Select Packages screen, add the following packages (by expanding the Devel category, then clicking once on the Skip label next to each package name):
make
mingw64-x86_64-gcc-core (only needed for hsdis-amd64.dll)
mingw64-i686-gcc-core (only needed for hsdis-i386.dll)
diffutils (in Utils category)
Run the Cygwin Terminal. This can be done using the Desktop or Start Menu icon created by the installer, and will create your Cygwin home directory (C:\cygwin\home\<username>\ or C:\cygwin64\home\<username>\ by default).
Download the latest GNU binutils source package and extract its contents to your Cygwin home directory. At the time of writing, the latest package is binutils-2.25.tar.bz2. This should result in a directory named binutils-2.25 (or whatever the latest version is) in your Cygwin home directory.
Download the OpenJDK source by going to the JDK 8 Updates repository, selecting the tag corresponding to your installed JRE version, and clicking bz2. Extract the hsdis directory (found in src\share\tools) to your Cygwin home directory.
In the Cygwin Terminal, enter cd ~/hsdis.
To build hsdis-amd64.dll, enter
make OS=Linux MINGW=x86_64-w64-mingw32 'AR=$(MINGW)-ar' BINUTILS=~/binutils-2.25
To build hsdis-i386.dll, enter
make OS=Linux MINGW=i686-w64-mingw32 'AR=$(MINGW)-ar' BINUTILS=~/binutils-2.25
In either case, replace 2.25 with the binutils version you downloaded. OS=Linux is necessary because, although Cygwin is a Linux-like environment, the hsdis makefile fails to recognize it as such.
The build will fail with messages ./chew: No such file or directory and gcc: command not found. Edit <Cygwin home directory>\hsdis\build\Linux-amd64\bfd\Makefile in a text editor like Wordpad or Notepad++ to change SUBDIRS = doc po (line 342, if using binutils 2.25) to SUBDIRS = po. Re-run the previous command.
The DLL can now be installed by copying it from hsdis\build\Linux-amd64 or hsdis\build\Linux-i586 to your JRE's bin\server or bin\client directory. You can find all such directories on your system by searching for java.dll.
Bonus tip: if you prefer Intel ASM syntax to AT&T, specify -XX:PrintAssemblyOptions=intel alongside any other PrintAssembly options you use.
*page license is Creative Commons

Assuming you're using the Sun Hotspot JVM (i.e. the one provided on java.com by Oracle), you can add the flag
-XX:+PrintOptoAssembly
when running your code. This will print out the optimized code generated by the JIT compiler and leaves out the rest.
If you want see the entire bytecode, including the unoptimized parts, add
-XX:CompileThreshold=#
when you're running your code.
You can read more about this command and the functionality of JIT in general here.

You need an hsdis plugin to use PrintAssembly. A convenient choice is the hsdis plugin based on FCML library.
It can be compiled for UNIX-like systems and on Windows you can use pre-built libraries available in the FCML download section on Sourceforge:
To install in Windows:
Extract the dll (it can be found in hsdis-1.1.2-win32-i386.zip and hsdis-1.1.2-win32-amd64.zip).
Copy the dll to wherever exists java.dll (use Windows search). On my system, I found it at two locations:
C:\Program Files\Java\jre1.8.0_45\bin\server
C:\Program Files\Java\jdk1.8.0_45\jre\bin\server
To install in Linux:
Download source code, extract it
cd <source code dir>
./configure && make && sudo make install
cd example/hsdis && make && sudo make install
sudo ln -s /usr/local/lib/libhsdis.so <JDK PATH>/lib/amd64/hsdis-amd64.so
sudo ln -s /usr/local/lib/libhsdis.so <JDK PATH>/jre/lib/amd64/hsdis-amd64.so
On my system, the JDK is in /usr/lib/jvm/java-8-oracle
How to run it:
java -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly
-XX:+LogCompilation -XX:PrintAssemblyOptions=intel,mpad=10,cpad=10,code
-jar fcml-test.jar
Additional configuration parameters:
code Print machine code before the mnemonic.
intel Use the Intel syntax.
gas Use the AT&T assembler syntax (GNU assembler compatible).
dec Prints IMM and displacement as decimal values.
mpad=XX Padding for the mnemonic part of the instruction.
cpad=XX Padding for the machine code.
seg Shows the default segment registers.
zeros Show leading zeros in case of HEX literals.
The Intel syntax is a default one in case of Windows, whereas the AT&T one is a default for the GNU/Linux.
For more details see the FCML Library Reference Manual

For the HotSpot (was Sun) JVM, even in product modes:
http://wikis.oracle.com/display/HotSpotInternals/PrintAssembly
Some assembly required: it needs a plugin.

I believe WinDbg would be helpful if you are running it on windows machine.
I have just run one jar.
Then I attached to the java process
through Windbg
Examined threads by ~ command; There were 11 threads, 0 thread was main worker thread
Switched to 0-thread - ~0s
Looked through unmanmaged callstack by kb there was:
0008fba8 7c90e9c0 ntdll!KiFastSystemCallRet
0008fbac 7c8025cb ntdll!ZwWaitForSingleObject+0xc
0008fc10 7c802532 kernel32!WaitForSingleObjectEx+0xa8
0008fc24 00403a13 kernel32!WaitForSingleObject+0x12
0008fc40 00402f68 java+0x3a13
0008fee4 004087b8 java+0x2f68
0008ffc0 7c816fd7 java+0x87b8
0008fff0 00000000 kernel32!BaseProcessStart+0x23
Highlighted lines is direct running JIT-ed code on JVM.
Then we can look for method address:
java+0x2f68 is 00402f68
On WinDBG: Click View -->
Disassembly. Click Edit --> Go to
Address. Put 00402f68 there
and got
00402f68 55 push ebp
00402f69 8bec mov ebp,esp
00402f6b 81ec80020000 sub esp,280h
00402f71 53 push ebx
00402f72 56 push esi
00402f73 57 push edi
... and so on
For additional info here is the Example how to trace back JIT-ed code from memory dumps using process explorer and WinDbg.

Another way to see machine code and some performance data is to use AMD's CodeAnalyst or OProfile, which have a Java plugin to visualize executing Java code as machine code.

Print the assembly of your hotspots with JMH's perfasm profilers (LinuxPerfAsmProfiler or WinPerfAsmProfiler). JMH does require the hsdis library since it relies on PrintAssembly.