Hadoop build Failing in Windows: zconf.h is missing from native.sln? - java

I'm trying to build the following hadoop version on development computer with Windows 10 Home Edition
hadoop-2.7.3-src
Here are the details about my local development environment:
-Windows 10 Home Edition
-Intel Core i5-6200U CPU #2.30GHz
-RAM 16 GB
-64-bit Operating System, x64-based processor
-Microsoft Visual Studio Community 2015 Version 14.0.25431.01 Update 3
-Also added MSBUILD location as C:\Program Files (x86)\MSBuild\14.0\Bin\amd64 to Windows System Environment Variable Path
-.NET Framework 4.6.01586
-cmake version 3.7.2
-CYGWIN_NT-10.0 LTPBCV82DUG 2.7.0(0.306/5/3) 2017-02-12 13:18 x86_64 Cygwin
-java version "1.8.0_121"
-Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
-Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
-Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00)
-Google Protocol Buffers protoc --version libprotoc 2.5.0
Also, I've created and system environment variable called Platform and set it to x64
I opened up Developer Command Prompt for Visual Studio 2015 (VS2015)
c:\hadoop\hadoop-2.7.3-src> mvn package -Pdist,native-win -DskipTests -Dtar -X
Unfortunately, I'm getting the following error:
[C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
ZlibDecompressor.c
c:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36): fatal error C1083: Cannot open include file: 'zlib.h': No such file or directory [C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
Done Building Project "C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj" (default targets) -- FAILED.
Done Building Project "C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.sln" (default targets) -- FAILED.
Build FAILED.
"C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.sln" (default target) (1) ->
"C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj" (default target) (2) ->
(ClCompile target) ->
c:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36): fatal error C1083: Cannot open include file: 'zlib.h': No such file or directory [C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
c:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\src\org\apache\hadoop\io\compress\zlib\org_apache_hadoop_io_compress_zlib.h(36): fatal error C1083: Cannot open include file: 'zlib.h': No such file or directory [C:\hadoop\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.vcxproj]
0 Warning(s)
2 Error(s)
Time Elapsed 00:00:02.49
The aforementioned error has to do with zlib tool.
After researching online, someone said that the following Visual Studio solution file needs to be built successfully in Visual Studio:
....\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\native\native.sln
Using Visual Studio 2015 in Administrator mode, I opened up the native.sln file, and immediately saw an error:
enter image description here
Could someone please tell me what steps I have to take to resolve said error?

So there were quite a few steps I had to take in order to resolve the problems.
Within the ....\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\winutils directory, I opened up the following solution in Visual Studio 2015:
winutils.sln
Within .....\hadoop-2.7.3-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.c , I commented out the following line of code, and made a modified copy of it as shown below:
//const WCHAR* wsceConfigRelativePath = WIDEN_STRING(STRINGIFY(WSCE_CONFIG_DIR)) L"\\" WIDEN_STRING(STRINGIFY(WSCE_CONFIG_FILE));
const WCHAR* wsceConfigRelativePath = WIDEN_STRING("../etc/hadoop") L"\\" WIDEN_STRING("wsce-site.xml");
Also, In the winutils solution's property window, I had to set the platform value to x64 as the screenshot below shows:
Next, I opened Dos command prompt, and checked the exact version of my Windows OS:
ver
Microsoft Windows [Version 10.0.14393]
Also, I opened up the property window of the libwinutils project, and ensured that properties that are marked in the following snapshot had the proper values:
Also, I took the same steps for the properties of the winutils project:
(Sorry, stackoverflow would not allow me to place another picture snapshot, but all you basically have to do is make sure the the winutils project's property are set properly )
I downloaded zlib version 1.2.11 source code. Using Developer Command Prompt for VS2015 ( Visual Studio 2015 ) I built zlib from zlib version 1.2.11 source code using cmake
c:\zlib\zlib-1.2.11>cmake -G "Visual Studio 14 2015" -A x64 c:\zlib\zlib-1.2.11\
-- The C compiler identification is MSVC 19.0.24215.1
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - failed
-- Looking for fseeko
-- Looking for fseeko - not found
-- Looking for unistd.h
-- Looking for unistd.h - not found
-- Configuring done
-- Generating done
-- Build files have been written to: C:/zlib/zlib-1.2.11
Finally, run the build with cmake
c:\zlib\zlib-1.2.11>cmake --build .
In Windows System Variables I have the following variable defined:
ZLIB_HOME is set to C:\zlib\zlib-1.2.11

Related

Vim YouCompleteMe Java setup

I'm trying to set up Java autocompletion for Vim using the YouCompleteMe plugin. I ran the install script and got the following message saying that I need to set an option called java_binary_path:
$ python3 install.py --java-completer --clang-completer
Generating ycmd build configuration...OK
Compiling ycmd target: ycm_core...OK
Building regex module...OK
Building watchdog module...OK
Installing jdt.ls for Java support...
*** WARNING ***: jdt.ls requires Java 17. You must set the option java_binary_path to point to a working java 17.
OK
I tried setting this in my ~/.vimrc file as follows, but now I'm getting the error E518: Unknown option: java_binary_path=/usr/local/opt/openjdk#17/bin/java:
set java_binary_path=/usr/local/opt/openjdk#17/bin/java
Does anyone know how to set this option?

CMake could not find JNI

I created a new Android Studio Project with C++ support and did not add anything to it. The project builds, compiles and runs with CMake. I then added the following two lines of code to CMakeLists.txt at the bottom of the file and I get a CMake build error.
find_package(Java COMPONENTS Development) # Line 47
find_package(JNI REQUIRED) # Line 48
And this is the error I get when building the project
-- Found Java: /usr/lib/jvm/java-10-oracle/bin/java (found version "10.0.1") found components: Development
-- Configuring incomplete, errors occurred!
CMake Error at /home/xxxxx/Android/Sdk/cmake/3.6.4111459/share/cmake-3.6/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
Could NOT find JNI (missing: JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY
See also "/home/xxxxx/AndroidStudioProjects/MyApplication/app/.externalNativeBuild/cmake/release/x86_64/CMakeFiles/CMakeOutput.log".
JAVA_INCLUDE_PATH2 JAVA_AWT_INCLUDE_PATH)
Call Stack (most recent call first):
/home/xxxxxx/Android/Sdk/cmake/3.6.4111459/share/cmake-3.6/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
/home/xxxxxx/Android/Sdk/cmake/3.6.4111459/share/cmake-3.6/Modules/FindJNI.cmake:314 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:48 (find_package)
External native generate JSON release: JSON generation completed with problems
CMake runs when being called form the command line, but fails inside Android Studio for some reason and I am not sure why.
EDIT 1
Here is the CMakeOutput.log file
EDIT 2
This is all that is in CMakeLists.txt file. It compiles in cmd with the command cmake.
cmake_minimum_required(VERSION 3.4.1)
find_package(Java COMPONENTS Development)
find_package(JNI REQUIRED)
This is the cmd output
E:\Users\xxxxx\AndroidStudioProjects\MyApplication\app\build>cmake ..
-- Configuring done
-- Generating done
-- Build files have been written to: E:/Users/xxxxx/AndroidStudioProjects/MyApplication/app/build
When I try to add it compile it in Android Studio using gradle I get the following error on Windows 10 pro 64 bit
Microsoft Windows [Version 10.0.17134.112]
(c) 2018 Microsoft Corporation. All rights reserved.
E:\Users\xxxxx\AndroidStudioProjects\MyApplication>gradlew build --stacktrace
> Task :app:generateJsonModelDebug
External native generate JSON debug: starting JSON generation
External native generate JSON debug: using platform version 24 for ABI ARMEABI_V7A and min SDK version 24
External native generate JSON debug: noticing that build file 'E:\Users\xxxxx\AndroidStudioProjects\MyApplication\app\CMakeLists.txt' is out of date with respect to E:\Users\xxxxx\AndroidStudioProjects\MyApplication\app\.extern
alNativeBuild\cmake\debug\armeabi-v7a\android_gradle_build.json
External native generate JSON debug: rebuilding JSON E:\Users\xxxxx\AndroidStudioProjects\MyApplication\app\.externalNativeBuild\cmake\debug\armeabi-v7a\android_gradle_build.json due to:
External native generate JSON debug: - a dependent build file changed
External native generate JSON debug: keeping json folder 'E:\Users\xxxxx\AndroidStudioProjects\MyApplication\app\.externalNativeBuild\cmake\debug\armeabi-v7a' but regenerating project
External native generate JSON debug: executing cmake Executable : E:\Android\SDK\cmake\3.6.4111459\bin\cmake.exe
arguments :
-HE:\Users\xxxxx\AndroidStudioProjects\MyApplication\app
-BE:\Users\xxxxx\AndroidStudioProjects\MyApplication\app\.externalNativeBuild\cmake\debug\armeabi-v7a
-DANDROID_ABI=armeabi-v7a
-DANDROID_PLATFORM=android-24
-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=E:\Users\xxxxx\AndroidStudioProjects\MyApplication\app\build\intermediates\cmake\debug\obj\armeabi-v7a
-DCMAKE_BUILD_TYPE=Debug
-DANDROID_NDK=E:\Android\SDK\ndk-bundle
-DCMAKE_CXX_FLAGS=
-DCMAKE_TOOLCHAIN_FILE=E:\Android\SDK\ndk-bundle\build\cmake\android.toolchain.cmake
-DCMAKE_MAKE_PROGRAM=E:\Android\SDK\cmake\3.6.4111459\bin\ninja.exe
-GAndroid Gradle - Ninja
jvmArgs :
CMake Error at E:/Android/SDK/cmake/3.6.4111459/share/cmake-3.6/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
Could NOT find JNI (missing: JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY
JAVA_INCLUDE_PATH2 JAVA_AWT_INCLUDE_PATH)
Call Stack (most recent call first):
E:/Android/SDK/cmake/3.6.4111459/share/cmake-3.6/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
E:/Android/SDK/cmake/3.6.4111459/share/cmake-3.6/Modules/FindJNI.cmake:314 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:4 (find_package)
-- Configuring incomplete, errors occurred!
See also "E:/Users/xxxxx/AndroidStudioProjects/MyApplication/app/.externalNativeBuild/cmake/debug/armeabi-v7a/CMakeFiles/CMakeOutput.log".
Could NOT find JNI (missing: JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY
Solved with
sudo apt-get install -y openjdk-8-jdk
sudo apt-get install -y default-jdk
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
As the CMake version bundled with Android tries to tell you, it can't find the JNI package because some parts were missing:
missing: JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY JAVA_INCLUDE_PATH2 JAVA_AWT_INCLUDE_PATH
Checking the documentation for FindJNI (https://cmake.org/cmake/help/latest/module/FindJNI.html) these variables are set to the locations of libraries and headers not shipped with the Android version of JNI (to little surprise, Android does not included the AWT library for instance).
When running find_package(JNI REQUIRED), the FindJNI code checks if these variables are set and if not, issues an error.
A workaround is to set these variables yourself, before calling find_package:
# We are only interested in finding jni.h: we do not care about extended JVM
# functionality or the AWT library.
set(JAVA_AWT_LIBRARY NotNeeded)
set(JAVA_JVM_LIBRARY NotNeeded)
set(JAVA_INCLUDE_PATH2 NotNeeded)
set(JAVA_AWT_INCLUDE_PATH NotNeeded)
find_package(JNI REQUIRED)
Be aware though, that your code will only be able to use jni.h and its functionality: if it tries to access any other part of the JNI package it will fail (probably at compile time) because essentially, you have tricked CMake into thinking that the entire package was found, when in reality only a part of it exists in the Android setup.
I found a solution that works for me:
cmake_minimum_required(VERSION 3.10)
project("ktaglib")
set(JAVA_AWT_LIBRARY "$ENV{JAVA_HOME}/lib/libjawt.so")
set(JAVA_JVM_LIBRARY "$ENV{JAVA_HOME}/lib/server/libjvm.so")
set(JAVA_INCLUDE_PATH "$ENV{JAVA_HOME}/include")
set(JAVA_INCLUDE_PATH2 "$ENV{JAVA_HOME}/include/linux")
set(JAVA_AWT_INCLUDE_PATH "$ENV{JAVA_HOME}/include")
find_package(JNI REQUIRED)
You must set the environment variable JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64" beforehand.
More info: https://github.com/dockcross/dockcross/issues/576
Same on alpine:edge docker image, fixed using:
ENV JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk
RUN apk add --no-cache openjdk8
ref: https://pkgs.alpinelinux.org/contents?branch=edge&name=openjdk8&arch=x86_64&repo=community

java.lang.NoClassDefFoundError error running RNetLogo on 64-bit Ubuntu 16.04 - but is the issue rJava or NetLogo?

The issue
I am attempting to start an instance of NetLogo in R using the RNetLogo package, which has rJava as a dependency.
During installation of rJava 0.9-9 (the latest development snapshot from rforge), I get the following errors:
warning: [options] bootstrap class path not set in conjunction with -source 1.6
This appears to refer to Java version 6, even though I have version 8 only on my machine. However, the developer of rJava appears to say here that as long as the package installs and loads correctly, which it does, then users should ignore the warnings. Furthermore, .jinit() appears to run correctly:
> .jinit()
[1] 0
and the correct version of Java is detected:
> .jcall("java/lang/System", "S", "getProperty", "java.runtime.version")
[1] "1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12"
So, having loaded RNetLogo, I attempt to start a NetLogo instance. Here is the minimal code I'm running:
library(RNetLogo)
nl.path <- "~/NetLogo 6.0.2/app"
NLStart(nl.path, gui = FALSE, nl.jarname='netlogo-6.0.2.jar')
Which returns the following errors:
java.lang.NoClassDefFoundError: org/nlogo/workspace/Controllable
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
Caused by: java.lang.ClassNotFoundException
at RJavaClassLoader.findClass(RJavaClassLoader.java:383)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 2 more
I get the same issue using RStudio or running R from the terminal (including running as root).
My full sessionInfo():
R version 3.4.3 (2017-11-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.3 LTS
Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
locale:
[1] LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=C LC_TIME=en_GB.UTF-8 LC_COLLATE=en_GB.UTF-8
[5] LC_MONETARY=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8 LC_PAPER=en_GB.UTF-8 LC_NAME=en_GB.UTF-8
[9] LC_ADDRESS=en_GB.UTF-8 LC_TELEPHONE=en_GB.UTF-8 LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=en_GB.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] RNetLogo_1.0-4 igraph_1.1.2 rJava_0.9-9
loaded via a namespace (and not attached):
[1] compiler_3.4.3 magrittr_1.5 tools_3.4.3 pkgconfig_2.0.1
Attempted fixes
Based on other users' issues that appear to be related, I also tried to following:
° Setting environment variables in /etc/profile.d/:
export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"
export PATH="$PATH:$HOME/bin:$JAVA_HOME/bin"
export LD_LIBRARY_PATH="/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server"
export CLASSPATH="$ClASSPATH:$HOME/R/x86_64-pc-linux-gnu-library/3.4/rJava/java"
° Running
sudo R CMD javareconf -e
° Adding a couple of lines proposed as mac OS fixes to the start of my script:
Sys.setenv(NOAWT=1)
dyn.load('/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so')
Thanks in advance.
I spent a lot of time staring at this question, assuming I had the same issue because I was getting the same error. So, for future readers: On Ubuntu 17.10, with rJava 0.9-9, NetLogo 6.0.3 and RNetLogo 1.0-4, the same error results when using the Oracle 9 Java SDK. Going back to 8 solves it.
For the OP: The ~/ in nl.path might be the problem. On my system it doesn't work, but specifying the full path (that is, home/user_name/...) does.
I had the same problem with rNetLogo 1.0-4 for NetLogo 5.3.1 in combination with openjdk-8 on Ubuntu 16.04. I tried openjdk-8 and Oracle Java 8 (using ppa:webupd8team/java) with no luck.
In the end it works with rJava as Ubuntu package (r-cran-rjava) and going back to rNetLogo 1.0-0 (with I suppose did the trick), still using Oracle Java 8.
url = 'https://cran.r-project.org/src/contrib/Archive/RNetLogo/RNetLogo_1.0-0.tar.gz'
install.packages(url, repos=NULL, type="source")

Upgraded Fedora 24, Gradle status text does not show

Prior to upgrading to Fedora 24, I would see output from Gradle like the following:
> Loading
> Configuring project 0/20
> Executing test 39 of 4049, 12 skipped.
I've read that users often set TERM=dumb in order to disable this output; however, my TERM environment variable is set to xterm-256color.
Could someone tell me what I need to set or change under Fedora 24 (which seemed to exist in Fedora 23) to allow me to get this Gradle output?
The problem is that Gradle is wanting to use ncurses5 where it seems Fedora 24 install comes with a newer version, thus causing the issue. This can be seen by running gradle --debug and seeing the following output:
[DEBUG] [org.gradle.internal.nativeintegration.services.NativeServices] Unable to load from native-platform backed Consol
eDetector. Continuing with fallback. Failure: net.rubygrapefruit.platform.NativeException: Failed to load native library 'libnative-pl
atform-curses.so' for Linux amd64.
caused by: java.lang.UnsatisfiedLinkError: /home/dkowis/.gradle/native/19/linux-amd64/libnative-platform-curses.so: libncurses.so.5: c
annot open shared object file: No such file or directory
The temporary solution is to create a symbolic link to the new libncurses.so.6 library as follows:
sudo ln -s /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5
After doing this, Gradle now outputs the build status text as it did previously.

Playscape publishing kir (1.17) error: Failed to execture lib2dex

We have problem with PlayscapeSDK 1.17.411 in our Unity Android project.
When we build the game, on "Publishing kit post-process" step we have this error:
An error occured while applying post-build logic: Failed to execture lib2dex. Error description: jar2dex.
PlayscapeSDK we add in project according to official instruction.
We build project on OS Windows7 x64.
System variables ANDROID_HOME, JAVA_HOME, JDK_HOME added.
Also path to JAVA exist in PATH variable.
Google Play Service lib(google-play-services.jar) we stripped (3,79 MB)
Using soft:
Playscape sdk 1.17.411
Unity 5.3.2f1
JDK jdk1.8.0_71
By the way, the same error we have, when try to build app on MacOS (El Capitan)

Categories

Resources