Background & Problem
I am having a bit of trouble running the examples in Spark's MLLib on a machine running Fedora 23. I have built Spark 1.6.2 with the following options per Spark documentation:
build/mvn -Pnetlib-lgpl -Pyarn -Phadoop-2.4 \
-Dhadoop.version=2.4.0 -DskipTests clean package
and upon running the binary classification example:
bin/spark-submit --class org.apache.spark.examples.mllib.BinaryClassification \
examples/target/scala-*/spark-examples-*.jar \
--algorithm LR --regType L2 --regParam 1.0 \
data/mllib/sample_binary_classification_data.txt
I receive the following error:
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.92-1.b14.fc23.x86_64/jre/bin/java: symbol lookup error: /tmp/jniloader5830472710956533873netlib-native_system-linux-x86_64.so: undefined symbol: cblas_dscal
Errors of this form (symbol lookup error with netlib) are not limited to this particular example. On the other hand, the Elastic Net example (./bin/run-example ml.LinearRegressionWithElasticNetExample) runs without a problem.
Attempted Solutions
I have tried a number of solutions to no avail. For example, I went through some of the advice here https://datasciencemadesimpler.wordpress.com/tag/blas/, and while I can successfully import from com.github.fommil.netlib.BLAS and LAPACK, the aforementioned symbol lookup error persists.
I have read through the netlib-java documentation at fommil/netlib-java, and have ensured my system has the libblas and liblapack shared object files:
$ ls /usr/lib64 | grep libblas
libblas.so
libblas.so.3
libblas.so.3.5
libblas.so.3.5.0
$ ls /usr/lib64 | grep liblapack
liblapacke.so
liblapacke.so.3
liblapacke.so.3.5
liblapacke.so.3.5.0
liblapack.so
liblapack.so.3
liblapack.so.3.5
liblapack.so.3.5.0
The most promising advice I found was here http://fossdev.blogspot.com/2015/12/scala-breeze-blas-lapack-on-linux.html, which suggests including
JAVA_OPTS="- Dcom.github.fommil.netlib.BLAS=com.github.fommil.netlib.NativeRefBLAS"
in the sbt script. So, I included appended those options to _COMPILE_JVM_OPTS="..." in the build/mvn script, which also did not resolve the problem.
Finally, a last bit of advice I found online suggested passing the following flags to sbt:
sbt -Dcom.github.fommil.netlib.BLAS=com.github.fommil.netlib.F2jBLAS \
-Dcom.github.fommil.netlib.LAPACK=com.github.fommil.netlib.F2jLAPACK \
-Dcom.github.fommil.netlib.ARPACK=com.github.fommil.netlib.F2jARPACK
and again the issue persists. I am limited to two links in my post, but the advice can be found as the README.md of lildata's 'scaladatascience' repo on github.
Has anybody suffered this issue and successfully resolved it? Any and all help or advice is deeply appreciated.
It's been a couple months, but I got back to this problem and was able to get a functioning workaround (posting here in case anybody else has the same issue).
It came down to library precedence; so, by calling:
$ export LD_PRELOAD=/path/to/libopenblas.so
prior to launching Spark, everything works as expected.
I figured out the solution after reading:
https://github.com/fommil/netlib-java/issues/88 (directly addresses this issue)
JNI "symbol lookup error" in shared library on Linux (similar linking issue, doesn't have to do with Spark but answers are informative with regards to linking)
Related
I'm here to write about an issue that I've tried to fix for 3 weeks.
I think I updated a tool in the SDK within Android Studio (not sure which one).After this, I saw a lot of different issues that caused all my projects to stop working. I uninstalled it and deleted all my data of installations SDK NDK and reinstalled it, but I see the same error.
When I create a new Android Studio project I get these issues:
Access Deniel: I followed all instructions which results in the second issue.
you should use version Kotlin 1.2.51 or higher: I
don't use Kotlin in my project and the version of Gradle is 3.1.0, which was working correctly before.
Following the instructions, I updated the version of Kotlin in my project to 1.2.51, I received a lot of issues:
Android resource compilation failed
Output: C:\Users\safin\Desktop\Project\myproject 02-05-
2018\myproject\myproject\myproject\app\build\intermediates\incremental\mergeDebugResources\merged.dir\values\values.xml:977: error: unescaped apostrophe in
string
"Safin click,
it is a oneself.\n\n
All have made
and click.\n\n
After days.\n
Thank you safineas.".
C:\Users\safin\Desktop\Project\myproject
02-05-2018\myprojectr\myproject\myproject\app\build\intermediates
\incremental\mergeDebugResources\merged.dir\values\values.xml:963:
error: not a valid string.
Command: C:\Users\safin\.gradle\caches\transforms-1\files-1.1\aapt2-3.2.0-
4818971-windows.jar\016347ebf5a2e9c410c8e5c96l7d7b16\aapt2-3.2.0-4818971-windows\aapt2.exe compile --legacy \
-o \
C:\Users\safin\Desktop\Project\myproject 02-05-2018\myproject\myproject\myproject\app\build\intermediates\res\merged\debug \
C:\Users\safin\Desktop\Project\myproject 02-05-2018\myproject\myproject\myproject\app\build\intermediates\incremental\mergeDebugResources\merged.dir\values\values.xml
Daemon: AAPT2 aapt2-3.2.0-4818971-windows Daemon #0
See attached screenshots below:
I don't get your whole problem but in that case - error: unescaped apostrophe in string, the string you provided SHOULD BE MODIFIED. You must add backslash before the apostrophe like this-
\"Safin click, it is a oneself.\n\n All have made and click.\n\n After days.\n Thank you safineas.\"
Our application is a RoR app, and currently uses JRuby version 1.7.22, and JRE 8_65. Our app is an on-prem solution, so we use JRuby to host our application on JVM at the target, Windows Server 2012 R2 system. We compile our ruby code, using
jruby -S jrubyc
This takes the .rb file and compiles it to a .class file. In the original .rb, it loads in the class file, like so.
load __FILE__.sub(/\.rb$/, ".class")
This all works with JRuby 1.7.22
Now, we want to update JRuby to 9.0.5.0, but are experiencing some problems when it comes to deploying our application. Basically, that line of code above inside of the .rb file is not working anymore, and we get the error when trying to run a rake db:setup
rake aborted!
LoadError: C:/appname/app/models/app_attribute.class is not compiled Ruby; use java_import to load normal classes
C:/appname/app/models/app_attribute.rb:1:in `<top>'
C:/appname/db/seeds.rb:10:in `<top>'
C:/appname/db/seeds.rb:9:in `block in (root)'
Tasks: TOP => db:setup => db:seed
(See full trace by running task with --trace)
Great. So I replace load with java_import
rake aborted!
ArgumentError: not a valid Java identifier: C:/appname/app/models/app_attribute.class
uri:classloader:/jruby/java/core_ext/object.rb:43:in `block in java_import'
uri:classloader:/jruby/java/core_ext/object.rb:34:in `java_import'
C:/appname/app/models/app_attribute.rb:1:in `<top>'
C:/appname/db/seeds.rb:10:in `<top>'
C:/appname/db/seeds.rb:9:in `block in (root)'
Tasks: TOP => db:setup => db:seed
(See full trace by running task with --trace)
Still not working, no matter what I try. I looked at this post: https://github.com/jruby/jruby/issues/3018
I tried to pass the parameter
jruby -Xaot.loadClasses=true
But I get a warning saying that aot.LoadClasses is not recognized. EVEN THOUGH I see it in the properties when I type
jruby -Xproperties
I have done A LOT of research on this, and have probably have looked at everything on the internet regarding this. Any input will be greatly appreciated. Is there something I missing? I am not fully adept in Java.
Thank you.
might be the same issue as https://github.com/jruby/jruby/issues/3651
which means you'll need to wait for 9.1 or use a snapshot http://ci.jruby.org/
since, the error is slightly different you should look into reproducing with snapshot and if it fails (might be Windows related) a step-by-step reproduction might speed-up getting the issue resolved.
jruby -Xaot.loadClasses=true
this is not needed with Warbler
But I get a warning saying that aot.LoadClasses is not recognized. EVEN THOUGH I see it in the properties when I type
hmm, could you reproduce this with an empty script and no JRUBY_OPTS ?
I have done A LOT of research on this, and have probably have looked at everything on the internet regarding this. Any input will be greatly appreciated.
you might want to try looking into the issue next time :) or considering getting some support
Is there something I missing? I am not fully adept in Java.
you shouldn't be missing anything - its not a Java issue ...
Can you please tell me, is this possible to remove Java JDK package with Chef, with windows_package, or I have to execute command to silent uninstall? Much better would be the first option.
I've tried this way:
windows_package node['name']['JDK1.6'] do
action :remove
end
and even added option installer_type :custom, still got this error:
FATAL: Mixlib::ShellOut::ShellCommandFailed: windows_package[Java(TM) SE Development Kit 6 Update 35] (line 4) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0, 42, 127], but received '1603'
---- Begin output of MsiExec.exe /I{32A3A4F4-B792-11D6-A78A-00B0D0160350} /qn ----
STDOUT:
STDERR:
---- End output of MsiExec.exe /I{32A3A4F4-B792-11D6-A78A-00B0D0160350} /qn ----
Ran MsiExec.exe /I{32A3A4F4-B792-11D6-A78A-00B0D0160350} /qn returned 1603
Well, I am out of ideas. On official Opscode site I've found this information:
:remove: remove a package. The remove action is completely hit or miss as many application uninstallers do not support a full silent/quiet mode.
But as far as I know, JDK supports silent uninstall.
So - how should I do this in order to properly uninstall packages? Is this even possible?
Many thanks for every help.
Specification:
Chef 12.4.1
Windows 7
Java JDK in versions: 6u35, 7u79, 8u45
If you need additional information, feel free to ask.
Well, I've ended up with my own implementation. For those, who would need this, too:
# variable 'code' is for JDK version code from registry.
# Uninstall comand: MsiExec.exe /quiet /X[CODE]
# Codes are in regkey HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall. Be aware to check key for JDK, not JRE!
remove_command = "MsiExec.exe /quiet /X#{code}"
execute "Remove JDK from system" do
command remove_command
returns [0,1605] # 1605 for non-existing product error
end
For example, for JDK 1.6.35 (mentioned in question) code should be
code = '{32A3A4F4-B792-11D6-A78A-00B0D0160350}'
So it seems like this is different command than used by Chef:
used by Chef
MsiExec.exe /I{32A3A4F4-B792-11D6-A78A-00B0D0160350} /qn
this command
MsiExec.exe /quiet /X{32A3A4F4-B792-11D6-A78A-00B0D0160350}
Maybe somebody will find this useful.
This looks like it might not be a chef problem as such. The uninstaller is hitting an error for some reason. Try googling for "java install error 1603" and you'll see a bunch of posts, including one that's apparently a bug that's currently under investigation https://www.java.com/en/download/help/error_1603.xml - though it sounds like that might be install not uninstall.
Try removing it manually and debug that, then once you understand it you can chef it.
I did something similar to your answer, but it only requires the package name rather than the code from the registry.
execute 'uninstall_jdk8u72' do
command "wmic product where name=\"#{node[:java8u72][:package_name]}\" call uninstall"
end
I got the idea from https://github.com/chef-cookbooks/windows/issues/89
I'm following the tutorial at http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-using-sdk-java-add-data-to-stream.html - translating the PutRecords example fom java to scala.
I've included amazon-kinesis-client v1.20 and aws-java-sdk-kinesis v1.9.16 as dependencies.
Compiling using sbt gives me the following error:
value putRecords is not a member of com.amazonaws.services.kinesis.AmazonKinesisClient
Experimenting using the console confirms that this method does not seem to exist on the version of AmazonKinesisClient included with these libraries. Has anyone else had this problem and does anyone know how to resolve it?
The method appears to exist in the aws-java-sdk-kinesis 1.9.16 JAR currently available through Maven:
$ javap -cp aws-java-sdk-kinesis-1.9.16.jar com.amazonaws.services.kinesis.AmazonKinesisClient | grep PutRecords
public com.amazonaws.services.kinesis.model.PutRecordsResult putRecords(com.amazonaws.services.kinesis.model.PutRecordsRequest);
Is it possible that your IDE or classpath contains a reference to another (older) version of the Kinesis or AWS SDKs?
I installed DBfit as specified here: http://dbfit.github.io/dbfit/docs/getting-started.html
Configured the test and the database;
Ran the startFitnesse.bat;
Defined a test;
But when the I run the test the output is this:
Including the suggested variable: !define TEST_SYSTEM {slim} gives me this:
IF I DELETE THE !path lib/*.jar - IT STILL GIVES ME THE SAME.
I realize that DBfit can't find the .CLASS files - just don't know how to solve it.
Is the path variable incorrect?
Any suggestions on what to do will be rewarded.
P.S. my test script:
!define TEST_SYSTEM {slim}
!path lib/*.jar
!|dbfit.SQLServerTest|
!|Connect|localhost:2256|mindaugasb|MyPassword|TEST1_DB|
!|query|select * from dbo.Employees|
The problem was with !|dbfit.SQLServerTest| parameter. I have copied it from somewhere incorreclty !|dbfit.SqlServerTest|.
If anyone reads in this with similar issues here is a helpful forum where the issue was solved, and the responses where very prompt: https://groups.google.com/forum/#!topic/dbfit/4mrHAHvW4M0