All versions of Java require the user to close resources manually - usually handled in the finally block. Java 7 is about to get ARM (automatic resource management) blocks.
Firstly I don't see a finally block concept in python. Or do you close resources in the catch for each raised exceptions ?
Is there a library that performs ARM in python ? If not, then what is the pythonic way to do ARM ?
There is a try: except: finally: in python.
You can also use the with: statement, which I believe is what you are after for ARM. These are called context managers. http://www.python.org/dev/peps/pep-0343/
There is some automated resource management in Python. Most objects who open resources will close them when they get garbage collected. When that happens is undefined, and it may not happen at all, so this only works if you don't use very many resources, don't care if they are open long and the resources will be closed by the operating system when the program exits.
Otherwise, use context managers and the with statement as per Matthews answer.
Here is a simple example that redirects stdout:
>>> import sys
>>> class redirect_stdout:
... def __init__(self, target):
... self.stdout = sys.stdout
... self.target = target
...
... def __enter__(self):
... sys.stdout = self.target
...
... def __exit__(self, type, value, tb):
... sys.stdout = self.stdout
...
>>> from StringIO import StringIO
>>> out = StringIO()
>>> with redirect_stdout(out):
... print 'Test'
...
>>> out.getvalue() == 'Test\n'
True
Related
I want to parallelize my data writing process. I am writing a data frame to Oracle Database. This data has 4 million rows and 8 columns. It takes 6.5 hours without parallelizing.
When I try to go parallel, I get the error
Error in checkForRemoteErrors(val) :
7 nodes produced errors; first error: No running JVM detected. Maybe .jinit() would help.
I know this error. I can solve it when I work with single cluster. But I do not know how to tell other clusters the location of Java. Here is my code
Sys.setenv(JAVA_HOME='C:/Program Files/Java/jre1.8.0_181')
library(rJava)
library(RJDBC)
library(DBI)
library(compiler)
library(dplyr)
library(data.table)
jdbcDriver =JDBC("oracle.jdbc.OracleDriver",classPath="C:/Program Files/directory/ojdbc6.jar", identifier.quote = "\"")
jdbcConnection =dbConnect(jdbcDriver, "jdbc:oracle:thin:#//XXXXX", "YYYYY", "ZZZZZ")
By using Sys.setenv(JAVA_HOME='C:/Program Files/Java/jre1.8.0_181') I solve the same problem for single core. But when I go parallel
library(parallel)
no_cores <- detectCores() - 1
cl <- makeCluster(no_cores)
clusterExport(cl, varlist = list("jdbcConnection", "brand3.merge.u"))
clusterEvalQ(cl, .libPaths("C:/Users/onur.boyar/Documents/R/win-library/3.5"))
clusterEvalQ(cl, library(RJDBC))
clusterEvalQ(cl, library(rJava))
parLapply(cl, 1:length(brand3.merge.u$CELL_PH_NUM), function(x) dbSendUpdate(jdbcConnection, "INSERT INTO xxnvdw.an_cust_analytics VALUES(?,?,?,?,?,?,?,?)", brand3.merge.u[x, 1], brand3.merge.u[x,2], brand3.merge.u[x,3],brand3.merge.u[x,4],brand3.merge.u[x,5],brand3.merge.u[x,6],brand3.merge.u[x,7],brand3.merge.u[x,8]))
#brand3.merge.u is my data frame that I try to write.
I get the above error and I do not know how to set my Java location for other nodes.
I want to use parLapply since it is faster than foreach. Any help would be appreciated. Thanks!
JAVA_HOME environment variable
If the problem really is with the location of Java, you could set the environment variable in your .Renviron file. It is likely located in ~/.Renviron. Add a line to that file and this will be propagated to all R session that run via your user:
JAVA_HOME='C:/Program Files/Java/jre1.8.0_181'
Alternatively, you can just add that location to your PATH environment variable.
JVM Initialization via rJava
On the other hand the error message may point to just a JVM not being initialized, which you can solve with .jinit, a minimal example:
library(parallel)
cl <- makeCluster(detectCores())
parallel::parLapply(cl, 1:5, function(x) {
rJava::.jinit()
rJava::.jnew(class = "java/lang/Integer", x)$toString()
})
Working around Java use
This was not specifically asked, but you can also work around the need for Java dependency using ODBC drivers, which for Oracle should be accessible here:
con <- DBI::dbConnect(
odbc::odbc(),
Driver = "[your driver's name]",
...
)
I need to find a solution to a complex system of linear equations.
Current implementation is in C++ and calls LAPACK zgesv function.
I thought about rewriting it in Scala and using Breeze for linear algebra.
I try an example
import breeze.linalg._
import breeze.math._
val a = DenseMatrix(
(Complex(-1.34, 2.55), Complex(0.28, 3.17), Complex(-6.39, -2.20), Complex(0.72, -0.92)),
(Complex(-0.17, -1.41), Complex(3.31, -0.15), Complex(-0.15, 1.34), Complex(1.29, 1.38)),
(Complex(-3.29, -2.39), Complex(-1.91, 4.42), Complex(-0.14, -1.35), Complex(1.72, 1.35)),
(Complex(2.41, 0.39), Complex(-0.56, 1.47), Complex(-0.83, -0.69), Complex(-1.96, 0.67))
)
val b = DenseVector(Complex(26.26, 51.78), Complex(6.43, -8.68), Complex(-5.75, 25.31), Complex(1.16, 2.57))
val x = a \ b
I get an error
Error:(14, 17) not enough arguments for method : (implicit op: breeze.linalg.operators.OpSolveMatrixBy.Impl2[breeze.linalg.DenseMatrix[breeze.math.Complex],breeze.linalg.DenseVector[breeze.math.Complex],That])That.
Unspecified value parameter op.
lazy val x = a \ b
^
Looks like Breeze doesn't support complex type for this operation.
I also found that complex routines are not supported by netlib-java, which is used by Breeze.
Do I understand correctly and it is currently not supported by Breeze?
Sorry, Breeze doesn't support lapack calls for Complex numbers right now.
In both the Joy of Clojure and on Alex Miller's Pure Danger Tech blog-post it is recommended that you can print the last stack using something like the following:
(use 'clojure.stacktrace)
(java.util.Date. "foo")
(.printStackTrace *e 5)
But I can't get any of their examples to work, and instead just get
java.lang.NullPointerException: null
Reflector.java:26 clojure.lang.Reflector.invokeInstanceMethod
(Unknown Source) jtown$eval9755.invoke
What's up with this? .printStackTrace seems to be a Java function from the looks of it, so I am not sure why I am bringing clojure.stacktrace into my namespace, in the first place. I read through the clojure.stacktrace API, though, and see an e function, which seems similar too but is not the *e function, which is in core and is supposed to be binding to the last exception, but isn't. Could somebody straighten me out on the best way to check stack-traces?
There are some special vars available when using the REPL and
*e - holds the result of the last exception.
For instance:
core=> (java.util.Date. "foo")
IllegalArgumentException java.util.Date.parse (Date.java:615)
core=> (class *e)
java.lang.IllegalArgumentException
core=> (.printStackTrace *e)
java.lang.IllegalArgumentException
at java.util.Date.parse(Date.java:615)
<not included.....>
You are right, .printStackTrace is the java method that is invoked on the exception class. This is not very straightforward (since its java interop) so clojure.stacktrace namespace has some utilities about working with stack traces
So after
(use 'clojure.stacktrace)
you can use the stacktrace library instead of java interop:
core=> (print-stack-trace *e)
java.lang.IllegalArgumentException: null
at java.util.Date.parse (Date.java:615)
<not included.....>
Obviously in an app, instead of *e, you can do a try - catch and use the related functions as necessary
I use
(.printStackTrace *e *out*)
That seems to work.
I have a Perl script that uses Inline::Java and just has to fork (it is a server and I want it to handle multiple connections simultaneously).
So I wanted to implement this solution which makes use of a shared JVM with SHARED_JVM => 1. Since the JVM is not shutdown when the script exits, I want to reuse it with START_JVM => 0. But since it might just be the first time I start the server, I would also like to have a BEGIN block make sure a JVM is running before calling use Inline.
My question is very simple, but I couldn't find any answer on the web: How do I simply start a JVM? I've looked at man java and there seems to be just no option that means "start and just listen for connections".
Here is a simplified version of what I'm trying to do in Perl, if this helps:
BEGIN {
&start_jvm unless &jvm_is_running;
}
use Inline (
Java => 'STUDY',
SHARED_JVM => 1,
START_JVM => 0,
STUDY => ['JavaStuff'],
);
if (fork) {
JavaStuff->do_something;
wait;
}
else {
Inline::Java::reconnect_JVM();
JavaStuff->do_something;
}
What I need help with is writing the start_jvm subroutine.
If you've got a working jvm_is_running function, just use it to determine whether Inline::Java should start the JVM.
use Inline (
Java => 'STUDY',
SHARED_JVM => 1,
START_JVM => jvm_is_running() ? 0 : 1,
STUDY => ['JavaStuff'],
);
Thanks to details provided by tobyink, I am able to answer my own question, which was based on a the erroneous assumption that the JVM itself provides a server and a protocole.
As a matter of fact, one major component of Inline::Java is a server, written in Java, which handles request by the Inline::Java::JVM client, written in Perl.
Therefore, the command-line to launch the server is:
$ java org.perl.inline.java.InlineJavaServer <DEBUG> <HOST> <PORT> <SHARED_JVM> <PRIVATE> <NATIVE_DOUBLES>
where all parameters correspond to configuration options described in the Inline::Java documentation.
Therefore, in my case, the start_jvm subroutine would be:
sub start_jvm {
system
'java org.perl.inline.java.InlineJavaServer 0 localhost 7891 true false false';
}
(Not that it should be defined: tobyink's solution, while it did not directly address the question I asked, is much better.)
As for the jvm_is_running subroutine, this is how I defined it:
use Proc::ProcessTable;
use constant {
JAVA => 'java',
INLINE_SERVER => 'org.perl.inline.java.InlineJavaServer',
};
sub jvm_is_running {
my $pt = new Proc::ProcessTable;
return grep {
$_->fname eq JAVA && ( split /\s/, $_->cmndline )[1] eq INLINE_SERVER
} #{ $pt->table };
}
I am calling a function in a java library from jython which prints to stdout. I would like to suppress this output from the jython script. I attempt the python idiom replacing sys.stdout with a file like object (StringIO), but this does not capture the output of the java library. I'm guessing sys.stdout does not affect the java program. Is there a standard convention for redirecting or suppressing this output programatically in jython? If not what ways can I accomplish this?
You can use System.setOut, like this:
>>> from java.lang import System
>>> from java.io import PrintStream, OutputStream
>>> oldOut = System.out
>>> class NoOutputStream(OutputStream):
... def write(self, b, off, len): pass
...
>>> System.setOut(PrintStream(NoOutputStream()))
>>> System.out.println('foo')
>>> System.setOut(oldOut)
>>> System.out.println('foo')
foo
Note that this won't affect Python output, because Jython grabs System.out when it starts up so you can reassign sys.stdout as you'd expect.
I've created a context manager to mimic (Python3's) contextlib's redirect_stdout (gist here):
'''Wouldn't it be nice if sys.stdout knew how to redirect the JVM's stdout? Shooting star.
Author: Sean Summers <seansummers#gmail.com> 2015-09-28 v0.1
Permalink: https://gist.githubusercontent.com/seansummers/bbfe021e83935b3db01d/raw/redirect_java_stdout.py
'''
from java import io, lang
from contextlib import contextmanager
#contextmanager
def redirect_stdout(new_target):
''' Context manager for temporarily redirecting sys.stdout to another file or file-like object
see contextlib.redirect_stdout documentation for usage
'''
# file objects aren't java.io.File objects...
if isinstance(new_target, file):
new_target.close()
new_target = io.PrintStream(new_target.name)
old_target, target = lang.System.out, new_target
try:
lang.System.setOut(target)
yield None
finally:
lang.System.setOut(old_target)