Say, I have 2 .xqy files. One .xqy references functions from another file by importing module.
I want to port my .xqy files to use Java API of BaseX. I found some Java_Examples of BaseX. In RunQueries.java the query is placed in a string variable. Could I reference a function in another module like:
String query = "mymodule:testFunction()";
// Process the query by using the database command
System.out.println("\n* Use the database command:");
query(query);
Related
I am wondering if using loadjava to load the Java package called JSch.jar in an Oracle database and then loading another .java file, that utilizes the JSch package to connect over SSH, would be able to be executed within an Oracle database through a function or procedure.
I ask this before trying because I need to reach out to a DBA to try and load everything. I want to make sure it is doable because I am not very skilled in java as of yet and wouldn't know if something was impossible or if it just needs fixed.
Thanks.
Yes
Use something like:
loadjava -user USERNAME/PASSWORD#SID JSch.jar
Then create a static class method which uses the classes loaded from the Jar file:
CREATE OR REPLACE AND COMPILE JAVA SOURCE NAMED MyJavaSource AS
import org.millea9805.jsch.JSchSomething;
public class MyClass {
public static String function_name()
{
JSchSomething.doSomething();
return "Something";
}
}
/
Then you can create a PL/SQL wrapper around the static Java method:
CREATE OR REPLACE FUNCTION DO_SOMETHING()
RETURN VARCHAR2
AS LANGUAGE JAVA
NAME 'MyClass.function_name() return java.lang.String';
/
A more detailed example using the XZ library to unzip BLOBs is here.
in order to make use of Machine Learning in Java, I'm trying to train a model in TensorFlow, save it as ONNX file and then use the file for inference in Java. While this works fine with simple models, it's getting more complicated using pre-processing layers, as they seem to depend on custom operators.
https://www.tensorflow.org/tutorials/keras/text_classification
As an example, this Colab deals with text classification and uses an TextVectorization layer this way:
#tf.keras.utils.register_keras_serializable()
def custom_standardization2(input_data):
lowercase = tf.strings.lower(input_data)
stripped_html = tf.strings.regex_replace(lowercase, '<br />',' ')
return tf.strings.regex_replace(stripped_html, '[%s]' % re.escape(string.punctuation), '')
vectorize_layer = layers.TextVectorization(
standardize=custom_standardization2,
max_tokens=max_features,
output_mode='int',
output_sequence_length=sequence_length
)
It is used as pre-processing layer in the compiled model:
export_model = tf.keras.Sequential([
vectorize_layer,
model,
layers.Activation('sigmoid')
])
export_model.compile(loss=losses.BinaryCrossentropy(from_logits=False), optimizer="adam", metrics=['accuracy'])
In order to create the ONNX file I save the model as protobuf and then convert it to ONNX:
export_model.save("saved_model")
python -m tf2onnx.convert --saved-model saved_model --output saved_model.onnx --extra_opset ai.onnx.contrib:1 --opset 11
Using onnxruntime-extensions it is now possible to register the custom ops and to run the model in Python for inference.
import onnxruntime
from onnxruntime import InferenceSession
from onnxruntime_extensions import get_library_path
so = onnxruntime.SessionOptions()
so.register_custom_ops_library(get_library_path())
session = InferenceSession('saved_model.onnx', so)
res = session.run(None, { 'text_vectorization_2_input': example_new })
This raises the question if it's possible to use the same model in Java in a similar way. Onnxruntime for Java does have a SessionOptions#registerCustomOpLibrary function, so I thought of something like this:
OrtEnvironment env = OrtEnvironment.getEnvironment();
OrtSession.SessionOptions options = new OrtSession.SessionOptions();
options.registerCustomOpLibrary(""); // reference the library
OrtSession session = env.createSession("...", options);
Does anyone have an idea if the use case described is feasable or how to use models with pre-processing layers in Java (without using TensorFlow Java)?
UPDATE:
Spotted a potential solution. If I understand the comments in this GitHub Issue correctly, one possibility is to build the ONNXRuntime Extensions package from source (see this explanation) and reference the generated library file by calling registerCustomOpLibrary in the ONNX Runtime Library for Java. However, as I have no experience with tools like cmake this might become a challenge for me.
The solution you propose in your update is correct, you need to compile the ONNX Runtime extension package from source to get the dll/so/dylib, and then you can load that into ONNX Runtime in Java using the session options. The Python whl doesn't distribute the binary in a format that can be loaded outside of Python, so compiling from source is the only option. I wrote the ONNX Runtime Java API, so if this approach fails open an issue on Github and we'll fix it.
Has anyone attempted to "link" in the Rascal command line jar in a java executable and call REPL commands from this java executable?
I found a similar question on stackoverflow (Running a Rascal program from outside the REPL), but that doesn't go into details unfortunately.
I also checked the Rascal tutor site, but couldn't find any examples on how to do this. Tijs told me that it's something along the lines of "instantiate an interpreter and then call the import() function, after which the call() function can be called to inject REPL commands).
Is there any example code on how to do, e.g. the following from the tutor site on the REPL but from a java programming context instead of on the command line:
rascal>import demo::lang::Exp::Concrete::NoLayout::Syntax;
ok
rascal>import ParseTree;
ok
rascal>parse(#Exp, "2+3");
sort("Exp"): `2+3`
The following would do the trick; a utility class for the same can be found in rascal/src/org/rascalmpl/interpreter/JavaToRascal.java:
GlobalEnvironment heap = new GlobalEnvironment();
IValueFactory vf = ValueFactoryFactory.getValueFactory();
TypeFactory TF = TypeFactory.getInstance();
IRascalMonitor mon = new NullRascalMonitor();
Evaluator eval = new Evaluator(vf, new PrintWriter(System.err), new PrintWriter(System.out), new ModuleEnvironment(ModuleEnvironment.SHELL_MODULE, heap), heap);
eval.doImport(mon, "demo::lang::Exp::Concrete::NoLayout::Syntax");
eval.doImport(mon, "ParseTree");
eval.eval(mon, "parse(#Exp, \"2+3\");", URIUtil.rootLocation("unknown"));
There is also more efficient ways of interacting with the evaluator, via the pdb.values IValue interfaces to build data and ICalleableValue to call Rascal functions. You can use the above heap object to query its environments to get references to functions and you can use the low level pdb.values API to construct values to pass to these functions.
Caveat emptor: this code is "internal" API with no guarantee for backward compatibility. I can guarantee that something like this will always be possible.
I want to extend the Sysdeo Tomcat Plugin to be capable of resolving variables like you can specify in run configurations for Java applications, e.g. ${workspace_loc}.
I have tried to use this, but the resulting array has no contents:
ResourcesPlugin.getWorkspace().getPathVariableManager().getPathVariableNames();
Using this one does not give me the wanted variables:
JavaCore.getClasspathVariableNames();
Further I search for the code doing the actual variable replacement in a string.
Use the IStringVariableManager to access variables such as ${workspace_loc}. You get the manager with:
IStringVariableManager manager = VariablesPlugin.getDefault().getStringVariableManager();
To process the variables in a string use:
String newString = manager.performStringSubstitution(string);
You can also use the org.eclipse.core.variables.dynamicVariables and org.eclipse.core.variables.valueVariables extension points to add new variables.
I need to get a list of chunks after sharding inside my Java code. My code is simple and looks like this:
Mongo m = new Mongo( "localhost" , 27017 );
DB db = m.getDB( "admin" );
Object cr = db.eval("db.printShardingStatus()", 1);
A call of eval() returns an error:
Exception in thread "main" com.mongodb.CommandResult$CommandFailure: command failed [$eval]: { "serverUsed" : "localhost/127.0.0.1:27017" , "errno" : -3.0 , "errmsg" : "invoke failed: JS Error: ReferenceError: printShardingStatus is not defined src/mongo/shell/db.js:891" , "ok" : 0.0}
at com.mongodb.CommandResult.getException(CommandResult.java:88)
at com.mongodb.CommandResult.throwOnError(CommandResult.java:134)
at com.mongodb.DB.eval(DB.java:340)
at org.sm.mongodb.MongoTest.main(MongoTest.java:35)
And, really, if we look into the code of db.js, in line 891 there is a call to a method printShardingStatus() that is not defined inside a file. Inside of sh.status() method in utils_sh.js file, there is even a comment:
// TODO: move the actual commadn here
Important to mention, when I run these commands in mongo command line, everything works properly!
My questions are:
Is there any other possibility of getting a full sharding status within Java code? (eg. with DB.command() method)
If not, any other suggestions how to avoid my problem?
Many of the shell's helper functions are not available for server-side code execution. In the case of printShardingStatus(), it makes sense because there isn't a console to use for printing output and you'd rather have a string returned. Thankfully, you should be able to pull up the source of the shell function and reimplement it in your application (e.g. concatenating a returned string instead of printing directly).
$ mongo
MongoDB shell version: 2.2.0
connecting to: test
> db.printShardingStatus
function (verbose) {
printShardingStatus(this.getSiblingDB("config"), verbose);
}
So, let's look at the printShardingStatus() function...
> printShardingStatus
function (configDB, verbose) {
if (configDB === undefined) {
configDB = db.getSisterDB("config");
}
var version = configDB.getCollection("version").findOne();
// ...
}
Before turning all of the output statements into string concatenation, you'd want to make sure the other DB methods are all available to you. Performance-wise, I think the best option is to port the innards of this function to Java and avoid server-side JS evaluation altogether. If you dive deeper into the printShardingStatus() function, you'll see it's just issuing find() on the config database along with some group() queries.
If you do want to stick with evaluating JS and would rather not keep this code within your Java application, you can also look into storing JS functions server-side.
Have you deployed a shard cluster properly?
If so, you could connect to a mongo database that has sharding enabled.
Try calling the method db.printShardingStatus() with a that database within the mongo shell and see what happens.
Apparently the Javascript function 'printShardingStatus' is only available for the mongo shell and not for execution with server commands, to see the code start mongo.exe and type only 'printShardingStatus' and press enter.
In this case writing an extension method would be the best for solving this...
Javascript way of printing output of MongoDB query to a file
1] create a javascript file
test.js
cursor = db.printShardingStatus();
while(cursor.hasNext()){
printjson(cursor.next());
}
2] run
mongo admin --quiet test.js > output.txt