I'm writing an application in java that will facilitate creating wireless sensor networks using off the shelf micro controllers, sensors, and radios. Each sensor and radio will most likely require unique code. I'm planning on creating skeletons for each platform and and then having modular bits of code for each sensor and radio that can be plugged into these skeletons. This will result in a library of static information that will be used to dynamically generate code for these sensors.
I'm not sure what the best way to store and organize this data would be. I started off trying to create classes for each sensor encapsulating its unique properties but using objects for data storage only seems weird. I feel like SQL would be overkill as the data isn't really changing and I would also like to keep everything in version control. Should I just use flat files? XML? Any advice on how to architect this project would be very welcome.
Instead of generating source, I'd go binary. Conceptually, that is.
Why would the source code need to change if a device is plugged in or out? Simply compile binary device driver libraries and link them to the main app.
There is an assembler, so likely there is a linker.
If there is no linker, and you are forced to use a monolithic source file, then at least we can use the concepts of a linker.
Linking Source Code
For inspiration and details I'd look into Operating System Design a little bit, for the concepts of device drivers, and IO devices, and network sockets. I'd use this to take a hard look at the source that would be generated, and what exactly changes if a device is changed, and fix it so that as little as possible, ideally nothing, has to be changed.
The code for the app running on the (presumably embedded) system should be maintained separate from the device drivers, so here is where the abstraction needs to begin. It needs to be refactored to abstract away the particulars of the devices into abstract device classes.
So this is the first step: refactor the generated source to abstract out the particulars of the device drivers so that you have a main
application that calls functions via symbols.
This allows the main app to work regardless of the number and kind of devices available.
Next, I'd look into compiler theory, particularly the concepts of symbol resolution and static/dynamic linking, and stub. Since the generated source is refactored so that there is a main application and a list of device drivers, all that is left is to make the devices available to the application.
Illustration
The application could generate the source code to be assembled by concatenating the source for the main application with the source for the device drivers.
It would provide a stub as well: a small library providing a function to iterate the devices and interrogate their classes.
Your application then becomes so simple that a one-liner on a *NIX prompt could do it. No Java required:
cat program stub drivers/foo drivers/bar > generated-source-or-binary
In it's simplest form, the program would contain a call to an iterate_devices label in stub.
Here's a layout of the source and/or binary image:
// application
main() {
for ( device in list_devices() ) {
switch ( device.class ) {
....
}
}
}
// stub
list_devices() {
for ( device = first; device != null; device += *cur )
yield device;
}
first: // drivers follow
// drivers/foo
dev_foo: .long dev_foo_end - . // size
....
dev_foo_end
// drivers/bar
dev_bar: .long dev_bar_end - .
....
dev_bar_end
Organizing Driver Sources
This shouldn't have to be more complicated than a directory with files.
A simple approach would be to include these in the .jar in a specific package. For instance, having a class provide driver sources like this:
package myapp.drivers;
public class DriverSource {
public static InputStream getDriverSource( String identifier ) {
return this.getClass().getClassLoader().getResourceAsStream(
this.getClass().getPackage().getName().replace('.', '/')
+ '/' + identifier + '.source'
);
}
}
would require the driver sources to be put in myapp/drivers/{identifier}.source. In a standard eclipse project, you'd place the files in src/myapp/drivers/. Using Maven, you'd put them in src/main/resources/myapp/drivers/. You can also put them in another directory, as long as they are copied as resources to the proper package directory.
The above class could also serve as a basis for more complex storage: you could query a remote service and download the source files, or query an SQL database. But resource files will be a decent start.
Related
I've been using Nashorn for awk-like bulk data processing. The idea is, that there's a lot of incoming data, coming row by row, one by another. And each row consists of named fields. These data are processed by user-defined scripts stored somewhere externally and editable by users. Scripts are simple, like if( c>10) a=b+3, where a, b and c are fields in the incoming data rows. The amount of data is really huge. Code is like that (an example to show the use case):
ScriptEngine engine = new NashornScriptEngineFactory().getScriptEngine(
new String[]{"-strict", "--no-java", "--no-syntax-extensions", "--optimistic-types=true"},
null,
scr -> false);
CompiledScript cs;
Invocable inv=(Invocable) engine;
Bindings bd=engine.getBindings(ScriptContext.ENGINE_SCOPE);
bd.remove("load");
bd.remove("loadWithNewGlobal");
bd.remove("exit");
bd.remove("eval");
bd.remove("quit");
String scriptText=readScriptText();
cs = ((Compilable) engine).compile("function foo() {\n"+scriptText+"\n}");
cs.eval();
Map params=readIncomingData();
while(params!=null)
{
Map<String, Object> res = (Map) inv.invokeFunction("foo", params);
writeProcessedData(res);
params=readIncomingData();
}
Now nashorn is obsolete and I'm looking for alternatives. Was googling for a few days but didn't found exact match for my needs. The requirements are:
Speed. There's a lot of data so it shall be really fast. So I assume as well, precompilation is the must
Shall work under linux/openJDK
Support sandboxing at least for data access/code execution
Nice to have:
Simple, c-like syntax (not lua;)
Support sandboxing for CPU usage
So far I found that Rhino is still alive (last release dated 13 Jan 2020) but I'm not sure is it still supported and how fast it is - as I remember, one of reasons Java switched to Nashorn was speed. And speed is very important in my case. Also found J2V8 but linux is not supported. GraalVM looks like a bit overkill, also didn't get how to use it for such a task yet - maybe need to explore further if it is suitable for that, but looks like it is complete jvm replacement and cannot be used as a library.
It's not necessary shall be javascript, maybe there are other alternatives.
Thank you.
GraalVM's JavaScript can be used as a library with the dependencies obtained as any Maven artifact. While the recommended way to run it is to use the GraalVM distribution, there are some explanations how to run it on OpenJDK.
You can restrict things script should have access to, like Java classes, creating threads, etc:
From the documentation:
The following access parameters may be configured:
* Allow access to other languages using allowPolyglotAccess.
* Allow and customize access to host objects using allowHostAccess.
* Allow and customize host lookup to host types using allowHostLookup.
* Allow host class loading using allowHostClassLoading.
* Allow the creation of threads using allowCreateThread.
* Allow access to native APIs using allowNativeAccess.
* Allow access to IO using allowIO and proxy file accesses using fileSystem.
And it is several times faster than Nashorn. Some measurements can be found for example in this article:
GraalVM CE provides performance comparable or superior to Nashorn with
the composite score being 4 times higher. GraalVM EE is even faster.
I am new to java and DLL-s
I need to access DLL's methods from java. So go easy on me.
I have tried using JNA to access the DLL here is what I have done.
import com.sun.jna.Library;
public class mapper {
public interface mtApi extends Library {
public boolean IsStopped();
}
public static void main(String []args){
mtApi lib = (mtApi) Native.loadLibrary("MtApi", mtApi.class);
boolean test = lib.IsStopped();
System.out.println(test);
}
}
When I run the code, I am getting the following error:
Exception in thread "main" java.lang.UnsatisfiedLinkError:Error looking up function 'IsStopped':The specified procedure could not be found.
I understand that this error is saying it cannot find the function, but I have no idea how to fix it.
I am trying to use this API mt4api
and here is the method, I am attempting to access MQL4
Can anyone tell me what I am doing wrong?
I have looked at other alternatives, like jni4net, but I cannot get this working either.
If anyone can link me to a tutorial that shows me how to set this up, or knows how to, I would be greatfull.
Trading?Hunting for milliseconds to shave-off?Go rather into Distributed Processing... Definitely safer than relying on API !
While your OP was directed onto how bend java to call .NET DLL-functions,
let me sketch a much future-safer solution.
Using AI/ML-regression based predictors for FOREX trading, I was hunting in the same forest. The best solution found within the last about 12-years, having spent about a few hundreds man*years of experience, was setup in the following manner:
Host A executes trades: operates MetaTrader Terminal 4, with both Script and EA --- the distributed-processing system communicates with with a use of ZeroMQ low-latency messaging/signalling framework ( about a few tens of microseconds needed )
Host B executes AI/ML processing of predictions for a traded instrument ( about a few hundreds of microseconds apply )
Cluster C executes continuous AI/ML predictor re-trainings and HyperParameterSPACE model selections ( many CPU-hours indeed needed, continuous model self-adapting process running 24/7 )
Signalling / Messaging layer with ZeroMQ has ports and/or bindings available and ready for most of the mainstream and many of niche programming languages, including java.
Hidden dangers of going just against a published API:
While the efforts for system integration and testing are immense, the API specifications are always dangerous for specification creeping.
This said, add countless man*months consumed on debugging after a silent change in MT4 language specifications that de-rail your previous tools + libraries. Why? Just imagine. Some time ago, MQL4 stopped to be MQL4 and was silently shifted towards MQL5, under a name New-MQL4. Among other changes in compilation, there were many small and big nails in the coffin -- string surprisingly ceased to be a string and was hidden as an internal struct -- which one could guess what will cause with all DLL-calls.
So, beware of API creepings.
Does it hurt a distributed processing solution?
No.
With a wise message-layout design, there are no adverse effects of MetaTrader Terminal 4 behaviour and all the logic ( incl. the strategy decision ) is put outside this creeping platform.
Doable. Fast and smart. Also could use remote-GPU-cluster processing, if your budget allows.
Does it work even in Strategy Tester?
Yes, it does.
If anyone has the gut to rely on the in-built Strategy Tester, the distributed-processing model still works there. Performance depends on the preferred style of modelling, a full one year, tick-by-tick simulation, with a quite complex AI/ML components took a few days on a common COTS desktops PC-systems ( after years of Quant R&D, we do not use Strategy Tester internally at all, but the request was to batch-test the y/y tick-data, so could be commented here ).
I have a Java application that I want to automate for testing. Unfortunately, the app window only registers as a SunAWTFrame, which means none of the controls are exposed to typical window analysis and automation tools.
My search has lead me to C# and utilising Java Access Bridge DLLS in a C# program to automate it.
Has anyone had any experience of this?
Oracle provides JavaAccessBridge (JAB) with some DLLS to help with this as I understand it after reading a few articles around the internet. There are some code examples but I'm really not groking it right now. By breaking it down, I think this is what needs to be achieved:
Import / load / parse the JAB dlls
Map functions in the JAB dll to methods / calls within my program
Have the Java application to automate run (with JAB enabled) and get handle of it to my program
Utilise the JAB functions to control the Java application
I don't know C# as well as I know Java, but that's not going to stop me.
If anyone can provide help, guidance, pointers or anything to get me started, that'd be truly awesome.
As of 2019 we have a great tool AccessBridgeExplorer created and open-sourced by google. It's a really good staring point that contains a WindowsAccessBridgeInterop project which encapsulates almost every JAB API into a class oriented, .NET friendly assembly.
One notable thing, AccessBridge.Initialize() must be called in WPF/WinForm UI thread or in your own messaging pump thread, otherwise some methods like AccessBridge.EnumJvms() will always return false/empty.
Building on Stackia's great tip to use Google's AccessBridgeExplorer, Here are some tips to get you going:
Download Access Bridge Explorer
Use the WindowsAccessBridgeInterop.dll in your own (WinForms not Console) project (Add> Project Reference> Select the DLL)
Create a new access bridge object
AccessBridge Java = new AccessBridge();
Initialize the Access Bridge object
Java.Initialize();
Call Application.DoEvents() - A hack to wait for Java.Initialize to complete (My simple understanding is Java Access Bridge Uses a hidden window or similar)
Application.DoEvents(); //this waits for Java Bridge to initilize ;)
Get the handle of the Java Window
(plenty of examples online of how to get a Window Handle in C#)
Get Access to the Java Object that represents the window:
Java.Functions.GetAccessibleContextFromHWND(Handle, out int vmid, out JavaObjectHandle javaObjectHandle);
Get AccessibleWindow Object for Window (so you can find its children)
AccessibleWindow win = Java.CreateAccessibleWindow(handle);
Come up with your own way to cycle through the children, and the childrens children until you find the object you are after:
//Similar to:
foreach(var child in win.GetChildren())
JavaObjectHandle? javaObject = Java.Functions.GetAccessibleChildFromContext(node.JvmId, parentJavaObject, child.GetIndexInParent());
//to get the label or title of the object:
child.GetTitle();
To Interact with an object (eg click a button), do similar to the following:
(please note where it says JavaObject - it means the child java object (eg. to click a button you need to get the JavaObject for that button using GetAccessibleChildFromContext as i mentioned above)
//Get Possible Actions
JavaAutomation.Java.Functions.GetAccessibleActions(VMID, JavaObject, out AccessibleActions accessibleActions);
foreach( var action in accessibleActions.actionInfo)
{
Log.Info($"DoAction: {action.name}");
}
AccessibleActionsToDo accessibleActionsToDo = new AccessibleActionsToDo();
accessibleActionsToDo.actions = accessibleActions.actionInfo;
accessibleActionsToDo.actionsCount = accessibleActions.actionsCount;
//Do Actions
JavaAutomation.Java.Functions.DoAccessibleActions(VMID, JavaObject, ref accessibleActionsToDo, out int failure);
I have to use already developed JAR in order to keep my code running. The JAR provides me with functionality of transforming file formats . My code looks something like this:
public class Transformer {
//some fields
//constructor
public List<MyFile> tranformFiles(List<MyFile> files){
JarClassUsed used = new JarClassUsed();
List<MyFile> data = new ArrayList<>();
foreach(MyFile file : files){
data.add(used.TransformFileFormat(file));
}
return data;
}
}
It's working fine, but still very slow. The problem is that the JAR I have to use is making a hidden/internal DB connection call when transforming files (reading predefined configurations). When count is around 100 is acceptable, but I have one case with more than 1000. And as already you guessed is quite problematic.
How can I keep this hidden connection Opened - still DB connections are expensive to create and Singleton or static sound like good solution. It's not smart to create it for each file that is being processed. Can I use reflection somehow here?
So we talked about it in the office... it seems the best course of action is to find a library that does the same formats, if thats not possible externalize the configuration and utilize a different library.
I know this does not answer your question in any way... and you probably already knew that... but unless you have the raw source and are allowed to change that jar, the "hidden" connection is something written into the library.
I really wish I had better advice, but I think the best course of action is to investigate other implementations.
If you can list the formats I would be more than happy to assist in this investigation in a different medium.
I need to create a desktop application that will run third party code, and I need to avoid the third party code from export by any way (web, clipboard, file io) informations from the application.
Somethig like:
public class MyClass {
private String protectedData;
public void doThirdPartyTask() {
String unprotedtedData = unprotect(protectedData);
ThirdPartyClass.doTask(unprotectedData);
}
private String unprotect(String data) {
// ...
}
}
class ThirdPartyClass {
public static void doTask(String unprotectedData) {
// Do task using unprotected data.
// Malicious code may try to externalize the data.
}
}
I'm reading about SecurityManager and AccessControler, but I'm still not sure what's the best approach to handle this.
What should I read about to do this implementation?
First of all, there is pretty much no way you can stop every information leak on a local computer. You can certainly restrict network access, and even a lot of file system access, but there is nothing that would stop the gui from popping up a dialog showing the information to the user on the screen, or any other 100 ways you could "leak" data.
Secondly, you keep talking about the policy file being changeable by the user. yes, it is. it sounds like you are basically trying to recreate DRM. I'd suggest reading up on DRM and the general futility of it. It essentially boils down to giving someone a locked box and the key to the box and telling them not to open it. If someone has physical access to your program, there is almost nothing you can do to stop them from getting data out of it, in java or pretty much any other programming language (at least, not on computers as they are built today).
A general approach would be to run your jvm with a security policy that grants java.security.AllPermission to your codebase (i.e. jar) and no permissions whatsoever to the third-party codebase. Here is some documentation on how to run with a policy file and what to put in said file.