Would it be possible to "emulate" the dynamic code generation that is done via Spring AOP, Dependency Injection and other Spring mechanisms when we recompile a java application to a native one? How about Hibernate and pure java reflection? Some of the tools that are capable of translating Java to native code include the GCC and the Excelsior compilers, but do they provide reliable native code that will work in the same way as does the Java application?
The whole story
Currently, we are facing a weird scenario at work. We have a server-side java web application that uses Spring with its dependency injection, messaging, MVC and AOP. The application is quite a large one, with complex structure and dependencies, involving lots of third parties and has a huge code base.
The problem is that we have to target a Windows system (Windows 7 embedded; Core2 Duo; 4GB RAM) that should host the above application. We have certain limitations (hardware, software and legal) which disallow us to use pure java code on the target hardware. The hardware operates a Windows operating system, and the only allowed option for us is a native executable (EXE).
So, we have to convert (a soft word for it) the java server application (or a minimal subset of it) to native one for the device. According to certain resources on the internet, this might be possible trough certain compilers that produce native (non-JAVA) binaries. However, due to severe difference between a dynamic platform (as Java is) and the static nature of the native one, some of the JVM features will not be available - for instance dynamic compilation and proxies (according to this page). So it seems a nearly impossible task for an application that is relying on dynamic fundamental technologies like Spring's DI, AOP and transaction management. Not to mention that we are using Hibernate as an ORM.
I must say that the whole application will be trimmed as much as possible and only the desired functionality will be left (as it is quite standalone - in terms of not relying on third party services to work), but still we cannot say goodbye to core concepts like Spring, Hibernate and AOP. Therefore, my question is: is it somehow possible successfully convert such an application to a native code (having the above properties and effective limitations)? Would it be possible to "emulate" the dynamic code generation that is done via the AOP (like writing our proxies as static instead if relying on the AOP) and other Spring mechanisms? Some of the tools that are mentioned to do this include the GCC and the Excelsior compilers. Do they provide reliable result with the dynamic technologies I mentioned? Is there any alternative besides rewriting the application on a more-static language like C/C++ (which is unacceptable for our time frame)? I'd also appreciate any arguments in support for not doing so, as we are currently just evaluating the opportunity. Any well-argumented rejection is acceptable for me, as myself personally have never done such native code conversion and am not aware of the full impacts of it over the software.
Your only real target option are things like IKVM and JNBridgePro -- however, i'm guessing a .net application is not really reasonable either given the way you described your constraints, and depending on the technical details of the application IKVM
In that case, you're pretty much asking for the impossible. There is no sane strategy for compiling java down to win32 native code that supports any significant subset of the average enterprise application tool chain.
It will be cheaper to re-write the application from scratch for the target platform then it will be to create the infrastructure and do the testing required to hack/port it from it's target runtime to a wholly unrelated foreign runtime.
UPDATE: Some things have changed: There is now graalvm which will generate native images for your java code. It's free and fancy and does largely what OP wanted.
This is a very interesting problem: How does the compiler determine dynamically loaded classes up front?
I took a look at the Excelsior documentation and it seems to handle this by Mixed Compilation Model. The JET runtime involves a JIT compiler that kicks in for dynamic scenarios. I think converting a dynamic Java program into native one is a really hard problem and not feasible to solve in your particular scenario.
Nevertheless, .NET Team recently showcased a new compile time technology that does what you asked for Microsoft managed apps (i.e for .NET not Java). Currently, it is only supported for Windows Store apps so it is not useful for your scenario. However, there is a blog you may want to check out: http://blogs.msdn.com/b/dotnet/archive/2014/05/20/net-native-deep-dive-dynamic-features-in-static-code.asp to get insight about their solution to this interesting problem.
They discuss the mechanism they introduced to support dynamic scenarios in a static compilation model. In a nutshell, the native .NET compiler does its best to predict all the classes that will be used by the app. For classes that can't be predicted statically, it requires "hints" to be supplied to the compiler. App developers and library developers specify runTime directives to enable dynamic scenarios for specified types. Runtime directives allow adding Reflection metadata or serialization support for specified types. In this model, there is no JIT, so you would run into a runtime exception if your type was not inferred by the compiler or not included in Runtime Directives xml file.
Related
Can I make a Java program to generate another java application at runtime.
I want to make a "installer" program, which takes user input and generates an application as per user requirement, instead of just configuring the pre-built application according to the user needs.
I came across this solution - how to compile & run java program in another java program?, but I don't want to make clients install JDK on there computer.
Dynamically create table and Java classes at runtime -
which also need JDK, but I got a work around:
ToolProvider.getSystemJavaCompiler() returns null - usable with only JRE installed?
Can I make a complete application using above methods?
Is it a bad idea to generate such program?
Can I make Spring and Hibernate applications like that?
Or is there any existing framework for doing so?
(if possible it should create tables in db and generate html files as well. I came across http://velocity.apache.org/, so is it possible to generate java code using that.)
Your goal doesn't make a lot of sense from a practical perspective. I hope that my answer will help you to understand why.
Can I make a java program to generate another java application at runtime.
Yes you can. But it is a lot of work, especially if the application if complicated.
I want to make a "installer" program, which takes user input and generate an application as per user requirement, instead of just configuring the pre-build application according to the user needs.
That is possible ... in theory.
The problem is that you have to write a program that is capable of reading and understanding the user's requirements, and can then converting those requirements into code. Normally ... this is what a programmer does. Writing a program to do what a programmer does is not practical. (My guess is that it is 20 or more years beyond the "state of the art" of artificial intelligence to do such a thing.)
Now if the problem domain was sufficiently restricted, and the requirements were tightly specified in an unambiguous notation, then it might be feasible to do this. However, benefits of generating a program rather than configuring an existing one (based on the same requirement notation) are pretty small. And probably not worth the effort.
... but I don't want to make clients install JDK on their computer.
If you are generating Java programs you need a Java compiler. So if you insist on using a JRE (in Java 8), you need to include a 3rd party Java compiler in your application.
However, for Java 9 onward this is moot:
Oracle no longer provides JRE distributions for Java 9+ so you would need to get your client to use a 3rd-party source for their JRE.
You could (should) be using the Java 9+ jlink utility to produce a custom JRE for you application, and that can include the standard Java compiler.
If you are trying to generate code at the bytecode level, your problem is immediately ten times harder.
Sorry, I am using Java 8
Are you aware that Java 8 is "end of life" for commercial use? That is likely to affect your clients.
Can I make a complete application using above methods?
Maybe yes, maybe no. It depends on the problem domain. The more complicated it is, and the more diverse / general the requirements, the harder it will be.
Is it a bad idea to generate such program?
Yes. It is a bad idea. It is a lot more work than writing an application that is configured in the conventional way. (Noting that the configuration could include writing plugins in Java, rules in some scripting language, and so on.)
I would advise only generating source code or bytecodes if you already have a conventional application with most / all of the required functionality that you can use as a prototype for the generated generated code. (If you can't write such a prototype by hand, then writing a generator that will create one is not realistic.)
And even when it is feasible, I would question the wisdom of building a generator. There doesn't seem to be a significant pay-off for the extra effort. (For example, where is the benefit for the end user?)
Can I make spring and hibernate application like that?
I don't see why you couldn't generate such an application. But see 1) and 2).
Or is there any existing frameworks for doing so?
There are frameworks that could be used in some cases:
Templating frameworks like Velocity1 can be used to generate Java source code.
Bytecode engineering frameworks could be used to generate code directly.
1 - Indeed, I have used Velocity for Java source code generation. It worked, though I'm not convinced it was an ideal solution.
Sure you can. You can also leverage a project like GraalVM to generate native binaries for a given platform.
However, it is a lot of work, and the end result won't probably be as useful as you think. Any use case you have in mind will probably be a lot better served by an app that you just configure to do different tasks, so your efforts are probably best spent in that direction.
I am an embedded programmer and working with an embedded JVM.
This enables running Java files on constrained devices.
These Java files are first compiled to bytecode into .class files which are then further optimized and uploaded to the device which has a micro JVM to run the optimized bytecode.
The micro JVM does not support all features, e.g., no reflection.
The main benefit is obvious: this allows programming in Java for constrained devices.
However, I was thinking that plenty of languages compile to bytecode, some are listed https://en.wikipedia.org/wiki/Java_bytecode.
So in theory these languages could also be used to program.
I'd like to obtain a list of common languages that compile down to bytecode and was wondering if you could help.
For example, Python has special implementations that reduce to Java Bytecode, if I'm not mistaken, and stuff like C to Java virtual machine compilers also exist.
So what languages would you think are logical to try and run on the devices? Any pointers on how to or similar experiences?
Also, I'm not clear what the difference is from reading Wikipedia between (Python) bytecode and Java bytecode, could anybody help explain that?
I'm agree with you about the overall idea and it would be nice to develop an embedded application using any language that can run on a JVM. But there are some practical issues that you should consider and I think that's why none of major vendors or open source initiatives have any active/serious project on this (as far as I know).
As you mentioned, a JVM implementations that can run on embedded devices, each have their own constraints and limitations. The most obvious one is that some packages may not be available at runtime. In order to apply such a constraint, you should either control it in the compile process or have a toolchain (sort of an SDK) which accepts the bytecode and checks such constraints.
This situation would be worth when a developer tries to use a third party library that is available for that specific language. It's not easy to guess if a library is safe for use against such a framework or not.
One great facility for developers would be to have their IDE check such issues on the fly (something like inspection in IntelliJ Idea). This makes it much more smoother to move toward using such a solution. But again the problem is that for each such languages there need to be a specific plugin compatible with their own syntax.
Also some of JVM languages that are actually implementation of an existing language (e.g. Jython or JRuby) are most of the time out of sync with the original language in case of supporting libraries/syntax changes of that language.
Anyway, I think in order to have a list of JVM languages you could easily find them on Wikipedia. Maybe you mean those who may worth considering in this regard by having a large community and tools support. In my opinion, you should focus on the following JVM languages as those who may worth to include in your final list:
Groovy
Kotlin
Scala
These are all pure JVM languages and are only using different syntax than Java.
Regarding the topic in general, I should say that when you search for embedded JVM implementations, you'll notice that it's also a fairly academic concepts and they're so many publications in this topics regarding the overall architecture, threading support, toolchain, error handling, memory management, etc. This means that you should have a very great experiences/background on both embedded systems and programming language concepts and implementation to be able to devise a proper architecture for such a platform.
About your last question regarding the difference between Python bytecode and Java bytecode (if I understand your question correctly), these are both conceptually the same but each has its own syntax and constraints. The bytecode concept refers to the piece of software that is the output of the compiler and is the platform independent representation of the original code and can be run/interpreted at runtime by another software component which is the virtual machine. In Java world, this software is called the Java Virtual Machine (JVM). I'm from the Java world so I don't know what it's called in Python vocabulary but it should be something similar (e.g. Python virtual machine).
I think due to the complexity of developing such a toolchain and also considering the unprecedented development of new IoT and SoC devices, many of them capable of running a more higher level operating systems, maybe in a long run most developers prefer to develop for a more high end devices using more high level APIs and SDKs. Who knows! In that case, we would have a same situation that we're in today for PCs. Languages like C and Assembly are still in use, but they have their own domain of applications. I mean throughout the time, layers of abstraction are being added on top of the previous one. The same thing can happen for embedded devices.
We want to use existing C# sources within our Java project. So far, this would not be a great problem since using e.g. Java Native Interface (JNI) is quite straight forward.
The problem is that the software shall also run on non-windows OS. So, we can compile the C# sources with Mono in order to make them executable on e.g. Linux. But how about the integration within Java? JNI or any COM-based solutions for C# <-> Java interoperability are OS-dependent and only work e.g. on Windows.
One possible solution would be the implementation of webservices. Has anybody another idea of how to solve this problem? I would be very thankful for alternative suggestions!
Thanks very much!
Regards
This is maybe not an "answer" as such, more a bit of discussion of how I viewed a similar (I think) situation.
I had a major investment in a C#/.Net-based client-server style system. So when I decided that I also wanted to support an Android "client" app I looked into various options. To me the most important factor was to maintain my C# classes as the defining classes for the object interchange between the existing system and the to-be-written Java Android app.
What I eventually settled on, and tweaked to my liking, was a system where Google Protocol Buffers is the interchange media. (If you're not familiar with them they are a sort of JSON-like interchange format.)
https://developers.google.com/protocol-buffers/
At the .Net end I use ProtoBuf-Net, written by Marc Gravell (he works here at SO, I believe). It includes the ability to take .Net objects and generate .proto files, the defining file for Protocol Buffers.
https://code.google.com/p/protobuf-net/
At the Android end I use ProtoStuff, written by David Yu. There is a part of his code that takes a .proto file and generates the corresponding Java classes.
https://code.google.com/p/protostuff/
One problem I encountered was that this didn't work well for my .Net classes that are derived classes, which was most of them. I created a workaround that is described in my comment to the answer here:
How to get protobuf-net to flatten and unflatten inherited classes in .Net?
This is now working to my satisfaction.
Note that I haven't talked at all about how the Android app connects to the Windows-based system and how the communications is performed. That was secondary for me - my primary consideration was making the C# class definitions the definitive definitions and having Java classes created from them automatically, and then the object-to-object interchange. (In the event I'm using a home-made TCP/IP communications link, but the actual communications could be anything, probably also web services.)
Hope this helps.
So I did a lot of research on this topic and want to share my findings with you:
One (from a technical point very attractive) option is to use commercial bridges between Java and .Net. For sure, the most popular products are JNBridge and Javonet. Both products seem to be quite easy-to-use, have good support and seem to be very sophisticated. Especially JNBridge already supports bridging between Java and Mono too, which allows the portation to also non-Windows OS, which is one of our main requirements as stated above. Javonet also wants to integrate Mono and is going to release this feature soon. However, both solutions are commercial and one needs to weigh their features against the respective costs. Nevertheless, from a pure technical point of view, they look great and also state to enable very fast communication between Java and .Net (faster than with web services).
Another option is to connect Java and .NET via COM. Since COM is generelly defined platform-independently, this could work on multiple OS. There are lots of open source projects that could be used for such an implementation, such as EZJCOM, J-Interop, JACOB or JCOM. The main restriction (expecially for our project) is that Mono only supports COM-interoperability under Windows (yet). So, this is not really an option for us. But if you want to create Java-.NET interoperability on Windows only, this is a good way.
The straighforward way of integrating Java and C# is to use Java Native Interface (JNI). You can also find manifold implementations that make JNI more easy to use, the most popular one is probably jni4net which seems to be a very active and frequently used project. But there are also others with specific pros and cons, such as Caffeine, Espresso or csjni. Finally, JNI is not 100% platform independet. It is applicable on different platforms, but you have to generate platform-specific code which makes it clearly less usable for our purposes. If you limit your application to Windows, jni4net seems to be a very good choice.
The third option could be to run both the Java and the .Net part within a Common Language Runtime. Ikvm.net is one possible and very popular solution therefore (as mentioned above by Samuel Audet). The drawback of this option is the loss of features and efficiency of the JDK.
The last and surely most generic alternative is to set up webservices between the Java and the .Net world. For this solution, one needs to find appropriate ways for serializing/deserializing objects from/to Java and .Net. There are manifold possible solutions for that available. RenniePet mentioned a sophisticated solution based on Protocol Buffers. Others exist as well such as http://java-cs-bridge.sourceforge.net/. This option might have a potential drawback when considering communication runtime, but may be the way to go for us.
Hope this may help anyone in the future that is confronted with the same problem.
I have an open source Java database migration tool (http://www.liquibase.org) which I am considering porting to .Net.
The majority of the tool (at least from a complexity side) is around logic like "if you are adding a primary key and the database is Oracle use this SQL. If database is MySQL use this SQL. If the primary key is named and the database is Postgres use this SQL".
I could fork the Java codebase and covert it (manually and/or automatically), but as updates and bug fixes to the above logic come in I do not want to have to apply it to both versions. What I would like to do is move all that logic into a form that can be compiled and used by both Java and .Net versions naively.
The code I am looking to convert does not contain any advanced library usage (JDBC, System.out, etc) that would vary significantly from Java to .Net, so I don't think that will be an issue (at worst it can be designed around).
So what I am looking for is:
A language in which I can code common parts of my app in and compile it into classes usable by the "standard" languages on the target platform
Does not add any runtime requirements to the system
Nothing so strange that it scares away potential contributors
I know Python and Ruby both have implementations on for the JVM and CLR. How well do they fit my requirements? Has anyone been successful (or unsuccesful) using this technique for cross-platform applications? Are there any gotcha's I need to worry about?
Check out the Fantom programming language. It has its own Java-like/C#-like syntax but can target either the Java VM or .NET CLR.
Their "Why Fantom" page gives a high-level overview of their approach to portability versus dynamic languages running on a VM.
You might have some luck using IKVM.NET. I'm not sure on its exact status, but it's worth a try if you're insistent on running Java code on the .NET Framework. It includes a .NET implementation of the Java base class library, so it seems reasonably complete.
The only other option I might suggest is porting the code to the J# language, a full .NET language (although not first class in the sense that C# or VB.NET is). The language was designed so that the differences with Java were minimal.
If you are thinking about an emdedded approach, you might look at Lua.
Alright everyone,
Let's say i wanted to make a system that used only assembly and java, my question would be, as long as i included all of the jvm folders, classes, jars, etc.. java should still function effectively?
i understand there are those things that are compiled platform specifically but this is why i am asking, is it possible, using assembly to replicate all of the .exe, or other executable files that java has included into a pure assembly/java system?
If you are asking whether it is possible to build a system in Java and assembly language that runs on "bare metal", the answer is yes. There are a couple of current examples:
JavaOS is targeted primarily at the embedded systems domain. (Sun consider SunOS to be a "legacy" product line these days.)
JNode aims are broader, and encompass embedded systems, desktop systems, servers and cloud computing.
Be aware that building a system of this kind is a multi-year, multi-person project requiring deep understanding of virtual machine internals, compilers, garbage collectors, hardware architectures, device driver writing and so on.
If you are asking about something else, please be more explicit.
EDIT: responding to the OP's followup question:
It is not practical to use the Java and other "exe" files per se. They require a fully fledged operating system underneath them; e.g. Windows, Linux, whatever. If you had access to the source code, you could conceivably rewrite as required to make them run on "bare metal", but that would entail significant architectural changes, especially if you want to write device drivers, etc in Java. (Besides, the core of Sun's JRE is implemented in C++ ... ).
You cannot directly use the existing Java class library JAR files, because they include a significant amount of platform specific code. However, you can build your own Java class library JARs from an existing open-source version of the Java class libraries (e.g. the OpenJDK 6.0 J2SE libraries). You deal with the platform specific code by providing your own versions as native libraries or (as JNode does) as Java classes.
If I understand your question correctly, you mean something like JavaOS. Sure, its possible to implement the JVM raw on the hardware, not sure why you would, though. And if you did, why you wouldn't use C instead of Assembly for most of the work.
Its theoretically possible to implement the jvm in a whole other language. The best example I can think of is Python/Jpython where there is the original C implementation and a pure Java implementation of the language.
The main argumant against this is -- its a ton of work for not much benefit.
The official Sun jvm and supporting jni libraries are written mostly in C, you would need to provide native assembler implementations for most of the C POSIX APIs at the very least.
Also the original design goal of C was 'a portable assembly language' and to a large extent it still meets these goals. C produces efficient machine code and most C compilers will let code machine instructions inline with the C code.
Another benefit of C is the number of cross compilers available, you dont need to run the development environment on tHe target architecture, you can deveop and unit test on your favourite paltform/IDE, when you are ready you can then export your executables to the target platform.
Jikes RVM and Sun's Maxine provide a JVM implementation with little (of the order of 1 kloc) native code. However, both VMs require an OS and are only research implementations. The process of creating a stream of octets that form machine code, is obviously achievable in Java.
Have a look at JNode. They have been working on this for years.
http://www.jnode.org/