I'm trying to make an addresslist in Java, which saves its contents in a Sqlite database.
Therefor (and for other future uses), I tried to create my own library for all kinds of database connections ("PentagonsDatabaseConnector-1.0.jar"). It currently supports Sqlite and MySql.
It references other libraries for them to provide the JDBC-drivers ("mysql-connector-java-8.0.16.jar" and "sqlite-jdbc-3.30.1.jar").
Problem: My Library works just fine if I'm accessing it from its own project folder, but as soon as I compile it and add it to the "Adressliste"-project, it isn't able to find the JDBC-drivers anymore (I can access the rest of my self-written library without problems though). Also, as shown in the screenshot, "PentagonsDatabaseConnector-1.0.jar" brings the JDBC-libraries with itself in "lib"-folder.
LINK TO THE SCREENSHOT
Do you guys have an idea whats wrong?
Thank you for your help!
Ps: Sorry for bad English, I'm German :)
Java cannot read jars-in-jars.
Dependencies come in a few flavours. In this case, PentagonsDC is a normal dependency; it must be there at runtime, and also be there at compile time.
The JDBC libraries are a bit special; they are runtime-only deps. You don't need them to be around at compile time. You want this, because JDBC libraries are, as a concept, pluggable.
Okay, so what do I do?
Use a build system to manage your dependencies is the answer 90%+ of java programmers go to, and what I recommend you do here. Particularly for someone starting out, I advise Maven. Here you'd just put in a text file the names of your dependencies and maven just takes care of it, at least at compile time.
For the runtime aspect, you have a few options. It depends on how your java app runs.
Some examples:
Manifest-based classpaths
You run your java application 'stand alone', as in, you wrote the psv main(String[]) method that starts the app and you distribute it everywhere it needs to run. In this case, the usual strategy is to have an installer (you need a JVM on the client to run your application and neither oracle nor any OS vendor supports maintaining a functioning JVM on end-user's systems anymore; it is now your job – this is unfortunately non-trivial), and given that you have that, you should deploy your jars such that they contain in the manifest (jars are zips, the manifest ends up at META-INF/MANIFEST.MF):
Main-Class: com.of.yourproj.Main
Class-Path: lib/sqlite-jdbc.jar lib/mysql-jdbc.jar lib/guava.jar
And then have a directory stucture like so:
C:\Program Files\yourapp\yourapp.jar
C:\Program Files\yourapp\lib\sqlite-jdbc.jar
C:\Program Files\yourapp\lib\mysql-jdbc.jar
Or the equivalent on any other OS. The classpath entries in the manifest are space separated and resolved relative to the dir that 'yourapp.jar' is in. Done this way, you can run yourapp.jar from anywhere and it along with all entries listed in Class-Path are now available to it.
Build tools can make this manifest for you.
Shading / Uberjars
Shading is the notion of packing everything into a single giant jar; not jars-in-jars, but unpack the contents of your dependency jars into the main app jar. This can be quite slow in the build (if you have a few hundred MB worth of deps, those need to be packed in and all class files need analysis for the shade rewrite, that's a lot of bits to process, so it always takes some time). The general idea behind shading is that deployment 'is as simple as transferring one jar file', but this is not actually practical, given that you can no longer assume that end users have a JVM installed, and even if they do, you cannot rely on it being properly up to date. I mention it here because you may hear this from others, but I wouldn't recommend it.
If you really do want to go for this, the only option is build systems: They have a plugin to do it; there is no command line tool that ships with java itself that can do this. There are also caveats about so-called 'signed jars' which cannot just be unpacked into a single uberjar.
App container
Not all java apps are standalone where you provide the main. If you're writing a web service, for example, you have no main at all; the framework does. Instead of a single entrypoint ('main' - the place where your code initially begins execution), web services have tons of entrypoints: One for every URL you want to respond to. The framework takes care of invoking them, and usually these frameworks have their own documentation and specs for how dependencies are loaded. Usually it is a matter of putting a jar in one place and its dependencies in a subdir named 'lib', or you build a so-called war file, but, really, so many web frameworks and so many options on how they do this. The good news is, usually its simple and the tutorial of said framework will cover it.
This advice applies to any 'app container' system; those are usually web frameworks, but there are non-web related frameworks that take care of launching your app.
Don't do these
Don't force your users to manually supply the -classpath option or mess with the CLASSPATH environment variable.
Don't try to write a custom classloader that loads jars-in-jars.
NB: Sqlite2 is rather complicated for java; it's not getting you many of the benefits that the 'lite' is supposed to bring you, as it is a native dependency. The simple, works everywhere solution in the java sphere is 'h2', which is written in all java, thus shipping the entire h2 engine as part of the java app is possible with zero native components.
Related
We have a web application made in Java, which uses struts2, spring and JasperReport. This application runs on glassfish 4.0.
The libraries of the application are in the WEB-INF/lib folder, and also in glassfish are installed 4 more than uses the same libraries.
Glassfish is configured to use 1024mb for heapspace and 512m for permgen, and the most of the memory consumption when i use libraries per application is in the struts actions and spring aop classes (using netbeans profiler).
The problem we are having is the amount of memory consumed by having libraries in the classloader per application because is to high and generates PermGen errors and we have also noticed that the application run slower with more users.
because of that we try to use shared-libraries, put it in domain1/lib folder and found that with a single deployed application the load time and memory consumption is much lower, and the application works faster in general. But when we deploy the rest of the applications on the server only the first application loaded works well and the rest has errors when we calls struts2 actions.
We believe that is because each application has slightly different settings on struts2 and log4j.
We have also tried to put only certain libraries on glassfish and leaving only struts2 in the application but it shows InvocationTargetException errors because all libraries depend the lib from apache-common and it dont matter if we put those lib on one place or another. Also if we put it in both places the application don’t start.
there any special settings or best practices for using shared-libraries?
Is there a way to use shared-libraries but load settings per application? or we have to change the settings to make them all the same?
Is there any special settings or best practices for using shared-libraries? Is there a way to use shared-libraries but load settings per application? or we have to change the settings to make them all the same?
These are actually interesting questions... I don't use GlassFish but, according to the documentation :
Application-Specific Class Loading
[...]
You can specify module- or application-specific library classes [...] Use the asadmin deploy command with the --libraries option and specify comma-separated paths
[...]
Circumventing Class Loader Isolation
Since each application or individually deployed module class loader universe is isolated, an application or module cannot load classes from another application or module. This prevents two similarly named classes in different applications or modules from interfering with each other.
To circumvent this limitation for libraries, utility classes, or individually deployed modules accessed by more than one application, you can include the relevant path to the required classes in one of these ways:
Using the Common Class Loader
Sharing Libraries Across a Cluster
Packaging the Client JAR for One Application in Another Application
Using the Common Class Loader
To use the Common class loader, copy the JAR files into the domain-dir/lib or as-install/lib directory or copy the .class files (and other needed files, such as .properties files) into the domain-dir/lib/classes directory, then restart the server.
Using the Common class loader makes an application or module accessible to all applications or modules deployed on servers that share the same configuration. However, this accessibility does not extend to application clients. For more information, see Using Libraries with Application Clients. [...]
Then I would try:
Solution 1
put all the libraries except Struts2 jars under domain1/lib ,
put only Struts2 jars under domain1/lib/applibs,
then run
$ asadmin deploy --libraries struts2-core-2.3.15.2.jar FooApp1.war
$ asadmin deploy --libraries struts2-core-2.3.15.2.jar FooApp2.war
To isolate Struts2 libraries classloading while keeping the rest under Common Classloader's control.
Solution 2
put all the libraries except Struts2 jars under domain1/lib ,
put only Struts2 jars under domain1/lib/applibs, in different copies with different names, eg appending the _appname at the jar names
then run
$ asadmin deploy --libraries struts2-core-2.3.15.2_FooApp1.jar FooApp1.war
$ asadmin deploy --libraries struts2-core-2.3.15.2_FooApp2.jar FooApp2.war
To prevent sharing of the libraries by istantiating (mock) different versions of them.
Hope that helps, let me know if some of the above works.
You can try to create what is known as a skinny WAR. Pack all your WARs inside an EAR and move all the common JARs from WEB-INF/lib to the lib/ folder in the EAR (don't forget to set <library-directory> in the application.xml).
I'd bet that placing the libs under lib/ or lib/ext won't resolve your performance issues. You did not write anything about the applications or server settings, like size of application, available Heap and PermGen space, but nonetheless I would recommend to stay with separate libs per app.
If you place the libs in server dirs, they will be shared among all apps. You will loose the option to upgrade only one of your applications to a new framework or to get rid away of any of them. Your deployment will be bound to a specific server architecture.
And you wrote it did not solve your problems, it even may raise new ones.
I would recommend to invest some hours into tuning the server. If it runs with defaults, allocate more PermGen and HeapSpace.
If this does not help, you should analyze in deep what's going wrong. Shared libs might be a solution, but you don't know the problem, yet. IBM offer some cool and free tools to analyze heap dumps, this could be a good starting point.
I came here in search of guidance about installing libraries that are shared among multiple applications or projects. I am deeply disappointed to read that the accepted practice favors installing a copy of every shared library into each project. So, if you have ten Web application, all of which use, e. g., httpcomponents-client, mysql-connector-java, etc., then your installation contains ten copies of each.
This behavior reminds me, painfully, of the way of thinking that motivated me to abandon the mainframe in favor of the PC; the thinking seemed to be "I don't care how many resources my application consumes. In fact, I'd like to be able to brag about what a resource hog it is." Excuse me, please, while I hurl.
The interface exposed by a library is an immutable contract that is not subject to change at the developer's whim.
There is this concept called backwards compatibility. If you break it, you create a new interface.
I know of at least two types of interfaces that adhere to the letter and spirit of these rules.
By far the oldest is the IBM System/370 system libraries. You might have Foo and Foo2, where the latter extends and/or breaks the contract made by the Foo interface in some way that made it incompatible.
From its beginnings in the Bell Labs Unix project, the standard C runtime library has adhered to the above rules.
Though it is much newer, the Microsoft COM interface specification enforces the same rule.
To their credit, Microsoft generally adheres to those rules in the Win32 API, too, although there are a handful of exceptions in that API. To a degree, they went backwards with the .NET Framework, which seems slavishly to follow in the footsteps of the Java environment that it so eagerly seeks to replace.
I've been using libraries since 1978, and my understanding was and is that the goal of putting code into a library was to make it reusable. While maintaining copies of the library code in each application eliminates the need to implement it again for each new project, it severely complicates upgrading, since you now have ten (or more) copies of the library, each of which must be updated.
If libraries adhere to the rule that an interface is an immutable contract, why shouldn't they live in a shared library directory, as do the Unix system libraries that live in its /lib directory, from which everything that runs on the host shares a single copy of the standard C runtime library, Zlib, and so forth.
Color me seriously disappointed.
I have a Swing/Java application that is being used for clients and has weekly updates.
The problem is the setup just lay out the classes and their respective directories and on the update I just update the classes.
I want to do a single jar containing all the classes, but I'm not sure how would I be able to update it...
Also some clients need some updates during the week where only one or two classes would be updated.
What is the right way of doing this ?
Im using Eclipse.
EDIT: I know how to create the jar, I just dont know how to dynamically update it.
I would suggest you look into Java WebStart which is designed to do exactly what you need.
You need to first create the finished deployment and then you can look into creating a JNLP file for the deployment, and a way to put all the files to a web server for the user to access. For now just let the user download the whole thing every time you update. When you get more experienced you can look into how you can make incremental updates.
I would strongly recommend including a build number or timestamp in the deployment paths so a jar file is not cached incorrectly in the client by accident.
The general way of doing this, even if only small changes were made, would be to repackage your JAR after each update and give that JAR to a user. The user would replace the existing JAR. How you produce your JAR is up to you. Many IDEs support it, you could write a shell script, use existing build systems like ant or maven or even make, whatever. (See edit below)
If your JAR is very large and deployment is cumbersome, you may be able to split your project into smaller subcomponents, each with their own JAR, and only redistribute the ones containing changes. That may or may not be a major refactoring for you, and it might not even be appropriate. For most small or average size projects, this is generally unnecessary.
As for deployment, there are also a zillion ways to make that easier on the user. You could just give them a JAR file. You could use e.g. install4j. You could use Java Web Start (although its kind of clunky). There are others.
Both install4j and JWS support automatically checking for updates. If you choose to support that feature, all you would need to do is update your distribution site, and users would receive updates automatically. That's also up to you.
But a short answer to your question is: If you have all of your classes packaged in a JAR, no matter how many classes change, you'll want to give the entire updated JAR to the user. The benefit that counters this cost is that a JAR is a nice, compressed, self-contained collection of your application/library's source, which makes management and deployment very convenient.
Edit: Responding to your edit where you specify that you are using Eclipse, Josh M gives you instructions in his comment on your answer. To add to his comment, to export a Runnable Jar you'll have to have a Run Configuration set up which, if you've been running your application in Eclipse already, you probably already have. If not you can create one in the Run menu.
Edit 2: Please refer to Thorbjørn Ravn Andersen's answer as well for some good JWS tips.
I am developing a java web application and that includes an applet. That applet is
dependent on two jar files:
JFreeChart (for plotting graphs at the client side) - 1.7 mb(size of
jar file)
MySqlJdbcConnector (for storing data, captured at the client side, to
a remote database) - .7 mb (size of
jar file)
Now, the problem is the size of above
two jar files. The total size of my
applet jar (myApplet.jar) is 2.5
mb out of which 2.4 mb is
because of the above two jar files.
I am not using all the classes in
those jar files. Specifically, for
jfreechart, I am using a very small number of classes from that
library.
Questions
Q1. For creating myApplet.jar file, what I have done is I have unzipped both of the jar files (jfreechart and mySQLJdbcConnector) and then packed the unzipped version of the jar files with the source code of my applet code to create one single jar file (i.e myApplet.jar). Is it the correct way of packing third party jar files with your applet code? Is there any way by which I can optimize this?
Q2. I tried to find the dependencies of the classes of jfreechart library which I am using in my application so as to pack only those dependencies in myApplet.jar. For that purpose, I used DependencyAnalyzer to find the dependencies of all the classes. But later I found it difficult to do so manually because every class (class of jfreechart that I am using in my application) has lot of dependencies and I am using some 15 classes of jfreechart so doing this for every class will be very difficult. So any suggestion on this?
Q3. Is this situation very common that developers encounter or I am missing something because of which I have to do this?
I'd suggest trying out ProGuard. You can exclude parts of jar files you're not using.
Yes you can save space by creating a JAR containing only the classes that your applet requires. (I've seen this called an uber-JAR.)
There are various tools for doing this; e.g. ProGuard, Zelix ClassMaster, a Maven plugin whose name I forget and so on.
There are however a couple of issues, at least in the general case:
If your code uses dynamic loading (e.g. by calling Class.forName(className)), these tools generally cannot detect the dependency. So to avoid dynamically loaded classes being left out of the final JAR, you need to tell the tool the names of all of all classes that your application might explicitly load that way.
You need to take a look at the license of the third party library. IIRC, some licenses require you to include the library in your distributed artifacts in a way that allows people to substitute a different version of the library. One could argue that an uber-JAR makes this hard to do, and therefore could be problematic.
JFreeChart is LGPL, and LGPL is a license that has the requirement above. However MySQL is GPL, which trumps LGPL, and this means that your applet must be GPL'ed ... if you distribute it.
Finally, if you want to minimize the size of your applet JAR, you should NOT include your source code in the JAR. Source code should be in a separate JAR (or ZIP, TAR or whatever) file.
A1:
You can create an ant script or use Eclipse or any other IDE to automatically package your applet. But your way is correct, too
A2:
I wouldn't do these things manually. Finding transitive dependencies is very complex. Maybe darioo's answer is a better way to do this.
A3:
This is very common indeed. A couple of hints:
You can always re-build those third party libraries without debug information. That should slightly decrease the size of those libraries.
On the other hand, maybe you shouldn't have a direct connection from your applet to a database. You could create an RMI interface (or something similar) to transfer your SQL and result data to an application server, which actually executes your SQL. This is an important security aspect for your applet, if you don't run this in a safe intranet.
I am very new to java and android development and to learn I am trying to start with an application to gather statistics and information like munin does. I am trying to be able to load "plugins" in my application. These plugins are already in the application but I don't want to have to invoke them all separately, but be able to iterate over them. I was trying to use serviceloader but could never get the META-INF/services into my apk. So I am wondering if it is possible to use serviceloader on android
Thanks
EDIT: I am asking about java.util.ServiceLoader, I think it should, but I can't figure out how to get my services folder into META-INF on the apk
There is an open bug report against this issue. See https://code.google.com/p/android/issues/detail?id=59658
The META-INF folder is deliberately excluded from the APK by ApkBuilder; the only comment in ApkBuilder.java is "we need to exclude some other folder (like /META-INF)" but there is no other explanation.
Even after adding META-INF with ant, you will still get in trouble if you want to use Proguard, which refuses to replace the content of META-INF/services/* files or rename them (that's another story, the author wants to keep Proguard agnostic).
However, people using maven may want to check https://github.com/pa314159/maven-android-plugin (the branch named "modified"), that tries to solve both issues. It is a fork from the original "android-maven-plugin" I modified one month ago for my own Android projects.
It also provides a patch for Proguard-4.7
Hope this helps, any feedback is welcome.
I've figured out a solution that may work for some situations. Instead of ServiceLoader, I'm using the org.openide.util.Lookup class / library that comes with NetBeans - it is a superset of ServiceLoader. It does not require NetBeans itself and seems to work ok with Eclipse. It is necessary to replace whatever ServiceLoader functionality you are using in your application with Lookup equivalents, and add the org-openide-util-lookup library. Then, you can just do something like this:
Lookup lookup = new ProxyLookup(Lookup.getDefault(),
Lookups.metaInfServices(myClass.getClassLoader(), "services/"));
And move your ServiceLoader files from META-INF/services/ to services/.
Note that, because of the ProxyLookup, this will continue to work on standard Java environments unchanged (i.e., in those cases it will continue to look in META-INF/services).
Here is a link to the documentation for the library: http://bits.netbeans.org/dev/javadoc/org-openide-util-lookup/org/openide/util/lookup/Lookups.html
UPDATE
After working with this for a couple of days, it seems to function well - I move between environments (standard Java and Android) and it works properly in each location. The primary downside is having to manually copy the files to the /services directory.
It is possible. You may want to check http://developer.android.com/reference/java/util/ServiceLoader.html
ServiceLoader is stuff from the Java language that is not really relevant on Android. I recommend not using it. If you just want to find a list of classes within your .apk to load, there are all kinds of ways to do this -- put in XMl file in res/xml that lists them, use reflection, annotations, etc.
Are there any disadvantages in using Java 6 wildcards in my classpath ? e.g.
C:> set CLASSPATH=.\lib\*
I can see that where there are two jars that both contain a class with the same path then using a wildcard may lead to results that are hard to track down.
But other than that, is there anything else to be aware of?
If it's what you want to do, then do it. As long as you are aware of the consequences. Keep in mind that if anyone else has to maintain the project, they may copy a bunch of jars into that folder not realizing that they'll be linked by default. It shouldn't take them too long to see what's going on, though.
I generally try to minimize the number of jar files I use, and link them all in manually. I realize this is personal preference.
You might load undesired classes by doing so, and if there is two versions of the same library; well, kaboom.
An explicit classpath can server as a documentation of what libraries (and perhaps what versions thereof!) the application depends on.
You lose this if you use wildcards - if it's not documented elsewhere, then if someone gets a copy of the app without the lib folder (or you delete it accidentally), they'll have a very hard time tracking down all the dependencies via repeatedly running the app, looking at ClassNotFoundErrors and hoping that all libraries used sensible package names.
You are potentially giving the JVM a lot of places to search, this might give some overhead as classes are loaded - I guess a clever JVM will do this efficiently.
My first reflex was don't use env.CLASSPATH, but on second thought and while thinking about why one shouldn't do it I started likeing the idea, at least for a local development and test environment (and at least for a while)
The advantage of this approach is that you can keep a local folder with all your common libraries (log4j, dom4j, joda time, google collections, the apache commons zoo, ...). So you can compile and execute all your applications from the shell without wasting time typing long classpath arguments.
And your still free to use a -cp argument, because it replaces the global CLASSPATH setting.
I would never use it on a production system. The risk is just to high that someone changes the content of that folder or the CLASSPATH variable and my application doesn't work anymore.
So for production, no global 'CLASSPATH' and no wildcards in the classpath string.
A disadvantage of using the wildcard path in an environment like above: After a while too many projects depend upon the single library folder. You don't know the sideeffects of updating a library or deleting an old one. And for large application it might be hard to find out which libraries from the pool are really needed. You might end up adding unused libs to the product just because you're unsure if the application will run without that lib.
So my conclusion - a nice shortcut for development, testing, prototyping but risky for production. For production I'd prefer (autogenerated) classpath strings without wildcards.