I have two Java Servlets: DataFetcherServlet and UploaderServlet. Both servlets call 2 different Java methods which in turn call their corresponding Matlab functions through JNI, and each of which was compiled into a separate Java jar file to use as a library. The application is powered by AJAX to create a Desktop like feel. For the UploaderServlet, the users can upload an excel file to this servlet, the parsed data then get passed to a Java method which then calls the compiled Matlab function to generate and save alot of images (currently over 5000 images), because this will take alot of time, I use an ExecutorService to execute it in the background. But new requests sent the DataFetcherServlet which will also call another compiled Matlab function is blocked until the image generation part is completed. I don't know why it is blocking new requests even though the requests are sent to a different servlet.
DataFetcherServlet.java
public class DataFetcherServlet extends HttpServlet {
#Inject
private CdfReader reader; // An EJB to get a data array from Matlab
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
try {
String filePath = "path/to/file";
Object[] result = reader.read(filePath); // reader.read() is just a wrapper around the method in the jar file mentioned above that actually calls the matlab function to return an array of number
MWNumericArray array = (MWNumericArray)result[0] // This will block while the other Matlab function is generating the images.
.
.
.
} catch (MWException ex) {
Logger.getLogger(DataFetcherServlet.class.getName()).log(Level.SEVERE, null, ex);
}
}
UploaderServlet.java
public class UploaderServlet extends HttpServlet {
#Inject
private ExcelIonImageGenerator generator; // An EJB to call Matlab to generate the images
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
try {
String dir = "path/to/parent/directory";
Path excel = Paths.get(dir+ "excel", part.getSubmittedFileName()); // Path to where the uploaded excel file is stored
if (!Files.exists(excel))
Files.copy(part.getInputStream(), excel);
// ExcelExtractor is a helper class to parse the excel file.
Double[][] ranges = ExcelExtractor.extractSheet(WorkbookFactory.create(excel.toFile()));
// This will call a Java library method which in turns call the Matlab function
// to generate the images (over 5000 in this case)
// See the code for this method below.
generator.generate(dir+ "images" + File.separator, ranges);
} catch (MWException | InvalidFormatException ex) {
Logger.getLogger(UploaderServlet.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
ExcelIonImageGenerator.java
import com.mathworks.toolbox.javabuilder.*; // Matlab SDK needed to integrate with Java
import java.util.concurrent.*;
import java.util.logging.*;
import javax.annotation.PreDestroy;
import javax.ejb.Stateless;
import save_ion_image_for_all_ranges_in_spreadsheet.Class1; // The jar file which contains code to call Matlab code through JNI
#Stateless
public class ExcelIonImageGenerator {
private final Class1 clazz1;
private ExecutorService pool;
public ExcelIonImageGenerator() throws MWException {
clazz1 = new Class1();
pool = Executors.newFixedThreadPool(1);
}
public void generate(String path, Double[][] ranges) throws MWException {
// Submit this task to the ExecutorService so it can be processed
// in a different thread than the caller thread
pool.submit(() -> generateHelper(path, ranges, clazz1), 1);
}
private void generateHelper(String path, Double[][] ranges, Class1 clazz) {
try {
// This method was generated by Matlab tool, it calls the native
// Matlab code through JNI, and it will block any request that will call
// other Matlab functions until it finishes.
clazz.save_ion_image_for_all_ranges_in_spreadsheet(path, ranges);
} catch (MWException ex) {
Logger.getLogger(ExcelIonImageGenerator.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
You have three options:
Launch multiple processes of your Java application that makes calls to Matlab. Calls from a single process use the same MCR which has a process-wide lock, however calls from different processes would run on separate MCR computational engines.
Use Matlab Production Server, which basically facilitates the use of multiple MCRs. This is a toolkit that requires a separate license and installation.
You don't necessarily have to limit yourself to running MCR / compiled code unless you have very specific performance concerns. You can actually install Matlab itself on the server, launch multiple instances (headless, etc) from the same Java process, and communicate with them e.g. through MatlabControl or the new official MATLAB Engine API for Java.
There's a very good answer from MathWorks Support Team on MatlabCentral, explaining these limitations of MCR in detail.
Related
I have brief understanding on multi threading in Java web apps. However, I came across an issue while developing our android app which communicates with the server via REST.
Our web application is based on Apache Wicket 8.6 and contains additional HttpServlet endpoints. One endpoint is applied while uploading images from the app via post. This works fine as long as I am uploading only one image at a time. If I perform multiple upload requests in my android app in a quick succession (some milliseconds), only the last upload is performed successfully (it is working fine when I put a second break between the uploads). It seems, as if all requests except the last one are missing the image content read from the input stream of the servlet request. Thus, I am guessing, that I am having a threading problem with my servlet. I would appreciate, if someone could guide me in the right direction to solve this issue.
public class MyServlet extends HttpServlet{
#Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
boolean proceed =true;
Map<String,String[]> parameters = req.getParameterMap();
if(parameters!=null){
//read some parameters which specify the request
}
if(proceed) {
InputStream is = req.getInputStream();
if(is!=null) {
//The result of this output is 0 for the first requests only for the last request it is 1
System.err.println("Stream size: "+is.available());
//do something
}
}
//do something....
}
}
I could, of course, write the images in my android app in one request using multipart but still I'd like to make the servlet thread safe for the occasion that it gets two requests at the same time.
I appreciate your help.
So, after some additional research I discovered, that the input stream is not empty even is.available() returns 0
My problem was something different. I save the uploaded image in a ModeShape repository. The repository session is stored in the wicket application instance. THus, only one ModeSHape repository session exists. When writing the image, it seems as if there was a problem with the ModeShape session. Thus, I put the ModeShape session in a synchronized block and now everything runs fine.
public class MyServlet extends HttpServlet{
#Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
boolean proceed =true;
Map<String,String[]> parameters = req.getParameterMap();
if(parameters!=null){
//read some parameters which specify the request
}
if(proceed) {
String fileName = req.getHeader("fileName");
if(!StringUtils.isEmpty(fileName)) {
InputStream is = req.getInputStream();
if(is!=null) {
//The result of this output is 0 for the first requests only for the last request it is 1
System.err.println("Stream size: "+is.available());
WicketApplication app=null;
try {
app = (WicketApplication)Application.get("wicket");
} catch (Exception e) {
LOG.error("Error while generating WicketApplication object for project "+project.getName(),e);
}
if(app!=null) {
final Session repoSession = app.getRepoSession();
synchronized (repoSession) {
//write image to repository
}
}
}
}
}
//do something....
}
}
I'm not that good in Java but I have my webApplication running on a Wildfly.
I have 3 threads who just call a function that insert logs in in and the function saves the logs to a Database and after that every thread sends a time how long did it takes to do this.
They send the data to another programm I wrote what has 3 threads to call one of the 3 server threads.
So now I try to do some ByteCode Manipulation every thread on the server saves the datetime call the log function waits 1 second and returns then the time they needed.
1 Thread write something in an logfile before or after they waitet 1 second.
But this part where they wait a second and call the log function I want that to inject to every 3 threads with Bytecode manipulation.
public class MyTransformer implements ClassFileTransformer {
#Override
public byte[] transform(ClassLoader loader, String className, Class redefiningClass, ProtectionDomain protectionDomain, byte[] bytes) throws IllegalClassFormatException {
return transformClass(redefiningClass, bytes);
}
private byte[] transformClass(Class classToTransform, byte[] b) {
ClassPool pool = ClassPool.getDefault();
CtClass cl = null;
try {
cl = pool.get("de.soptim.ws.MyApplication");
} catch (javassist.NotFoundException e) {
e.printStackTrace();
}
try {
assert cl != null;
CtMethod[] methods = cl.getMethods();
for (int i = 0; i < methods.length; i++) {
if (methods[i].isEmpty() == false) {
changeMethod(methods[i]);
}
}
b = cl.toBytecode();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (cl != null) {
cl.detach();
}
}
return b;
}
private void changeMethod(CtMethod method) throws NotFoundException, CannotCompileException {
if (method.hasAnnotation(Loggable.class)) {
method.insertBefore("threadLogger.info(\"ADDED THIS FROM BYTECODE !!!\");");
method.insertAfter("threadLogger.info(\"ADDED THIS FROM BYTECODE !!!\");");
}
}}
Thats my transformer class it should increase the Code my Methods need it checks what Method has an #Loggable annotation an then adds the code into it("at the moment it's just some log statments for checking if it works")
My biggest problem is now that I don't know how to call my agent ... I googled hwo to call an agent at runtime with agentmain() but I think I dind't really understood how it works.
Agent Class
public class LogAgent {
public static void agentmain(String agentArgs, Instrumentation inst) {
System.out.println("Starting the agent");
inst.addTransformer(new MyTransformer());
}}
Hope you understand My problem :) I and if you answere pleas try to stay noob friendly :D.
you don't call your agent explicitly, you should specify additional argument to your JVM :
java -javaagent:jarpath[=options]
where jarpath is a path to jar containing your agent. JVM will invoke premain method before main method of java program.
And transform method will be called before classloading by JVM (you don't call it explicitly).
Last remark : you should implement premain method, not agentmain.
agentmain is used during attaching to running vm, while premain is used when you start JVM with -javaagent method.
And make sure that your jar have valid manifest, as described : https://docs.oracle.com/javase/8/docs/api/java/lang/instrument/package-summary.html
I haven't using javaassit so I cannot say that your code is valid, but instrumenting webapp server like Wildfly is much harder than normal java app (mostly due to classloaders visibility and hierarchy).
See also :
http://www.tomsquest.com/blog/2014/01/intro-java-agent-and-bytecode-manipulation/
Tutorials about javaagents
This question already has an answer here:
Downloading file/files in Java. Multithreading, this works?
(1 answer)
Closed 8 years ago.
I'm experiencing some issue at working multithreading in java. I am a student, java beginner and just developing this for entertainment purposes (sorry for bad grammar, english is not my first lang).
I'm doing a tiny downloader for personal use that accepts a maximum of 5 simultaneous downloads. Each download is handled by a different thread (not the common threads, just swingworker, which I also use to avoid UI freezing.).
public static String PATH; //File path
public static String NAME; //File name
//Download Method
public void download() {
//download code...
}
This method works fine (it downloads a file and then saves it into the hard drive as expected). But the issue comes when I want to do two (or more) downloads simultaneously. Consider I am downloading two files at the same time, file A and file B. When I start the download of A, strings PATH and NAME obtain its values according to file A, all OK. Then, I start downloading B and the previous stored values from A are replaced with the values that correspond to B. So when the download of A is complete, the file name is the same that B should have when download B is completed.
Resuming, I need different ìnstances of the same variable, that will contain different and indepentant values.
I started to research about the topic and led to ThreadLocal variables. This type of variable is supposed to change in each running thread, just what I need.
I tried to implement this to my code.
public static String PATH; //File path
public static String NAME; //File name
public ThreadLocal<String> TL_PATH = new ThreadLocal<String>();
public ThreadLocal<String> TL_NAME = new ThreadLocal<String>();
public void download() {
//Try to set ThreadLocal to PATH and NAME variables.
TL_NAME.set(NAME);
TL_PATH.set(PATH);
//download code...
}
Once I did this, everything was the same. What's wrong in my code? (No exceptions are thrown in any case, just what I explained before)
You are doing it wrong
You should implement Runnable and have it take the url and path to download to.
Use ExecutorService set to 5 threads in the pool.
To manage the individual instances of the class that implements Runnable.
Here is an example:
import javax.annotation.Nonnull;
import java.io.File;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class Q24591731
{
private static final ExecutorService EXECUTOR_SERVICE;
static
{
EXECUTOR_SERVICE = Executors.newFixedThreadPool(5);
}
public static void main(final String[] args)
{
final Download d1;
try { d1 = new Download(new URL("www.someurl.to.download.com"), new File("dest/file/name")); }
catch (MalformedURLException e) { throw new RuntimeException(e); }
// create as many downloads as you need how ever you want.
EXECUTOR_SERVICE.submit(d1);
// when all are submitted
EXECUTOR_SERVICE.shutdown();
try
{
EXECUTOR_SERVICE.awaitTermination(1, TimeUnit.MINUTES);
}
catch (InterruptedException e)
{
System.exit(1);
}
}
public static class Download implements Runnable
{
private final URL url;
private final File dest;
public Download(#Nonnull final URL url, #Nonnull final File dest)
{
this.url = url;
this.dest = dest;
}
#Override
public void run()
{
// download the file and write it to disk
}
}
}
Exercise for the reader:
ExecutorCompletionService is a more appropriate solution, but it is a little more involved and works best with Callable instead of Runnable. Especially for batch processing like this. But I have examples on other answers here and don't feel like repeating that solution again.
How can I call a function when a specific event occurs in my java Applet ?
In my Javascript I has the following code, that always return a empty value.
$(function () {
alert(document.applets[0].returnClientId());
});
I need to call this alert, just when a specific method is executed. To be more specific, the method called identify, that exists in my java Applet.
The identify method, exists in my Util.class, that don't extends from JApplet.
My FormMain.class extends from JApplet and I call some methods (including the identify method) from this Util.class.
UPDATE: My Java Code
public String getClientid() {
return clientid;
}
public void setClientid(String clientid) {
this.clientid = clientid;
}
public String returnClientId() {
return getClientid();
}
public void identify() {
try {
fingerprintSDK.prepareForIdentification(this.template);
ResultSet rs = identifyStmt.executeQuery();
while (rs.next()) {
byte[] templateBuffer = rs.getBytes("template");
Template referenceTemplate = new Template(templateBuffer);
boolean matched = fingerprintSDK.identify(referenceTemplate);
if (matched) {
// ui is my FormMain instance
ui.showImage(GrFingerJava.getBiometricImage(template,
fingerprint, fingerprintSDK));
ui.writeLog("Found. Client = "
+ rs.getString("Name"));
ui.setClienteid(rs.getString("Cliente_Id"));
ui.disableTemplate();
return;
}
}
ui.writeLog("Not Found.");
ui.enableTemplate();
} catch (SQLException e) {
ui.writeLog(e.getMessage());
} catch (GrFingerJavaException e) {
ui.writeLog(e.getMessage());
}
}
The Identify method is executed just when my User put the finger in the biometric device.
Someone has some idea ?
Suppose you have the following JS function
function foo(client) {
alert(client);
}
you'll modify your Utils.java as follows
public void identify() {
// the rest of your code
String id = rs.getString("Cliente_Id");
ui.setClienteid(id);
// call the Javascript function
JSObject.getWindow(ui).eval(String.format("foo(%s)", id));
}
Invoking Javascript code from an Applet
Java to Javascript communication (Official Oracle docs)
To compile the code using netscape.javascript.* package you need the $JAVA_HOME/jre/lib/plugin.jar. See here
In order to call JavaScript, Java code uses the
netscape.javascript.JSObject and netscape.javascript.JSException
classes. Since the release of Java 2, Standard Edition version 1.4,
these classes are provided in the jar file jre/lib/plugin.jar within
either the Java Development Kit or Java Runtime Environment. If you
reference these JavaScript classes, you will need to add plugin.jar to
your compilation classpath. This can be done through your Java IDE, if
you use one, or by passing the -classpath command-line argument to the
Java compiler javac.
At run-time, the Java Plug-In automatically makes these classes
available to applets, so no changes to the applet or how it is set up
are necessary.
I am writing a simple application that lets a user upload images. After the upload, the user can tag them or remove them.
I figured out how to upload the files and save them once the files are uploaded. I am keeping tracking of a global path where images are kept. In the database I keep the meta data about the images like file name, tags, etc.
I am using Java/JSP (specifically Stripes framework but my problem is generic).
My question is where do I keep these image files once they are uploaded?
Right now I have two web applications deployed on a Tomcat server. One main web application and other one is the where I upload the images.
But this does not work as I can not see the uploaded images in the main application until I redeploy/restart Tomcat.
It seems like Tomcat does not pick newly uploaded images automatically.
Does anyone have any solutions?
This is a simple project, so I do not want to store them in a database or use Apache for images. That is all just too complicated for this small project.
Thanks.
Definitely don't store the images in the database, but you will want to store the image path in the database. This will allow you to store the image just about anywhere.
Since you are using two tomcat applications, your best bet may be to store the images outside of either app and stream the image back to the user instead of letting tomcat manage the files. Otherwise, I would ask why you are trying to do this with two web apps.
However, storing uploaded images inside the web-app directory is not a wise thing to do, and you know it.
By the way, you might want to look this stackoverflow thread, lately discussed where to store the images. It might not solve your issue, surely will give you more confidence on what you are doing.
I've solved this in different ways.
First, the non-portable way, is that Glassfish (and I do believe Tomcat as well) allows you to map an external directory in to the webapps hierarchy. This works really well and does exactly what you want. It lets you store your images in an external directory away from your webapp, yet still serve them up.
However, this technique is not portable.
The way to I've done it portably is by creating a filter.
You place the filter someplace obvious, say "/images".
What the filter does is this:
it checks for the image (or anything, it works with any static resource) in a special directory within the webapp. For our example we'll use the url "/webapp/images".
if the file DOES NOT exist, we copy the file from your external location in to the appropriate spot within the webapp. So, let's say the reqyest url is "/images/banner.gif". And that your files are stored on disk at "/home/app/images". So, our source file is "/home/app/images/banner.gif". We then copy it to where we want it in the webapp tree. We use "ServletContext.getRealPath" for this. So, the destination will be "ServletContext.get RealPath("/webapp/images/banner.gif"). Just copy the source to the destination.
if the file already existed, or now exists, simply forward to the actual image at /webapp/images/banner.gif.
Effectively you end up having a file cache within your webapps deployment tree. The down side is that it's a cache, so it needs to be maintained (i.e. you should check if the original is newer than your cache, make sure you delete if the source is deleted, etc.). Also, it duplicates your resources, so your images will consume, eventually, twice as much disk space. Finally, there's the initial copy cost at start up.
However, it DOES work, and it prevents you from having to serve up static resources using your own code. (Which is the 3rd solution, map a filter/servlet to intercept the URLs and simply stream it your self.)
I would look at the construct within Tomcat (assuming it exists) to do the mapping for you. I know it exists in Glassfish. (Google alternatedocroot for Glassfish to see how it works.)
I was using two web applications to avoid over writing the uploaded images in case I redeploy a new main application war file.
But as you mention there is no other option but to stream them through a Servlet or something I guess I can keep them outside tomcat directory.
I wanted to avoid writing this Streaming Servlet. Just too small project to deal with all the mess (like proper content type, 404, etc.) while writing the streaming servlet.
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
/**
* Image streaming Servlet.
*/
public class ImageDisplayServlet extends HttpServlet {
private static final long serialVersionUID = 1L;
/**
* #see HttpServlet#HttpServlet()
*/
public ImageDisplayServlet() {
super();
}
/**
* #see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse response)
*/
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
String relativePath = trimToEmpty(request.getPathInfo());
// Make sure no one try to screw with us.
// This is important as user can literally access any file if we are not careful
if(isXSSAttack(relativePath) == false) {
String pathToFile = this.getServletContext().getRealPath(request.getPathInfo());
File file = new File(pathToFile);
System.out.println("Looking for file " + file.getAbsolutePath());
// show a 404 page
if(!file.exists() || !file.isFile()) {
httpError(404, response);
} else {
try {
streamImageFile(file, response);
} catch(Exception e) {
// Tell the user there was some internal server error.\
// 500 - Internal server error.
httpError(500, response);
e.printStackTrace();
}
}
} else {
// what to do if i think it is a XSS attack ?!?
}
}
private void streamImageFile(File file, HttpServletResponse response) {
// find the right MIME type and set it as content type
response.setContentType(getContentType(file));
BufferedInputStream bis = null;
BufferedOutputStream bos = null;
try {
response.setContentLength((int) file.length());
// Use Buffered Stream for reading/writing.
bis = new BufferedInputStream(new FileInputStream(file));
bos = new BufferedOutputStream(response.getOutputStream());
byte[] buff = new byte[(int) file.length()];
int bytesRead;
// Simple read/write loop.
while (-1 != (bytesRead = bis.read(buff, 0, buff.length))) {
bos.write(buff, 0, bytesRead);
}
} catch (Exception e) {
throw new RuntimeException(e);
} finally {
if (bis != null) {
try {
bis.close();
} catch (IOException e) {
e.printStackTrace();
// To late to do anything about it now, we may have already sent some data to user.
}
}
if (bos != null) {
try {
bos.close();
} catch (IOException e) {
e.printStackTrace();
// To late to do anything about it now, we may have already sent some data to user.
}
}
}
}
private String getContentType(File file) {
if(file.getName().length() > 0) {
String[] parts = file.getName().split("\\.");
if(parts.length > 0) {
// only last part interests me
String extention = parts[parts.length - 1];
if(extention.equalsIgnoreCase("jpg")) {
return "image/jpg";
} else if(extention.equalsIgnoreCase("gif")) {
return "image/gif";
} else if(extention.equalsIgnoreCase("png")) {
return "image/png";
}
}
}
throw new RuntimeException("Can not find content type for the file " + file.getAbsolutePath());
}
private String trimToEmpty(String pathInfo) {
if(pathInfo == null) {
return "";
} else {
return pathInfo.trim();
}
}
private void httpError(int statusCode, HttpServletResponse response) {
try {
response.setStatus(statusCode);
response.setContentType("text/html");
PrintWriter writer = response.getWriter();
writer.append("<html><body><h1>Error Code: " + statusCode + "</h1><body></html>");
writer.flush();
} catch (IOException e) {
e.printStackTrace();
}
}
private boolean isXSSAttack(String path) {
boolean xss = false;
// Split on the bases of know file separator
String[] parts = path.split("/|\\\\");
// Now verify that no part contains anything harmful
for(String part : parts) {
// No double dots ..
// No colons :
// No semicolons ;
if(part.trim().contains("..") || part.trim().contains(":") || part.trim().contains(";")) {
// Fire in the hole!
xss = true;
break;
}
}
return xss;
}
/**
* #see HttpServlet#doPost(Ht/promotions/some.jpgtpServletRequest request, HttpServletResponse response)
*/
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
doGet(request, response);
}
}
Ok Here is a Servlet that I quickly wrote that can stream images:
Here is the List of limitations and know issues:
May have XSS vulnerability use with care
Not production ready use as reference
Images need to in the web application directory. Can be easily change but I too lazy (it is not worth it the project is too small)
Only stream jpg,gif or png files.
Usage:
Let say you deploy this web application called images as separate application.
http://www.example.com/images/promotions/promo.jpg
means there should be a directory in "promotions" with image "promo.jpg" with in this images web application.
PS: Do not ask why I am doing this Servlet Container only solution that sucks big time.
<servlet>
<description></description>
<display-name>ImageDisplayServlet</display-name>
<servlet-name>ImageDisplayServlet</servlet-name>
<servlet-class>com.example.images.ImageDisplayServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>ImageDisplayServlet</servlet-name>
<url-pattern>/*</url-pattern>
</servlet-mapping>
Oh ya configure your Servlet like above for best results :P