This question already has an answer here:
Downloading file/files in Java. Multithreading, this works?
(1 answer)
Closed 8 years ago.
I'm experiencing some issue at working multithreading in java. I am a student, java beginner and just developing this for entertainment purposes (sorry for bad grammar, english is not my first lang).
I'm doing a tiny downloader for personal use that accepts a maximum of 5 simultaneous downloads. Each download is handled by a different thread (not the common threads, just swingworker, which I also use to avoid UI freezing.).
public static String PATH; //File path
public static String NAME; //File name
//Download Method
public void download() {
//download code...
}
This method works fine (it downloads a file and then saves it into the hard drive as expected). But the issue comes when I want to do two (or more) downloads simultaneously. Consider I am downloading two files at the same time, file A and file B. When I start the download of A, strings PATH and NAME obtain its values according to file A, all OK. Then, I start downloading B and the previous stored values from A are replaced with the values that correspond to B. So when the download of A is complete, the file name is the same that B should have when download B is completed.
Resuming, I need different ìnstances of the same variable, that will contain different and indepentant values.
I started to research about the topic and led to ThreadLocal variables. This type of variable is supposed to change in each running thread, just what I need.
I tried to implement this to my code.
public static String PATH; //File path
public static String NAME; //File name
public ThreadLocal<String> TL_PATH = new ThreadLocal<String>();
public ThreadLocal<String> TL_NAME = new ThreadLocal<String>();
public void download() {
//Try to set ThreadLocal to PATH and NAME variables.
TL_NAME.set(NAME);
TL_PATH.set(PATH);
//download code...
}
Once I did this, everything was the same. What's wrong in my code? (No exceptions are thrown in any case, just what I explained before)
You are doing it wrong
You should implement Runnable and have it take the url and path to download to.
Use ExecutorService set to 5 threads in the pool.
To manage the individual instances of the class that implements Runnable.
Here is an example:
import javax.annotation.Nonnull;
import java.io.File;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
public class Q24591731
{
private static final ExecutorService EXECUTOR_SERVICE;
static
{
EXECUTOR_SERVICE = Executors.newFixedThreadPool(5);
}
public static void main(final String[] args)
{
final Download d1;
try { d1 = new Download(new URL("www.someurl.to.download.com"), new File("dest/file/name")); }
catch (MalformedURLException e) { throw new RuntimeException(e); }
// create as many downloads as you need how ever you want.
EXECUTOR_SERVICE.submit(d1);
// when all are submitted
EXECUTOR_SERVICE.shutdown();
try
{
EXECUTOR_SERVICE.awaitTermination(1, TimeUnit.MINUTES);
}
catch (InterruptedException e)
{
System.exit(1);
}
}
public static class Download implements Runnable
{
private final URL url;
private final File dest;
public Download(#Nonnull final URL url, #Nonnull final File dest)
{
this.url = url;
this.dest = dest;
}
#Override
public void run()
{
// download the file and write it to disk
}
}
}
Exercise for the reader:
ExecutorCompletionService is a more appropriate solution, but it is a little more involved and works best with Callable instead of Runnable. Especially for batch processing like this. But I have examples on other answers here and don't feel like repeating that solution again.
Related
/*
this method will create the required manifest file in compatible format such that
quicksight can import data from specified s3 bucket
*/
private JSONObject CreateManifestFileJSONObject(JSONObject ManifestFile){
JSONArray URIPrefixArray= new JSONArray();
URIPrefixArray.put(PrefixLocation);
JSONObject URIPrefixJSONObject= new JSONObject();
URIPrefixJSONObject.put("URIPrefixes",URIPrefixArray);
JSONArray FileLocationsArray= new JSONArray();
FileLocationsArray.put(URIPrefixJSONObject);
JSONObject globalUploadSettings= new JSONObject();
globalUploadSettings.put("format","JSON");
ManifestFile.put("globalUploadSettings",globalUploadSettings);
ManifestFile.put("fileLocations",FileLocationsArray);
return(ManifestFile);
}
/*
this method will upload the ManifestFile to same S3 Bucket in which data files is stored
*/
private void UploadManifestFileJSONObjectToS3(JSONObject ManifestFile){
try {
AmazonS3 S3Client = new Utility().SetUpS3Client();
byte[] fileContentBytes = (ManifestFile.toString()).getBytes();
InputStream fileInputStream = new ByteArrayInputStream(fileContentBytes);
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(fileContentBytes.length);
S3Client.putObject(new PutObjectRequest(Bucket_Name, ManifestFileName, fileInputStream, objectMetadata).withCannedAcl(CannedAccessControlList.PublicRead));
}
catch(Exception exception){
exception.printStackTrace();
}
}
public void handler() {
System.out.println("inside the manifest file");
try {
JSONObject ManifestFile = new JSONObject();
ManifestFile = CreateManifestFileJSONObject(ManifestFile);
UploadManifestFileJSONObjectToS3(ManifestFile);
}
catch(Exception exception){
exception.printStackTrace();
}
I want to test handler method but handlor method is calling private methods so I do not know how to write the test class for this code.
i want to write unit test for this class please help
this is the test class I am able to create up to this point but it will not surely mock s3 behaviour
#Test
public void handler() {
ManifestFileHandler manifestFileHandler=new ManifestFileHandler();
manifestFileHandler.handler();
}
You can test private methods with the help of PowerMock provides utilities which uses reflection to do certain things.
check the below example,
https://examples.javacodegeeks.com/core-java/mockito/mockito-mock-private-method-example-with-powermock/
There are two schools of thought on unit testing private functions. The first is that you make them public (or protected or package accessible) and test them as you would a public function. The second is that if they are private they are part of the encapsulated implementation detail and you only need to test them through the public functions.
My personal view is that complicated private functions are often a sign that you are breaking the single responsibility principle and it's likely you should have logic in private functions that should be split into a separate class that can then be tested through its public methods.
With respect to the code you've posted you have a larger problem than how to test the private functions: your class depends on other classes that you don't have control over. You have no way of mocking the behaviour of those classes to test various scenarios or to verify that they have been called correctly. I suspect it is this problem that is really behind your question.
As an example, I would suggest you inject a S3Client into your class rather than create it internally through new Utility().SetUpS3Client(). That way you can mock its behaviour and verify it is called correctly by your code. Attempting to do that with the real version of this class will be challenging.
So using this model, your code might look something like:
public class ManifestFileHandler {
private final S3Client client;
public ManifestFileHandler(S3Client client) {
this.client = client;
}
private void upload(JSONObject manifestFile) {
...
client.putObject(...);
}
public void handleManifest() {
...
upload(manifestFile);
...
}
}
And your test code (using mockito):
#Test
void testManifestUpload() {
S3Client client = mock(S3Client.class);
ManifestFileHandler handler = new ManifestFileHandler(client);
handler.handleManifest();
verify(client).putObject(expectedObject);
}
If you need to capture the argument passed to putObject and assert various aspects of it then that is possible with most mocking tools (including mockito) but is beyond the scope of your question.
I have two Java Servlets: DataFetcherServlet and UploaderServlet. Both servlets call 2 different Java methods which in turn call their corresponding Matlab functions through JNI, and each of which was compiled into a separate Java jar file to use as a library. The application is powered by AJAX to create a Desktop like feel. For the UploaderServlet, the users can upload an excel file to this servlet, the parsed data then get passed to a Java method which then calls the compiled Matlab function to generate and save alot of images (currently over 5000 images), because this will take alot of time, I use an ExecutorService to execute it in the background. But new requests sent the DataFetcherServlet which will also call another compiled Matlab function is blocked until the image generation part is completed. I don't know why it is blocking new requests even though the requests are sent to a different servlet.
DataFetcherServlet.java
public class DataFetcherServlet extends HttpServlet {
#Inject
private CdfReader reader; // An EJB to get a data array from Matlab
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
try {
String filePath = "path/to/file";
Object[] result = reader.read(filePath); // reader.read() is just a wrapper around the method in the jar file mentioned above that actually calls the matlab function to return an array of number
MWNumericArray array = (MWNumericArray)result[0] // This will block while the other Matlab function is generating the images.
.
.
.
} catch (MWException ex) {
Logger.getLogger(DataFetcherServlet.class.getName()).log(Level.SEVERE, null, ex);
}
}
UploaderServlet.java
public class UploaderServlet extends HttpServlet {
#Inject
private ExcelIonImageGenerator generator; // An EJB to call Matlab to generate the images
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
try {
String dir = "path/to/parent/directory";
Path excel = Paths.get(dir+ "excel", part.getSubmittedFileName()); // Path to where the uploaded excel file is stored
if (!Files.exists(excel))
Files.copy(part.getInputStream(), excel);
// ExcelExtractor is a helper class to parse the excel file.
Double[][] ranges = ExcelExtractor.extractSheet(WorkbookFactory.create(excel.toFile()));
// This will call a Java library method which in turns call the Matlab function
// to generate the images (over 5000 in this case)
// See the code for this method below.
generator.generate(dir+ "images" + File.separator, ranges);
} catch (MWException | InvalidFormatException ex) {
Logger.getLogger(UploaderServlet.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
ExcelIonImageGenerator.java
import com.mathworks.toolbox.javabuilder.*; // Matlab SDK needed to integrate with Java
import java.util.concurrent.*;
import java.util.logging.*;
import javax.annotation.PreDestroy;
import javax.ejb.Stateless;
import save_ion_image_for_all_ranges_in_spreadsheet.Class1; // The jar file which contains code to call Matlab code through JNI
#Stateless
public class ExcelIonImageGenerator {
private final Class1 clazz1;
private ExecutorService pool;
public ExcelIonImageGenerator() throws MWException {
clazz1 = new Class1();
pool = Executors.newFixedThreadPool(1);
}
public void generate(String path, Double[][] ranges) throws MWException {
// Submit this task to the ExecutorService so it can be processed
// in a different thread than the caller thread
pool.submit(() -> generateHelper(path, ranges, clazz1), 1);
}
private void generateHelper(String path, Double[][] ranges, Class1 clazz) {
try {
// This method was generated by Matlab tool, it calls the native
// Matlab code through JNI, and it will block any request that will call
// other Matlab functions until it finishes.
clazz.save_ion_image_for_all_ranges_in_spreadsheet(path, ranges);
} catch (MWException ex) {
Logger.getLogger(ExcelIonImageGenerator.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
You have three options:
Launch multiple processes of your Java application that makes calls to Matlab. Calls from a single process use the same MCR which has a process-wide lock, however calls from different processes would run on separate MCR computational engines.
Use Matlab Production Server, which basically facilitates the use of multiple MCRs. This is a toolkit that requires a separate license and installation.
You don't necessarily have to limit yourself to running MCR / compiled code unless you have very specific performance concerns. You can actually install Matlab itself on the server, launch multiple instances (headless, etc) from the same Java process, and communicate with them e.g. through MatlabControl or the new official MATLAB Engine API for Java.
There's a very good answer from MathWorks Support Team on MatlabCentral, explaining these limitations of MCR in detail.
I want to build call graphs on the fly, starting at an arbitrary method call or with a new thread, which ever is easier, from within the running JVM itself. (this piece of software is going to be a test fixture for load testing another piece of software that consumes call graphs)
I understand there are some SPI interfaces, but it looks like you need to run -javaagent flag with them. I want to access this directly in the VM itself.
Ideally, I'd like to get a callback for entry and exit of each method call, parameters to that method call, and time in that method. Within a single thread obviously.
I know AOP could probably do this, but I'm just wondering if there are tools within the JDK that would allow me to capture this.
There is no such API provided by the JVM— even for agents started with -javaagent. The JVM TI is a native interface provided for native agents started with the -agent option or for debuggers. Java agents might use the Instrumentation API which provides the lowlevel feature of class instrumentation but no direct profiling capability.
There are two types of profiling implementations, via sampling and via instrumentation.
Sampling works by recording stack traces (samples) periodically. This does not trace every method call but still detect hot spots as they occur multiple times in the recorded stack traces. The advantage is that it does not require agents nor special APIs and you have the control over the profiler’s overhead. You can implement it via the ThreadMXBean which allows you to get stack traces of all running threads. In fact, even a Thread.getAllStackTraces() would do but the ThreadMXBean provides more detailed information about the threads.
So the main task is to implement an efficient storage structure for the methods found in the stack traces, i.e. collapsing occurrences of the same method into single call tree items.
Here is an example of a very simple sampler working on its own JVM:
import java.lang.Thread.State;
import java.lang.management.ManagementFactory;
import java.lang.management.ThreadInfo;
import java.lang.management.ThreadMXBean;
import java.util.*;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
public class Sampler {
private static final ThreadMXBean TMX=ManagementFactory.getThreadMXBean();
private static String CLASS, METHOD;
private static CallTree ROOT;
private static ScheduledExecutorService EXECUTOR;
public static synchronized void startSampling(String className, String method) {
if(EXECUTOR!=null) throw new IllegalStateException("sampling in progress");
System.out.println("sampling started");
CLASS=className;
METHOD=method;
EXECUTOR = Executors.newScheduledThreadPool(1);
// "fixed delay" reduces overhead, "fixed rate" raises precision
EXECUTOR.scheduleWithFixedDelay(new Runnable() {
public void run() {
newSample();
}
}, 150, 75, TimeUnit.MILLISECONDS);
}
public static synchronized CallTree stopSampling() throws InterruptedException {
if(EXECUTOR==null) throw new IllegalStateException("no sampling in progress");
EXECUTOR.shutdown();
EXECUTOR.awaitTermination(Long.MAX_VALUE, TimeUnit.DAYS);
EXECUTOR=null;
final CallTree root = ROOT;
ROOT=null;
return root;
}
public static void printCallTree(CallTree t) {
if(t==null) System.out.println("method not seen");
else printCallTree(t, 0, 100);
}
private static void printCallTree(CallTree t, int ind, long percent) {
long num=0;
for(CallTree ch:t.values()) num+=ch.count;
if(num==0) return;
for(Map.Entry<List<String>,CallTree> ch:t.entrySet()) {
CallTree cht=ch.getValue();
StringBuilder sb = new StringBuilder();
for(int p=0; p<ind; p++) sb.append(' ');
final long chPercent = cht.count*percent/num;
sb.append(chPercent).append("% (").append(cht.cpu*percent/num)
.append("% cpu) ").append(ch.getKey()).append(" ");
System.out.println(sb.toString());
printCallTree(cht, ind+2, chPercent);
}
}
static class CallTree extends HashMap<List<String>, CallTree> {
long count=1, cpu;
CallTree(boolean cpu) { if(cpu) this.cpu++; }
CallTree getOrAdd(String cl, String m, boolean cpu) {
List<String> key=Arrays.asList(cl, m);
CallTree t=get(key);
if(t!=null) { t.count++; if(cpu) t.cpu++; }
else put(key, t=new CallTree(cpu));
return t;
}
}
static void newSample() {
for(ThreadInfo ti:TMX.dumpAllThreads(false, false)) {
final boolean cpu = ti.getThreadState()==State.RUNNABLE;
StackTraceElement[] stack=ti.getStackTrace();
for(int ix = stack.length-1; ix>=0; ix--) {
StackTraceElement ste = stack[ix];
if(!ste.getClassName().equals(CLASS)||!ste.getMethodName().equals(METHOD))
continue;
CallTree t=ROOT;
if(t==null) ROOT=t=new CallTree(cpu);
for(ix--; ix>=0; ix--) {
ste = stack[ix];
t=t.getOrAdd(ste.getClassName(), ste.getMethodName(), cpu);
}
}
}
}
}
Profilers hunting for every method invocation without going through the debugging API use instrumentation to add notification code to every method they are interested in. The advantage is that they never miss a method invocation but on the other hand they are adding a significant overhead to the execution which might influence the result when searching for hot spots. And it’s way more complicated to implement. I can’t give you a code example for such a byte code transformation.
The Instrumentation API is provided to Java agents only but in case you want to go into the Instrumentation direction, here is a program which demonstrates how to connect to its own JVM and load itself as a Java agent:
import java.io.*;
import java.lang.instrument.Instrumentation;
import java.lang.management.ManagementFactory;
import java.nio.ByteBuffer;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.util.UUID;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
// this API comes from the tools.jar of your JDK
import com.sun.tools.attach.*;
public class SelfAttacher {
public static Instrumentation BACK_LINK;
public static void main(String[] args) throws Exception {
// create a special property to verify our JVM connection
String magic=UUID.randomUUID().toString()+'/'+System.nanoTime();
System.setProperty("magic", magic);
// the easiest way uses the non-standardized runtime name string
String name=ManagementFactory.getRuntimeMXBean().getName();
int ix=name.indexOf('#');
if(ix>=0) name=name.substring(0, ix);
VirtualMachine vm;
getVM: {
try {
vm = VirtualMachine.attach(name);
if(magic.equals(vm.getSystemProperties().getProperty("magic")))
break getVM;
} catch(Exception ex){}
// if the easy way failed, try iterating over all local JVMs
for(VirtualMachineDescriptor vd:VirtualMachine.list()) try {
vm=VirtualMachine.attach(vd);
if(magic.equals(vm.getSystemProperties().getProperty("magic")))
break getVM;
vm.detach();
} catch(Exception ex){}
// could not find our own JVM or could not attach to it
return;
}
System.out.println("attached to: "+vm.id()+'/'+vm.provider().type());
vm.loadAgent(createJar().getAbsolutePath());
synchronized(SelfAttacher.class) {
while(BACK_LINK==null) SelfAttacher.class.wait();
}
System.out.println("Now I have hands on instrumentation: "+BACK_LINK);
System.out.println(BACK_LINK.isModifiableClass(SelfAttacher.class));
vm.detach();
}
// create a JAR file for the agent; since our class is already in class path
// our jar consisting of a MANIFEST declaring our class as agent only
private static File createJar() throws IOException {
File f=File.createTempFile("agent", ".jar");
f.deleteOnExit();
Charset cs=StandardCharsets.ISO_8859_1;
try(FileOutputStream fos=new FileOutputStream(f);
ZipOutputStream os=new ZipOutputStream(fos)) {
os.putNextEntry(new ZipEntry("META-INF/MANIFEST.MF"));
ByteBuffer bb = cs.encode("Agent-Class: "+SelfAttacher.class.getName());
os.write(bb.array(), bb.arrayOffset()+bb.position(), bb.remaining());
os.write(10);
os.closeEntry();
}
return f;
}
// invoked when the agent is loaded into the JVM, pass inst back to the caller
public static void agentmain(String agentArgs, Instrumentation inst) {
synchronized(SelfAttacher.class) {
BACK_LINK=inst;
SelfAttacher.class.notifyAll();
}
}
}
You can modify bytecode of each method adding routine to log method's enter/exit events. Javassist will help you http://www.csg.ci.i.u-tokyo.ac.jp/~chiba/javassist/
Also check out a nice tutorial: https://today.java.net/pub/a/today/2008/04/24/add-logging-at-class-load-time-with-instrumentation.html
I have two public methods that I'd like to trace. One of the methods calls the other repeatedly. What I'd like to do is trace only the method that was called from the outside.
Here's a simple class to demonstrate what I mean:
public class LoggingExample {
private static final Logger logger = LoggerFactory.getLogger(LoggingExample.class);
public static final String USER_ROOT = "/home/waisbrot";
/** could be called by fileExistsRobust *or* from outside */
public static boolean fileExists(String filename) {
logger.trace("Checking for file {}", filename);
File f = new File(filename);
return f.exists();
}
/** always gets called from outside */
public static boolean fileExistsRobust(String filename) {
logger.trace("Checking for any varient of {}", filename);
if (fileExists(filename))
return true;
for (String prefix : prefixes) { // this list is 100 items long
if (fileExists(prefix + filename));
return true;
}
return false;
}
}
Elsewhere in my code, I might call fileExists, in which case I want its logging message to get printed (assuming I'm tracing it). But if I call fileExistsRobost than I want that log message, but not fileExists.
I want to have both methods traced, but I'm getting buried in output when I call the second one. I was hoping Logback could be configured to understand what I want, but I'm not seeing anything useful in the documentation. I could flip a flag when I enter fileExistsRobust and then test for it in fileExists, but that's going to get ugly with more than one thread (since these are static methods) and it seems like it starts polluting the class with lots of logging infrastructure. I could use MDC to store the info, but that seems like an abuse of MDC.
Anyone run into this situation before? How'd you deal with it?
I assume that you are able to change the code. Then the simplest way in my opinion is avoiding the problem by introducing another internalFileExists(String filename) or overloading fileExists(String filename) with a logging toogle:
public static boolean fileExists(String filename, boolean doLog) {
if (doLog) logger.trace("Checking for file {}", filename);
File f = new File(filename);
return f.exists();
}
and let fileExistsRobust use the overloaded version with doLog = false, while the single argument version redirects to fileExists(filename, true).
That does not really address the problem, but mitigates it.
I made a windows service from a jar file using WinRun4J, so far it's very basic.
package org.boris.winrun4j.test;
import java.io.BufferedWriter;
import java.io.FileWriter;
import org.boris.winrun4j.Service;
import org.boris.winrun4j.ServiceException;
public class ServiceWrite implements Service
{
private volatile boolean shutdown = false;
public int serviceMain(String[] args) throws ServiceException {
int count = 0;
while (!shutdown) {
try {
Thread.sleep(6000);
} catch (InterruptedException e) {
}
try {
FileWriter fstream = new FileWriter("result.txt");
BufferedWriter out = new BufferedWriter(fstream);
out.write("Counts: " + count);
out.close();
} catch (Exception e){
}
count++;
}
return 0;
}
public int serviceRequest(int control) throws ServiceException {
switch (control) {
case SERVICE_CONTROL_STOP:
case SERVICE_CONTROL_SHUTDOWN:
shutdown = true;
break;
}
return 0;
}
}
When the service is started it just keeps writing every couple of seconds to result.txt located in the root folder.. (Just for trying out WinRun4J)
Now my question is, can I do a method in the service jar, like this
public void write(String s){
//Write some string s to result.txt
}
And then invoke this method from a different java file on the system, i.e
java WriteToFile SomeString
Where WriteToFile is supposed to invoke write with some argument.
Is it possible? if so, how ?
The overall purpose of this is to have a service running where I can invoke methods via a GUI.
to "invoke methods via a GUI", you can't do it with WinRun4J.
in general rule, a Windows Service can't have a GUI for security reason (except for special cases).
However, there are other tools to create a windows service from a Java application, with which it will be possible to have a service with GUI and able to interact with the Desktop.