I am working on an application that retrieves files from different URL's.
There is a TreeSet that contains the target to download. This is processed in a loop with each item being called with an ExecutorService. Here's some code:
private void retrieveDataFiles() {
if (this.urlsToRetrieve.size() > 0) {
System.out.println("Target URLs to retrieve: " + this.urlsToRetrieve.size());
ExecutorService executorProcessUrls = Executors.newFixedThreadPool(this.urlsToRetrieve.size());//could use fixed pool based on size of urls to retrieve
for (Entry target : this.urlsToRetrieve.entrySet()) {
final String fileName = (String) target.getKey();
final String url = (String) target.getValue();
String localFile = localDirectory + File.separator + fileName;
System.out.println(localFile);
executorProcessUrls.submit(new WikiDumpRetriever(url, localFile));
dumpFiles.add(localFile);
//TODO: figure out why only 2 files download
}
executorProcessUrls.shutdown();
try {
executorProcessUrls.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException ex) {
System.out.println("retrieveDataFiles InterruptedException: " + ex.getMessage());
}
} else {
System.out.println("No target URL's were retrieved");
}
}
Then the WikiDumpRetriever:
private static class WikiDumpRetriever implements Runnable {
private String wikiUrl;
private String downloadTo;
public WikiDumpRetriever(String targetUrl, String localDirectory) {
this.downloadTo = localDirectory;
this.wikiUrl = targetUrl;
}
public void downloadFile() throws FileNotFoundException, IOException, URISyntaxException {
HTTPCommunicationGet httpGet = new HTTPCommunicationGet(wikiUrl, "");
httpGet.downloadFiles(downloadTo);
}
#Override
public void run() {
try {
downloadFile();
} catch (FileNotFoundException ex) {
System.out.println("WDR: FileNotFound " + ex.getMessage());
} catch (IOException ex) {
System.out.println("WDR: IOException " + ex.getMessage());
} catch (URISyntaxException ex) {
System.out.println("WDR: URISyntaxException " + ex.getMessage());
}
}
}
As you can see this is an inner class. The TreeSet contains:
Key : Value
enwiki-latest-pages-articles.xml.bz2 : http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
elwiki-latest-pages-articles.xml.bz2 : http://dumps.wikimedia.org/enwiki/latest/elwiki-latest-pages-articles.xml.bz2
zhwiki-latest-pages-articles.xml.bz2 : http://dumps.wikimedia.org/enwiki/latest/zhwiki-latest-pages-articles.xml.bz2
hewiki-latest-pages-articles.xml.bz2 : http://dumps.wikimedia.org/enwiki/latest/hewiki-latest-pages-articles.xml.bz2
The problem is that this process downloads 2 of the four files. I know that all four are available and I know that they can be downloaded. However, only 2 of them process at any time.
Can anyone shed any light on this for me please - what am I missing or what am I getting wrong?
Thanks
nathj07
Thanks to ppeterka - it was a limit from the source. So, to overcome this I set the fixed thread pool size to 2. This means that only 2 files are downloaded simultaneously.
The answer then was to find the vendor imposed limit and set the thread pool:
ExecutorService executorProcessUrls = Executors.newFixedThreadPool(2);
I wanted to accept an answer but couldn't seem to do it with the comments. Sorry if this was the wrong way to do it.
Thanks for all the pointers - the 'group think' really helped solve this for me.
Related
I have my client trying to lookup a JMS server. Here is my class JmsTest.java:
public static void main(String[] aInArgs)
{
boolean bContinue = true;
try
{
// determine JmsTest configuration based on command line arguments.
JmsTest jmsTest = parseCommandLine(aInArgs);
// connect to the server.
//jmsTest.initializeConnection();
Thread jmsFaultClientThread = null;
jmsFaultClientThread = new Thread("RUN") {
#Override
public void run() {
try
{
System.out.println("jmsFaultClient starting...");
jmsTest.initializeConnection();
}
catch (Exception e)
{
System.out.println("Exception: " + e.toString());
}
System.out.println("jmsFaultClient started.");
}
};
jmsFaultClientThread.start();
And my method initializeConnection():
public void initializeConnection() throws Exception
{
try
{
Hashtable env = new Hashtable();
env.put(Context.SECURITY_PRINCIPAL, user );
env.put(Context.SECURITY_CREDENTIALS, password);
jndiContext = new InitialContext(env);
System.out.println("Initializing Topic (" + strName + ")...");
try
{
topicConnectionFactory = (TopicConnectionFactory) jndiContext.lookup(CONNECTION_FACTORY);
}
catch (Exception e)
{
topicConnectionFactory = getExternalFactory(jndiContext);
}
When I run jmsTest.initializeConnection() like this everything works, and the lookup is working. However, the problem is when it's run inside the thread it gets stuck without any exception or any error. It's just stuck.
In my logs i'm seeing:
System.out.println("Initializing Topic (" + strName + ")...");
Which is a log inside my try / catch, and nothing else.
In dependencies, I have 2 jars, contening javax\jms. With the first one it's work inside the thread, and with the second one it doesn't. But I don't know why my jar can "block" the thread.
UPDATE 1 :
#AnotherJavaprogrammer said me to print the error:
here is my lookup with print :
try
{
getLogger().debug("TRY context");
Context lInitialContext = (Context) jndiContext.lookup(JMS_CONTEXT);
lInitialContext.lookup("SAMConnectionFactory");
getLogger().debug("END trying context");
}
catch (Exception e)
{
getLogger().debug("Catch");
getLogger().debug("Exception", e);
}
The output from getLogger().debug("END trying context") never comes, and I don't see the getLogger().debug("Catch") one either. So it appears I'm really "stuck" inside the lookup(). I can't go further, and it doesn't throw an exception.
WHAT?
I am trying to build a tool that will reads a text file and publishes the text, after doing some string transformation.
HOW?
The tool reads the file line by line and populates a LinkedBlockingQueue. At the same time I initiate multiple threads that will then take a message each from the LBQ, do some processing and publish them.
Main
private static LinkedBlockingQueue<String> lbQueue = new LinkedBlockingQueue<>();
private static Boolean keepPublisherActive = Boolean.TRUE;
public static void main(String[] args) {
try {
tool.initMessagePublish();
tool.searchUsingScanner();
} catch (Exception ex) {
logger.error("Exception in Tool Main() " + ex.toString());
throw ex;
}
}
File Reader
private void searchUsingScanner() {
Scanner scanner = null;
try {
scanner = new Scanner(new File(LOG_FILE_PATH));
while (scanner.hasNextLine()) {
String line = scanner.nextLine().trim();
if (StringUtils.isNotBlank(line)) {
lbQueue.offer(line);
}
}
} catch (Exception e) {
logger.error("Error while processing file: " + e.toString());
} finally {
try {
if (scanner != null) {
scanner.close();
}
// end thread execution
keepPublisherActive = false;
} catch (Exception e) {
logger.error("Exception while closing file scanner " + e.toString());
throw e;
}
}
}
Multi-threaded Publisher
private void initMessagePublish() throws InterruptedException {
ExecutorService service = Executors.newFixedThreadPool(6);
try {
while (keepPublisherActive || lbQueue.getSize() > 0) {
service.execute(messagePublisher); // messagePublisher implements Runnable
}
} catch (Exception ex) {
logger.error("Multi threaded message publish failed " + ex.toString());
throw ex;
} finally {
service.shutdown();
}
}
THE PROBLEM
The intention behind calling initMessagePublish() fist is that the publisher need not wait for all lines to be read from the file before starting to publish. It should start publishing as soon as something becomes available in the LBQ.
But with the current implementation, the control never comes out of the initMessagePublish and start searchUsingScanner. How do I solve this? Basically, the two methods should execute parallely.
Just start messagePublisher in a new Thread (Line no #5 in Main class):
new Thread(()->tool.initMessagePublish()).start();
It should solve your problem.
From time to time I come across a class where a bigger part of code are logging calls.
e.g.
public init(Config config) {
logger.info("Configuring ...");
if (config.hasInitInterval()) {
initInterval = config.getInitInterval();
logger.info("Set initInterval to " + initInterval);
}
...
try {
logger.info("Updating access points " + config.getAccessPoints());
updateAccessPoints(config.getAccessPoints())
} catch (Throwable e) {
logger.warn("Init failed due to ", e);
}
...
if (logger.isDebugEnabled ()) {
for(int i = 0; i < config.getModifiers().size(); i++) {
try {
isValidModifier(config.getModifiers().get(i));
} catch (Exception e) {
throw new IllegalArgumentException ("Wrong modifier: " config.getModifiers().get(i));
}
}
}
}
When a class is not formatted well plus contains comments, it's hard to read the code.
I used proxy pattern to partially improve it but it's suitable only to log something before or after a call of method.
What are the best practices to separate functionality from logging?
I am kinda new to BlackBerry OS development, my code is trying to get name of the city while using LocationProvider after passing Criteria parameter.
i was following according to this link from RIM itself i tried "JSR-179" and "JSR-179 Extension"
My code is as follows:
"with JSR 179"
public String CurrentLocation()
{
try
{
//LBS(Location Based Service) JSR-179
Criteria criteria = new Criteria();
criteria.isAddressInfoRequired();
criteria.setCostAllowed(false);
criteria.setAddressInfoRequired(true);
criteria.setHorizontalAccuracy(200);
criteria.setVerticalAccuracy(200);
locationProvider = LocationProvider.getInstance(criteria);
location = locationProvider.getLocation(10);
cityName = location.getAddressInfo().getField(AddressInfo.CITY);
}
catch (LocationException le)
{
new Tracer("CurrentLocation() caught LocationException : " + le.getMessage());
le.printStackTrace();
}
catch (InterruptedException ire)
{
new Tracer("CurrentLocation() caught InterruptedException: " + ire.getMessage());
ire.printStackTrace();
}
return cityName;
}
with JSR-179 Extension
public String CurrentLocaiton(){
try
{
BlackBerryCriteria BBCriteria = new BlackBerryCriteria();
BBCriteria.setAddressInfoRequired(ProvideAddressInfoTExt);
BBCriteria.setAltitudeRequired(true);
BBCriteria.setSatelliteInfoRequired(true, true);
BBCriteria.setCostAllowed(true);
BBCriteria.setPreferredPowerConsumption(BlackBerryCriteria.POWER_USAGE_HIGH); //testing for high power
BlackBerryLocationProvider bbLocationProvider
= (BlackBerryLocationProvider)
BlackBerryLocationProvider.getInstance(BBCriteria);
location = bbLocationProvider.getLocation(9000);
QualifiedCoordinates qualCoor = location.getQualifiedCoordinates();
cityName = location.getAddressInfo().getField(AddressInfo.CITY);
}
catch(LocationException le)
{
le.printStackTrace();
System.out.println(le.getMessage());
}
catch(InterruptedException ie)
{
ie.printStackTrace();
System.out.println(ie.getMessage());
}
catch(NullPointerException npe)
{
npe.printStackTrace();
cityName = "Caught Null Pointer";
Dialog.alert(npe.getMessage());
}
return cityName;
}
Am I missing something there or is there something wrong I have been scratching on this issue for hours now.
I tried to catch Null Pointer Exception from this code snippet as well and if catching from "static void main" then Uncaught Exception: pushModalScreen called by a non-event thread
....
try{
String location = cellLocation.CurrentLocation();
LabelField towerlocationText = new LabelField(towerString + location,
LabelField.FOCUSABLE|LabelField.DEFAULT_POSITION);
add(towerlocationText);
}
catch(NullPointerException npe)
{
npe.printStackTrace();
System.out.println(npe.getMessage());
Dialog.alert(npe.getMessage() + "" + "AddresInfo/LocationProvider/Locaiton gave null pointer exception" );
}
...
tried on BB-9700 BB-9790 Devices as well on Simulator BB-9900
Thanks,
use
Either :-
/***/
UiApplication.getUiApplication().invokeLater(new Runnable()
{
public void run()
{
Dialog.alert("Hello");
}
});
/***/
or
synchronized (UiApplication.getEventLock())
{
Dialog.alert("Hello");
}
wherever you are trying to show Dialog on Non-event thread , you should need a Event Lock.
You can use above method to make Event Lock
read the article for more info .
This builds up on my previous question.
My ftp server has 10 files, say test1.txt, test2.txt and so on. I want to be able to download multiple files (max 3) at the same time. I am calling downloadFilesByPattern(....) If I dont use synchronized on downloadFile() then only some of the file are downloaded not all. If I use synchronized, then all the files are downloaded, but I don't think they are happening in parallel. Is the issue because an instance varible is passed to all threads and a method on that instance is called by all threads.
public class FTPClientService implements IClient {
private String username;
private String password;
private String port;
private String host;
private String path;
FTPClient client = new FTPClient();
private static class DownloadTask implements Runnable {
private String name;
private String toPath;
private IClient client;
public DownloadTask(String name, String toPath, IClient client) {
this.name = name;
this.toPath = toPath;
this.client = client;
}
#Override
public void run() {
System.out.println("download = " + name);
client.downloadFile(name, toPath);
}
}
public void downloadFilesByPattern(String fileNamePattern, final String outputFilePath) {
if(!changeDirectory()){
return;
}
try {
//get a list of file names that match the pattern
String[] names = client.listNames();
ExecutorService pool = Executors.newFixedThreadPool(3);
for (String name : names) {
//check if the filename matches the pattern
Pattern pattern = Pattern.compile(fileNamePattern);
Matcher matcher = pattern.matcher(name);
if(matcher.find()){
System.out.println("Match found = " + name);
pool.submit(new DownloadTask(name, outputFilePath, this));
}else{
System.out.println("No match = " + name);
}
}
pool.shutdown();
try {
pool.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
} catch (InterruptedException ex) {
}
} catch (IOException ex) {
}
}
public synchronized void downloadFile(String fileName, String outputFilePath) {
FileOutputStream fos = null;
try {
fos = new FileOutputStream(outputFilePath+"/"+fileName);
if(this.isFilePresent(fileName)){
//look for fileName in the path and write it to
client.retrieveFile(fileName, fos);
System.out.println("Downloading file " + fileName + " ...");
}else{
System.out.println("Could not find file " + fileName);
}
} catch (IOException ex) {
} finally {
try {
fos.close();
} catch (IOException ex) {
}
}
}
}
It's because they all use the same instance of
FTPClient client
You need to either create new instance of FTPClientService for every download/thread or have an instance of FTPClient for every thread. I personally prefer the second variant which can be easily implemented using ThreadLocal.
FTPClient is probably not thread safe (what product is it coming from anyway?). You may want to create it right before download or create a pool of FTP clients if you need to reuse it.
Also, I recommend you modify your naming convention a bit as it's very hard to distinguish in the code ftpclient vs your own client.