This builds up on my previous question.
My ftp server has 10 files, say test1.txt, test2.txt and so on. I want to be able to download multiple files (max 3) at the same time. I am calling downloadFilesByPattern(....) If I dont use synchronized on downloadFile() then only some of the file are downloaded not all. If I use synchronized, then all the files are downloaded, but I don't think they are happening in parallel. Is the issue because an instance varible is passed to all threads and a method on that instance is called by all threads.
public class FTPClientService implements IClient {
private String username;
private String password;
private String port;
private String host;
private String path;
FTPClient client = new FTPClient();
private static class DownloadTask implements Runnable {
private String name;
private String toPath;
private IClient client;
public DownloadTask(String name, String toPath, IClient client) {
this.name = name;
this.toPath = toPath;
this.client = client;
}
#Override
public void run() {
System.out.println("download = " + name);
client.downloadFile(name, toPath);
}
}
public void downloadFilesByPattern(String fileNamePattern, final String outputFilePath) {
if(!changeDirectory()){
return;
}
try {
//get a list of file names that match the pattern
String[] names = client.listNames();
ExecutorService pool = Executors.newFixedThreadPool(3);
for (String name : names) {
//check if the filename matches the pattern
Pattern pattern = Pattern.compile(fileNamePattern);
Matcher matcher = pattern.matcher(name);
if(matcher.find()){
System.out.println("Match found = " + name);
pool.submit(new DownloadTask(name, outputFilePath, this));
}else{
System.out.println("No match = " + name);
}
}
pool.shutdown();
try {
pool.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
} catch (InterruptedException ex) {
}
} catch (IOException ex) {
}
}
public synchronized void downloadFile(String fileName, String outputFilePath) {
FileOutputStream fos = null;
try {
fos = new FileOutputStream(outputFilePath+"/"+fileName);
if(this.isFilePresent(fileName)){
//look for fileName in the path and write it to
client.retrieveFile(fileName, fos);
System.out.println("Downloading file " + fileName + " ...");
}else{
System.out.println("Could not find file " + fileName);
}
} catch (IOException ex) {
} finally {
try {
fos.close();
} catch (IOException ex) {
}
}
}
}
It's because they all use the same instance of
FTPClient client
You need to either create new instance of FTPClientService for every download/thread or have an instance of FTPClient for every thread. I personally prefer the second variant which can be easily implemented using ThreadLocal.
FTPClient is probably not thread safe (what product is it coming from anyway?). You may want to create it right before download or create a pool of FTP clients if you need to reuse it.
Also, I recommend you modify your naming convention a bit as it's very hard to distinguish in the code ftpclient vs your own client.
Related
I am new at stackoverflow and I am sorry if this kind of a question is asked before but did a quick search and I could not find any title like mine. I am working on a multi-client chat application on Java. I was following the tutorials and I can send messages that every user in the application can see. But I wonder how to create and send a private message to a spesific user into the chat.
import java.io.*;
import java.net.*;
import java.util.HashSet;
import java.util.Set;
public class ChatServer {
private int port;
private Set<String> userNames = new HashSet<>();
private Set<UserThread> userThreads = new HashSet<>();
public ChatServer(int port) {
this.port = port;
}
public static void main(String[] args) {
new ChatServer(9999).execute();
}
private void execute() {
try {
ServerSocket serverSocket = new ServerSocket(9999);
System.out.println("Server is running");
while (true) {
Socket socket = serverSocket.accept();
System.out.println("New user connected");
UserThread newUser = new UserThread(socket, this);
userThreads.add(newUser);
newUser.start();
}
} catch (IOException e) {
e.printStackTrace();
}
}
public void addUserName(String s) {
this.userNames.add(s);
}
public void broadcast(String serverMessage, UserThread excludeUser) {
for (UserThread aUser : userThreads) {
if (aUser != excludeUser)
aUser.sendMessage(serverMessage);
}
}
}
The code above is my server code.
public void run() {
Console console = System.console();
String userName = console.readLine("Enter your username : ");
writer.println(userName);
String text;
do {
text = console.readLine("[" + userName + "]: ");
if (text.startsWith("[")) {
isTargeted = true;
this.aimUserName = text.substring(text.indexOf("[") + 1, text.indexOf("]"));
//System.out.println("Private Message to: " + aimUserName);
} else {
isTargeted = false;
}
writer.println(text);
} while (!text.equals("bye"));
try {
socket.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
and this code above is a part of my write thread class. As you can see, if a message starts with '[name]' part, the "name" means the user that we want to send a private message. By doing this, I can get the name of the user that I want to send a private message but I could not figure out how to broadcast this message just to that spesific user. I believe I need to configure my broadcast function in ChatServer class but I don't really know how to do. What steps should I follow?
--Edit--
I've been working on my question and I did some additions to solve my problem. First of all, I think I should share everything I have to you. I shared my ChatServer class previously. Other classes I have are:
import java.io.IOException;
import java.net.Socket;
public class ChatClient {
public static void main(String[] args) {
new ChatClient().execute();
}
private void execute() {
try {
Socket socket = new Socket("localhost", 3);
System.out.println("Connected to chat server");
new ReadThread(socket, this).start();
new WriteThread(socket, this).start();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
import java.net.*;
import java.io.*;
public class ReadThread extends Thread{
private BufferedReader reader;
private Socket socket;
private ChatClient client;
public ReadThread(Socket socket, ChatClient client) {
this.socket = socket;
this.client = client;
InputStream input;
try {
input = this.socket.getInputStream();
this.reader = new BufferedReader(new InputStreamReader(input));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void run() {
while(true) {
try {
String response = this.reader.readLine();
System.out.println("\n" + response);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
break;
}
}
}
}
import java.net.Socket;
import java.io.*;
public class UserThread extends Thread {
private Socket socket;
private ChatServer server;
PrintWriter writer = null;
public String userName;
public UserThread(Socket socket, ChatServer chatServer) {
this.socket = socket;
this.server = chatServer;
}
public void run() {
try {
InputStream input = socket.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
OutputStream output = socket.getOutputStream();
writer = new PrintWriter(output,true);
String userName = reader.readLine();
this.userName = userName;
server.addUserName(userName);
String serverMessage = "New user connected: " + userName;
server.broadcast(serverMessage,this);
String clientMessage;
do {
clientMessage = reader.readLine();
serverMessage = "[" + userName + "] : " + clientMessage;
server.broadcast(serverMessage, this);
}while(!clientMessage.equals("bye"));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void sendMessage(String serverMessage) {
writer.println(serverMessage);
}
}
import java.net.*;
import java.io.*;
public class WriteThread extends Thread {
private Socket socket;
private ChatClient client;
private PrintWriter writer;
public WriteThread(Socket socket, ChatClient client) {
this.socket = socket;
this.client = client;
OutputStream output;
try {
output = socket.getOutputStream();
this.writer = new PrintWriter(output, true);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void run() {
Console console = System.console();
String userName = console.readLine("Enter your username : ");
writer.println(userName);
String text;
do {
text = console.readLine("[" + userName + "]: ");
if(text.startsWith("[")){
String aimUserName = text.substring(text.indexOf("[")+1,text.indexOf("]"));
System.out.println("Private Message to: " + aimUserName);}
writer.println(text);
}while(!text.equals("bye"));
/*do {
text = console.readLine("[" + userName + "]: ");
writer.println(text);
}while(!text.equals("bye"));*/
try {
socket.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
These codes work properly and I can multi-chat very clean. But while working on private chat stuff, I added to the ChatServer the line of:
public void privatebr(String serverMessage, String targetUserName){
for(UserThread aUser: userThreads){
if(aUser.userName == targetUserName)
aUser.sendMessage(serverMessage);
}
to the UserThread, I edited the part as:
String clientMessage;
do {
clientMessage = reader.readLine();
serverMessage = "[" + userName + "] : " + clientMessage;
if(clientMessage.startsWith("[")){
String targetUserName = clientMessage.substring(clientMessage.indexOf("[")+1,clientMessage.indexOf("]"));
serverMessage = "[" + userName + "] : " + clientMessage;
server.privatebr(serverMessage, targetUserName);
}else{
server.broadcast(serverMessage, this);
}
}while(!clientMessage.equals("bye"));
But when I did all these edits, the normal multi-chat progress became broken where is my fault? Why everything has broken?
Good question! To answer the question you asked is that you should maintain a Map of Users to their Socket connections, so that way with DMs you can just select the user(s) that you want to message. You will also need a messaging protocol for that (see below)
...But I have to tell you that using Sockets and SocketServer classes in today's day and age is like re-inventing the wheel. The place to start in doing a chat server is using the web sockets protocol. Even under this, you will probably want to define a message protocol (like I did - I created a messaging protocol using JSON and message types, where the string message in the websocket event onMessage first gets parsed into an object)
There are implementations for supporting WS on all platforms: java, .net, python, php etc. This should be your starting point.
--- Update ---
I understand where you are coming from. To help you along in understanding Sockets / ServerSockets, here are a couple of pointers & resources
DatagramSockets (aka UDP): This is a different transmission protocol than the regular TCP, used by Shockwave and then Flash, and is the fundamental reason that Flash is problematic. I strongly recommend against this
Data & Object Input/OutputStreams: "Data" streams are Java only (can't connect to technologgy built on other platforms). Object streams are similar, except you are transporting actual whole objects through the stream (also Java only) No one* (almost no one) uses these anymore.
SocketException: Using java.net.[Server]Socket(s), you are likely to encounter this exception. It happens when you are waiting for more data (through a read / readLine call) on a socket, and the socket closes. It took me a long time to figure this out, but THIS EXCEPTION IS YOUR FRIEND! You get it when the connection has closed (either on the client or server side). It allows the thread that was waiting on the socket to wake up, and allows you to do whatever clean-up you need to do. SocketException is a subclass of IOException, so you may not even realize what this is. But now at least I have warned you
Streams vs. Writers and Readers: Writers and Readers are for interpreting raw bytes as Java characters and Strings. This is necessary, as there are multiple text formats (i.e. ascii, windows-xx, utf-8, utf-16). Readers and Writers help you read and write text in different text formats (and also interpreting Images from image formats).
Buffered Writers and Streams: These are for INEFFICIENT reading and writing. For writing, this means enabling you to write part of a message and not send it until you are ready. For reading, this means reading streams line by line for example rather than reading everything at one go.
TUS: tjacobs/io - https://sourceforge.net/p/tus/code/HEAD/tree/tjacobs/io/ this is an old collection of Java libraries I put on SourceForge years ago, but a lot of the classes here pertain to dealing with Sockets. In particular, see SocketServerEx, DataFetcher, Main/App, Timeout, and maybe IOUtils. And of everything, really look at DataFetcher which is a lightweight threadableframework for Callback I/O listening.
Good luck and have fun!
I have a program that starts by creating a GUI to handle user input and display the output.
The first thing that happens is the window is created and then the Functions Class method initServer() is called to initialize some variables for the input and output portion
private JFrame frame;
public static Functions func = new Functions();
public static void main(String[] args) {
EventQueue.invokeLater(new Runnable() {
public void run() {
Thread.currentThread().setName("Console");
System.out.println(Thread.currentThread().getName() + " [" + Thread.currentThread().getId() + "] Started");
try {
Console window = new Console();
window.frame.setVisible(true);
} catch (Exception e) {
e.printStackTrace();
}
func.initServer();
}
});
}
With the Functions Class method initServer() being called, We start the process then follow by initializing the i/o variable that will handle all the streams being used to communicate with the process. Then we start the two threads - ConsoleInputWriter and ConsoleOutputReader - responsible for handling Input and Output to the process.
public class Functions {
private ConsoleOutputReader cor = new ConsoleOutputReader();
private ConsoleInputWriter ciw = new ConsoleInputWriter();
private OutputStreamWriter osw;
private InputStreamReader isr;
private BufferedWriter bw;
private BufferedReader br;
private BufferedReader input = new BufferedReader(new InputStreamReader(System.in));
private File serverJar;
private String serverPath;
private ProcessBuilder builder;
private Process proc;
private boolean init = false;
public void initServer()
{
updateConsole("Server Initiated Status: " + serverStatus());
builder = new ProcessBuilder("/bin/bash");
try {
proc = builder.start();
} catch (IOException e) {
e.printStackTrace();
}
osw = new OutputStreamWriter(proc.getOutputStream());
bw = new BufferedWriter(osw);
isr = new InputStreamReader(proc.getInputStream());
br = new BufferedReader(isr);
serverStatus(true);
updateConsole("Server Initiated Status: " + serverStatus());
cor.start();
ciw.start();
}
public String recieveInput()
{
String s = null;
try {
s = input.readLine();
} catch (IOException e) {
e.printStackTrace();
}
return s;
}
public boolean serverStatus()
{
return init;
}
public void serverStatus(boolean status)
{
init = status;
}
public void exec(String cmd)
{
try {
bw.write(cmd);
bw.newLine();
bw.flush();
} catch (IOException e) {
updateConsole("Cant run: [" + cmd + "] :::::::: " + e);
e.printStackTrace();
}
}
public void updateConsole()
{
//edit to print to textPane
try {
System.out.println(br.readLine());
} catch (IOException e) {
}
}
public void updateConsole(String s)
{
System.out.println(s);
}
public File getJar(/**String s**/)
{
serverJar = new File(Functions.class.
getResource("CraftBukkit.jar").getPath());
return serverJar;
}
public void setPath(String s)
{
serverPath = s;
}
public String getPath()
{
return serverPath;
}
}
Once called the Class ConsoleOutputReader starts and executes a command for the process to start a Jar File and confirms that the i/o streams have been initialized before it tries to get any output. If it continues to the while loop we should be getting output.
public class ConsoleOutputReader extends Thread{
private static Functions func = new Functions();
public void run()
{
currentThread().setName("cor");
System.out.println(currentThread().getName() + " [" + Thread.currentThread().getId() + "] Started");
func.exec("cd " + "~/Desktop/Bukkit" + " && java -Xmx1024M -jar " + func.getJar() + " -o true");
while(func.serverStatus())
func.updateConsole();
}
}
and the Class ConsoleInputWriter follows right after ConsoleOutputReader also confirming that the serverInit() boolean is true, then to wait in a while loop for an input from the user.
public class ConsoleInputWriter extends Thread{
public static Functions func = new Functions();
public void run()
{
currentThread().setName("ciw");
func.updateConsole(currentThread().getName() + " [" + Thread.currentThread().getId() + "] Started");
while(func.serverStatus())
func.exec(func.recieveInput());
}
}
The main issue I have is that with minimal knowledge on threads I seemed to have made the serverStatus() boolean from the Functions class equal two different things. where the output from ConsoleOutputReader is true and the output from ConsoleInputWriter is false. How would I make sure that when I start both threads they're seeing the same value when they call the method?
I've gotten this code to work with two threads where the main thread ran the inputs and a second thread was used to run the outputs, but I wanted to try it setup like this.
Any tips to my style and or patterns I use are also very welcome.
Edit: I realized with all my frantic changes that whatever class calls initServer() is that class that get true when they call serverStatus().
Any methods that change an object for both threads need to be synchronized. When the method runs, if the object is being read by one or both of the threads while it changes, The threads could read different values.
e.x.:
public static synchronized void initServer(boolean bool) { init = bool; }
i am creating a class(CheckCon.java) that implements callable interface which basically returns the no. of devices Connected to the network. the problem is that i donot know how to execute it correctly because the results returned are really slow as compared to traditional multithreading. and i need to return values to a class in (NetScan.java). Kindly Help me execute it properly.
code for CheckCon.java (callable implementing class):
public class CheckCon implements Callable<Object>{
int startindex,endindex;
ArrayList<Object> list;
byte[] ip;
public CheckCon(int startindex, int endindex, byte[] ip) {
this.startindex = startindex;
this.endindex = endindex;
this.ip = ip;
list = new ArrayList<>();
}
#Override
public ArrayList<Object> call() throws Exception {
for(int i =startindex;i<endindex;i++){
try {
ip[3] = (byte)i;
InetAddress address = InetAddress.getByAddress(ip);
if (address.isReachable(1000))
{
System.out.println("Name is......"+address.getHostName()+"\tIP is......."+address.getHostAddress());
}
else if (!address.getHostAddress().equals(address.getHostName()))
{
String host = address.getCanonicalHostName();
String ipaddress =address.toString().replace(host+"/", "");
Object[] data = {host,ipaddress};
list.add(data);
System.out.println("Name is......"+address.getHostName()+"\tIP is......."+address.getHostAddress());
}
else
{
System.out.println("nothing");
// the host address and host name are equal, meaning the host name could not be resolved
} } catch (UnknownHostException ex) {
Logger.getLogger(NetScan.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(NetScan.class.getName()).log(Level.SEVERE, null, ex);
}
}
return list;
}
}
and the class Calling NetScan.java
private void getDataForTable() throws InterruptedException, ExecutionException {
ExecutorService executor =Executors.newCachedThreadPool();
for(int i =0; i<26 ; i++){
if(s == 250){
Future<Object> f =executor.submit(new CheckCon(s,s+5,ip));
list.add(f);
break;
}else{
Future<Object> f =executor.submit(new CheckCon(s,s+10,ip));
list.add(f);
s= s+10;
}
}
dm = new DefaultTableModel(ColumnName,0){
#Override
public boolean isCellEditable(int row, int column) {
return false;
}
};
for(Future<Object> f : list){
dm.addRow((Object[]) f.get());
}}
I am creating 25 threads checking 10 IP's in each thread.
I removed the CheckCon.java and edited the NetScan to get what i Want thanks to #amit-bera . i Created a thread and asked the completablefuture to runasync with executor defining the no.of threads i require.
ExecutorService e =Executors.newFixedThreadPool(25);
for(int i =0;i<255;i++){
final int j =i;
f = CompletableFuture.runAsync(new Runnable() {
#Override
public void run() {
try {
ip[3] = (byte)j;
InetAddress address = InetAddress.getByAddress(ip);
if (address.isReachable(1000))
{
System.out.println("Name is......"+address.getHostName()+"\tIP is......."+address.getHostAddress()+j);
}
else if (!address.getHostAddress().equals(address.getHostName()))
{
String host = address.getCanonicalHostName();
String ipaddress =address.toString().replace(host+"/", "");
Object[] data = {host,ipaddress};
dm.addRow(data);
System.out.println("Name is......"+address.getHostName()+"\tIP is......."+address.getHostAddress()+j);
}
else
{
System.out.println("nothing"+j);
// the host address and host name are equal, meaning the host name could not be resolved
} } catch (UnknownHostException ex) {
Logger.getLogger(NetScan.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(NetScan.class.getName()).log(Level.SEVERE, null, ex);
}
}
},e);
}
added this content to a method.
As per my obervationa please used singolton java pettern.example i am giving below.singolton alwaus retun instance so you save request response time on thread.
Thread is light-wight process but still initilised thread at whenever need.please make sure that thread work based on your server configrationa.
if you are using linux system
used TOP command to usage of your system perfomance and process execution on threadiwse.
i try and its work fine on my system.
class Singleton
{
private static Singleton single_instance = null;
public String s;
private Singleton()
{
s = "Hello I am a string part of Singleton class";
}
public static Singleton getInstance()
{
if (single_instance == null)
single_instance = new Singleton();
return single_instance;
}
}
I've working on an assignment that asks me to alter a method in a class to take content from a textfile and use it to create multiple instances of various subclasses of the Event Class. Here is the text file:
Event=ThermostatNight,time=0
Event=LightOn,time=2000
Event=WaterOff,time=8000
Event=ThermostatDay,time=10000
Event=Bell,time=9000
Event=WaterOn,time=6000
Event=LightOff,time=4000
Event=Terminate,time=12000
The Event=* is the name of the subclass, while time=* is a parameter that is used in the subclass' constructor. The Event class itself is an abstract class and is used for inheritance.
public class Restart extends Event {
Class eventClass;
String eventInput;
Long timeDelay;
public Restart(long delayTime, String filename) {
super(delayTime);
eventsFile = filename;
}
public void action() {
List<String> examples = Arrays.asList("examples1.txt", "examples2.txt", "examples3.txt", "examples4.txt");
for (String example : examples) {
//finding pattern using Regex
Pattern pattern = Pattern.compile(example);
Matcher matcher1 = pattern.matcher(eventsFile);
if (matcher1.find()) {
File file = new File(example);
String line;
try {
FileReader fileReader = new FileReader(file);
Scanner sc = new Scanner(file);
BufferedReader bufferedReader =
new BufferedReader(fileReader);
while ((line = bufferedReader.readLine()) != null) {
sc.useDelimiter("\n");
//Parsing through text
while (sc.hasNext()) {
String s = sc.next();
String[] array1 = s.split(",");
String[] array2 = array1[0].split("=");
eventInput = array2[1];
String[] array3 = array1[1].split("=");
String timeInput = array3[1];
try {
eventClass = Class.forName(eventInput);
timeDelay = Long.parseLong(timeInput);
try {
addEvent(new eventClass(timeDelay));
}
//catch block
catch(NoSuchMethodException e){
System.out.println("No Such Method Error");
} catch (Exception e) {
System.out.println("error");
}
//catch block
} catch (ClassNotFoundException ex) {
System.out.println("Unable to locate Class");
} catch (IllegalAccessException ex) {
System.out.println("Illegal Acces Exception");
} catch (InstantiationException ex) {
System.out.println("Instantiation Exception");
}
}
}
//Close bufferedReader
bufferedReader.close();
}
//catch block
catch (FileNotFoundException ex) {
System.out.println(
"Unable to open file '" +
file + "'");
} catch (IOException ex) {
ex.printStackTrace();
}
break;
}
//if input match is not found
else {
System.out.println("No Match Found");}
}
}
I seem to be able to parse fine, and find the strings i'm looking for, but I'm not able to use eventInput which I've pulled from the text file as a parameter to create a new event.
eventClass = Class.forName(eventInput);
doesn't seem to be turning my string into an acceptable parameter either.
Any help would be much appreciated!
I know I'm probably missing something key here, but I've been staring at it too long that it seems like a lost cause.
Here is the Event class:
public abstract class Event {
private long eventTime;
protected final long delayTime;
public Event(long delayTime) {
this.delayTime = delayTime;
start();
}
public void start() { // Allows restarting
eventTime = System.currentTimeMillis() + delayTime;
}
public boolean ready() {
return System.currentTimeMillis() >= eventTime;
}
public abstract void action();
} ///:~
I think you've misunderstood how reflection works. Once you have a Class object (the output from Class.forName(), you have to find the appropriate constructor with
Constructor<T> constructor = eventClass.getConstructor(parameter types)
and then create a new instance with
constructor.newInstance(parameters);
For a no-arg constructor there's a shortcut
eventClass.newInstance();
I strongly suggest you read the tutorials on reflection before proceeding.
I am working on an application that retrieves files from different URL's.
There is a TreeSet that contains the target to download. This is processed in a loop with each item being called with an ExecutorService. Here's some code:
private void retrieveDataFiles() {
if (this.urlsToRetrieve.size() > 0) {
System.out.println("Target URLs to retrieve: " + this.urlsToRetrieve.size());
ExecutorService executorProcessUrls = Executors.newFixedThreadPool(this.urlsToRetrieve.size());//could use fixed pool based on size of urls to retrieve
for (Entry target : this.urlsToRetrieve.entrySet()) {
final String fileName = (String) target.getKey();
final String url = (String) target.getValue();
String localFile = localDirectory + File.separator + fileName;
System.out.println(localFile);
executorProcessUrls.submit(new WikiDumpRetriever(url, localFile));
dumpFiles.add(localFile);
//TODO: figure out why only 2 files download
}
executorProcessUrls.shutdown();
try {
executorProcessUrls.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException ex) {
System.out.println("retrieveDataFiles InterruptedException: " + ex.getMessage());
}
} else {
System.out.println("No target URL's were retrieved");
}
}
Then the WikiDumpRetriever:
private static class WikiDumpRetriever implements Runnable {
private String wikiUrl;
private String downloadTo;
public WikiDumpRetriever(String targetUrl, String localDirectory) {
this.downloadTo = localDirectory;
this.wikiUrl = targetUrl;
}
public void downloadFile() throws FileNotFoundException, IOException, URISyntaxException {
HTTPCommunicationGet httpGet = new HTTPCommunicationGet(wikiUrl, "");
httpGet.downloadFiles(downloadTo);
}
#Override
public void run() {
try {
downloadFile();
} catch (FileNotFoundException ex) {
System.out.println("WDR: FileNotFound " + ex.getMessage());
} catch (IOException ex) {
System.out.println("WDR: IOException " + ex.getMessage());
} catch (URISyntaxException ex) {
System.out.println("WDR: URISyntaxException " + ex.getMessage());
}
}
}
As you can see this is an inner class. The TreeSet contains:
Key : Value
enwiki-latest-pages-articles.xml.bz2 : http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
elwiki-latest-pages-articles.xml.bz2 : http://dumps.wikimedia.org/enwiki/latest/elwiki-latest-pages-articles.xml.bz2
zhwiki-latest-pages-articles.xml.bz2 : http://dumps.wikimedia.org/enwiki/latest/zhwiki-latest-pages-articles.xml.bz2
hewiki-latest-pages-articles.xml.bz2 : http://dumps.wikimedia.org/enwiki/latest/hewiki-latest-pages-articles.xml.bz2
The problem is that this process downloads 2 of the four files. I know that all four are available and I know that they can be downloaded. However, only 2 of them process at any time.
Can anyone shed any light on this for me please - what am I missing or what am I getting wrong?
Thanks
nathj07
Thanks to ppeterka - it was a limit from the source. So, to overcome this I set the fixed thread pool size to 2. This means that only 2 files are downloaded simultaneously.
The answer then was to find the vendor imposed limit and set the thread pool:
ExecutorService executorProcessUrls = Executors.newFixedThreadPool(2);
I wanted to accept an answer but couldn't seem to do it with the comments. Sorry if this was the wrong way to do it.
Thanks for all the pointers - the 'group think' really helped solve this for me.