Develop Reference

How to MultiThread HttpURLConnection in Java

Hi guys, I have a client that would like to check the variations around their website. They have 5 million URLs to check. If I was to send requests/pings synchronously, it would take me 23 days. So I'm looking for a multithreaded solution. I originally started this problem off in Python, but didn't see much improvement/couldn't scale well, so here I am in Java, and if this fails too, I'll try in Go before throwing in the towel.
The issue is I'm not seeing any improvements at all with multithreading. Perhaps I'm implementing it wrong, could anyone please help me?
Edits:
I'll just be making edits here and new comers can look at the history of this post to see how I've progressed through the problem.
This is the socket suggestion, fails when I try to run it in a thread, unsure what I'm doing wrong here too.
Main Class:
package com.company;
import java.io.IOException;
import java.util.ArrayList;
import java.util.concurrent.TimeUnit;
public class Main extends Thread{
public static void main(String[] args) throws IOException {
long startTime = System.nanoTime();
Helpers.get("www.google.com", 80); // works here
String path = "test.txt";
boolean append = true;
for (int x = 0; x < 1; x++) {
ArrayList<String> urls = new ArrayList<String>();
// when x = 0, y = 0 | 10 /\ when x = 1, y = 10 | 20
for (int y= x * 10;y < ((x + 1) * 10); y++){
urls.add(String.format("www.google%d.com/", y)); // doesn't work here
}
Thread thread = new Thread(new Helpers(path, append, urls, 80));
thread.start();
thread.interrupt();
}
long endTime = System.nanoTime();
long duration = TimeUnit.NANOSECONDS.toMillis(endTime - startTime);
System.out.println(duration + " ms");
}
}
Helpers Class:
package com.company;
import java.io.IOException;
import java.net.*;import java.io.FileWriter;
import java.io.PrintWriter;import java.util.ArrayList;
public class Helpers extends Thread{
public Helpers(String path, boolean append, ArrayList<String> urls, int port) throws IOException {
this.run(path, append, urls, port);
}
public void run(String path, boolean append, ArrayList<String> urls, int port) throws IOException {
for (String url : urls) {
String status = Helpers.get(url, port);Helpers.writeToFile(path, append, status);
System.out.println(status);
}
}
public static String get(String url, int port) throws IOException {
try {
Socket conn = new Socket(url, 80);
conn.close();
return url + " | Success";
}catch (UnknownHostException error){return url + " | Failed";
}
}

You are implementing it wrong. You should make Helpers class extend thread or implement runnable. Pass everything you need to this class, for example, url, file pointer etc.
In your main class, create object of Helper and then run it as a thread.

You can try a different approach. Instead of making HTTPConnection for every call, you can try creating a socket connection to the webserver and then make multiple calls (GET/HEAD) to different URLs.
/**
* hostname of the webserver e.g. www.w3.org
* #param hostname
* #param urlList
* #throws IOException
*/
public static void makingHTTPCall( String hostname , List<String> urlList) throws IOException {
SSLSocketFactory factory = (SSLSocketFactory)SSLSocketFactory.getDefault();
SSLSocket socket =
(SSLSocket)factory.createSocket(hostname, 443);
BufferedReader in
= new BufferedReader(
new InputStreamReader(socket.getInputStream()));
PrintWriter out
= new PrintWriter(socket.getOutputStream(), true);
/**
* if required create different url List and pass those list to separate thread for better performance
*/
urlList.forEach(
url -> {
System.out.println("Making call to url /" + url);
out.println("HEAD " + url + " HTTP/1.1\r\n");
out.flush();
String line = "";
try {
while ((line = in.readLine()) != null) {
System.out.println("Response" + line);
break;
}
} catch (IOException e) {
e.printStackTrace();
}
}
);
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
out.close();
}
I have tried with the SSL socket you can change this according to your configuration.

How to edit a file inside a zip remotely on Secure_CRT?

I have a task to edit a file inside of a zip on SecureCRT.
I am able to run Linux commands remotely using JSCH library (com.jcraft.jsch)
Here is part of my code:
Session session = setUpSession(testParameters, softAsserter);
Channel channel = session.openChannel("exec");
((ChannelExec)channel).setCommand(command);
channel.setInputStream(null);
((ChannelExec)channel).setErrStream(System.err);
InputStream inputStream = channel.getInputStream();
channel.connect();
I wish to know what is the best way, or the right commands in order to edit a file (for example Test.txt) inside of a zip file on a SecureCRT server.

The contends inside the zip file can be modified in significant number of ways.
I have mentioned some ways which actually might work for you. In order to do that
We should securely transfer the source file/compiled file from local machine to server. The below link would help to transfer the file securely .
https://www.vandyke.com/int/drag_n_drop.html
As a first step , We should develop a snippet which is capable of modifying the contends of the zip file, Then we should copy the file to the server . Then we execute the command to run the file so that the contends inside the zip gets modified.
The below approach has been mentioned only to modify the zip contends.
Approach 1: Using a Simple Java snippet to achieve
We can write a simple java snippet which can open the zip file and edit , Keep the file in the machine and then execute the class file by just running "java filename" which would actually modify contends in the zip file.
Link which would help :
Modifying a text file in a ZIP archive in Java
import java.io.*;
import java.nio.file.*;
class RemoteEditFileContends {
/**
* Edits the text file in zip.
*
* #param zipFilePathInstance
* the zip file path instance
* #throws IOException
* Signals that an I/O exception has occurred.
*/
public static void editTextFileInZip(String zipFilePathInstance) throws IOException {
Path pathInstance = Paths.get(zipFilePathInstance);
try (FileSystem fileSystemIns = FileSystems.newFileSystem(pathInstance, null)) {
Path pathSourceInstance = fileSystemIns.getPath("/abc.txt");
Path tempCopyIns = generateTempFile(fileSystemIns);
Files.move(pathSourceInstance, tempCopyIns);
streamCopy(tempCopyIns, pathSourceInstance);
Files.delete(tempCopyIns);
}
}
/**
* Generate temp file.
*
* #param fileSystemIns
* the file system ins
* #return the path
* #throws IOException
* Signals that an I/O exception has occurred.
*/
public static Path generateTempFile(FileSystem fileSystemIns) throws IOException {
Path tempCopyIns = fileSystemIns.getPath("/___abc___.txt");
if (Files.exists(tempCopyIns)) {
throw new IOException("temp file exists, generate another name");
}
return tempCopyIns;
}
/**
* Stream copy.
*
* #param sourecInstance
* the src
* #param destinationInstance
* the dst
* #throws IOException
* Signals that an I/O exception has occurred.
*/
public static void streamCopy(Path sourecInstance, Path destinationInstance) throws IOException {
try (
BufferedReader bufferInstance = new BufferedReader(new InputStreamReader(Files.newInputStream(sourecInstance)));
BufferedWriter writerInstance = new BufferedWriter(
new OutputStreamWriter(Files.newOutputStream(destinationInstance)))) {
String currentLine = null;
while ((currentLine = bufferInstance.readLine()) != null) {
currentLine = currentLine.replace("key1=value1", "key1=value2");
writerInstance.write(currentLine);
writerInstance.newLine();
}
}
}
public static void main(String[] args) throws IOException {
editTextFileInZip("test.zip");
}
}
Approach 2: Using python to modify the zip files
How to update one file inside zip file using python
Approach 3 : Writing a shell script to modify the contends of zip file directly, So that we can copy the shell script to the server and then execute directly the shell script.
https://superuser.com/questions/647674/is-there-a-way-to-edit-files-inside-of-a-zip-file-without-explicitly-extracting
The below snippet would help you to connect and execute using the library.
import java.io.IOException;
import java.io.InputStream;
import java.util.logging.Level;
import java.util.logging.Logger;
import com.jcraft.jsch.Channel;
import com.jcraft.jsch.ChannelExec;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.JSchException;
import com.jcraft.jsch.Session;
public class ConnetionManager {
private static final Logger _logger = Logger.getLogger(ConnetionManager.class.getName());
private JSch jschSSHChannel;
private String strUserName;
private String strConnectionIP;
private int intConnectionPort;
private String strPassword;
private Session sesConnection;
private int intTimeOut;
private void doCommonConstructorActions(String userNameInstance, String tokenpassword, String connetionServerIo,
String hostFileName) {
jschSSHChannel = new JSch();
try {
jschSSHChannel.setKnownHosts(hostFileName);
} catch (JSchException exceptionInstance) {
_logError(exceptionInstance.getMessage());
}
strUserName = userNameInstance;
strPassword = tokenpassword;
strConnectionIP = connetionServerIo;
}
public ConnetionManager(String userName, String password, String connectionIP, String knownHostsFileName) {
doCommonConstructorActions(userName, password, connectionIP, knownHostsFileName);
intConnectionPort = 22;
intTimeOut = 60000;
}
public ConnetionManager(String userName, String password, String connectionIP, String knownHostsFileName,
int connectionPort) {
doCommonConstructorActions(userName, password, connectionIP, knownHostsFileName);
intConnectionPort = connectionPort;
intTimeOut = 60000;
}
public ConnetionManager(String userName, String password, String connectionIP, String knownHostsFileName,
int connectionPort, int timeOutMilliseconds) {
doCommonConstructorActions(userName, password, connectionIP, knownHostsFileName);
intConnectionPort = connectionPort;
intTimeOut = timeOutMilliseconds;
}
public String connect() {
String errorMessage = null;
try {
sesConnection = jschSSHChannel.getSession(strUserName, strConnectionIP, intConnectionPort);
sesConnection.setPassword(strPassword);
sesConnection.connect(intTimeOut);
} catch (JSchException exceptionInstance) {
errorMessage = exceptionInstance.getMessage();
}
return errorMessage;
}
private String _logError(String errorMessage) {
if (errorMessage != null) {
_logger.log(Level.SEVERE, "{0}:{1} - {2}", new Object[] { strConnectionIP, intConnectionPort, errorMessage });
}
return errorMessage;
}
private String _logWarnings(String warnMessage) {
if (warnMessage != null) {
_logger.log(Level.WARNING, "{0}:{1} - {2}", new Object[] { strConnectionIP, intConnectionPort, warnMessage });
}
return warnMessage;
}
public String sendCommand(String executionCommand) {
StringBuilder outputBuffer = new StringBuilder();
try {
Channel channelInstance = sesConnection.openChannel("exec");
((ChannelExec) channelInstance).setCommand(executionCommand);
InputStream commandOutputStream = channelInstance.getInputStream();
channelInstance.connect();
int readByte = commandOutputStream.read();
while (readByte != 0xffffffff) {
outputBuffer.append((char) readByte);
readByte = commandOutputStream.read();
}
channelInstance.disconnect();
} catch (IOException ioExceptionInstance) {
_logWarnings(ioExceptionInstance.getMessage());
return null;
} catch (JSchException schExceptionInstance) {
_logWarnings(schExceptionInstance.getMessage());
return null;
}
return outputBuffer.toString();
}
public void close() {
sesConnection.disconnect();
}
}

Java and SSH looking to execute multiple commands from a file, across a list of hosts in another file and write the output to a file

“my-thoughts” Provided the code below. but it’s not working 100%. 1) if I have more than one command, the program won’t run. 2) one command in the command file, the script won’t execute properly. for example with “ls” the log file will have the following "Last login: Tue Jun 9 14:30:11 2015 from localhost lsmyhost:~ myaccount$ ls "
JSSH class:
public class JSSH {
private static final String user = "UID";
private static final String password = "pass";
public static void main(String args[]) throws JSchException,
InterruptedException, IOException {
JSSH jssh = new JSSH();
JSch jsch = new JSch();
for (String host : jssh.listOfhost()) {
Session session = jsch.getSession(user, host, 22);
session.setPassword(password);
session.setConfig(getProperties());
session.connect(10 * 1000);
Channel channel = session.openChannel("shell");
for(String command : jssh.listOfCommand()) {
channel.setInputStream(new ByteArrayInputStream(command.getBytes()));
channel.setOutputStream(new FileOutputStream(new File(OUTPUT_FILE)));
channel.connect(15 * 1000);
TimeUnit.SECONDS.sleep(3);
}
channel.disconnect();
session.disconnect();
}
}
private static Properties getProperties() {
Properties properties = new Properties();
properties.put("StrictHostKeyChecking", "no");
return properties;
}
private List<String> listOfCommand() throws IOException {
return new LineBuilder("command_file.txt").build();
}
private List<String> listOfhost() throws IOException {
return new LineBuilder("host_file.txt").build();
}
}
LineBuilder Class:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public class LineBuilder {
private String fileName;
public LineBuilder(String fileName) {
this.fileName = fileName;
}
public List<String> build() throws IOException {
BufferedReader reader = new BufferedReader(new FileReader(new File(fileName)));
List<String> lines = new ArrayList<String>();
String line = null;
try {
while((line = reader.readLine()) != null) {
lines.add(line);
}
} catch(IOException e) {
throw e;
} finally {
reader.close();
}
return lines;
}
}

Every time you call new FileOutputStream, you are overwriting the file. So you definitely should not be calling it for every command, as that would result in a file which contains only the output of the last command executed.
A FileOutputStream should almost always be wrapped in a BufferedOutputStream, as it often improves performance. This has been true for a very long time, since before Java even existed.
I'm guessing you want the output from all commands on all hosts to be in that log file. If that's the case, you want to create the OutputStream once, outside of any loops.
You probably should not create a new connection for every command. Move these two lines outside of your inner for-loop, so they're right after the creation of the Channel:
channel.setOutputStream(outputStream);
channel.connect(15 * 1000);
Note that the documentation for setOutputStream states that you should be calling it before you call Channel.connect(). Also, since we're using a single OutputStream for all commands from all hosts, you want to pass true as the second argument, so JSch won't close that OutputStream.
In fact, the documentation for setInputStream says the same thing: It has to be called before calling connect().
So how does one manage that? You'll need to create a background Thread that "feeds" lines to the InputStream through a pipe. This is accomplished with PipedInputStream and PipedOutputStream, which allow one thread to read from a stream fed by another thread.
So, the revised version might look like this:
JSSH jssh = new JSSH();
final String[] commands = jssh.listOfCommand();
// Local class for feeding commands to a pipe in a background thread.
class CommandSender
implements Runnable {
private final OutputStream target;
IOException exception;
CommandSender(OutputStream target) {
this.target = Objects.requireNonNull(target);
}
#Override
public void run() {
try {
for (String command : commands) {
target.write(command.getBytes());
target.write(10); // newline
TimeUnit.SECONDS.sleep(3);
}
} catch (IOException e) {
exception = e;
} catch (InterruptedException e) {
System.out.println("Interrupted, exiting prematurely.");
}
}
}
try (OutputStream log = new BufferedOutputStream(
new FileOutputStream(OUTPUT_FILE))) {
JSch jsch = new JSch();
for (String host : jssh.listOfhost()) {
Session session = jsch.getSession(user, host, 22);
session.setPassword(password);
session.setConfig(getProperties());
session.connect(10 * 1000);
Channel channel = session.openChannel("shell");
channel.setOutputStream(log, true);
try (PipedInputStream commandSource = new PipedInputStream();
OutputStream commandSink = new PipedOutputStream(commandSource)) {
CommandSender sender = new CommandSender(commandSink);
Thread sendThread = new Thread(sender);
sendThread.start();
channel.setInputStream(commandSource);
channel.connect(15 * 1000);
sendThread.join();
if (sender.exception != null) {
throw sender.exception;
}
}
channel.disconnect();
session.disconnect();
}
}
A word of caution: Calling String.getBytes() converts the String's characters to bytes using the platform's default charset. In Windows, this is usually UTF16-LE (two bytes per character), so if you are making ssh connections from Windows to Unix or Linux machines, which typically expect characters encoded in UTF-8, you may get a lot of failures. The easy solution is to specify an explicit charset, assuming you know the charset used by the target machines:
command.getBytes(StandardCharsets.UTF_8)

If you are using the shell channel and sending raw command data (i.e., a ByteArrayInputStream in your case) you need to make sure to include a newline character \n after each command. Otherwise, the command will not execute, but just sit and wait for more input.
See my response to this question for a similar example.

Run an interactive Dos program from java

I would like to run a Dos program from a web server. The Dos program has to be run interactively as the user interface is via a series of questions and answers. The answer to one question will determine the next question. I will have to use ajax on the web server, but I think I can do that.
I found one java program on Stackoverflow which seems to do something similar to what I want. However when I compile the program I get an error ie.
javac PipeRedirection.java
PipeRedirection.java:43: package InputProcess does not exist
InputProcess.Gobbler outGobbler = new InputProcess.Gobbler(p.getInputStream());
The stack overflow question url was
How can I write large output to Process getOutputStream?
The Java file was
/*
####### PipeRedirection.java
*/
import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.PrintWriter;
import java.util.ArrayList;
import java.util.List;
public class PipeRedirection {
public static void main(String[] args) throws FileNotFoundException {
if(args.length < 2) {
System.err.println("Need at least two arguments");
System.exit(1);
}
try {
String input = null;
for(int i = 0; i < args.length; i++) {
String[] commandList = args[i].split(" ");
ProcessBuilder pb = new ProcessBuilder(commandList);
//pb.redirectErrorStream(true);
Process p = pb.start();
if(input != null) {
PrintWriter writer = new PrintWriter(new OutputStreamWriter(new BufferedOutputStream(p.getOutputStream())), true);
writer.println(input);
writer.flush();
writer.close();
}
InputProcess.Gobbler outGobbler = new InputProcess.Gobbler(p.getInputStream());
InputProcess.Gobbler errGobbler = new InputProcess.Gobbler(p.getErrorStream());
Thread outThread = new Thread(outGobbler);
Thread errThread = new Thread(errGobbler);
outThread.start();
errThread.start();
outThread.join();
errThread.join();
int exitVal = p.waitFor();
System.out.println("\n****************************");
System.out.println("Command: " + args[i]);
System.out.println("Exit Value = " + exitVal);
List<String> output = outGobbler.getOuput();
input = "";
for(String o: output) {
input += o;
}
}
System.out.println("Final Output:");
System.out.println(input);
} catch (IOException ioe) {
// TODO Auto-generated catch block
System.err.println(ioe.getLocalizedMessage());
ioe.printStackTrace();
} catch (InterruptedException ie) {
// TODO Auto-generated catch block
System.err.println(ie.getLocalizedMessage());
ie.printStackTrace();
}
}
public static class Gobbler implements Runnable {
private BufferedReader reader;
private List<String> output;
public Gobbler(InputStream inputStream) {
this.reader = new BufferedReader(new InputStreamReader(inputStream));
}
public void run() {
String line;
this.output = new ArrayList<String>();
try {
while((line = this.reader.readLine()) != null) {
this.output.add(line + "\n");
}
this.reader.close();
}
catch (IOException e) {
// TODO
System.err.println("ERROR: " + e.getMessage());
}
}
public List<String> getOuput() {
return this.output;
}
}
}
Does anyone know why I get the compile error? Can I substitute some other code for InputProcess?
Thanks for any help
Peter

I think it's pretty obvious that you're missing parts to this code. A package named InputProcess which has a class called Gobbler was not included in the OP's post. Probably because it was not relevant to their question.
The error message essentially says that it can not find this package/code that it is looking for.
What this class does exactly, only the OP can tell you. At its most basic, though, it appears to read from an InputStream and convert it to a List<String>. I would read up on Java IO and try to replicate similar functionality.
Edit:
Looks like the Gobbler class is indeed included in the example above. Remove the InputProcess package name from your code (or put the Gobbler class in an InputProcess package) and you should be good to go.

Writing Console Output to File in Java

So I was wondering if it was possible to write all the console output to a separate file outside of Java? I know about the Printwriter and Filewriter method. However, in my experience those would work if I was using them all within one method, but I don't think I can do that with the code I have right now. Below is what I have...
Java Code
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
import org.xml.sax.*;
import org.xml.sax.helpers.DefaultHandler;
import javax.xml.parsers.SAXParser;
import javax.xml.parsers.SAXParserFactory;
public class XMLTagParser extends DefaultHandler
{
private int i;
public XMLTagParser()
{
traverse(new File("C:/Documents and Settings/user/workspace/Intern Project/Proposals/Converted Proposals/Extracted Items"));
}
private static final class SaxHandler extends DefaultHandler
{
private StringBuffer buffer;
private String heading;
private boolean inHeading;
public void startElement(String uri, String localName, String qName, Attributes attrs)
{
if ("w:pStyle".equals(qName))
{
String val = attrs.getValue("w:val");
if (val.contains("Heading"))
{
if (isHeading(val))
{
System.out.println(val);
inHeading = true;
}
}
}
if("w:t".equals(qName))
{
if (inHeading == true)
{
buffer = new StringBuffer();
}
}
}
public void characters(char buff[], int offset, int length) throws SAXException
{
String s = new String(buff, offset, length);
if(buffer != null)
{
buffer.append(s);
heading = heading += s;
}
}
public void endElement(String uri, String localName, String qName)
{
buffer = null;
//if the qName is "w:p" and it is in the heading, print out the heading and then reset
if ("w:p".equals(qName) && inHeading == true)
{
System.out.println(heading);
heading = "";
inHeading = false;
}
}
// method to verify whether element is an actual heading
private static boolean isHeading(String heading)
{
String headingNumber = heading.substring(7,8);
String headingName = heading.substring(0,7);
if (headingName.equals("Heading"))
{
if (headingNumber.equals("1")
|| headingNumber.equals("2")
|| headingNumber.equals("3")
|| headingNumber.equals("4")
|| headingNumber.equals("5")
|| headingNumber.equals("6"))
{
return true;
}
}
return false;
}
}
/*private void writeFile(File file)
{
try
{
PrintWriter out = new PrintWriter(new FileWriter(file + "/" + i++));
out.close();
}
catch (IOException e)
{
e.printStackTrace(System.out);
}
}*/
private void traverse(File directory)
{
//Get all files in directory
File[] files = directory.listFiles();
for (File file : files)
{
if (file.getName().equals("document.xml"))
{
try
{
// creates and returns new instance of SAX-implementation:
SAXParserFactory factory = SAXParserFactory.newInstance();
// create SAX-parser...
SAXParser parser = factory.newSAXParser();
// prints out the current working proposal, traversing up the directory structure
System.out.println(file.getParentFile().getParentFile().getName());
// .. define our handler:
SaxHandler handler = new SaxHandler();
// and parse:
parser.parse(file.getAbsolutePath(), handler);
try
{
// instantiates new printwriter which writes out to a file
PrintWriter out = new PrintWriter(new FileWriter(file.getParentFile().getParentFile() + "/" + i++ + ".txt"));
out.close();
}
catch (IOException e)
{
e.printStackTrace(System.out);
}
}
catch (Exception ex)
{
ex.printStackTrace(System.out);
}
}
else if (file.isDirectory())
{
//It's a directory so (recursively) traverse it
traverse(file);
}
}
}
}
So I've instantiated the printwriter in there, but obviously it's no good if I have nothing to write to it. So I'm not really sure how I can get what's printing out the console to be written to that file. Any ideas? Thanks in advance.

If you really want to you can redirect System.out to any PrintStream like this:
PrintStream stream = new PrintStream("filename.txt");
System.setOut(stream);

If you get into rolling your own file logger you'll spend more time dealing with io issues, rolling files, file sizes, ect. You should use log4j instead! It will handle things like this and make your logging more flexible. It's pretty much the standard for java logging.

The System.out is basically an OutputStream; which by default points to the console. Instead, you could just create a new FileOutputStream instance pointing to the file of your choice, and identify this stream by setting it through System.setOut. That will do it for you, throughout the life-cycle of the program/application. Check this link for a complete code.

Instead of using System.out, you could use a FileWriter, write to it and flush it. It is unclear, why you increment i in your code. I guess you want to write everything to just one file.
Also, it looks like you never write to the Writer that you initialize.

You could keep using System.out, and redirect it using ">" to the file when you invoke the application. You can still retain stderr for direct console output.
Or do you mean something else when you write "outside java"?