I have an executor service that runs new threads based upon reflection. I have one method that when run does not exit the thread and will hang program execution. I am unsure why this is, can anyone point me to what I'm missing?
Also, if any of the concurrency experts out there notice any problems I may run into, please let me know, I am rather green in concurrency....
Notes:
The connectToFTP method will be refactored to return ChannelSftp in the future.
downloadFromFTP returns false at the end as the method is not completed. I feel that this is the method that is causing the thread to hang. I just don't know why.
The goal of the method is to list each file within an SFTP directory.
from NetworkingShopCa.
#Override
public Object connectToFTP(String username, String password, String host, String port, FtpTypes ftpTypes) {
switch(ftpTypes){
case FTP:
LOGGER.error("Plain FTP is not implemented yet (if ever)");
break;
case FTPS:
FTPSClient client = new FTPSClient();
client.setTrustManager(TrustManagerUtils.getAcceptAllTrustManager());
try {
client.connect(host);
client.enterLocalPassiveMode();
client.login(username, password);
} catch (IOException e) {
LOGGER.error(e.toString());
}
return client;
case SFTP:
JSch jsch = new JSch();
Session session = null;
try {
session = jsch.getSession(username, host);
session.setConfig("StrictHostKeyChecking", "no");
session.setPassword(password);
session.connect();
Channel channel = session.openChannel("sftp");
channel.connect();
ChannelSftp sftpChannel = (ChannelSftp) channel;
return sftpChannel;
} catch (JSchException e) {
// TODO Auto-generated catch block
LOGGER.error(e.toString());
}
break;
default:
LOGGER.error("Invalid FtpType");
break;
}
return false;
}
#Override
public boolean downloadFromFTP(String directory, String filename, boolean downloadAll,Object activeConnection) {
if(activeConnection instanceof ChannelSftp){
ChannelSftp sftpChannel = (ChannelSftp) activeConnection;
try {
sftpChannel.cd(directory);
//List our files within the directory
Vector vv = sftpChannel.ls(directory);
if (vv != null) {
for (int ii = 0; ii < vv.size(); ii++) {
Object obj = vv.elementAt(ii);
if (obj instanceof ChannelSftp.LsEntry) {
LOGGER.debug("[" + ((LsEntry) obj).getFilename() + "]");
}
}
}
} catch (SftpException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
return false;
}
From main
runtimes.put(config.getInt("ESE_PRIORITY"),"RUN_ESE");
ExecutorService threadPool = Executors.newFixedThreadPool(totalRunnables);
LOGGER.info("Executing runtimes in order of priority.");
for(final int priority : runtimes.keySet()){
if(!threadPool.isShutdown() && !threadPool.isTerminated()){
//run the method denoted by the property
final java.lang.reflect.Method method = m.getClass().getMethod(runtimes.get(priority));
Future<?> f = threadPool.submit(new Runnable() {
#Override
public void run() {
try {
method.invoke(m);
} catch (IllegalAccessException | InvocationTargetException e) {
e.printStackTrace();
}
}
});
f.get();
}
}
public void RUN_ESE(){
LOGGER.info("Running ESE");
Networking networking = new NetworkingShopCa();
networking.downloadFromFTP("/toclient/order/processed", "", true, networking.connectToFTP("user", "password", "host", "", FtpTypes.SFTP));
}
--Edit--
Following a debugger downloadFromFTP executes fully, returns out of the method and goes to close the runnable when it hangs:
Line 1153 of ThreadPoolExecutor.java shows:
afterExecute(task, thrown);
Any ideas? For what its worth my build environment is:
Ubuntu 13.04 64 bit
OpenJDK 7 (ubuntu default) with attached sources.
Eclipse kepler
Related
I'm using JSch for file upload over SFTP. In its current state each thread opens and closes connection when needed.
If it possible to use connection pooling with JSch in order to avoid overhead caused by large number of connection opening and closing?
Here is a example of function called from inside of thread
public static void file_upload(String filename) throws IOException {
JSch jsch = new JSch();
Session session = null;
try {
session = jsch.getSession("user", "server_name", 22);
session.setConfig("StrictHostKeyChecking", "no");
session.setPassword("super_secre_password");
session.connect();
Channel channel = session.openChannel("sftp");
channel.connect();
ChannelSftp sftpChannel = (ChannelSftp) channel;
FileInputStream inputSrr = new FileInputStream(filename);
try {
sftpChannel.put(inputSrr, "/var/temp/"+filename);
} catch (SftpException e) {
e.printStackTrace();
} finally {
if (inputSrr != null) {
inputSrr.close();
}
}
sftpChannel.exit();
session.disconnect();
} catch (JSchException e) {
e.printStackTrace();
} catch (SftpException e) {
e.printStackTrace();
}
}
For that I would prefer commons-pool. ;)
Here's an implementation of Ssh Connection pool
http://www.javacodegeeks.com/2013/02/pool-of-ssh-connections-using-apache-keyedobjectpool.html
you can use grep4j to use this pool
https://code.google.com/p/grep4j/source/browse/trunk/src/main/java/org/grep4j/core/command/linux/SessionFactory.java?r=354
Also make sure you can access the server from the execution machine. For instance if the target server is not in your reach. It'll throw connection timeout.
I wold like to share with you our implementation, We have used Session Manager of jsch-extension library
First of all you need to implement pool object factory that is responsible for lifecycle of pooled objects:
public class ChannelSftpConnectionsFactory extends BasePooledObjectFactory<ChannelSftp> {
private SessionManager sessionManager;
public ChannelSftpConnectionsFactory(final SessionManager sessionManager) {
this.sessionManager = sessionManager;
}
//Create and open channel
#Override
public ChannelSftp create() throws JSchException {
ChannelSftp channelSftp = (ChannelSftp) sessionManager.getSession().openChannel("sftp");
channelSftp.connect();
return channelSftp;
}
//wrapping
#Override
public PooledObject<ChannelSftp> wrap(final ChannelSftp channelSftp) {
return new DefaultPooledObject<>(channelSftp);
}
#Override
//disconnect channel on destroy
public void destroyObject(final PooledObject<ChannelSftp> pooledObject) {
ChannelSftp sftp = pooledObject.getObject();
disconnectChannel(sftp);
}
void disconnectChannel(final ChannelSftp sftp) {
if (sftp.isConnected()) {
sftp.disconnect();
}
}
#Override
//reset channel current folder to home if someone was walking on another folders
public void passivateObject(final PooledObject<ChannelSftp> p) {
ChannelSftp sftp = p.getObject();
try {
sftp.cd(sftp.getHome());
} catch (SftpException ex) {
log.error("Could not reset channel to home folder, closing it");
disconnectChannel(sftp);
}
}
#Override
//validate object before it is borrowed from pool. If false object will be removed from pool
public boolean validateObject(final PooledObject<ChannelSftp> p) {
ChannelSftp sftp = p.getObject();
return sftp.isConnected() && !sftp.isClosed();
}
}
Now you could create pool using configured factory:
ObjectPool<ChannelSftp> createPool(final SessionManager sessionManager, final GenericObjectPoolConfig<ChannelSftp> poolConfig) {
return PoolUtils.synchronizedPool(new GenericObjectPool<>(buildFactory(sessionManager), poolConfig));
}
PooledObjectFactory<ChannelSftp> buildFactory(final SessionManager sessionManager) {
return PoolUtils.synchronizedPooledFactory(new ChannelSftpConnectionsFactory(sessionManager));
}
This java doc would help you to configure pool properly : https://commons.apache.org/proper/commons-pool/api-2.6.0/org/apache/commons/pool2/impl/BaseGenericObjectPool.html
Do not forget about correct borrowing and returning of object into pool: https://commons.apache.org/proper/commons-pool/api-2.6.0/org/apache/commons/pool2/ObjectPool.html
Object obj = null;
try {
obj = pool.borrowObject();
try {
//...use the object...
} catch(Exception e) {
// invalidate the object
pool.invalidateObject(obj);
// do not return the object to the pool twice
obj = null;
} finally {
// make sure the object is returned to the pool
if(null != obj) {
pool.returnObject(obj);
}
}
} catch(Exception e) {
// failed to borrow an object
}
I am trying to makeAConnection with the server using this class. This class gets the list of parameters needed to perform operation on Images in HashMaps. Then in doInBackground , I perform the operations required on Image one by one. The code for one of the classes which is OVFImage Deployer is also pasted below
public class ImageDeployer extends SwingWorker<Boolean,String> {
public ImageDeployer(){
}
public ImageDeployer(HashMap<String, String> volIDMap, HashMap<String, String> osMap) {
// TODO Auto-generated constructor stub
this.volIDMap = volIDMap;
this.osMap = osMap;
System.out.println(volIDMap);
System.out.println(osMap);
makeAConnection();
try {
doInBackground();
System.out.println("Do In Background");
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void makeAConnection(){
inputFile = RESTEngine.getFilePath();
Properties defaultProps = new Properties();
try {
fin = new FileInputStream(inputFile);
defaultProps.load(fin);
fin.close();
}
catch(FileNotFoundException e1){
System.out.println("The properties file supposed to contain Authorization parameters was not found.");
e1.printStackTrace();
System.exit(-1);
}
catch(IOException e1){
System.out.println("An exception occured while trying to open the properties file");
e1.printStackTrace();
System.exit(-1);
}
// assign variables from Input file with default value as null
user = defaultProps.getProperty("UserID", null);
host = defaultProps.getProperty("PowerVC_IP_ADDRESS", null);
password = defaultProps.getProperty("UserPass" ,null );
jsch = new JSch();
try {
session = jsch.getSession(user, host, 22);
session.setPassword(password);
java.util.Properties config = new java.util.Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.connect();
channel=session.openChannel("exec");
channel.setInputStream(null);
try {
in = channel.getInputStream();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println("Connection Successful");
} catch (JSchException e) {
// TODO Auto-generated catch block
System.out.println("Unable to connect");
e.printStackTrace();
}
}
#Override
protected Boolean doInBackground() throws Exception {
ImageDeployer imageDeployer = new ImageDeployer();
imageDeployer.makeAConnection();
for(String imageName : volIDMap.keySet()){
String volID = volIDMap.get(imageName);
String oS = osMap.get(imageName);
if (oS.equalsIgnoreCase("aix")){
imageDeployer = new OVFImageDeployer(volID, oS, imageName);
}
// Other Cases depending upon the OS Type
}
return null;
}
}
The code for OVFImage Deployer
public class OVFImageDeployer extends PowerVCImageDeployer {
public OVFImageDeployer(String VolID,String oS,String imageName){
String command="/usr/bin/powervc-devtools/powervc-devcli glance image-create json "+imageName+" "+oS+" "+VolID;
try {
((ChannelExec)channel).setCommand(command);
channel.connect();
} catch (JSchException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Now when I run the code I get a NullPointerException on line ((ChannelExec)channel).setCommand(command).
I know if I put makeAConnection after the try block in OVFImageDeployer the code would work, but then I don't want to make a connection again and again . I want a connection to be initialized just once and all operations to be performed using that connection only.
You should remove the call to doInBackground from inside the constructor of ImageDeployer:
public ImageDeployer(HashMap<String, String> volIDMap, HashMap<String, String> osMap) {
....
makeAConnection();
//doInBackground();
...
}
This will initialize the channel when you create an instance of ImageDeployer. And you can add channel to the list of constructor arguments of OVFImageDeployer:
public OVFImageDeployer(String VolID,String oS,String imageName, Channel channel){
this.channel = channel;
...
}
This will create an instance of OVFImageDeployer with the channel that is present in the ImageDeployer instance. You need to remove these two statements from inside the doInBackground method and pass channel along with the other parameters while constructing an instance of OVFImageDeployer:
#Override
protected Boolean doInBackground() throws Exception {
//ImageDeployer imageDeployer = new ImageDeployer();
//imageDeployer.makeAConnection();
...
ImageDeployer imageDeployer = new OVFImageDeployer(volID, oS, imageName, channel);
...
}
Now the client code can create an instance of ImageDeployer and can execute doInBackground on it:
ImageDeployer imageDeployer = new ImageDeployer();
imageDeployer.doInBackground();
With this, every time you create an instance of OVFImageDeployer inside the doInBackground method, you can use the same channel which was created by the makeAConnection method while constructing the ImageDeployer instance.
I want to check if server application is available. After server is started I want to stop checking until the server changes status. How to do that with my code:
private static final String SERVER_ADDRESS = "192.144.10.10";
private static final int TCP_SERVER_PORT = 8890;
private static boolean connected = false;
static Socket s;
public static void main(String[] args) {
Timer timer = new Timer();
timer.schedule(task, 01, 5001); }
static TimerTask task = new TimerTask() {
#Override
public void run() {
// TODO Auto-generated method stub
if (connected == false)
{
System.out.println(hostAvailabilityCheck());
}
}
};
public static boolean hostAvailabilityCheck()
{
boolean available = true;
try {
if (connected == false)
{ (s = new Socket(SERVER_ADDRESS, TCP_SERVER_PORT)).close();
}
}
catch (UnknownHostException e)
{ // unknown host
available = false;
s = null;
}
catch (IOException e) { // io exception, service probably not running
available = false;
s = null;
}
catch (NullPointerException e) {
available = false;
s=null;
}
return available;
}
Is there any better way to solve this?
The check method can be rewritten as follows (Java 7 and later):
public static boolean hostAvailabilityCheck() {
try (Socket s = new Socket(SERVER_ADDRESS, TCP_SERVER_PORT)) {
return true;
} catch (IOException ex) {
/* ignore */
}
return false;
}
In addition to simplifying the exception handling, this eliminates a pesky Socket leak. (If you are concerned with the time taken to do this check, then set a connection timeout before attempting to connect: see Setting a timeout for socket operations)
But the problems with this approach are many-fold:
It only tests that something is listening for connections. If your service is behind a proxy ... or is managed by something like the inetd service ... then the accepted connections don't mean your service is actually working.
This is going to cause your service to "see" connections that close down without sending a request. So you'd better code your service to deal with this "gracefully".
Doing this repeatedly adds to network and server load.
If you set a short timeout because you don't want the test to "freeze", then you risk setting it too short and judging the host to be down when it isn't.
After server is started I want to stop checking until the server changes status
That is next to impossible. The reason is that you won't be able to tell whether the server has "changed status" without checking. Or at least, you won't be able to do this without implementing an elaborate status notification service where the server calls the client to tell it is changing status. (And if "change status" includes "die" or "lost network connection", then you won't be able to make that notification reliable ... if at all.)
public static boolean hostAvailabilityCheck() {
try (Socket s = new Socket(SERVER_ADDRESS, TCP_SERVER_PORT)) {
return true;
} catch (IOException ex) {
/* ignore */
}
return false;
}
working, but the problem is that when you turn on the phone throught WI-FI it comes to a "screeching halt" and no action. for thought...=)
next code will be to work through WI-FI ... if you increase the connection time -
public static boolean isOnline() {
boolean b = true;
try{
InetSocketAddress sa = new InetSocketAddress("SERVER_IP_ADDRESS", PORT);
Socket ss = new Socket();
ss.connect(sa, 1); --> change from 1 to 500 (for example)
ss.close();
}catch(Exception e) {
b = false;
}
return b;
}
First check if server is running and the server accepts the connection.
public static boolean hostAvailabilityCheck()
{
s = new Socket(SERVER_ADDRESS, TCP_SERVER_PORT);
boolean available = true;
try {
if (s.isConnected())
{ s.close();
}
}
catch (UnknownHostException e)
{ // unknown host
available = false;
s = null;
}
catch (IOException e) { // io exception, service probably not running
available = false;
s = null;
}
catch (NullPointerException e) {
available = false;
s=null;
}
return available;
}
I used this method for my ServerUtil.
public static boolean isOnline() {
boolean b = true;
try{
InetSocketAddress sa = new InetSocketAddress("SERVER_IP_ADDRESS", PORT);
Socket ss = new Socket();
ss.connect(sa, 1);
ss.close();
}catch(Exception e) {
b = false;
}
return b;
}
What is the correct procedure to follow when an exception is thrown on an FTP Client in Java i.e. does the FTP session stay active or does it automatically 'quit' when an exception is thrown?
So I have this:
public boolean testHost(Host host, String path) {
boolean success = false;
try {
FTPClient ftp = new FTPClient();
ftp.setRemoteHost(host.getIpaddress());
ftp.connect();
ftp.login(host.getUsername(), host.getPassword());
success = ftp.connected();
if (success && path != null){
ftp.chdir(path);
}
ftp.quit();
} catch (UnknownHostException e) {
LOG.info("Host IPAddress cannot be reached on " + host.getIpaddress());
success = false;
} catch (IOException e) {
e.printStackTrace();
success = false;
} catch (FTPException e) {
success = false;
}
return success;
}
The quit command doesnt get hit when any of the exceptions get called - is this a problem? Could there potentially be 100's of active connections open to the FTP Client if this method keeps getting hit? Or am I worrying about nothing?
Move your ftp.quit() statement so it is just above the return statement
Like this:
public boolean testHost(Host host, String path) {
boolean success = false;
try {
FTPClient ftp = new FTPClient();
ftp.setRemoteHost(host.getIpaddress());
ftp.connect();
ftp.login(host.getUsername(), host.getPassword());
success = ftp.connected();
if (success && path != null){
ftp.chdir(path);
}
} catch (UnknownHostException e) {
LOG.info("Host IPAddress cannot be reached on " + host.getIpaddress());
success = false;
} catch (IOException e) {
e.printStackTrace();
success = false;
} catch (FTPException e) {
success = false;
}
ftp.quit();
return success;
}
Since none of your catches terminate the method, execution will continue to the ftp.quit() statement and finally return with the success result.
Optionally, you can use the finally clause at the end of the try and put the ftp.quit() statement in that.
AFAIK the choice is preferential.
I need to execute(run) a shell script which resides in the server(Solaris) from java. Please help me how to execute a file from java.? I have tried with sendCommand() of TelnetToClient. So please help me in running a file from my GUI.
The program goes like this.
TelnetToPort tele = new TelnetToPort("opmer3");
tele.login("root","root");
String command_ = "/usr/bin/bash /opt/nrl/logs/applications/ns/lccommands.sh";
tele.runComm(command_);
If you are looking for optimized solution for executing any scripts for your java class, then you can use Jsch with Google Expect4j libraries.
For jsch, go to http://www.jcraft.com/jsch/
For Expect4j, go to http://code.google.com/p/expect4j/
Following is small code sample for log in and executing file fro remote java class.
private Expect4j SSH(String hostname, String username,String password, int port) throws Exception {
JSch jsch = new JSch();
Session session = jsch.getSession(username, hostname, port);
if (password != null) {
session.setPassword(password);
}
Hashtable<String,String> config = new Hashtable<String,String>();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.connect(60000);
channel = (ChannelShell) session.openChannel("shell");
Expect4j expect = new Expect4j(channel.getInputStream(), channel.getOutputStream());
channel.connect();
return expect;
}
This method will open up SSH stream to remote server which will be used by expect4j for sending commands.
private boolean executeCommands() {
boolean isSuccess = true;
Closure closure = new Closure() {
public void run(ExpectState expectState) throws Exception {
buffer.append(expectState.getBuffer());
expectState.exp_continue();
}
};
List<Match> lstPattern = new ArrayList<Match>();
String[] regEx = SSHConstants.linuxPromptRegEx;
if (regEx != null && regEx.length > 0) {
synchronized (regEx) {
for (String regexElement : regEx) {//list of regx like, :>, /> etc. it is possible command prompts of your remote machine
try {
RegExpMatch mat = new RegExpMatch(regexElement, closure);
lstPattern.add(mat);
} catch (MalformedPatternException e) {
return false;
} catch(Exception e) {
return false;
}
}
lstPattern.add(new EofMatch( new Closure() { // should cause entire page to be collected
public void run(ExpectState state) {
}
}));
lstPattern.add(new TimeoutMatch(defaultTimeOut, new Closure() {
public void run(ExpectState state) {
}
}));
}
}
try {
Expect4j expect = SSH(objConfig.getHostAddress(), objConfig.getUserName(), objConfig.getPassword(), SSHConstants.SSH_PORT);
expect.setDefaultTimeout(defaultTimeOut);
if(isSuccess) {
for(String strCmd : lstCmds)
isSuccess = isSuccess(lstPattern,strCmd);
}
boolean isFailed = checkResult(expect.expect(lstPattern));
return !isFailed;
} catch (Exception ex) {
return false;
} finally {
closeConnection();
}
}
private boolean isSuccess(List<Match> objPattern,String strCommandPattern) {
try {
boolean isFailed = checkResult(expect.expect(objPattern));
if (!isFailed) {
expect.send(strCommandPattern);
expect.send("\r");
return true;
}
return false;
} catch (MalformedPatternException ex) {
return false;
} catch (Exception ex) {
return false;
}
}
Hope this help.
Thanks.
new ProcessBuilder("ssh", "root#opmer3", command_).start();