I'm using JSch for file upload over SFTP. In its current state each thread opens and closes connection when needed.
If it possible to use connection pooling with JSch in order to avoid overhead caused by large number of connection opening and closing?
Here is a example of function called from inside of thread
public static void file_upload(String filename) throws IOException {
JSch jsch = new JSch();
Session session = null;
try {
session = jsch.getSession("user", "server_name", 22);
session.setConfig("StrictHostKeyChecking", "no");
session.setPassword("super_secre_password");
session.connect();
Channel channel = session.openChannel("sftp");
channel.connect();
ChannelSftp sftpChannel = (ChannelSftp) channel;
FileInputStream inputSrr = new FileInputStream(filename);
try {
sftpChannel.put(inputSrr, "/var/temp/"+filename);
} catch (SftpException e) {
e.printStackTrace();
} finally {
if (inputSrr != null) {
inputSrr.close();
}
}
sftpChannel.exit();
session.disconnect();
} catch (JSchException e) {
e.printStackTrace();
} catch (SftpException e) {
e.printStackTrace();
}
}
For that I would prefer commons-pool. ;)
Here's an implementation of Ssh Connection pool
http://www.javacodegeeks.com/2013/02/pool-of-ssh-connections-using-apache-keyedobjectpool.html
you can use grep4j to use this pool
https://code.google.com/p/grep4j/source/browse/trunk/src/main/java/org/grep4j/core/command/linux/SessionFactory.java?r=354
Also make sure you can access the server from the execution machine. For instance if the target server is not in your reach. It'll throw connection timeout.
I wold like to share with you our implementation, We have used Session Manager of jsch-extension library
First of all you need to implement pool object factory that is responsible for lifecycle of pooled objects:
public class ChannelSftpConnectionsFactory extends BasePooledObjectFactory<ChannelSftp> {
private SessionManager sessionManager;
public ChannelSftpConnectionsFactory(final SessionManager sessionManager) {
this.sessionManager = sessionManager;
}
//Create and open channel
#Override
public ChannelSftp create() throws JSchException {
ChannelSftp channelSftp = (ChannelSftp) sessionManager.getSession().openChannel("sftp");
channelSftp.connect();
return channelSftp;
}
//wrapping
#Override
public PooledObject<ChannelSftp> wrap(final ChannelSftp channelSftp) {
return new DefaultPooledObject<>(channelSftp);
}
#Override
//disconnect channel on destroy
public void destroyObject(final PooledObject<ChannelSftp> pooledObject) {
ChannelSftp sftp = pooledObject.getObject();
disconnectChannel(sftp);
}
void disconnectChannel(final ChannelSftp sftp) {
if (sftp.isConnected()) {
sftp.disconnect();
}
}
#Override
//reset channel current folder to home if someone was walking on another folders
public void passivateObject(final PooledObject<ChannelSftp> p) {
ChannelSftp sftp = p.getObject();
try {
sftp.cd(sftp.getHome());
} catch (SftpException ex) {
log.error("Could not reset channel to home folder, closing it");
disconnectChannel(sftp);
}
}
#Override
//validate object before it is borrowed from pool. If false object will be removed from pool
public boolean validateObject(final PooledObject<ChannelSftp> p) {
ChannelSftp sftp = p.getObject();
return sftp.isConnected() && !sftp.isClosed();
}
}
Now you could create pool using configured factory:
ObjectPool<ChannelSftp> createPool(final SessionManager sessionManager, final GenericObjectPoolConfig<ChannelSftp> poolConfig) {
return PoolUtils.synchronizedPool(new GenericObjectPool<>(buildFactory(sessionManager), poolConfig));
}
PooledObjectFactory<ChannelSftp> buildFactory(final SessionManager sessionManager) {
return PoolUtils.synchronizedPooledFactory(new ChannelSftpConnectionsFactory(sessionManager));
}
This java doc would help you to configure pool properly : https://commons.apache.org/proper/commons-pool/api-2.6.0/org/apache/commons/pool2/impl/BaseGenericObjectPool.html
Do not forget about correct borrowing and returning of object into pool: https://commons.apache.org/proper/commons-pool/api-2.6.0/org/apache/commons/pool2/ObjectPool.html
Object obj = null;
try {
obj = pool.borrowObject();
try {
//...use the object...
} catch(Exception e) {
// invalidate the object
pool.invalidateObject(obj);
// do not return the object to the pool twice
obj = null;
} finally {
// make sure the object is returned to the pool
if(null != obj) {
pool.returnObject(obj);
}
}
} catch(Exception e) {
// failed to borrow an object
}
Related
I am trying to find a bug is some RabbitMQ client code that was developed six or seven years ago. The code was modified to allow for delayed messages. It seems that connections are created to the RabbitMQ server and then never destroyed. Each exists in a separate thread so I end up with 1000's of threads. I am sure the problem is very obvious / simple - but I am having trouble seeing it. I have been looking at the exchangeDeclare method (the commented out version is from the original code which seemed to work), but I have been unable to find the default values for autoDelete and durable which are being set in the modified code. The method below in within a Spring service class. Any help, advice, guidance and pointing out huge obvious errors appreciated!
private void send(String routingKey, String message) throws Exception {
String exchange = applicationConfiguration.getAMQPExchange();
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-delayed-type", "fanout");
Map<String, Object> headers = new HashMap<String, Object>();
headers.put("x-delay", 10000); //delay in miliseconds i.e 10secs
AMQP.BasicProperties.Builder props = new AMQP.BasicProperties.Builder().headers(headers);
Connection connection = null;
Channel channel = null;
try {
connection = myConnection.getConnection();
}
catch(Exception e) {
log.error("AMQP send method Exception. Unable to get connection.");
e.printStackTrace();
return;
}
try {
if (connection != null) {
log.debug(" [CORE: AMQP] Sending message with key {} : {}",routingKey, message);
channel = connection.createChannel();
// channel.exchangeDeclare(exchange, exchangeType);
channel.exchangeDeclare(exchange, "x-delayed-message", true, false, args);
// channel.basicPublish(exchange, routingKey, null, message.getBytes());
channel.basicPublish(exchange, routingKey, props.build(), message.getBytes());
}
else {
log.error("Total AMQP melt down. This should never happen!");
}
}
catch(Exception e) {
log.error("AMQP send method Exception. Unable to get send.");
e.printStackTrace();
}
finally {
channel.close();
}
}
This is the connection class
#Service
public class PersistentConnection {
private static final Logger log = LoggerFactory.getLogger(PersistentConnection.class);
private static Connection myConnection = null;
private Boolean blocked = false;
#Autowired ApplicationConfiguration applicationConfiguration;
#PreDestroy
private void destroy() {
try {
myConnection.close();
} catch (IOException e) {
log.error("Unable to close AMQP Connection.");
e.printStackTrace();
}
}
public Connection getConnection( ) {
if (myConnection == null) {
start();
}
return myConnection;
}
private void start() {
log.debug("Building AMQP Connection");
ConnectionFactory factory = new ConnectionFactory();
String ipAddress = applicationConfiguration.getAMQPHost();
String user = applicationConfiguration.getAMQPUser();
String password = applicationConfiguration.getAMQPPassword();
String virtualHost = applicationConfiguration.getAMQPVirtualHost();
String port = applicationConfiguration.getAMQPPort();
try {
factory.setUsername(user);
factory.setPassword(password);
factory.setVirtualHost(virtualHost);
factory.setPort(Integer.parseInt(port));
factory.setHost(ipAddress);
myConnection = factory.newConnection();
}
catch (Exception e) {
log.error("Unable to initialise AMQP Connection.");
e.printStackTrace();
}
myConnection.addBlockedListener(new BlockedListener() {
public void handleBlocked(String reason) throws IOException {
// Connection is now blocked
log.warn("Message Server has blocked. It may be resource limitted.");
blocked = true;
}
public void handleUnblocked() throws IOException {
// Connection is now unblocked
log.warn("Message server is unblocked.");
blocked = false;
}
});
}
public Boolean isBlocked() {
return blocked;
}
}
I would like to stop my current transfer file with using method stopUpload() :
private ChannelSftp channelSftp
private ChannelSftp setupJsch() throws JSchException {
JSch jsch = new JSch();
jsch.setKnownHosts("/Users/john/.ssh/known_hosts");
Session jschSession = jsch.getSession(username, remoteHost);
jschSession.setPassword(password);
jschSession.connect();
return (ChannelSftp) jschSession.openChannel("sftp");
}
public void stopUpload()
{
channelSftp.disconnect();
}
public void whenUploadFileUsingJsch_thenSuccess() throws JSchException, SftpException {
ChannelSftp channelSftp = setupJsch();
channelSftp.connect();
String localFile = "src/main/resources/sample.txt";
String remoteDir = "remote_sftp_test/";
channelSftp.put(localFile, remoteDir + "jschFile.txt");
channelSftp.exit();
}
When stopUpload() run I have this error : Exception in thread "JavaFX Application Thread" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
To cleanly cancel a JSch SFTP transfer, when you need, implement the SftpProgressMonitor interface:
public class CancellableProgressMonitor implements SftpProgressMonitor {
private boolean cancelled;
public CancellableProgressMonitor() {}
public void cancel() {
this.cancelled = true;
}
public bool wasCancelled() {
return this.cancelled;
}
public void init(int op, java.lang.String src, java.lang.String dest, long max) {
this.cancelled = false;
}
public boolean count(long bytes) {
return !this.cancelled;
}
public void end() {
}
}
And pass it to ChannelSftp.put:
CancellableProgressMonitor monitor = new CancellableProgressMonitor()
channelSftp.put(localFile, remoteDir + "jschFile.txt", monitor);
Call monitor.cancel() when you need to cancel the transfer.
public void stopUpload() {
monitor.cancel();
}
If you want to cleanup the partially transferred file:
String remoteFile = remoteDir + "jschFile.txt";
try {
channelSftp.put(localFile, remoteFile, monitor);
} catch (IOException e) {
if (monitor.wasCancelled() && channelSftp.getSession().isConnected()) {
try {
channelSftp.rm(remoteFile);
} catch (SftpException e) {
if (e.id == SSH_FX_NO_SUCH_FILE) {
// can happen if the transfer was cancelled
// before the file was even created
} else {
throw e;
}
}
}
throw e;
}
I have written a code to download file from the Remote machine through SFTP and it is working fine if I provide
sftpBean.downloadFile("/dd1.csv","C:\\test\\SFTP_1");
Filename and the directory where I want to download the file which is C drive.
My requirement is let say If I have multiple folder in side my sftproot(The folder which I shared for SFTP access) folder how can i traverse inside and outside the folder/directory and download the list of (for example all .CSV)files from that location.
Thanks in advance for your help.
*****************************************************************************
**This is the code that have logic.**
package sftp.bean;
import java.util.Vector;
import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.ChannelSftp.LsEntry;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.Session;
//class using for sftp function
public class SFTPBean {
// variable for sftp channel
private JSch mJschSession = null;
private Session mSSHSession = null;
// sftp channel
private ChannelSftp mChannelSftp = null;
// connect fucntion let connect to sftp server
public boolean connect(String strHostAddress, int iPort,
String strUserName, String strPassword) {
boolean blResult = false;
try {
// creating a new jsch session
this.mJschSession = new JSch();
// set sftp server no check key when login
java.util.Properties config = new java.util.Properties();
config.put("StrictHostKeyChecking", "no");
this.mJschSession.setConfig(config);
// creating session with user, host port
this.mSSHSession = mJschSession.getSession(strUserName,
strHostAddress, iPort);
// set password
this.mSSHSession.setPassword(strPassword);
// connect to host
this.mSSHSession.connect();
// open sftp channel
this.mChannelSftp = (ChannelSftp) this.mSSHSession
.openChannel("sftp");
// connect to sftp session
this.mChannelSftp.connect();
if (this.mChannelSftp != null) {
blResult = true;
}
} catch (Exception exp) {
exp.printStackTrace();
}
return blResult;
}
// list file on sftp server
public Vector<LsEntry> listFile(String strPath) {
Vector<LsEntry> vtFile = null;
try {
vtFile = this.mChannelSftp.ls(strPath);
} catch (Exception exp) {
exp.printStackTrace();
}
return vtFile;
}
// download file
public boolean downloadFile(String strSftpFile, String strLocalFile) {
boolean blResult = false;
try {
this.mChannelSftp.get(strSftpFile, strLocalFile);
blResult = true;
} catch (Exception exp) {
exp.printStackTrace();
}
return blResult;
}
// upload file
public boolean uploadFile(String strLocalFile, String strSftpFile) {
boolean blResult = false;
try {
this.mChannelSftp.put(strLocalFile, strSftpFile);
blResult = true;
} catch (Exception exp) {
exp.printStackTrace();
}
return blResult;
}
// close session
public void close() {
try {
this.mChannelSftp.disconnect();
} catch (Exception exp) {
}
try {
this.mSSHSession.disconnect();
} catch (Exception exp) {
}
this.mChannelSftp = null;
this.mSSHSession = null;
this.mJschSession = null;
}
}
*************************************************************************
**this the code having main method to run the upper code :**
package sftp.main.demo;
import java.util.Vector;
import sftp.bean.SFTPBean;
import com.jcraft.jsch.ChannelSftp.LsEntry;
public class SFTPMainDemo {
// main class let exec demo
public static void main(String[] args) {
// TODO Auto-generated method stub
// now we start up run
SFTPBean sftpBean = new SFTPBean();
boolean blResult = sftpBean.connect("10.14.173.298", 22, "test",
"ab123");
if (blResult) {
System.out.println("Connect successed");
// now we will download file
// blResult =
// sftpBean.downloadFile("/dd1.csv","C:\\Work_old eclipse\\SFTP_Demo");
if (blResult) {
System.out.println("download successed");
} else {
System.out.println("u failed");
}
Vector<LsEntry> vtFiles = sftpBean.listFile("*.txt");
if (vtFiles != null) {
for (LsEntry lsEntry : vtFiles) {
System.out.println(lsEntry.getFilename() + "");
// System.out.println(lsEntry. + "");
}
}
sftpBean.close();
} else {
System.out.println("Connect failed.");
}
}
}
I have an executor service that runs new threads based upon reflection. I have one method that when run does not exit the thread and will hang program execution. I am unsure why this is, can anyone point me to what I'm missing?
Also, if any of the concurrency experts out there notice any problems I may run into, please let me know, I am rather green in concurrency....
Notes:
The connectToFTP method will be refactored to return ChannelSftp in the future.
downloadFromFTP returns false at the end as the method is not completed. I feel that this is the method that is causing the thread to hang. I just don't know why.
The goal of the method is to list each file within an SFTP directory.
from NetworkingShopCa.
#Override
public Object connectToFTP(String username, String password, String host, String port, FtpTypes ftpTypes) {
switch(ftpTypes){
case FTP:
LOGGER.error("Plain FTP is not implemented yet (if ever)");
break;
case FTPS:
FTPSClient client = new FTPSClient();
client.setTrustManager(TrustManagerUtils.getAcceptAllTrustManager());
try {
client.connect(host);
client.enterLocalPassiveMode();
client.login(username, password);
} catch (IOException e) {
LOGGER.error(e.toString());
}
return client;
case SFTP:
JSch jsch = new JSch();
Session session = null;
try {
session = jsch.getSession(username, host);
session.setConfig("StrictHostKeyChecking", "no");
session.setPassword(password);
session.connect();
Channel channel = session.openChannel("sftp");
channel.connect();
ChannelSftp sftpChannel = (ChannelSftp) channel;
return sftpChannel;
} catch (JSchException e) {
// TODO Auto-generated catch block
LOGGER.error(e.toString());
}
break;
default:
LOGGER.error("Invalid FtpType");
break;
}
return false;
}
#Override
public boolean downloadFromFTP(String directory, String filename, boolean downloadAll,Object activeConnection) {
if(activeConnection instanceof ChannelSftp){
ChannelSftp sftpChannel = (ChannelSftp) activeConnection;
try {
sftpChannel.cd(directory);
//List our files within the directory
Vector vv = sftpChannel.ls(directory);
if (vv != null) {
for (int ii = 0; ii < vv.size(); ii++) {
Object obj = vv.elementAt(ii);
if (obj instanceof ChannelSftp.LsEntry) {
LOGGER.debug("[" + ((LsEntry) obj).getFilename() + "]");
}
}
}
} catch (SftpException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
return false;
}
From main
runtimes.put(config.getInt("ESE_PRIORITY"),"RUN_ESE");
ExecutorService threadPool = Executors.newFixedThreadPool(totalRunnables);
LOGGER.info("Executing runtimes in order of priority.");
for(final int priority : runtimes.keySet()){
if(!threadPool.isShutdown() && !threadPool.isTerminated()){
//run the method denoted by the property
final java.lang.reflect.Method method = m.getClass().getMethod(runtimes.get(priority));
Future<?> f = threadPool.submit(new Runnable() {
#Override
public void run() {
try {
method.invoke(m);
} catch (IllegalAccessException | InvocationTargetException e) {
e.printStackTrace();
}
}
});
f.get();
}
}
public void RUN_ESE(){
LOGGER.info("Running ESE");
Networking networking = new NetworkingShopCa();
networking.downloadFromFTP("/toclient/order/processed", "", true, networking.connectToFTP("user", "password", "host", "", FtpTypes.SFTP));
}
--Edit--
Following a debugger downloadFromFTP executes fully, returns out of the method and goes to close the runnable when it hangs:
Line 1153 of ThreadPoolExecutor.java shows:
afterExecute(task, thrown);
Any ideas? For what its worth my build environment is:
Ubuntu 13.04 64 bit
OpenJDK 7 (ubuntu default) with attached sources.
Eclipse kepler
I need to execute(run) a shell script which resides in the server(Solaris) from java. Please help me how to execute a file from java.? I have tried with sendCommand() of TelnetToClient. So please help me in running a file from my GUI.
The program goes like this.
TelnetToPort tele = new TelnetToPort("opmer3");
tele.login("root","root");
String command_ = "/usr/bin/bash /opt/nrl/logs/applications/ns/lccommands.sh";
tele.runComm(command_);
If you are looking for optimized solution for executing any scripts for your java class, then you can use Jsch with Google Expect4j libraries.
For jsch, go to http://www.jcraft.com/jsch/
For Expect4j, go to http://code.google.com/p/expect4j/
Following is small code sample for log in and executing file fro remote java class.
private Expect4j SSH(String hostname, String username,String password, int port) throws Exception {
JSch jsch = new JSch();
Session session = jsch.getSession(username, hostname, port);
if (password != null) {
session.setPassword(password);
}
Hashtable<String,String> config = new Hashtable<String,String>();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.connect(60000);
channel = (ChannelShell) session.openChannel("shell");
Expect4j expect = new Expect4j(channel.getInputStream(), channel.getOutputStream());
channel.connect();
return expect;
}
This method will open up SSH stream to remote server which will be used by expect4j for sending commands.
private boolean executeCommands() {
boolean isSuccess = true;
Closure closure = new Closure() {
public void run(ExpectState expectState) throws Exception {
buffer.append(expectState.getBuffer());
expectState.exp_continue();
}
};
List<Match> lstPattern = new ArrayList<Match>();
String[] regEx = SSHConstants.linuxPromptRegEx;
if (regEx != null && regEx.length > 0) {
synchronized (regEx) {
for (String regexElement : regEx) {//list of regx like, :>, /> etc. it is possible command prompts of your remote machine
try {
RegExpMatch mat = new RegExpMatch(regexElement, closure);
lstPattern.add(mat);
} catch (MalformedPatternException e) {
return false;
} catch(Exception e) {
return false;
}
}
lstPattern.add(new EofMatch( new Closure() { // should cause entire page to be collected
public void run(ExpectState state) {
}
}));
lstPattern.add(new TimeoutMatch(defaultTimeOut, new Closure() {
public void run(ExpectState state) {
}
}));
}
}
try {
Expect4j expect = SSH(objConfig.getHostAddress(), objConfig.getUserName(), objConfig.getPassword(), SSHConstants.SSH_PORT);
expect.setDefaultTimeout(defaultTimeOut);
if(isSuccess) {
for(String strCmd : lstCmds)
isSuccess = isSuccess(lstPattern,strCmd);
}
boolean isFailed = checkResult(expect.expect(lstPattern));
return !isFailed;
} catch (Exception ex) {
return false;
} finally {
closeConnection();
}
}
private boolean isSuccess(List<Match> objPattern,String strCommandPattern) {
try {
boolean isFailed = checkResult(expect.expect(objPattern));
if (!isFailed) {
expect.send(strCommandPattern);
expect.send("\r");
return true;
}
return false;
} catch (MalformedPatternException ex) {
return false;
} catch (Exception ex) {
return false;
}
}
Hope this help.
Thanks.
new ProcessBuilder("ssh", "root#opmer3", command_).start();