The following method is used to parse log files and find errors in them. It works fine until the number of errors run into the several million - 30 million to be specific. Then the server runs out of memory trying to run the application. I get a GC out of memory error. I can change the parameters to java so that it can use more memory, but I would like to optimise the code a bit first.
for (String logFile : fileList) {
Map<String, Long> messageErrorCount = new HashMap<String, Long>();
LOGGER.info("Mining data from: " + logFile);
BufferedReader reader = null;
try {
String hostName = getHostName(logFile);
reader = new BufferedReader(new FileReader(logFile));
long errorCountPerFile = 0;
String logLineStr;
long firstErrorOccurance = 1;
while ((logLineStr = reader.readLine()) != null) {
String errorResult = isErrorLine(logLineStr);
if (errorResult != null) {
errorCountPerFile++;
if (messageErrorCount.containsKey(errorResult)) {
messageErrorCount.put(errorResult, messageErrorCount.get(errorResult) + 1);
} else {
messageErrorCount.put(errorResult, firstErrorOccurance);
}
Pattern pattern = Pattern.compile("^(.*?)\\s-");
Matcher matcher = pattern.matcher(errorResult);
if (matcher.find()) {
String formattedError = matcher.group(1);
if (totalErrorMessages.containsKey(formattedError)) {
totalErrorMessages.put(formattedError, totalErrorMessages.get(formattedError) + 1);
} else {
totalErrorMessages.put(formattedError, firstErrorOccurance);
}
}
}
}
// Need to add the file to list only if count is non zero.
if (errorCountPerFile > 0) {
String fileName = new File(logFile).getName();
fileErrorMaps.put(fileName, messageErrorCount);
putErrorLogRecord(hostName, fileName, errorCountPerFile, messageErrorCount);
}
} catch (IOException e) {
LOGGER.error("File operation failed: " + logFile);
throw new ParserException("File operation failed: " + logFile, e);
} finally {
try {
if (reader != null) {
reader.close();
}
} catch (IOException ex) {
LOGGER.error("File operation failed: " + logFile);
throw new ParserException("File operation failed: " + logFile, ex);
}
}
}
and the isErrorLine method:
public String isErrorLine(String logString) {
if (isWhitelistedError(logString)) {
return null;
} else {
Matcher matcher = ULOG_LINE_PATTERN_QUOTES.matcher(logString);
if (matcher.find()) {
return (matcher.group(2) + " - " + matcher.group(3)).replace("\"", "");
}
matcher = ULOG_LINE_PATTERN_ENDLINE.matcher(logString);
if (matcher.find()) {
return (matcher.group(2) + " - " + matcher.group(3)).replace("\"", "");
}
}
return null;
}
These lines create new objects every time they are used:
Pattern pattern = Pattern.compile("^(.*?)\\s-");
Matcher matcher = pattern.matcher(errorResult);
Is there any better way of doing this method so it uses less memory?
Related
I am making an application for file uploading in Java using jSch. I want to put my file in different directories based on their creation date etc.
I have a main directory "/var/local/recordingsbackup/" in which I am creating other directories and putting data in them.
To achieve this:
I have to create Dir'y like
"/var/local/recordingsbackup/20140207/root/SUCCESS/WN/" and put
data in it.
I've tried this so far:
private void fileTransfer(ChannelSftp channelTarget, temp_recording_log recObj, String filePath) {
int fileNameStartIndex = filePath.lastIndexOf("/") + 1;
String date = new SimpleDateFormat("yyyyMMdd").format(recObj.getCalldate());
String fileName = filePath.substring(fileNameStartIndex);
String staticPath = "/var/local/recordingsbackup/";
String completeBackupPath = staticPath + date + "/" + recObj.getUsername() + "/" + recObj.getStatus() + "/" + recObj.getDisposition() + "/";
try {
InputStream get = SourceChannel.get(filePath);
try {
channelTarget.put(get, completeBackupPath + fileName);
} catch (SftpException e) {
System.out.println("Creating Directory...");
channelTarget.mkdir(completeBackupPath); // error on this line
channelTarget.put(get, completeBackupPath + fileName);
}
} catch (SftpException e) {
log.error("Error Occured ======== File or Directory dosen't exists === " + filePath);
e.printStackTrace();
}
}
If I'm creating single dir like /var/local/recordingsbackup/ then no error occurs and files successfully uploaded.
Please help me in this...how can I create these Nested Directories???
Finally, I've done it.
This is how I got succeed :
try {
channelTarget.put(get, completeBackupPath + fileName);
} catch (SftpException e) {
System.out.println("Creating Directory...");
String[] complPath = completeBackupPath.split("/");
channelTarget.cd("/");
for (String dir : complPath) {
if (folder.length() > 0) {
try {
System.out.println("Current Dir : " + channelTarget.pwd());
channelTarget.cd(folder);
} catch (SftpException e2) {
channelTarget.mkdir(folder);
channelTarget.cd(folder);
}
}
}
channelTarget.cd("/");
System.out.println("Current Dir : " + channelTarget.pwd());
channelTarget.put(get, completeBackupPath + fileName);
}
I don't think what you want to do is possible in the SFTP protocol. You will have to create each sub-directory in turn.
public static void mkdirs(ChannelSftp ch, String path) {
try {
String[] folders = path.split("/");
if (folders[0].isEmpty()) folders[0] = "/";
String fullPath = folders[0];
for (int i = 1; i < folders.length; i++) {
Vector ls = ch.ls(fullPath);
boolean isExist = false;
for (Object o : ls) {
if (o instanceof LsEntry) {
LsEntry e = (LsEntry) o;
if (e.getAttrs().isDir() && e.getFilename().equals(folders[i])) {
isExist = true;
}
}
}
if (!isExist && !folders[i].isEmpty()) {
ch.mkdir(fullPath + folders[i]);
}
fullPath = fullPath + folders[i] + "/";
}
} catch (Exception e) {
e.printStackTrace();
}
}
I used this implementation to create nested folders.
I tried not to use Exception. Keep in mind that filesystem is linux based.
The OP already found a solution but I wanted to append to it.
Simply I do mkdir if the folder that I wanted to create doesn't exist in "ls" result.
Correction of the previous script:
public static void mkdirs(ChannelSftp ch, String path) {
try {
String[] folders = path.split("/");
if (folders[0].isEmpty()) folders[0] = "/";
String fullPath = folders[0];
for (int i = 1; i < folders.length; i++) {
Vector ls = ch.ls(fullPath);
boolean isExist = false;
for (Object o : ls) {
if (o instanceof LsEntry) {
LsEntry e = (LsEntry) o;
if (e.getAttrs().isDir() && e.getFilename().equals(folders[i])) {
isExist = true;
}
}
}
if (!isExist && !folders[i].isEmpty()) {
// Add separator path
ch.mkdir(fullPath + "/" + folders[i]);
}
// Add separator path
fullPath = fullPath + "/" + folders[i] + "/";
}
} catch (Exception e) {
e.printStackTrace();
}
}
Another solution is execute shell command:
String remotePath = "fake/folders/recursive/on/sftp/server";
ChannelExec channel = (ChannelExec) session.openChannel("exec");
// NOTE: the provided paths are expected to require no escaping
channel.setCommand("mkdir -p " + remotePath);
channel.connect();
while (!channel.isClosed()) {
// dir creation is usually fast, so only wait for a short time
Thread.sleep(SHORT_WAIT_MSEC);
}
channel.disconnect();
if (channel.getExitStatus() != 0) {
throw new IOException("Creating directory failed: " + remotePath);
}
I am making an application for file uploading in Java using jSch. I want to put my file in different directories based on their creation date etc.
I have a main directory "/var/local/recordingsbackup/" in which I am creating other directories and putting data in them.
To achieve this:
I have to create Dir'y like
"/var/local/recordingsbackup/20140207/root/SUCCESS/WN/" and put
data in it.
I've tried this so far:
private void fileTransfer(ChannelSftp channelTarget, temp_recording_log recObj, String filePath) {
int fileNameStartIndex = filePath.lastIndexOf("/") + 1;
String date = new SimpleDateFormat("yyyyMMdd").format(recObj.getCalldate());
String fileName = filePath.substring(fileNameStartIndex);
String staticPath = "/var/local/recordingsbackup/";
String completeBackupPath = staticPath + date + "/" + recObj.getUsername() + "/" + recObj.getStatus() + "/" + recObj.getDisposition() + "/";
try {
InputStream get = SourceChannel.get(filePath);
try {
channelTarget.put(get, completeBackupPath + fileName);
} catch (SftpException e) {
System.out.println("Creating Directory...");
channelTarget.mkdir(completeBackupPath); // error on this line
channelTarget.put(get, completeBackupPath + fileName);
}
} catch (SftpException e) {
log.error("Error Occured ======== File or Directory dosen't exists === " + filePath);
e.printStackTrace();
}
}
If I'm creating single dir like /var/local/recordingsbackup/ then no error occurs and files successfully uploaded.
Please help me in this...how can I create these Nested Directories???
Finally, I've done it.
This is how I got succeed :
try {
channelTarget.put(get, completeBackupPath + fileName);
} catch (SftpException e) {
System.out.println("Creating Directory...");
String[] complPath = completeBackupPath.split("/");
channelTarget.cd("/");
for (String dir : complPath) {
if (folder.length() > 0) {
try {
System.out.println("Current Dir : " + channelTarget.pwd());
channelTarget.cd(folder);
} catch (SftpException e2) {
channelTarget.mkdir(folder);
channelTarget.cd(folder);
}
}
}
channelTarget.cd("/");
System.out.println("Current Dir : " + channelTarget.pwd());
channelTarget.put(get, completeBackupPath + fileName);
}
I don't think what you want to do is possible in the SFTP protocol. You will have to create each sub-directory in turn.
public static void mkdirs(ChannelSftp ch, String path) {
try {
String[] folders = path.split("/");
if (folders[0].isEmpty()) folders[0] = "/";
String fullPath = folders[0];
for (int i = 1; i < folders.length; i++) {
Vector ls = ch.ls(fullPath);
boolean isExist = false;
for (Object o : ls) {
if (o instanceof LsEntry) {
LsEntry e = (LsEntry) o;
if (e.getAttrs().isDir() && e.getFilename().equals(folders[i])) {
isExist = true;
}
}
}
if (!isExist && !folders[i].isEmpty()) {
ch.mkdir(fullPath + folders[i]);
}
fullPath = fullPath + folders[i] + "/";
}
} catch (Exception e) {
e.printStackTrace();
}
}
I used this implementation to create nested folders.
I tried not to use Exception. Keep in mind that filesystem is linux based.
The OP already found a solution but I wanted to append to it.
Simply I do mkdir if the folder that I wanted to create doesn't exist in "ls" result.
Correction of the previous script:
public static void mkdirs(ChannelSftp ch, String path) {
try {
String[] folders = path.split("/");
if (folders[0].isEmpty()) folders[0] = "/";
String fullPath = folders[0];
for (int i = 1; i < folders.length; i++) {
Vector ls = ch.ls(fullPath);
boolean isExist = false;
for (Object o : ls) {
if (o instanceof LsEntry) {
LsEntry e = (LsEntry) o;
if (e.getAttrs().isDir() && e.getFilename().equals(folders[i])) {
isExist = true;
}
}
}
if (!isExist && !folders[i].isEmpty()) {
// Add separator path
ch.mkdir(fullPath + "/" + folders[i]);
}
// Add separator path
fullPath = fullPath + "/" + folders[i] + "/";
}
} catch (Exception e) {
e.printStackTrace();
}
}
Another solution is execute shell command:
String remotePath = "fake/folders/recursive/on/sftp/server";
ChannelExec channel = (ChannelExec) session.openChannel("exec");
// NOTE: the provided paths are expected to require no escaping
channel.setCommand("mkdir -p " + remotePath);
channel.connect();
while (!channel.isClosed()) {
// dir creation is usually fast, so only wait for a short time
Thread.sleep(SHORT_WAIT_MSEC);
}
channel.disconnect();
if (channel.getExitStatus() != 0) {
throw new IOException("Creating directory failed: " + remotePath);
}
I have a CSV file that I am having trouble parsing. I am using the opencsv library. Here is what my data looks like and what I am trying to achieve.
RPT_PE,CLASS,RPT_MKT,PROV_CTRCT,CENTER_NM,GK_TY,MBR_NM,MBR_PID
"20150801","NULL","33612","00083249P PCP602","JOE SMITH ARNP","NULL","FRANK, LUCAS E","50004655200"
The issue I am having is the member name ("FRANK, LUCAS E") is being split into two columns and the member name should be one. Again I'm using opencsv and a comma as the separator. Is there any way I can ignore the commas inside the double-quotes?
public void loadCSV(String csvFile, String tableName,
boolean truncateBeforeLoad) throws Exception {
CSVReader csvReader = null;
if (null == this.connection) {
throw new Exception("Not a valid connection.");
}
try {
csvReader = new CSVReader(new FileReader(csvFile), this.seprator);
} catch (Exception e) {
e.printStackTrace();
throw new Exception("Error occured while executing file. "
+ e.getMessage());
}
String[] headerRow = csvReader.readNext();
if (null == headerRow) {
throw new FileNotFoundException(
"No columns defined in given CSV file."
+ "Please check the CSV file format.");
}
String questionmarks = StringUtils.repeat("?,", headerRow.length);
questionmarks = (String) questionmarks.subSequence(0, questionmarks
.length() - 1);
String query = SQL_INSERT.replaceFirst(TABLE_REGEX, tableName);
System.out.println("Base Query: " + query);
String headerRowMod = Arrays.toString(headerRow).replaceAll(", ]", "]");
String[] strArray = headerRowMod.split(",");
query = query
.replaceFirst(KEYS_REGEX, StringUtils.join(strArray, ","));
System.out.println("Add Headers: " + query);
query = query.replaceFirst(VALUES_REGEX, questionmarks);
System.out.println("Add questionmarks: " + query);
String[] nextLine;
Connection con = null;
PreparedStatement ps = null;
try {
con = this.connection;
con.setAutoCommit(false);
ps = con.prepareStatement(query);
if (truncateBeforeLoad) {
//delete data from table before loading csv
con.createStatement().execute("DELETE FROM " + tableName);
}
final int batchSize = 1000;
int count = 0;
Date date = null;
while ((nextLine = csvReader.readNext()) != null) {
System.out.println("Next Line: " + Arrays.toString(nextLine));
if (null != nextLine) {
int index = 1;
for (String string : nextLine) {
date = DateUtil.convertToDate(string);
if (null != date) {
ps.setDate(index++, new java.sql.Date(date
.getTime()));
} else {
ps.setString(index++, string);
}
}
ps.addBatch();
}
if (++count % batchSize == 0) {
ps.executeBatch();
}
}
ps.executeBatch(); // insert remaining records
con.commit();
} catch (SQLException | IOException e) {
con.rollback();
e.printStackTrace();
throw new Exception(
"Error occured while loading data from file to database."
+ e.getMessage());
} finally {
if (null != ps) {
ps.close();
}
if (null != con) {
con.close();
}
csvReader.close();
}
}
public char getSeprator() {
return seprator;
}
public void setSeprator(char seprator) {
this.seprator = seprator;
}
public char getQuoteChar() {
return quoteChar;
}
public void setQuoteChar(char quoteChar) {
this.quoteChar = quoteChar;
}
}
Did you try the the following?
CSVReader reader = new CSVReader(new FileReader("yourfile.csv"), ',');
I wrote a following program and it works for me, I got the following result:
[20150801] [NULL] [33612] [00083249P PCP602] [JOE SMITH ARNP] [NULL]
[FRANK, LUCAS E] [50004655200]
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import au.com.bytecode.opencsv.CSVReader;
public class CVSTest {
/**
* #param args
*/
public static void main(String[] args) {
CSVReader reader = null;
try {
reader = new CSVReader(new FileReader(
"C:/Work/Dev/Projects/Pure_Test/Test/src/cvs"), ',');
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
String[] nextLine;
try {
while ((nextLine = reader.readNext()) != null) {
// nextLine[] is an array of values from the line
System.out.println("[" + nextLine[0] + "] [" + nextLine[1]
+ "] [" + nextLine[2] + "] [" + nextLine[3] + "] ["
+ nextLine[4] + "] [" + nextLine[5] + "] ["
+ nextLine[6] + "] [" + nextLine[7] + "]");
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
According to the documentation, you can supply custom separator and quote characters in the constructor, which should deal with it:
CSVReader(Reader reader, char separator, char quotechar)
Construct your reader with , as separator and " as quotechar.
It is simple to load your CSV as an SQL table into HSQLDB, then select rows from the table to insert into another database. HSQLDB handles commas inside quotes. You need to define your text source as "quoted". See this:
http://hsqldb.org/doc/2.0/guide/texttables-chapt.html
Your case should be handled out of the box with no special configuration required.
If you can't make it work, then just switch to uniVocity-parsers to do this for you - it's twice as fast in comparison to OpenCSV, requires much less code and is packed with features.
CsvParserSettings settings = new CsvParserSettings(); // you have many configuration options here - check the tutorial.
CsvParser parser = new CsvParser(settings);
List<String[]> allRows = parser.parseAll(new FileReader(new File("C:/Work/Dev/Projects/Pure_Test/Test/src/cvs")));
Disclosure: I am the author of this library. It's open-source and free (Apache V2.0 license).
I have written a multithreaded application that analyzes rows in a database with regex and updates them appropriately. I am writing each row to a log file for logging purposes. I have noticed that the same row is being written to the log file several times...sometimes upwards of 15 times. Here are snippets of the code.
Setting up ThreadPoolExecuter:
private static BlockingQueue<Runnable> worksQueue = new ArrayBlockingQueue<Runnable>(blockingQueueSize);
private static ThreadPoolExecutor exec = new ThreadPoolExecutor(threadPoolSize, threadPoolSize, 10, TimeUnit.SECONDS, worksQueue);
In this part, I run a query, then go through the results:
rs = ps.executeQuery();
while (rs.next()) {
exec.execute(new UpdateMember(rs, conn, fileWriter));
if (worksQueue.size() == blockingQueueSize) {
//reach the maximum, stop refill
for (;;) {
Thread.yield();
//wait until the size of queue reached the minimum
if (worksQueue.size() == 0) {
//start refill
break;
}
}
}
}
UpdateMember (with only run and writeToLog methods showing):
public class UpdateMember implements Runnable {
ResultSet rs;
Connection conn;
FileWriter fw;
public UpdateMember(ResultSet rs, Connection conn, FileWriter fw) {
this.rs = rs;
this.conn = conn;
this.fw = fw;
}
#Override
public void run() {
try {
String regex = "((?<city>[a-zA-Z\\s\\.]+)\\s)?(?<provState>AB|ALB|Alta|alberta|BC|B\\.C\\.|British Columbia|LB|Labrador|MB|Man|Manitoba|N[BLTSU]|Nfld|NF|Newfoundland|NWT|Northwest Territories|Nova Scotia|New Brunswick|Nunavut|ON|ONT|Ontario|PE|PEI|Prince Edward Island|QC|PC|QUE|QU|Quebec|SK|Sask|Saskatchewan|YT|Yukon|Yukon Territories)(\\s(?<country>CA|CAN|CANADA))?$";
Pattern pattern = Pattern.compile(regex, Pattern.CASE_INSENSITIVE);
BigDecimal memrecno = rs.getBigDecimal(2);
String addressLineTwo = rs.getString(4);
String addressLineThree = rs.getString(5);
String addressLineFour = rs.getString(6);
BigDecimal attrrecno = rs.getBigDecimal(9);
String addressBeingParsed = "";
String city = null;
String province = null;
String country = null;
boolean usingAddressThree = false;
boolean usingAddressFour = false;
if (addressLineFour == null) {
if (addressLineThree == null) {
city = "Unknown";
}
else
{
addressBeingParsed = addressLineThree;
usingAddressThree = true;
}
}
else
{
addressBeingParsed = addressLineFour;
usingAddressFour = true;
}
if (usingAddressThree || usingAddressFour) {
Matcher matcher = pattern.matcher(addressBeingParsed);
// if matches are found
if (matcher.matches()) {
city = matcher.group("city");
province = matcher.group("provState");
country = matcher.group("country");
if (city == null || city.isEmpty()) {
// cities are alpha characters and spaces only
String cityRegex = "(?<city>^[a-zA-Z\\s\\.]+$)";
Pattern cityPattern = Pattern.compile(cityRegex, Pattern.CASE_INSENSITIVE);
if (usingAddressFour && (addressLineThree != null) && !addressLineThree.isEmpty()) {
Matcher cityMatcher = cityPattern.matcher(addressLineThree);
if (cityMatcher.matches()) {
city = cityMatcher.group("city");
}
else
{
city = "Unknown";
}
}
else if (usingAddressThree && (addressLineTwo != null) && !addressLineTwo.isEmpty()) {
Matcher cityMatcher = cityPattern.matcher(addressLineTwo);
if (cityMatcher.matches()) {
city = cityMatcher.group("city");
}
else
{
city = "Unknown";
}
}
else
{
city = "Unknown";
}
}
if (province != null && !province.isEmpty()) {
province = createProvinceCode(province);
}
}
else
{
city = "Unknown";
}
}
// update attributes in database
boolean success = updateRow(memrecno, attrrecno, city, province);
String logLine = memrecno.toString() + "|" + attrrecno.toString() + "|" + addressLineTwo + "|" + addressLineThree + "|" + addressLineFour + "|" + city + "|" + province + "|" + country + "|" + success + "|" + String.valueOf(Thread.currentThread().getId());
writeToLog(logLine);
}
catch (Exception e)
{
e.printStackTrace();
}
}
private synchronized void writeToLog(String line) {
try {
fw.write(line + "\r\n");
fw.flush();
}
catch (IOException ex)
{
System.out.println("Error writing to log file. " + ex.getMessage());
}
}
}
I don't know if the threads are also calling the updateRow method multiple times, but I'm assuming they are and that's really bad.
Any ideas as to why it would be doing this?
I don't think ResultSet is thread safe. From your code, you should get the value first and then pass the value instead of rs into the thread.
I am making an application for file uploading in Java using jSch. I want to put my file in different directories based on their creation date etc.
I have a main directory "/var/local/recordingsbackup/" in which I am creating other directories and putting data in them.
To achieve this:
I have to create Dir'y like
"/var/local/recordingsbackup/20140207/root/SUCCESS/WN/" and put
data in it.
I've tried this so far:
private void fileTransfer(ChannelSftp channelTarget, temp_recording_log recObj, String filePath) {
int fileNameStartIndex = filePath.lastIndexOf("/") + 1;
String date = new SimpleDateFormat("yyyyMMdd").format(recObj.getCalldate());
String fileName = filePath.substring(fileNameStartIndex);
String staticPath = "/var/local/recordingsbackup/";
String completeBackupPath = staticPath + date + "/" + recObj.getUsername() + "/" + recObj.getStatus() + "/" + recObj.getDisposition() + "/";
try {
InputStream get = SourceChannel.get(filePath);
try {
channelTarget.put(get, completeBackupPath + fileName);
} catch (SftpException e) {
System.out.println("Creating Directory...");
channelTarget.mkdir(completeBackupPath); // error on this line
channelTarget.put(get, completeBackupPath + fileName);
}
} catch (SftpException e) {
log.error("Error Occured ======== File or Directory dosen't exists === " + filePath);
e.printStackTrace();
}
}
If I'm creating single dir like /var/local/recordingsbackup/ then no error occurs and files successfully uploaded.
Please help me in this...how can I create these Nested Directories???
Finally, I've done it.
This is how I got succeed :
try {
channelTarget.put(get, completeBackupPath + fileName);
} catch (SftpException e) {
System.out.println("Creating Directory...");
String[] complPath = completeBackupPath.split("/");
channelTarget.cd("/");
for (String dir : complPath) {
if (folder.length() > 0) {
try {
System.out.println("Current Dir : " + channelTarget.pwd());
channelTarget.cd(folder);
} catch (SftpException e2) {
channelTarget.mkdir(folder);
channelTarget.cd(folder);
}
}
}
channelTarget.cd("/");
System.out.println("Current Dir : " + channelTarget.pwd());
channelTarget.put(get, completeBackupPath + fileName);
}
I don't think what you want to do is possible in the SFTP protocol. You will have to create each sub-directory in turn.
public static void mkdirs(ChannelSftp ch, String path) {
try {
String[] folders = path.split("/");
if (folders[0].isEmpty()) folders[0] = "/";
String fullPath = folders[0];
for (int i = 1; i < folders.length; i++) {
Vector ls = ch.ls(fullPath);
boolean isExist = false;
for (Object o : ls) {
if (o instanceof LsEntry) {
LsEntry e = (LsEntry) o;
if (e.getAttrs().isDir() && e.getFilename().equals(folders[i])) {
isExist = true;
}
}
}
if (!isExist && !folders[i].isEmpty()) {
ch.mkdir(fullPath + folders[i]);
}
fullPath = fullPath + folders[i] + "/";
}
} catch (Exception e) {
e.printStackTrace();
}
}
I used this implementation to create nested folders.
I tried not to use Exception. Keep in mind that filesystem is linux based.
The OP already found a solution but I wanted to append to it.
Simply I do mkdir if the folder that I wanted to create doesn't exist in "ls" result.
Correction of the previous script:
public static void mkdirs(ChannelSftp ch, String path) {
try {
String[] folders = path.split("/");
if (folders[0].isEmpty()) folders[0] = "/";
String fullPath = folders[0];
for (int i = 1; i < folders.length; i++) {
Vector ls = ch.ls(fullPath);
boolean isExist = false;
for (Object o : ls) {
if (o instanceof LsEntry) {
LsEntry e = (LsEntry) o;
if (e.getAttrs().isDir() && e.getFilename().equals(folders[i])) {
isExist = true;
}
}
}
if (!isExist && !folders[i].isEmpty()) {
// Add separator path
ch.mkdir(fullPath + "/" + folders[i]);
}
// Add separator path
fullPath = fullPath + "/" + folders[i] + "/";
}
} catch (Exception e) {
e.printStackTrace();
}
}
Another solution is execute shell command:
String remotePath = "fake/folders/recursive/on/sftp/server";
ChannelExec channel = (ChannelExec) session.openChannel("exec");
// NOTE: the provided paths are expected to require no escaping
channel.setCommand("mkdir -p " + remotePath);
channel.connect();
while (!channel.isClosed()) {
// dir creation is usually fast, so only wait for a short time
Thread.sleep(SHORT_WAIT_MSEC);
}
channel.disconnect();
if (channel.getExitStatus() != 0) {
throw new IOException("Creating directory failed: " + remotePath);
}