I want to create a Quartz job which reads .csv files and moves them when file is processed. I tried this:
#Override
public void execute(JobExecutionContext context) {
File directoryPath = new File("C:\\csv\\nov");
// Create a new subfolder called "processed" into source directory
try {
Files.createDirectory(Path.of(directoryPath.getAbsolutePath() + "/processed"));
} catch (IOException e) {
throw new RuntimeException(e);
}
FilenameFilter textFileFilter = (dir, name) -> {
String lowercaseName = name.toLowerCase();
if (lowercaseName.endsWith(".csv")) {
return true;
} else {
return false;
}
};
// List of all the csv files
File filesList[] = directoryPath.listFiles(textFileFilter);
System.out.println("List of the text files in the specified directory:");
Optional<File> csvFile = Arrays.stream(filesList).findFirst();
File file = csvFile.get();
for(File file : filesList) {
try {
List<CsvLine> beans = new CsvToBeanBuilder(new FileReader(file.getAbsolutePath(), StandardCharsets.UTF_16))
.....
.build()
.parse();
for(CsvLine item: beans){
....... sql queries
Optional<ProcessedWords> isFound = processedWordsService.findByKeyword(item.getKeyword());
......................................
}
} catch (Exception e){
e.printStackTrace();
}
// Move here file into new subdirectory when file processing is finished
Path copied = Paths.get(file.getAbsolutePath() + "/processed");
Path originalPath = file.toPath();
try {
Files.move(originalPath, copied, StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
Folder processed is created when the job is started but I get exception:
2022-11-17 23:12:51.470 ERROR 16512 --- [cessor_Worker-4] org.quartz.core.JobRunShell : Job DEFAULT.keywordPostJobDetail threw an unhandled Exception:
java.lang.RuntimeException: java.nio.file.FileSystemException: C:\csv\nov\11_42_33.csv -> C:\csv\nov\processed\11_42_33.csv: The process cannot access the file because it is being used by another process
at com.wordscore.engine.processor.ImportCsvFilePostJob.execute(ImportCsvFilePostJob.java:127) ~[main/:na]
at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz-2.3.2.jar:na]
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) ~[quartz-2.3.2.jar:na]
Caused by: java.nio.file.FileSystemException: C:\csv\nov\11_42_33.csv -> C:\csv\nov\processed\11_42_33.csv: The process cannot access the file because it is being used by another process
at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92) ~[na:na]
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) ~[na:na]
at java.base/sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:403) ~[na:na]
at java.base/sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:293) ~[na:na]
at java.base/java.nio.file.Files.move(Files.java:1432) ~[na:na]
at com.wordscore.engine.processor.ImportCsvFilePostJob.execute(ImportCsvFilePostJob.java:125) ~[main/:na]
... 2 common frames omitted
Do you know how I can release the file and move it into a sub directory?
EDIT: Update code with try-catch
#Override
public void execute(JobExecutionContext context) {
File directoryPath = new File("C:\\csv\\nov");
// Create a new subfolder called "processed" into source directory
try {
Path path = Path.of(directoryPath.getAbsolutePath() + "/processed");
if (!Files.exists(path) || !Files.isDirectory(path)) {
Files.createDirectory(path);
}
} catch (IOException e) {
throw new RuntimeException(e);
}
FilenameFilter textFileFilter = (dir, name) -> {
String lowercaseName = name.toLowerCase();
if (lowercaseName.endsWith(".csv")) {
return true;
} else {
return false;
}
};
// List of all the csv files
File filesList[] = directoryPath.listFiles(textFileFilter);
System.out.println("List of the text files in the specified directory:");
Optional<File> csvFile = Arrays.stream(filesList).findFirst();
File file = csvFile.get();
for(File file : filesList) {
try {
try (var br = new FileReader(file.getAbsolutePath(), StandardCharsets.UTF_16)){
List<CsvLine> beans = new CsvToBeanBuilder(br)
......
.build()
.parse();
for (CsvLine item : beans) {
.....
if (isFound.isPresent()) {
.........
}}
} catch (Exception e){
e.printStackTrace();
}
// Move here file into new subdirectory when file processing is finished
Path copied = Paths.get(file.getAbsolutePath() + "/processed");
Path originalPath = file.toPath();
try {
Files.move(originalPath, copied, StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
Quartz config:
#Configuration
public class SchedulerConfig {
private static final Logger LOG = LoggerFactory.getLogger(SchedulerConfig.class);
private ApplicationContext applicationContext;
#Autowired
public SchedulerConfig(ApplicationContext applicationContext) {
this.applicationContext = applicationContext;
}
#Bean
public JobFactory jobFactory() {
AutowiringSpringBeanJobFactory jobFactory = new AutowiringSpringBeanJobFactory();
jobFactory.setApplicationContext(applicationContext);
return jobFactory;
}
#Bean
public SchedulerFactoryBean schedulerFactoryBean(Trigger simpleJobTrigger) throws IOException {
SchedulerFactoryBean schedulerFactory = new SchedulerFactoryBean();
schedulerFactory.setQuartzProperties(quartzProperties());
schedulerFactory.setWaitForJobsToCompleteOnShutdown(true);
schedulerFactory.setAutoStartup(true);
schedulerFactory.setTriggers(simpleJobTrigger);
schedulerFactory.setJobFactory(jobFactory());
return schedulerFactory;
}
#Bean
public SimpleTriggerFactoryBean simpleJobTrigger(#Qualifier("keywordPostJobDetail") JobDetail jobDetail,
#Value("${simplejob.frequency}") long frequency) {
LOG.info("simpleJobTrigger");
SimpleTriggerFactoryBean factoryBean = new SimpleTriggerFactoryBean();
factoryBean.setJobDetail(jobDetail);
factoryBean.setStartDelay(1000);
factoryBean.setRepeatInterval(frequency);
factoryBean.setRepeatCount(4); // factoryBean.setRepeatCount(SimpleTrigger.REPEAT_INDEFINITELY);
return factoryBean;
}
#Bean
public JobDetailFactoryBean keywordPostJobDetail() {
JobDetailFactoryBean factoryBean = new JobDetailFactoryBean();
factoryBean.setJobClass(ImportCsvFilePostJob.class);
factoryBean.setDurability(true);
return factoryBean;
}
public Properties quartzProperties() throws IOException {
PropertiesFactoryBean propertiesFactoryBean = new PropertiesFactoryBean();
propertiesFactoryBean.setLocation(new ClassPathResource("/quartz.properties"));
propertiesFactoryBean.afterPropertiesSet();
return propertiesFactoryBean.getObject();
}
}
Quartz config:
org.quartz.scheduler.instanceName=wordscore-processor
org.quartz.scheduler.instanceId=AUTO
org.quartz.threadPool.threadCount=5
org.quartz.jobStore.class=org.quartz.simpl.RAMJobStore
As you can see I wan to have 5 threads in order to execute 5 parallel jobs. Do you know how I can process the files without this exception?
new FileReader(file.getAbsolutePath(), StandardCharsets.UTF_16)
This parts creates a resource. A resource is an object that represents an underlying heavy thing - a thing that you can have very few of. In this case, it represents an underlying OS file handle.
You must always safely close these. There are really only 2 ways to do it correctly:
Use try-with-resources
Save it to a field, and make yourself AutoClosable so the code that uses of instances of this class can use try-with-resources
try (var br = new FileReader(file, StandardCharsets.UTF_16)) {
List<CsvLine> beans = new CsvToBeanBuilder(br)
.....
.build()
.parse();
}
Is the answer.
Although I agree completely with the answer and comments of #rzwitserloot, note the following in your error stack trace:
java.nio.file.FileSystemException: C:\csv\nov\07_06_26.csv -> C:\csv\nov\07_06_26.csv\processed: The process cannot access the file because it is being used by another process
You are trying moving your file to the backup directory, but note you are doing it to the wrong path, C:\csv\nov\07_06_26.csv\processed, in the example.
Please, try the following:
#Override
public void execute(JobExecutionContext context) {
File directoryPath = new File("C:\\csv\\nov");
// Create a new subfolder called "processed" into source directory
// Hold a reference to the processed files directory path, we will
// use it later
Path processedDirectoryPath;
try {
processedDirectoryPath = Path.of(directoryPath.getAbsolutePath() + "/processed");
if (!Files.exists(processedDirectoryPath) || !Files.isDirectory(processedDirectoryPath)) {
Files.createDirectory(processedDirectoryPath);
}
} catch (IOException e) {
throw new RuntimeException(e);
}
FilenameFilter textFileFilter = (dir, name) -> {
String lowercaseName = name.toLowerCase();
if (lowercaseName.endsWith(".csv")) {
return true;
} else {
return false;
}
};
// List of all the csv files
File filesList[] = directoryPath.listFiles(textFileFilter);
System.out.println("List of the text files in the specified directory:");
for(File file : filesList) {
try {
try (var br = new FileReader(file.getAbsolutePath(), StandardCharsets.UTF_16)){
List<CsvLine> beans = new CsvToBeanBuilder(br)
......
.build()
.parse();
for (CsvLine item : beans) {
.....
if (isFound.isPresent()) {
.........
}}
} catch (Exception e){
e.printStackTrace();
}
// Move here file into new subdirectory when file processing is finished
// In my opinion, here is the error:
// Path copied = Paths.get(file.getAbsolutePath() + "/processed");
Path originalPath = file.toPath();
try {
// Note the use of the path we defined before
Files.move(originalPath, processedDirectoryPath.resolve(originalPath.getFileName()),
StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
If you need to increase the throughput of files processed, you could try splitting them in batches, say for certain pattern in their name like a month name or a job number, for instance. The simple solution could be to use the provided JobExecutionContext of every job to include some split criteria. That criteria will be used in your FilenameFilter causing every job to process only a certain portion of the whole amount of files that need to be processed. I think the solution is preferable to any kind of locking or similar mechanism..
For example, consider the following:
#Override
public void execute(JobExecutionContext context) {
File directoryPath = new File("C:\\csv\\nov");
// Create a new subfolder called "processed" into source directory
// Hold a reference to the processed files directory path, we will
// use it later
Path processedDirectoryPath;
try {
processedDirectoryPath = Path.of(directoryPath.getAbsolutePath() + "/processed");
if (!Files.exists(processedDirectoryPath) || !Files.isDirectory(processedDirectoryPath)) {
Files.createDirectory(processedDirectoryPath);
}
} catch (IOException e) {
throw new RuntimeException(e);
}
// We obtain the file processing criteria using a job parameter
JobDataMap data = context.getJobDetail().getJobDataMap();
String filenameProcessingCriteria = data.getString("FILENAME_PROCESSING_CRITERIA");
// Use the provided criteria to restrict the files that this job
// will process
FilenameFilter textFileFilter = (dir, name) -> {
String lowercaseName = name.toLowerCase();
if (lowercaseName.endsWith(".csv") && lowercaseName.indexOf(filenameProcessingCriteria) > 0) {
return true;
} else {
return false;
}
};
// List of all the csv files
File filesList[] = directoryPath.listFiles(textFileFilter);
System.out.println("List of the text files in the specified directory:");
for(File file : filesList) {
try {
try (var br = new FileReader(file.getAbsolutePath(), StandardCharsets.UTF_16)){
List<CsvLine> beans = new CsvToBeanBuilder(br)
......
.build()
.parse();
for (CsvLine item : beans) {
.....
if (isFound.isPresent()) {
.........
}}
} catch (Exception e){
e.printStackTrace();
}
// Move here file into new subdirectory when file processing is finished
// In my opinion, here is the error:
// Path copied = Paths.get(file.getAbsolutePath() + "/processed");
Path originalPath = file.toPath();
try {
// Note the use of the path we defined before
Files.move(originalPath, processedDirectoryPath.resolve(originalPath.getFileName()),
StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
You need to pass the required parameter to your jobs:
JobDetail job1 = ...;
job1.getJobDataMap().put("FILENAME_PROCESSING_CRITERIA", "job1pattern");
An even simpler approach, based on the same idea, could be splitting the files in different folders and pass the folder name that need to be processed as a job parameter:
#Override
public void execute(JobExecutionContext context) {
// We obtain the directory path as a job parameter
JobDataMap data = context.getJobDetail().getJobDataMap();
String directoryPathName = data.getString("DIRECTORY_PATH_NAME");
File directoryPath = new File(directoryPathName);
// Create a new subfolder called "processed" into source directory
// Hold a reference to the processed files directory path, we will
// use it later
Path processedDirectoryPath;
try {
processedDirectoryPath = Path.of(directoryPath.getAbsolutePath() + "/processed");
if (!Files.exists(processedDirectoryPath) || !Files.isDirectory(processedDirectoryPath)) {
Files.createDirectory(processedDirectoryPath);
}
} catch (IOException e) {
throw new RuntimeException(e);
}
FilenameFilter textFileFilter = (dir, name) -> {
String lowercaseName = name.toLowerCase();
if (lowercaseName.endsWith(".csv")) {
return true;
} else {
return false;
}
};
// List of all the csv files
File filesList[] = directoryPath.listFiles(textFileFilter);
System.out.println("List of the text files in the specified directory:");
for(File file : filesList) {
try {
try (var br = new FileReader(file.getAbsolutePath(), StandardCharsets.UTF_16)){
List<CsvLine> beans = new CsvToBeanBuilder(br)
......
.build()
.parse();
for (CsvLine item : beans) {
.....
if (isFound.isPresent()) {
.........
}}
} catch (Exception e){
e.printStackTrace();
}
// Move here file into new subdirectory when file processing is finished
// In my opinion, here is the error:
// Path copied = Paths.get(file.getAbsolutePath() + "/processed");
Path originalPath = file.toPath();
try {
// Note the use of the path we defined before
Files.move(originalPath, processedDirectoryPath.resolve(originalPath.getFileName()),
StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
And pass a different folder to every different job:
JobDetail job1 = ...;
job1.getJobDataMap().put("DIRECTORY_PATH_NAME", "C:\\csv\\nov");
Please, consider refactor your code and define methods for file processing, file backup, etc, it will make your code easy to understand and handle.
Assuming we have File file = new File("c:/test.txt"), and print the the following paths:
Path copied = Paths.get(file.getAbsolutePath() + "/processed");
Path originalPath = file.toPath();
We will get the result:
copied: C:\test.txt\processed
originalPath: C:\test.txt
So its incorrect. You should try to get the parent path plus the processed folder plus the file name.
Path copied = Paths.get(file.getParentFile().getAbsolutePath() + "/processed/" + file.getName());
Path originalPath = file.toPath();
The line in error message
Caused by: java.lang.RuntimeException:
java.nio.file.FileSystemException: C:\csv\nov\07_06_26.csv ->
C:\csv\nov\07_06_26.csv\processed: The process cannot access the file
because it is being used by another process
I think you want to move the file from C:\csv\nov to C:\csv\nov\processed, so
you have to change following line:
Path copied = Paths.get(file.getAbsolutePath() + "/processed");
to
Path copied = Paths.get(file.getParent() + "/processed");
because file.getAbsolutePath() returns the complete path, include the name of file.
I’m pretty sure that the file is being locked by the file reader that you create but never close in the following line:
List<CsvLine> beans = new CsvToBeanBuilder(new FileReader(file.getAbsolutePath(), StandardCharsets.UTF_16))
Refactor your code so that you have that reader in a try finally block or close it explicitly.
The unintuitive behavior you might see is that those files are released at seemingly random times. This is because when the garbage collector frees up those readers, they will then release the files. Clean them up explicitly instead.
Related
I use this code to list files into directory and move it based on a found value.
#Override
public void execute(JobExecutionContext context) {
File directoryPath = new File("C:\\csv\\nov");
// Create a new subfolder called "processed" into source directory
try {
Path processedFolderPath = Path.of(directoryPath.getAbsolutePath() + "/processed");
if (!Files.exists(processedFolderPath) || !Files.isDirectory(processedFolderPath)) {
Files.createDirectory(processedFolderPath);
}
Path invalidFilesFolderPath = Path.of(directoryPath.getAbsolutePath() + "/invalid_files");
if (!Files.exists(invalidFilesFolderPath) || !Files.isDirectory(invalidFilesFolderPath)) {
Files.createDirectory(invalidFilesFolderPath);
}
} catch (IOException e) {
throw new RuntimeException(e);
}
FilenameFilter textFileFilter = (dir, name) -> {
String lowercaseName = name.toLowerCase();
if (lowercaseName.endsWith(".csv")) {
return true;
} else {
return false;
}
};
// List of all the csv files
File filesList[] = directoryPath.listFiles(textFileFilter);
System.out.println("List of the text files in the specified directory:");
for(File file : filesList) {
try {
try (var br = new FileReader(file.getAbsolutePath(), StandardCharsets.UTF_16)){
List<CsvLine> beans = new CsvToBeanBuilder(br)
.withType(CsvLine.class)
.withSeparator('\t')
.withSkipLines(3)
.build()
.parse();
for (CsvLine item : beans)
{
Path originalPath = file.toPath();
if(item.getValue().equals(2)
|| item.getValue().equals(43)
|| item.getValue().equals(32))
{
// Move here file into new subdirectory when file is invalid
Path copied = Paths.get(file.getParent() + "/invalid_files");
try {
// Use resolve method to keep the "processed" as folder
br.close();
Files.move(originalPath, copied.resolve(originalPath.getFileName()), StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
throw new RuntimeException(e);
}
}}
} catch (Exception e){
e.printStackTrace();
}
Path originalPath = file.toPath();
System.out.println(String.format("\nProcessed file : %s, moving the file to subfolder /processed\n",
originalPath));
}
// Move here file into new subdirectory when file processing is finished
Path copied = Paths.get(file.getParent() + "/processed");
try {
// Use resolve method to keep the "processed" as folder
br.close();
Files.move(originalPath, copied.resolve(originalPath.getFileName()), StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
The problem is that I try to move a file. I get error:
java.lang.RuntimeException: java.nio.file.NoSuchFileException: C:\csv\nov\12_21_39.csv
at com.wordscore.engine.processor.DataValidationCheckJob.execute(DataValidationCheckJob.java:94)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.nio.file.NoSuchFileException: C:\csv\nov\12_21_39.csv
How I can edit the code in way that it can be moved properly?
It looks like you have omitted break; after both Files.move(originalPath, ...); calls.
Without a break to exit the for (CsvLine item : beans) loop, processing continues over the next iteration. The second iteration will attempt to perform Files.move(originalPath, ...) again - and fails because the file has already been moved.
Therefore you get the NoSuchFileException which you have re-thrown as RuntimeException in your try.. Files.move .. catch block.
The handling would be cleaner if you dealt with move in one place and such that it avoids the untidy br.close(), leaving file handling to the auto-close try() block. Something like this:
Path invalidDir = Paths.get(file.getParent() + "/invalid_files");
Path processedDir = Paths.get(file.getParent() + "/processed");
for(File file : filesList) {
Path originalPath = file.toPath();
Path moveTo = processedDir.resolve(originalPath.getFileName());
try (var br = new FileReader(file.getAbsolutePath(), StandardCharsets.UTF_16)){
List<CsvLine> beans = new CsvToBeanBuilder(br)
.withType(CsvLine.class)
.withSeparator('\t')
.withSkipLines(3)
.build()
.parse();
for (CsvLine item : beans) {
if(item.getValue().equals(2)
|| item.getValue().equals(43)
|| item.getValue().equals(32)) {
// file is invalid, skip it
moveTo = invalidDir.resolve(originalPath.getFileName());
// LOG MSG HERE
break;
}
}
}
Files.move(originalPath, moveTo, StandardCopyOption.REPLACE_EXISTING);
System.out.format("%nProcessed file : %s, moving the file to subfolder %s%n", originalPath, moveTo);
}
I've a Java program that is copying a file from Unix to hdfs. It is running fine however I am looking for impersonating a different account when it runs and copies file.
Input: Apart form input file and target hdfs directory path, another input should be properties file containing account, keytab directory, domain
Please kindly let me know the best way to move forward.
I am currently exploring using a shell to first issue a kinit command and then run the jar
I am also reading about Jaas and how this can be done in Java itself - from - https://henning.kropponline.de/2016/02/14/a-secure-hdfs-client-example/
Need inputs and any reference of available options.
My Java program that copies file is as below:
public class FileCopy implements Runnable {
#Option(names = {"-i","--input"}, required=true, description="file name to copy to hadoop")
String input;
#Option(names = {"-o","--output"}, required=true, description="hdfs directory path to be copied into")
String output;
public void run() {
Properties hadoop_properties = new Properties();
HdfsFileDeploy hdfsFileDeploy = new HdfsFileDeploy();
try {
hadoop_properties.load(FileCopy.class.getClassLoader().getResourceAsStream("hadoop.properties"));
} catch (IOException e) {
e.printStackTrace();
}
FileSystem fs = hdfsFileDeploy.configureFilesystem(hadoop_properties.getProperty("coreSitePath"),hadoop_properties.getProperty("hdfsSitePath"));
String status = hdfsFileDeploy.writeToHDFS(fs,input,output);
if (status == "SUCCESS") {
System.out.println("completed copying");
} else {
System.out.println("copying error");
}
hdfsFileDeploy.closeFileSystem(fs);
}
public static void main(String[] args) throws IOException {
CommandLine.run(new FileCopy(), args);
}
}
public class HdfsFileDeploy {
public FileSystem configureFilesystem(String coreSitePath, String hdfsSitePath) {
FileSystem fileSystem = null;
try {
Configuration conf = new Configuration();
Path hdfsCoreSitePath = new Path(coreSitePath);
Path hdfsHDFSSitePath = new Path(hdfsSitePath);
conf.addResource(hdfsCoreSitePath);
conf.addResource(hdfsHDFSSitePath);
fileSystem = FileSystem.get(conf);
System.out.println(fileSystem);
return fileSystem;
} catch (Exception ex) {
ex.printStackTrace();
return fileSystem;
}
}
public void closeFileSystem(FileSystem fileSystem) {
try {
fileSystem.close();
} catch (Exception ex) {
System.out.println("Unable to close Hadoop filesystem : " + ex);
}
}
//
public String writeToHDFS(FileSystem fileSystem, String sourcePath, String destinationPath) {
String failure = "FAILURE";
String success = "SUCCESS";
Boolean doNotDelSrc = false;
Boolean overwrite = true;
try {
Path inputPath = new Path(sourcePath);
Path outputPath = new Path(destinationPath);
if(!fileSystem.exists(outputPath)) {
System.out.println("Output path " + outputPath + " does not exist. Creating outputPath directory now..");
if (fileSystem.mkdirs(outputPath)) {
System.out.println("Output path " + outputPath + " created...");
}
}
System.out.println("about to copy from " + inputPath + " to " + outputPath);
fileSystem.copyFromLocalFile(doNotDelSrc, overwrite, inputPath, outputPath);
return success;
} catch (IOException ex) {
System.out.println("Some exception occurred while writing file to hdfs");
ex.printStackTrace();
return failure;
}
}
}
Input1: input file
Input2: target hdfs directory
Reference Input: file (say yaml) containing account, domain, keytab path.
jar should impersonate and copy the input file to target hdfs directory.
I am using the common source code for setting up a directory poller in Spring Integration. The Poller uses NioLocker to lock and file, and per JavaDocs, I am responsible for unlocking the file using NioFileLocker.unlock(file).
My main questions are when should this take place and of course how?
My initial thought is after my job listener moves the file to a new directory. At this time I only know the file path name and do not have a handle for the File itself.
The directory poller code:
#Bean
#InboundChannelAdapter(value = "claimInputChannel", poller = #Poller(fixedDelay = "5000"))
public FileReadingMessageSource claimPollingFileSource(){
logger.debug("Setting up inbound channel adapter ===> claimInputChannel");
CompositeFileListFilter<File> compositeFileListFilter= new CompositeFileListFilter<File>();
compositeFileListFilter.addFilter(new RegexPatternFileListFilter("(?i).*_CLAIM_.*[.]txt"));
compositeFileListFilter.addFilter(lastModifiedFilter());
compositeFileListFilter.addFilter(new IgnoreHiddenFileListFilter());
compositeFileListFilter.addFilter(new AcceptOnceFileListFilter<File>());
FileReadingMessageSource pollDirectory = new FileReadingMessageSource();
pollDirectory.setDirectory(new File(pollingDirectory));
pollDirectory.setAutoCreateDirectory(false);
pollDirectory.setFilter(compositeFileListFilter);
pollDirectory.setLocker(new NioFileLocker());
return pollDirectory;
}
AfterJob code in the listener:
#Override
public void afterJob(JobExecution jobExecution) {
logger.debug("In afterJob routine");
if (jobExecution.getStatus() == BatchStatus.COMPLETED) {
// check if directory exists
File dir = new File(processedDirectory);
if (dir.exists()) {
Path sourcePath = Paths.get(jobExecution.getJobParameters().getString("fileName"));
String target = processedDirectory + "/" + sourcePath.getFileName();
logger.debug("Moving processed file " + sourcePath.toString() + " to " + target);
try {
Files.move(sourcePath, Paths.get(target), StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
logger.error("Something went wrong in moving file to processed directory.");
e.printStackTrace();
}
} else {
logger.error("The " + processedDirectory + " does not exist.");
}
} else {
logger.debug("Batch job is not yet complete.");
}
}
I was wondering whether there is a way to take a given .jar file, selected with a JFileChooser, extract it and put it into a new directory. Then, take all the files from another directory, add it to the directory with the extracted .jar file, and then take all that and package it back up again.
I'm doing this because I want a really easy way to install mods for that game, minecraft, where you can just select your minecraft.jar, and make sure the files for the mod are in a folder, and wait a bit, as indicated by a JProgressBar.
This is all I have so far
import java.io.*;
import java.util.jar.*;
import javax.swing.*;
public class Main extends JFrame {
public Main() {
super("Auto-mod installer");
setSize(300, 60);
setLocationRelativeTo(null);
setResizable(false);
setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
JProgressBar bar = new JProgressBar(0, 100);
add(bar);
setVisible(true);
}
public static void main(String[] args) {
Main m = new Main();
}
private void extract(File f) {
//Hrm...
}
private void addModFiles() {
//Uh...
}
private void repackage(File f) {
//What?
}
}
As you can see, I have no idea what I'm doing. I do know what the imports needed are, but that's about it. Help would be appreciated, ranting about anything I did wrong would get me mad. Thanks!
EDIT: If you know a way to get the same results, and it's not the way that I was looking for, please let me know how to do so. As long as I get the results I was looking for, it would be great. Thanks again!
The idea is relatively simple. You have a few gotchas (like what to do if files already exist and that kind of thing), but otherwise...
I'd start by having a look at JarFile
(I'm in the middle of another example, but when I get time, I'll post some stuff)
UPDATE with Example
public class JarTest {
protected static final String OUTPUT_PATH = "..."; // The place you want to extact the jar to
/**
* #param args the command line arguments
*/
public static void main(String[] args) {
new JarTest();
}
public JarTest() {
try {
unjar();
// Copy new contents in...
jar();
} catch (IOException exp) {
exp.printStackTrace();
}
}
// This just recursivly lists through all the files to be included in the new jar
// We don't care about the directories, as we will create them from the file
// references in the Jar ourselves
protected List<File> getFiles(File path) {
List<File> lstFiles = new ArrayList<File>(25);
// If you want the directories, add the "path" to the list now...
File[] files = path.listFiles();
if (files != null && files.length > 0) {
for (File file : files) {
if (file.isDirectory()) {
lstFiles.addAll(getFiles(file));
} else {
lstFiles.add(file);
}
}
}
return lstFiles;
}
// Re-Jar the contents
// You should always attempt to jar back to a new file, as you may not want to effect the original ;)
public void jar() throws IOException {
JarOutputStream jos = null;
try {
String outputPath = OUTPUT_PATH;
// Create a new JarOutputStream to the file you want to create
jos = new JarOutputStream(new FileOutputStream("...")); // Add your file reference
List<File> fileList = getFiles(new File(OUTPUT_PATH));
System.out.println("Jaring " + fileList.size() + " files");
// Okay, I cheat. I make a list of all the paths already added to the Jar only create
// them when I need to. You could use "file.isDirectory", but that would mean you would need
// to ensure that the files were sorted to allow all the directories to be first
// or make sure that the directory reference is added to the start of each recursion list
List<String> lstPaths = new ArrayList<String>(25);
for (File file : fileList) {
// Replace the Windows file seperator
// We only want the path to this element
String path = file.getParent().replace("\\", "/");
// Get the name of the file
String name = file.getName();
// Remove the output path from the start of the path
path = path.substring(outputPath.length());
// Remove the leading slash if it exists
if (path.startsWith("/")) {
path = path.substring(1);
}
// Add the path path reference to the Jar
// A JarEntry is considered to be a directory if it ends with "/"
if (path.length() > 0) {
// At the trailing path seperator
path += "/";
// Check to see if we've already added it out not
if (!lstPaths.contains(path)) {
// At the path entry...we need need this to make it easier to
// extract the files at a later state. There is a way to cheat,
// but I'll let you figure it out
JarEntry entry = new JarEntry(path);
jos.putNextEntry(entry);
jos.closeEntry();
// Make sure we don't try to add the same path entry again
lstPaths.add(path);
}
}
System.out.println("Adding " + path + name);
// Create the actual entry for this file
JarEntry entry = new JarEntry(path + name);
jos.putNextEntry(entry);
// Write the entry to the file
FileInputStream fis = null;
try {
fis = new FileInputStream(file);
byte[] byteBuffer = new byte[1024];
int bytesRead = -1;
while ((bytesRead = fis.read(byteBuffer)) != -1) {
jos.write(byteBuffer, 0, bytesRead);
}
jos.flush();
} finally {
try {
fis.close();
} catch (Exception e) {
}
}
jos.closeEntry();
}
jos.flush();
} finally {
try {
jos.close();
} catch (Exception e) {
}
}
}
public void unjar() throws IOException {
JarFile jarFile = null;
try {
String outputPath = OUTPUT_PATH;
File outputPathFile = new File(outputPath);
// Make the output directories.
// I'll leave it up to you to decide how best to deal with existing content ;)
outputPathFile.mkdirs();
// Create a new JarFile reference
jarFile = new JarFile(new File("C:/hold/Java_Harmony.jar"));
// Get a list of all the entries
Enumeration<JarEntry> entries = jarFile.entries();
while (entries.hasMoreElements()) {
// Get the next entry
JarEntry entry = entries.nextElement();
// Make a file reference
File path = new File(outputPath + File.separator + entry.getName());
if (entry.isDirectory()) {
// Make the directory structure if we can
if (!path.exists() && !path.mkdirs()) {
throw new IOException("Failed to create output path " + path);
}
} else {
System.out.println("Extracting " + path);
// Extract the file from the Jar and write it to disk
InputStream is = null;
OutputStream os = null;
try {
is = jarFile.getInputStream(entry);
os = new FileOutputStream(path);
byte[] byteBuffer = new byte[1024];
int bytesRead = -1;
while ((bytesRead = is.read(byteBuffer)) != -1) {
os.write(byteBuffer, 0, bytesRead);
}
os.flush();
} finally {
try {
os.close();
} catch (Exception e) {
}
try {
is.close();
} catch (Exception e) {
}
}
}
}
} finally {
try {
jarFile.close();
} catch (Exception e) {
}
}
}
}
You can use this very simple library to pack/unpack jar file
JarManager
Very simple
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import fr.stevecohen.jarmanager.JarPacker;
import fr.stevecohen.jarmanager.JarUnpacker;
public class MyClass {
public void addFileToJar(String jarPath, String otherFilePath) {
try {
JarUnpacker jarUnpacker = new JarUnpacker();
File myJar = new File("./myfile.jar");
File otherFile = new File(otherFilePath);
Path unpackDir = Files.createTempDirectory(myJar.getName()); //create a temp directory to extract your jar
System.out.println("Unpacking in " + unpackDir.toString());
jarUnpacker.unpack(jarPath, unpackDir.toString()); //extraxt all files contained in the jar in temp directory
Files.copy(otherFile.toPath(), new File(unpackDir.toFile(), otherFile.getName()).toPath()); //copy your file
JarPacker jarRepacker = new JarPacker();
File newJar = new File("./maNewFile.jar");
System.out.println("Packing jar in " + newJar.getAbsolutePath());
jarRepacker.pack(unpackDir.toString(), newJar.getAbsolutePath()); //repack the jar with the new files inside
} catch (IOException e) {
e.printStackTrace();
}
}
}
You can also use maven dependency
<dependency>
<groupId>fr.stevecohen.jarmanager</groupId>
<artifactId>JarManager</artifactId>
<version>0.5.0</version>
</dependency>
You also need my repository
<repository>
<id>repo-reapersoon</id>
<name>ReaperSoon's repo</name>
<url>http://repo-maven.stevecohen.fr</url>
</repository>
Check the last version with the link bellow to use the last dependency
Please use my public issue tracker if you find some bugs
File content[] = new File("C:/FilesToGo/").listFiles();
for (int i = 0; i < content.length; i++){
String destiny = "C:/Kingdoms/"+content[i].getName();
File desc = new File(destiny);
try {
Files.copy(content[i].toPath(), desc.toPath(), StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
e.printStackTrace();
}
}
This is what I have. It copies everything just fine.
But among the contents there are some folders. The folders are copied but the folder's contents are not.
Would recommend using FileUtils in Apache Commons IO:
FileUtils.copyDirectory(new File("C:/FilesToGo/"),
new File("C:/Kingdoms/"));
Copies directories & contents.
Recursion. Here is a method the uses rescursion to delete a system of folders:
public void move(File file, File targetFile) {
if(file.isDirectory() && file.listFiles() != null) {
for(File file2 : file.listFiles()) {
move(file2, new File(targetFile.getPath() + "\\" + file.getName());
}
}
try {
Files.copy(file, targetFile.getPath() + "\\" + file.getName(), StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
e.printStackTrace();
}
}
Didn't test the code, but it should work. Basically, it digs down into the folders, telling it to move the item, if its a folder, go through all its children, and move them, etc.
Just to clarify what needs to be changed in Alex Coleman's answer, for the code to work. Here is the modified version of Alex's code that I tested and that works fine for me:
private void copyDirectoryContents(File source, File destination){
try {
String destinationPathString = destination.getPath() + "\\" + source.getName();
Path destinationPath = Paths.get(destinationPathString);
Files.copy(source.toPath(), destinationPath, StandardCopyOption.REPLACE_EXISTING);
}
catch (UnsupportedOperationException e) {
//UnsupportedOperationException
}
catch (DirectoryNotEmptyException e) {
//DirectoryNotEmptyException
}
catch (IOException e) {
//IOException
}
catch (SecurityException e) {
//SecurityException
}
if(source.isDirectory() && source.listFiles() != null){
for(File file : source.listFiles()) {
copyDirectoryContents(file, new File(destination.getPath() + "\\" + source.getName()));
}
}
}
The question is old by now, but I wanted to share my copy methods using java.nio.file.
Copying source directory: src into a container directory: dst.
"Directory" is just a helper class. In this example you can think of it as "Path" container.
We separate the directory structure from the file content.
It's explicit, and easy to imagine. (Also, it avoids some potential Exceptions thrown by the Files.copy() method if you instead copied all files in "one go")
public static void copy(Directory src, Directory dst, boolean replace) throws IOException {
if (src == null || dst == null) throw new IllegalArgumentException("...");
Path sourcePath = src.path();
Path targetPath = dst.path().resolve(sourcePath.getFileName());
copyStructure(sourcePath,targetPath);
copyContent(sourcePath,targetPath,replace);
}
I.e. we want to copy the folder "top" into the "dst" folder "container".
sourcePath = ...some/location/top
targetPath = ...another/location/container/top
copyStructure: Iterates through the source files. If the source file is a directory and the a target file with equivalent name does not exist, we create
the target folder. (So "copy" is not accurate. We "assure" structure)
private static void copyStructure(final Path source, final Path target) throws IOException {
if (source == null || target == null) throw new IllegalArgumentException("...");
final String tarString = target.toString();
final String srcString = source.toString();
Files.walk(source).forEach(new Consumer<Path>() {
#Override
public void accept(Path srcPath) {
if (Files.isDirectory(srcPath)) {
String subString = srcPath.toString().substring(srcString.length());
Path newFolder = Path.of(tarString,subString);
if (!Files.exists(newFolder)) {
try { Files.createDirectory(newFolder);
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
});
}
Now that we know that a target directory structure exists, we iterate the source files again. But now we only copy the "content" (regular files). Choosing whether to replace existing content. copyContent:
private static void copyContent(final Path source, final Path target, boolean replace) throws IOException {
if (source == null || target == null) throw new IllegalArgumentException("...");
final String tarString = target.toString();
final String srcString = source.toString();
Files.walk(source).forEach(new Consumer<Path>() {
#Override
public void accept(Path srcPath) {
if (Files.isRegularFile(srcPath)) {
String subString = srcPath.toString().substring(srcString.length());
Path newFile = Path.of(tarString,subString);
if (!Files.exists(newFile)) {
try { Files.copy(srcPath,newFile);
} catch (IOException e) {
e.printStackTrace();
}
} else {
if (replace) {
try { Files.copy(srcPath,newFile,
StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
});
}