I want to decompress a large folder in ZIP format with nested subdirectories in a directory that already exists. The files inside the ZIP folder can exist in the decompressed directory. I need to keep the previous files only when the date of that file is newer than the date of the file in the ZIP folder. If the file in the ZIP is newer, then I want to overwrite it.
There is some good strategy for doing this? I already checked truezip and zip4j, but I can't find the option (the best option for me so far is modifying the zip4j sources, but it should be a better way.
P.S. If I haven't explained this correctly, please feel free to ask. English is not my native language and I could have expressed anything wrong..
Thanks.
With Zip4j, this is how it can be done:
import java.io.File;
import java.util.Date;
import java.util.List;
import net.lingala.zip4j.core.ZipFile;
import net.lingala.zip4j.model.FileHeader;
import net.lingala.zip4j.util.Zip4jUtil;
public class ExtractWithoutOverwriting {
public static void main(String[] args) {
try {
String outputPath = "yourOutputPath";
ZipFile zipFile = new ZipFile(new File("yourZipFile.zip"));
if (zipFile.isEncrypted()) {
zipFile.setPassword("yourPassword".toCharArray());
}
#SuppressWarnings("unchecked")
List<FileHeader> fileHeaders = zipFile.getFileHeaders();
for (FileHeader fileHeader : fileHeaders) {
if (fileHeader.isDirectory()) {
File file = new File(outputPath + System.getProperty("file.separator") + fileHeader.getFileName());
file.mkdirs();
} else {
if (canWrite(outputPath, fileHeader)) {
System.out.println("Writing file: " + fileHeader.getFileName());
zipFile.extractFile(fileHeader, outputPath);
} else {
System.out.println("Not writing file: " + fileHeader.getFileName());
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
private static boolean canWrite(String outputPath, FileHeader fileHeader) {
File file = new File(outputPath + System.getProperty("file.separator") + fileHeader.getFileName());
//time stamps are stored in dos format in a zip file
//convert it to java format
long lastModifiedFromZip = Zip4jUtil.dosToJavaTme(fileHeader.getLastModFileTime());
//If the file exists, it can be overwritten only if the file in the destination path
//is newer than the one in the zip file
return !(file.exists() && isLastModifiedDateFromFileNewer(file.lastModified(), lastModifiedFromZip));
}
public static boolean isLastModifiedDateFromFileNewer(long lastModifiedFromFile, long lastModifiedFromZip) {
Date lastModifiedDateFromFile = new Date(lastModifiedFromFile);
Date lastModifiedDateFromZip = new Date(lastModifiedFromZip);
return lastModifiedDateFromFile.after(lastModifiedDateFromZip);
}
}
What we do here is:
Create a new instance of the ZipFile
If the zip file is encrypted, set a password
Loop over all files in the zip file
Check if a file with this name exists in the output path and if this file's last modification time is "newer" than the one in the zip file. This check is done in the method: canWrite()
This code is not completely tested, but I hope it gives you an idea of a solution.
Related
I have below code where i am reading the file from particular directory, processing it and once processed i am moving the file to archive directory. This is working fine. I am receiving new file everyday and i am using Control-M scheduler job to run this process.
Now in next run i am reading the new file from that particularly directory again and checking this file with the file in the archive directory and if the content is different then only process the file else dont do anything. There is shell script written to do this job and we dont see any log for this process.
Now i want to produce log message in my java code if the files are identical from the particular directory and in the archive directory then generate log that 'files are identical'. But i dont know exactly how to do this. I dont want to write the the logic to process or move anything in the file ..i just need to check the files are equal and if it is then
produce log message. The file which i recieve are not very big and the max size can be till 10MB.
Below is my code:
for(Path inputFile : pathsToProcess) {
// read in the file:
readFile(inputFile.toAbsolutePath().toString());
// move the file away into the archive:
Path archiveDir = Paths.get(applicationContext.getEnvironment().getProperty(".archive.dir"));
Files.move(inputFile, archiveDir.resolve(inputFile.getFileName()),StandardCopyOption.REPLACE_EXISTING);
}
return true;
}
private void readFile(String inputFile) throws IOException, FileNotFoundException {
log.info("Import " + inputFile);
try (InputStream is = new FileInputStream(inputFile);
Reader underlyingReader = inputFile.endsWith("gz")
? new InputStreamReader(new GZIPInputStream(is), DEFAULT_CHARSET)
: new InputStreamReader(is, DEFAULT_CHARSET);
BufferedReader reader = new BufferedReader(underlyingReader)) {
if (isPxFile(inputFile)) {
Importer.processField(reader, tablenameFromFilename(inputFile));
} else {
Importer.processFile(reader, tablenameFromFilename(inputFile));
}
}
log.info("Import Complete");
}
}
Based on the limited information about the size of file or performance needs, something like this can be done. This may not be 100% optimized, but just an example. You may also have to do some exception handling in the main method, since the new method might throw an IOException:
import org.apache.commons.io.FileUtils; // Add this import statement at the top
// Moved this statement outside the for loop, as it seems there is no need to fetch the archive directory path multiple times.
Path archiveDir = Paths.get(applicationContext.getEnvironment().getProperty("betl..archive.dir"));
for(Path inputFile : pathsToProcess) {
// Added this code
if(checkIfFileMatches(inputFile, archiveDir); {
// Add the logger here.
}
//Added the else condition, so that if the files do not match, only then you read, process in DB and move the file over to the archive.
else {
// read in the file:
readFile(inputFile.toAbsolutePath().toString());
Files.move(inputFile, archiveDir.resolve(inputFile.getFileName()),StandardCopyOption.REPLACE_EXISTING);
}
}
//Added this method to check if the source file and the target file contents are same.
// This will need an import of the FileUtils class. You may change the approach to use any other utility file, or read the data byte by byte and compare. If the files are very large, probably better to use Buffered file reader.
private boolean checkIfFileMatches(Path sourceFilePath, Path targetDirectoryPath) throws IOException {
if (sourceFilePath != null) { // may not need this check
File sourceFile = sourceFilePath.toFile();
String fileName = sourceFile.getName();
File targetFile = new File(targetDirectoryPath + "/" + fileName);
if (targetFile.exists()) {
return FileUtils.contentEquals(sourceFile, targetFile);
}
}
return false;
}
I have a Java program which searches for a folder with the date of yesterday and compresses it to a 7zip file and deletes it at the end. Now I have noticed that the generated 7zip archive files by my program are way too big. When I use a program like 7-Zip File Manager to compress my files it generates an archive which is 5 kb big while my program generates an archive which is 737 kb big for the same files (which have a 873 kb size). Now I am afraid that my program does not compress it to a 7zip file but do a usual zip file. Is there a way to change something in my code so that it generates a smaller 7zip file like 7-Zip File Manager would do it?
package SevenZip;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.concurrent.TimeUnit;
import org.apache.commons.compress.archivers.sevenz.SevenZArchiveEntry;
import org.apache.commons.compress.archivers.sevenz.SevenZOutputFile;
public class SevenZipUtils {
public static void main(String[] args) throws InterruptedException, IOException {
String sourceFolder = "C:/Users/Ferid/Documents/Dates/";
String outputZipFile = "/Users/Ferid/Documents/Dates";
int sleepTime = 0;
compress(sleepTime, outputZipFile, sourceFolder);
}
public static boolean deleteDirectory(File directory, int sleepTime) throws InterruptedException {
if (directory.exists()) {
File[] files = directory.listFiles();
if (null != files) {
for (int i = 0; i < files.length; i++) {
if (files[i].isDirectory()) {
deleteDirectory(files[i], sleepTime);
System.out.println("Folder deleted: " + files[i]);
} else {
files[i].delete();
System.out.println("File deleted: " + files[i]);
}
}
}
}
TimeUnit.SECONDS.sleep(sleepTime);
return (directory.delete());
}
public static void compress(int sleepTime, String outputZipFile, String sourceFolder)
throws IOException, InterruptedException {
// finds folder of yesterdays date
final Calendar cal = Calendar.getInstance();
cal.add(Calendar.DATE, -1); // date of yesterday
String timeStamp = new SimpleDateFormat("yyyyMMdd").format(cal.getTime()); // format the date
System.out.println("Yesterday was " + timeStamp);
if (sourceFolder.endsWith("/")) { // add yesterday folder to sourcefolder path
sourceFolder = sourceFolder + timeStamp;
} else {
sourceFolder = sourceFolder + "/" + timeStamp;
}
if (outputZipFile.endsWith("/")) { // add yesterday folder name to outputZipFile path
outputZipFile = outputZipFile + " " + timeStamp + ".7z";
} else {
outputZipFile = outputZipFile + "/" + timeStamp + ".7z";
}
File file = new File(sourceFolder);
if (file.exists()) {
try (SevenZOutputFile out = new SevenZOutputFile(new File(outputZipFile))) {
addToArchiveCompression(out, file, ".");
System.out.println("Files sucessfully compressed");
deleteDirectory(new File(sourceFolder), sleepTime);
}
} else {
System.out.println("Folder does not exist");
}
}
private static void addToArchiveCompression(SevenZOutputFile out, File file, String dir) throws IOException {
String name = dir + File.separator + file.getName();
if (file.isFile()) {
SevenZArchiveEntry entry = out.createArchiveEntry(file, name);
out.putArchiveEntry(entry);
FileInputStream in = new FileInputStream(file);
byte[] b = new byte[1024];
int count = 0;
while ((count = in.read(b)) > 0) {
out.write(b, 0, count);
}
out.closeArchiveEntry();
in.close();
System.out.println("File added: " + file.getName());
} else if (file.isDirectory()) {
File[] children = file.listFiles();
if (children != null) {
for (File child : children) {
addToArchiveCompression(out, child, name);
}
}
System.out.println("Directory added: " + file.getName());
} else {
System.out.println(file.getName() + " is not supported");
}
}
}
I am using the Apache Commons Compress library
EDIT: Here is a link where I have some of the Apache Commons Compress code from.
Commons Compress is starting a new block in the container file for each archive entry. Note the block counter here:
Not quite the answer you were hoping for, but the docs say it doesn't support "solid compression" - writing several files to a single block. See paragraph 5 in the docs here.
A quick look around found a few other Java libraries that support LZMA compression, but I couldn't spot one that could do so within the parent container file format for 7-Zip. Perhaps someone else knows of an alternative...
It sounds like a normal zip file format (e.g. via ZipOutputStream) is not an option?
Use 7-Zip file archiver instead, it compresses 832 KB file to 26.0 KB easily:
Get its Jar and SDK.
Choose LZMA Compression .java related files.
Add Run arguments to project properties: e "D:\\2017ASP.pdf" "D:\\2017ASP.7z", e stands for encode, "input path" "output path".
Run the project [LzmaAlone.java].
Results
Case1 (.pdf file ):
From 33,969 KB to 24,645 KB.
Case2 (.docx file ):
From 832 KB to 26.0 KB.
I don't have enough rep to comment anymore so here are my thoughts:
I don't see where you set the compression ratio so it could be that SevenZOutputFile uses no (or very low) compression. As #CristiFati said, the difference in compression is odd, especially for text files
As noted by #df778899, there is no support for solid compression, which is how the best compression ratio is achieved, so you won't be able to do as well as the 7z command line
That said, if zip really isn't an option, your last resort could be to call the proper command line directly within your program.
If pure 7z is not mandatory, another option would be to use a "tgz"-like format to emulate solid compression: first compress all files to a non-compressed file (e.g. tar format, or zip file with no compression), then compress that single file in zip mode with standard Java Deflate algorithm. Of course that will be viable only if that format is recognized by further processes using it.
I am new in Java and I have a question regarding the method readAlllines for the class Files. The file "Testfile.txt" is saved in the same directory as my Java class changeFiles. I want to read the lines out of it.
Here is my example code:
import java.io.*;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
public class changeFiles {
public static void main(String[] args) {
File temp =new File("Testfile.txt");
Path p = temp.toPath();
try{
List<String> zeilen = Files.readAllLines(p);
for(String line : zeilen){
System.out.println(line);
}
} catch (IOException e) {
System.out.println(e);
}
}
}
Unfortunately, the method can't find the file. How do I get the correct path to my file in readAllLines?
You're trying to get file from working directory, check yours printing this in some way
System.getProperty("user.dir")
Place "Testfile.txt" there, run and enjoy.
Another solution will be put folder when reading file using File(folder, file) constructor:
// imagine your file is placed in: c:\tmp\Testfile.txt
final String folder = "C:\\tmp\\";
File temp = new File(folder, "Testfile.txt");
Or maybe merge both:
final String folder = System.getProperty("user.dir");
File temp = new File(folder, "Testfile.txt");
Java class location is not the same as current directory.
For example current directory is something like:
C:\Users\userName\project (This is where txt file shoud be)
And java class is something like C:\Users\userName\project\src\packageName\Java.java
to find out what the current directory is you can run: System.getProperty("user.dir")
I am trying to copy files, folders, sub folders, zip files etc from a given location to another location. I used the code below.
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
public class CopyDirectoryExample
{
public static void main(String[] args)
{
File srcFolder = new File("C:\\Users\\Yohan\\Documents");
File destFolder = new File("D:\\Test");
//make sure source exists
if(!srcFolder.exists()){
System.out.println("Directory does not exist.");
//just exit
System.exit(0);
}else{
try{
copyFolder(srcFolder,destFolder);
}catch(IOException e){
e.printStackTrace();
//error, just exit
System.exit(0);
}
}
System.out.println("Done");
}
public static void copyFolder(File src, File dest)
throws IOException{
if(src.isDirectory()){
//if directory not exists, create it
if(!dest.exists()){
dest.mkdir();
System.out.println("Directory copied from "
+ src + " to " + dest);
}
//list all the directory contents
String files[] = src.list();
for (String file : files) {
//construct the src and dest file structure
File srcFile = new File(src, file);
File destFile = new File(dest, file);
//recursive copy
copyFolder(srcFile,destFile);
}
}else{
//if file, then copy it
//Use bytes stream to support all file types
InputStream in = new FileInputStream(src);
OutputStream out = new FileOutputStream(dest);
byte[] buffer = new byte[1024];
int length;
//copy the file content in bytes
while ((length = in.read(buffer)) > 0){
out.write(buffer, 0, length);
}
in.close();
out.close();
System.out.println("File copied from " + src + " to " + dest);
}
}
}
Now, I used the above code to take a copy of "My Documents". But unfortunatly, it ended up with NullPointerException after running for a while.
The reason for the error is it tried to take a copy of "My Music" folder, which is not even inside of the "My Documents" folder. I tested this code in 2 different machines running windows 7, got the same error in both.
A windows specific solution is fine for me, as I am targeting windows machines at the moment. What have I done wrong?
The error I am getting is below
Directory copied from C:\Users\Yohan\Documents\My Music to D:\Test\My Music
Exception in thread "main" java.lang.NullPointerException
at CopyDirectoryExample.copyFolder(CopyDirectoryExample.java:51)
at CopyDirectoryExample.copyFolder(CopyDirectoryExample.java:56)
at CopyDirectoryExample.main(CopyDirectoryExample.java:25)
The reason this isn't working is because "My Music", "My Pictures" (or Images) and other directories are just symbolic links. See this post on how to detect symbolic links: Java 1.6 - determine symbolic links
Unfortunately, these folders (Images, Music, Videos) are NOT considered symbolic links in Java. Using Java 8,
Files.isSymbolicLink(srcFile.toPath())
While return false, and Files.readSymbolicLink(srcFile.toPath()) will fail with an Access Denied Exception.
So you can't process them automatically. Fix your code so that you handle properly the case where srcFile.isDirectory() returns true, but srcFile.listFiles() return null.
On my Windows 8 machine, three folders were in that case. I'm on a French machine, so I got a "Ma Musique" folder that gave null for listFiles. However,
new File("C:\\Users\\<user>\\Music").listFiles()
Does NOT return null. So I'm afraid you'll have to hardcode special code for the three folders (Music, Videos, Images) if you want to copy the data too.
You are not handling the empty directories -- try making the following change,
It will work after making the below change.
//list all the directory contents
String files[] = src.list();
if (files!=null && files.length>0) {
for (String file : files) {
//construct the src and dest file structure
File srcFile = new File(src, file);
File destFile = new File(dest, file);
//recursive copy
copyFolder(srcFile,destFile);
}
}
I want to know the list of files under the 'META-INF/config' directory in a JAR file.
I am using the below code to retrieve the files list. But it is failing.
Enumeration<URL> path = Thread.currentThread().getContextClassLoader().getResources("META-INF/config/");
while(path.hasMoreElements())
{
URL path1 = path.nextElement();
System.out.println("File =" +path1.getFile());
File configFolder = new File(path1.getPath());
File[] files = configFolder.listFiles();
for (File file : files)
{
System.out.println("\nFile Name =" + file.getName());
}
}
Can somebody help me in fixing this?
Thanks In Advance,
Maviswa
try below code
import java.util.*;
import java.util.jar.*;
import java.util.zip.*;
import java.io.*;
public class JarContents{
public static void main(String[] args) throws IOException{
JarContents mc = new JarContents();
}
public JarContents() throws IOException{
BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
System.out.print("Enter jar file name: ");
String filename = in.readLine();
File file = new File(filename);
if(!filename.endsWith(".jar")){
System.out.println("Invalid file name!");
System.exit(0);
}
else if(!file.exists()){
System.out.println("File not exist!");
System.exit(0);
}
try{
JarFile jarfile = new JarFile(filename);
Enumeration em = jarfile.entries();
for (Enumeration em1 = jarfile.entries(); em1.hasMoreElements();) {
System.out.println(em1.nextElement());
}
}
catch(ZipException ze){
System.out.println(ze.getMessage());
System.exit(0);
}
}
}
Good Luck!!!
I remember having to do this a while back to read in a jar's manifest.mf to extract its version information to display. Given that all properly built jars have manifests, trying to access them as a resource is impossible (they all have the same path), and as such, had to examine the jar individually as a zip file.
Given that you aren't providing information as to where the failure is, it is difficult to guess as to what your problem is. I'm not sure if it is not finding the file that you are expecting, or if it is reading the wrong file, or if you are getting NPEs, etc.
try adding a "/" or "./" before the META-INF in the getResources() call
e.g.
...
read.currentThread().getContextClassLoader().getResources("./META-INF/config/");