Is there a way to do parallel zipping in Java?
I'm currently using ParallelScatterZipCreator but unfortunately it does parallel zipping per file. So if there's a single file that is much larger than other files, the parallel zipping only happens for smaller files. Then it has to wait until the large file is zipped serialy.
Is there a better library out there that utilizes all CPU cores even if we're zipping a single file?
TL;DR: You may not need compression at all. If you do, then you probably don't want to use the zip format, it's outdated tech with sizable downsides, and clearly you have some fairly specific needs. You probably want ZStandard (zstd).
How does compression work
Compression works by looking at a blob of bytes and finding repetitions of some form within it. Thus, it is not possible to compress a single byte.
This makes the job fundamentally at odds with parallelising: If you take a blob of 1 million bytes and compress them by lopping that up into 10 chunks of 100k bytes each, compressing each miniblob individually, then any repetition such that one of the reptitions was in one miniblob and another was in another, means that you have missed an opportunity to compress data, that you would not have missed if you had compressed this data in one blob instead.
The only reason ZIP does let you parallellize a little bit, is because it is an old format - sensible at the time, but in this age, just about every part of the ZIP format is crap.
Why is ZIP bad?
ZIP is a mixed bag, conflating two unrelated jobs.
A bundler. A bundling tool is some software that takes a bunch of files and turns that into a single stream (a single sack of bytes). To do so, the bundling tool will take the metadata about a file (its name, its owner/group or other access info, its last-modified time, etcetera), and the data within it, and serializes this into a single stream. zip does this, as does e.g. the posix tar tool.
A compressor. A compressor takes a stream of data and compresses it by finding repeated patterns.
zip in essence is only #1, but as part of the bundler, the part with 'the data within this file' has a flag to indicate that a compression algorithm has been applied to the bytes representing the data. In theory you can use just about any algorithm, but in practice, every zip file has all entries compressed with the DEFLATE algorithm, which is vastly inferior to more modern algorithms.
.tar.gz is the exact same technology (first bundle it: tar file, then gzip that tar file. gzip is the DEFLATE algorithm), but vastly more efficient in certain cases (it applies compression to the entire stream vs. restarting from scratch for every file. If you take a sack of 1000 small, similar files, then that in .tar.gz form is orders of magnitude smaller than that in .zip form).
Also, zip is old, and it made choices defensible at the time but silly in modern systems: You can't 'stream' zips (you can't meaningfully start unpacking one until the whole file has been received), because the bundler's info is at the end of the file.
So why can I parallellize zips?
Because zips 'restart' their compression window on every file. This is inefficient and hurts the compression ratio of zip files.
You can apply the exact same principle to any block of data, if you want. Trade compression efficiency for paralellizability. ZIP is the format that doesn't do it in a useful way; as you said, if you have one much larger file, the point is moot.
'restart window at' is a principle that can be generalized, and various compression formats support it in a much more useful fashion (restart at every X bytes, vs. ZIPs unreliable 'restart at every file').
What is the bottleneck?
Multiple aspects are involved when sending data: The speed at which the source provides the bytes you want to send, the speed at which the bytes are processed into a package that can then be sent (e.g. a zip tool, but can be anything, including just sending it verbatim, uncompressed), the speed at which the packaged-up bytes are transited to the target system, the speed at which the target can unpack it, and the speed at which the target can then process the unpacked result.
Are you sure that the compression aspect is the bottleneck?
In the base case where you read the bytes off of a harddisk, zip them up, send them across a residential internet pipe to another system, that system unzips, and saves them on a HDD, it is rather likely that the bottleneck is the network. Parallellizing the compression step is a complete waste and in fact only slows things down by reducing compression ratios.
If you're reading files off of a spinning platter, then the speed of the source is likely the bottleneck, and parallel processing considerably slows things down: You're now asking the read head to bounce around, and this is much slower than reading the data sequentially in one go.
If you have a fast source, and a fast pipe, then the bottleneck is doubtlessly the compression and uncompression, but the solution is then not to compress at all: If you are transferring data off of SSDs or from a USB3-connected byte-spewing sensor and transfer it across a 10M CAT6 cable from one gigabit ethernet port to another, then why compress at all? Just send those bytes. Compression isn't going to make it any faster, and as long as you don't saturate that 1Gb connection, you gain absolutely nothing whatsoever by attempting to compress it.
If you have a slow pipe, then the only way to make things faster is to compress as much as you can. Which most definitely involves not using the DEFLATE algorithm (e.g. don't use zip). Use another algorithm and configure it to get better compression rates, at the cost of CPU performance. Parallelising is irrelevant; it's not the bottleneck, so there is no point whatsoever in doing it.
Conclusions
Most likely you want to either send your files over uncompressed, or ZStandard your files over, tweaking the compression v. speed ratio as needed. I'm not aware of any ZStandard (zstd) impl for java itself, but the zstd-jni project gives you a java-based API for invoking the C zstd library.
If you insist on sticking with ZIP, the answer is a fairly basic 'nope, you cannot really do that', though you could in theory write a parallel ZIP compressor that has worse compression power but parallelizes better (by restarting the window within a single file for larger files in addition to the forced-upon-you-by-the-format restart at every file), and produces ZIP files that are still compatible with just about every unzip tool on the planet. I'm not aware of one, I don't think one exists, and writing one yourself would be a decidedly non-trivial exercise.
I need to compress PNG images on PC for data transmission in Python, transfer it to mobile and read it there in Java. I need lossless compression.
The hard part is compression and decompression in different environments and programming languages. I need to use something available for both languages. I've tried zlib, which technically should work, but it only decreases size by about 0.001% (with "best" compression setting 9).
What am I doing wrong with zlib, if anything?
What are the possible alternatives?
Is there any other way to go about this problem? I just need to transfer data as byte stream and the compression was my first thought to optimizing it.
compressing a a compressed file (like png / jpg) will normally not yield a lot and can even occasionally increase the file size.
It's not worth the effort.
I am exploring if i can change compression level of compression.
I think the level shouldn't affect compatibility, i.e, when decompressing, deflator can handle both best and default compression level, but wanted to confirm there is no gotchas.
I am aware that there is additional processing time needed to do best compression, but is the same true when decompressing?
thanks
No. Decompression is a smidge faster for more highly compressed data, simply because there is less compressed data to process.
I wrote a backup program using Deflater and SHA-1 to store files and a hash value. I see that Java's Deflater uses zlib. If I explicit set the Deflater's level, could I expect to always get the same series of bytes regardless of platform and JRE version?
If not then what do I use? Are there any stable and fast pure-Java implementations?
Do the SHA-1 before compression. Then you verify the correctness of the compression and decompression as well.
There is no assurance that what a compressor produces today will be the same as what a later version of the compressor produces tomorrow for the same input. And there should be no such assurance, since that would preclude improvements in compression.
The only assurance is that the compression-decompression process is lossless, so that what you get from the decompressor is exactly what you fed the compressor. For that reason, you need to compute signatures on the input of the compressor and the output of the decompressor. Ignore the intermediate compressed stream.
I am compressing files of over 2GB in Java using a consecutive application of two compression algorithms; one LZ based and one Huffman-based. (This is similar to DEFLATE).
Since 2GB is too large to be held in any buffer, I have to pass the file through one algorithm outputting a temporary file, then pass that temporary file through the second algorithm outputting the final file.
An alternative is to compress the file in 8MB blocks (the size where I don't get an Out-Of-Memory error) but then I have an inability to take full advantage of the redundancy within the entire file.
Any ideas how to perform these operations neater. No temporary files, and no compressing in blocks? Do any other compression tools compress in blocks? How do they deal with this issue? Regards
Java comes with “java.util.zip” library to perform data compression in ZIp format.
The overall concept is quite straightforward.
Library reads file with “FileInputStream”.
And add the file name to “ZipEntry” and output it to “ZipOutputStream“
import java.util.zip.ZipEntry and import java.util.zip.ZipOutputStream are used for importing Zip folder to a program.
But how can decompress a file
?
What's wrong with piping of streams? You can read from InputStream, compress the bytes and write them to output stream that is connected to input stream of the next algorithm. Take a look on PipeInputStream and PipeOutputStream.
I hope that these algorithms can work incrementally.
You could use two levels of java.util.zip. First, just concatenate all files (without compression). If possible, sort the entries by file type so that similar files are next to each other (this will increase compression ratio). Second, compress this stream. You don't need to run two separate phases; instead, you can wrap the first within the second stage, like CompressStream(ConcatenateFiles(directory)). That way you have a zip file within another zip file: the outer zip file is compressed, the inner is not and contains all the actual files.
It is true that java.util.zip used to have problems with files larger than 2 GB (I did run into those problems). However, I believe that was only the case for ZipFile and not for ZipIn/OutputStream. Also, I think those problems are fixed with recent Java versions.
Buffer size: regular compression algorithms such as Deflate will not benefit from chunk sizes larger than about 64 KB. More advanced algorithms can benefit from using larger chunk sizes, for example bzip2 up to 900 KB, or LZMA2 up to 2 MB. Anything beyond that is more likely the domain of data deduplication, which might or might not make sense for what you want to do.