Is there a way to do parallel zipping in Java?
I'm currently using ParallelScatterZipCreator but unfortunately it does parallel zipping per file. So if there's a single file that is much larger than other files, the parallel zipping only happens for smaller files. Then it has to wait until the large file is zipped serialy.
Is there a better library out there that utilizes all CPU cores even if we're zipping a single file?
TL;DR: You may not need compression at all. If you do, then you probably don't want to use the zip format, it's outdated tech with sizable downsides, and clearly you have some fairly specific needs. You probably want ZStandard (zstd).
How does compression work
Compression works by looking at a blob of bytes and finding repetitions of some form within it. Thus, it is not possible to compress a single byte.
This makes the job fundamentally at odds with parallelising: If you take a blob of 1 million bytes and compress them by lopping that up into 10 chunks of 100k bytes each, compressing each miniblob individually, then any repetition such that one of the reptitions was in one miniblob and another was in another, means that you have missed an opportunity to compress data, that you would not have missed if you had compressed this data in one blob instead.
The only reason ZIP does let you parallellize a little bit, is because it is an old format - sensible at the time, but in this age, just about every part of the ZIP format is crap.
Why is ZIP bad?
ZIP is a mixed bag, conflating two unrelated jobs.
A bundler. A bundling tool is some software that takes a bunch of files and turns that into a single stream (a single sack of bytes). To do so, the bundling tool will take the metadata about a file (its name, its owner/group or other access info, its last-modified time, etcetera), and the data within it, and serializes this into a single stream. zip does this, as does e.g. the posix tar tool.
A compressor. A compressor takes a stream of data and compresses it by finding repeated patterns.
zip in essence is only #1, but as part of the bundler, the part with 'the data within this file' has a flag to indicate that a compression algorithm has been applied to the bytes representing the data. In theory you can use just about any algorithm, but in practice, every zip file has all entries compressed with the DEFLATE algorithm, which is vastly inferior to more modern algorithms.
.tar.gz is the exact same technology (first bundle it: tar file, then gzip that tar file. gzip is the DEFLATE algorithm), but vastly more efficient in certain cases (it applies compression to the entire stream vs. restarting from scratch for every file. If you take a sack of 1000 small, similar files, then that in .tar.gz form is orders of magnitude smaller than that in .zip form).
Also, zip is old, and it made choices defensible at the time but silly in modern systems: You can't 'stream' zips (you can't meaningfully start unpacking one until the whole file has been received), because the bundler's info is at the end of the file.
So why can I parallellize zips?
Because zips 'restart' their compression window on every file. This is inefficient and hurts the compression ratio of zip files.
You can apply the exact same principle to any block of data, if you want. Trade compression efficiency for paralellizability. ZIP is the format that doesn't do it in a useful way; as you said, if you have one much larger file, the point is moot.
'restart window at' is a principle that can be generalized, and various compression formats support it in a much more useful fashion (restart at every X bytes, vs. ZIPs unreliable 'restart at every file').
What is the bottleneck?
Multiple aspects are involved when sending data: The speed at which the source provides the bytes you want to send, the speed at which the bytes are processed into a package that can then be sent (e.g. a zip tool, but can be anything, including just sending it verbatim, uncompressed), the speed at which the packaged-up bytes are transited to the target system, the speed at which the target can unpack it, and the speed at which the target can then process the unpacked result.
Are you sure that the compression aspect is the bottleneck?
In the base case where you read the bytes off of a harddisk, zip them up, send them across a residential internet pipe to another system, that system unzips, and saves them on a HDD, it is rather likely that the bottleneck is the network. Parallellizing the compression step is a complete waste and in fact only slows things down by reducing compression ratios.
If you're reading files off of a spinning platter, then the speed of the source is likely the bottleneck, and parallel processing considerably slows things down: You're now asking the read head to bounce around, and this is much slower than reading the data sequentially in one go.
If you have a fast source, and a fast pipe, then the bottleneck is doubtlessly the compression and uncompression, but the solution is then not to compress at all: If you are transferring data off of SSDs or from a USB3-connected byte-spewing sensor and transfer it across a 10M CAT6 cable from one gigabit ethernet port to another, then why compress at all? Just send those bytes. Compression isn't going to make it any faster, and as long as you don't saturate that 1Gb connection, you gain absolutely nothing whatsoever by attempting to compress it.
If you have a slow pipe, then the only way to make things faster is to compress as much as you can. Which most definitely involves not using the DEFLATE algorithm (e.g. don't use zip). Use another algorithm and configure it to get better compression rates, at the cost of CPU performance. Parallelising is irrelevant; it's not the bottleneck, so there is no point whatsoever in doing it.
Conclusions
Most likely you want to either send your files over uncompressed, or ZStandard your files over, tweaking the compression v. speed ratio as needed. I'm not aware of any ZStandard (zstd) impl for java itself, but the zstd-jni project gives you a java-based API for invoking the C zstd library.
If you insist on sticking with ZIP, the answer is a fairly basic 'nope, you cannot really do that', though you could in theory write a parallel ZIP compressor that has worse compression power but parallelizes better (by restarting the window within a single file for larger files in addition to the forced-upon-you-by-the-format restart at every file), and produces ZIP files that are still compatible with just about every unzip tool on the planet. I'm not aware of one, I don't think one exists, and writing one yourself would be a decidedly non-trivial exercise.
I need to compress PNG images on PC for data transmission in Python, transfer it to mobile and read it there in Java. I need lossless compression.
The hard part is compression and decompression in different environments and programming languages. I need to use something available for both languages. I've tried zlib, which technically should work, but it only decreases size by about 0.001% (with "best" compression setting 9).
What am I doing wrong with zlib, if anything?
What are the possible alternatives?
Is there any other way to go about this problem? I just need to transfer data as byte stream and the compression was my first thought to optimizing it.
compressing a a compressed file (like png / jpg) will normally not yield a lot and can even occasionally increase the file size.
It's not worth the effort.
Our application reads and writes lots of encrypted files and we have created classes that implement InputStream and OutputStream to handle all these details. These implementation work fine (and are very convenient to use, by the way).
To be a bit more specific, when we write (in our OutputStream implementation), we utilize the Bouncy Castle (version 1.53) CMS classes and we compress, sign, encrypt, and sign again.
The problem we have is that we occasionally write large (>1GB after compression, 10+GB before compression) files and these can take 1+ hours to write. During this process, one CPU is pegged; the bottleneck is the compression step (based on profiling). It is not an option to not compress - we get good compression (better than 10x), these files are transported across a network (so much smaller is much faster), and we pay for storing these files essentially forever, so much smaller is cheaper.
The servers that we typically run this code on are AWS EC2 c5.4xlarge instances, which have lots of memory and 16 CPUs.
Here is the code for where we set up the compression:
import org.bouncycastle.cms.CMSCompressedDataStreamGenerator;
CMSCompressedDataStreamGenerator compressedGenerator = new CMSCompressedDataStreamGenerator();
compressedStream = compressedGenerator.open(signedStream, new ZlibCompressor());
We have run into similar situation where we GZIP (rather than encrypt) large files. We have successfully worked around this performance bottleneck by using a parallel GZIP writer. With the parallel GZIP writer, we can effectively utilize all available CPUs and complete the compression in a fraction of the time compared to using the JDK default implementation.
The question:
Is there a way to configure/invoke Bouncy Castle to parallelize the ZlibCompressor?
Is there a different Bouncy Castle compressor that we can/should use that will provide equivalent compression and significantly improve throughput?
I am exploring if i can change compression level of compression.
I think the level shouldn't affect compatibility, i.e, when decompressing, deflator can handle both best and default compression level, but wanted to confirm there is no gotchas.
I am aware that there is additional processing time needed to do best compression, but is the same true when decompressing?
thanks
No. Decompression is a smidge faster for more highly compressed data, simply because there is less compressed data to process.
I'm trying to compare the output of running strings through GZIP on both Java, and the CLI gzip command. The outputs are not the same, and I've figure out why, but not sure how to get them to jive with each other.
I've read a number of questions on SO, have read the man pages for gzip, and the code for both GZIPOutputStream and DeflaterOutputStream. The default compression level for GZIPOutputStream (set through Deflator) is "-1", and there's little explanation as to what that means. Furthermore, gzip CLI only allows for values between 1 and 9, inclusive.
So is there a way I can change the compression settings in either Java or the gzip command to make them produce the same output?
No. Java is using the zlib deflator, which is derived from but not exactly the same as the older gzip command-line utility deflator. They will generally not produce the same output and there are no settings to coerce them to do so.
Compression level -1 requests the default compression level, which in the current zlib implementation is level 6.
I would have to ask why you care to get their outputs to be the same. All that matters is that the compression is lossless, i.e., that both the gzip and Java compressed streams produce the same original data when decompressed. There is no requirement that, for example, different versions of zlib produce the same output at the same compression level.