Develop Reference

Parallel zipping of a single large file

Is there a way to do parallel zipping in Java?
I'm currently using ParallelScatterZipCreator but unfortunately it does parallel zipping per file. So if there's a single file that is much larger than other files, the parallel zipping only happens for smaller files. Then it has to wait until the large file is zipped serialy.
Is there a better library out there that utilizes all CPU cores even if we're zipping a single file?

TL;DR: You may not need compression at all. If you do, then you probably don't want to use the zip format, it's outdated tech with sizable downsides, and clearly you have some fairly specific needs. You probably want ZStandard (zstd).
How does compression work
Compression works by looking at a blob of bytes and finding repetitions of some form within it. Thus, it is not possible to compress a single byte.
This makes the job fundamentally at odds with parallelising: If you take a blob of 1 million bytes and compress them by lopping that up into 10 chunks of 100k bytes each, compressing each miniblob individually, then any repetition such that one of the reptitions was in one miniblob and another was in another, means that you have missed an opportunity to compress data, that you would not have missed if you had compressed this data in one blob instead.
The only reason ZIP does let you parallellize a little bit, is because it is an old format - sensible at the time, but in this age, just about every part of the ZIP format is crap.
Why is ZIP bad?
ZIP is a mixed bag, conflating two unrelated jobs.
A bundler. A bundling tool is some software that takes a bunch of files and turns that into a single stream (a single sack of bytes). To do so, the bundling tool will take the metadata about a file (its name, its owner/group or other access info, its last-modified time, etcetera), and the data within it, and serializes this into a single stream. zip does this, as does e.g. the posix tar tool.
A compressor. A compressor takes a stream of data and compresses it by finding repeated patterns.
zip in essence is only #1, but as part of the bundler, the part with 'the data within this file' has a flag to indicate that a compression algorithm has been applied to the bytes representing the data. In theory you can use just about any algorithm, but in practice, every zip file has all entries compressed with the DEFLATE algorithm, which is vastly inferior to more modern algorithms.
.tar.gz is the exact same technology (first bundle it: tar file, then gzip that tar file. gzip is the DEFLATE algorithm), but vastly more efficient in certain cases (it applies compression to the entire stream vs. restarting from scratch for every file. If you take a sack of 1000 small, similar files, then that in .tar.gz form is orders of magnitude smaller than that in .zip form).
Also, zip is old, and it made choices defensible at the time but silly in modern systems: You can't 'stream' zips (you can't meaningfully start unpacking one until the whole file has been received), because the bundler's info is at the end of the file.
So why can I parallellize zips?
Because zips 'restart' their compression window on every file. This is inefficient and hurts the compression ratio of zip files.
You can apply the exact same principle to any block of data, if you want. Trade compression efficiency for paralellizability. ZIP is the format that doesn't do it in a useful way; as you said, if you have one much larger file, the point is moot.
'restart window at' is a principle that can be generalized, and various compression formats support it in a much more useful fashion (restart at every X bytes, vs. ZIPs unreliable 'restart at every file').
What is the bottleneck?
Multiple aspects are involved when sending data: The speed at which the source provides the bytes you want to send, the speed at which the bytes are processed into a package that can then be sent (e.g. a zip tool, but can be anything, including just sending it verbatim, uncompressed), the speed at which the packaged-up bytes are transited to the target system, the speed at which the target can unpack it, and the speed at which the target can then process the unpacked result.
Are you sure that the compression aspect is the bottleneck?
In the base case where you read the bytes off of a harddisk, zip them up, send them across a residential internet pipe to another system, that system unzips, and saves them on a HDD, it is rather likely that the bottleneck is the network. Parallellizing the compression step is a complete waste and in fact only slows things down by reducing compression ratios.
If you're reading files off of a spinning platter, then the speed of the source is likely the bottleneck, and parallel processing considerably slows things down: You're now asking the read head to bounce around, and this is much slower than reading the data sequentially in one go.
If you have a fast source, and a fast pipe, then the bottleneck is doubtlessly the compression and uncompression, but the solution is then not to compress at all: If you are transferring data off of SSDs or from a USB3-connected byte-spewing sensor and transfer it across a 10M CAT6 cable from one gigabit ethernet port to another, then why compress at all? Just send those bytes. Compression isn't going to make it any faster, and as long as you don't saturate that 1Gb connection, you gain absolutely nothing whatsoever by attempting to compress it.
If you have a slow pipe, then the only way to make things faster is to compress as much as you can. Which most definitely involves not using the DEFLATE algorithm (e.g. don't use zip). Use another algorithm and configure it to get better compression rates, at the cost of CPU performance. Parallelising is irrelevant; it's not the bottleneck, so there is no point whatsoever in doing it.
Conclusions
Most likely you want to either send your files over uncompressed, or ZStandard your files over, tweaking the compression v. speed ratio as needed. I'm not aware of any ZStandard (zstd) impl for java itself, but the zstd-jni project gives you a java-based API for invoking the C zstd library.
If you insist on sticking with ZIP, the answer is a fairly basic 'nope, you cannot really do that', though you could in theory write a parallel ZIP compressor that has worse compression power but parallelizes better (by restarting the window within a single file for larger files in addition to the forced-upon-you-by-the-format restart at every file), and produces ZIP files that are still compatible with just about every unzip tool on the planet. I'm not aware of one, I don't think one exists, and writing one yourself would be a decidedly non-trivial exercise.

Write 1 million rows of CSV into S3 by batches

I'm trying to build a very large CSV file on S3.
I want to build this file on S3
I want to append rows to this file in batches.
Number of rows could be anywhere between 10k to 1M
Size of each batch could be < 5Mb(So multi-part upload is not feasible)
What would be the right way of accomplishing something like this?

Traditionally in Big Data processing ("Data Lakes"), information related to a single table are stored in a directory rather than a single file. So, appending information to a table is as simple as adding another file to a directory. All files within the directory will need to be the same schema (such as CSV columns, or JSON data).
The directory of files can then be used with tools such as:
Spark, Hive and Presto on Hadoop
Amazon Athena
Amazon Redshift Spectrum
A benefit of this method is that the above systems can process multiple files in parallel rather than being restricted to processing a single file in a single-threaded method.
Also common is to compress the files using technologies like gzip. This lowers storage requirements and makes it faster to read data from disk. Adding additional files is easy (just add another csv.gz file) rather than having to unzip, append and re-zip a file.
Bottom line: It would be advisable to re-think your requirements for "one great big CSV file".

'One big file' isn't going to work for you - you can't append rows to an s3 file, without first downloading the entire file, adding the rows, and then uploading the new file over the old one - for small files, it will work, but as the file gets larger, the bandwidth and processing is going to go up geometrically on you, and may get very slow and possibly expensive.
Better off refactoring your design to work with lots of little files instead of one big one.

Leave a 5MB garbage object sitting on S3 and do concatenation with it where part 1 = 5MB garbage object, part 2 = your file that you want to upload and concatenate. Keep repeating this for each fragment and finally use the range copy to strip out the 5MB garbage.

Best way storing multiple files with various size

My server application stores files of different size. Database size goes up to 2-4 TB, where about 90% of all files have less than 100kb, the rest up to 2 GB.
Which would be the best way of storing these files according to following conditions:
* The database should be portable.
It is at the actual solution, as the index, metadata etc. is stored a SQLite3 database. Now it is a pest copying 10 million single small files so I'd prefer store small files inside archives, like 1000x 2gb files.
* Some files are quite big, so BLOB storing may get to a performance problem
Additionally some of the bigger(5 mb - 2gb) files may need RandomAccess reading, which won't be possible with most archive systems
* Some transactions need to read thousands of small files at once
..which works good with BLOB, but I'm not sure if something like a zip archive would perform well in this case. The server can in most cases put all of these files in one archive, but the objects may have to be read in random order.
* Files stored should not be (directly) executable, but no need of encryption
Which system would you recommend for that?

Java: How to decompress files (zip or tar.gz) with low memory usage?

I have the following requirement:
I need to unpack a zip or tar.gz file. The target platform where this code will run is an AEM 6.1 environment. The system had some performance issues in the past. Especially memory usage was much to high. Therefore I have to save memory. The zip/tar.gz file contains some text-Files, SVG, PNG and EPS files as well as more files I don't need. The archive file will be downloaded and available as an ByteArrayInputStream.
I did some research and tried to figure out, which is the best way to do that. Apache commons provides libraries to unpack archives as well as the JDK. But I could not figure out, what implementation uses less memory.
I think, it would be the best, if I could open the archive while it is still compressed, and read and unpack the containing files separately. So I would just have the compressed archive and the one of its containing file in in memory.
But I'm not sure, which implementation provides this possibility or if there is a better way to do that.
Has somebody a good advice?
Thank you and best regards.

ZipInputStream from the JDK does just what you need: https://docs.oracle.com/javase/8/docs/api/java/util/zip/ZipInputStream.html
You can find the entry you need via getNextEntry().getName(), and read the bytes just for that entry. The ZipInputStream.read method allows you to implement buffered read, so you can easily limit the memory consumption if you don't need the whole decompressed entry in memory (i.e. if you write the entry into output as you read it).
In this case you can minimize the footprint of you application as well, since you won't need any extra libraries.

Compressing large files using blocks in Java

I am compressing files of over 2GB in Java using a consecutive application of two compression algorithms; one LZ based and one Huffman-based. (This is similar to DEFLATE).
Since 2GB is too large to be held in any buffer, I have to pass the file through one algorithm outputting a temporary file, then pass that temporary file through the second algorithm outputting the final file.
An alternative is to compress the file in 8MB blocks (the size where I don't get an Out-Of-Memory error) but then I have an inability to take full advantage of the redundancy within the entire file.
Any ideas how to perform these operations neater. No temporary files, and no compressing in blocks? Do any other compression tools compress in blocks? How do they deal with this issue? Regards

Java comes with “java.util.zip” library to perform data compression in ZIp format.
The overall concept is quite straightforward.
Library reads file with “FileInputStream”.
And add the file name to “ZipEntry” and output it to “ZipOutputStream“
import java.util.zip.ZipEntry and import java.util.zip.ZipOutputStream are used for importing Zip folder to a program.
But how can decompress a file
?

What's wrong with piping of streams? You can read from InputStream, compress the bytes and write them to output stream that is connected to input stream of the next algorithm. Take a look on PipeInputStream and PipeOutputStream.
I hope that these algorithms can work incrementally.

You could use two levels of java.util.zip. First, just concatenate all files (without compression). If possible, sort the entries by file type so that similar files are next to each other (this will increase compression ratio). Second, compress this stream. You don't need to run two separate phases; instead, you can wrap the first within the second stage, like CompressStream(ConcatenateFiles(directory)). That way you have a zip file within another zip file: the outer zip file is compressed, the inner is not and contains all the actual files.
It is true that java.util.zip used to have problems with files larger than 2 GB (I did run into those problems). However, I believe that was only the case for ZipFile and not for ZipIn/OutputStream. Also, I think those problems are fixed with recent Java versions.
Buffer size: regular compression algorithms such as Deflate will not benefit from chunk sizes larger than about 64 KB. More advanced algorithms can benefit from using larger chunk sizes, for example bzip2 up to 900 KB, or LZMA2 up to 2 MB. Anything beyond that is more likely the domain of data deduplication, which might or might not make sense for what you want to do.