I have a simple Spring Boot microservice that takes care of uploading, retrieving and deleting images to/from Google Cloud Storage. I have the following code for the get request in my service:
public StorageObject getImage(String fileName) throws IOException {
StorageObject object = storage.objects().get(bucketName, fileName).execute();
File file = new File("./" + fileName);
FileOutputStream os = new FileOutputStream(file);
storage.getRequestFactory()
.buildGetRequest(new GenericUrl(object.getMediaLink()))
.execute()
.download(os);
object.set("file", file);
return object;
}
And this is my controller part:
#GetMapping("/get/image/{id}")
public ResponseEntity<byte[]> getImage(#PathVariable("id") Long id) {
try {
String fileName = imageService.findImageById(id);
StorageObject object = gcsService.getImage(fileName);
byte[] res = Files.toByteArray((File) object.get("file"));
return ResponseEntity.ok()
.contentType(MediaType.IMAGE_JPEG)
.body(res);
} catch (IOException e) {
e.printStackTrace();
throw new RuntimeException("No such file or directory");
}
}
It all works fine in terms of getting the image in the response, but my problem is that the images get downloaded at the root directory of the project too. Many images are going to be uploaded through this service so this is an issue. I only want to display the images in the response (as a byteArray), without having them download. I tried playing with the code but couldn't manage to get it to work as I want.
I'd suggest to instead stream the download, while skipping the FileChannel operation:
public static void streamObjectDownload(
String projectId, String bucketName, String objectName, String targetFile
) {
Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();
try (ReadChannel reader = storage.reader(BlobId.of(bucketName, objectName));
FileChannel targetFileChannel = FileChannel.open(Paths.get(targetFile), StandardOpenOption.WRITE)) {
ByteStreams.copy(reader, targetFileChannel);
System.out.println(
"Downloaded object " + objectName
+ " from bucket " + bucketName
+ " to " + targetFile
+ " using a ReadChannel.");
}
} catch (IOException e) {
e.printStacktrace()
}
}
One can eg. obtain a FileChannel from a RandomAccessFile:
RandomAccessFile file = new RandomAccessFile(Paths.get(targetFile), StandardOpenOption.WRITE);
FileChannel channel = file.getChannel();
While the Spring framework similarly has a GoogleStorageResource:
public OutputStream getOutputStream() throws IOExceptionReturns the output stream for a Google Cloud Storage file.
Then convert from OutputStream to byte[] (this may be binary or ASCII data):
byte[] bytes = os.toByteArray();
Would it work for you to create Signed URLs in Cloud Storage to display your images? These URLs give access to storage bucket files for a limited time, and then expire, so you would rather not store temporary copies of the image locally as is suggested in this post.
public class GzExtractor implements Extractor {
Logger logger = LoggerFactory.getLogger(GzExtractor.class);
private static final int BUFFER_SIZE = 1024;
byte[] buff = new byte[BUFFER_SIZE];
private File file;
private String destinationPath;
public GzExtractor(File file, String destinationPath) {
this.file = file;
this.destinationPath = destinationPath;
}
public void extract() {
try {
File destDir = new File(destinationPath);
if (!destDir.exists()) {
destDir.mkdir();
}
GZIPInputStream gZipObj = new GZIPInputStream(new FileInputStream(file));
String extractedFilename = file.getName().split(".gz")[0];
OutputStream fosObj = new FileOutputStream(destinationPath + extractedFilename);
int len;
while ((len = gZipObj.read(buff)) > 0) {
fosObj.write(buff, 0, len);
}
gZipObj.close();
fosObj.close();
} catch (Exception e) {
logger.info("GZ Exception : {}",e.getMessage());
}
}
}
I'm getting the error of unexpected ZLIB stream but the file is extracted successfully.
I tried some solutions but none of them solved this. I tried closing the gzip stream before reading as I found that from one of the answers here. But that throws another error of course.
I am confused why I'm getting this and I want to basically eliminate the error statement.
[pool-1-thread-1] INFO service.ExtractorImpl.GzExtractor - GZ Exception : Unexpected end of ZLIB input stream
Okay so probably the compressed file was not in the correct format. I was using an FTP server where I was uploading different kinds of files i.e zip, gzip, CSV, etc. It was in my logic that the decompression would occur on the file according to the compression type of the file. While downloading from my FTP server, I forgot to mention the type of file that must be binary to include zip files.
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
So the file which was being decompressed must not be in a correct format. Maybe that is why I was getting this error.
After setting the file type to this, it worked okay.
I'm trying to return a list of file names from inside of a tar file. I'm using the code below, but when it gets to the while loop, it immediately goes to the catch exception and says "java.io.IOException: Error detected parsing the header
Below is the code I'm using. Can you help me figure out why this doesn't work?
public List<String> getFilesInTar(String filename) {
List<String> foundFiles = Lists.newArrayList();
String filePath = System.getProperty("user.home") + File.separator + "Downloads" + File.separator + filename;
try {
TarArchiveInputStream tarInput = new TarArchiveInputStream(new FileInputStream(filePath));
TarArchiveEntry entry;
while ((entry = tarInput.getNextTarEntry()) != null) {
if (!entry.isDirectory()) {
foundFiles.add(entry.getName());
}
}
tarInput.close();
} catch (IOException ex) {
log.error(ex.getMessage());
}
return foundFiles;
}
Your file is not a tar file. It’s a compressed archive of a tar file.
You cannot open it as a tar file, as is, for the same reason you can’t read a text file while it’s in a zip archive: the bytes representing compressed data are not themselves readable.
The .gz extension of your filename indicates that it was compressed using gzip, which is common when compressing tar files. You can use the GZIPInputStream class to uncompress it:
TarArchiveInputStream tarInput = new TarArchiveInputStream(
new GZIPInputStream(
new BufferedInputStream(
new FileInputStream(filePath))));
I am currently extracting the contents of a war file and then adding some new files to the directory structure and then creating a new war file.
This is all done programatically from Java - but I am wondering if it wouldn't be more efficient to copy the war file and then just append the files - then I wouldn't have to wait so long as the war expands and then has to be compressed again.
I can't seem to find a way to do this in the documentation though or any online examples.
Anyone can give some tips or pointers?
UPDATE:
TrueZip as mentioned in one of the answers seems to be a very good java library to append to a zip file (despite other answers that say it is not possible to do this).
Anyone have experience or feedback on TrueZip or can recommend other similar libaries?
In Java 7 we got Zip File System that allows adding and changing files in zip (jar, war) without manual repackaging.
We can directly write to files inside zip files as in the following example.
Map<String, String> env = new HashMap<>();
env.put("create", "true");
Path path = Paths.get("test.zip");
URI uri = URI.create("jar:" + path.toUri());
try (FileSystem fs = FileSystems.newFileSystem(uri, env))
{
Path nf = fs.getPath("new.txt");
try (Writer writer = Files.newBufferedWriter(nf, StandardCharsets.UTF_8, StandardOpenOption.CREATE)) {
writer.write("hello");
}
}
As others mentioned, it's not possible to append content to an existing zip (or war). However, it's possible to create a new zip on the fly without temporarily writing extracted content to disk. It's hard to guess how much faster this will be, but it's the fastest you can get (at least as far as I know) with standard Java. As mentioned by Carlos Tasada, SevenZipJBindings might squeeze out you some extra seconds, but porting this approach to SevenZipJBindings will still be faster than using temporary files with the same library.
Here's some code that writes the contents of an existing zip (war.zip) and appends an extra file (answer.txt) to a new zip (append.zip). All it takes is Java 5 or later, no extra libraries needed.
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Enumeration;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipOutputStream;
public class Main {
// 4MB buffer
private static final byte[] BUFFER = new byte[4096 * 1024];
/**
* copy input to output stream - available in several StreamUtils or Streams classes
*/
public static void copy(InputStream input, OutputStream output) throws IOException {
int bytesRead;
while ((bytesRead = input.read(BUFFER))!= -1) {
output.write(BUFFER, 0, bytesRead);
}
}
public static void main(String[] args) throws Exception {
// read war.zip and write to append.zip
ZipFile war = new ZipFile("war.zip");
ZipOutputStream append = new ZipOutputStream(new FileOutputStream("append.zip"));
// first, copy contents from existing war
Enumeration<? extends ZipEntry> entries = war.entries();
while (entries.hasMoreElements()) {
ZipEntry e = entries.nextElement();
System.out.println("copy: " + e.getName());
append.putNextEntry(e);
if (!e.isDirectory()) {
copy(war.getInputStream(e), append);
}
append.closeEntry();
}
// now append some extra content
ZipEntry e = new ZipEntry("answer.txt");
System.out.println("append: " + e.getName());
append.putNextEntry(e);
append.write("42\n".getBytes());
append.closeEntry();
// close
war.close();
append.close();
}
}
I had a similar requirement sometime back - but it was for reading and writing zip archives (.war format should be similar). I tried doing it with the existing Java Zip streams but found the writing part cumbersome - especially when directories where involved.
I'll recommend you to try out the TrueZIP (open source - apache style licensed) library that exposes any archive as a virtual file system into which you can read and write like a normal filesystem. It worked like a charm for me and greatly simplified my development.
You could use this bit of code I wrote
public static void addFilesToZip(File source, File[] files)
{
try
{
File tmpZip = File.createTempFile(source.getName(), null);
tmpZip.delete();
if(!source.renameTo(tmpZip))
{
throw new Exception("Could not make temp file (" + source.getName() + ")");
}
byte[] buffer = new byte[1024];
ZipInputStream zin = new ZipInputStream(new FileInputStream(tmpZip));
ZipOutputStream out = new ZipOutputStream(new FileOutputStream(source));
for(int i = 0; i < files.length; i++)
{
InputStream in = new FileInputStream(files[i]);
out.putNextEntry(new ZipEntry(files[i].getName()));
for(int read = in.read(buffer); read > -1; read = in.read(buffer))
{
out.write(buffer, 0, read);
}
out.closeEntry();
in.close();
}
for(ZipEntry ze = zin.getNextEntry(); ze != null; ze = zin.getNextEntry())
{
out.putNextEntry(ze);
for(int read = zin.read(buffer); read > -1; read = zin.read(buffer))
{
out.write(buffer, 0, read);
}
out.closeEntry();
}
out.close();
tmpZip.delete();
}
catch(Exception e)
{
e.printStackTrace();
}
}
I don't know of a Java library that does what you describe. But what you described is practical. You can do it in .NET, using DotNetZip.
Michael Krauklis is correct that you cannot simply "append" data to a war file or zip file, but it is not because there is an "end of file" indication, strictly speaking, in a war file. It is because the war (zip) format includes a directory, which is normally present at the end of the file, that contains metadata for the various entries in the war file. Naively appending to a war file results in no update to the directory, and so you just have a war file with junk appended to it.
What's necessary is an intelligent class that understands the format, and can read+update a war file or zip file, including the directory as appropriate. DotNetZip does this, without uncompressing/recompressing the unchanged entries, just as you described or desired.
As Cheeso says, there's no way of doing it. AFAIK the zip front-ends are doing exactly the same as you internally.
Anyway if you're worried about the speed of extracting/compressing everything, you may want to try the SevenZipJBindings library.
I covered this library in my blog some months ago (sorry for the auto-promotion). Just as an example, extracting a 104MB zip file using the java.util.zip took me 12 seconds, while using this library took 4 seconds.
In both links you can find examples about how to use it.
Hope it helps.
See this bug report.
Using append mode on any kind of
structured data like zip files or tar
files is not something you can really
expect to work. These file formats
have an intrinsic "end of file"
indication built into the data format.
If you really want to skip the intermediate step of un-waring/re-waring, you could read the war file file, get all the zip entries, then write to a new war file "appending" the new entries you wanted to add. Not perfect, but at least a more automated solution.
Yet Another Solution: You may find code below useful in other situations as well. I have used ant this way to compile Java directories, generating jar files, updating zip files,...
public static void antUpdateZip(String zipFilePath, String libsToAddDir) {
Project p = new Project();
p.init();
Target target = new Target();
target.setName("zip");
Zip task = new Zip();
task.init();
task.setDestFile(new File(zipFilePath));
ZipFileSet zipFileSet = new ZipFileSet();
zipFileSet.setPrefix("WEB-INF/lib");
zipFileSet.setDir(new File(libsToAddDir));
task.addFileset(zipFileSet);
task.setUpdate(true);
task.setProject(p);
task.init();
target.addTask(task);
target.setProject(p);
p.addTarget(target);
DefaultLogger consoleLogger = new DefaultLogger();
consoleLogger.setErrorPrintStream(System.err);
consoleLogger.setOutputPrintStream(System.out);
consoleLogger.setMessageOutputLevel(Project.MSG_DEBUG);
p.addBuildListener(consoleLogger);
try {
// p.fireBuildStarted();
// ProjectHelper helper = ProjectHelper.getProjectHelper();
// p.addReference("ant.projectHelper", helper);
// helper.parse(p, buildFile);
p.executeTarget(target.getName());
// p.fireBuildFinished(null);
} catch (BuildException e) {
p.fireBuildFinished(e);
throw new AssertionError(e);
}
}
this a simple code to get a response with using servlet and send a response
myZipPath = bla bla...
byte[] buf = new byte[8192];
String zipName = "myZip.zip";
String zipPath = myzippath+ File.separator+"pdf" + File.separator+ zipName;
File pdfFile = new File("myPdf.pdf");
ZipOutputStream out = new ZipOutputStream(new FileOutputStream(zipPath));
ZipEntry zipEntry = new ZipEntry(pdfFile.getName());
out.putNextEntry(zipEntry);
InputStream in = new FileInputStream(pdfFile);
int len;
while ((len = in.read(buf)) > 0) {
out.write(buf, 0, len);
}
out.closeEntry();
in.close();
out.close();
FileInputStream fis = new FileInputStream(zipPath);
response.setContentType("application/zip");
response.addHeader("content-disposition", "attachment;filename=" + zipName);
OutputStream os = response.getOutputStream();
int length = is.read(buffer);
while (length != -1)
{
os.write(buffer, 0, length);
length = is.read(buffer);
}
Here are examples how easily files can be appended to existing zip using TrueVFS:
// append a file to archive under different name
TFile.cp(new File("existingFile.txt"), new TFile("archive.zip", "entry.txt"));
// recusively append a dir to the root of archive
TFile src = new TFile("dirPath", "dirName");
src.cp_r(new TFile("archive.zip", src.getName()));
TrueVFS, the successor of TrueZIP, uses Java 7 NIO 2 features under the hood when appropriate but offers much more features like thread-safe async parallel compression.
Beware also that Java 7 ZipFileSystem by default is vulnerable to OutOfMemoryError on huge inputs.
Here is Java 1.7 version of Liam answer which uses try with resources and Apache Commons IO.
The output is written to a new zip file but it can be easily modified to write to the original file.
/**
* Modifies, adds or deletes file(s) from a existing zip file.
*
* #param zipFile the original zip file
* #param newZipFile the destination zip file
* #param filesToAddOrOverwrite the names of the files to add or modify from the original file
* #param filesToAddOrOverwriteInputStreams the input streams containing the content of the files
* to add or modify from the original file
* #param filesToDelete the names of the files to delete from the original file
* #throws IOException if the new file could not be written
*/
public static void modifyZipFile(File zipFile,
File newZipFile,
String[] filesToAddOrOverwrite,
InputStream[] filesToAddOrOverwriteInputStreams,
String[] filesToDelete) throws IOException {
try (ZipOutputStream out = new ZipOutputStream(new FileOutputStream(newZipFile))) {
// add existing ZIP entry to output stream
try (ZipInputStream zin = new ZipInputStream(new FileInputStream(zipFile))) {
ZipEntry entry = null;
while ((entry = zin.getNextEntry()) != null) {
String name = entry.getName();
// check if the file should be deleted
if (filesToDelete != null) {
boolean ignoreFile = false;
for (String fileToDelete : filesToDelete) {
if (name.equalsIgnoreCase(fileToDelete)) {
ignoreFile = true;
break;
}
}
if (ignoreFile) {
continue;
}
}
// check if the file should be kept as it is
boolean keepFileUnchanged = true;
if (filesToAddOrOverwrite != null) {
for (String fileToAddOrOverwrite : filesToAddOrOverwrite) {
if (name.equalsIgnoreCase(fileToAddOrOverwrite)) {
keepFileUnchanged = false;
}
}
}
if (keepFileUnchanged) {
// copy the file as it is
out.putNextEntry(new ZipEntry(name));
IOUtils.copy(zin, out);
}
}
}
// add the modified or added files to the zip file
if (filesToAddOrOverwrite != null) {
for (int i = 0; i < filesToAddOrOverwrite.length; i++) {
String fileToAddOrOverwrite = filesToAddOrOverwrite[i];
try (InputStream in = filesToAddOrOverwriteInputStreams[i]) {
out.putNextEntry(new ZipEntry(fileToAddOrOverwrite));
IOUtils.copy(in, out);
out.closeEntry();
}
}
}
}
}
this works 100% , if you dont want to use extra libs ..
1) first, the class that append files to the zip ..
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class AddZip {
public void AddZip() {
}
public void addToZipFile(ZipOutputStream zos, String nombreFileAnadir, String nombreDentroZip) {
FileInputStream fis = null;
try {
if (!new File(nombreFileAnadir).exists()) {//NO EXISTE
System.out.println(" No existe el archivo : " + nombreFileAnadir);return;
}
File file = new File(nombreFileAnadir);
System.out.println(" Generando el archivo '" + nombreFileAnadir + "' al ZIP ");
fis = new FileInputStream(file);
ZipEntry zipEntry = new ZipEntry(nombreDentroZip);
zos.putNextEntry(zipEntry);
byte[] bytes = new byte[1024];
int length;
while ((length = fis.read(bytes)) >= 0) {zos.write(bytes, 0, length);}
zos.closeEntry();
fis.close();
} catch (FileNotFoundException ex ) {
Logger.getLogger(AddZip.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(AddZip.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
2) you can call it in your controller ..
//in the top
try {
fos = new FileOutputStream(rutaZip);
zos = new ZipOutputStream(fos);
} catch (FileNotFoundException ex) {
Logger.getLogger(UtilZip.class.getName()).log(Level.SEVERE, null, ex);
}
...
//inside your method
addZip.addToZipFile(zos, pathFolderFileSystemHD() + itemFoto.getNombre(), "foto/" + itemFoto.getNombre());
Based on the answer given by #sfussenegger above, following code is used to append to a jar file and download it:
public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
Resource resourceFile = resourceLoader.getResource("WEB-INF/lib/custom.jar");
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try (ZipOutputStream zos = new ZipOutputStream(baos, StandardCharsets.ISO_8859_1);) {
try (ZipFile zin = new ZipFile(resourceFile.getFile(), StandardCharsets.ISO_8859_1);) {
zin.stream().forEach((entry) -> {
try {
zos.putNextEntry(entry);
if (!entry.isDirectory()) {
zin.getInputStream(entry).transferTo(zos);
}
zos.closeEntry();
} catch (Exception ex) {
ex.printStackTrace();
}
});
}
/* build file records to be appended */
....
for (FileContents record : records) {
zos.putNextEntry(new ZipEntry(record.getFileName()));
zos.write(record.getBytes());
zos.closeEntry();
}
zos.flush();
}
response.setContentType("application/java-archive");
response.setContentLength(baos.size());
response.setHeader(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"custom.jar\"");
try (BufferedOutputStream out = new BufferedOutputStream(response.getOutputStream())) {
baos.writeTo(out);
}
}
Hi i have been trying to upload image file in Play Framework. I have been trying out with Java File Upload since morning but unable to do so. I have seen [JavaFileUpload][1] tutorial available on framework website. But i am still not successful. Here is my code which i am trying to run:
Http.MultipartFormData body = request().body().asMultipartFormData();
List<Http.MultipartFormData.FilePart> fileParts = body.getFiles();
for (Http.MultipartFormData.FilePart filePart : fileParts) {
String filename = filePart.getFilename();
File file = filePart.getFile(); //error comes on this line
if (filePart.getFilename().toLowerCase().endsWith(".png")) {
//saving here but how?
} else {
return badRequest("Invalid request, only PNGs are allowed.");
}
}
but problem is that whenever i try to get the file i am having this conversion error:
java.lang.Object cannot be converted to java.io.File
Anyone can guide me in the direction? if we see the official document there is no proper documentation on how to upload multiple files. If anyone can show me some website which can helps me in that direction that will be also helpful
I'm using Play 2.4 and
FilePart filePart = request().body().asMultipartFormData()
.getFile("myFileKey");
File file = filePart.getFile();
With Play 2.2 I used for multiple file uploads:
MultipartFormData mfd = request().body().asMultipartFormData();
List<FilePart> filePartList = mfd.getFiles();
FilePart filePart = filePartList.get(0);
So after lots of trouble i was able to figure out the answer to my question. Here i am going to post the answer so it helps other people searching the answer to the same problem i faced
The controller function call which will upload the files looks like this:
Http.MultipartFormData body = request().body().asMultipartFormData();
List<Http.MultipartFormData.FilePart> fileParts = body.getFiles();
for (Http.MultipartFormData.FilePart filePart : fileParts) {
if (filePart.getFilename().toLowerCase().endsWith(".png")) {
String filename = filePart.getFilename();
Files.write(Paths.get(filename + ".png"), readContentIntoByteArray((File) filePart.getFile()));
} else {
return badRequest("Invalid request, only PNGs are allowed.");
}
}
I am using a function call to read the content of the file into byte array and save them inside the file:
private static byte[] readContentIntoByteArray(File file) {
FileInputStream fileInputStream = null;
byte[] bFile = new byte[(int) file.length()];
try {
//convert file into array of bytes
fileInputStream = new FileInputStream(file);
fileInputStream.read(bFile);
fileInputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
return bFile;
}
Remember you can choose whatever the path you want to save the file at Paths.get(filename + ".png")