I'm trying to pull in JSON from a lambda, compress it to gzip format and upload to s3. I can do all of this except compress it to gzip. I have pulled various bit of code from here (S.O.) the first code but does not seem to work correctly. Here is what I have tried and the outcome:
this first method seems to make the file much smaller and is gzip format:
public void compressAndUpload(AmazonS3 s3, InputStream in) throws IOException {
Path tmpPath = Files.createTempFile("atest", ".json.gz");
OutputStream out = Files.newOutputStream(tmpPath);
GzipCompressorOutputStream gzOut = new GzipCompressorOutputStream(out);
IOUtils.copy(in, gzOut);
InputStream fileIn = Files.newInputStream(tmpPath);
long size = Files.size(tmpPath);
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType("application/x-gzip");
metadata.setContentLength(size);
s3.putObject(bucketName, "atest.json.gz", fileIn, metadata);
}
However, when I pull it to my local machine, but when I try to use 'gunzip' on it i get the following error message:
gzip: atest.json.gz: unexpected end of file
this next method when is not actually compressing the file and when i pull it down locally it says "not in gzip format"
public String handleRequest(Input input, Context context) {
try {
byte[] btArr = compress(input.getMessage());
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentType("application/x-gzip");
metadata.setContentLength(btArr.length);
AmazonS3ClientBuilder.defaultClient().putObject(new PutObjectRequest(bucketName, "test22.json.gz",
new ByteArrayInputStream(btArr), metadata));
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
public static byte[] compress(String str) throws Exception {
if (str == null || str.length() == 0) {
return null;
}
System.out.println("String length : " + str.length());
ByteArrayOutputStream obj=new ByteArrayOutputStream();
GzipCompressorOutputStream gzip = new GzipCompressorOutputStream(obj);
gzip.write(str.getBytes("UTF-8"));
gzip.flush(); <-------******Update: This was missing.. caused it to fail.
gzip.close();
return obj.toByteArray();
}
Am I missing a step here? I feel like this should be a fairly straight forward thing...
Related
I've found many ways of converting a file to a byte array and writing byte array to a file on storage.
What I want is to convert java.io.File to a byte array and then convert a byte array back to a java.io.File.
I don't want to write it out to storage like the following:
//convert array of bytes into file
FileOutputStream fileOuputStream = new FileOutputStream("C:\\testing2.txt");
fileOuputStream.write(bFile);
fileOuputStream.close();
I want to somehow do the following:
File myFile = ConvertfromByteArray(bytes);
Otherwise Try this :
Converting File To Bytes
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
public class Temp {
public static void main(String[] args) {
File file = new File("c:/EventItemBroker.java");
byte[] b = new byte[(int) file.length()];
try {
FileInputStream fileInputStream = new FileInputStream(file);
fileInputStream.read(b);
for (int i = 0; i < b.length; i++) {
System.out.print((char)b[i]);
}
} catch (FileNotFoundException e) {
System.out.println("File Not Found.");
e.printStackTrace();
}
catch (IOException e1) {
System.out.println("Error Reading The File.");
e1.printStackTrace();
}
}
}
Converting Bytes to File
public class WriteByteArrayToFile {
public static void main(String[] args) {
String strFilePath = "Your path";
try {
FileOutputStream fos = new FileOutputStream(strFilePath);
String strContent = "Write File using Java ";
fos.write(strContent.getBytes());
fos.close();
}
catch(FileNotFoundException ex) {
System.out.println("FileNotFoundException : " + ex);
}
catch(IOException ioe) {
System.out.println("IOException : " + ioe);
}
}
}
I think you misunderstood what the java.io.File class really represents. It is just a representation of the file on your system, i.e. its name, its path etc.
Did you even look at the Javadoc for the java.io.File class? Have a look here
If you check the fields it has or the methods or constructor arguments, you immediately get the hint that all it is, is a representation of the URL/path.
Oracle provides quite an extensive tutorial in their Java File I/O tutorial, with the latest NIO.2 functionality too.
With NIO.2 you can read it in one line using java.nio.file.Files.readAllBytes().
Similarly you can use java.nio.file.Files.write() to write all bytes in your byte array.
UPDATE
Since the question is tagged Android, the more conventional way is to wrap the FileInputStream in a BufferedInputStream and then wrap that in a ByteArrayInputStream.
That will allow you to read the contents in a byte[]. Similarly the counterparts to them exist for the OutputStream.
You can't do this. A File is just an abstract way to refer to a file in the file system. It doesn't contain any of the file contents itself.
If you're trying to create an in-memory file that can be referred to using a File object, you aren't going to be able to do that, either, as explained in this thread, this thread, and many other places..
Apache FileUtil gives very handy methods to do the conversion
try {
File file = new File(imagefilePath);
byte[] byteArray = new byte[file.length()]();
byteArray = FileUtils.readFileToByteArray(file);
}catch(Exception e){
e.printStackTrace();
}
There is no such functionality but you can use a temporary file by File.createTempFile().
File temp = File.createTempFile(prefix, suffix);
// tell system to delete it when vm terminates.
temp.deleteOnExit();
You cannot do it for File, which is primarily an intelligent file path. Can you refactor your code so that it declares the variables, and passes around arguments, with type OutputStream instead of FileOutputStream? If so, see classes java.io.ByteArrayOutputStream and java.io.ByteArrayInputStream
OutputStream outStream = new ByteArrayOutputStream();
outStream.write(whatever);
outStream.close();
byte[] data = outStream.toByteArray();
InputStream inStream = new ByteArrayInputStream(data);
...
1- Traditional way
The traditional conversion way is through using read() method of InputStream as the following:
public static byte[] convertUsingTraditionalWay(File file)
{
byte[] fileBytes = new byte[(int) file.length()];
try(FileInputStream inputStream = new FileInputStream(file))
{
inputStream.read(fileBytes);
}
catch (Exception ex)
{
ex.printStackTrace();
}
return fileBytes;
}
2- Java NIO
With Java 7, you can do the conversion using Files utility class of nio package:
public static byte[] convertUsingJavaNIO(File file)
{
byte[] fileBytes = null;
try
{
fileBytes = Files.readAllBytes(file.toPath());
}
catch (Exception ex)
{
ex.printStackTrace();
}
return fileBytes;
}
3- Apache Commons IO
Besides JDK, you can do the conversion using Apache Commons IO library in 2 ways:
3.1. IOUtils.toByteArray()
public static byte[] convertUsingIOUtils(File file)
{
byte[] fileBytes = null;
try(FileInputStream inputStream = new FileInputStream(file))
{
fileBytes = IOUtils.toByteArray(inputStream);
}
catch (Exception ex)
{
ex.printStackTrace();
}
return fileBytes;
}
3.2. FileUtils.readFileToByteArray()
public static byte[] convertUsingFileUtils(File file)
{
byte[] fileBytes = null;
try
{
fileBytes = FileUtils.readFileToByteArray(file);
}
catch(Exception ex)
{
ex.printStackTrace();
}
return fileBytes;
}
Server side
#RequestMapping("/download")
public byte[] download() throws Exception {
File f = new File("C:\\WorkSpace\\Text\\myDoc.txt");
byte[] byteArray = new byte[(int) f.length()];
byteArray = FileUtils.readFileToByteArray(f);
return byteArray;
}
Client side
private ResponseEntity<byte[]> getDownload(){
URI end = URI.create(your url which server has exposed i.e. bla
bla/download);
return rest.getForEntity(end,byte[].class);
}
public static void main(String[] args) throws Exception {
byte[] byteArray = new TestClient().getDownload().getBody();
FileOutputStream fos = new
FileOutputStream("C:\\WorkSpace\\testClient\\abc.txt");
fos.write(byteArray);
fos.close();
System.out.println("file written successfully..");
}
//The file that you wanna convert into byte[]
File file=new File("/storage/0CE2-EA3D/DCIM/Camera/VID_20190822_205931.mp4");
FileInputStream fileInputStream=new FileInputStream(file);
byte[] data=new byte[(int) file.length()];
BufferedInputStream bufferedInputStream=new BufferedInputStream(fileInputStream);
bufferedInputStream.read(data,0,data.length);
//Now the bytes of the file are contain in the "byte[] data"
/*If you want to convert these bytes into a file, you have to write these bytes to a
certain location, then it will make a new file at that location if same named file is
not available at that location*/
FileOutputStream fileOutputStream =new FileOutputStream(Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS).toString()+"/Video.mp4");
fileOutputStream.write(data);
/* It will write or make a new file named Video.mp4 in the "Download" directory of
the External Storage */
I have the following task to obtain a PDF from URL and return a BASE64 string.
What I have currently (sorry I am not a Java Expert):
public String readPDFSOAP(String var, Container container) throws StreamTransformationException{
try {
//get the url page from the arguments array
URL url = new URL("URLPDF");
try {
//get input Stream from URL
InputStream in = new BufferedInputStream(url.openStream());
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buf = new byte[131072];
int n = 0;
while (-1 != (n = in.read(buf))) {
out.write(buf, 0, n);
}
out.close();
in.close();
byte[] response = out.toByteArray();
String string = new String(response);
} catch (Exception e) {
e.printStackTrace();
}
} catch (Exception e) {
e.printStackTrace();
}return String;}
But the string can't be returned.
Any help is appreciated.
Thanks,
Julian
Your code is all kinds of wrong. For starters, use the Base64 class to handle encoding your byte array. And no need to assign it to a variable, just return it.
return Base64.getEncoder().encodeToString(response)
and on your last line, outside of your try/catch block, just throw an exception. If you get there then you weren't able to properly retrieve and encoded the response, so no need to return a value. You're in an error condition.
Use java.util.Base64.
PDFs can be pretty large. Instead of reading it into memory, encode the InputStream directly:
ByteArrayOutputStream out = new ByteArrayOutputStream();
try (InputStream in = new BufferedInputStream(url.openStream())) {
in.transferTo(Base64.getEncoder().wrap(out));
}
String base64 = out.toString(StandardCharsets.US_ASCII);
The Base64 encoded version is even larger than the original file. I don’t know what you plan to do with the encoded version, but if you’re planning to write it somewhere, you want to avoid keeping any version of the file—original or encoded—in memory. You can do that by having your method accept an OutputStream as an argument:
public void readPDFSOAP(OutputStream destination,
String var,
Container container)
throws StreamTransformationException,
IOException {
URL url = new URL("https://example.com/doc.pdf");
try (InputStream in = new BufferedInputStream(url.openStream())) {
in.transferTo(Base64.getEncoder().wrap(destination));
}
}
Update:
Since you have said you cannot use a try-with-resources statement:
A try-with-resources statement is just a convenient way to guarantee an InputStream (or other closeable resource) is closed. This:
try (InputStream in = new BufferedInputStream(url.openStream())) {
// code that uses 'in'
}
is (nearly) equivalent to this:
InputStream in = null;
try {
in = new BufferedInputStream(url.openStream());
// code that uses 'in'
} finally {
if (in != null) {
try {
in.close();
} catch (IOException e) {
// Suppress
}
}
}
import java.io.*;
import java.nio.*;
import java.util.Base64;
import java.util.UUID;
import java.io.UnsupportedEncodingException;
public class Abc {
public static String readFileAsString(String filePath) throws IOException {
DataInputStream dis = new DataInputStream(new FileInputStream(filePath));
try {
long len = new java.io.File(filePath).length();
if (len > Integer.MAX_VALUE) throw new IOException("File " + filePath + " too large")
byte[] bytes = new byte[(int) len];
dis.readFully(bytes);
String ans = new String(bytes, "UTF-8");
return ans;
} finally {
dis.close();
}
}
public static void main(String args[]) throws IOException {
String base64encodedString = null;
FileOutputStream stream = new FileOutputStream("C:\\Users\\EMP142738\\Desktop\\New folder\\Readhjbdsdsefd.pdf");
String filePath = new String("C:\\Users\\EMP142738\\Desktop\\New folder\\Readers Quick Ref Card.pdf");
try {
base64encodedString = java.util.Base64.getUrlEncoder().encodeToString(new Abc().readFileAsString(filePath).getBytes("utf-8"));
} catch (IOException e) {
e.printStackTrace();
}
try {
byte[] base64decodedBytes = java.util.Base64.getUrlDecoder().decode(base64encodedString);
stream.write(base64decodedBytes);
} catch(IOException e){
e.printStackTrace();}
finally {
stream.close();
}//catch (FileNotFoundException e) {
// e.printStackTrace();
}
}
I'm trying to encode and decode a PDF file using Base64. What I'm doing is converting a PDF(Binary File) to a ByteArray, then returning the ByteArray as a string. I'm then encoding this string in Base64, using java.util.Base64. When I try to backtrack through the process, I'm able to convert a PDF(Binary File) but the File is corrupted/damaged. Also, the output file after the entire process ( Encode- Decode) is significantly larger than the input file. I expected that both of them would be of the same size. What am I doing wrong here?
Edit 1( 7/13/16):
In the main method, I modified the code as per Jim's suggestion.
I tried using Base64.encode(byte[] src) after reading the documentation of the same. However it keeps giving the error "cannot find symbol Base64.encode(byte[])". But I've used the encodetoString method from the same Class( java.util.Base64.Encoder). I'm unable to understand the issue here.
Here's the modified main method used after returning a byte[] from the readFileAsString method.
public void main(String args[]) throws IOException {
String filePath = new String("C:\\Users\\EMP142738\\Desktop\\New folder\\Readers Quick Ref Card.pdf");
byte[] src = new Abc().readFileAsString(filePath);
byte[] destination = Base64.encode(src);
}
The problem is in your flow
byte[] -> String -> base64 string
You need to omit the conversion to String and go directly:
byte[] -> base64 string
Converting to String will corrupt a binary stream as it involves a decode operation from the input character set to 16-bit Unicode characters.
On server (C++), binary data is compressed using ZLib function:
compress2()
and it's sent over to client (Java).
On client side (Java), data should be decompressed using the following code snippet:
public static String unpack(byte[] packedBuffer) {
InflaterInputStream inStream = new InflaterInputStream(new ByteArrayInputStream( packedBuffer);
ByteArrayOutputStream outStream = new ByteArrayOutputStream();
int readByte;
try {
while((readByte = inStream.read()) != -1) {
outStream.write(readByte);
}
} catch(Exception e) {
JMDCLog.logError(" unpacking buffer of size: " + packedBuffer.length);
e.printStackTrace();
// ... the rest of the code follows
}
Problem is that when it tries to read in while loop it always throws:
java.util.zip.ZipException: invalid stored block lengths
Before I check for other possible causes can someone please tell me can I compress on one side with compress2 and decompress it on the other side using above code, so I can eliminate this as a problem? Also if someone has a possible clue about what might be wrong here (I know I didn't provide too much of of the code in here but projects are rather big.
Thanks.
I think the problem is not with unpack method but in packedBuffer content. Unpack works fine
public static byte[] pack(String s) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
DeflaterOutputStream dout = new DeflaterOutputStream(out);
dout.write(s.getBytes());
dout.close();
return out.toByteArray();
}
public static void main(String[] args) throws Exception {
byte[] a = pack("123");
String s = unpack(a); // calls your unpack
System.out.println(s);
}
output
123
public static String unpack(byte[] packedBuffer) {
try (GZipInputStream inStream = new GZipInputStream(
new ByteArrayInputStream(packedBuffer));
ByteArrayOutputStream outStream = new ByteArrayOutputStream()) {
inStream.transferTo(outStream);
//...
return outStream.toString(StandardCharsets.UTF_8);
} catch(Exception e) {
JMDCLog.logError(" unpacking buffer of size: " + packedBuffer.length);
e.printStackTrace();
throw new IllegalArgumentException(e);
}
}
ZLib is the zip format, hence a GZipInputStream is fine.
A you seem to expect the bytes to represent text, hence be in some encoding, add that encoding, Charset, to the conversion to String (which always holds Unicode).
Note, UTF-8 is the encoding of the bytes. In your case it might be an other encoding.
The ugly try-with-resources syntax closes the streams even on exception or here the return.
I rethrowed a RuntimeException as it seems dangerous to do something with no result.
Basically i compress video using the customized compressor class in Java. I have assembled my complete code snippets here. My actually problem is, generated video [ A.mp4] from the decompressed byte array is not running. I actually i got this compressor class code over the internet. As i new to Java platform, i am struggling to resolve this problem. Could you please any one help me on this.?
public class CompressionTest
{
public static void main(String[] args)
{
Compressor compressor = new Compressor();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
FileInputStream fis=null;
File file=null;
try
{
URL uri=CompressionTest.class.getResource("/Files/Video.mp4");
file=new File(uri.getPath());
fis = new FileInputStream(file);
}
catch ( FileNotFoundException fnfe )
{
System.out.println( "Unable to open input file");
}
try
{
byte[] videoBytes = getBytesFromFile(file);
System.out.println("CompressionVideoToCompress is: '" +videoBytes + "'");
byte[] bytesCompressed = compressor.compress(videoBytes);
System.out.println("bytesCompressed is: '" +bytesCompressed+ "'");
byte[] bytesDecompressed=compressor.decompress(bytesCompressed);
System.out.println("bytesDecompressed is: '" +bytesDecompressed+ "'");
FileOutputStream out = new FileOutputStream("A.mp4");
out.write(bytesDecompressed,0,bytesDecompressed.length-1);
out.close();
}
catch (IOException e)
{
// TODO Auto-generated catch block
System.out.println("bytesCompressed is: '");
}
}
public static byte[] getBytesFromFile(File file) throws IOException
{
InputStream is = new FileInputStream(file);
// Get the size of the file
long length = file.length();
// You cannot create an array using a long type.
// It needs to be an int type.
// Before converting to an int type, check
// to ensure that file is not larger than Integer.MAX_VALUE.
if (length > Integer.MAX_VALUE) {
// File is too large
}
// Create the byte array to hold the data
byte[] bytes = new byte[1064];
// Read in the bytes
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0)
{
offset += numRead;
}
// Ensure all the bytes have been read in
if (offset < bytes.length) {
throw new IOException("Could not completely read file "+file.getName());
}
// Close the input stream and return bytes
is.close();
return bytes;
}
}
class Compressor
{
public Compressor()
{}
public byte[] compress(byte[] bytesToCompress)
{
Deflater deflater = new Deflater();
deflater.setInput(bytesToCompress);
deflater.finish();
byte[] bytesCompressed = new byte[Short.MAX_VALUE];
int numberOfBytesAfterCompression = deflater.deflate(bytesCompressed);
byte[] returnValues = new byte[numberOfBytesAfterCompression];
System.arraycopy
(
bytesCompressed,
0,
returnValues,
0,
numberOfBytesAfterCompression
);
return returnValues;
}
public byte[] decompress(byte[] bytesToDecompress)
{
Inflater inflater = new Inflater();
int numberOfBytesToDecompress = bytesToDecompress.length;
inflater.setInput
(
bytesToDecompress,
0,
numberOfBytesToDecompress
);
int compressionFactorMaxLikely = 3;
int bufferSizeInBytes =
numberOfBytesToDecompress
* compressionFactorMaxLikely;
byte[] bytesDecompressed = new byte[bufferSizeInBytes];
byte[] returnValues = null;
try
{
int numberOfBytesAfterDecompression = inflater.inflate(bytesDecompressed);
returnValues = new byte[numberOfBytesAfterDecompression];
System.arraycopy
(
bytesDecompressed,
0,
returnValues,
0,
numberOfBytesAfterDecompression
);
}
catch (DataFormatException dfe)
{
dfe.printStackTrace();
}
inflater.end();
return returnValues;
}
}
I've tested your code by compressing and decompressing a simple TXT file. The code is broken, since the compressed file, when uncompressed, is different from the original one.
Take for granted that the code is broken at least in the getBytesFromFile function. Its logic is tricky and troublesome, since it only allows files up to length 1064 and the check (throwing IOException when a longer file is read) does not work at all. The file gets read only partially and no exception is thrown.
What you are trying to achieve (file compression/decompression) can be done this way. I've tested it and it works, you just need this library.
import java.io.*;
import java.util.zip.*;
import org.apache.commons.io.IOUtils; // <-- get this from http://commons.apache.org/io/index.html
public class CompressionTest2 {
public static void main(String[] args) throws IOException {
File input = new File("input.txt");
File output = new File("output.bin");
Compression.compress(input, output);
File input2 = new File("input2.txt");
Compression.decompress(output, input2);
// At this point, input.txt and input2.txt should be equal
}
}
class Compression {
public static void compress(File input, File output) throws IOException {
FileInputStream fis = new FileInputStream(input);
FileOutputStream fos = new FileOutputStream(output);
GZIPOutputStream gzipStream = new GZIPOutputStream(fos);
IOUtils.copy(fis, gzipStream);
gzipStream.close();
fis.close();
fos.close();
}
public static void decompress(File input, File output) throws IOException {
FileInputStream fis = new FileInputStream(input);
FileOutputStream fos = new FileOutputStream(output);
GZIPInputStream gzipStream = new GZIPInputStream(fis);
IOUtils.copy(gzipStream, fos);
gzipStream.close();
fis.close();
fos.close();
}
}
This code doesn't come from "credible and/or official sources" but at least it works. :)
Moreover, in order to get more answers, adjust the title stating your real problem: your compressed files don't decompress the right way. There is no 'video' stuff here. Moreover, zipping a .mp4 file is no achievement (compression ratio will likely be around 99.99%).
Two tips:
1) Replace getBytesFromFile with a well known API call, either using Apache commons (IOUtils) or java 7 now provides such a method, too.
2) Test compress and decompress by writing a Junit test:
Create a random huge byte array, write it out, read it back and compare it with the created one.