Develop Reference

Java Base64 encoding of Excel File missing last byte

I am base64 encoding an excel file, send it somewhere where it is saved. Apparently after this, excel complains that the file is incorrect and if I want to attept a restore. The code I am doing (actually I did a quick test main method) is:
public static void main(String[] args) throws IOException {
Path p = Paths.get("C:\\VariousJunk\\excel-test", "test.xlsx");
ByteArrayOutputStream base64StringOutputStream = new ByteArrayOutputStream();
OutputStream base64EncodingStream = Base64.getEncoder().wrap(base64StringOutputStream);
Files.copy(p, base64EncodingStream);
base64StringOutputStream.close();
String b64 = base64StringOutputStream.toString();
byte[] data = Base64.getDecoder().decode(b64);
FileOutputStream fos = new FileOutputStream("C:\\VariousJunk\\excel-test\\test-backup.xlsx");
fos.write(data);
fos.close();
}
Now I have compared binary data of both files and it appears, that the output file is only missing one last byte with value 0. I have added the last bit for the test
fos.write(data);
fos.write(0);
fos.close();
And it works fine. The problem is I will be using this for any other type of data, and so I am not sure whether hardcoding a last byte is a good idea, possibly it might crash other filetypes. Is this a feature of this Base64 encoding method or am I doing something wrong?

Apparently the missing bit was base64EncodingStream.close() just after Files.copy()
OutputStream base64EncodingStream = Base64.getEncoder().wrap(base64StringOutputStream);
Files.copy(p, base64EncodingStream);
base64EncodingStream.close();
String b64 = base64StringOutputStream.toString();

How can i get pdf files from FTP server as base64 encoding format on Java spring mvc project?

I'm trying getting files from FTP server on Java spring mvc project. I'm study on windows but my tomcat server is in linux machine. Following code returns base64 encoding files and created base64 url for front end side and this files temporarily held. This code works fine windows but works bad on linux machine. Getting pdf files are issueless in windows, corrupted in linux machine. Works differently according to file size.
This result in linux machine, this file is issueless in folder of FTP.
This result on windows machine
Can the problem be caused by base64 encoding?
public List<String> getMultipleBase64PDF(String workingDir) {
List<String> fileList = new ArrayList<>();
try {
FTPFile[] files = client.listFiles(workingDir);
for (FTPFile file : files) {
if (file.getName().endsWith(".pdf")) {
fileNames.add(file.getName());
}
}
if (fileNames.size() > 0) {
for (String filename : fileNames) {
String encodedFile = "";
InputStream is = client.retrieveFileStream( workingDir);
File file = File.createTempFile("tmp", null);
FileUtils.copyInputStreamToFile(is, file);
byte[] bytes = new byte[(int) file.length()];
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
bis.read(bytes, 0, (int) file.length());
bis.close();
encodedFile += new String(Base64.getEncoder().withoutPadding().encode(bytes), "UTF-8");
fileList.add(encodedFile);
file.delete();
client.completePendingCommand();
}
}
disconnect();
} catch (IOException e) {
e.printStackTrace();
}
return fileList;
}

FTPClient is set to ASCII by default. While configuring FTPClient bean, try switching your client from using ASCII to binary, like so:
client.setFileType(FTP.BINARY_FILE_TYPE);
Here's the original answer for the similair problem

Java: how to compress a byte[] using ZipOutputStream without intermediate file

Requirement: compress a byte[] to get another byte[] using java.util.zip.ZipOutputStream BUT without using any files on disk or in-memory(like here https://stackoverflow.com/a/18406927/9132186). Is this even possible?
All the examples I found online read from a file(.txt) and write to a file(.zip). ZipOutputStream needs a ZipEntry to work with and that ZipEntry needs a file.
However, my use case is as follows: I need to compress a chunk (say 10MB) of a file at a time using a zip format and append all these compressed chunks to make a .zip file. But, when I unzip the .zip file then it is corrupted.
I am using in-memory files as suggested in https://stackoverflow.com/a/18406927/9132186 to avoid files on disk but need a solution without these files also.
public void testZipBytes() {
String infile = "test.txt";
FileInputStream in = new FileInputStream(infile);
String outfile = "test.txt.zip";
FileOutputStream out = new FileOutputStream(outfile);
byte[] buf = new byte[10];
int len;
while ((len = in.read(buf)) > 0) {
out.write(zipBytes(buf));
}
in.close();
out.close();
}
// ACTUAL function that compresses byte[]
public static class MemoryFile {
public String fileName;
public byte[] contents;
}
public byte[] zipBytesMemoryFileWORKS(byte[] input) {
MemoryFile memoryFile = new MemoryFile();
memoryFile.fileName = "try.txt";
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ZipOutputStream zos = new ZipOutputStream(baos);
ZipEntry entry = new ZipEntry(memoryFile.fileName);
entry.setSize(input.length);
zos.putNextEntry(entry);
zos.write(input);
zos.finish();
zos.closeEntry();
zos.close();
return baos.toByteArray();
}
Scenario 1:
if test.txt has small amount of data (less than 10 bytes) like "this" then unzip test.txt.zip yeilds try.txt with "this" in it.
Scenario 2:
if test.txt has larger amount of data (more than 10 bytes) like "this is a test for zip output stream and it is not working" then unzip test.txt.zip yields try.txt with broken pieces of data and is incomplete.
this 10 bytes is the buffer size in testZipBytes and is the amount of data that is compressed at a time by zipBytes
Expected (or rather desired):
1. unzip test.txt.zip does not use the "try.txt" filename i gave in the MemoryFile but rather unzips to filename test.txt itself.
2. unzipped data is not broken and yields the input data as is.
3. I have done the same with GzipOutputStream and it works perfectly fine.

Requirement: compress a byte[] to get another byte[] using java.util.zip.ZipOutputStream BUT without using any files on disk or in-memory(like here https://stackoverflow.com/a/18406927/9132186). Is this even possible?
Yes, you've already done it. You don't actually need MemoryFile in your example; just delete it from your implementation and write ZipEntry entry = new ZipEntry("try.txt") instead.
But you can't concatenate the zips of 10MB chunks of file and get a valid zip file for the combined file. Zipping doesn't work like that. You could have a solution which minimizes how much is in memory at once, perhaps. But breaking the original file up into chunks seems unworkable.

Extract tar.gz file in memory in Java

I'm using the Apache Compress library to read a .tar.gz file, something like this:
final TarArchiveInputStream tarIn = initializeTarArchiveStream(this.archiveFile);
try {
TarArchiveEntry tarEntry = tarIn.getNextTarEntry();
while (tarEntry != null) {
byte[] btoRead = new byte[1024];
BufferedOutputStream bout = new BufferedOutputStream(new FileOutputStream(destPath)); //<- I don't want this!
int len = 0;
while ((len = tarIn.read(btoRead)) != -1) {
bout.write(btoRead, 0, len);
}
bout.close();
tarEntry = tarIn.getNextTarEntry();
}
tarIn.close();
}
catch (IOException e) {
e.printStackTrace();
}
Is it possible not to extract this into a seperate file, and read it in memory somehow? Maybe into a giant String or something?

You could replace the file stream with a ByteArrayOutputStream.
i.e. replace this:
BufferedOutputStream bout = new BufferedOutputStream(new FileOutputStream(destPath)); //<- I don't want this!
with this:
ByteArrayOutputStream bout = new ByteArrayOutputStream();
and then after closing bout, use bout.toByteArray() to get the bytes.

Is it possible not to extract this into a seperate file, and read it in memory somehow? Maybe into a giant String or something?
Yea sure.
Just replace the code in the inner loop that is openning files and writing to them with code that writes to a ByteArrayOutputStream ... or a series of such streams.
The natural representation of the data that you read from the TAR (like that) will be bytes / byte arrays. If the bytes are properly encoded characters, and you know the correct encoding, then you can convert them to strings. Otherwise, it is better to leave the data as bytes. (If you attempt to convert non-text data to strings, or if you convert using the wrong charset/encoding you are liable to mangle it ... irreversibly.)
Obviously, you are going to need to think through some of these issues yourself, but basic idea should work ... provided you have enough heap space.

copy the value of btoread to a String like
String s = String.valueof(byteVar);
and goon appending the byte value to the string untill end of the file reaches..

How to create ByteArrayInputStream from a file in Java?

I have a file that can be any thing like ZIP, RAR, txt, CSV, doc etc. I would like to create a ByteArrayInputStream from it.
I'm using it to upload a file to FTP through FTPClient from Apache Commons Net.
Does anybody know how to do it?
For example:
String data = "hdfhdfhdfhd";
ByteArrayInputStream in = new ByteArrayInputStream(data.getBytes());
My code:
public static ByteArrayInputStream retrieveByteArrayInputStream(File file) {
ByteArrayInputStream in;
return in;
}

Use the FileUtils#readFileToByteArray(File) from Apache Commons IO, and then create the ByteArrayInputStream using the ByteArrayInputStream(byte[]) constructor.
public static ByteArrayInputStream retrieveByteArrayInputStream(File file) {
return new ByteArrayInputStream(FileUtils.readFileToByteArray(file));
}

The general idea is that a File would yield a FileInputStream and a byte[] a ByteArrayInputStream. Both implement InputStream so they should be compatible with any method that uses InputStream as a parameter.
Putting all of the file contents in a ByteArrayInputStream can be done of course:
read in the full file into a byte[]; Java version >= 7 contains a convenience method called readAllBytes to read all data from a file;
create a ByteArrayInputStream around the file content, which is now in memory.
Note that this may not be optimal solution for very large files - all the file will stored in memory at the same point in time. Using the right stream for the job is important.

A ByteArrayInputStream is an InputStream wrapper around a byte array. This means you'll have to fully read the file into a byte[], and then use one of the ByteArrayInputStream constructors.
Can you give any more details of what you are doing with the ByteArrayInputStream? Its likely there are better ways around what you are trying to achieve.
Edit:
If you are using Apache FTPClient to upload, you just need an InputStream. You can do this;
String remote = "whatever";
InputStream is = new FileInputStream(new File("your file"));
ftpClient.storeFile(remote, is);
You should of course remember to close the input stream once you have finished with it.

This isn't exactly what you are asking, but is a fast way of reading files in bytes.
File file = new File(yourFileName);
RandomAccessFile ra = new RandomAccessFile(yourFileName, "rw"):
byte[] b = new byte[(int)file.length()];
try {
ra.read(b);
} catch(Exception e) {
e.printStackTrace();
}
//Then iterate through b

This piece of code comes handy:
private static byte[] readContentIntoByteArray(File file)
{
FileInputStream fileInputStream = null;
byte[] bFile = new byte[(int) file.length()];
try
{
//convert file into array of bytes
fileInputStream = new FileInputStream(file);
fileInputStream.read(bFile);
fileInputStream.close();
}
catch (Exception e)
{
e.printStackTrace();
}
return bFile;
}
Reference: http://howtodoinjava.com/2014/11/04/how-to-read-file-content-into-byte-array-in-java/