I have few files in my local folder. I want to store the file-names as the key and the content of the corresponding file as value.
HashMap<String,String> hm = new HashMap<String,String>();
hm.put(filename,filecontent);
Can someone tell me is this the right way to do?
When storing file contents as a String, you have to make sure the encoding is respected, I would recommend to use byte array instead:
Map<String, byte[]> hm = new HashMap<String, byte[]>();
Also: depending on how many files you are manipulating, you may want to consider using file streams to avoid keeping everything in memory.
There are a couple of steps to what you would like.
I am going to assume you have the filename already as a String.
HashMap<String, byte[]> hm = new HashMap<String, byte[]>(); //Initialize our hashmap with String as key, and byte array as data (binary data)
FileInputStream fileStream = new FileInputStream(filename); //Get a stream of the file
byte[] buff = new byte[512]; //A buffer for our read loop
ByteArrayOutputStream byteStream = new ByteArrayOutputStream(); //Where to write buff content to, we can convert this into the output byte array with toByteArray()
while(fileStream.read(buff) > 0) { //read 512 bytes of file at a time, until end of file
byteStream.write(buff); //write buff content to byte stream
}
fileStream.close(); //Close our file handle, if we don't do this we may not be able to see changes!
hm.put(filename, byteStream.toByteArray()); //insert filename and filecontent to hashmap
As others have suggested, however, this is less than ideal. You are holding multiple files in memory for an arbitrary length of time. You can eat a lot of ram and not realize it doing this, and quickly run into an out of memory exception.
You would be better off reading the file content only when needed, so there isn't a whole file sitting in your ram for god knows how long. The only plausible reason I could see to store file contents would be if you were reading it a lot, and you could afford the ram to cache the file in memory.
Update for Binary Data
HashMap<String,String> hm = new HashMap<String, byte[]>();
final File folder = new File("/home/you/Desktop");
listFilesForFolder(folder);
public void listFilesForFolder(final File folder) {
for (final File fileEntry : folder.listFiles()) {
if (fileEntry.isDirectory()) {
listFilesForFolder(fileEntry);
} else {
String name = fileEntry.getName();
byte[] fileData = new byte[(int) fileEntry.length()];
DataInputStream dis = new DataInputStream(new FileInputStream(fileEntry));
dis.readFully(fileData);
dis.close();
hm.put(name,fileData);
}
}
}
Tested for Zip file for OP:
public static void main(String[] args) throws FileNotFoundException, IOException {
File file = new File("D:\\try.zip");
System.out.println(file.length());
byte[] fileData = new byte[(int) file.length()];
DataInputStream dis = new DataInputStream(new FileInputStream(file));
dis.readFully(fileData);
dis.close();
}
Related
Requirement: compress a byte[] to get another byte[] using java.util.zip.ZipOutputStream BUT without using any files on disk or in-memory(like here https://stackoverflow.com/a/18406927/9132186). Is this even possible?
All the examples I found online read from a file(.txt) and write to a file(.zip). ZipOutputStream needs a ZipEntry to work with and that ZipEntry needs a file.
However, my use case is as follows: I need to compress a chunk (say 10MB) of a file at a time using a zip format and append all these compressed chunks to make a .zip file. But, when I unzip the .zip file then it is corrupted.
I am using in-memory files as suggested in https://stackoverflow.com/a/18406927/9132186 to avoid files on disk but need a solution without these files also.
public void testZipBytes() {
String infile = "test.txt";
FileInputStream in = new FileInputStream(infile);
String outfile = "test.txt.zip";
FileOutputStream out = new FileOutputStream(outfile);
byte[] buf = new byte[10];
int len;
while ((len = in.read(buf)) > 0) {
out.write(zipBytes(buf));
}
in.close();
out.close();
}
// ACTUAL function that compresses byte[]
public static class MemoryFile {
public String fileName;
public byte[] contents;
}
public byte[] zipBytesMemoryFileWORKS(byte[] input) {
MemoryFile memoryFile = new MemoryFile();
memoryFile.fileName = "try.txt";
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ZipOutputStream zos = new ZipOutputStream(baos);
ZipEntry entry = new ZipEntry(memoryFile.fileName);
entry.setSize(input.length);
zos.putNextEntry(entry);
zos.write(input);
zos.finish();
zos.closeEntry();
zos.close();
return baos.toByteArray();
}
Scenario 1:
if test.txt has small amount of data (less than 10 bytes) like "this" then unzip test.txt.zip yeilds try.txt with "this" in it.
Scenario 2:
if test.txt has larger amount of data (more than 10 bytes) like "this is a test for zip output stream and it is not working" then unzip test.txt.zip yields try.txt with broken pieces of data and is incomplete.
this 10 bytes is the buffer size in testZipBytes and is the amount of data that is compressed at a time by zipBytes
Expected (or rather desired):
1. unzip test.txt.zip does not use the "try.txt" filename i gave in the MemoryFile but rather unzips to filename test.txt itself.
2. unzipped data is not broken and yields the input data as is.
3. I have done the same with GzipOutputStream and it works perfectly fine.
Requirement: compress a byte[] to get another byte[] using java.util.zip.ZipOutputStream BUT without using any files on disk or in-memory(like here https://stackoverflow.com/a/18406927/9132186). Is this even possible?
Yes, you've already done it. You don't actually need MemoryFile in your example; just delete it from your implementation and write ZipEntry entry = new ZipEntry("try.txt") instead.
But you can't concatenate the zips of 10MB chunks of file and get a valid zip file for the combined file. Zipping doesn't work like that. You could have a solution which minimizes how much is in memory at once, perhaps. But breaking the original file up into chunks seems unworkable.
I have 2 java classes. Let them be class A and class B.
Class A gets String input from user and stores the input as byte into the FILE, then Class B should read the file and display the Byte as String.
CLASS A:
File file = new File("C:\\FILE.txt");
file.createNewFile();
FileOutputStream fos = new FileOutputStream(file);
String fwrite = user_input_1+"\n"+user_input_2;
fos.write(fwrite.getBytes());
fos.flush();
fos.close();
In CLASS B, I wrote the code to read the file, but I don't know how to read the file content as bytes.
CLASS B:
fr = new FileReader(file);
br = new BufferedReader(fr);
arr = new ArrayList<String>();
int i = 0;
while((getF = br.readLine()) != null){
arr.add(getF);
}
String[] sarr = (String[]) arr.toArray(new String[0]);
The FILE.txt has the following lines
[B#3ce76a1
[B#36245605
I want both these lines to be converted into their respective string values and then display it. How to do it?
Are you forced to save using a String byte[] representation to save data? Take a look at object serialization (Object Serialization Tutorial), you don't have to worry about any low level line by line read or write methods.
Since you are writing a byte array through the FileOutputStream, the opposite operation would be to read the file using the FileInputStream, and construct the String from the byte array:
File file = new File("C:\\FILE.txt");
Long fileLength = file.length();
byte[] bytes = new byte[fileLength.intValue()]
try (FileInputStream fis = new FileInputStream(file)) {
fis.read(bytes);
}
String result = new String(bytes);
However, there are better ways of writing the String to a file.
You could write it using the FileWriter, and read using FileReader (possibly wrapping them by the corresponding BufferedReader/Writer), this will avoid creating intermediate byte array. Or better yet, use Apache Commons' IOUtils or Google's Guava libraries.
I have file with names like ex.zip. In this example, the Zip file contains only one file with the same name(ie. `ex.txt'), which is quite large. I don't want to extract the zip file every time.Hence I need to read the content of the file(ex.txt) without extracting the zip file. I tried some code like below But i can only read the name of the file in the variable.
How do I read the content of the file and stores it in the variable?
Thank you in Advance
fis=new FileInputStream("C:/Documents and Settings/satheesh/Desktop/ex.zip");
ZipInputStream zis = new ZipInputStream(new BufferedInputStream(fis));
ZipEntry entry;
while((entry = zis.getNextEntry()) != null) {
i=i+1;
System.out.println(entry);
System.out.println(i);
//read from zis until available
}
Your idea is to read the zip file as it is into a byte array and store it in a variable.
Later when you need the zip you extract it on demand, saving memory:
First read the content of the Zip file in a byte array zipFileBytes
If you have Java 1.7:
Path path = Paths.get("path/to/file");
byte[] zipFileBytes= Files.readAllBytes(path);
otherwise use Appache.commons lib
byte[] zipFileBytes;
zipFileBytes = IOUtils.toByteArray(InputStream input);
Now your Zip file is stored in a variable zipFileBytes, still in compressed form.
Then when you need to extract something use
ByteArrayInputStream bis = new ByteArrayInputStream(zipFileBytes));
ZipInputStream zis = new ZipInputStream(bis);
Try this:
String zipFile = "ex.zip";
try (ZipFile zip = new ZipFile(zipFile)) {
int i = 0;
for (Enumeration<? extends ZipEntry> e = zip.entries(); e.hasMoreElements(); ) {
ZipEntry entry = (ZipEntry) e.nextElement();
System.out.println(entry);
System.out.println(i);
InputStream in = zip.getInputStream(entry);
}
}
For example, if the file contains text, and you want to print it as a String, you can read the InputStream like this: How do I read / convert an InputStream into a String in Java?
I think that in your case the fact that a zipfile is a container that can hold many files (and thus forces you to navigate to the right contained file each time you open it) seriously complicates things, as you state that each zipfile only contains one textfile. Maybe it's a lot easier to just gzip the text file (gzip is not a container, just a compressed version of your data). And it's very simple to use:
GZIPInputStream gis = new GZIPInputStream(new FileInputStream("file.txt.gz"));
// and a BufferedReader on top to comfortably read the file
BufferedReader in = new BufferedReader(new InputStreamReader(gis) );
Producing them is equally simple:
GZIPOutputStream gos = new GZIPOutputStream(new FileOutputStream("file.txt.gz"));
I have a file that can be any thing like ZIP, RAR, txt, CSV, doc etc. I would like to create a ByteArrayInputStream from it.
I'm using it to upload a file to FTP through FTPClient from Apache Commons Net.
Does anybody know how to do it?
For example:
String data = "hdfhdfhdfhd";
ByteArrayInputStream in = new ByteArrayInputStream(data.getBytes());
My code:
public static ByteArrayInputStream retrieveByteArrayInputStream(File file) {
ByteArrayInputStream in;
return in;
}
Use the FileUtils#readFileToByteArray(File) from Apache Commons IO, and then create the ByteArrayInputStream using the ByteArrayInputStream(byte[]) constructor.
public static ByteArrayInputStream retrieveByteArrayInputStream(File file) {
return new ByteArrayInputStream(FileUtils.readFileToByteArray(file));
}
The general idea is that a File would yield a FileInputStream and a byte[] a ByteArrayInputStream. Both implement InputStream so they should be compatible with any method that uses InputStream as a parameter.
Putting all of the file contents in a ByteArrayInputStream can be done of course:
read in the full file into a byte[]; Java version >= 7 contains a convenience method called readAllBytes to read all data from a file;
create a ByteArrayInputStream around the file content, which is now in memory.
Note that this may not be optimal solution for very large files - all the file will stored in memory at the same point in time. Using the right stream for the job is important.
A ByteArrayInputStream is an InputStream wrapper around a byte array. This means you'll have to fully read the file into a byte[], and then use one of the ByteArrayInputStream constructors.
Can you give any more details of what you are doing with the ByteArrayInputStream? Its likely there are better ways around what you are trying to achieve.
Edit:
If you are using Apache FTPClient to upload, you just need an InputStream. You can do this;
String remote = "whatever";
InputStream is = new FileInputStream(new File("your file"));
ftpClient.storeFile(remote, is);
You should of course remember to close the input stream once you have finished with it.
This isn't exactly what you are asking, but is a fast way of reading files in bytes.
File file = new File(yourFileName);
RandomAccessFile ra = new RandomAccessFile(yourFileName, "rw"):
byte[] b = new byte[(int)file.length()];
try {
ra.read(b);
} catch(Exception e) {
e.printStackTrace();
}
//Then iterate through b
This piece of code comes handy:
private static byte[] readContentIntoByteArray(File file)
{
FileInputStream fileInputStream = null;
byte[] bFile = new byte[(int) file.length()];
try
{
//convert file into array of bytes
fileInputStream = new FileInputStream(file);
fileInputStream.read(bFile);
fileInputStream.close();
}
catch (Exception e)
{
e.printStackTrace();
}
return bFile;
}
Reference: http://howtodoinjava.com/2014/11/04/how-to-read-file-content-into-byte-array-in-java/
Getting data onto inputStream object from web url
inputStream = AWSFileUtil.getInputStream(
AWSConnectionUtil.getS3Object(null),
"cdn.generalsentiment.com", filePath);
If they are mutliple files then i want to zip them and sent the filetype as "zip" to struts.xml which does the download.
actually am converting the inputstream into byteArrayInputStream
ByteArrayInputStream byteArrayInputStream = new
ByteArrayInputStream(inputStream.toString().getBytes());
while (byteArrayInputStream.read(inputStream.toString().getBytes()) > 0) {
zipOutputStream.write(inputStream.toString().getBytes());
}
and then
zipOutputStream.close();
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
fileInputStream = new FileInputStream(file);
while (fileInputStream.read(buffer) > 0) {
byteArrayOutputStream.write(buffer);
}
byteArrayOutputStream.close();
inputStream = new ByteArrayInputStream(byteArrayOutputStream.toByteArray());
reportName = "GS_MediaValue_Reports.zip";
fileType = "zip";
}
return fileType;
But the downloaded zip when extracted gives corrupt files.
Please suggest me a solution for this issue.
The short answer is that it's not how ZipOutputStream works. Since it was designed to store multiple files, along with their file names, directory structures and so on, you need to tell the stream about that explicitly.
Furthermore, converting a stream to a string is a bad idea in general, plus it's slow, especially when you're doing it in a loop.
So your solution will be something like:
ZipEntry entry = new ZipEntry( fileName ); // You have to give each entry a different filename
zipOutputStream.putNextEntry( entry );
byte buffer[] = new byte[ 1024 ]; // 1024 is the buffer size here, but it could be anything really
int count;
while( (count = inputStream.read( buffer, 0, 1024 ) ) != -1 ) {
zipOutputStream.write( buffer, 0, count );
}