I am trying to read one big file in chunk . So the read operation will be called multiple times being offset one of the parameter . The read is working perfectly fine .
But the real problem is starts when I try to delete the file after read is complete . It is throwing IO exception .
I do not want to forcefully garbage collect(System.gc()) .
Read Code :
public static GenericExcelRead ReadFileContent(String fileName, int offset, String status) throws IOException
{
GenericExcelRead aGenericExcelRead = new GenericExcelRead();
//FileInputStream fileStream = null;
FileChannel fileChannel = null;
MappedByteBuffer buffer;
try(FileInputStream fileStream = new FileInputStream(fileName)) {
fileChannel = fileStream.getChannel();
buffer = null;
if (status != "Completed")
{
if(fileChannel.size()>=(offset+1048756))
{
buffer = fileChannel.map(FileChannel.MapMode.READ_ONLY, offset, 1048756);
aGenericExcelRead.setStatus("Partial");
aGenericExcelRead.setEndOffset(offset+1048756);
}
else
{
buffer = fileChannel.map(FileChannel.MapMode.READ_ONLY, offset, (fileChannel.size()-offset));
aGenericExcelRead.setStatus("Completed");
aGenericExcelRead.setEndOffset((int)fileChannel.size());
}
byte[] b = new byte[buffer.remaining()];
buffer.get(b);
String encodedcontent = new String(Base64.encodeBase64(b));
buffer.clear();
fileChannel.close();
aGenericExcelRead.setData(encodedcontent);
fileStream.close();
}
} catch (IOException e) {
throw new IOException("IO Exception/File not found");
}finally {
if(fileChannel != null)
fileChannel.close();
}
return aGenericExcelRead;
}
Related
In my applet I have GET call to download file from a remote location. When I am trying to download some large file of around 13MB, then my Applet memory consumption is increasing more than 50MB. I am using the below code to get my memory consumption:
public static long getMemoryUsage()
{
long memory = 0;
// Get the Java runtime
Runtime runtime = Runtime.getRuntime();
memory = runtime.totalMemory() - runtime.freeMemory();
return memory;
}
Code for my get call is
public void getFiles(String filePath, long fileSize)throws MyException
{
InputStream objInputStream = null;
HttpURLConnection conn = null;
BufferedReader br = null;
try
{
URL fileUrl=new URL(filePath);
final String strAPICall=fileUrl.getPath();
final String strHost="some.test.com";
final int iPort=1000;
URL url = null;
url = new java.net.URL
( "https",
strHost, iPort , "/" + strAPICall,
new myHandler() );
conn = (HttpURLConnection)new HttpsURLConn(url);
conn.setRequestMethod("GET");
conn.connect();
if (conn.getResponseCode() != 200) {
objInputStream=conn.getInputStream();
br = new BufferedReader(new InputStreamReader(
(objInputStream)));
String output;
while ((output = br.readLine()) != null) {
System.out.println(output);
}
throw new MyException("Bad response from server",
MyError.BAD_RESPONSE_ERROR);
}
else
{
notifyProgressToObservers(0);
System.out.println("conn.getResponseCode()"+conn.getResponseCode());
System.out.println("conn.getResponseMessage()"+conn.getResponseMessage());
objInputStream = conn.getInputStream();
int count=objInputStream.available();
System.out.println("Stream size: "+count);
System.out.println("fileSize size: "+fileSize);
byte []downloadedData = getBytesFromInputStream
(objInputStream, count,fileSize);
notifyChunkToObservers(downloadedData);
notifyIndivisualFileEndToObservers(true, null);
}
}
catch (MyException pm)
{
throw new MyException
(pm, MyError.CONNECTION_TIMEOUT);
}
catch (IOException pm)
{
throw new MyException
(pm, MyError.CONNECTION_TIMEOUT);
}
catch (Exception e)
{
notifyIndivisualFileEndToObservers(false,new MyException(e.toString()));
}
finally
{
System.out.println("Closing all the streams after getting file");
if(conn !=null)
{
try
{
conn.disconnect();
}
catch(Exception e)
{
}
}
if(objInputStream != null)
{
try {
objInputStream.close();
} catch (IOException e) {
}
}
if (br != null)
{
try {
br.close();
} catch (IOException e) {
}
}
}
}
In the above method, I tried putting the log for memory consumption after each line and found that after conn.connect();, the memory consumption of applet increases by atleast 50MB even though the file I am trying to download is only 13MB.
Is there any memory leak anywhere?
EDIT: Added Implementation for getBytesFromInputStream()
public byte[] getBytesFromInputStream(InputStream is, int len, long fileSize)
throws IOException
{
byte[] readBytes= new byte[8192];
ByteArrayOutputStream getBytes= new ByteArrayOutputStream();
int numRead = 0;
while ((numRead = is.read(readBytes)) != -1) {
getBytes.write(readBytes, 0, numRead);
}
return getBytes.toByteArray();
}
it's because of this line:
byte []downloadedData = getBytesFromInputStream(objInputStream, count,fileSize);
here you are reading the complete amount of bytes of file into the heap. After that you need to track down what happens with this array. Maybe you are copying it somewhere and the GC needs some time to kick in even if you do not use the reference to the object anymore.
Large files should never be read completly to memory, but rather streamed directly to some processor of the data.
The only way to optimize getBytesFromInputStream() is if you know beforehand exactly how many by bytes there are to read. Then you allocate a byte[] of the required size, and read from the input directly into the byte[]. For example:
byte[] buffer = new byte[len];
int pos = 0;
while (pos < len) {
int nosRead = is.read(buffer, pos, len - pos);
if (nosRead == -1) {
throw new IOException("incomplete response");
}
pos += nosRead;
}
return buffer;
(For more information, read the javadoc.)
Unfortunately, your (apparent) attempt at getting the size is incorrect.
int count = objInputStream.available();
This doesn't return the total number of bytes that can be read from the stream. It returns the number of bytes that can be read right now without the possibility of blocking.
If the server is setting the Content-Length header in the response, then you could use that; call getContentLength() (or getContentLengthLong() in other use-cases) once you have the response. But be prepared for the case where that gives you -1.
My application streams twitter data and writes them to files.
while(true){
Status status = queue.poll();
if (status == null) {
Thread.sleep(100);
}
if(status!=null){
list.add(status);
}
if(list.size()==10){
FileOutputStream fos = null;
ObjectOutputStream out = null;
try {
String uuid = UUID.randomUUID().toString();
String filename = "C:/path/"+topic+"-"+uuid+".ser";
fos = new FileOutputStream(filename);
out = new ObjectOutputStream(fos);
out.writeObject(list);
tweetsDownloaded += list.size();
if(tweetsDownloaded % 100==0)
System.out.println(tweetsDownloaded+" tweets downloaded");
// System.out.println("File: "+filename+" written.");
out.close();
} catch (IOException e) {
e.printStackTrace();
}
list.clear();
}
I have this code which gets data from files.
while(true){
File[] files = folder.listFiles();
if(files != null){
Arrays.sort(//sorting...);
//Here we manage each single file, from data-load until the deletion
for(int i = 0; i<files.length; i++){
loadTweets(files[i].getAbsolutePath());
//TODO manageStatuses
files[i].delete();
statusList.clear();
}
}
}
The method loadTweets() does the following operations:
private static void loadTweets(String filename) {
FileInputStream fis = null;
ObjectInputStream in = null;
try{
fis = new FileInputStream(filename);
in = new ObjectInputStream(fis);
statusList = (List<Status>) in.readObject();
in.close();
}
catch(IOException | ClassNotFoundException ex){
ex.printStackTrace();
}
}
Unfortunately, I don't know why sometimes it throws a
EOFException
when running this line
statusList = (List<Status>) in.readObject();
Anybody knows how I can solve this? Thank you.
I've seen that you're passing the file correctly with the getAbsolutePath() based on a previous question of yours
From what I've read that can be a couple of things, one of them the file being null.
Explaining this idea, you might have written the file but something caused the file to have nothing inside, this might cause an EOFException. The file in fact exists it's just empty
EDIT
Try to enclose the code in while(in.available() > 0)
It would look like this
private static void loadTweets(String filename) {
FileInputStream fis = null;
ObjectInputStream in = null;
try{
fis = new FileInputStream(filename);
in = new ObjectInputStream(fis);
while(in.available() > 0) {
statusList = (List<Status>) in.readObject();
}
in.close();
}
catch(IOException | ClassNotFoundException ex){
ex.printStackTrace();
}
}
Found out what was necessary to solve this. Thanks to #VGR's comment, I thought to pause the executing thread for 0.2 seconds if the file has been created less than a second ago.
if(System.currentTimeMillis()-files[i].lastModified()<1000){
Thread.sleep(200);
This prevents the exception and the application works now fine.
I have written a java controller which handles any download request from the server. My files are present in the server however I am getting 0 kB file downloaded every time. The file is getting downloaded but the size is always 0 Kb. Please help me. Here is my code -
#RequestMapping(value="/downloadFile/{docId}")
public void getDownloadFile(#PathVariable(value = "docId") Integer docId, HttpServletResponse response) throws Exception {
String userName = getUserName();
try {
DocVault documentsVault = documentsVaultRepository.findDocumentAttachment(docId);
String fileName = documentsVault.getDocumentName();
int customerId = documentsVault.getCustomerId();
Map<Integer, String> customerInfo = cspUtils.getCustomersInfo(userName);
Set<Integer> customerIds = customerInfo.keySet();
for (int custId : customerIds) {
if (custId == customerId) {
String path = env.getProperty("doc.rootfolder") + File.separator + documentsVault.getFileName();
service.downloadFile(fileName, path, response);
} else {
logger.info("Customer not linked to user");
}
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
Implementation -
public void downloadFile(String fileName, String path, HttpServletResponse response) {
try {
File downloadFile = new File(path);
FileInputStream inputStream = new FileInputStream(downloadFile);
response.setContentLength((int) downloadFile.length());
response.setHeader("Content-Disposition", "attachment;filename=" + fileName);
// get output stream of the response
OutputStream outStream = response.getOutputStream();
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead = -1;
// write bytes read from the input stream into the output stream
while ((bytesRead = inputStream.read(buffer)) != -1) {
outStream.write(buffer, 0, bytesRead);
}
inputStream.close();
outStream.close();
} catch (Exception e) {
e.printStackTrace();
}
}
I am not getting any exceptions. Please help.
Thank you in advance.
response.setContentLength((int) downloadFile.length());
Remove this. The container will set it automatically.
int bytesRead = -1;
You don't need to initialize this variable. It gets assigned in the very next line.
Thank you for your help. What I figured out was there was a problem with my Content Disposition header. I was passing only the file name without extension. When I passed the full file name then it worked perfectly. Size and Extension of the file were proper.
I am trying to query from database with Java JDBC and compress data in one column to gzip file in specific directory. I have tested my JDBC query and it working fine, but the Gzip code not going with the while loop, it's run with the loop firt row and stuck there. Why it's stuck? help me please!
These folders already existed: D:\Directory\My\year\id1\id2
//Some query JDBC code here, it's work well. I query all rows Data, year, id1,id2,id3
while (myRs1.next()){
String str = Data;
File myGzipFile = new File("D:\\Directory\\My\\"+year+"\\"+id1+"\\"+id2+"\\"+id3+".gzip");
GZIPOutputStream gos = null;
InputStream is = new ByteArrayInputStream(str.getBytes());
gos = new GZIPOutputStream(new FileOutputStream(myGzipFile));
byte[] buffer = new byte[1024];
int len;
while ((len = is.read(buffer)) != -1) {
gos.write(buffer, 0, len);
System.out.print("done for:"+id3);
}
try { gos.close(); } catch (IOException e) { }
}
Try formatting the source like this to catch exceptions.
public class InputStreamDemo {
public static void main(String[] args) throws Exception {
InputStream is = null;
int i;
char c;
try{
is = new FileInputStream("C://test.txt");
System.out.println("Characters printed:");
// reads till the end of the stream
while((i=is.read())!=-1)
{
// converts integer to character
c=(char)i;
// prints character
System.out.print(c);
}
}catch(Exception e){
// if any I/O error occurs
e.printStackTrace();
}finally{
// releases system resources associated with this stream
if(is!=null)
is.close();
}
}
}
I have a question regarding internal files in android.. I tried to write some data into a file and then read it back however, it seems like I can't write data to a file unless I cast it to an integer first.. is there anyway that I can save double or float values.. I added the code I'm trying to use below:
FormatCluster formatCluster = ((FormatCluster)objectCluster.returnFormatCluster(ofFormats,"Calibrated"));
if (formatCluster != null) {
//Obtain data for text view
calibratedDataArray[0] = formatCluster.mData;
calibratedUnits = formatCluster.mUnits;
A.setText("data: " + formatCluster.mData);
String filename = "myfile";
//String string = "Hello world!";
FileOutputStream outputStream;
try {
outputStream = openFileOutput(filename, Context.MODE_PRIVATE);
outputStream.write((int)formatCluster.mData);//here I don't want to cast the value to integer
outputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
//testing.setText)
double ch;
StringBuffer fileContent = new StringBuffer("");
FileInputStream fis;
try {
fis = context.openFileInput( filename );
try {
while( (ch = fis.read()) != -1)
testing.setText(fileContent.append(ch));
} catch (IOException e) {
e.printStackTrace();
}
} catch (FileNotFoundException e) {
e.printStackTrace();
}
Envelope your InputStream and OutputStream with a DataInputStream and a DataOutputStream. These classes have the method you need