I'm using MemoryMapped buffer to read a file. Initially I'm getting the channel size and using the same size I"m mapping the file on memory and here the initial position is 0 as I want to map the file from the beginning. Now another 400KB of data is added to that file, now I want to map that 400kb alone. But something is wrong in my code, I'm not able to figure it out and I'm getting this
260java.io.IOException: Channel not open for writing - cannot extend file to required size
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:812)
at trailreader.main(trailreader.java:55
So here's my code
BufferedWriter bw;
FileInputStream fileinput = null;
try {
fileinput = new FileInputStream("simple.csv");
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
FileChannel channel = fileinput.getChannel();
MappedByteBuffer ByteBuffer;
try {
ByteBuffer = fileinput.getChannel().map(FileChannel.MapMode.READ_ONLY, 0, channel.size());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
/*
* Add some 400 bytes to simple.csv. outside of this program...
*/
//following line throw exception.
try {
ByteBuffer = fileinput.getChannel().map(FileChannel.MapMode.READ_ONLY, channel.size(), 400);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
So in my code I'm trying to reread the additional data that has been added but its not working, I know the prob is channel.size(), but I'm not able to rectify it.
channel.size() is always the current end of file. You are attempting to map 400 bytes past it. It isn't there. You need something like:
ByteBuffer = fileinput.getChannel().map(FileChannel.MapMode.READ_ONLY, channel.size()-400, 400);
Related
class HelloWorld {
public static void main(String args[]) {
File file = new File("d://1.mp4");
FileInputStream fr = null;
FileOutputStream fw = null;
byte a[] = new byte[(int) file.length()];
try {
fr = new FileInputStream(file);
fw = new FileOutputStream("d://2.mp4");
fr.read(a);
fw.write(a);
fw.write(a);
fw.write(a);
fw.write(a);
fw.write(a);
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
try {
fr.close();
fw.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
Here i write fw.write(a) five times, the size of the file increases to 5x but the original 1.mp4 and copy 2.mp4 both have same length i.e. 3:30 minutes ?
Simply duplicating the bytes of certain files does not necessarily mean it simply duplicates things when inspecting them with software. For example, the video player might read the data until some terminal is encountered and not look forward. This terminal would then exist at the end of the first file data block.
You could open the new file with a hex editor and check if you can see the data of the original video file five times in a row.
FileOutputStream fooStream = new FileOutputStream("FilePath", false);
This will overwrite the content and the size of the file created will be same size as of original file.
I'm learning about Memory Mapped files in java.
I would like to know how to write/read to a MappedByteBuffer.
Here's the code I'm using for writing to MappedByteBuffer.
private static void write(String strFilePath) {
File fl = new File(strFilePath);
FileChannel fChannel = null;
RandomAccessFile rf = null;
MappedByteBuffer mBBuffer = null;
try {
rf = new RandomAccessFile(fl, "rw");
fChannel = rf.getChannel();
mBBuffer = fChannel.map(FileChannel.MapMode.READ_WRITE, 0, 1024 );
for(int i =0;i<10030;i++) {
if(i == mBBuffer.limit()) {
mBBuffer = fChannel.map(FileChannel.MapMode.READ_WRITE,i+1,1024);
}
mBBuffer.put((byte)'a');
}
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
try {
if(fChannel != null) {
fChannel.close();
}
if(rf != null) {
rf.close();
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
I get BufferOverflow Exception while trying to do it.
How do I increase the size of the MappedByteBuffer after reaching the limit while writing, if I do not the know the size of the contents I will write to the file in advance?
For reading, let's take this case,
I create an MappedByteBuffer with an intial buffer size, and if reached the end of that initial size, how do i map a different portion of the file
More generally, I have a file , with byte offsets from one part of the file to another, When I read an offset (y) at a point (x), I would like to jump to (y) from (x). How do I do it?
Thanks in advance for helping me out.
I am trying to insert and retrieve small files in couchbase, insertion is successful but when I try to fetch the content and write it to a file am getting below error.
BinaryDocument responsefromDB = bucket.get("KESAVAN", BinaryDocument.class);
try {
FileOutputStream ostream = new FileOutputStream("C:\\Satz\\Test - Copy\\Output.txt");
ostream.write(responsefromDB.content().array());
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Error :
Exception in thread "main" java.lang.UnsupportedOperationException: direct buffer
at com.couchbase.client.deps.io.netty.buffer.PooledUnsafeDirectByteBuf.array(PooledUnsafeDirectByteBuf.java:363)
at com.couchbase.client.deps.io.netty.buffer.SlicedByteBuf.array(SlicedByteBuf.java:97)
at com.couchbase.client.deps.io.netty.buffer.CompositeByteBuf.array(CompositeByteBuf.java:463)
at com.util.task.CouchbaseClient.main(CouchbaseClient.java:52)
You can only access the array() if hasArray() returns true. Otherwise the Netty buffer itself is backed by native memory. In this case you will need to use one of its getBytes(...) methods to copy the content to an array.
Don't forget to release() the buffer after obtaining it (in the finally block of your try catch for instance).
You seem to be outputing the content into a text file, so is BinaryDocument really what you're after? Maybe StringDocument would be a better, less hurdle, fit? (see http://docs.couchbase.com/developer/java-2.1/documents-basics.html).
Note that if you still have a compelling reason to use BinaryDocument and want the output as a String, you can use ByteBuf.toString(Charset) for this instead of getBytes.
i am getting out of memory error in asynctask which loop to stringbuilder . My target for using this to download image from server and store inside my sd card.My code as below :
HttpClient httpclient = new DefaultHttpClient();
httpclient.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
HttpPost httppost = new HttpPost(severPath);
httppost.setEntity(params[0]);
System.out.println("executing request " + httppost.getRequestLine());
HttpResponse response = null;
try {
response = httpclient.execute(httppost);
} catch (ClientProtocolException e6) {
// TODO Auto-generated catch block
e6.printStackTrace();
} catch (IOException e6) {
// TODO Auto-generated catch block
e6.printStackTrace();
}
String output;
System.out.println("Output from Server .... \n");
BufferedReader br = null;
try {
br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
} catch (IllegalStateException e5) {
// TODO Auto-generated catch block
e5.printStackTrace();
} catch (IOException e5) {
// TODO Auto-generated catch block
e5.printStackTrace();
}
OutputStreamWriter outputStreamWriter = null;
try {
outputStreamWriter = new OutputStreamWriter(context.openFileOutput("LargeImages.txt", context.MODE_PRIVATE));
} catch (FileNotFoundException e6) {
// TODO Auto-generated catch block
e6.printStackTrace();
}
int i = 0;
StringBuilder builder = new StringBuilder();
String Result = "";
try {
for (String line = null; (line = br.readLine()) != null ; ) {
builder.append(line.toString());
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
outputStreamWriter.close();
i am getting out of memory allocation error. please help. i try many method but also not getting the right.
if you are downloading an image, then you should not use Reader/Writer/StringBuilder to store it's content. Because the file is binary content will be scrambled because of the character encoding used by Reader/Writer classes.
Try using InputStream/OutputStream and store the content directly to sdcard without storing it in memory.
Try out the below code:
InputStream in = response.getEntity().getContent();
OutputStream out = context.openFileOutput("LargeImages.txt", context.MODE_PRIVATE);
byte b[] = new byte[4096];
int i;
while ((i = in.read(b)) >= 0) {
out.write(b, 0, i);
}
There may be two problems.
The first - the cycle for (String line = null; (line = br.readLine()) != null ; ) is not terminated properly. Try to find it out by opening a small file(e.g. with 10 lines total).
The second - it's actually a memory insufficient case. Probably it's not the best idea to get image via strings as images may be very heavy and creating a plenty of Strings causes natural memory error. Try to find another approach.
I don't see code that is actually writing to the output stream. Shouldn't there be a line before the close, that is like outputStreamWriter.print(builder)?
About your question. Instead of collecting the whole data in memory in a StringBuilder and than write it at once, you should write directly each line you get within your for-loop. You don't need the StringBuilder at all. Here's a code snippet:
try {
for (String line = br.readLine(); line != null; line = br.readLine()) {
outputStreamWriter.append(line);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return;
}
Three more remarks:
When you get an Exception you should also stop the action, e.g. return from your method. Your code above would print the Stacktrace (which is definitely helpful) but would then continue, which would be not so helpful. Just add return after each printstackTrace.
There's still a chance that one line is too long for memory, but the risk is minimized.
Is the data you download binary image or text? You name it image but you download text. Please be aware that there's a difference between bytes and characters (encoded with character set) and stay within what you actually receive.
EDIT: children is an array of directories. This code loops trough this array in order to enter to each directory and load into the array webs all the files listed. Then, for each file, the readFile function is supposed to read the file.
My code is:
for (File cat: children) {
File[] webs = cat.listFiles();
System.out.println(" Indexing category: " + cat.getName());
for (File f: webs) {
Web w = readFile(f);
// Do things with w
}
}
I'm getting this error:
org.htmlparser.util.ParserException: Error in opening a connection to 209800.webtrec
209801.webtrec
...
422064.webtrec
422071.webtrec
422087.webtrec
422089.webtrec
422112.webtrec
422125.webtrec
422127.webtrec
;
java.io.IOException: File Name Too Long
at java.io.UnixFileSystem.canonicalize0(Native Method)
at java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:172)
at java.io.File.getCanonicalPath(File.java:576)
at org.htmlparser.http.ConnectionManager.openConnection(ConnectionManager.java:848)
at org.htmlparser.Parser.setResource(Parser.java:398)
at org.htmlparser.Parser.<init>(Parser.java:317)
at org.htmlparser.Parser.<init>(Parser.java:331)
at IndexGenerator.IndexGenerator.readFile(IndexGenerator.java:156)
at IndexGenerator.IndexGenerator.main(IndexGenerator.java:101)
It's strange because I don't see any of those files in that directory.
Thanks!
EDIT2: This is the readFile function. It loads the contents of the file into a string and parses it. Actually, files are html files.
private static Web readFile(File file) {
try {
FileInputStream fin = new FileInputStream(file);
FileChannel fch = fin.getChannel();
// map the contents of the file into ByteBuffer
ByteBuffer byteBuff = fch.map(FileChannel.MapMode.READ_ONLY,
0, fch.size());
// convert ByteBuffer to CharBuffer
// CharBuffer chBuff = Charset.defaultCharset().decode(byteBuff);
CharBuffer chBuff = Charset.forName("UTF-8").decode(byteBuff);
String f = chBuff.toString();
// Close imputstream. By doing this you close the channel associated to it
fin.close();
Parser parser = new Parser(f);
Visitor visit = new Visitor();
parser.visitAllNodesWith((NodeVisitor)visit);
return new Web(visit.getCat(), visit.getBody(), visit.getTitle());
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ParserException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}
Okay, finally I got the solution.
It was a very stupid error. I had a file in that directory that contained the names of all empty html files that I had deleted in a previous task. So, I was trying to parse it, and then the parser would interpret it like an URL and not as an htmlfile (since there aren't tags and a lot of points...). I couldn't find the file easily because I have millions of files in that folder.