Develop Reference

Writing and Reading a MappedByteBuffer in Java

I'm learning about Memory Mapped files in java.
I would like to know how to write/read to a MappedByteBuffer.
Here's the code I'm using for writing to MappedByteBuffer.
private static void write(String strFilePath) {
File fl = new File(strFilePath);
FileChannel fChannel = null;
RandomAccessFile rf = null;
MappedByteBuffer mBBuffer = null;
try {
rf = new RandomAccessFile(fl, "rw");
fChannel = rf.getChannel();
mBBuffer = fChannel.map(FileChannel.MapMode.READ_WRITE, 0, 1024 );
for(int i =0;i<10030;i++) {
if(i == mBBuffer.limit()) {
mBBuffer = fChannel.map(FileChannel.MapMode.READ_WRITE,i+1,1024);
}
mBBuffer.put((byte)'a');
}
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
try {
if(fChannel != null) {
fChannel.close();
}
if(rf != null) {
rf.close();
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
I get BufferOverflow Exception while trying to do it.
How do I increase the size of the MappedByteBuffer after reaching the limit while writing, if I do not the know the size of the contents I will write to the file in advance?
For reading, let's take this case,
I create an MappedByteBuffer with an intial buffer size, and if reached the end of that initial size, how do i map a different portion of the file
More generally, I have a file , with byte offsets from one part of the file to another, When I read an offset (y) at a point (x), I would like to jump to (y) from (x). How do I do it?
Thanks in advance for helping me out.

Write more than 65535 rows in a csv file

[Update]Problem is different, nothing about csv file format. Question
would be "While using File Writer to write a csv file last few records
are missing.
In my java application i need to append more than 65535 rows in a csv file. but it only writes 65535 rows in a sheet. I haven't used any libraries. some final records missing. how to resolve this..........
public void writeSubmission(){
try {
writer = new FileWriter("res/sample.csv");
writer.append("PhraseId");
writer.append(',');
writer.append("Sentiment");
writer.append('\n');
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
public void readTestData(){
String path="res/test.tsv";
Calculation cal=new Calculation();
int counter=0;
try {
BufferedReader bReader = new BufferedReader(new FileReader(path));
String line;
writeSubmission();
bReader.readLine();
while ((line = bReader.readLine()) != null) {
String datavalue[] = line.split("\t");
writer.append(datavalue[0]);
writer.append(',');
try {
double value=cal.calculate(datavalue[2]);
System.out.println(value);
String val;
if(value<-0.4)
{
val="0";
}
else if(value>-0.4 && value<-0.1)
{
val="1";
}
else if(value>-0.1 && value<+0.1)
{
val="2";
}
else if(value>+0.1 && value<+0.40)
{
val="3";
}
else{
val="4";
}
counter++;
writer.append(val);
} catch (Exception e) {
// TODO Auto-generated catch block
System.out.println(e);
}
writer.append('\n');
}
System.out.println(counter);
bReader.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
System.out.println(e);
} catch (IOException e){
e.printStackTrace();
System.out.println(e);
}
}

I think your issue is probably that the tool you're opening up the CSVs with on the other end doesn't want more than 65535 rows, not that Java's doing anything wrong. It's a bug on the other end, not your Java code, almost certainly. (FileWriter wouldn't care at all about 65535 lines, for example.)
If you're using Excel 2003, for example, you'd see this issue: How to get around 64k row limit in Excel

Finally i fixed the error.File Writer should be flush after used in the code push existing stream. OR should be close the writer it automatically flush and close
The flush method flushes the output stream and forces any buffered
output bytes to be written out. The general contract of flush is that
calling it is an indication that, if any bytes previously written have
been buffered by the implementation of the output stream, such bytes
should immediately be written to their intended destination.
writer = new FileWriter("res/sample.csv");
writer.flush();
writer.close();

How to know offset of a begining of a line in text file in java?

I want to know the offset of every line present in a text file.
For now I have tried,
path=FileSystems.getDefault().getPath(".",filename);
br=Files.newBufferedReader(path_doc_title_index_path, Charset.defaultCharset());
int offset=0; //offset of first line.
String strline=br.readline();
offset+=strline.length()+1; //offset of second line
In this way I can loop through entire file to know offset of begining of lines in entire text file. But if I use RandomAccessFile to seek through file and access a line using offset calulated by above method then I found myself in the middle of some line. That is it seems that offset are not correct.
What's wrong? Is this method incorrect to calculate offset? Any better and fast methods please?

Your code will only work for ASCII encoded text. Since some characters need more than one byte, you have to change following line
offset += strline.length() + 1;
to
offset += strline.getBytes(Charset.defaultCharset()).length + 1;
As stated in my comments below your question, you have to specifiy the correct encoding of your file. E.g. Charset.forName("UTF-8") here and also where you initialize your BufferedReader.

Apparently, this gives me the expected result. In the following program I print out each line of a file through a set of offsets that I collect through the BufferedReader. Is this your case?
public static void main(String[] args) {
File readFile = new File("/your/file/here");
BufferedReader reader = null;
try
{
reader = new BufferedReader( new FileReader(readFile) );
}
catch (IOException ioe)
{
System.err.println("Error: " + ioe.getMessage());
}
List<Integer> offsets=new ArrayList<Integer>(); //offset of first line.
String strline;
try {
strline = reader.readLine();
while(strline!=null){
offsets.add(strline.length()+System.getProperty("line.separator").length()); //offset of second line
strline = reader.readLine();
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
try {
RandomAccessFile raf = new RandomAccessFile(readFile, "rw");
for(Integer offset : offsets){
try {
raf.seek(offset);
System.out.println(raf.readLine());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}

Re-reading a file after EOF using NIO (Java)

I'm using MemoryMapped buffer to read a file. Initially I'm getting the channel size and using the same size I"m mapping the file on memory and here the initial position is 0 as I want to map the file from the beginning. Now another 400KB of data is added to that file, now I want to map that 400kb alone. But something is wrong in my code, I'm not able to figure it out and I'm getting this
260java.io.IOException: Channel not open for writing - cannot extend file to required size
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:812)
at trailreader.main(trailreader.java:55
So here's my code
BufferedWriter bw;
FileInputStream fileinput = null;
try {
fileinput = new FileInputStream("simple.csv");
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
FileChannel channel = fileinput.getChannel();
MappedByteBuffer ByteBuffer;
try {
ByteBuffer = fileinput.getChannel().map(FileChannel.MapMode.READ_ONLY, 0, channel.size());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
/*
* Add some 400 bytes to simple.csv. outside of this program...
*/
//following line throw exception.
try {
ByteBuffer = fileinput.getChannel().map(FileChannel.MapMode.READ_ONLY, channel.size(), 400);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
So in my code I'm trying to reread the additional data that has been added but its not working, I know the prob is channel.size(), but I'm not able to rectify it.

channel.size() is always the current end of file. You are attempting to map 400 bytes past it. It isn't there. You need something like:
ByteBuffer = fileinput.getChannel().map(FileChannel.MapMode.READ_ONLY, channel.size()-400, 400);

Strange error reading files from a directory in Java

EDIT: children is an array of directories. This code loops trough this array in order to enter to each directory and load into the array webs all the files listed. Then, for each file, the readFile function is supposed to read the file.
My code is:
for (File cat: children) {
File[] webs = cat.listFiles();
System.out.println(" Indexing category: " + cat.getName());
for (File f: webs) {
Web w = readFile(f);
// Do things with w
}
}
I'm getting this error:
org.htmlparser.util.ParserException: Error in opening a connection to 209800.webtrec
209801.webtrec
...
422064.webtrec
422071.webtrec
422087.webtrec
422089.webtrec
422112.webtrec
422125.webtrec
422127.webtrec
;
java.io.IOException: File Name Too Long
at java.io.UnixFileSystem.canonicalize0(Native Method)
at java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:172)
at java.io.File.getCanonicalPath(File.java:576)
at org.htmlparser.http.ConnectionManager.openConnection(ConnectionManager.java:848)
at org.htmlparser.Parser.setResource(Parser.java:398)
at org.htmlparser.Parser.<init>(Parser.java:317)
at org.htmlparser.Parser.<init>(Parser.java:331)
at IndexGenerator.IndexGenerator.readFile(IndexGenerator.java:156)
at IndexGenerator.IndexGenerator.main(IndexGenerator.java:101)
It's strange because I don't see any of those files in that directory.
Thanks!
EDIT2: This is the readFile function. It loads the contents of the file into a string and parses it. Actually, files are html files.
private static Web readFile(File file) {
try {
FileInputStream fin = new FileInputStream(file);
FileChannel fch = fin.getChannel();
// map the contents of the file into ByteBuffer
ByteBuffer byteBuff = fch.map(FileChannel.MapMode.READ_ONLY,
0, fch.size());
// convert ByteBuffer to CharBuffer
// CharBuffer chBuff = Charset.defaultCharset().decode(byteBuff);
CharBuffer chBuff = Charset.forName("UTF-8").decode(byteBuff);
String f = chBuff.toString();
// Close imputstream. By doing this you close the channel associated to it
fin.close();
Parser parser = new Parser(f);
Visitor visit = new Visitor();
parser.visitAllNodesWith((NodeVisitor)visit);
return new Web(visit.getCat(), visit.getBody(), visit.getTitle());
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ParserException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return null;
}

Okay, finally I got the solution.
It was a very stupid error. I had a file in that directory that contained the names of all empty html files that I had deleted in a previous task. So, I was trying to parse it, and then the parser would interpret it like an URL and not as an htmlfile (since there aren't tags and a lot of points...). I couldn't find the file easily because I have millions of files in that folder.