I'm programming a conversion tool that should take a json file as an input. Due to the problem that json input files may be very large I'll perform a split at the beginning. I have to resort the structure of the input file and because I don't want to keep the whole large file in my memory all the time, I'll split it.
I'm proofing the occurences of a json block by counting the { and }. If the counter is 0, a split should be performed. That's where the problem is: if the file is about 40MBs large the JVM throws a StackOverFlowError and I don't know why.
Here's what I do to perform a split:
/*
* Utility-function to read a json file part by part and save the parts to a separate json file.
* #param scanner The scanner which contains the file and which returns the lines from the file.
* #param j The counter of the file. As the file should change whenever the counter changes.
* #return jsonString The content of the jsonString.
*/
public String readPartOfFileAndSave(String filepath, Scanner scanner, int j) {
String jsonString = "";
int i = 0;
++j;
while (scanner.hasNext()) {
String token = scanner.next();
jsonString += token + System.lineSeparator();
if (token.contains("{")) {
i++;
}
if (token.contains("}")) {
i--;
}
if (i == 0) {
// DEBUG
// if (j % 1000 == 0) {
// System.gc();
// System.out.println("Garbage Collector called manually");
// }
saveFile(filepath, "actor", j, jsonString);
readPartOfFileAndSave(filepath, scanner, j);
}
}
return "";
}
/*
* #param filename The name of the target file where the content is saved to.
* #param j The counter of the file. As the file should change whenever the counter changes.
* #param content The content of the file.
*/
public void saveFile(String filepath, String fileprefix, int j, String content) {
File file = null;
if (this.osValidator.isWindows()) {
file = new File(filepath + "\\" + fileprefix + "" + j + ".json");
} else {
file = new File(filepath + "/" + fileprefix + "" + j + ".json");
}
try {
// if file doesnt exists, then create it
if (!file.exists()) {
file.createNewFile();
}
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(file.getAbsoluteFile()), "UTF-8"));
bw.write(content);
bw.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
The exception looks like this:
Exception in thread "AWT-EventQueue-0" java.lang.StackOverflowError
at java.util.Hashtable.get(Hashtable.java:363)
at java.util.Properties.getProperty(Properties.java:969)
at java.lang.System.getProperty(System.java:717)
at sun.security.action.GetPropertyAction.run(GetPropertyAction.java:84)
at sun.security.action.GetPropertyAction.run(GetPropertyAction.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.BufferedWriter.<init>(BufferedWriter.java:109)
at java.io.BufferedWriter.<init>(BufferedWriter.java:88)
at utility.ReadFileAndSave.saveFile(ReadFileAndSave.java:183)
at utility.ReadFileAndSave.readPartOfFileAndSave(ReadFileAndSave.java:158)
at utility.ReadFileAndSave.readPartOfFileAndSave(ReadFileAndSave.java:159)
splitFile() is called in a separate class. But nothing else happens there.
public void splitFile() {
try {
ReadFileAndSave reader = new ReadFileAndSave();
String jsonFilePath = this.converterView.sourceTextField.getText();
File jsonFile = new File(jsonFilePath);
Scanner scanner = new Scanner(new FileInputStream(jsonFile), "UTF-8");
int j = 0;
File newDir = null;
if (this.osValidator.isWindows()) {
newDir = new File(System.getProperty("user.home") + "\\temp\\");
} else {
newDir = new File(System.getProperty("user.home") + "/temp/");
}
if (!newDir.exists()) {
newDir.mkdir();
}
if (this.osValidator.isWindows()) {
reader.readPartOfFileAndSave(System.getProperty("user.home") + "\\temp\\", scanner, j);
} else {
reader.readPartOfFileAndSave(System.getProperty("user.home") + "/temp/", scanner, j);
}
} catch (FileNotFoundException ex) {
Logger.getLogger(FilterController.class.getName()).log(Level.SEVERE, null, ex);
}
}
The input file has about 38.000 blocks in it. About 5.200 will be splitted to separate files. Then the exception is thrown.
The JVM seems to have a problem with the BufferedWriter. Does anyone know how to fix this issue?
Related
I am trying to delete a json file I just created, The file is in the same directory as the workplace so I didn't include it when creating the new File object.
I also closed the output stream before trying to delete the file by using .close() method.
import java.io.File;
public static void processFilesForValidation(String name, Scanner obj, PrintWriter logFile) {
int count = 0;
PrintWriter outputStream = null;
String[] line = obj.nextLine().split(",", -1);
if (count == 0) {
for (int i = 0; i < line.length; i++) {
try {
if (line[i].equals(" ")) {
throw new CSVFileInvalidException();
} else {
try {
outputStream = new PrintWriter((new FileOutputStream(name + ".json", true)));
} catch (FileNotFoundException e) {
System.out.println("Could not create json file " + name
+ ". Program will terminate after deleting any corrupted files and closing any opened files.");
obj.close();
outputStream.close();
String fname = name + ".json";
File file = new File(fname);
file.delete();
System.exit(0);
}
}
} catch (CSVFileInvalidException eCsvFileInvalidException) {
System.out.println(
"File " + name + ".CSV is invlaid: field is missing. \nFile is not converted to JSON.");
int detected = line.length - 1;
logFile.println("\nFile " + name + ".CSV is invalid. \nMissing field:" + detected
+ " detected, 1 missing");
for (int j = 0; j < line.length; j++) {
if (line[j].equals(" ")) {
logFile.print(line[j] + "***, ");
} else {
logFile.print(line[j] + ", ");
}
}
logFile.close();
outputStream.close();
String fname= name+".JSON";
File file = new File(fname);
file.delete();
break;
}
}
}
}
The code is a little long but this is the method, I hope you can help me.
You open a new outputStream every time around the loop. As far as I can tell, you only close it on a couple of errors. So, on loop #2, the first outputStream is left dangling and not closed.
I have a I/O java file, a SDive file which contains main, a .txt Directory file that has random words in it and a Sorted .txt to return the random words in order ascending to descending. It print all the words in the Sorted but it is not sorted.
//Sort.java
// required for input
import java.io.FileReader;
import java.io.BufferedReader;
import java.io.FileNotFoundException;
// required for output
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
class Sort{
private String[] tArr = new String[100];
public void swap(int j) {
String temp = tArr[j-1];
tArr[j-1] = tArr[j];
tArr[j] = temp;
}
///Bubble sort
public void sort() {
int n = tArr.length;
for(int i = 0; i < n; i++) {
for(int j = 1; j < n-i; j++) {
if(tArr[j-1].compareTo(tArr[j]) > 0) {
swap(j);
}
}
}
}
public void read() {
System.out.println("in read()");
String pwd = System.getProperty("user.dir");
String fileName = pwd + "/directory.txt";
System.out.println("Looking for: " + fileName);
String fileLine = "";
try {
File f = new File(fileName);
if(f.exists()) {
System.out.println("Directory profile found, loading data...");
}
else {
System.out.println("Directory profile not found, loading default...");
return; // done, return back to the caller
}
// Read file
FileReader data = new FileReader(fileName);
// Wrap FileReader with BufferedReader
BufferedReader br = new BufferedReader(data);
//String tmp;
int i=0;
while ((fileLine = br.readLine()) != null) {
tArr[i++] = fileLine;
}
// ok, time to load an existing profile
// close the file
br.close();
} catch (FileNotFoundException fnf) {
System.out.println("File not found: " + fileName);
} catch (IOException ioe) {
System.out.println("Error reading file: " + fileName);
} catch (Exception exc) {
System.out.println("Something went wrong: " + fileName);
}
}
public void write() {
System.out.println("in write()");
String pwd = System.getProperty("user.dir");
String fileName = pwd + "/Sorted.txt";
try {
System.out.println("Writing out to: " + fileName);
File file = new File(fileName);
// creates the file
file.createNewFile();
// create FileWriter object
FileWriter writer = new FileWriter(file);
// output to file
// ADD pIdx, pArr, mood, and anything else here...
for(int i = 0; i < tArr.length; i++) {
writer.write(tArr[i] + "\n");
}
writer.flush();
writer.close();
} catch (IOException ex) {
System.out.println("Error writing to file: " + fileName);
}
}
}
And my main file is SDrive:
class SDriver{
public static void main(String args []){
Sort io = new Sort();
io.read();
io.write();
}
}
You should add the io.sort() method after io.read(); line and just before io.write(); line.
I see you use bubble sort, if you really want to implement your own sorting method take a look at quick sort and merge sort which are much much faster than bubble sort on larger arrays but also harder to implement. Insertion and Selection sort are not as fast as merge or quick but still faster than bubble and still as easy to self implement. Or use Arrays.sort(tArr); if you want to do it quick.
I am have a project that need to modify some text in the text file.
Like BB,BO,BR,BZ,CL,VE-BR
I need make it become BB,BO,BZ,CL,VE.
and HU, LT, LV, UA, PT-PT/AR become HU, LT, LV, UA,/AR.
I have tried to type some code, however the code fail to loop and also,in this case.
IN/CI, GH, KE, NA, NG, SH, ZW /EE, HU, LT, LV, UA,/AR, BB
"AR, BB,BO,BR,BZ,CL, CO, CR, CW, DM, DO,VE-AR-BR-MX"
I want to delete the AR in second row, but it just delete the AR in first row.
I got no idea and seeking for helps.
Please
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileWriter;
import java.util.Scanner;
public class tomy {
static StringBuffer stringBufferOfData = new StringBuffer();
static StringBuffer stringBufferOfData1 = stringBufferOfData;
static String filename = null;
static String input = null;
static String s = "-";
static Scanner sc = new Scanner(s);
public static void main(String[] args) {
boolean fileRead = readFile();
if (fileRead) {
replacement();
writeToFile();
}
System.exit(0);
}
private static boolean readFile() {
System.out.println("Please enter your files name and path i.e C:\\test.txt: ");
filename = "C:\\test.txt";
Scanner fileToRead = null;
try {
fileToRead = new Scanner(new File(filename));
for (String line; fileToRead.hasNextLine()
&& (line = fileToRead.nextLine()) != null;) {
System.out.println(line);
stringBufferOfData.append(line).append("\r\n");
}
fileToRead.close();
return true;
} catch (FileNotFoundException ex) {
System.out.println("The file " + filename + " could not be found! "+ ex.getMessage());
return false;
} finally {
fileToRead.close();
return true;
}
}
private static void writeToFile() {
try {
BufferedWriter bufwriter = new BufferedWriter(new FileWriter(
filename));
bufwriter.write(stringBufferOfData.toString());
bufwriter.close();
} catch (Exception e) {// if an exception occurs
System.out.println("Error occured while attempting to write to file: "+ e.getMessage());
}
}
private static void replacement() {
System.out.println("Please enter the contents of a line you would like to edit: ");
String lineToEdit = sc.nextLine();
int startIndex = stringBufferOfData.indexOf(lineToEdit);
int endIndex = startIndex + lineToEdit.length() + 2;
String getdata = stringBufferOfData.substring(startIndex + 1, endIndex);
String data = " ";
Scanner sc1 = new Scanner(getdata);
Scanner sc2 = new Scanner(data);
String lineToEdit1 = sc1.nextLine();
String replacementText1 = sc2.nextLine();
int startIndex1 = stringBufferOfData.indexOf(lineToEdit1);
int endIndex1 = startIndex1 + lineToEdit1.length() + 3;
boolean test = lineToEdit.contains(getdata);
boolean testh = lineToEdit.contains("-");
System.out.println(startIndex);
if (testh = true) {
stringBufferOfData.replace(startIndex, endIndex, replacementText1);
stringBufferOfData.replace(startIndex1, endIndex1 - 2,
replacementText1);
System.out.println("Here is the new edited text:\n"
+ stringBufferOfData);
} else {
System.out.println("nth" + stringBufferOfData);
System.out.println(getdata);
}
}
}
I wrote a quick method for you that I think does what you want, i.e. remove all occurrences of a token in a line, where that token is embedded in the line and is identified by a leading dash.
The method reads the file and writes it straight out to a file after editing for the token. This would allow you to process a huge file without worrying about about memory constraints.
You can simply rename the output file after a successful edit. I'll leave it up to you to work that out.
If you feel you really must use string buffers to do in memory management, then grab the logic for the line editing from my method and modify it to work with string buffers.
static void onePassReadEditWrite(final String inputFilePath, final String outputPath)
{
// the input file
Scanner inputScanner = null;
// output file
FileWriter outputWriter = null;
try
{
// open the input file
inputScanner = new Scanner(new File(inputFilePath));
// open output file
File outputFile = new File(outputPath);
outputFile.createNewFile();
outputWriter = new FileWriter(outputFile);
try
{
for (
String lineToEdit = inputScanner.nextLine();
/*
* NOTE: when this loop attempts to read beyond EOF it will throw the
* java.util.NoSuchElementException exception which is caught in the
* containing try/catch block.
*
* As such there is NO predicate required for this loop.
*/;
lineToEdit = inputScanner.nextLine()
)
// scan all lines from input file
{
System.out.println("START LINE [" + lineToEdit + "]");
// get position of dash in line
int dashInLinePosition = lineToEdit.indexOf('-');
while (dashInLinePosition != -1)
// this line has needs editing
{
// split line on dash
String halfLeft = lineToEdit.substring(0, dashInLinePosition);
String halfRight = lineToEdit.substring(dashInLinePosition + 1);
// get token after dash that is to be removed from whole line
String tokenToRemove = halfRight.substring(0, 2);
// reconstruct line from the 2 halves without the dash
StringBuilder sb = new StringBuilder(halfLeft);
sb.append(halfRight.substring(0));
lineToEdit = sb.toString();
// get position of first token in line
int tokenInLinePosition = lineToEdit.indexOf(tokenToRemove);
while (tokenInLinePosition != -1)
// do for all tokens in line
{
// split line around token to be removed
String partLeft = lineToEdit.substring(0, tokenInLinePosition);
String partRight = lineToEdit.substring(tokenInLinePosition + tokenToRemove.length());
if ((!partRight.isEmpty()) && (partRight.charAt(0) == ','))
// remove prefix comma from right part
{
partRight = partRight.substring(1);
}
// reconstruct line from the left and right parts
sb.setLength(0);
sb = new StringBuilder(partLeft);
sb.append(partRight);
lineToEdit = sb.toString();
// find next token to be removed from line
tokenInLinePosition = lineToEdit.indexOf(tokenToRemove);
}
// handle additional dashes in line
dashInLinePosition = lineToEdit.indexOf('-');
}
System.out.println("FINAL LINE [" + lineToEdit + "]");
// write line to output file
outputWriter.write(lineToEdit);
outputWriter.write("\r\n");
}
}
catch (java.util.NoSuchElementException e)
// end of scan
{
}
finally
// housekeeping
{
outputWriter.close();
inputScanner.close();
}
}
catch(FileNotFoundException e)
{
e.printStackTrace();
}
catch(IOException e)
{
inputScanner.close();
e.printStackTrace();
}
}
I have a Method that calls a second method, the second method will:
Create any missing directories
Create a file
Decode a 2D String[] to a String (Not working)
Write content
Write the decoded String to the file with a header (Not working)
First method
public static boolean store(Exception exception, String[][] flags){
return storePrivate(exception, location, flags);
}
Second Method (Not all code just relevant code)
private static boolean storePrivate(Exception exception, String dir, String[][] flags){
String flag = "";
for(int i = 0; i >= flags.length; i++){
flag = flag + "" + flags[i][0] + ": " + flags[i][1] + "\n";
}
try {
File directory = new File(dir);
File file = new File(dir + id + ".txt");
if (!directory.exists()) {
directory.mkdirs();
}
file.createNewFile();
FileWriter filewriter = new FileWriter(file.getAbsoluteFile());
BufferedWriter writer = new BufferedWriter(filewriter);
if(flag != ""){
writer.write("Flags by Developer: ");
writer.write(flag);
}
writer.flush();
writer.close();
return true;
} catch (IOException e) {
return false;
}
}
Call to the first method
public static void main(String[] args) {
try {
test();
} catch (Exception e) {
// TODO Auto-generated catch block
ExceptionAPI.store(e, new String[][]{{"flag1", "Should be part of flag1"}, {"flag2", "this should be flag 2 contence"}});
}
}
public static void test() throws IOException{
throw new IOException();
}
I cant find why this won't work. I think it has to do with the second method, particularly
if(flag != ""){
writer.write("Flags by Developer: ");
writer.write(flag);
}
Thanks if anyone can help me.
Curlip
Try this if you want to just convert an array of strings into a single string:
String[] stringTwoD = //... I think this is a 1D array, and Sting[][] is a 2D, anyway
String stringOneD = "";
for (String s : stringTwoD)
stringOneD += s;//add the string with the format you want
BTW, your loop condition seems wrong and ,so you may change it to :
for(int i = 0; i < flags.length; i++){
flag += flags[i][0] + ": " + flags[i][1] + "\n";
}
I have a list of log files, and I need to find which one has a latest edition of a specific line, and all or none could have this line.
The lines in the files look like this:
2013/01/06 16:01:00:283 INFO ag.doLog: xxxx xxxx xxxx xxxx
And I need a line lets say
xx/xx/xx xx:xx:xx:xxx INFO ag.doLog: the line i need
I know how to get an array of files, and if I scan backwards I could find the latest latest line in each file (if it exists).
Biggest problem is that the file could be big (2k lines?) and I want to find the line in a relative fast way (a few seconds), so I am open for suggestion.
Personal ideas:
If a file has the line at X time, then any file that has not found the line before X time should not be scan anymore. This will require to search all files at the same time, which i dont know how.
Atm the code breaks, and I suppose if lack of memory.
Code:
if(files.length>0) { //in case no log files exist
System.out.println("files.length: " + files.length);
for(int i = 0; i < files.length; i++) { ///for each log file look for string
System.out.println("Reading file: " + i + " " + files[i].getName());
RandomAccessFile raf = new RandomAccessFile(files[i].getAbsoluteFile(), "r"); //open log file
long lastSegment = raf.length(); //Finds how long is the files
lastSegment = raf.length()-5; //Sets a point to start looking
String leido = "";
byte array[] = new byte[1024];
/*
* Going back until we find line or file is empty.
*/
while(!leido.contains(lineToSearch)||lastSegment>0) {
System.out.println("leido: " + leido);
raf.seek(lastSegment); //move the to that point
raf.read(array); //Reads 1024 bytes and saves in array
leido = new String(array); //Saves what is read as a string
lastSegment = lastSegment-15; //move the point a little further back
}
if(lastSegment<0) {
raf.seek(leido.indexOf(lineToSearch) - 23); //to make sure we get the date (23 characters long) NOTE: it wont be negative.
raf.read(array); //Reads 1024 bytes and saves in array
leido = new String(array); //make the array into a string
Date date = new SimpleDateFormat("MMMM d, yyyy", Locale.ENGLISH).parse(leido.substring(0, leido.indexOf(" INFO "))); //get only the date part
System.out.println(date);
//if date is bigger than the other save file name
}
}
}
I find the code difficult to verify. One could split the task in a backwards reader, which reads lines from file end to start. And use that for parsing dates line wise.
Mind, I am not going for nice code, but something like this:
public class BackwardsReader implements Closeable {
private static final int BUFFER_SIZE = 4096;
private String charset;
private RandomAccessFile raf;
private long position;
private int readIndex;
private byte[] buffer = new byte[BUFFER_SIZE];
/**
* #param file a text file.
* #param charset with bytes '\r' and '\n' (no wide chars).
*/
public BackwardsReader(File file, String charset) throws IOException {
this.charset = charset;
raf = new RandomAccessFile(file, "r");
position = raf.length();
}
public String readLine() throws IOException {
if (position + readIndex == 0) {
raf.close();
raf = null;
return null;
}
String line = "";
for (;;) { // Loop adding blocks without newline '\n'.
// Search line start:
boolean lineStartFound = false;
int lineStartIndex = readIndex;
while (lineStartIndex > 0) {
if (buffer[lineStartIndex - 1] == (byte)'\n') {
lineStartFound = true;
break;
}
--lineStartIndex;
}
String line2;
try {
line2 = new String(buffer, lineStartIndex, readIndex - lineStartIndex,
charset).replaceFirst("\r?\n?", "");
readIndex = lineStartIndex;
} catch (UnsupportedEncodingException ex) {
Logger.getLogger(BackwardsReader.class.getName())
.log(Level.SEVERE, null, ex);
return null;
}
line = line2 + line;
if (lineStartFound) {
--readIndex;
break;
}
// Read a prior block:
int toRead = BUFFER_SIZE;
if (position - toRead < 0) {
toRead = (int) position;
}
if (toRead == 0) {
break;
}
position -= toRead;
raf.seek(position);
raf.readFully(buffer, 0, toRead);
readIndex = toRead;
if (buffer[readIndex - 1] == (byte)'\r') {
--readIndex;
}
}
return line;
}
#Override
public void close() throws IOException {
if (raf != null) {
raf.close();
}
}
}
And a usage example:
public static void main(String[] args) {
try {
File file = new File(args[0]);
BackwardsReader reader = new BackwardsReader(file, "UTF-8");
int lineCount = 0;
for (;;) {
String line = reader.readLine();
if (line == null) {
break;
}
++lineCount;
System.out.println(line);
}
reader.close();
System.out.println("Lines: " + lineCount);
} catch (IOException ex) {
Logger.getLogger(App.class.getName()).log(Level.SEVERE, null, ex);
}
}