I have a huge CSV file.I have to read it and store in a database using java.below code only read about 2000 rows from that file and store it into database.why? note the below calculation is to change the seconds to minutes with approximation.dont think a lot about that
import com.opencsv.CSVReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;
public class ReadCSV {
public static void main(String[] args) throws SQLException{
MainController mc = new MainController();
boolean status=false;
String csvFile = "C:/Users/Thanushiya/Desktop/mobios/internship/csvfile/csvfile/Master.csv";
CSVReader reader = null;
List myList = new ArrayList();
String[] row = null;
int duration_m = 0;
try {
reader = new CSVReader (new FileReader(csvFile), ',', '\'', 17);
myList = reader.readAll();
int i=0;
for (Object object : myList) {
row = (String[]) object;
float duration_float = Float.parseFloat(row[12]) / 60 ;
float duration_mod = duration_float % 1 ;
if(Integer.parseInt(row[12]) <= 60){
duration_m = 1;
}
else{
if(duration_mod == 0 ){
duration_m = (int) duration_float;
}
else{
duration_m = ((int) duration_float) + 1;
}
}
String query = "INSERT INTO master(msisdn,serv,start,end,ringwithdurartion,duration_s,status,cid,duration_m) values ('"+row[0]+"','"+row[4]+"','"+row[8]+"','"+row[10]+"','"+row[11]+"','"+row[12]+"','"+row[13]+"','"+row[15]+"','"+duration_m+"')";
status=mc.insertData(query);
}//end for
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
System.out.println("Done");
}
}
With readAll() you're loading (or at least trying to load) the entire file into memory at once. That will severely limit the number of lines you can read before running out of memory. As indicated on the OpenCSV homepage, there's also an iterator that reads line per line:
CSVReader reader = new CSVReader(new FileReader("C:/Users/Thanushiya/Desktop/mobios/internship/csvfile/csvfile/Master.csv"), ',', '\'', 17);
String [] nextLine;
while ((nextLine = reader.readNext()) != null) {
// put the code from your for loop here
}
Related
I have text file data like :
2,2,1
data1,123,89,1
data2,124,90,2
data3,125,91,3
data4,126,92,4
data5,127,93,5
data6,128,94,6
data7,129,95,7
data8,130,96,8
data9,131,97,9
data10,132,98,10
The first line 2,2,1 indicate 2 lines from 1st set of lines and store it in nodeFile, 2 lines from 2nd set of lines store it in linkFile and 1 line from 3rd set of lines store it in moduleFile. However for example purpose I have shows small number of lines but its a larger file.
import java.io.File;
import java.io.FileNotFoundException;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;
public class ReadFile {
static List<String> moduleFile = new ArrayList<>();
static List<String> linkFile = new ArrayList<>();
static List<String> nodeFile = new ArrayList<>();
static int a[];
public static void main(String[] args) {
File file11 = new File("/home/madhu/Desktop/node.txt");
Scanner scAll = null;
try {
scAll = new Scanner(file11);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
String[] numberOfLines = (scAll.nextLine()).split(",");
int flag = 0;
int counter = 1;
while (scAll.hasNext()) {
if (flag == 0 && "\\n\\n".equals(scAll.nextLine()) && counter <= Integer.parseInt(numberOfLines[0].trim())) {
for (int i = 0; i < Integer.parseInt(numberOfLines[0].trim()); i++) {
System.out.println(scAll.nextLine());
nodeFile.add(scAll.nextLine());
counter++;
}
if (counter > Integer.parseInt(numberOfLines[0].trim())) {
flag = 1;
counter = 1;
}
} else if (flag == 1 && "\\n\\n".equals(scAll.nextLine())
&& counter <= Integer.parseInt(numberOfLines[1].trim())) {
for (int i = 0; i < Integer.parseInt(numberOfLines[1].trim()); i++) {
System.out.println(scAll.nextLine());
linkFile.add(scAll.nextLine());
counter++;
}
if (counter > Integer.parseInt(numberOfLines[1].trim())) {
flag = 2;
counter = 1;
}
} else if (flag == 2 && "\\n\\n".equals(scAll.nextLine())
&& counter <= Integer.parseInt(numberOfLines[2].trim())) {
for (int i = 0; i < Integer.parseInt(numberOfLines[2].trim()); i++) {
System.out.println(scAll.nextLine());
moduleFile.add(scAll.nextLine());
counter++;
}
} else {
continue;
}
}
scAll.close();
}
}
I have written the above code, but this code gets terminated during execution. How to get the desired result? Please help.
Hopefully I am not misunderstanding, but this is what I'd do.
I didn't check to see if this is fully working example, but it should work more or less.
import java.io.File;
import java.io.FileNotFoundException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Scanner;
import java.util.stream.Collectors;
class ReadFile {
// basically just do what you did
static List<String> nodeFile;
static List<String> linkFile;
static List<String> moduleFile;
public static void main(String[] args) throws FileNotFoundException {
final File file = new File("/home/madhu/Desktop/node.txt");
final Scanner scanner = new Scanner(file);
// make it a little better for indexing
final List<Integer> selections = Arrays
.stream(scanner.nextLine().split(","))
.map(Integer::parseInt)
.collect(Collectors.toList());
// this is the meat of the code
// basically each split up block of lines is a block
final List<List<String>> blocks = new ArrayList<>();
while (scanner.hasNextLine()) {
List<String> lines = new ArrayList<>();
String line;
while (!(line = scanner.nextLine()).equals("\n")) {
lines.add(line);
}
if (!lines.isEmpty()) {
blocks.add(lines);
}
}
// allocate your files now
nodeFile = blocks.get(0).subList(0, selections.get(0));
linkFile = blocks.get(1).subList(0, selections.get(1));
moduleFile = blocks.get(2).subList(0, selections.get(2));
scanner.close();
}
}
try this code , i use BufferedReader because its more cleaner :
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public class ReadFile {
static List<String> moduleFile = new ArrayList<>();
static List<String> linkFile = new ArrayList<>();
static List<String> nodeFile = new ArrayList<>();
public static void main(String[] args) {
File file = new File("/home/madhu/Desktop/node.txt");
BufferedReader reader = null;
try {
reader = new BufferedReader(new FileReader(file));
String data[] = reader.readLine().split(",");
String s;
int nbline=0, i=0,block =0;
while ((s = reader.readLine())!=null && block < data.length) {
if(s.equals("")){
block++;
nbline = Integer.parseInt(data[block-1]);
i = 0;
}
for(;i<nbline;i++){
s = reader.readLine();
if(s == null) break;
else if(s.equals("")){
block++;
break;
}
switch(block){
case 1 :
nodeFile.add(s);
break;
case 2:
linkFile.add(s);
break;
default: moduleFile.add(s);
}
}
}
} catch (IOException | NumberFormatException ex) {
System.err.println(ex.getStackTrace());
}
finally{
closeReader(reader);
}
System.out.println("nodeFile : "+nodeFile);
System.out.println("linkFile : "+linkFile);
System.out.println("moduleFile : "+moduleFile);
}
public static void closeReader(BufferedReader reader) {
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
}
}
}
}
output :
nodeFile : [data1,123,89,1, data2,124,90,2]
linkFile : [data5,127,93,5, data6,128,94,6]
moduleFile : [data8,130,96,8]
I am making a Library System in Java, I am able to add new books, view and save them. However, I now want to search them using a Search window box. The saved data is located in a txt file. I would like to search for specific fields. I am thinking of implementing a linear search method, but am not too sure how to do it.
package bcu.storer;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import bcu.model.Book;
public class BookStorer {
public void StoreBooks(ArrayList<Book> booksList) throws IOException
{
FileWriter fw = new FileWriter(".\\data\\books.txt");
BufferedWriter bw = new BufferedWriter(fw);
try {
for (int i = 0; i < booksList.size(); i++)
{
String content = "";
Book book = booksList.get(i);
content += book.getIsbn()+"::";
content += book.getTitle()+"::";
content += book.getAuthor()+"::";
content += book.getPublisher()+"::";
content += book.getPudDate()+"::";
content += book.getStatus()+"\n";
bw.write(content);
}
System.out.println("Complete storing all books!");
} catch (IOException ae) {
ae.printStackTrace();
} finally {
try {
if (bw != null)
bw.close();
if (fw != null)
fw.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}
This is the code which stores the book information to the TXT file, I would like to access this data in the search results.
I'm not sure if that's what you want. I help it helps you.
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
public class ReadFromFile {
private static final String FILENAME = "pathToFile";
public static void main(String[] args) {
BufferedReader br = null;
FileReader fr = null;
try {
//br = new BufferedReader(new FileReader(FILENAME));
fr = new FileReader(FILENAME);
br = new BufferedReader(fr);
String sCurrentLine;
while ((sCurrentLine = br.readLine()) != null) {
Array[String] tmpBook = sCurrentLine.split("::");
Book myBook = new Book(tmpBook(0),tmpBook(1),tmpBook(2), tmpBook(3), tmpBook(4), tmpBook(5))
/*Check here if is the book you are looking for*/
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (br != null)
br.close();
if (fr != null)
fr.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}
This will read the file, line by line, and will get all the lines that match the search criteria.
This map provides the position (column) of each of the fields in the Book object.
private static Map<String, Integer> fieldToPositionMap = ImmutableMap.<String, Integer>builder()
.put("Isbn", 0)
.put("Title", 1)
.put("Author", 2)
.put("Publisher", 3)
.put("PudDate", 4)
.put("Status", 5)
.build();
Note: I have used GoogleGuava's ImmutableMap to construct the map, but you can build it in a traditional way.
The search method takes the name of the field and the value that you want to search for (eg, Isbn=ABC or Publisher=XYZ) and returns all the rows (as Book objects) that match the criteria.
public List<Book> search(String fieldName, String fieldValue) {
try {
return Files.lines(Paths.get("/path/to/txt/file")) //reads a file line by line
.filter(line -> {
String[] blocks = line.split("::");
//filter (choose) a row if value of the searched field equals the provided value
return blocks[fieldToPositionMap.get(fieldName)].equals(fieldValue);
})
.map(this::deserialize) //convert the line to a Book object
.collect(Collectors.toList()); //collect the result
} catch (IOException e) {
throw new RuntimeException(e);
}
}
//To convert a row from the file to a Book instance
private Book deserialize(String line) {
String [] blocks = line.split("::");
Book book = new Book();
book.setIsbn(blocks[0]);
book.setTitle(blocks[1]);
book.setAuthor(blocks[2]);
book.setPublisher(blocks[3]);
book.setPudDate(blocks[4]);
book.setStaus(blocks[5]);
}
Note: You might have to handle cases when a row does not have all fields
Here is my code
package sequentialFilePractice;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.logging.Level;
import java.util.logging.Logger;
public class ReadFile{
static String line = "";
ReadFile() throws FileNotFoundException{
readTheFile();
CSVtoArrayList();
}
public String readTheFile() throws FileNotFoundException{
String csvFile = "H:\\S6\\AH Computing\\Java Practice\\test.csv";
BufferedReader br = null;
String cvsSplitBy = ",";
try {
br = new BufferedReader(new FileReader(csvFile));
while ((line = br.readLine()) != null) {
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return line;
}
public static ArrayList<String> CSVtoArrayList() {
ArrayList<String> splitCSV = new ArrayList<>();
if (line != null) {
String[] splitData = line.split("\\s*,\\s*");
for (int i = 0; i < splitData.length; i++) {
if (!(splitData[i] == null) || !(splitData[i].length() == 0)) {
splitCSV.add(splitData[i].trim());
}
}
}
for(int j = 0;j < splitCSV.size();j++){
System.out.println(splitCSV.get(j));
}
return splitCSV;
}
public static void main(String[]args) throws IOException{
ReadFile f = new ReadFile();
}
}
The code compiles and the file exists. I can print line and it prints the contents of the file however when I print the arrayList, nothing is output so it has not been copied. This is my first use of sequential files in java.
Do you HAVE to read the file manually? If not, you should check out http://opencsv.sourceforge.net/, it allows you to read a CSV directly into a List<String[]> instead of having to deal with the admin of looping, splitting the line and creating a list.
In essence reducing your code to:
CSVReader reader = new CSVReader(new FileReader("yourfile.csv"));
List myEntries = reader.readAll();
This question already has answers here:
Java Scanner to print previous and next lines
(2 answers)
Closed 6 years ago.
I have a text file from which according to some keywords I have to read that line and write that line into excel file. I did this, now I need to read next line and previous line and that lines also I need to write into excel sheet in different columns. How can I do this.
rahul1.txt
ABCD1 abhishek1 duplicatevalue jgf
ABCD2 abhishek2 duplicatevalue jgf
ABCD3 abhishek3 duplicatevalue jgf
ABCD4 abhishek4 duplicatevalue jgf
while (st1.hasMoreTokens()) {String txt = st1.nextToken();if (txt.contains("abhishek2")) {l1.add(txt);}
How can I print
ABCD1 abhishek1 duplicatevalue jgf prev Line
and
ABCD3 abhishek3 duplicatevalue jgf Next Line
in different column?
import java.io.BufferedReader;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;`
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import org.apache.poi.hssf.usermodel.HSSFSheet;
import org.apache.poi.hssf.usermodel.HSSFWorkbook;
import org.apache.poi.ss.usermodel.Cell;
import org.apache.poi.ss.usermodel.Row;
public class ReadWrite {
int rownum = 1;
HSSFSheet firstSheet;
Collection<File> files;
HSSFWorkbook workbook;
File exactFile;
boolean retu;
// BufferedReader reader = null;
{
workbook = new HSSFWorkbook();
firstSheet = workbook.createSheet("SampleSheet");
Row headerRow = firstSheet.createRow(0);
// headerRow.createCell(0).setCellValue("#");
headerRow.createCell(0).setCellValue("ID");
headerRow.createCell(1).setCellValue(Message");
headerRow.createCell(2).setCellValue("name");
headerRow.createCell(3).setCellValue("address");
headerRow.createCell(4).setCellValue("contact");
headerRow.createCell(5).setCellValue("Next");
headerRow.createCell(6).setCellValue("Prev");
}
public static void main(String args[]) {
ReadWrite class2 = new ReadWrite();
class2.readfile();
}
void readfile() {
try {
FileInputStream fInput = new FileInputStream(
"D:\\Rahul\\rahul1.txt");
DataInputStream dis = new DataInputStream(fInput);
BufferedReader br = new BufferedReader(new InputStreamReader(dis));
String lineStr;
String prevStr = "";
String nextStr = "";
//List<String> l1 = new ArrayList<String>();
int i;
int seqno = 1;
while ((lineStr = br.readLine()) != null) {
List<String> l1 = new ArrayList<String>();
if (lineStr.contains("Oracle")
|| lineStr.contains("SAP")
|| lineStr.contains("J2EE")) {
l1.add("C1");
l1.add(lineStr);
l1.add("M1");
l1.add("R1");
l1.add("V1");
l1.add(nextStr);
l1.add(prevStr);
} else {
prevStr = lineStr;
}
try {
if (l1 != null && l1.size() > 0)
retu = writenameinsheet(l1);
} catch (Exception e) {
e.printStackTrace();
}
seqno++;
i = 1;
}br.close();
FileOutputStream fos = null;
try {
File excelFile = new File("D:\\Rahul\\rahul.xls");
fos = new FileOutputStream(excelFile);
workbook.write(fos);
} catch (IOException e) {
e.printStackTrace();
} finally {
if (fos != null) {
try {
fos.flush();
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
catch (Exception e) {
e.printStackTrace();
}
}
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
}
}
boolean writenameinsheet(List<String> l1) throws Exception {
try {
Row row = firstSheet.createRow(rownum);
for (int j = 0; j < l1.size(); j++){
Cell cell = row.createCell(j);
cell.setCellValue(l1.get(j));
}rownum++;
} catch (Exception e) {
e.printStackTrace();
} finally {
}
return true;
}
}
I need to make my program read a file, then take the numbers in the string and sort them into an array. I can get my program to read the file and put it to a string, but that's where I'm stuck. All the numbers are on different lines in the file, but appear as one long number in the string. This is what I have so far:
public static void main(String[] args) {
String ipt1;
Scanner fileInput;
File inFile = new File("input1.dat");
try {
fileInput = new Scanner(inFile);
//Reads file contents
while (fileInput.hasNext()) {
ipt1 = fileInput.next();
System.out.print(ipt1);
}
fileInput.close();
}
catch (FileNotFoundException e) {
System.out.println(e);
}
}
I recommend reading the values in as numeric types using fileInput.nextInt() or whatever type you want them, putting them in an array and using a built in sort like Arrays.sort. Unless I'm missing a more subtle point about the question.
If your task is just to get input from some file and you're sure the file has integers, use an ArrayList.
import java.util.*;
Scanner fileInput;
ArrayList<Double>ipt1 = new ArrayList<Double>();
File inFile = new File("input1.dat");
try {
fileInput = new Scanner(inFile);
//Reads file contents
while (fileInput.hasNext()){
ipt1.add(fileInput.nextDouble()); //Adds the next Double to the ArrayList
System.out.print(ipt1.get(ipt1.size()-1)); //Prints out what you just got.
}
fileInput.close();
}
catch (FileNotFoundException e){
System.out.println(e);
}
//Sorting time
//This uses the built-in Array sorting.
Collections.sort(ipt1);
However, if you DO need to come up with a simple array in the end, but CAN use ArrayLists, you can add the following:
Double actualResult[] = new Double[ipt1.size()]; //Declare array
for(int i = 0; i < ipt1.size(); ++i){
actualResult[i] = ipt1.get(i);
}
Arrays.sort(actualResult[]);
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.time.LocalDateTime;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.stream.Collectors;
public class SortNumberFromFile {
public static void main(String[] args) throws IOException {
BufferedReader br = null;
try {
System.out.println("Started at " + LocalDateTime.now());
br = new BufferedReader(new FileReader("/folder/fileName.csv"));//Read data from file named /folder/fileName.csv
List<Long> collect = br.lines().mapToLong(a -> Long.parseLong(a)).boxed().collect(Collectors.toList());//Collect all read data in list object
Collections.sort(collect);//Sort the data
writeRecordsToFile(collect, "/folder/fileName.txt");//Write sorted data to file named /folder/fileName.txt
System.out.println("Ended at " + LocalDateTime.now());
}
finally {
br.close();
}
}
public static <T> void writeRecordsToFile(Collection<? extends T> items, String filePath) {
BufferedWriter writer = null;
File file = new File(filePath);
try {
if(!file.exists()) {
file.getParentFile().mkdirs();
file.createNewFile();
}
writer = new BufferedWriter(new FileWriter(filePath, true));
if(items != null && items.size() > 0) {
for(T eachItem : items) {
if(eachItem != null) {
writer.write(eachItem.toString());
writer.newLine();
}
}
}
} catch (IOException ex) {
}finally {
try {
writer.close();
} catch (IOException e) {
}
}
}
}