I've got a Spring MVC app with a file upload capability. Files are passed to the controller as MultipartFile from which it's easy to get an InputStream. I'm uploading zip files that contain CSVs and I'm struggling to find a way to open the CSVs and read them a line at a time. There are plenty of examples on the 'net of reading into a fixed sizes buffer. I've tried this, but the buffers don't concatenate very well and it soon gets out of sync and uses a lot of memory:
ZipEntry entry = input.getNextEntry();
while(entry != null)
{
if (entry.getName().matches("Data/CSV/[a-z]{0,1}[a-z]{0,1}.csv"))
{
final String fullPath = entry.getName();
final String filename = fullPath.substring(fullPath.lastIndexOf('/') + 1);
visitor.startFile(filename);
final StringBuilder fileContent = new StringBuilder();
final byte[] buffer = new byte[1024];
while (input.read(buffer) > 0)
fileContent.append(new String(buffer));
final String[] lines = fileContent.toString().split("\n");
for(String line : lines)
{
final String[] columns = line.split(",");
final String postcode = columns[0].replace(" ", "").replace("\"", "");
if (columns.length > 3)
visitor.location(postcode, "", "");
}
visitor.endFile();
}
entry = input.getNextEntry();
}
There must be a better way that actually works.
Not clear if this suits your need, but have you tried opencsv (http://opencsv.sourceforge.net)? Their example is really intuitive:
CSVReader reader = new CSVReader(new FileReader("yourfile.csv"));
String [] nextLine;
while ((nextLine = reader.readNext()) != null) {
// nextLine[] is an array of values from the line
System.out.println(nextLine[0] + nextLine[1] + "etc...");
}
For your case, all you will need is to wrap the zipped file stream into a buffered reader and pass the reader to create a CSVReader and use it:
FileInputStream fis = new FileInputStream(file);
GZIPInputStream gis = new GZIPInputStream(fis);
InputStreamReader isr = new InputStreamReader(gis);
BufferedReader br = new BufferedReader(isr);
CSVReader reader = new CSVReader(br);
You could use a BufferedReader which includes the convenient readLine() method and wont load the entire contents of the file into memory e.g.
BufferedReader in = new BufferedReader(new InputStreamReader(input), 1024);
String line=null;
while((line=br.readLine())!=null) {
String[] columns = line.split(",");
//rest of your code
}
Related
I downloaded an xml file from web service. If I open file from file system is formed correctly, but when I run my code isn't formed correctly.
A part of xml file formed correctly, it opened from file system:
<?xml version="1.0" encoding="UTF-8"?><ns3:FatturaElettronica xmlns:ns3="http://ivaservizi.agenziaentrate.gov.it/docs/xsd/fatture/v1.2" xmlns:ns2="http://www.w3.org/2000/09/xmldsig#" versione="FPR12">
Here the same xml file managed by my code:
ÿþ<
I can't copy the code and I put an image of what I see on the eclipse console.
I tryed different ways to manage this file, but nothing worked.
This is the code that it manages files. I put all ways I tryed to solve the error.
private static String readFile(File file, Writer writerArg) throws FileNotFoundException, IOException,Exception
{
FileInputStream fis = null;
InputStreamReader isr = null;
String typeEncoding = null;
/*
* First way
*
* BufferedReader br = new BufferedReader(new FileReader(fileName));
String nextLine = "";
StringBuffer sb = new StringBuffer();
while ((nextLine = br.readLine()) != null)
{
// System.out.println("Writing: " + nextLine);
writerArg.write(nextLine);
// sb.append(nextLine);
sb.append(nextLine+"\n");
} // Convert the content into to a string
String clobData = sb.toString().trim();
*/
/*
* Second way
*
* fis = new FileInputStream(file);
isr = new InputStreamReader(fis);
typeEncoding = isr.getEncoding();
Charset inputCharset = Charset.forName(typeEncoding);
BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(file), inputCharset));
String str;
String nextLine = "";
StringBuffer sb = new StringBuffer();
while ((nextLine = in.readLine()) != null) {
System.out.println(nextLine);
writerArg.write(nextLine);
// sb.append(nextLine);
sb.append(nextLine+"\n");
}
String clobData = sb.toString().trim();
// Return the data.
return clobData;
*/
/* Third way */
String data = "";
data = new String(Files.readAllBytes(Paths.get(file.getAbsolutePath())));
System.out.println(data);
return data;
}
And when the below code receives the string I get the error: "String index out of range: -1"
schema=stringXml.substring(0,stringXml.indexOf("<FatturaElettronicaHeader")).trim();
The first way downloaded thousands of files and managed them. Only this file gives my this error. It's from yesterday that I'm looking for a way to solve the error.
Can anyone give my any idea?
I want to save the contents of a tar.gz archive inside a database table.
The archive contains txt files in CSV format.
The idea is to insert a new line in the database for each line in the txt files.
The problem is that I can't read the contents of a file separately then move on to the next file.
Below EntryTable and EntryTableLine are Hibernate entities.
EntryTable is in a OneToMany relationship with EntryTableLine (a file -EntryTable- can have many lines -EntryTableLine-).
public static final int TAB = 9;
FileInputStream fileInputStream = new FileInputStream(fileLocation);
GZIPInputStream gzipInputStream = new GZIPInputStream(fileInputStream);
TarArchiveInputStream tar = new TarArchiveInputStream(gzipInputStream);
BufferedReader reader = new BufferedReader(new InputStreamReader(tar));
// Columns are delimited with TAB
CSVFormat csvFormat = CSVFormat.TDF.withHeader().withDelimeter((char) TAB);
CSVParser parser = new CSVParser(reader, csvFormat);
TarArchiveEntry tarEntry = tar.getNextTarEntry();
while(tarEntry != null){
EntryTable entryTable = new EntryTable();
entryTable.setFilename(tarEntry.getName());
if(reader != null){
// Here is the problem
for(CSVRecord record : parser){
//this could have been a StringBuffer
String line;
int i = 1;
for(String val : record){
line = "<column" + i + ">" + val + "</column" + i + ">";
}
EntryTableLine entryTableLine = new EntryTableLine();
entryTableLine.setContent(line);
entryDao.saveLine(entryTableLine);
}
}
tarEntry = tar.getNextTarEntry();
}
I tried converting tarEntry.getFile() to InputStream, but tarEntry.getFile() is null unfortunately.
Let's say I have 4 files in the archive. Each file has 3 lines inside. However, in the database, some entries have 5 lines while others have none.
Thank you !
You can use the TarArchiveInputStream of Apache Commons Compress as shown below(Reference):
TarArchiveInputStream input = new TarArchiveInputStream(new GzipCompressorInputStream(new FileInputStream("C:\\Users\\User\\Desktop\\Books\\test\\CoverLetter-Version2.gz")));
TarArchiveEntry entry = input.getNextTarEntry();
System.out.println(entry.getName()); // prints the name of file inside the tar
BufferedReader br = null;
StringBuilder sb = new StringBuilder();
while (entry != null) {
br = new BufferedReader(new InputStreamReader(input)); // Read directly from tarInput
System.out.println("For File = " + currentEntry.getName());
String line;
while ((line = br.readLine()) != null) {
System.out.println("line="+line);
}
entry = input.getNextTarEntry();
}
Try to read directly from inputstream:
BufferedReader br = null;
while(tarEntry != null){
br = new BufferedReader(new InputStreamReader(tarEntry));
Doing something similar to this solved the problem:
TarArchiveEntry entry = tarInput.getNextTarEntry();
byte[] content = new byte[entry.getSize()];
LOOP UNTIL entry.getSize() HAS BEEN READ {
tarInput.read(content, offset, content.length - offset);
}
Reference mentioned in the comments
I am writing a little app where I go to an api, get some json data and fill that into a csv file.
It works so far, that I get a csv file, with the correct rows, but instead of columns there are still commas in text form in file.
I am using opencsv.
private void writeCsv(InputStream input, String name) throws IOException {
String line;
BufferedReader reader = new BufferedReader(new InputStreamReader(input, "UTF-8"));
StringBuilder text = new StringBuilder();
while ((line = reader.readLine()) != null) {
text.append(line);
}
JSONArray docs = new JSONArray(text.toString());
File file=new File(name + ".csv");
String csv = CDL.toString(docs);
CSVWriter writer = new CSVWriter(new FileWriter(file));
BufferedReader sreader = new BufferedReader(new StringReader(csv));
String csvline;
while ((csvline = sreader.readLine()) != null) {
writer.writeNext(csvline);
}
writer.close();
System.out.println("done");
}
You need to add column mapping strategy.
ColumnPositionMappingStrategy mappingStrategy = new ColumnPositionMappingStrategy();
check this blog for better understanding.
http://www.javainterviewpoint.com/csvtobean-and-beantocsv-example-using-opencsv/
I am trying to read a huge file where a new line is indicated by no space, comma, new line character, or anything.
Example: line1element1, line1element2, line1element3, line2element1, line2element2, and so on..
The file is a csv and I am reading it like following:
public static void main(String[] args) throws Exception {
ArrayList<String> list = new ArrayList<>();
String element;
String filename = "E:\\csv.csv";
Scanner scanner = new Scanner(new File(filename));
scanner.useDelimiter(",");
for (int i = 0; i < 50; i++) {
element = scanner.next();
list.add(element);
}
System.out.print(list);
}
This causes issues because the element50 in a line gets combined with element51, although it should be a new line.
Use a BufferedReader for this:
String filename = "E:\\csv.csv";
BufferedReader fileReader = null;
//Delimiter used in CSV file
final String DELIMITER = ",";
String line = "";
//Create the file reader
fileReader = new BufferedReader(new FileReader(filename ));
//Read the file line by line
while ((line = fileReader.readLine()) != null)
{
//Get all tokens available in line
String[] tokens = line.split(DELIMITER);
for(String token : tokens)
{
//Print all tokens
System.out.println(token);
}
}
USe a BufferedReader, not Scanner
File f= ...;
BufferedReader br = new BufferedReader(new FileReader(f));
String line;
while ((line = br.nextLine()) != null) {
String[] columns = line.split(",");
}
Why not use CSVParser from Apache Commons or OpenCSV?
Examples here:
OpenCSV Example
Apache Commons example
If you insist on doing this manually, use BufferedReader as the other comments mention.
From your description it seems your file does not have headers for each column. Use uniVocity-parsers to do this for you - it's 3 times faster than Commons CSV & OpenCSV and packed with features.
// you have many configuration options here - check the tutorial. By default values are trimmed and blank lines skipped.
CsvParserSettings settings = new CsvParserSettings();
CsvParser parser = new CsvParser(settings);
List<String[]> allRows = parser.parseAll(new FileReader(new File("/path/to/your/file.csv")));
Disclosure: I am the author of this library. It's open-source and free (Apache V2.0 license).
How would I read a .txt file in Java and put every line in an array when every lines contains integers, strings, and doubles? And every line has different amounts of words/numbers.
I'm a complete noob in Java so sorry if this question is a bit stupid.
Thanks
Try the Scanner class which no one knows about but can do almost anything with text.
To get a reader for a file, use
File file = new File ("...path...");
String encoding = "...."; // Encoding of your file
Reader reader = new BufferedReader (new InputStreamReader (
new FileInputStream (file), encoding));
... use reader ...
reader.close ();
You should really specify the encoding or else you will get strange results when you encounter umlauts, Unicode and the like.
Easiest option is to simply use the Apache Commons IO JAR and import the org.apache.commons.io.FileUtils class. There are many possibilities when using this class, but the most obvious would be as follows;
List<String> lines = FileUtils.readLines(new File("untitled.txt"));
It's that easy.
"Don't reinvent the wheel."
The best approach to read a file in Java is to open in, read line by line and process it and close the strea
// Open the file
FileInputStream fstream = new FileInputStream("textfile.txt");
BufferedReader br = new BufferedReader(new InputStreamReader(fstream));
String strLine;
//Read File Line By Line
while ((strLine = br.readLine()) != null) {
// Print the content on the console - do what you want to do
System.out.println (strLine);
}
//Close the input stream
fstream.close();
To learn more about how to read file in Java, check out the article.
Your question is not very clear, so I'll only answer for the "read" part :
List<String> lines = new ArrayList<String>();
BufferedReader br = new BufferedReader(new FileReader("fileName"));
String line = br.readLine();
while (line != null)
{
lines.add(line);
line = br.readLine();
}
Common used:
String line = null;
File file = new File( "readme.txt" );
FileReader fr = null;
try
{
fr = new FileReader( file );
}
catch (FileNotFoundException e)
{
System.out.println( "File doesn't exists" );
e.printStackTrace();
}
BufferedReader br = new BufferedReader( fr );
try
{
while( (line = br.readLine()) != null )
{
System.out.println( line );
}
#user248921 first of all, you can store anything in string array , so you can make string array and store a line in array and use value in code whenever you want. you can use the below code to store heterogeneous(containing string, int, boolean,etc) lines in array.
public class user {
public static void main(String x[]) throws IOException{
BufferedReader b=new BufferedReader(new FileReader("<path to file>"));
String[] user=new String[500];
String line="";
while ((line = b.readLine()) != null) {
user[i]=line;
System.out.println(user[1]);
i++;
}
}
}
This is a nice way to work with Streams and Collectors.
List<String> myList;
try(BufferedReader reader = new BufferedReader(new FileReader("yourpath"))){
myList = reader.lines() // This will return a Stream<String>
.collect(Collectors.toList());
}catch(Exception e){
e.printStackTrace();
}
When working with Streams you have also multiple methods to filter, manipulate or reduce your input.
For Java 11 you could use the next short approach:
Path path = Path.of("file.txt");
try (var reader = Files.newBufferedReader(path)) {
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
}
Or:
var path = Path.of("file.txt");
List<String> lines = Files.readAllLines(path);
lines.forEach(System.out::println);
Or:
Files.lines(Path.of("file.txt")).forEach(System.out::println);