Develop Reference

Spring batch filtering data inside item reader

I'm writing a batch that reads log files which should take many types (format of log log file ) then I want to read every file based on some characters inside log files for example
15:31:44,437 INFO <NioProcessor-32> Send to <SLE-
15:31:44,437 INFO <NioProcessor-32> [{2704=5, 604=1, {0=023pdu88mW00007z}]
15:31:44,437 DEBUG <NioProcessor-32> SCRecord 2944
In such a log file I want to read only log lines which contain ' [{}] ' and ignore all others. I have tried to read it in item reader and split it to object but I can't figure how. I think that I should create a custom item reader or something like that; my Logline class looks too simple:
public class logLine {
String idOrder;
String time;
String Tags;
}
and my item reader look like:
public FlatFileItemReader<logLine> customerItemReader() {
FlatFileItemReader<logLine> reader = new FlatFileItemReader<>();
reader.setResource(new ClassPathResource("/data/customer.log"));
DefaultLineMapper<LogLine> customerLineMapper = new DefaultLineMapper<>();
DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
tokenizer.setNames(new String[] {"idOrder", "date", "tags"});
customerLineMapper.setLineTokenizer(tokenizer);
customerLineMapper.setFieldSetMapper(new CustomerFieldSetMapper());
reader.setLineMapper(customerLineMapper);
return reader;
}
How can I add a filter in this item reader to read only lines which contain [{
without doing the job in the
item processor

filtering should be responsibility of processor and not the reader. You can use composite item processor and add First processor as Filtering.
Filtering processor should return null for log lines which does not contain ' [{}] ' .
These rows will be automatically ignore in next processor and in writer.

can implement a customfilereader extending FlatFileItemReader with partition num or filter criteria passed in constructor from config, and override the read() method -> https://github.com/spring-projects/spring-batch/blob/main/spring-batch-infrastructure/src/main/java/org/springframework/batch/item/support/AbstractItemCountingItemStreamItemReader.java#L90
Every slave step would instantiate based on a different constructor param.

How to modify a given String (from CSV)

I need to write a program for a project at university which should cut some specific parts out of a given CSV File. I've started already but I don't know how to keep only the content (sentence and vote values) or min. to remove the date part.
PARENT,"Lorem ipsum...","3","0","Town","09:17, 29/11/2016"
REPLY,"Loren ipsum...”,"2","0","Town","09:18, 29/11/2016"
After the program ran I want to have it like this:
Lorem ipsum... (String) 3 (int) 0 (int)
Loren ipsum... (String) 2 (int) 0 (int)
I have no problem with writing a parser (read in, remove separators) but I don't know how realize this thing.

You can create your own data structure that contains a string, and two integers and then do the following while reading from the csv file. Only include the stuff you want from the csv based on the column number which is the index of the String array returned by the split() method.
Scanner reader = new Scanner(new File("path to your CSV File"));
ArrayList<DataStructure> csvData = new ArrayList<>();
while(reader.hasNextLine())
{
String[] csvLine = reader.nextLine().split(",");
DataStructure data = new DataStructure(
csvLine[1],
Integer.parseInt(csvLine[2]),
Integer.parseInt(csvLine[3]));
csvData.add(data);
}

How to skip a part of the file then read a line?

I have a code that reads a file using buffered reader and split, said file was created via a method that automatically adds 4KB of empty space at the beginning of the file, this results in when I read the following happens:
First the Code:
BufferedReader metaRead = new BufferedReader(new FileReader(metaFile));
String metaLine = "";
String [] metaData = new String [100000];
while ((metaLine = metaRead.readLine()) != null){
metaData = metaLine.split(",");
for (int i = 0; i < metaData.length; i++){
System.out.println(metaData[i]);
}
}
This is the result, keep in mind this file already exists and contains the values:
//4096 spaces then the first actual word in the document which is --> testTable2
Name
java.lang.String
true
No Reference
Is there a way to skip the first 4096 spaces, and get straight to the actual value within the file so I can get the result regularly? Because I'll be using the metaData array later in other operations, and I'm pretty sure the spaces will mess up the number of slots within the array. Any suggestions would be appreciated.

If you're using Eclipse, the auto-completion should help.
metaRead.skip(4096);
https://docs.oracle.com/javase/7/docs/api/java/io/BufferedReader.html

You could (as mentioned) simply do:
metaRead.skip(4096);
If the whitespace always occupies that many characters, or you could just avoid lines which are empty
while ((metaLine = metaRead.readLine()) != null){
if(metaLine.trim().length() > 0){
metaData = metaLine.split(",");
for (int i = 0; i < metaData.length; i++){
System.out.println(metaData[i]);
}
}
}

jvm throwing NoSuchElementException when using Scanner

I'm facing a problem that i really don't understand.
I designed my program to read info from files and then generate reports about it.
At the first time, I open all the files that I need:
clientesarq = new File(args[1]);
fornecedoresarq = new File(args[3]);
produtosarq = new File(args[5]);
then I use a java.util.Scanner to loop through them:
leitor = new Scanner(clientesarq);
leitor.nextLine();
/* leitura e armazenamento dos clientes em mapa */
while(leitor.hasNextLine()) {
Cliente c = pd.novoCliente(leitor);
clientes.addCliente(c);
}
leitor = new Scanner(fornecedoresarq);
leitor.nextLine();
/* leitura e arazenaento dos fornecedores em mapa */
while(leitor.hasNextLine()) {
Fornecedor f = pd.novoFornecedor(leitor);
fornecedores.addFornecedor(f);
}
when my program reaches this part of the code, the JVM throws to me NoSuchElementException.
leitor = new Scanner(produtosarq);
leitor.nextLine(); /* EXCEPTION HERE */
/* leitura e armazenamento dos produtos em mapa */
while(leitor.hasNextLine()) {
Produto p = pd.novoProduto(leitor);
produtos.addProduto(p);
}
I really want to know why the hell I am getting this excpetion, as you can see, the code is exactly the same as the other ones. Any?
You can get all needed files here: https://www.dropbox.com/sh/c48roudfwuj7qzu/AAAMn_OFGXJFHEjVJyZ7piCPa

Use UTF-8 character set to read from file which is able to represent every character in Unicode Format.
Here it will convert Bytes from the file into characters using the specified charset(Here UTF-8) and will make it readable for Scanner I guess.
Scanner input = new Scanner(new File("filename"),"UTF-8");

Specifying the charset did not fix the issue for me. This post is comment-like as I have not fully discovered the issue, but I am making an attempt at explaining the why portion of the question with what I have found thus far.
I found another post with the same problem as a secondary resource. In both cases, the first line of the files ends with ASCII character 13 carriage return which I would have thought should be picked up by the Scanner as a line separator. As a sanity check, I performed a BufferedReader readLine() and it works fine. Digging deeper, I cloned the source of Scanner and wound up on readInput line 849 : n = source.read(buf);.
After determining the type of source was sun.nio.cs.StreamDecoder, I cloned the class and investigated readImpl. In both cases, line 324 int n = readBytes(); reads the entire file into the buffer hitting the continue and coming back around to line 316 CoderResult cr = decoder.decode(bb, cb, eof); At this point cr is wrapping a MalformedInputException which gets consumed by Scanner catch (IOException ioe).
Since I was able to reproduce this issue in my cloned source, I printed the stacktrace:
java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
at javaapplication1.StreamDecoder.implRead(StreamDecoder.java:349)
at javaapplication1.StreamDecoder.read(StreamDecoder.java:188)
at java.io.Reader.read(Reader.java:100)
at javaapplication1.MyScanner.readInput(MyScanner.java:844)
at javaapplication1.MyScanner.findWithinHorizon(MyScanner.java:1789)
at javaapplication1.MyScanner.nextLine(MyScanner.java:1629)
The decoder at runtime is sun.nio.cs.UTF_8. I've stopped digging at this point as it is way into the AM hours, I may edit in further findings.
The main question now is why the final single character of input kills the decoder.

Just a guess but maybe the file you are trying to read doesnt have any information. Always check if it has a next line before getting that next line.
if(scanner.hasNextLine()){
String line = scanner.nextLine();
}

Java: CSV File Easy Read/Write

I'm working on a program that requires quick access to a CSV comma-delimited spreadsheet file.
So far I've been able to read from it easily using a BufferedReader.
However, now I want to be able to edit the data it reads, then export it BACK to the CSV.
The spreadsheet contains names, phone numbers, email addresses, etc. And the program lists everyone's data, and when you click on them it brings up a page with more detailed information, also pulled from the CSV. On that page you can edit the data, and I want to be able to click a "Save Changes" button, then export the data back to its appropriate line in the CSV--or delete the old one, and append the new.
I'm not very familiar with using a BufferedWriter, or whatever it is I should be using.
What I started to do is create a custom class called FileIO. It contains both a BufferedReader and a BufferedWriter. So far it has a method that returns bufferedReader.readLine(), called read(). Now I want a function called write(String line).
public static class FileIO {
BufferedReader read;
BufferedWriter write;
public FileIO (String file) throws MalformedURLException, IOException {
read = new BufferedReader(new InputStreamReader (getUrl(file).openStream()));
write = new BufferedWriter (new FileWriter (file));
}
public static URL getUrl (String file) throws IOException {
return //new URL (fileServer + file).openStream()));
FileIO.class.getResource(file);
}
public String read () throws IOException {
return read.readLine();
}
public void write (String line) {
String [] data = line.split("\\|");
String firstName = data[0];
// int lineNum = findLineThatStartsWith(firstName);
// write.writeLine(lineNum, line);
}
};
I'm hoping somebody has an idea as to how I can do this?

Rather than reinventing the wheel you could have a look at OpenCSV which supports reading and writing of CSV files. Here are examples of reading & writing

Please consider Apache commons csv.
To fast understand the api, there are four important classes:
CSVFormat
Specifies the format of a CSV file and parses input.
CSVParser
Parses CSV files according to the specified format.
CSVPrinter
Prints values in a CSV format.
CSVRecord
A CSV record parsed from a CSV file.
Code Example:
Unit test code:

The spreadsheet contains names, phone numbers, email addresses, etc. And the program lists everyone's data, and when you click on them it brings up a page with more detailed information, also pulled from the CSV. On that page you can edit the data, and I want to be able to click a "Save Changes" button, then export the data back to its appropriate line in the CSV--or delete the old one, and append the new.
The content of a file is a sequence of bytes. CSV is a text based file format, i.e. the sequence of byte is interpreted as a sequence of characters, where newlines are delimited by special newline characters.
Consequently, if the length of a line increases, the characters of all following lines need to be moved to make room for the new characters. Likewise, to delete a line you must move the later characters to fill the gap. That is, you can not update a line in a csv (at least not when changing its length) without rewriting all following lines in the file. For simplicity, I'd rewrite the entire file.
Since you already have code to write and read the CSV file, adapting it should be straightforward. But before you do that, it might be worth asking yourself if you're using the right tool for the job. If the goal is to keep a list of records, and edit individual records in a form, programs such as Microsoft Access or whatever the Open Office equivalent is called might be a more natural fit. If you UI needs go beyond what these programs provide, using a relational database to keep your data is probably a better fit (more efficient and flexible than a CSV).

Add Dependencies
implementation 'com.opencsv:opencsv:4.6'
Add Below Code in onCreate()
InputStreamReader is = null;
try {
String path= "storage/emulated/0/Android/media/in.bioenabletech.imageProcessing/MLkit/countries_image_crop.csv";
CSVReader reader = new CSVReader(new FileReader(path));
String[] nextLine;
int lineNumber = 0;
while ((nextLine = reader.readNext()) != null) {
lineNumber++;
//print CSV file according to your column 1 means first column, 2 means
second column
Log.e(TAG, "onCreate: "+nextLine[2] );
}
}
catch (Exception e)
{
Log.e(TAG, "onCreate: "+e );
}

I solved it using
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-csv</artifactId>
<version>2.8.6</version>
</dependency>
and
private static final CsvMapper mapper = new CsvMapper();
public static <T> List<T> readCsvFile(MultipartFile file, Class<T> clazz) throws IOException {
InputStream inputStream = file.getInputStream();
CsvSchema schema = mapper.schemaFor(clazz).withHeader().withColumnReordering(true);
ObjectReader reader = mapper.readerFor(clazz).with(schema);
return reader.<T>readValues(inputStream).readAll();
}