I'm trying to use PostgreSQL COPY command to import data without creating temporary files. Initial data is in a format that requires conversion so I have something like this:
void itemsToCsv(String filename, Writer writer) {
/* Start processing */
CSVPrinter printer = new CSVPrinter(writer, CSVFormat.DEFAULT)
for(Array[String] row: processedFile) {
printer.printRecord(row);
}
}
Now here comes the problem. I was thinking about using something like this code
CopyManager copyMgr = (BaseConnection) conn.getCopyAPI();
Writer w = new PipedWriter();
Reader r = new PipedReader(w);
CopyIn cpin = copyMgr.copyIn("COPY items_import (id, name)FROM STDIN CSV", r);
itemsToCsv("items_in_weird_format.xml", w)
As far as I understand, the copyIn method will block, because there's nothing to read yet. Is there some clever way to wire readers and writers together without resorting to CopyIn.writeToCopy method?
UPD. Solved by wrapping call to itemsToCsv in a thread.
Related
I am new to Java and I need help.
When I reopen GUI, it doesn't shows what has been saved into the file, which was serialization. The file is saving successful, but when I close and reopen and run the application it doesn't show on JList, what was saved into this file.
try
{
FileInputStream jos = new FileInputStream("jokam.ser");
GZIPInputStream gis = new GZIPInputStream(jos);
ObjectInputStream hehe = new ObjectInputStream(gis);
v1= (Vector<Vector>)hehe.readObject();
Vpredmeti.addAll((Collection<? extends Predmet>)v1.get(0));
Vvlak.addAll((Collection<? extends Vlak>)v1.get(1));
jos.close();
hehe.close();
gis.close();
v1.addAll(0, v1);
for(Predmet pr : predmetAR){
System.out.println(pr);
}
}
catch (Exception e)
{
}
}
These Vectors are before try code.
Vector <Predmet> Vpredmeti = new Vector (predmetAR);
Vector <Vlak> Vvlak= new Vector();
Vector <Vector> v1 = new Vector<>();
This is where I add to JList.
private void DodajPredmetMouseClicked(java.awt.event.MouseEvent evt) {
// TODO add your handling code here:
// DefaultListModel list = new DefaultListModel();
String praznoPolje=ImePredmeta.getText();
String drugoPraznoPolje=ZnacilnostPredmeta.getText();
int tretjoPraznoPolje = (int)ComboBoxZabojnika.getSelectedIndex();
Predmet novPredmet = new Predmet();
novPredmet.ime = ImePredmeta.getText();
novPredmet.znacilnosti = drugoPraznoPolje;
novPredmet.tipZabojnika=tretjoPraznoPolje;
//list.addElement(novPredmet);
predmetAR.add(novPredmet);
Save code
Vector<Predmet> Vpredmet = new Vector<>(predmetAR);
Vector<Vlak> Vvlak = new Vector<>(vlakAR);
Vector<Vector> v = new Vector<>();
v.add(0,Vpredmet);
v.add(1,Vvlak);
try
{
FileOutputStream fos = new FileOutputStream("jokam.ser");
GZIPOutputStream gos = new GZIPOutputStream(fos);
ObjectOutputStream oos = new ObjectOutputStream(gos);
oos.writeObject(v);
gos.close();
fos.close();
oos.close();
}
catch(Exception e)
{
}
}
Those exceptions you noted are almost definitely problems where a stream was closed early, either on the write part or the read part. It's also indicative of a layering problem with the streams, but I don't see that here.
To first step in solving these problems is making sure all the data is written before the stream is closed, and in the proper order. I usually flush() the highest level stream before closing it or underlying parts. flush() the highest level OutputStream (here, the ObjectOutputStream), and it will flush all the underlying streams (here the GZIPOutputStream and FileOutputStream). Technically close() also flush()es the stream so this may not be necessary.
Also, make sure to close() streams in the correct order. Same as flush(), close the higher level stream and the underlying streams get close()d (and flush()ed) automatically.
The code you already have close()es the GZIPOutputStream first, which precludes the closing bits of the ObjectOutputStream. Later, the ObjectOutputStream is close()d which will try to write those bits but the underlying stream has already been closed so, so an IOException is thrown.
When writing, I suggest trying just:
objectOutputStream.close();
As for the reading, just this should be good:
objectInputStream.close()
As I mentioned in the comments, you should close() in a finally block so that any Exception thrown in the try block still results in the close() being called. Be aware that close() can also throw an Exception ;)
To investigate this on your own, I suggest looking into the source code of all these streams to see what's happening inside. The JDK includes an optional jdk/lib/src.zip, which most IDE's will let you jump into. Try 'go to definition' on your objectOutputStream.close() and you should see the source code.
Is there anyway to reuse an inputStream by changing its content? (Without new statement).
For instance, I was able to something very close to my requirement, but not enough
In the following code I am using a SequenceInputStream, and everytime I am adding a new InputStream to that sequence.
But I would like to do the same thing by using the same inputStream (I don't care which implementation of InputStream).
I thought about mark()/reset() APIs, but i still need to change the content to be read.
The idea to avoid new InputStream creations is because of performance issues
//Input Streams
List<InputStream> inputStreams = new ArrayList<InputStream>();
try{
//First InputStream
byte[] input = new byte[]{24,8,102,97};
inputStreams.add(new ByteArrayInputStream(input));
Enumeration<InputStream> enu = Collections.enumeration(inputStreams);
SequenceInputStream is = new SequenceInputStream(enu);
byte [] out = new byte[input.length];
is.read(out);
for (byte b : out){
System.out.println(b);//Will print 24,8,102,97
}
//Second InputStream
input = new byte[]{ 4,66};
inputStreams.add(new ByteArrayInputStream(input));
out = new byte[input.length];
is.read(out);
for (byte b : out){
System.out.println(b);//will print 4,66
}
is.close();
}catch (Exception e){//
}
No, You can't restart reading the input stream after it reaches to the end of the stream as it is uni-directional i.e. moves only in single direction.
But Refer below links, they may help:
How to Cache InputStream for Multiple Use
Getting an InputStream to read more than once, regardless of markSupported()
You could create your own implementation (subclass) of InputStream that would allow what you require. I doubt there is an existing implementation of this.
I highly doubt you'll get any measurable performance boost from this though, there's not much of logic in e.g. FileInputStream that you wouldn't need to perform anyways, and Java is well optimized for garbage-collecting short-lived objects.
I started working with Apache poi just recently. What I'm trying to do:
My method receives a HashMap, with the key being a long Vendor (a class I've created to represent a customer of my company) ID, and the value being the Vendor object itself.
My goal is to use the boolean attribute 'managed' in the Vendor object, and output into one file the list of managed Vendors (the ones that have 'true' in 'managed') and then another one for the unmanaged. Here's my code:
public static void loadVendorsIntoFile(XSSFWorkbook workbook, HashMap<Long,Vendor> vendors, boolean managed){
XSSFSheet vendorsSheet = workbook.createSheet("Vendors");
int rowNum = 1;
Row row = vendorsSheet.createRow(0);
row.createCell(0).setCellValue("vendor_id");
row.createCell(1).setCellValue("num_items");
row.createCell(2).setCellValue("vendor_username");
row.createCell(3).setCellValue("Segment");
row.createCell(4).setCellValue("AM");
for (long vendor : vendors.keySet()){
if (vendors.get(vendor) != null && vendors.get(vendor).getIfManaged() == managed){
row = vendorsSheet.createRow(rowNum);
row.createCell(0).setCellValue(vendors.get(vendor).getVendorId());
row.createCell(1).setCellValue(vendors.get(vendor).getCountItems());
row.createCell(2).setCellValue(vendors.get(vendor).getVendorName());
row.createCell(3).setCellValue(vendors.get(vendor).getSegment());
row.createCell(4).setCellValue(vendors.get(vendor).getAccountManager());
rowNum++;
}
}
}
public static void topvendorsOutput (HashMap<Long,Vendor> vendors) throws ClassNotFoundException, SQLException, IOException{
XSSFWorkbook managedWorkbook = new XSSFWorkbook();
loadVendorsIntoFile(managedWorkbook,vendors,true);
OutputStream outManaged = new FileOutputStream("C:\\Users\\User\\managed_out.xlsx");
managedWorkbook.write(outManaged);
outManaged.close();
XSSFWorkbook nonManagedWorkbook = new XSSFWorkbook();
loadVendorsIntoFile(nonManagedWorkbook,vendors,false);
OutputStream outNonManaged = new FileOutputStream("C:\\Users\\User\\unmanaged_out.xlsx");
managedWorkbook.write(outNonManaged);
outNonManaged.close();
}
However, for some reason I get two files with the same info - both show only managed Vendors. I've been testing (printing to the console) and I see that in the second call to loadVendorsIntoFile(), the unmanaged Vendors are really being printed when the loop runs - but for some reason the what is written to the second file is the info that should come only from the first call (and first file) to the method.
Also, the second call (which creates the second file) comes out with corruption problems, excel warns me that it has "found problems with the content" and needs to restore it.
I guess I'm doing something wrong here.
So in short - the first file is OK, the second one isn't.
Would appreciate any help with this. Thanks.
Looking at your code, we see a few problems:
XSSFWorkbook managedWorkbook = new XSSFWorkbook();
loadVendorsIntoFile(managedWorkbook,vendors,true);
OutputStream outManaged = new FileOutputStream("C:\\Users\\User\\unmanaged_out.xlsx");
managedWorkbook.write(outManaged);
outManaged.close();
This writes managedWorkbook into C:\\Users\\User\\unmanaged_out.xlsx
XSSFWorkbook nonManagedWorkbook = new XSSFWorkbook();
loadVendorsIntoFile(nonManagedWorkbook,vendors,false);
OutputStream outNonManaged = new FileOutputStream("C:\\Users\\User\\unmanaged_out.xlsx");
managedWorkbook.write(outNonManaged);
outNonManaged.close();
This also writes managedWorkbook into C:\\Users\\User\\unmanaged_out.xlsx !
I suspect that you meant that last bit to instead be something like:
OutputStream outNonManaged = new FileOutputStream("C:\\Users\\User\\unmanaged_out2.xlsx");
nonManagedWorkbook.write(outNonManaged);
outNonManaged.close();
Note how the changed snippet both writes into a different file, and importantly, writes out the second workbook you created, rather than outputting the first one twice!
I'm working on a program that requires quick access to a CSV comma-delimited spreadsheet file.
So far I've been able to read from it easily using a BufferedReader.
However, now I want to be able to edit the data it reads, then export it BACK to the CSV.
The spreadsheet contains names, phone numbers, email addresses, etc. And the program lists everyone's data, and when you click on them it brings up a page with more detailed information, also pulled from the CSV. On that page you can edit the data, and I want to be able to click a "Save Changes" button, then export the data back to its appropriate line in the CSV--or delete the old one, and append the new.
I'm not very familiar with using a BufferedWriter, or whatever it is I should be using.
What I started to do is create a custom class called FileIO. It contains both a BufferedReader and a BufferedWriter. So far it has a method that returns bufferedReader.readLine(), called read(). Now I want a function called write(String line).
public static class FileIO {
BufferedReader read;
BufferedWriter write;
public FileIO (String file) throws MalformedURLException, IOException {
read = new BufferedReader(new InputStreamReader (getUrl(file).openStream()));
write = new BufferedWriter (new FileWriter (file));
}
public static URL getUrl (String file) throws IOException {
return //new URL (fileServer + file).openStream()));
FileIO.class.getResource(file);
}
public String read () throws IOException {
return read.readLine();
}
public void write (String line) {
String [] data = line.split("\\|");
String firstName = data[0];
// int lineNum = findLineThatStartsWith(firstName);
// write.writeLine(lineNum, line);
}
};
I'm hoping somebody has an idea as to how I can do this?
Rather than reinventing the wheel you could have a look at OpenCSV which supports reading and writing of CSV files. Here are examples of reading & writing
Please consider Apache commons csv.
To fast understand the api, there are four important classes:
CSVFormat
Specifies the format of a CSV file and parses input.
CSVParser
Parses CSV files according to the specified format.
CSVPrinter
Prints values in a CSV format.
CSVRecord
A CSV record parsed from a CSV file.
Code Example:
Unit test code:
The spreadsheet contains names, phone numbers, email addresses, etc. And the program lists everyone's data, and when you click on them it brings up a page with more detailed information, also pulled from the CSV. On that page you can edit the data, and I want to be able to click a "Save Changes" button, then export the data back to its appropriate line in the CSV--or delete the old one, and append the new.
The content of a file is a sequence of bytes. CSV is a text based file format, i.e. the sequence of byte is interpreted as a sequence of characters, where newlines are delimited by special newline characters.
Consequently, if the length of a line increases, the characters of all following lines need to be moved to make room for the new characters. Likewise, to delete a line you must move the later characters to fill the gap. That is, you can not update a line in a csv (at least not when changing its length) without rewriting all following lines in the file. For simplicity, I'd rewrite the entire file.
Since you already have code to write and read the CSV file, adapting it should be straightforward. But before you do that, it might be worth asking yourself if you're using the right tool for the job. If the goal is to keep a list of records, and edit individual records in a form, programs such as Microsoft Access or whatever the Open Office equivalent is called might be a more natural fit. If you UI needs go beyond what these programs provide, using a relational database to keep your data is probably a better fit (more efficient and flexible than a CSV).
Add Dependencies
implementation 'com.opencsv:opencsv:4.6'
Add Below Code in onCreate()
InputStreamReader is = null;
try {
String path= "storage/emulated/0/Android/media/in.bioenabletech.imageProcessing/MLkit/countries_image_crop.csv";
CSVReader reader = new CSVReader(new FileReader(path));
String[] nextLine;
int lineNumber = 0;
while ((nextLine = reader.readNext()) != null) {
lineNumber++;
//print CSV file according to your column 1 means first column, 2 means
second column
Log.e(TAG, "onCreate: "+nextLine[2] );
}
}
catch (Exception e)
{
Log.e(TAG, "onCreate: "+e );
}
I solved it using
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-csv</artifactId>
<version>2.8.6</version>
</dependency>
and
private static final CsvMapper mapper = new CsvMapper();
public static <T> List<T> readCsvFile(MultipartFile file, Class<T> clazz) throws IOException {
InputStream inputStream = file.getInputStream();
CsvSchema schema = mapper.schemaFor(clazz).withHeader().withColumnReordering(true);
ObjectReader reader = mapper.readerFor(clazz).with(schema);
return reader.<T>readValues(inputStream).readAll();
}
I'm a newbie in Spring Batch, and I would appreciate some help to resolve this situation: I read some files with a MultiResourceItemReader, make some marshalling work, in the ItemProcessor I receive a String and return a Map<String, List<String>>, so my problem is that in the ItemWriter I should iterate the keys of the Map and for each one of them generate a new file containing the value associated with that key, can someone point me out in the right direction in order to create the files?
I'm also using a MultiResourceItemWriter because I need to generates files with a maximum of lines.
Thanks in advance
Well, finaly got a solution, I'm not really excited about it but it's working and I don't have much more time, so I've extended the MultiResourceItemWriter and redefined the "write" method, processing the map's elements and writing the files by myself.
In case anyone out there needs it, here it is.
#Override
public void write(List items) throws Exception {
for (Object o : items) {
//do some processing here
writeFile(anotherObject);
}
private void writeFile (AnotherObject anotherObject) throws IOException {
File file = new File("name.xml");
boolean restarted = file.exists();
FileUtils.setUpOutputFile(file, restarted, true, true);
StringBuffer sb = new StringBuffer();
sb.append(xStream.toXML(anotherObject));
FileOutputStream os = new FileOutputStream(file, true);
BufferedWriter bufferedWriter = new BufferedWriter(new OutputStreamWriter(os, Charset.forName("UTF-8")));
bufferedWriter.write(sb.toString());
bufferedWriter.close();
}
And that's it, I want to believe that there is a better option that I don't know, but for the moment this is my solution. If anyone knows how can I enhance my implementation, I'd like to know it.