Develop Reference

Why there is a 't' in front of column while using stream resolution return byte array outputstream? [duplicate]

This question already has an answer here:
Writing LinkedList into text file via ObjectOutputStream but output is garbage
(1 answer)
Closed 1 year ago.
I am using Spring framework, and use streamresolution to return a .txt file for user to download.
The result of data is fine, however, there is a 't' in front of every column of data,
and besides the last column, there is a 'w' in the end of every column.
I can't not understand why because the data seems fine, and I didn't told the program to create the letter.
Here is my code:
// A list of String, which are the data, it might looks like 20200810,a,b,c,100,55,.....
// the whole is a String contains comma
List<String> dataList = (List<String>) parameters.get("myData");
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
StreamingResolution streamingResolution = null;
ObjectOutputStream oos = new ObjectOutputStream(outputStream);
oos.writeObject("\n");
for (String s : dataList) {
oos.writeObject(s.trim());
oos.writeUTF("\n");
}
streamingResolution = new StreamingResolution("text/plain", new ByteArrayInputStream(outputStream.toByteArray()));
streamingResolution.setCharacterEncoding(CharEncoding.UTF_8);
String year = Integer.toString((Integer.parseInt(end.substring(0, 4));
String day = year + end.substring(4, 6);
oos.close();
return streamingResolution.setFilename(day + ".txt");
while I download the data, 202108.txt
it might looks like
t ?0210810,a,b,c,100,55w
t ?0210810,d,e,f,99,60
could anyone please tell me why there would be a 't' in the front
and a 'w' in the end?
And how to fix this?
Thanks a lot.

This code uses an ObjectOutputStream, which is used to write serialized Java data in a binary format. It is not a plain text format, and should not be used in this way. The extra characters are bytes that are defined in the Java Object Serialization Specification.
To write plain text, you can use the java.io.PrintStream class instead. For example:
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
PrintStream printStream = new PrintStream(outputStream, false, StandardCharsets.UTF_8);
printStream.println();
for (String s : dataList) {
printStream.println(s.trim());
}
printStream.flush();
StreamingResolution streamingResolution = new StreamingResolution("text/plain", new ByteArrayInputStream(outputStream.toByteArray()));
streamingResolution.setCharacterEncoding(CharEncoding.UTF_8);
Note that I also simplified the code by moving the streamingResolution local variable declaration to where it is assigned.
This is a straightforward translation of the code provided, to show you how to use the PrintStream class, however it may not be the best way to write it. The StreamingResolution class appears to be part of the Stripes Framework. It is intended for streaming large responses to the client. However, this implementation does not actually stream the response, it accumulates it into a byte array. A better way to implement this would be to subclass the StreamingResponse class, as described in the Stripe documentation, to write directly to the response:
return new StreamingResolution("text/plain") {
public void stream(HttpServletResponse response) throws Exception {
response.setCharacterEncoding("UTF-8");
PrintWriter out = response.getWriter();
out.println();
for (String s : dataList) {
out.println(s.trim());
}
out.flush();
}
}.setFilename(day + ".txt");

Re-opening GUI doesn't show saved data

I am new to Java and I need help.
When I reopen GUI, it doesn't shows what has been saved into the file, which was serialization. The file is saving successful, but when I close and reopen and run the application it doesn't show on JList, what was saved into this file.
try
{
FileInputStream jos = new FileInputStream("jokam.ser");
GZIPInputStream gis = new GZIPInputStream(jos);
ObjectInputStream hehe = new ObjectInputStream(gis);
v1= (Vector<Vector>)hehe.readObject();
Vpredmeti.addAll((Collection<? extends Predmet>)v1.get(0));
Vvlak.addAll((Collection<? extends Vlak>)v1.get(1));
jos.close();
hehe.close();
gis.close();
v1.addAll(0, v1);
for(Predmet pr : predmetAR){
System.out.println(pr);
}
}
catch (Exception e)
{
}
}
These Vectors are before try code.
Vector <Predmet> Vpredmeti = new Vector (predmetAR);
Vector <Vlak> Vvlak= new Vector();
Vector <Vector> v1 = new Vector<>();
This is where I add to JList.
private void DodajPredmetMouseClicked(java.awt.event.MouseEvent evt) {
// TODO add your handling code here:
// DefaultListModel list = new DefaultListModel();
String praznoPolje=ImePredmeta.getText();
String drugoPraznoPolje=ZnacilnostPredmeta.getText();
int tretjoPraznoPolje = (int)ComboBoxZabojnika.getSelectedIndex();
Predmet novPredmet = new Predmet();
novPredmet.ime = ImePredmeta.getText();
novPredmet.znacilnosti = drugoPraznoPolje;
novPredmet.tipZabojnika=tretjoPraznoPolje;
//list.addElement(novPredmet);
predmetAR.add(novPredmet);
Save code
Vector<Predmet> Vpredmet = new Vector<>(predmetAR);
Vector<Vlak> Vvlak = new Vector<>(vlakAR);
Vector<Vector> v = new Vector<>();
v.add(0,Vpredmet);
v.add(1,Vvlak);
try
{
FileOutputStream fos = new FileOutputStream("jokam.ser");
GZIPOutputStream gos = new GZIPOutputStream(fos);
ObjectOutputStream oos = new ObjectOutputStream(gos);
oos.writeObject(v);
gos.close();
fos.close();
oos.close();
}
catch(Exception e)
{
}
}

Those exceptions you noted are almost definitely problems where a stream was closed early, either on the write part or the read part. It's also indicative of a layering problem with the streams, but I don't see that here.
To first step in solving these problems is making sure all the data is written before the stream is closed, and in the proper order. I usually flush() the highest level stream before closing it or underlying parts. flush() the highest level OutputStream (here, the ObjectOutputStream), and it will flush all the underlying streams (here the GZIPOutputStream and FileOutputStream). Technically close() also flush()es the stream so this may not be necessary.
Also, make sure to close() streams in the correct order. Same as flush(), close the higher level stream and the underlying streams get close()d (and flush()ed) automatically.
The code you already have close()es the GZIPOutputStream first, which precludes the closing bits of the ObjectOutputStream. Later, the ObjectOutputStream is close()d which will try to write those bits but the underlying stream has already been closed so, so an IOException is thrown.
When writing, I suggest trying just:
objectOutputStream.close();
As for the reading, just this should be good:
objectInputStream.close()
As I mentioned in the comments, you should close() in a finally block so that any Exception thrown in the try block still results in the close() being called. Be aware that close() can also throw an Exception ;)
To investigate this on your own, I suggest looking into the source code of all these streams to see what's happening inside. The JDK includes an optional jdk/lib/src.zip, which most IDE's will let you jump into. Try 'go to definition' on your objectOutputStream.close() and you should see the source code.

Saving a Hashset to a file in Java

I know this question has been asked a million times and I have seen a million solutions but none that work for me. I have a hashet that I want to write to a file but I want each element in the Hashset in a separate line.
Here is my code:
Collection<String> similar4 = new HashSet<String>(file268List);
Collection <String> different4 = new HashSet<String>();
different4.addAll(file268List);
different4.addAll(sqlFileList);
similar4.retainAll(sqlFileList);
different4.removeAll(similar4);
Iterator hashSetIterator = different.iterator();
while(hashSetIterator.hasNext()){
System.out.println(hashSetIterator.next());
}
ObjectOutputStream writer = new ObjectOutputStream(new FileOutputStream("HashSet.txt"));
while(hashSetIterator.hasNext()){
Object o = hashSetIterator.next();
writer.writeObject(o);
}

Where you got it wrong is that you are trying to serialize the strings instead of just printing them to the file, exactly the same way you print them to the screen:
PrintStream out = new PrintStream(new FileOutputStream("HashSet.txt")));
Iterator hashSetIterator = different.iterator();
while(hashSetIterator.hasNext()){
out.println(hashSetIterator.next());
}

ObjectOutputStream will try to serialize the String as an object (binary format). I think you you want to use a PrintWriter instead. Example:
PrintWriter writer= new PrintWriter( new OutputStreamWriter( new FileOutputStream( "HashSet.txt"), "UTF-8" ));
while(hashSetIterator.hasNext()) {
String o = hashSetIterator.next();
writer.println(o);
}
Note that per this answer and the answer from Marko, you can use PrintStream or PrintWriter to output strings (characters). There is little difference between the two, but be sure to specify a character encoding if you work with non standard characters or need to read/write files across different platforms.

reverse chunks in a file

My basic Java problem is this: I need to read in a file by chunks, then reverse the order of the chunks, then write that out to a new file. My first (naive) attempt followed this approach:
read a chunk from the file.
reverse the bytes of the chunk
push the bytes one at a time to the front of a results list
repeat for all chunks
write result list to new file.
So this is basically a very stupid and slow way to solve the problem, but generates the correct output that I am looking for. To try to improve the situation, I change to this algorithm:
read a chunk from the file
push that chunk onto the front of a list of arrays
repeat for all chunks
foreach chunk, write to new file
And to my mind, that produces the same output. except it doesn't and I am quite confused. The first chunk in the result file matches with both methods, but the rest of the file is completely different.
Here is the meat of the Java code I am using:
FileInputStream in;
FileOutputStream out, out2;
Byte[] t = new Byte[0];
LinkedList<Byte> reversed_data = new LinkedList<Byte>();
byte[] data = new byte[bufferSize];
LinkedList<byte[]> revd2 = new LinkedList<byte[]>();
try {
in = new FileInputStream(infile);
out = new FileOutputStream(outfile1);
out2 = new FileOutputStream(outfile2);
} catch (FileNotFoundException e) {
e.printStackTrace();
return;
}
while(in.read(data) != -1)
{
revd2.addFirst(data);
byte[] revd = reverse(data);
for (byte b : revd)
{
reversed_data.addFirst(b);
}
}
for (Byte b : reversed_data)
{
out.write(b);
}
for (byte[] b : revd2)
{
out2.write(b);
}
At http://pastie.org/3113665 you can see a complete example program (a long with my debugging attempts). For simplicity I am using a bufferSize that divides evenly the size of the file so all chunks will be the same size, but this won't hold in the real world. My question is, WHY don't these two methods generate the same output? It's driving me crazy because I can't grok it.

You're constantly overwriting the data you've read previously.
while(in.read(data) != -1)
{
revd2.addFirst(data);
// ignore byte-wise stuff
}
You're adding the same object repeatedly to the list revd2, so each list node will finally contain a reference to data filled with the result of the last read. I suggest replacing that with revd2.addFirst(data.clone()).

My guess is you want to change
revd2.addFirst(data);
byte[] revd = reverse(data);
for the following so the reversed copy is added to the start of the list.
byte[] revd = reverse(data);
revd2.addFirst(revd);

Java data object for bidirectional I/O

I am developing an interface that takes as input an encrypted byte stream -- probably a very large one -- that generates output of more or less the same format.
The input format is this:
{N byte envelope}
- encryption key IDs &c.
{X byte encrypted body}
The output format is the same.
Here's the usual use case (heavily pseudocoded, of course):
Message incomingMessage = new Message (inputStream);
ProcessingResults results = process (incomingMessage);
MessageEnvelope messageEnvelope = new MessageEnvelope ();
// set message encryption options &c. ...
Message outgoingMessage = new Message ();
outgoingMessage.setEnvelope (messageEnvelope);
writeProcessingResults (results, message);
message.writeToOutput (outputStream);
To me, it seems to make sense to use the same object to encapsulate this behaviour, but I'm at a bit of a loss as to how I should go about this. It isn't practical to load all of the encrypted body in at a time; I need to be able to stream it (so, I'll be using some kind of input stream filter to decrypt it) but at the same time I need to be able to write out new instances of this object. What's a good approach to making this work? What should Message look like internally?

I won't create one class to handle in- and output - one class, one responsibility. I would like two filter streams, one for input/decryption and one for output/encryption:
InputStream decrypted = new DecryptingStream(inputStream, decryptionParameters);
...
OutputStream encrypted = new EncryptingStream(outputSream, encryptionOptions);
They may have something like a lazy init mechanism reading the envelope before first read() call / writing the envelope before first write() call. You also use classes like Message or MessageEnvelope in the filter implementations, but they may stay package protected non API classes.
The processing will know nothing about de-/encryption just working on a stream. You may also use both streams for input and output at the same time during processing streaming the processing input and output.

Can you split the body at arbitrary locations?
If so, I would have two threads, input thread and output thread and have a concurrent queue of strings that the output thread monitors. Something like:
ConcurrentLinkedQueue<String> outputQueue = new ConcurrentLinkedQueue<String>();
...
private void readInput(Stream stream) {
String str;
while ((str = stream.readLine()) != null) {
outputQueue.put(processStream(str));
}
}
private String processStream(String input) {
// do something
return output;
}
private void writeOutput(Stream out) {
while (true) {
while (outputQueue.peek() == null) {
sleep(100);
}
String msg = outputQueue.poll();
out.write(msg);
}
}
Note: This will definitely not work as-is. Just a suggestion of a design. Someone is welcome to edit this.

If you need to read and write same time you either have to use threads (different threads reading and writing) or asynchronous I/O (the java.nio package). Using input and output streams from different threads is not a problem.
If you want to make a streaming API in java, you should usually provide InputStream for reading and OutputStream for writing. This way those can then be passed for other APIs so that you can chain things and so get the streams go all the way as streams.
Input example:
Message message = new Message(inputStream);
results = process(message.getInputStream());
Output example:
Message message = new Message(outputStream);
writeContent(message.getOutputStream());
The message needs to wrap the given streams with a classes that do the needed encryption and decryption.
Note that reading multiple messages at same time or writing multiple messages at same time would need support from the protocol too. You need to get the synchronization correct.

You should check Wikipedia article on different block cipher modes supporting encryption of streams. The different encryption algorithms may support a subset of these.
Buffered streams will allow you to read, encrypt/decrypt and write in a loop.
Examples demonstrating ZipInputStream and ZipOutputStream could provide some guidance on how you may solve this. See example.

What you need is using Cipher Streams (CipherInputStream). Here is an example of how to use it.

I agree with Arne, the data processor shouldn't know about encryption, it just needs to read the decrypted body of the message, and write out the results, and stream filters should take care of encryption. However, since this is logically operating on the same piece of information (a Message), I think they should be packaged inside one class which handles the message format, although the encryption/decryption streams are indeed independent from this.
Here's my idea for the structure, flipping the architecture around somewhat, and moving the Message class outside the encryption streams:
class Message {
InputStream input;
Envelope envelope;
public Message(InputStream input) {
assert input != null;
this.input = input;
}
public Message(Envelope envelope) {
assert envelope != null;
this.envelope = envelope;
}
public Envelope getEnvelope() {
if (envelope == null && input != null) {
// Read envelope from beginning of stream
envelope = new Envelope(input);
}
return envelope
}
public InputStream read() {
assert input != null
// Initialise the decryption stream
return new DecryptingStream(input, getEnvelope().getEncryptionParameters());
}
public OutputStream write(OutputStream output) {
// Write envelope header to output stream
getEnvelope().write(output);
// Initialise the encryption
return new EncryptingStream(output, getEnvelope().getEncryptionParameters());
}
}
Now you can use it by creating a new message for the input, and one for the output:
OutputStream output; // This is the stream for sending the message
Message inputMessage = new Message(input);
Message outputMessage = new Message(inputMessage.getEnvelope());
process(inputMessage.read(), outputMessage.write(output));
Now the process method just needs to read chunks of data as required from the input, and write results to the output:
public void process(InputStream input, OutputStream output) {
byte[] buffer = new byte[1024];
int read;
while ((read = input.read(buffer) > 0) {
// Process buffer, writing to output as you go.
}
}
This all now works in lockstep, and you don't need any extra threads. You can also abort early without having to process the whole message (if the output stream is closed for example).