Develop Reference

Manipulate big XML file in Java Spring

I have a Java program (a war) that runs out of memory when manipulating a big XML file.
The program is a REST API that returns the manipulated XML via a REST Controller.
First, the program gets an XML file from a remote URL.
Then it replaces the values of id attributes.
Finally, it returns the new XML to the caller via the API controller.
What I get from the remote URL is a byte[] body with XML data.
Then, I convert it to a String.
Next, I do a regexp search-replace on the whole string.
Then I convert it back to a byte[].
I'm guessing that the XML now is in memory 3 times (the incoming bytes, the string and the outgoing bytes).
I'm looking for ways to improve this.
I have no local copies on the filesystem btw.

You can delete the incoming bytes from memory after converting the bytes to String:
byte[] bytes = bytesFromURL;
String xml = new String(bytes);
{...manipulate xml}
bytes = null;
System.gc();
bytes = xml.getBytes();

Send binary file from Java Server to C# Unity3d Client with Protocol Buffer

I have asked this question https://stackoverflow.com/questions/32735189/sending-files-from-java-server-to-unity3d-c-sharp-client but I saw that it isn't an optimal solution to send files between Java and C# via built-in operations, because I also need also other messages, not only the file content.
Therefore, I tried using Protobuf, because it is fast and can serialize/deserialize objects platform independent. My .proto file is the following:
message File{
optional int32 fileSize = 1;
optional string fileName = 2;
optional bytes fileContent = 3;
}
So, I set the values for each variable in the generated .java file:
file.setFileSize(fileSize);
file.setFileName(fileName);
file.setFileContent(ByteString.copyFrom(fileContent, 0, fileContent.length);
I saw many tutorials about how to write the objects to a file and read from it. However, I can't find any example about how to send a file from server socket to client socket.
My intention is to serialize the object (file size, file name and file content) on the java server and to send these information to the C# client. So, the file can be deserialized and stored on the client side.
In my example code above, the server read the bytes of the file (image file) and write it to the output stream, so that the client can read and write the bytes to disk through input stream. I want to achieve the same thing with serialization of my generated .proto file.
Can anyone provide me an example or give me a hint how to do that?

As described in the documentation, protobuf does not take care of where a message start and stops, so when using a stream socket like TCP you'll have to do that yourself.
From the doc:
[...] If you want to write multiple messages to a single file or stream, it is up to you to keep track of where one message ends and the next begins. The Protocol Buffer wire format is not self-delimiting, so protocol buffer parsers cannot determine where a message ends on their own. The easiest way to solve this problem is to write the size of each message before you write the message itself. When you read the messages back in, you read the size, then read the bytes into a separate buffer, then parse from that buffer. [...]
Length-prefixing is a good candidate. Depending on what language you're writing, there are libraries that does length-prefixing for e.g. TCP that you can use, or you can define it yourself.
An example representation of the buffer on the wire might beof the format might be (beginning of buffer to the left):
[buf_length|serialized_buffer2]
So you code to pack the the buffer before sending might look something like (this is in javascript with node.js):
function pack(message) {
var packet = new Buffer(message.length + 2);
packet.writeIntBE(message.length, 0, 2);
message.copy(packet, 2);
return packet;
}
To read you would have to do the opposite:
client.on('data', function (data) {
dataBuffer = Buffer.concat([dataBuffer, data]);
var dataLen = dataBuffer.readIntBE(0, 2);
while(dataBuffer.length >= dataLen) {
// Message length excluding length prefix of 2 bytes
var msgLen = dataBuffer.readIntBE(0, 2);
var thisMsg = new Buffer(dataBuffer.slice(2, msgLen + 2));
//do something with the msg here
// Remove processed message from buffer
dataBuffer = dataBuffer.slice(msgLen + 2);
}
});
You should also be aware of that when sending multiple protobufs on a TCP socket, they are likely to be buffered for network optimizations (concatenated) and sent together. Meaning some sort of delimiter is needed anyway.

java printstream image file

Is a printstream appropriate for sending image files through a socket? I'm currently doing a homework assignment where I have to write a web proxy from scratch using basic sockets.
When I configure firefox to use my proxy everything works fine except images don't download. If I go to an image file directly firefox comes back with the error: The image cannot be displayed because it contains errors
Here is my code for sending the response from the server back to the client (firefox):
BufferedReader serverResponse = new BufferedReader(new InputStreamReader(webServer.getInputStream()));
String responseLine;
while((responseLine = serverResponse.readLine()) != null)
{
serverOutput.println(responseLine);
}
In the code above serverOutput is a PrintStream object. I am wondering if somehow the PrintStream is corrupting the data?

No, it is never appropriate to treat bytes as text unless you know they are text.
Specifically, the InputStreamReader will try to decode your image (which can be treated as a byte array) to a String. Then your PrintStream will try to encode the String back to a byte array.
There is no guarantee that this will produce the original byte array. You might even get an exception, depending on what encoding Java decides to use, if some of the image bytes aren't valid encoded characters.

Java as HTTP server - Is possible get a image via POST method?

I would like to know if is possible get the image via POST method with a HTTP server implemented in Java (With a simple input file form). I already implemented the Java server but I can only get text files via POST method it's because that the my application only copies the file content to another empty file creating the same file with the same characteristics. This does not work with image file or other files, this can only work with text file.
Anyone know how to implement it with images?
Some coordinates would be of great help!
Thanks in advance!

As far as i know you should create something like it:
Server-side: If you use a servlet that receive data in post you have to get the outputStream from the response. Once you have it it is done because you write the data image on the stream.
For example let's suppose your image is a file stored in the server you could do:
response.setContentLength((int) fileSize);
byte b[] = new byte[1024];
while ( fOutStream.read(b) != -1)
response.getOutputStream().write(b);
fOutStream.close() ;
Where the fOutStream is the source stream (your image).

Python equivalent of java ObjectOutputStream and ObjectInputStream?

In java I can transfer objects between server and client by using Object Output Stream and Object Input Stream. Is there anything equivalent in python?
Related:
python equivalent of java OutputStream?

The pickle module in Python provides object serialization and deserialization functionality. http://docs.python.org/library/pickle.html
It's not particularly secure, so you should always validate the incoming data, but it should support your needs.

The multiprocessing module has the Pipe() function that handles serializing and passing objects between processes.
http://docs.python.org/library/multiprocessing.html#multiprocessing.Pipe
example (pipes work within the same process too)
import multiprocessing
class ObjectToSend:
def __init__(self,data):
self.data = data
obj = ObjectToSend(['some data'])
#create 2 read/write ends to a pipe
a, b = multiprocessing.Pipe()
#serialize and send obj across the pipe using send method
a.send(obj)
#deserialize object at other end of the pipe using recv method
print(b.recv().data)