I'm currently using a ByteBuffer to store a chain of primitives before sending it all over the wire.
I'm using data.order(ByteOrder.LITTLE_ENDIAN); because that's how data should be sent.
Is there a ByteBuffer equivalent that supports putShort()... using little endian, AND can be expanded?
I could write a wrapper that checks limit() and position(), creating a new and bigger ByteBuffer when necessary, but I can't believe there's no other existing class that can do that.
It appears that Apache Mina has what you want: http://mina.apache.org/report/1.1/apidocs/org/apache/mina/common/ByteBuffer.html
I know that Mina is used in many production messaging systems, so it's probably a better choice than the other results that Google returned. And as you seem to be writing network code, you may find additional pieces of Mina useful.
However, I think it would be far simpler to allocate a much-larger-than-needed JDK buffer, with a simple check to ensure that you don't overrun it. In the case where you do overrun, write the first buffer then allocate a replacement.
Related
Quick design question: I need to implement a form of communication between a client-server network in my game-engine architecture in order to send events between one another.
I had opted to create event objects and as such, I was wondering how efficient it would be to serialize these objects and pass them through an object stream over the simple socket network?
That is, how efficient is it comparatively to creating a string representation of the object, sending the string over via a char stream, and parsing the string client side?
The events will be sent every game loop, if not more; but the event object itself is just a simple wrapper for a few java primitives.
Thanks for your insight!
(tl;dr - are object streams over networks efficient?)
If performance is the primary issue, I suggest using Protocol Buffers over both your own custom serialization and Java's native serialization.
Jon Skeet gives a good explanation as well as benchmarks here: High performance serialization: Java vs Google Protocol Buffers vs ...?
If you can't use PBs, I suspect Java's native serialization will be more optimized than manually serializing/deserializing from a String. Whether or not this difference is significant is likely dependent on how complex of an object you're serializing. As always, you should benchmark to confirm your predictions.
The fact that you're sending things over a network shouldn't matter.
Edit: For time-critical applications Protocol Buffers appear a better choice. However, it appears to me that there is a significant increase in development time. Effectively you'll have to code every exchange message twice: Once as a .proto file which is compiled and spits out java wrappers, and once as a POJO which makes something useful out of these wrappers. But that's guessing from the documentation.
End of Edit
Abstract: Go for the Object Stream
So, what is less? The time it takes to code the object, send the byte stream, and decode it - all by hand - or the time it takes to code the object, send the byte stream, and decode it - all by the trusty and tried serialization mechanism?
You should make sure the objects you send are as small as possible. This can be achieved with enum values, lookup tables and the such, where possible. Might shave a few bytes off each transmission. The serialization algorithm appears very speedy to me, and anything you would code would do exactly the same. When you reinvent the wheel, more often than not you end up with triangles.
i'm writing a webserver for mobile android based devices in java.
This webserver is single-threaded and follow the idea behind nginx, node.js and similar: don't spawn multiple threads just use async operations in an event loop.
While using a multi-threaded webserver may give better performance on x86 recent cpus, on arm based single core cpu will need to do a lot of more job.
To clarify, i know quite well C and i've implemented single threaded webservers in plain c or multithreaded one in C#, taking advantage of IOPS on windows, but i've wrote only a simple webserver in java, the one i want to replace with this new one.
Right now, i'm using java nio and i've readed that ByteBuffer are quite slow when converted to string but this isn't a problem because i don't need to do, infact to gaix maximium performances i wanna to implement parsing and comparing at byte level.
My question is, which method for parsing byte buffer is faster?
I've seen that ByteBuffer supports get method, that give access to a single byte and move ahead the cursor, supports array method, that return the backing array, so my question is which method is faster?
I can work directly on backed array, or i should avoid and use get?
I wanna to implement a ByteBufferPool to reuse bytebuffer, i'll make thread-aware it, read below, can be this an issue?
In some cases i'll compare byte to byte, appling a mask to handle case sensitivity (i mean, if the first byte is G, the third is T and fourth is a space (0x47, 0x54 and 0x20) i can treath the request as a GET one) and in other cases i'll need to compare strings with byte array, like for headers (i'll loop through string chars, cast them to bytes and compare to bytes).
Sorry for these silly questions, but i don't know java specs and don't know internal java stuff, so i need infos :)
Someone can give an hint? :)
PS: obiviously, not all operation can be handled in a do-stuff-pause-continue-return manner, so i'll implement a ThreadPool to avoid thread creation penalty
Give Netty a try.
You can control the threading model and you can implement just what you need.
Some background:
I working on some java packages that need to take data, divide it and distribute to many Servers online. According to the user code (person using my packages), I will know which kind of buffer to use. I have seen The Netty Project that has some interesting concepts in mind and other solutions suggested here in the site, but not exactly what I'm looking for. I guess I can implement my requirements, but I like reusing better.
The actual Issue: Byte Buffer Requirements
Separate implementation from interface.
Change between the two implementation.
Fixed Size:
Direct memory usage.
Zero Copy IO on distribution.
Growing Size
Can grow to up to a certain Size.
Keeps balance between reallocation and amount of memory in use.
Update 1: Decided to go with the Netty project library due to a lot of other benefits.
I wanted to give some details about my system. I need to distribute large amounts of data throughout a network of servers. I have very low amount of knowledge regarding the size of my buffers. Sometimes it can be fixed sizes, sometimes it can be in unknown size mode. I want to be able to develop a mechanism that enjoys the benefit of both worlds. I do have a defined event when I receive the final buffer for inspection.
I understand that buffers can perform better when they are directly mapped to memory. Netty provides this for me, but I can't make this buffer grow. I have dynamic buffer which can grow -Netty provides that too.
Couple of questions:
should I copy the buffer to from dynamic to direct in case it is changed? Is there a way to switch modes?
What do you think of implementing a class that holds an array of ChannleBuffers and exposes the same interface like ChannleBuffer while containing "inside" an array of buffers allocating new direct memory as needed.
Do you have other solutions in mind ?
I would have a good look at scatter/gather NIO on arrays of direct byte buffers. The direct-ness gives you no-copy I/O, and the array gives you the mechanism to grow.
I found out that Netty supplied me with all the things I need and more.
What is the difference between a byte array & byte buffer ?
Also, in what situations should one be preferred over the other?
[my usecase is for a web application being developed in java].
There are actually a number of ways to work with bytes. And I agree that it's not always easy to pick the best one:
the byte[]
the java.nio.ByteBuffer
the java.io.ByteArrayOutputStream (in combination with other streams)
the java.util.BitSet
The byte[] is just a primitive array, just containing the raw data. So, it does not have convenient methods for building or manipulating the content.
A ByteBuffer is more like a builder. It creates a byte[]. Unlike arrays, it has more convenient helper methods. (e.g. the append(byte) method). It's not that straightforward in terms of usage. (Most tutorials are way too complicated or of poor quality, but this one will get you somewhere. Take it one step further? then read about the many pitfalls.)
You could be tempted to say that a ByteBuffer does to byte[], what a StringBuilder does for String. But there is a specific difference/shortcoming of the ByteBuffer class. Although it may appear that a bytebuffer resizes automatically while you add elements, the ByteBuffer actually has a fixed capacity. When you instantiate it, you already have to specify the maximum size of the buffer.
That's one of the reasons, why I often prefer to use the ByteArrayOutputStream because it automatically resizes, just like an ArrayList does. (It has a toByteArray() method). Sometimes it's practical, to wrap it in a DataOutputStream. The advantage is that you will have some additional convenience calls, (e.g. writeShort(int) if you need to write 2 bytes.)
BitSet comes in handy when you want to perform bit-level operations. You can get/set individual bits, and it has logical operator methods like xor(). (The toByteArray() method was only introduced in java 7.)
Of course depending on your needs you can combine all of them to build your byte[].
ByteBuffer is part of the new IO package (nio) that was developed for fast throughput of file-based data. Specifically, Apache is a very fast web server (written in C) because it reads bytes from disk and puts them on the network directly, without shuffling them through various buffers. It does this through memory-mapped files, which early versions of Java did not have. With the advent of nio, it became possible to write a web server in java that is as fast as Apache. When you want very fast file-to-network throughput, then you want to use memory mapped files and ByteBuffer.
Databases typically use memory-mapped files, but this type of usage is seldom efficient in Java. In C/C++, it's possible to load up a large chunk of memory and cast it to the typed data you want. Due to Java's security model, this isn't generally feasible, because you can only convert to certain native types, and these conversions aren't very efficient. ByteBuffer works best when you are just dealing with bytes as plain byte data -- once you need to convert them to objects, the other java io classes typically perform better and are easier to use.
If you're not dealing with memory mapped files, then you don't really need to bother with ByteBuffer -- you'd normally use arrays of byte. If you're trying to build a web server, with the fastest possible throughput of file-based raw byte data, then ByteBuffer (specifically MappedByteBuffer) is your best friend.
Those two articles may help you http://nadeausoftware.com/articles/2008/02/java_tip_how_read_files_quickly and http://evanjones.ca/software/java-bytebuffers.html
I have some binary data (pixel values) in a int[] (or a byte[] if you prefer) that I want to write to disk in an Android app. I only want to use a small amount of processing time but want as much compression as I can for this. What are my options?
In many cases the array will contain lots of consecutive zeros so something simple and fast like RLE compression would probably work well. I can't see any Android API functions for this though. If I have to loop over the array in Java, this will be very slow as there is no JIT on most Android devices. I could use the NDK but I'd rather avoid this if I can.
DeflatorOutputStream takes ~25 ms to compress 1 MB in Java. Its a native method so a JIT should not make much difference.
Do you have a requirement which says 0.2s or 0.5s is too slow?
Can you do it in a background thread so the user doesn't notice how long it takes?
GZIP is based on the Deflator + CRC32 so is likely to be much the same or slightly slower.
Deflator has several modes. The DEFAULT_STRATEGY is fastest in Java, but simpler compressions such as HUFFMAN_ONLY might be faster for you.
Android has Java's DeflaterOutputStream. Would that work?
Pass the byte array to
http://download.oracle.com/javase/6/docs/api/java/io/FileWriter.html
and chain
http://download.oracle.com/javase/1.4.2/docs/api/java/util/zip/GZIPOutputStream.html
to it
then when you need to read the data back in do the reverse
http://download.oracle.com/javase/1.4.2/docs/api/java/io/FileReader.html
and chain
http://download.oracle.com/javase/1.4.2/docs/api/java/util/zip/GZIPInputStream.html
Depending on the size of the file your saving you will see some compression Gzip is good like that, if your not seeing much of a trade of just write the data uncompressed using a buffered writer(That should be the fastest). Also if you do gzip it using a buffered writer reader could also speed it up a bit.
I've had to solve basically the same problem on another platform and my solution was to use a modified LZW compression. First, do some difference filtering (similar to PNG) on the 32bpp image. This will turn most of the image to black if there are large areas of common color. Then use a generic GIF compression algorithm treating the filtered image as if it's 8bpp. You'll get decent compression and it works very quickly. This will need to run in native code (NDK). It's really quite easy to get native code working on Android.
Random thought: if it's image data, try saving it as png. Standard java has it, i'm sure android will too, and probably optimized with native code. It has pretty good compression and it's lossless.