I want to determine JSON response payload size for REST API using Java. To describe further I have to design a bulk API that is exporting large data from application, pagination and security is in place but I want to limit the individual page size to ensure the service continuity and performance output.
your help in this regards is appreciated Thanks in advance!
You're likely to need to actually encode it to JSON to be able to calculate that. The data model may not accurately indicate exactly what properties are exposed via JSON, what they're called, or how they're encoded.
There are a few other questions on SO asking roughly the same thing, see
is there an easy way to estimate size of a json object? and
Calculate size in bytes of JSON payload including it in the JSON payload in PHP
You don't say how you're using JSON, but it's fairly likely it's using Jackson, and perhaps Spring, so something like this ought to put you in the basic ballpark
ObjectMapper om = new ObjectMapper();
Object example = ... // the object you want to find the JSON size for
String jsonString = om.writeValueAsString(example); // get the JSON string
int characterCount = json.length(); // count the number of characters
byte[] bytes = om.writeValueAsBytes(example); // get the JSON as UTF-8 bytes
int byteCount = bytes.length; // count the number of bytes
The difference between characterCount and byteCount may not be obvious, especially if you're not used to dealing with Unicode code-points. HTTP Content-Length indicates the number of octets (bytes), so byteCount would be most accurate, see What's the "Content-Length" field in HTTP header?.
The Server should set the Content-Length header in the response (see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Length). This is the size of the payload in bytes.
So, the client can read the content length like this:
HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection();
connection.connect();
int length = connection.getContentLength();
Related
I am working on a legacy web service client code where the JSON data is being sent to the web service. Recently it was found that for some requests in the JSON body, the service is giving HTTP 400 response due to invalid characters (non-UTF8) in the JSON Body.
Below is one example of the data which is causing the issue.
String value = "zu3z5eq tô‰U\f‹Á‹€z";
I am using org.json.JSONObject.toString() method to generate the JSON string. Can you please let me know how can I ensure that the JSON string is UTF-8 encoded?
I already tried few solutions like available online , like converting to byte array and then back, using java charset methods etc, but they did not work. Either they convert the valid values as well like chinese/japanese characters, or doesn't work at all.
Can you please provide some input on this?
You need to set the character encoding for OutputStreamWriter when you create it:
httpConn.connect();
wr = new OutputStreamWriter(httpConn.getOutputStream(), StandardCharsets.UTF_8);
wr.write(jsonObject.toString());
wr.flush();
Otherwise it defaults to the "platform default encoding," which is some encoding that has been used historically for text files on whatever system you are running.
Use Base64 encoding for converting the value to Byte[].
String value = "zu3z5eq tô‰U\f‹Á‹€z";
// WHILE SENDING ENCODE THE VALUE
byte[] encodedBytes = Base64.getEncoder().encode(value.getBytes("UTF-8"));
String encodedValue = new String(encodedBytes, "UTF-8");
// TRANSPORT....
// ON RECEIVING END DECODE THE VALUE
byte[] decodedBytes = Base64.getDecoder().decode(encodedValue.getBytes("UTF-8"));
System.out.println( new String(decodedBytes, "UTF-8"));
My idea is to divide a big response text into small parts to load them concurrently.
The following code helps me open stream from an URL but I want to load its whole content from multithreads to optimize performance, then I will merge them into a single result. However, the method return a ReadableByteChannel which cannot specify the start position and I have to transfer it linearly:
URL url = new URL("link");
InputStream fromStream = url.openStream();
ReadableByteChannel fromChannel = Channels.newChannel(fromStream);
Is there any way to specify the position like SeekableByteChannel (seem likes this interface only works with file)? Thanks you :D
If you can manipulate the request before it's a stream then yes, you would use the Range http header to specify the chunk of data you wanted...
See Downloading a portion of a File using HTTP Requests
If not then you will manually have to read past the data you don't need.
See Given a Java InputStream, how can I determine the current offset in the stream?
Seriously.
I've been scratching around trying to find the answer to this conundrum for a while.
The request size is too large if the String is encoded, and the company won't take Base64 anyway. They actually want the binary code, but in JSON. Can anyone shed any light on how they think that other people might do this? Currently I'm processing it like this;
String addressProof = null;
if (proofRequired)
{
Part filePart = request.getPart("proof_of_address");
addressFileName = getSubmittedFileName(filePart);
InputStream fileContent = filePart.getInputStream();
final byte[] bytes = IOUtils.toByteArray(fileContent);
addressProof = new String(bytes);
//byte[] bytes64 = Base64.encodeBase64(fileBytes);
//addressProof = new String(fileBytes);
fileContent.close();
}
Am I being dim, or is this whole request, just a little bit flawed.
Many thanks.
You can send it (or receive) as a hex string. See how-to-convert-a-byte-array-to-a-hex-string-in-java.
Example output would be (if enclosed by a JSON object):
{
"content": "C5192E4190E54F5985ED09C6CD0D4BCC"
}
or just plain hex string: "C5192E4190E54F5985ED09C6CD0D4BCC"
You don't have to write it (or read) all at once. You can open two streams (in and out) and then stream the data. From file to response output stream or from request input stream to file.
Sorry but I am not sure if You want to send the bytes or receive them.
I need to make a byte array in which I will have header values initially and my actual message will come after the header values.
My header values will have - data center which is a string, client_id which is integer, pool_id which is also integer and data_count is also an integer.
And my actual message which will come after header values is - hello world
In my case, my header length may grow so I need to initialize that as a variable so that I can increase it later on as needed.
I am little bit confuse in how to use Byte Array here. How can I represent this in a byte array in network byte order so that c++ program can decode this out properly on ubuntu 12.04 machine?
You can use Protocol Buffers to represent the messages (header and content). It will handle the transformations between languages and different platforms. Also, it is providing room for further expansion and support for multiple message versions.
For your example you can define the message format like (eg. messageModel.proto):
package common;
option java_package = "my.java.package";
option java_outer_classname = "MessageProto";
message MyMessage {
optional string dataCenter = 1 [default = DEFAULT_DC];
optional int64 clientId = 2;
optional int64 poolId = 3;
optional int64 dataCount = 4;
optional string body = 5;
}
Then using the protoc compile like:
protoc -I src/java/ --java_out=src/java/ messageModel.proto
You will generate the transport objects and the utility classes to marshal them from one endpoint to another (representing different messages even). Please check the java tutorial for more details.
To create a MyMessage from java you will be able to do something like:
MessageProto.MyMessage.Builder mb = MessageProto.MyMessage.newBuilder();
mb.setDataCenter("aDC");
mb.setClientId(12);
mb.setPoolId(14);
mb.setDataCount(2);
mb.setbody("hello world");
MessageProto.MyMessage message = mb.build();
To transform the message into a byte array, you will use: message.toByteArray()
If C++/C is your destination you will need to generate (from the same model) the C builders and objects too. And to decode the message you will do something like:
MessageProto.MyMessage message = MessageProto.MyMessage.parseFrom(buffer);
Where buffer will represent the received content.
If this is only a homework assignment then you can serialize your header and body message using
a DataOutputStream, but I would suggest investigating Protocol Buffers as well.
Try using a DataOutputStream that is targeted to a ByteArrayOutputStream. When you're done with writing the message to the DataOutputStream, you can obtain the constructed byte array from the ByteArrayOutputStream.
Like this:
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(baos);
dos.writeInt(client_id);
dos.writeUTF(data_center);
// etc...
byte[] message = baos.toByteArray();
Protocol Buffers are also a good option, if you want more flexibility and higher performance. It depends on what you want to get out of this application; if it needs higher performance, or whether it's a one-off throwaway app or something that you expect to grow and be maintained in the longer future. DataOutputStream and DataInputStream are simple to use and you can start right away, you need to invest a bit more of your time to learn Protocol Buffers.
I'm seeking a better way to extract data from a String that contains a HTTP header. For example, I'd like to get the number 160 from the content length portion of the string: "Content-Length: 160\r\n".
It appears that all the data in the HTTP header is preceded with a name, colon and space, and after the value immediately follows the '\r' and '\n' characters.
At the moment I am doing this:
int contentLengthIndex = serverHeader.lastIndexOf("Content-Length: ");
int contentLastIndex = serverHeader.length()-1;
String contentText = serverHeader.substring(contentLengthIndex + 16, contentLastIndex);
contentText = contentText.replaceAll("(\\r|\\n)", "");
int contentLength = Integer.parseInt(contentText);
But it seems messy and it is only good for getting the "Content-Length" at the end of the string. Is there a better more universal solution for extracting values from a String containing a HTTP header that can be adjusted to work for obtaining both int values or String values?
I should also mention that the connection needs to be able return data back to the browser after a request, which from my understanding prevents me from reaping the benefits of using HttpURLConnection.
A quick solution will be:
new Scanner(serverHeader).useDelimiter("[^\\d]+").nextInt());
The other way if you want to create a Hashtable of the headers:
String[] fields = serverHeader.trim().split(":");
String key = fields[0].trim();
String value = fields[1].trim();
I am not sure why you are doing this manual, there is already API for this!
use Class java.net.HttpURLConnection
edited: also methods URLConnection.getContentLength() and URLConnection.getContentLengthLong()
Have you tried just stripping all non-numeric characters from the string?
serverHeader = serverHeader.replaceAll("[^\\d.]", "");
If you are using Socket class to read HTTP data i suggest you to use HttpURLConnection as it provides you convenient method to parse the Header Fields.
It has public Map getHeaderFields() method which you can use to get all the fields.
If you want a guide to start using HttpURLConnection you can see it here.