I am getting an InputCoercionException from Jackson when reading JSONs that contain too large numbers.
Caused by: com.fasterxml.jackson.core.exc.InputCoercionException: Numeric value (2328461700) out of range of int (-2147483648 - 2147483647)
Is it possible to configure Jackson to replace values that are out of range - for example, with a -1? If not, is there any solution to parse the value without changing the type to double?
Thank you.
As far as I can see you cannot replace values out of range using Jackson's Deserialization Features.
But you can easily create a custom deserializer and handle the values as you need.
There are tutorials and good examples on Baeldnung's or Jenkov's blogs.
Related
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 2 years ago.
Improve this question
I read and write a set of numerical values to a JSON file - they are all the 'double' datatype in Java. An example is the following:
double length = 22.4
double height = 13.1
double width = 17.9
double mass = 15.0
A JSONArray, 'objects', will then contain JSONObjects, each of which will contain the above measurements. Using the same values as above, it will look something like this (pay attention ot the datatypes):
"objects" :
[
{
"length" : 22.4,
"width" : 17.9,
"height" : 13.1,
"mass" : 15
}
]
The first issue posed by such a document is the fact that reading 'length', 'width' and 'height' will return a datatype 'BigDecimal' which seems unnormal for the org.json library which to my knowledge only returns double and integer. This is a problem because I need to convert the 'BigDecimal' values back to double although I wrote the JSON object using standard java double values. Additionally, the org.json library seems to convert any 'double' datatypes to integers when read (notice how mass is '15' and not '15.0'? This is a problem because as you can probably tell, the numerical value of "mass" may be different for another JSONObject (and thus indeed be a double). You can probably imagine the hell, that is iterating through this JSONArray when random values are an integer...
In summary, my problem is the fact that reading JSON files with the org.json library means that numerical values are only returned as doubles or BigDecimals. My question is if I can specify what datatype I would like org.json to write? All I want, is to read and write all my numerical values as a double - I don't want org.json to choose which datatype it will write to the JSON file.
My ideas were the following:
Writing everything as a string and converting back to double when reading
My attempts to solve this that haven't worked or are redundant to me:
Using the valueOf method to change between datatypes
Writing 9999 conditions to find the right instances to convert back to double (which sucks)
Using doubleValue() method
All I want, is to read and write all my numerical values as a double - I don't want org.json to choose which datatype it will write to the JSON file.
For what it is worth, the JSON library is not choosing a data type. It is choosing a character based representation to express a value. And there is some logical justification for it doing what it is doing. JSON is (or at least originated as) a subset of Javascript. In Javascript, integer and floating point are not distinct types. There is just one primitive data type for all numbers in Javascript.
I don't think that this can be solved using the org.json code. The stripping of trailing zeros after a decimal point happens in the static JSONObject::numberToString. Since it is a static method, you can't override it in a subclass.
(Take a look at the source code.)
My suggestions:
Work with the current behavior. For example, your code could be written know which fields are supposed to have floating point values, and use doubleValue() to retrieve their values.
Find an alternative JSON library that doesn't do this. (Based on my reading of the code, I think Jackson would be OK. I haven't checked any other libraries ...)
You could create your own private fork of the org.json library and modify it, but then you would be stuck with the overhead of maintaining that fork indefinitely.
I think that 3 is a bad idea. I have only included it for completeness.
When I try to serialize an object to JSON using Jackson object mapper, it works perfectly.
{"id":1291741231928705024,"uuid":null,"email":"kannanrbk.r#gmail.com"}
Whereas, when I try to access it using spring rest controller. The long value numbers are rounded off, the last 3 digits.
I read existing questions in the stackoverflow, most of them suggest changing the datatype to string. But we used the Long value reference in most of the places, changing datatype will need some refactoring.
I did my initial analysis:
We are using Jackson ObjectMapper
From Spring, it indirectly calls MappingJackson2HttpMessageConverter
This problem might be somewhere around the JSONParser, where it treats any number as a double (15 digits) and after that, it's rounded off
Is there any way to fix this issue?
Is there any way to fix this issue?
There is no problem with Jackson/Java/Spring Boot, but with JavaScript/Browser.
Trying to reproduce the issue I serialized the same object and got this using curl:
$ curl localhost:8080
{"id":1291741231928705024,"uuid":null,"email":"kannanrbk.r#gmail.com"}
Here the number is correctly serialized.
The same json viewed in Firefox does truncate:
However the "Raw Data" tab displays the number correctly:
.
In JavaScript 1291741231928705024 is not safe integer (see Number.isSafeInteger()):
Number.isSafeInteger(1291741231928705024);
false
The number is greater than 2^53 - 1 so it gets rounded. Even more confusing situations are possible in JavaScript:
> 1291741231928705024 === 1291741231928705022
true
Possible solution
First of all check your client against this kind of problems. If it can safely deserialize such numbers then you're safe.
Or you can serialize longs as Strings (as you mentioned in the question), this is what Twitter proposes in its Twitter IDs (snowflake) article:
To allow Javascript and JSON parsers to read the IDs, Twitter objects include a string version of any ID when responding with JSON. Status, User, Direct Message, Saved Search and other IDs in the Twitter API are therefore returned as both an integer and a string in JSON responses.
Try with bigInt for your primary key
My goal is to synchronize abitrary rows of data by using the JSON-Format.
As I do not know the exact scheme for the rows (it is a general sync method), my datamodel apparently has to rely on "Object". So in Java I will have an array of Map<String,Object> to be synchronized with the server.
Translating such a row into JSON would give something like
{{"string":"stringvalue"},{"double1":1234.567},{"double2":1234.0},{"long":1234}}
so far, so good - no problem with moshi - everything works as expected.
Now the Problem: When I try to deserialize that JSON with moshi, I get back a double-value for the "long" member. Moshi converts all numbers to Doubles. But unfortunately not all numbers can be safely converted to doubles. Very big integers (aka longs) have a problem with the limited precision of doubles. And rounding-effects also might exist.
I opened an issue with moshi, but unfortunately that was closed. Maybe I wasn't clear enough. (Issue 192)
JSON has no concept of integer - only numbers and Strings. But the subtle detail from "double2" from the example above might lead to a solution for my problem:
If a number does not contain a decimal-point, it is an integer and should be converted to a long.
As longs can not be losslessly converted to doubles, I need a method to intercept the parser before the value is converted to double. But how to do that?
Moshi has this handy concept of JsonAdapters - but unfortunately I currently do not see how I can use them in this case:
The input-type of such an JsonAdapter would have to be Object because I can not cast a generated double to long. So I have to intercept the parser before he converts any value.
But how to return more than one type from there? (I would have to return String, Double or Long from there - or if I can limit the inputs to only numbers I would at least have to return Longs or Doubles.)
(My backend is written in PHP and automatically produces the desired output: Integers are written without a decimal-point.)
I am afraid it's not possible without changing Moshi source code. The JSON string source passes through JsonReader which converts all numbers to double. I could not find a way how to alter this behavior since all subclasses are package-protected.
Given a protobuf serialization is it possible to get a list of all tag numbers that are in the message? Generally is it possible to view the structure of the message without the defining .proto files?
Most APIs will indeed have some form of reader-based API that allows you to enumerate a raw protobuf stream. However, that by itself is not enough to fully understand the data, since without the schema the interpretation is ambiguous:
a varint could be zig-zag encoded (sint32/sint64), or not (int32/int64/uint32/uint64) - radically changing the meaning, or a boolean, or an enum
a fixed-32/fixed-64 could be a signed or unsigned integer, or could be an IEEE754 float/double
a length-prefixed chunk could be a UTF-8 string, a BLOB, a sub-message, or a "packed" repeated set of primitives; if it is a sub-message, you'll have to repeat recursively
So... yes and no. Certainly you can get the field numbers of the outermost message.
Another approach would be to use the regular API against a type with no members (message Naked {}), and then query the unexpected data (i.e. all of it) via the "extension" API that many implementations provide.
You can get all the tag numbers which appear in one particular message, but you won't get any nested messages - and you won't know the types of those fields, only some subset of possible types.
If you look at the wire encoding, you can see that (for example) byte arrays, strings and nested messages are all encoded the same way - so you may know that "field 3 is a length-prefixed binary field" but you won't know whether that means it's a nested message, a string or a byte array.
Hallo,
I do not understand, how to query cassandra with hector, but the column-values returned are not of one single type, but of many:
I put in ???? where I do not know what to do:
MultigetSliceQuery<String, String, ??????> multigetSliceQuery = HFactory.createMultigetSliceQuery(keyspace, stringSerializer, stringSerializer, ???????);
For for example, if all my column-values are of String type, I would put in String. But I have differnt ones like String, Integer, and byte[] in one single row. So I must pass in more then one Serializer.
How to solve this?
Thank you.
(can one pls create hector as tag? I am not allowed to do this)
We provide arbitrary typing via a template-method pattern. See the test case:
https://github.com/rantav/hector/blob/master/core/src/test/java/me/prettyprint/cassandra/service/template/ColumnFamilyTemplateTest.java
The new getting started guide should be helpful as well:
https://github.com/rantav/hector/wiki/Getting-started-%285-minutes%29
You can use the ByteBufferSerializer, and then convert the ByteBuffers returned from ByteBufferSerializer as the argument to StringSerializer and IntegerSerializer to convert the columns which are Strings and Integers.