I'm running a java code using Apache Avro. Some code gets deprecated in the java file and I'm not sure why.
I'm using Maven to run my Java program.
This is the java file
public class AvroAddressTest {
public int tempRand;
static String[] NAMES = { "Karthik", "Sam", "Joe", "Jess", "Tom",
"Huck", "Hector", "Duke", "Jill", "Natalie", "Chirsta", "Ramya" };
static String[] EMAILS = { "kar#gmail.com", "steve#gmail.com",
"garry#gmail.com", "kumar#hotmail.com", "dave#hotmail.com",
"will#hotmail.com", "rick#ymail.com", "vinod#ymail.com",
"basu#ymail.com", "sachin#ymail.com", "chester#ymail.com",
"anand#ymail.com" };
static String[] PHONE_NUMBERS = { "9940099321", "9940099456",
"9934099333", "9940099567", "9940077654", "9940088323",
"9940097543", "9940099776", "9940000981", "9940088444",
"9940099409", "9940033987" };
static int[] AGES = { 32, 43, 23, 21, 55, 34, 33, 31, 22, 41, 56, 62 };
static boolean[] STU = { true, false, true, true, false, false, true, false, true, false, false, true };
public void serializeGeneric() throws IOException {
// Create a datum to serialize.
Schema schema = new Schema.Parser().parse(getClass()
.getResourceAsStream("/AddressRec.avsc"));
GenericRecord datum = new GenericData.Record(schema);
Random random = new Random();
int randInt = random.nextInt(NAMES.length);
datum.put("name", new Utf8(NAMES[randInt]));
datum.put("email", new Utf8(EMAILS[randInt]));
datum.put("phone", new Utf8(PHONE_NUMBERS[randInt]));
datum.put("age", AGES[randInt]);
datum.put("student", STU[randInt]);
//datum.put("door",new Utf8(NAMES[randInt]) );
// Serialize it.
ByteArrayOutputStream out = new ByteArrayOutputStream();
DatumWriter<GenericRecord> writer = new GenericDatumWriter<GenericRecord>(
schema);
Encoder encoder = EncoderFactory.get().binaryEncoder(out, null);
writer.write(datum, encoder);
encoder.flush();
out.close();
System.out.println("\nSerialization: " + out);
// Deserialize it.
DatumReader<GenericRecord> reader = new GenericDatumReader<GenericRecord>(
schema);
BinaryDecoder decoder = DecoderFactory.get().binaryDecoder(
out.toByteArray(), null);
GenericRecord result = reader.read(null, decoder);
System.out.printf(
"Deserialized output:\nName: %s, Email: %s, Phone: %s, Age: %d, Student?: %s\n\n",
result.get("name"), result.get("email"), result.get("phone"),
result.get("age"), result.get("student"));
}
public void serializeSpecific() throws IOException {
// Create a datum to serialize.
AddressRec datum = new AddressRec();
Random random = new Random();
int randInt = random.nextInt(NAMES.length);
datum.**name** = new Utf8(NAMES[randInt]);
datum.**email** = new Utf8(EMAILS[randInt]);
datum.**phone** = new Utf8(PHONE_NUMBERS[randInt]);
datum.**age** = AGES[randInt];
datum.**student** = STU[randInt];
File tmpFile = File.createTempFile("AddressRecAvroExample", ".avro");
// Serialize it.
DataFileWriter<AddressRec> writer = new DataFileWriter<AddressRec>(
new SpecificDatumWriter<AddressRec>(AddressRec.class));
writer.create(AddressRec.SCHEMA$, tmpFile);
writer.append(datum);
writer.close();
System.out.println("\nSerialization to tempfile: " + tmpFile);
// Deserialize it.
FileReader<AddressRec> reader = DataFileReader.openReader(tmpFile,
new SpecificDatumReader<AddressRec>(AddressRec.class));
while (reader.hasNext()) {
AddressRec result = reader.next();
System.out.printf("Deserialized output:\nName: %s, Email: %s, Phone: %s, Age: %d, Student?: %s\n\n",
result.**name**, result.**email**, result.**phone**,
result.**age**, result.**student**);
}
reader.close();
}
#Test
public void serializeTest() throws IOException {
serializeGeneric();
serializeSpecific();
}
}
What is the problem? The code in block is getting deprecated.
This is the .avsc file
{
"type": "record",
"name": "AddressRec",
"namespace":"com.mycompany.samples.avro",
"fields": [
{"name": "name", "type": "string"},
{"name": "email", "type": "string"},
{"name": "phone", "type": "string"},
{"name": "age", "type": "int"},
{"name": "student", "type": "boolean"}
]
}
The program is running fine . Its just that some code is deprecated. The same code is not deprecated when i use version 1.5.1
The only thing I can think of (since you didn't provide us with the actual warning messages) is that instead of directly accessing the field values (datum.foo = x) you should use accessor methods.
Related
I want to add a new field to jsonObject and this new field's name will be based on a value of another field. To be clear, this an examples of what I want to achieve.
{
"values": [
{
"id": "1",
"properties": [
{
"stat": "memory",
"data": 8
},
{
"stat": "cpu",
"data": 4
}
]
},
{
"id": "2",
"properties": [
{
"stat": "status",
"data": "OK"
},
{
"stat": "cpu",
"data": 4
}
]
}
]
}
I want to add a new field to each json object that will have the value of field "stat" as name.
{
"values": [
{
"id": "1",
"properties": [
{
"stat": "memory",
"data": 8,
"memory": 8
},
{
"stat": "cpu",
"data": 4,
"cpu": 4
}
]
},
{
"id": "2",
"properties": [
{
"stat": "status",
"data": 0,
"status": 0
},
{
"stat": "cpu",
"data": 4,
"cpu": 4
}
]
}
]
}
I have tried to do the following with JsonPath library but for me it's an ugly solution as I will parse the json three times and I do some manual replacements.
val configuration = Configuration.builder().options(Option.DEFAULT_PATH_LEAF_TO_NULL, Option.ALWAYS_RETURN_LIST).build()
val jsonContext5 = JsonPath.using(configuration).parse(jsonStr)
val listData = jsonContext.read("$['values'][*]['properties'][*]['data']").toString
.replace("[", "").replace("]", "").split(",").toList
val listStat = jsonContext.read("$['values'][*]['properties'][*]['stat']").toString
.replace("[", "").replace("]", "")
.replace("\"", "").split(",").toList
// Replacing values of "stat" by values of "data"
jsonContext5.map("$['values'][*]['properties'][*]['stat']", new MapFunction() {
var count = - 1
override def map(currentValue: Any, configuration: Configuration): AnyRef = {
count += 1
listData(count)
}
})
// replace field stat by its value
for( count <- 0 to listStat.size - 1){
val path = s"['values'][*]['properties'][$count]"
jsonContext5.renameKey(path, "stat", s"${listStat(count)}")
}
This is the result obtained
{
"values": [
{
"id": "1",
"properties": [
{
"data": 8,
"memory": "8"
},
{
"data": 4,
"cpu": "4"
}
]
},
{
"id": "2",
"properties": [
{
"data": 0,
"memory": "0"
},
{
"data": 4,
"cpu": "4"
}
]
}
]
}
Is there any better method to achieve this result ? I tried to do it with gson but it's not good handling paths.
This a way to do it with Gson but I will lose the information about other columns since I'm creating another json.
val jsonArray = jsonObject.get("properties").getAsJsonArray
val iter = jsonArray.iterator()
val agreedJson = new JsonArray()
while(iter.hasNext) {
val json = iter.next().getAsJsonObject
agreedJson.add(replaceCols(json))
}
def replaceCols(json: JsonObject) = {
val fieldName = "stat"
if(json.has(fieldName)) {
val columnName = json.get(fieldName).getAsString
val value: String = if (json.has("data")) json.get("data").getAsString else ""
json.addProperty(columnName, value)
}
json
}
How about something like this?
private static void statDup(final JSONObject o) {
if (o.containsKey("properties")) {
final JSONArray a = (JSONArray) o.get("properties");
for (final Object e : a) {
final JSONObject p = (JSONObject) e;
p.put(p.get("stat"), p.get("data"));
}
} else {
for (final Object key : o.keySet()) {
final Object value = o.get(key);
if (value instanceof JSONArray) {
for (final Object e : (JSONArray) value) {
statDup((JSONObject) e);
}
}
}
}
}
Using Gson, what you should do is create a base class that represents your initial JSON object. Then, extend that class and add the additional attribute(s) you want to add, such as "stat". Then, load the JSON objects into memory, either one by one or all together, then make the necessary changes to each to encompass your changes. Then, map those changes to the new class if you didn't in the prior step, and serialize them to a file or some other storage.
This is type-safe, a pure FP circe implementation with circe-optics:
object CirceOptics extends App {
import cats.Applicative
import cats.implicits._
import io.circe.{Error => _, _}
import io.circe.syntax._
import io.circe.parser._
import io.circe.optics.JsonPath._
val jsonStr: String = ???
def getStat(json: Json): Either[Error, String] =
root.stat.string.getOption(json)
.toRight(new Error(s"Missing stat of string type in $json"))
def getData(json: Json): Either[Error, Json] =
root.data.json.getOption(json)
.toRight(new Error(s"Missing data of json type in $json"))
def setField(json: Json, key: String, value: Json) =
root.at(key).setOption(Some(value))(json)
.toRight(new Error(s"Unable to set $key -> $value to $json"))
def modifyAllPropertiesOfAllValuesWith[F[_]: Applicative](f: Json => F[Json])(json: Json): F[Json] =
root.values.each.properties.each.json.modifyF(f)(json)
val res = for {
json <- parse(jsonStr)
modifiedJson <- modifyAllPropertiesOfAllValuesWith { j =>
for {
stat <- getStat(j)
data <- getData(j)
prop <- setField(j, stat, data)
} yield prop
} (json)
} yield modifiedJson
println(res)
}
The previous answer from Gene McCulley gives a solution with Java and using class net.minidev.json. This answer is using class Gson and written in Scala.
def statDup(o: JsonObject): JsonObject = {
if (o.has("properties")) {
val a = o.get("properties").getAsJsonArray
a.foreach { e =>
val p = e.getAsJsonObject
p.add(p.get("stat").getAsString, p.get("data"))
}
} else {
o.keySet.foreach { key =>
o.get(key) match {
case jsonArr: JsonArray =>
jsonArr.foreach { e =>
statDup(e.getAsJsonObject)
}
}
}
}
o
}
Your task is to add a new field to each record under each properties in the JSON file, make the current stat value the field name and data values the new field values. The code will be rather long if you try to do it in Java.
Suggest you using SPL, an open-source Java package to get it done. Coding will be very easy and you only need one line:
A
1
=json(json(file("data.json").read()).values.run(properties=properties.(([["stat","data"]|stat]|[~.array()|data]).record())))
SPL offers JDBC driver to be invoked by Java. Just store the above SPL script as addfield.splx and invoke it in a Java application as you call a stored procedure:
…
Class.forName("com.esproc.jdbc.InternalDriver");
con= DriverManager.getConnection("jdbc:esproc:local://");
st = con.prepareCall("call addfield()");
st.execute();
…
My current JSON file:
{
"dev": {
"name": "James",
"address": "12 VK Street",
"city": "Austin"
},
"test": {
"name": "Roy",
"address": "15 VK Street",
"city": "New York"
}
}
I have another environment called beta.
Need to check if beta node exists in current JSON file.
If beta exists in current JSON file then its ok and no changes in JSON file.
If beta node does not exists then I need to add node for beta and update JSON file to following.
{
"dev": {
"name": "James",
"address": "12 VK Street",
"city": "Austin"
},
"test": {
"name": "Roy",
"address": "15 VK Street",
"city": "New York"
}
"beta": {
}
}
I tried following code but it didn't work.
Checking if JSON has desired object already. If not then create a new one
JSONObject main = new JSONObject(jsonString);
if (!main.has("beta")) {
main.put("beta", new JSONObject());
// main.put("beta", ""); didn't work
// main.put("beta", "{}"); didn't work
}
How can I achieve this in Java ? I tried using Jackson, Simple JSON but it didn't work out. Thanks!
===========================
Got it working
private void addReplaceValueInJson(String key, String value) throws Exception {
File jsonFile = new File("testdata.json");
String jsonString = FileUtils.readFileToString(jsonFile, Charset.defaultCharset());
String env = System.getProperty(Constant.ENV);
JsonElement mainElement = new JsonParser().parse(jsonString);
JsonObject mainObject = mainElement.getAsJsonObject();
// If env node is not present in json then create one
if (!mainObject.has(env)) {
JsonElement envElement = new JsonParser().parse(new JSONObject("{}").toString());
mainObject.add(env, envElement);
}
try {
mainObject.getAsJsonObject(env).addProperty(key, value);
String resultingJson = new Gson().toJson(mainElement);
FileUtils.writeStringToFile(jsonFile, resultingJson, Charset.defaultCharset());
} catch (Exception e) {
System.out.println("JSONObject: " + env + " missing in testdata.json. Please add " + env + " node.");
}
}
This will add child jsonobject if it is not present.
And update key's value or add new one in place of entire file flush n close
this code works perfectly if there is no field called _transferSize,
sample har :
"cookies": [
{
"name": "x-amz-captcha-1",
"value": "1577615972915416",
"path": "/",
"expires": "2020-12-28T08:39:32.000Z",
"httpOnly": false,
"secure": false
},
{
"name": "x-amz-captcha-2",
"value": "PKRgLIISQDY1ubrOgWIOQQ==",
"path": "/",
"expires": "2020-12-28T08:39:32.000Z",
"httpOnly": false,
"secure": false
}
],
"content": {
"size": 0,
"mimeType": "x-unknown"
},
"redirectURL": "/",
"headersSize": -1,
"bodySize": -1,
"_transferSize": 553
},
I was trying to get all properties by HarLib java library.
code
ObjectMapper mapper = new ObjectMapper();
mapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
System.out.println("Harpath : " + harPath);
File f = new File(harPath);
HarFileReader r = new HarFileReader();
HarLog log = r.readHarFile(f);
HarEntries entries = log.getEntries();
System.out.println(entries);
// HarHeaders s = entries.getEntries().get(1).getResponse().getHeaders();
// List<HarHeader> ss = s.getHeaders();
// for (HarHeader harHeader : ss) {
// System.out.println(harHeader.getValue());
// }
// System.out.println("elements on entries : " + entries.getEntries().size());
//System.out.println(ss.get(5).getName() +"="+ss.get(5).getValue());
can anyone help me to get values of each KEYS, so that can get map and get all values?
It seems like you tried to customize the ObjectMapper but actually never used it. Try something like this:
ObjectMapper mapper = new ObjectMapper();
mapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
// ...
File f = new File(harPath);
JsonParser parser = mapper.getFactory().createParser(f);
List<HarWarning> warnings = new ArrayList<HarWarning>();
HarLog log = r.readHarFile(parser, warnings);
// ...
I have complex API that i parse and show in list view,I will be struggle to parse JSONArray.Here i will be in struggle following Json Array which is inside the posts json object "tags_name": ["Activities"],,some object it will come like "tags_name": [], this.Kindly review my question. My API and code is below. Presently i will implemented parsing code with model class. Once fix this issue i have to write list view coding please help me. May be my question formation is in silly way. please it look like means give some suggestion to frame question. Thanks in Advance.
MyAPI:
{
"status": true,
"nextpage": 0,
"count": 31,
"data": {
"postlist": [{
"posts": {},
"tags_name": ["Activities"],
"images_count": 3,
"images": [],
"post_user": [],
"is_encourage_user": true,
"encourage_feed_id": "1093"
},
{
"posts": {"id": "4647"},
"tags_name": [],
"images_count": 0,
"images": [],
"post_user": [],
"is_encourage_user": true,
"encourage_feed_id": "992"
}
]
},
"token": "wqeeqweqweqweqweqsfsdfsdf"
}
My Method for Parsing
private void parsingPostValues(String responseStatus) throws JSONException {
JSONObject responseObject = new JSONObject(responseStatus);
JSONObject datObject = responseObject.getJSONObject("data");
JSONArray postArr = new JSONArray(datObject.getString("postlist"));
for (int i = 0; i < postArr.length(); i++) {
JSONObject tempPostObject = postArr.getJSONObject(i);
JSONObject postObject = tempPostObject.getJSONObject("posts");
//setTag Array- this is the functional area i'm in bottle-neck.
try {
JSONArray tagNameArr = tempPostObject.getJSONArray("tags_name");
//ArrayList<Bean> tagListdata = new ArrayList<Bean>(tagNameArr.length());
if (tagNameArr.length()>0) {
for (int tagInfo = 0; tagInfo < tagNameArr.length(); tagInfo++) {
// listdata.add(tagNameArr.get(i).toString());
String tagme = "";
//Bean tagBean = new Bean();
//tagBean.setTagsArray((tagme.isEmpty() ? tagNameArr.get(tagInfo).toString() : "null")); //tagBean.setTagsArray(tagNameArr.get(tagInfo).toString());
//tagListdata.add(tagBean);
//beanAccess.setTagsArray(tagNameArr.get(tagInfo));
System.out.println("Tags Array:"+tagInfo+":"+tagNameArr.get(tagInfo));
}
//beanAccess.setTagsArray(tagListdata);
}
} catch (Exception e) {
e.printStackTrace();
}
}
replace this
JSONArray postArr = new JSONArray(datObject.getString("postlist"));
To
JSONArray postArr = datObject.getJSONArray("postlist");
Replace
String imgCount = tempPostObject.getString("images_count");
String is_encourage_user = tempPostObject.getString("is_encourage_user");
To
String imgCount = postObject.getString("images_count");
String is_encourage_user = postObject.getString("is_encourage_user");
It will work for you.
I have a question about getting jsonobjects out of a jsonobject.
This is what im getting back:
{
"data": {
"someid": {
"air_by_date": 0,
"cache": {
"banner": 1,
"poster": 1
},
"language": "en",
"network": "somenetwork",
"next_ep_airdate": "somedate",
"paused": 0,
"quality": "somequality",
"show_name": "somename",
"status": "Continuing",
"tvdbid": someid,
"tvrage_id": someid,
"tvrage_name": "Showname"
},
"someid": {
"air_by_date": 0,
"cache": {
"banner": 1,
"poster": 1
},
"language": "en",
"network": "somenetwork",
"next_ep_airdate": "",
"paused": 0,
"quality": "somequality",
"show_name": "somename",
"status": "Continuing",
"tvdbid": someid,
"tvrage_id": someid,
"tvrage_name": "somename"
},
But how am i supposed to create "ShowObjects" of them. I know how it works with JSONArrays but i have never done this kind of JSON before.
This is what i got untill now:
String json = download(url);
JSONObject result = new JSONObject(json);
JSONObject resultData = result.getJSONObject("data");
Copied from this answer:
Use the keys() iterator to iterate over all the properties, and call get() for each.
Iterator<String> iter = json.keys();
while (iter.hasNext()) {
String key = iter.next();
try {
Object value = json.get(key);
} catch (JSONException e) {
// Something went wrong!
}
}
Try using Jackson or Gson for these tasks.
With Jackson you would do something like this:
class Response() {
public String message;
public String result;
public Map<String, Object> data;
}
ObjectMapper mapper = new ObjectMapper()
Resonse respone = mapper.readValue(json, Response.class)
or use a custom deserializer to read those values out.
I will give you an example how to fetch the objects and vars in your "data" json:
lets say the second "someid" is : "123". We will fetch the second jsonObject now :
JSONObject secondJsonSomeId = resultData.getJsonObject("123");
Now we will fetch the "banner" ( int =1 ) of the jsonObject named "cache" located in our secondJsonSomeId :
int bannerInsecondJsonSomeId = secondJsonSomeId.getJsonObject("cache").getInt("banner");// bannerInsecondJsonSomeId == 1
the code is not compiled so there might be errors , ask if you have additional questions.