Related
Option 1 :if i use the TALEND job i get the JSON results stored with backslashes in the JSON file for some reason(which is unparseable)
{"read":[{"Body":"{\"items\":[{\"executionId\":\"a0613a31-d16d-4c4d-9279-4564a86cdd44\",\"startTimestamp\":\"2021-09-15T22:30:26.854Z\",\"triggerTimestamp\":\"2021-09-15T22:30:27.209Z\",\"userId\":\"user1\",\"status\":\"dispatching\",\"runtime{\"type\":\"REMOTE_ENGINE_CLUSTER\"},\"executionStatus\":\"DISPATCHING_FLOW\"},{\"executionId\":\"49a56eb1-f3c7-4f26-9554-8fa88acde38b\",\"startTimestamp\":\"2021-09-15T22:29:15.999Z\",\"triggerTimestamp\":\"2021-09-15T22:29:16.447Z\",\"userId\":\"user1\",\"executionType\":\"MANUAL\",\"status\":\"dispatching\",\"runtime\":{\"type\":\"REMOTE_ENGINE_CLUSTER\"},\"executionStatus\":\"DISPATCHING_FLOW\"}],\"limit\":100,\"offset\":0,\"total\":2}","ERROR_CODE":null}]}
Option 2:Using the WEB URL:
After REST API call,i have the JSON results like:
BODY:context.statusbody
{
"items": [{
"executionId": "4f679c12-d8d7-4dd7-89d5-507a94503988",
"startTimestamp": "2021-09-14T19:05:01.854Z",
"triggerTimestamp": "2021-09-14T19:05:02.385Z",
"userId": "user1",
"taskId": "60b7f6d31c6e394de0163d35",
"status": "dispatching",
"runtime": {
"type": "REMOTE_ENGINE_CLUSTER"
},
"executionStatus": "DISPATCHING_FLOW"
},
{
"executionId": "4f40b04c-1ac1-42ea-9a36-7c25b1b17fe8",
"startTimestamp": "2021-09-14T19:00:24.769Z",
"triggerTimestamp": "2021-09-14T19:00:25.122Z",
"userId": "user1",
"taskId": "60b7f6d31c6e394de0163d35",
"executionType": "SCHEDULED",
"status": "dispatching",
"runtime": {
"type": "REMOTE_ENGINE_CLUSTER"
},
"executionStatus": "DISPATCHING_FLOW"
}
],
"limit": 100,
"offset": 0,
"total": 2
}
From this i just need to extract the executionID and triggerTimestamp for each iteration and store in a global variable
In my grayed out tjava3 i am using global variable as:
context.testexecutionID=((String)globalMap.get("executionID"));
context.triggerTimestamp=((String)globalMap.get("triggerTimestamp"));
context.executionID=context.testexecutionID.substring(5, context.testexecutionID.lastIndexOf("\"")-3);
context.triggerTime=context.triggerTimestamp.substring(5, context.triggerTimestamp.lastIndexOf("\"")-3);
#sarah Based on your valid JSON you added in your edit .
i suggest you to add after tRest to add tExtractJSONFields component paremetred as such :
Job Design should be in first test :
tRest -> tExtractJSONFields -> tLogRow .
If extraction is good . Second test you have to see my previous my first answer to get global Variables .
Okey , Json i Used for your case to do your requirement :
Note That your Json is not Valid
{
"items": [{
"executionId": "6e5fa777-9ede-42b9-b862-03b4b1b12375",
"startTimestamp": "2021-09-15T05:59:40.599Z",
"triggerTimestamp": "2021-09-15T05:59:41.006Z",
"userId": "user"
}],
"limit ": 100,
"offset ": 0,
"total ": 2
}
Without knowing your job design i just got the output that is posted right here .
My job design is as such :
Important !: your tFileInputJson should be like this to get the 2 fields
Same configuration if your want to used tExtractJSONFields .
to Stock your Global Variables :
to get the globalVariables easy way is to use tfixedFlowInput :
Output :
UserDetails and Failures response are obtained by two seperate db calls. I want to insert the failures object inside userdetails object. The two arrays has to be associated based on the context.
This is my current JSON response :-
"Userdetails": [
{
"context": "C",
"count": 6,
"avgResponseTime": 133.8333,
"minResponseTime": 16.0,
"maxResponseTime": 293.0,
"success": 6
},
{
"context": "B",
"count": 1,
"avgResponseTime": 44.0,
"minResponseTime": 44.0,
"maxResponseTime": 44.0,
"success": 0
},
{
"context": "A",
"count": 101,
"avgResponseTime": 68.7822,
"minResponseTime": 9.0,
"maxResponseTime": 404.0,
"success": 96
}
],
"failures": [
{
"statusCode": 0,
"context": "A"
},
{
"statusCode": 400,
"context": "B"
},
{
"statusCode": 404,
"context": "C"
}
]
Response that I need :-
"Userdetails": [
{
"context": "C",
"count": 6,
"avgResponseTime": 133.8333,
"minResponseTime": 16.0,
"maxResponseTime": 293.0,
"success": 6,
"failures": [
{
"statusCode": 0,
"context": "A"
}
]
},
], and so on...
How do I insert the failures response bean inside the Userdetails bean as shown above. Any help would be appreciated.
My suggestion below
Convert the failures array and to a hashmap of type where the key is the context and the value is the corresponding failure object
Stream the UserDetails array and pick the corresponding value of context from the failures hashmap created
How to use mongodb java driver to compare dayOfYear of two ISODate objects?
Here are my docs
{"name": "hello", "count": 4, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
{"name": "hello", "count": 5, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
{"name": "goodbye", "count": 6, "TIMESTAMP": ISODate("2017-10-01T02:00:35.098Z")}
{"name": "foo", "count": 6, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
I want to compare the day in "TIMESTAMP" to perform some aggregation
Bson match = Aggregates.match(eq("name": "hello"));
Bson group = Aggregates.group(new Document("name", "$name"), Accumulators.sum("total", 1));
collection.aggregate(Arrays.asList(match, group))
Now I am not sure how to do this aggregation for all the records that belongs to particular day?
so my expected result for "2017-10-02" is
[{"_id": {"name":"hello"}, "total": 9}, {"_id": {"name":"foo"}, "total": 6}]
Given the following documents:
{"name": "hello", "count": 4, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
{"name": "hello", "count": 5, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
{"name": "goodbye", "count": 6, "TIMESTAMP": ISODate("2017-10-01T02:00:35.098Z")}
{"name": "foo", "count": 6, "TIMESTAMP": ISODate("2017-10-02T02:00:35.098Z")}
The following command ...
db.getCollection('dayOfYear').aggregate([
// project dayOfYear as an attribute
{ $project: { name: 1, count: 1, dayOfYear: { $dayOfYear: "$TIMESTAMP" } } },
// match documents with dayOfYear=275
{ $match: { dayOfYear: 275 } },
// sum the count attribute for the selected day and name
{ $group : { _id : { name: "$name" }, total: { $sum: "$count" } } }
])
... will return:
{
"_id" : {
"name" : "foo"
},
"total" : 6
}
{
"_id" : {
"name" : "hello"
},
"total" : 9
}
I think this meets the requirement expressed in your OP.
Here's the same command expressed using the MongoDB Java driver:
MongoCollection<Document> collection = mongoClient.getDatabase("stackoverflow").getCollection("dayOfYear");
Document project = new Document("name", 1)
.append("count", 1)
.append("dayOfYear", new Document("$dayOfYear", "$TIMESTAMP"));
Document dayOfYearMatch = new Document("dayOfYear", 275);
Document grouping = new Document("_id", "$name").append("total", new Document("$sum", "$count"));
AggregateIterable<Document> documents = collection.aggregate(Arrays.asList(
new Document("$project", project),
new Document("$match", dayOfYearMatch),
new Document("$group", grouping)
));
for (Document document : documents) {
logger.info("{}", document.toJson());
}
Update based on this comment:
One of the problems with project is that it only include fields you specify . The above input is just an example. I have 100 fields in my doc I can't sepecify every single one so if I use project I have to specify all 100 fields in addition to "dayOfYear" field. – user1870400 11 mins ago
You can use the following command to return the same output but without a $project stage:
db.getCollection('dayOfYear').aggregate([
// ignore any documents which do not match dayOfYear=275
{ "$redact": {
"$cond": {
if: { $eq: [ { $dayOfYear: "$TIMESTAMP" }, 275 ] },
"then": "$$KEEP",
"else": "$$PRUNE"
}
}},
// sum the count attribute for the selected day
{ $group : { _id : { name: "$name" }, total: { $sum: "$count" } } }
])
Here's that command in its 'Java form':
MongoCollection<Document> collection = mongoClient.getDatabase("stackoverflow").getCollection("dayOfYear");
Document redact = new Document("$cond", new Document("if", new Document("$eq", Arrays.asList(new Document("$dayOfYear", "$TIMESTAMP"), 275)))
.append("then", "$$KEEP")
.append("else", "$$PRUNE"));
Document grouping = new Document("_id", "$name").append("total", new Document("$sum", "$count"));
AggregateIterable<Document> documents = collection.aggregate(Arrays.asList(
new Document("$redact", redact),
new Document("$group", grouping)
));
for (Document document : documents) {
logger.info("{}", document.toJson());
}
Note: Depending on the size of your collection/your non functional requirements/etc you may want to consider the performance of these solutions and either (a) add a match stage before you start projecting/redacting or (b) extract dayOfYear into its own attribute so that you can avoid this complexity entirely.
I'm going to select data which return value from third party website as below Json objects format and that website build in Java languages and tomcat database.
Issue How to get contain from that websites?
Server description
URL:http://103.5.126.24/PalmHallServer/coodraw/coodraw!queryAllProduct.action
Return data json objects, luckDrawProductList for all product information array,
{
"LuckDrawProductList": [
{
"BaseTime": 20140119133000,
"CommissionRatio": 10,
"DeductBetCount": 1,
"Detail": "You can enter any amount between ...",
"Enable": 1,
"FreeResDiscount": 1,
"LotteryTime": 20151016164500,
"LuckDrawDesc": "DRAW1",
"LuckDrawDiscount": 1,
"MaxLuckNumLen": 5,
"MaxMoney": 2000000,
"MinLuckNumLen": 5,
"MinMoney": 100,
"NextStartTime": "2015-10-16 12:00:00",
"NextStopTime": "2015-10-16 16:15:00",
"OpenDrawType": -1,
"PeriodUnit": 1,
"PeriodUnitNum": 1,
"Price": 40,
"ProductCode": "DRAW1",
"ProductDesc": "Draw1",
"ProductEndTime": 20990119163000,
"ProductId": 11111,
"ProductType": 11,
"Rate": 0,
"ResTmplProductId": 11111,
"StopTime": 1800,
"ValidPeriod": 2
}
],
"Period": "201510161645"
}
Please try code below
$URL = "http://103.5.126.24/PalmHallServer/coodraw/coodraw!queryAllProduct.action";
$content = file_get_contents($URL); // get json data using file_get_content
$content_arr = json_decode($content); // json data to php array.
OR if you wish to use curl
$s = curl_init();
curl_setopt($s,CURLOPT_URL,$this->_url);
curl_setopt($s,CURLOPT_RETURNTRANSFER,true);
$result = curl_exec($s);
curl_close($s);
$content_arr = json_decode($result); // json data to php array.
Then you could print_r($content_arr) and check "LuckDrawProductList" the key or access $content_arr['LuckDrawProductList][0]
I have json response and I generated java code from this json by tools .But this json object have key like #size ,#order etc ... ,So in my java code some invalid identifier has been generated. When I removed this # symbol with java code I am able to catch the response of json value in my java object expect these value.
I am sure some json annotation is available to do this , means if the json key is not matching with java Object then we can map through this json annotation.
I tried with some Json annotation but is not worked for me and I confused which annotation should I used with getter methods .
here my json code below :-
{
"HotelListResponse": {
"cachedSupplierResponse": {
"#cachedTime": "0",
"#candidatePreptime": "111",
"#matchedCurrency": "true",
"#matchedLocale": "true",
"#otherOverheadTime": "4",
"#supplierRequestNum": "211",
"#supplierResponseNum": "20",
"#supplierResponseTime": "405",
"#tpidUsed": "5001"
},
"cacheKey": "302c317:13443ffb599:-7712",
"cacheLocation": "10.186.168.61:7302",
"customerSessionId": "0ABAA83D-2C31-7913-4432-FFB599907714",
"HotelList": {
"#activePropertyCount": "237",
"#size": "1",
"HotelSummary": {
"#ubsScore": "1867",
"#order": "0",
"hotelId": 127092,
"name": "The Edgewater - A Noble House Hotel",
"address1": "Pier 67, 2411 Alaskan Way",
"city": "Seattle",
"stateProvinceCode": "WA",
"postalCode": 98121,
"countryCode": "US",
"airportCode": "SEA",
"supplierType": "E",
"propertyCategory": 1,
"hotelRating": 4,
"confidenceRating": 85,
"amenityMask": 6259019,
"tripAdvisorRating": 4,
"tripAdvisorReviewCount": 590,
"tripAdvisorRatingUrl": "http://www.tripadvisor.com/img/cdsi/img2/ratings/traveler/4.0-12345-4.gif",
"locationDescription": "Near Washington State Convention & Trade Center",
"shortDescription": "<p><b>Location. </b> <br />The Edgewater - A Noble House Hotel is a business-friendly hotel located in central Seattle, close to Odyssey - The Maritime Discovery Center, Washington State Convention &",
"highRate": 249,
"lowRate": 186.75,
"rateCurrencyCode": "USD",
"latitude": 47.61252,
"longitude": -122.35013,
"proximityDistance": 11.898841,
"proximityUnit": "MI",
"hotelInDestination": true,
"thumbNailUrl": "/hotels/1000000/20000/11200/11133/11133_73_t.jpg",
"deepLink": "http://travel.ian.com/index.jsp?pageName=hotAvail&cid=55505&hotelID=127092&mode=2&numberOfRooms=2&room-0-adult-total=1&room-0-child-total=1&room-0-child-0-age=3&room-1-adult-total=1&room-1-child-total=1&room-1-child-0-age=5&arrivalMonth=8&arrivalDay=4&departureMonth=8&departureDay=5&showInfo=true&locale=en_US¤cyCode=USD",
"RoomRateDetailsList": {
"RoomRateDetails": {
"roomTypeCode": 1160,
"rateCode": 1221260,
"maxRoomOccupancy": 2,
"quotedRoomOccupancy": 2,
"minGuestAge": 0,
"roomDescription": "City Lodge - Nonrefundable",
"promoId": 200803780,
"promoDescription": "7-Day Advance Purchase Special (Nonrefundable)",
"currentAllotment": 10,
"propertyAvailable": true,
"propertyRestricted": false,
"expediaPropertyId": 11133,
"rateKey": "f3525aff-9f4d-4d92-bc1c-144628fcaa30",
"nonRefundable": true,
"RateInfos": {
"#size": "1",
"RateInfo": {
"#rateChange": "false",
"#promo": "true",
"#priceBreakdown": "true",
"RoomGroup": {
"Room": [
{
"numberOfAdults": 1,
"numberOfChildren": 1,
"childAges": 3
},
{
"numberOfAdults": 1,
"numberOfChildren": 1,
"childAges": 5
}
]
},
"ChargeableRateInfo": {
"#commissionableUsdTotal": "373.5",
"#total": "441.74",
"#surchargeTotal": "68.24",
"#nightlyRateTotal": "373.5",
"#averageBaseRate": "249.0",
"#averageRate": "186.75",
"#maxNightlyRate": "186.75",
"#currencyCode": "USD",
"NightlyRatesPerRoom": {
"#size": "1",
"NightlyRate": {
"#promo": "true",
"#rate": "186.75",
"#baseRate": "249.0"
}
},
"Surcharges": {
"#size": "1",
"Surcharge": {
"#amount": "68.24",
"#type": "TaxAndServiceFee"
}
}
}
}
}
}
}
}
}
}
}
and I generated the java classes from above mention json by this tool: http://jsongen.byingtondesign.com/
So here how to map[ #key (value of key start with #) ] in our java classes. Initially java class have #identifier_name but java compiler not support variable name start with # so I have to remove it.And then I am not able to map json value to this variable.
Any help would be appreciated.
Thanks in Advance