Develop Reference

Filtering Shodan Query Results based on Location

I am attempting to parse Shodan query results and write a new JSON file with only the results that match the criteria I have set.
Example JSON entries:
{
"matches": [
{
"product": "Microsoft IIS httpd",
"hostnames": [],
"hash": -1722221328,
"ip": 1261462342,
"isp": "AT&T Internet Services",
"transport": "tcp",
"cpe": [
"cpe:/a:microsoft:iis:7.5",
"cpe:/o:microsoft:windows"
],
"data": "",
"asn": "AS7018",
"port": 631,
"version": "7.5",
"link": "Ethernet or modem",
"location": {
"city": null,
"region_code": null,
"area_code": null,
"longitude": -97.822,
"country_code3": "USA",
"latitude": 37.751000000000005,
"postal_code": null,
"dma_code": null,
"country_code": "US",
"country_name": "United States"
},
"timestamp": "2017-02-28T23:55:24.306344",
"domains": [],
"org": "AT&T Internet Services",
"os": null,
"_shodan": {
"crawler": "122dd688b363c3b45b0e7582622da1e725444808",
"id": null,
"module": "http-simple-new",
"options": {}
},
"ip_str": "75.48.99.70"
},
{
"hash": 605323305,
"ip": 1757819678,
"isp": "Google Cloud",
"transport": "tcp",
"data": "",
"asn": "AS15169",
"port": 9000,
"hostnames": [
"30.51.198.104.bc.googleusercontent.com"
],
"location": {
"city": "Mountain View",
"region_code": "CA",
"area_code": 650,
"longitude": -122.0574,
"country_code3": "USA",
"latitude": 37.41919999999999,
"postal_code": "94043",
"dma_code": 807,
"country_code": "US",
"country_name": "United States"
},
"timestamp": "2017-02-28T23:51:35.997036",
"domains": [
"googleusercontent.com"
],
"org": "Google Cloud",
"os": null,
"_shodan": {
"crawler": "545144fc95e7a7ef13ece5dbceb98ee386b37950",
"id": null,
"module": "https-simple-new",
"options": {}
},
"ip_str": "104.198.51.30"
}
],
"total": 2
}
My hope is to load the JSON file and iterate over the set of elements, removing an element if it does not match the criteria of having a location country_code of "US".
The code I have (courtesy of https://gist.github.com/madonnelly and Iterate over JsonObject properties) is as follows:
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.util.Map;
import java.util.Set;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
public class ParseJSON {
public static void main(String[] args) {
JsonObject shodanJSON = convertFileToJSON("<Path to JSON file>");
Set<Map.Entry<String,JsonElement>> queryResults = shodanJSON.entrySet();
for (Map.Entry<String, JsonElement> queryResult : queryResults) {
JsonArray locArray = queryResult.getValue().getAsJsonObject().getAsJsonArray("location");
for (JsonElement locData : locArray) {
if (locData.getAsJsonObject().getAsJsonPrimitive("country_code").equals("US")) {
System.out.println(locData.getAsString());
}
}
}
}
public static JsonObject convertFileToJSON(String fileName) {
// Read from File to String
JsonObject jsonObject = new JsonObject();
try {
JsonParser parser = new JsonParser();
JsonElement jsonElement = parser.parse(new FileReader(fileName));
jsonObject = jsonElement.getAsJsonObject();
} catch (FileNotFoundException e) {
}
return jsonObject;
}
}
When I run my code I am receiving the error
Exception in thread "main" java.lang.IllegalStateException: Not a JSON
Object: [{"product":"Microsoft IIS
httpd","hostnames":[],"hash":-1722221328,"ip":1261462342,"isp":"AT&T
Internet Services","transport":...}] at
com.google.gson.JsonElement.getAsJsonObject(JsonElement.java:90) at
com.cti.shodan.ParseJSON.main(ParseJSON.java:22)
I am sure I am making a ton of mistakes and am hoping someone can point out the error I am making. Thanks in advance!

You have some assumptions about the concrete JSON document parsing that do not meet its real structure. I'm assuming you're going to display the matched results ($.matches) filtering out on their subproperty values ($.matches.*.location.country_code).
for ( final Entry<String, JsonElement> queryResult : shodanJsonObject.entrySet() ) {
final JsonElement value = queryResult.getValue();
// This is necessary to skip <"total": 2>
if ( value.isJsonArray() ) {
// Here comes an array, and should be iterated, rather than taken as an object
for ( final JsonElement match : value.getAsJsonArray() ) {
// This was the root cause, not an array
final JsonObject location = match.getAsJsonObject().getAsJsonObject("location");
// Previously jsonPrimitive.equals("US") -- convert the JSON primitive to a string first
if ( location.getAsJsonPrimitive("country_code").getAsString().equals("US") ) {
// Previously getAsString() -- it requires a JSON string literal, just remove it
System.out.println(match);
}
}
}
}
With Java 8 it may be somewhat more simple:
shodanJsonObject.entrySet()
.stream()
.map(Entry::getValue)
.filter(JsonElement::isJsonArray)
.map(JsonElement::getAsJsonArray)
.flatMap(jsonElements -> StreamSupport.stream(jsonElements.spliterator(), false))
.peek(System.out::println)
.map(JsonElement::getAsJsonObject)
.map(jsonObject -> jsonObject.getAsJsonObject("location"))
.filter(location -> location.getAsJsonPrimitive("country_code").getAsString().equals("US"))
.forEach(jsonObject -> {
}); // forEach is a terminal operation and it "pushes" the entire chain above
And probably the most expressive way if it's possible to use querying libraries like JsonPath:
final JsonPath jsonPath = JsonPath.compile("$.matches.*[?(#.location.country_code=='US')]");
for ( final Object match : jsonPath.<JSONArray>read(JSON) ) {
System.out.println(match);
}

Print an Arraylist to RESTful webservices

first of all, my question would be a bit long but i dont think its complex. But i simply have not clue where the problem could be.
So let me start.
Im trying to print an Arraylist into webpages. I have a json file like this:
[
{
"id": 0,
"brand": "audi",
"model": "q8",
"color": "red",
"price": "123",
"available": false
},
{
"id": 1,
"brand": "audi",
"model": "r6",
"color": "sfg",
"price": "952",
"available": true
},
{
"id": 2,
"brand": "BMW",
"model": "IDK",
"color": "Red",
"price": "105",
"available": true
}
]
I read the json file and save the data into an arraylist "carList" with this method
try(JsonReader jsonReader = new JsonReader(new InputStreamReader(new FileInputStream(this.db)))){
Gson myGson = new Gson();
JsonParser jsonParser = new JsonParser();
JsonArray array = jsonParser.parse(jsonReader).getAsJsonArray();
this.carList = new ArrayList<Car>();
for (JsonElement element : array) {
Car car = myGson.fromJson(element, Car.class);
carList.add(car);
}
}catch(IOException e){
e.printStackTrace();
}
My "Car" class has those variables
private int id = 0;
private String brand;
private String model;
private String color;
private String price;
private boolean available = true;
And when i print my "carList" with this
public ArrayList<Car> printAllCars(){
return carList;
}
and this
#RequestMapping(value = "/cars", method = RequestMethod.GET)
#ResponseBody
ArrayList<Car> printAllCars() {
return cars.printAllCars();
}
I got this with postman
postman result
So as you can see, the problem is the price is not printed and "available" is printed as "state".
But when i print in eclipse the price is displayed.
eclipse result
Sorry for my long post, my bad english and my bad code but please help me guys !

Jackson looks at get...() and set...() methods, so don't forget to add these methods or us filed-based serialization.

How to get specific object from RestTemplate exchange method?

I have an end-point which returns me this response:
{
"head": {
"status": 200,
"ok": true,
"messages": [],
"errors": [],
"references": {}
},
"body": {
"id": "d57a9c7aef9842c2e31a0f49c",
"flowId": "f57979d06f9842c3e94f1f197",
"creationDate": 1470744494732,
"path": "/luafanti/test",
"version": 0,
"elems": {
"xxx": {
"type": "integer",
"value": 200
}
}
}
}
My question is, how to make a model that can be populated with only a part of my json response. For example, with this:
"xxx": {
"type": "integer",
"value": 200
}
or this:
"elems": {
"xxx": {
"type": "integer",
"value": 200
}
}

Using Jackson, you can define your model as the following:
#JsonIgnoreProperties(ignoreUnknown=true)
public class MyResponseModel {
private Body body;
public void setBody(Body body) {this.body = body;}
public Body getBody() {return body;}
#JsonIgnoreProperties(ignoreUnknown=true)
public static class Body {
private Elems elems;
// getter and setter for elems
}
#JsonIgnoreProperties(ignoreUnknown=true)
public static class Elems {
private Xxx xxx;
// getter and setter for xxx
}
#JsonIgnoreProperties(ignoreUnknown=true)
public static class Xxx {
private String type;
private String value;
// getter and setter for type and value
}
}
The above is quite verbose, particularly if you are only interested in a very small part of the response. It may be more practical to handle the response as a String and then use e.g. JsonPath to extract only the data you are interested in.

You can use simple-json.jar to extract that object from inside the JSONObject
Downloadable Jar Link - simple-json.jar Download Link
Maven Jar Import Maven Repository pom syntax
You actual object is
{
"head": {
"status": 200,
"ok": true,
"messages": [],
"errors": [],
"references": {}
},
"body": {
"id": "d57a9c7aef9842c2e31a0f49c",
"flowId": "f57979d06f9842c3e94f1f197",
"creationDate": 1470744494732,
"path": "/luafanti/test",
"version": 0,
"elems": {
"xxx": {
"type": "integer",
"value": 200
}
}
}
} // hold this complete object in any string reference Variable.
Here I Suppose String jsonString holds the complete json object as above described.
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.JSONObject;
// implemetation logic for extracting jsonObject.
private JSONObject extractFirstBlock(String jsonString) throws ......{
JSONObject jsonObj ;
JSONParser parser=new JSONParser(); // parser to parse string to JSONObject
jsonObj = (JSONObject) parser.parse(jsonString); // parse the Object using parse Method.
String desiredObject = (String) jsonObj.get("body"); // this Object contains your desired output which you wish to receive.
jsonObj = parser.parse(desiredObject);
desiredObject = jsonObj.get("elems"); // you will get your desired object as you expected.
}
Here in desiredObject you will get your expected Values as JSONObject.
"xxx": {
"type": "integer",
"value": 200
}

android JSON parsing How to make two http Connection one after the other

Im using Rotten Tomatoes api and for some reason if i want to get more information about a movie like the movie director or the studio I need another http page. this is the url that im getting movie-info. from this JSON i get the id ( "self": "http://api.rottentomatoes.com/api/public/v1.0/movies/771205893.json").
and i want to use that id- i want to make another http connection to that specific url and get more information about each movie.
this is the movie-info JSON:
{
"total": 591,
"movies": [{
"title": "Jack and Jill",
"year": 2011,
"runtime": "",
"release_dates": {"theater": "2011-11-11"},
"ratings": {
"critics_score": -1,
"audience_score": 90
},
"synopsis": "",
"posters": {
},
"abridged_cast": [
{
"name": "Al Pacino",
"characters": []
},
{
"name": "Adam Sandler",
"characters": []
},
{
"name": "Katie Holmes",
"characters": []
}
],
"links": {
"self": "http://api.rottentomatoes.com/api/public/v1.0/movies/771205893.json",
}
}],
This is my code:
if (response != null) {
try {
// convert the String response to a JSON object,
// because JSON is the response format Rotten Tomatoes uses
JSONObject jsonResponse = new JSONObject(response);
// fetch the array of movies in the response
JSONArray movies = jsonResponse.getJSONArray("movies");
// add each movie's title to an array
movieTitles = new String[movies.length()];
for (int i = 0; i < movies.length(); i++) {
JSONObject movie = movies.getJSONObject(i);
movieTitles[i] = movie.getString("title");
}
try {
movieId = new String[movies.length()];
for (int i = 0; i < movies.length(); i++) {
JSONObject movie = movies.getJSONObject(i);
movieId[i] = movie.getString("id");
}
After i get the movieId i want to get only the movie id that the user has pressed
so i use :
public void onItemClick(AdapterView<?> arg0, View arg1,
int position, long id) {
Intent i = new Intent(getApplicationContext(),
AfterClickingMovieFromInternet.class);
i.putExtra("movieName", movieTitles[position]);
getMovieInfo( movieId[position]);
and in the getMovieInfo( movieId[position]) im trying to make a second connection to the api but for some reason it wont execute.....
if anyone got any idea ill be happy to know!

What is the problem with 2 consecutive connections?
Did you check libraries?
I would recommend this one - https://github.com/kodart/Httpzoid
Is uses GSON internally and provides API that works with objects. All JSON details are hidden.
Http http = HttpFactory.create(context);
http.get("http://example.com/users")
.handler(new ResponseHandler<User[]>() {
#Override
public void success(User[] users, HttpResponse response) {
}
}).execute();

How to set http header in Json response

I've a CXF RESTful service which returns both XML and Json format. I need to add a custom http header in the RESTful service. Here's a sample code snippet.
#GET
#Path("/test")
#Produces("application/xml")
public Response test(
#QueryParam("p") String var
{
TestRequest req = new TestRequest();
req.setVar(var);
TestResponse res = p.getData(req);
return Response.ok(res).header("Result", res.getResult()).build();
}
The above code shows the XML response which sets the custom http header "Result". I'm able to see the new http header in the response header. So far so good.
Now, here's the Json version which internally calls the testService() method to get the result, then use google Gson API to send the result back. This has been working well, till I decided to return the new header. Here's the code snippet.
#GET
#Path("/test/jsonp")
public String testJSONP(
#QueryParam("p") String var,
#QueryParam("cb") String callBack
{
Response resp = test(var);
XStream xs = new XStream(new JsonHierarchicalStreamDriver());
xs.setMode(XStream.NO_REFERENCES);
xs.alias("TestResponse", TestResponse.class);
StringBuilder sb = new StringBuilder();
sb.append(callBack);
sb.append("(");
GsonBuilder gb = new GsonBuilder();
gb.registerTypeAdapter(XMLGregorianCalendar.class, new XMLGregorianCalenderSerializer());
gb.setPrettyPrinting();
Gson gson = gb.create();
sb.append(gson.toJson(resp));
sb.append(")");
return sb.toString();
}
I'm not able to see the http header in Json response.
Any feedback will be highly appreciated.
-Thanks
UPDATE
I added the following code in Json method for my testing.
#GET
#Path("/test/jsonp")
public String testJSONP(
#QueryParam("p") String var,
#QueryParam("cb") String callBack
{
Response resp = test(var);
XStream xs = new XStream(new JsonHierarchicalStreamDriver());
xs.setMode(XStream.NO_REFERENCES);
xs.alias("TestResponse", TestResponse.class);
StringBuilder sb = new StringBuilder();
sb.append(callBack);
sb.append("(");
GsonBuilder gb = new GsonBuilder();
gb.registerTypeAdapter(XMLGregorianCalendar.class, new XMLGregorianCalenderSerializer());
gb.setPrettyPrinting();
Gson gson = gb.create();
sb.append(gson.toJson(resp));
sb.append(")");
return Response.ok(sb.toString(), MediaType.APPLICATION_JSON).header("Result", "50").build();
}
This sets the header value correctly,but the issue is the Json response format seems to have changed. Since this is an existing service, I'm not allowed to do that.
Here's the existing response format
null({
"status": "Completed",
"totalResult": "252",
"bin": [
{
"type": "source",
"value": "documentation",
"ndocs": "243"
},
{
"type": "source",
"value": "wikihelp",
"ndocs": "6"
},
"entries": {
"item": [
{
"url": "http://test.com/test.htm",
"title": "\u003cspan class\u003d\"vivbold qt0\"\u003eXREF\u003c/span\u003e",
"snippet": " Test data.",
"source": "documentation",
"type": "html",
"shortDescription": "Starts the TEST command.",
"category": [
"User"
],
"publishDate": "2012-02-05T12:00:00-0500",
"lastUpdateDate": "2012-03-14T12:00:00-0400",
"topicId": "GUID-7DD70C3C-B8AD-40F1-8A69-5D1EECEAB013"
}
]
}
})
Here's the response after adding this change
null({
"status": 200,
"entity": {
"status": "Completed",
"totalResult": "252",
"bin": [
{
"type": "source",
"value": "documentation",
"ndocs": "243"
},
{
"type": "source",
"value": "wikihelp",
"ndocs": "6"
}
],
"entries": {
"item": [
{
"url": "http://test.com/test.htm",
"title": "\u003cspan class\u003d\"vivbold qt0\"\u003eXREF\u003c/span\u003e",
"snippet": " Test data.",
"source": "documentation",
"type": "html",
"shortDescription": "Starts the TEST command.",
"category": [
"User"
],
"publishDate": "2012-02-05T12:00:00-0800",
"lastUpdateDate": "2012-03-14T12:00:00-0700",
"topicId": "GUID-7DD70C3C-B8AD-40F1-8A69-5D1EECEAB013"
}
]
}
},
"metadata": {
"Result": {
}
}
})

You need to change signature of your method, to return an instance of Response class, instead of a String, and then built the response manually.
From the CXF wiki page:
#Path("/example")
public ExampleResource {
#GET
public Response getSomething() {
return Response.ok(/* some entity */).header("CustomHeader", "CustomValue").build();
}
}
Update
You can also inject HttpServletResponse into your handler using #Context annotation like this:
#Path("/example")
public class Welcome {
#GET
public String getSomething(
#QueryParam("p1") String param1,
#QueryParam("p2") String param2,
#Context HttpServletResponse response) {
response.addHeader("CustomHeader", "CustomValue");
return "my awesome response";
}
}
Note, that there is a CXF-1498 bug in versions prior to 2.1 that causes HttpServletResponse not being injected, so you need a newer CXF version.