So I am just trying to add data in my indice via java transport client api . My code runs without any exceptions.
It successfully connects to my server, I can see the connection details on my console.
I listed all indices available on the server and it did show that too.
Then I tried to insert docs into my indice called logs
code runs without any exceptions.
I am lost here and have no idea what I am missing. The data is not getting inserted in the indice. Any help is appreciated.
Below is my code :
public static void main(String[] args) {
try {
ElasticSearchMain es = new ElasticSearchMain();
ElasticSearchMain.configureClient();
JSONObject data = new JSONObject();
data.put("serverip", "bhavik");
data.put("classname", "bhavik");
data.put("methodname", "bhavik");
data.put("exception", "bhavik");
data.put("logexception", "4896681231231232");
data.put("timestamp", "1900-00-00");
es.setIndex(data.toString(), "logs");
} catch (Exception e) {
e.printStackTrace();
}
}
public IndexRequestBuilder setIndex(String data,String IndexName){
try{
return client.prepareIndex(IndexName, "event").setSource(data);
}catch(Exception e){
e.printStackTrace();
}
return null;
}
public static void configureClient(){
try{
if(client==null){
String address = "localhost";
int port = 9300;
BasicConfigurator.configure();
client = TransportClient.builder().build()
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(address), port));
String[] indices = client.admin().indices().getIndex(new GetIndexRequest()).actionGet().getIndices();
for (String s : indices) {
System.out.println("indice ->> " + s);
}
}
}catch(Exception e ){
e.printStackTrace();
EmgrLog.AppendExceptionToLog(e);
}
}
Indice mapping :
PUT logs
{
"mappings": {
"event": {
"properties": {
"serverip": {
"type": "integer",
"index": "not_analyzed"
},
"classname": {
"type": "string",
"analyzer": "english"
},
"methodname": {
"type": "string",
"index": "not_analyzed"
},
"exception": {
"type": "string",
"index": "not_analyzed"
},
"logexception": {
"type": "string"
},
"timestamp":{
"type": "date"
}
}
}
}
}
This code :
public IndexRequestBuilder setIndex(String data,String IndexName){
try{
return client.prepareIndex(IndexName, "event").setSource(data);
}catch(Exception e){
e.printStackTrace();
}
return null;
}
Returns a "request builder", but this request is never built, nor executed, and logically, nothing happens as a consequence.
To submit the request, you have to send it to the elasticsearch instance.
Elasticsearch JavaAPI documentation proposes this code :
IndexResponse response = client.prepareIndex("twitter", "tweet", "1")
.setSource(jsonBuilder()
.startObject()
.field("user", "kimchy")
.field("postDate", new Date())
.field("message", "trying out Elasticsearch")
.endObject()
)
.get();
Note the get() at the end of the snippet is what triggers the call.
Related
I am new to GCP storage. I want to read a large JSON file (2GB array of Objects) from GCP storage in my project (Spring Boot, Java). If I download the whole JSON file, may be I will get out of memory.
So any JSON streaming parsing available to parse without download whole file, like below.
#Component
public class TestA {
#Value("classpath:/data-array-load.json")
Resource resourceFile;
#Autowired
ObjectMapper objectMapper;
public void processData() {
int count = 0;
try {
JsonFactory factory = new JsonFactory();
File fileData = resourceFile.getFile();
ResponseObject classObject = null;
List<ResponseObject> listData = new ArrayList<>();
JsonParser parser = objectMapper.getFactory().createParser(fileData);
if ( parser.nextToken() != JsonToken.START_ARRAY ) {
throw new Exception( "no array" );
}
while ( parser.nextToken() == JsonToken.START_OBJECT ) {
classObject = objectMapper.readValue( parser, ResponseObject.class );
listData.add(classObject);
if(count == 50){
log.info("Data size: {}", listData.size());
// once 50 , will call another service
}
count++;
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
Sample JSON file will be as below
[
{
"id": "345345",
"info": [
{
"data": "info"
},
{
"data": "info2"
}
]
},
{
"id": "456456",
"info": [
{
"data": "info"
},
{
"data": "info2"
}
]
},
{
"id": "67868678",
"info": [
{
"data": "info"
},
{
"data": "info2"
}
]
}
]
I am not getting any best approach to do this. Can any one please guide?
I'm trying to read a file which is fetched from Opentdb and access data from only specific keys.
I've tried all of the given methods which includes GSON and Json Simple but ended up having errors.
Here's my JSON:
{
"response_code": 0,
"results": [
{
"category": "Sports",
"type": "multiple",
"difficulty": "easy",
"question": "Which of the following sports is not part of the triathlon?",
"correct_answer": "Horse-Riding",
"incorrect_answers": [
"Cycling",
"Swimming",
"Running"
]
},
{
"category": "Sports",
"type": "multiple",
"difficulty": "easy",
"question": "Which English football club has the nickname \u0026#039;The Foxes\u0026#039;?",
"correct_answer": "Leicester City",
"incorrect_answers": [
"Northampton Town",
"Bradford City",
"West Bromwich Albion"
]
},
I want to access only category, difficulty, question , correct_answer and incorrect_answers .
excample for you
try {
URL resource = App.class.getClassLoader().getResource("info.json");
System.out.println(resource.toString());
FileReader reader = new FileReader(resource.getFile());
// Read JSON file
Object obj = jsonParser.parse(reader);
JSONArray infoList = (JSONArray) obj;
System.out.println(infoList);
Information i = new Information();
List<Information> info = i.parseInformationObject(infoList);
saveInformation(info);
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (ParseException e) {
e.printStackTrace();
}
and
List<Information> parseInformationObject(JSONArray infoList) {
List<Information> in = new ArrayList<>();
infoList.forEach(emp -> {
JSONObject info = (JSONObject) emp;
String id = info.get("id").toString();
String state = info.get("state").toString();
String type = null;
if (info.get("type") != null) {
type = info.get("type").toString();
}
String host = null;
if (info.get("host") != null) {
host = info.get("host").toString();
}
long timestamp = (long) info.get("timestamp");
in.add(new Information(id, state, type, host, timestamp));
});
return in;
}
I have a below response which I am unable to validate due to response is starting from JSON object 0. Thats means if more than one object, the response will start from 0 to the number of objects you have.
I have tried this but its not working and ends up with stack over flow error.
public static void Location_ContactValidation(Response response, String code, String message, ExtentTest log) {
try {
softAssert = new SoftAssert();
org.json.JSONObject jsonObject = new org.json.JSONObject(response);
org.json.JSONObject getSth = jsonObject.getJSONObject("status");
status_Message = getSth.get("message");
softAssert.assertEquals(code, code);
softAssert.assertEquals(message, status_Message);
log.log(LogStatus.INFO, "Validation: The status code is " + code);
log.log(LogStatus.INFO, "Validation: The status message is " + status_Message.toString());
} catch (Exception e) {
log.log(LogStatus.INFO, "Validation: The status code is " + code);
if (status_Message != null) {
log.log(LogStatus.INFO, "Validation: The status message is " + status_Message.toString());
}
System.out.println(e.getMessage());
if (softAssert != null) {
softAssert.assertAll();
}
}
}
Stack overflow error as flows-
java.lang.StackOverflowError
at org.json.JSONObject.wrap(JSONObject.java:1746)
at org.json.JSONArray.<init>(JSONArray.java:176)
at org.json.JSONObject.wrap(JSONObject.java:1747)
at org.json.JSONObject.populateMap(JSONObject.java:1167)
And here is the response I want to parse
{
"0": {
"status": "OK",
"data": {
"id": "*************",
"mobile": "*************"
},
"message": "Submitted Successfully"
},
"status": "OK"
}
I need to validate the mobile number, both the status and message.
But not able to do it.
If one more number is send with request then the response increases and to array gets created first as shown with 0 then with 1.
I appreciate your help.
You can list all keys for given JSONObject using keySet method. After that you can directly dig into required fields.
Below codes shows how to read all required fields:
import org.json.JSONObject;
import java.io.File;
import java.nio.file.Files;
public class OrgJsonApp {
public static void main(String[] args) throws Exception {
File jsonFile = new File("./resource/test.json").getAbsoluteFile();
String json = String.join("", Files.readAllLines(jsonFile.toPath()));
JSONObject response = new JSONObject(json);
response.keySet().forEach(key -> {
JSONObject object = response.optJSONObject(key);
// if given key represents object it must be data
if (object != null) {
final String dataKey = "data";
JSONObject data = object.optJSONObject(dataKey);
// extract mobile from data and remove data
// this way JSON node is much simpler
if (data != null) {
final String mobileKey = "mobile";
String mobile = data.optString(mobileKey);
System.out.println("Mobile => " + mobile);
}
System.out.println("status => " + object.optString("status"));
System.out.println("message => " + object.optString("message"));
}
});
}
}
For below JSON payload:
{
"0": {
"status": "OK",
"data": {
"id": "1",
"mobile": "44-32-12"
},
"message": "Submitted Successfully"
},
"1": {
"status": "OK",
"data": {
"id": "2",
"mobile": "9981234-543"
},
"message": "Submitted Successfully"
},
"status": "OK"
}
prints:
Mobile => 44-32-12
status => OK
message => Submitted Successfully
Mobile => 9981234-543
status => OK
message => Submitted Successfully
I am trying to connect to elasticsearch and do some basic query in 6.3.2 version.
The code I am trying is this:
RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(new HttpHost("localhost", 30100, "http")));
SearchRequest sr = new SearchRequest(INDEX);
sr.indicesOptions(IndicesOptions.lenientExpandOpen());
SearchSourceBuilder ssb = new SearchSourceBuilder();
sr.source(ssb);
MatchQueryBuilder builder = QueryBuilders.matchQuery("logLevel.keyword", "ERROR");
QueryBuilder qb = QueryBuilders.boolQuery().must(builder);
ssb.query(qb);
SearchResponse response = null;
try {
response = client.search(sr);
System.out.println("total hits ::: " + response.getHits().getTotalHits());
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
client.close();
} catch (IOException e) {
e.printStackTrace();
}
}
System.out.println(response);
UPDATE
As suggested i am using the query only now. I tried the build query from APIs and i see the result but for some reason response.getHits().getTotalHits() is returning zero. The generated query looks like below and is giving me the expected result in kibana i.e. a total count of 1 :
GET /_search
{
"from": 0,
"query": {
"bool": {
"must": [
{
"match": {
"logLevel.keyword": {
"query": "ERROR",
"operator": "OR",
"prefix_length": 0,
"max_expansions": 50,
"fuzzy_transpositions": true,
"lenient": false,
"zero_terms_query": "NONE",
"auto_generate_synonyms_phrase_query": true,
"boost": 1
}
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
}
}
Am missing some configuration for the restclient ?
I don't see why you need all the parts in the query that you posted (like max_expansions etc). A simple query like below, should do the job if you only need the count of documents that match the criteria.
curl -XGET "http://localhost:9200/index_name/_search" -d'
{
"size": 0,
"query": {
"bool": {
"must": [
{
"match": {
"logLevel": "ERROR"
}
},
{
"range": {
"date": {
"gte": "enter_date_here"
}
}
}
]
}
}
}'
reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html
I have the following json file from which I am trying to extract data and store it to a database.
*//FirstBlockStart*
{
"_id": {
"$oid": "572395d8b1bf87f39645b94e"
},
"ReferenceDate": {
"$date": 1461949267500
},
"Url": "http://goTechhy.com",
"PublicationDate": {
"$date": 1461553200000
},
"LastUpdateDate": {
"$date": 1461553200000
},
}
//FirstBlockEnd
//SecondBlockStart
{
"_id": {
"$oid": "572394d8b1bf87f39645b94e"
},
"ReferenceDate": {
"$date": 1461969267500
},
"Url": "http://goTechhy.com",
"PublicationDate": {
"$date": 1461553200000
},
"LastUpdateDate": {
"$date": 1461353200000
},
}
//SecondBlockEnd
I need to extract the Oid, ReferenceData, PublicationData, LastUpdateDate from the JSON
Here is what I have tried.
public static void main(String[] args) {
JSONParser parser = new JSONParser();
try {
JSONObject Obj = (JSONObject) parser.parse(new InputStreamReader(new FileInputStream("E:\\Json2Db\\PlayStore1.json")));
//JSONObject Obj = (JSONObject) parser.parse(new FileReader("E:\\Json2Db\\PlayStore1.json"));
getAppID(Obj);
System.out.println(getAppID(Obj));
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (ParseException e) {
e.printStackTrace();
}
}
private static String getAppID(JSONObject obj) {
JSONObject appIDObj = (JSONObject) obj.get("_id");
String appID = (String) appIDObj.get("$oid");
return appID;
}
When I try to execute this code, I get the following error.
Unexpected token LEFT BRACE({) at position 4644.
at org.json.simple.parser.JSONParser.parse(JSONParser.java:146)
at org.json.simple.parser.JSONParser.parse(JSONParser.java:92)
at com.freelancer.kaja.JsonParse.main(JsonParse.java:18)
But at the same time, When there is only One Block (say FirstBlock) in JSON, it works fine.. I am new to this JSON thing.. I dont know what I am missing here.. Please give your suggestions.
I would like the output to be something like this.
oid: 572395d8b1bf87f39645b94e
ReferenceDate:1461949267500
Url:http://goTechhy.com
PublicationDate:1461553200000
LastUpdateDate: 1461553200000
oid: 572395d8b1bf87f39645b94e
ReferenceDate:1461949267500
Url:http://goTechhy.com
PublicationDate:1461553200000
LastUpdateDate: 1461553200000