I am totally new to Apache Beam and Java.
Been working on PHP for around 5 years but i haven't worked in Java for the last 5 years :), plus Apache Beam SDK in java is something that is also new so bear with me.
I would like to implement pipeline where i will get data from Google PubSub, map the relevant fields into array and then check it to MySql Db to see if the message belong to one table, after that i will need to send api call to our API that will update some data in our app db. Another pipeline will enrich the data from elasticsearch and insert it into BigQuery.
But as of this moment i am stuck with reading data from MySql, i simply cannot adopt the argument in PCollection using JdbcIO.
My plan is to check if in Mysql table is present value that i get from pubsub ( value listid ).
Here is my code so far, any help will be appreciated.
Pipeline p = Pipeline.create(options);
org.apache.beam.sdk.values.PCollection<PubsubMessage> messages = p.apply(PubsubIO.readMessagesWithAttributes()
.fromSubscription("*******"));
org.apache.beam.sdk.values.PCollection<String> messages2 = messages.apply("GetPubSubEvent",
ParDo.of(new DoFn<PubsubMessage, String>() {
#ProcessElement
public void processElement(ProcessContext c) {
Map<String, String> Map = new HashMap<String, String>();
PubsubMessage message = c.element();
String messageText = new String(message.getPayload(), StandardCharsets.UTF_8);
JSONObject jsonObj = new JSONObject(messageText);
String requestURL = jsonObj.getJSONObject("httpRequest").getString("requestUrl");
String query = requestURL.split("\\?")[1];
final Map<String, String> querymap = Splitter.on('&').trimResults().withKeyValueSeparator("=")
.split(query);
JSONObject querymapJson = new JSONObject(querymap);
int subscriberid = 0;
int listid = 0;
int statid = 0;
int points = 0;
String stattype = "";
String requesttype = "";
try {
subscriberid = querymapJson.getInt("emp_uid");
} catch (Exception e) {
}
try {
listid = querymapJson.getInt("emp_lid");
} catch (Exception e) {
}
try {
statid = querymapJson.getInt("emp_statid");
} catch (Exception e) {
}
try {
stattype = querymapJson.getString("emp_stattype");
Map.put("stattype", stattype);
} catch (Exception e) {
}
try {
requesttype = querymapJson.getString("type");
} catch (Exception e) {
}
try {
statid = querymapJson.getInt("leadscore");
} catch (Exception e) {
}
Map.put("subscriberid", String.valueOf(subscriberid));
Map.put("listid", String.valueOf(listid));
Map.put("statid", String.valueOf(statid));
Map.put("requesttype", requesttype);
Map.put("leadscore", String.valueOf(points));
Map.put("requestip", jsonObj.getJSONObject("httpRequest").getString("remoteIp"));
System.out.print("Hello from message 1");
c.output(Map.toString());
}
}));
org.apache.beam.sdk.values.PCollection<String> messages3 = messages2.apply("Test",
ParDo.of(new DoFn<String, String>() {
#ProcessElement
public void processElement(ProcessContext c) {
System.out.println(c.element());
System.out.print("Hello from message 2");
}
}));
org.apache.beam.sdk.values.PCollection<KV<String, String>> messages23 = messages2.apply(JdbcIO.<KV<String, String>>read()
.withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create("org.apache.derby.jdbc.ClientDriver",
"jdbc:derby://localhost:1527/beam"))
.withQuery("select * from artist").withRowMapper(new JdbcIO.RowMapper<KV<String, String>>() {
public KV<String, String> mapRow(ResultSet resultSet) throws Exception {
KV<String, String> kv = KV.of(resultSet.getString("label"), resultSet.getString("name"));
return kv;
}
#Override
public KV<String, String> mapRow(java.sql.ResultSet resultSet) throws Exception {
KV<String, String> kv = KV.of(resultSet.getString("label"), resultSet.getString("name"));
return kv;
}
}).withCoder(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of())));
p.run().waitUntilFinish();
I'm using Retrofit 2.0.0-beta1.
In tests i have an alternate scenario and expect error HTTP 400
I would like to have retrofit.Response<MyError> response
but response.body() == null
MyError is not deserialised - i see it only here
response.errorBody().string()
but it doesn't give me MyError as object
I currently use a very easy implementation, which does not require to use converters or special classes. The code I use is the following:
public void onResponse(Call<ResponseBody> call, Response<ResponseBody> response) {
DialogHelper.dismiss();
if (response.isSuccessful()) {
// Do your success stuff...
} else {
try {
JSONObject jObjError = new JSONObject(response.errorBody().string());
Toast.makeText(getContext(), jObjError.getJSONObject("error").getString("message"), Toast.LENGTH_LONG).show();
} catch (Exception e) {
Toast.makeText(getContext(), e.getMessage(), Toast.LENGTH_LONG).show();
}
}
}
A point to note here is that response.errorBody().string() will return the correct value only once. If you call it again, it will return an empty string. So in case you want to reuse it, store the value in a variable with the first call.
There is a way to get the error body string from the response without making it empty on the next call, by rolling your own implementation of toString() that does not update the errorBody Buffer's read-pointer.
See this answer for more info.
ErrorResponse is your custom response object
Kotlin
val gson = Gson()
val type = object : TypeToken<ErrorResponse>() {}.type
var errorResponse: ErrorResponse? = gson.fromJson(response.errorBody()!!.charStream(), type)
Java
Gson gson = new Gson();
Type type = new TypeToken<ErrorResponse>() {}.getType();
ErrorResponse errorResponse = gson.fromJson(response.errorBody.charStream(),type);
I solved it by:
if(!response.isSuccessful()){
Gson gson = new Gson();
MyErrorMessage message=gson.fromJson(response.errorBody().charStream(),MyErrorMessage.class);
if(message.getCode()==ErrorCode.DUPLICATE_EMAIL_ID_CODE){
//DO Error Code specific handling
}else{
//DO GENERAL Error Code Specific handling
}
}
MyErrorMessage Class:
public class MyErrorMessage {
private int code;
private String message;
public int getCode() {
return code;
}
public void setCode(int code) {
this.code = code;
}
public String getMessage() {
return message;
}
public void setMessage(String message) {
this.message = message;
}
}
It's actually very straight forward.
Kotlin:
val jsonObj = JSONObject(response.errorBody()!!.charStream().readText())
responseInterface.onFailure(jsonObj.getString("msg"))
Java:
if(response.errorBody()!=null){
JSONObject jsonObj = new JSONObject(TextStreamsKt.readText(response.errorBody().charStream()));
responseInterface.onFailure(jsonObj.getString("msg"));
}else{
responseInterface.onFailure("you might want to return a generic error message.");
}
Tested on retrofit:2.5.0.
Read the text from the charStream which will give you a String, then parse to JSONObject.
Adios.
In Retrofit 2.0 beta2 this is the way that I'm getting error responses:
Synchronous
try {
Call<RegistrationResponse> call = backendServiceApi.register(data.in.account, data.in.password,
data.in.email);
Response<RegistrationResponse> response = call.execute();
if (response != null && !response.isSuccess() && response.errorBody() != null) {
Converter<ResponseBody, BasicResponse> errorConverter =
MyApplication.getRestClient().getRetrofitInstance().responseConverter(BasicResponse.class, new Annotation[0]);
BasicResponse error = errorConverter.convert(response.errorBody());
//DO ERROR HANDLING HERE
return;
}
RegistrationResponse registrationResponse = response.body();
//DO SUCCESS HANDLING HERE
} catch (IOException e) {
//DO NETWORK ERROR HANDLING HERE
}
Asynchronous
Call<BasicResponse> call = service.loadRepo();
call.enqueue(new Callback<BasicResponse>() {
#Override
public void onResponse(Response<BasicResponse> response, Retrofit retrofit) {
if (response != null && !response.isSuccess() && response.errorBody() != null) {
Converter<ResponseBody, BasicResponse> errorConverter =
retrofit.responseConverter(BasicResponse.class, new Annotation[0]);
BasicResponse error = errorConverter.convert(response.errorBody());
//DO ERROR HANDLING HERE
return;
}
RegistrationResponse registrationResponse = response.body();
//DO SUCCESS HANDLING HERE
}
#Override
public void onFailure(Throwable t) {
//DO NETWORK ERROR HANDLING HERE
}
});
Update for Retrofit 2 beta3:
Synchronous - not changed
Asynchronous - Retrofit parameter was removed from onResponse
Call<BasicResponse> call = service.loadRepo();
call.enqueue(new Callback<BasicResponse>() {
#Override
public void onResponse(Response<BasicResponse> response) {
if (response != null && !response.isSuccess() && response.errorBody() != null) {
Converter<ResponseBody, BasicResponse> errorConverter =
MyApplication.getRestClient().getRetrofitInstance().responseConverter(BasicResponse.class, new Annotation[0]);
BasicResponse error = errorConverter.convert(response.errorBody());
//DO ERROR HANDLING HERE
return;
}
RegistrationResponse registrationResponse = response.body();
//DO SUCCESS HANDLING HERE
}
#Override
public void onFailure(Throwable t) {
//DO NETWORK ERROR HANDLING HERE
}
});
Create a model of the Error response & user Gson to convert the response to it. This will just work fine.
APIError.java
public class APIError {
private String message;
public String getMessage() {
return message;
}
}
MainActivity.java (inside request onResponse)
if (response.isSuccessful()) {
// Do your success stuff...
} else {
APIError message = new Gson().fromJson(response.errorBody().charStream(), APIError.class);
Toast.makeText(MainActivity.this, "" + message.getMessage(), Toast.LENGTH_SHORT).show();
}
If you use Kotlin another solution could be just create extension function for Response class:
inline fun <reified T>Response<*>.parseErrJsonResponse(): T?
{
val moshi = MyCustomMoshiBuilder().build()
val parser = moshi.adapter(T::class.java)
val response = errorBody()?.string()
if(response != null)
try {
return parser.fromJson(response)
} catch(e: JsonDataException) {
e.printStackTrace()
}
return null
}
Usage
val myError = response.parseErrJsonResponse<MyErrorResponse>()
if(myError != null) {
// handle your error logic here
// ...
}
#Override
public void onResponse(Call<Void> call, retrofit2.Response<Void> response) {
if (response.isSuccessful()) {
//Do something if response is ok
} else {
JsonParser parser = new JsonParser();
JsonElement mJson = null;
try {
mJson = parser.parse(response.errorBody().string());
Gson gson = new Gson();
MyError errorResponse = gson.fromJson(mJson, MyError.class);
} catch (IOException ex) {
ex.printStackTrace();
}
}
In https://stackoverflow.com/a/21103420/2914140 and https://futurestud.io/tutorials/retrofit-2-simple-error-handling this variant is shown for Retrofit 2.1.0.
call.enqueue(new Callback<MyResponse>() {
#Override
public void onResponse(Call<MyResponse> call, Response<MyResponse> response) {
if (response.isSuccessful()) {
...
} else {
Converter<ResponseBody, MyError> converter
= MyApplication.getRetrofit().responseBodyConverter(
MyError.class, new Annotation[0]);
MyError errorResponse = null;
try {
errorResponse = converter.convert(response.errorBody());
} catch (IOException e) {
e.printStackTrace();
}
}
}
if(!response.isSuccessful()) {
StringBuilder error = new StringBuilder();
try {
BufferedReader bufferedReader = null;
if (response.errorBody() != null) {
bufferedReader = new BufferedReader(new InputStreamReader(
response.errorBody().byteStream()));
String eLine = null;
while ((eLine = bufferedReader.readLine()) != null) {
error.append(eLine);
}
bufferedReader.close();
}
} catch (Exception e) {
error.append(e.getMessage());
}
Log.e("Error", error.toString());
}
I did it this way for asynchronous calls using Retrofit 2.0-beta2:
#Override
public void onResponse(Response<RegistrationResponse> response,
Retrofit retrofit) {
if (response.isSuccess()) {
// Do success handling here
} else {
try {
MyError myError = (MyError)retrofit.responseConverter(
MyError.class, MyError.class.getAnnotations())
.convert(response.errorBody());
// Do error handling here
} catch (IOException e) {
e.printStackTrace();
}
}
}
I was facing same issue. I solved it with retrofit. Let me show this...
If your error JSON structure are like
{
"error": {
"status": "The email field is required."
}
}
My ErrorRespnce.java
public class ErrorResponse {
#SerializedName("error")
#Expose
private ErrorStatus error;
public ErrorStatus getError() {
return error;
}
public void setError(ErrorStatus error) {
this.error = error;
}
}
And this my Error status class
public class ErrorStatus {
#SerializedName("status")
#Expose
private String status;
public String getStatus() {
return status;
}
public void setStatus(String status) {
this.status = status;
}
}
Now we need a class which can handle our json.
public class ErrorUtils {
public static ErrorResponse parseError (Response<?> response){
Converter<ResponseBody , ErrorResponse> converter = ApiClient.getClient().responseBodyConverter(ErrorResponse.class , new Annotation[0]);
ErrorResponse errorResponse;
try{
errorResponse = converter.convert(response.errorBody());
}catch (IOException e){
return new ErrorResponse();
}
return errorResponse;
}
}
Now we can check our response in retrofit api call
private void registrationRequest(String name , String email , String password , String c_password){
final Call<RegistrationResponce> registrationResponceCall = apiInterface.getRegistration(name , email , password , c_password);
registrationResponceCall.enqueue(new Callback<RegistrationResponce>() {
#Override
public void onResponse(Call<RegistrationResponce> call, Response<RegistrationResponce> response) {
if (response.code() == 200){
}else if (response.code() == 401){
ErrorResponse errorResponse = ErrorUtils.parseError(response);
Toast.makeText(MainActivity.this, ""+errorResponse.getError().getStatus(), Toast.LENGTH_SHORT).show();
}
}
#Override
public void onFailure(Call<RegistrationResponce> call, Throwable t) {
}
});
}
That's it now you can show your Toast
Here is elegant solution using Kotlin extensions:
data class ApiError(val code: Int, val message: String?) {
companion object {
val EMPTY_API_ERROR = ApiError(-1, null)
}
}
fun Throwable.getApiError(): ApiError? {
if (this is HttpException) {
try {
val errorJsonString = this.response()?.errorBody()?.string()
return Gson().fromJson(errorJsonString, ApiError::class.java)
} catch (exception: Exception) {
// Ignore
}
}
return EMPTY_API_ERROR
}
and usage:
showError(retrofitThrowable.getApiError()?.message)
There are many valid answers already. This is just an addition for a use case, when you need to consume same Retrofit response more than once. Neither of below can be used, as you can read response body only once, as it will be closed afterwards and you will get null each next time, when you try to read from the same response object:
response()?.errorBody()?.charStream()?.readText()
response()?.errorBody()?.string()
Instead, you can get read-only copy of response string (while the response itself can be passed over and eventually consumed later):
response()?.errorBody()?.source()?.buffer?.snapshot()?.utf8()
This way you do not need a Retrofit instance if you only are injecting a service created from Retrofit.
public class ErrorUtils {
public static APIError parseError(Context context, Response<?> response) {
APIError error = new APIError();
try {
Gson gson = new Gson();
error = gson.fromJson(response.errorBody().charStream(), APIError.class);
} catch (Exception e) {
Toast.makeText(context, e.getMessage(), Toast.LENGTH_LONG).show();
}
if (TextUtils.isEmpty(error.getErrorMessage())) {
error.setError(response.raw().message());
}
return error;
}
}
Use it like this:
if (response.isSuccessful()) {
...
} else {
String msg = ErrorUtils.parseError(fragment.getActivity(), response).getError(); // would be from your error class
Snackbar.make(someview, msg, Snackbar.LENGTH_LONG).show();
}
}
if your error response is a string you can deserialize it by using the following kotlin code :
val errorString = response.errorBody()?.byteStream()?.bufferedReader().use { it?.readText() } // defaults to UTF-8
This seems to be the problem when you use OkHttp along with Retrofit, so either you can remove OkHttp or use code below to get error body:
if (!response.isSuccessful()) {
InputStream i = response.errorBody().byteStream();
BufferedReader r = new BufferedReader(new InputStreamReader(i));
StringBuilder errorResult = new StringBuilder();
String line;
try {
while ((line = r.readLine()) != null) {
errorResult.append(line).append('\n');
}
} catch (IOException e) {
e.printStackTrace();
}
}
Tested and works
public BaseModel parse(Response<BaseModel> response , Retrofit retrofit){
BaseModel error = null;
Converter<ResponseBody, BaseModel> errorConverter =
retrofit.responseBodyConverter(BaseModel.class, new Annotation[0]);
try {
if (response.errorBody() != null) {
error = errorConverter.convert(response.errorBody());
}
} catch (IOException e) {
e.printStackTrace();
}
return error;
}
For people using Kotlin with Moshi and coroutines, this is what I did:
Error data class
#JsonClass(generateAdapter = true)
data class ApiResponseNoData(
val exito: Int,
val error: String?
)
Extension
fun ResponseBody.getApiError(): String? {
return try {
Moshi
.Builder()
.build()
.adapter(ApiResponseNoData::class.java)
.fromJson(string())
?.error
}catch(e: Exception) { null }
}
ViewModel
fun test() {
viewModelScope.launch(Dispatchers.IO) {
val response = repository.test()
withContext(Dispatchers.Main) {
if(response.isSuccessful) {
...
}else{
val errorMsg = response.errorBody()?.getApiError() ?: "Unexpected error occurred"
...
]
}
}
}
json response
{
"success": false,
"status_code": 32,
"status_message": "Email not verified: Your email address has not been verified."
}
Error class
data class ResponseError(
#SerializedName("status_code")
val statusCode: Int,
#SerializedName("status_message")
val statusMessage: String,
#SerializedName("success")
val success: Boolean
)
get error message
fun <T : Any> getResultOrError(response: Response<T>): T? {
if (response.isSuccessful) {
return response.body()
} else {
try {
val responseError = Gson().fromJson(
response.errorBody()?.string(),
ResponseError::class.java
)
throw Throwable(responseError.statusMessage)
} catch (e: Exception) {
throw Throwable("Unknown error")
}
}
}
solved it by:
Converter<MyError> converter =
(Converter<MyError>)JacksonConverterFactory.create().get(MyError.class);
MyError myError = converter.fromBody(response.errorBody());
try{
ResponseBody response = ((HttpException) t).response().errorBody();
JSONObject json = new JSONObject( new String(response.bytes()) );
errMsg = json.getString("message");
}catch(JSONException e){
return t.getMessage();
}
catch(IOException e){
return t.getMessage();
}
In Kotlin:
val call = APIClient.getInstance().signIn(AuthRequestWrapper(AuthRequest("1234567890z", "12341234", "nonce")))
call.enqueue(object : Callback<AuthResponse> {
override fun onResponse(call: Call<AuthResponse>, response: Response<AuthResponse>) {
if (response.isSuccessful) {
} else {
val a = object : Annotation{}
val errorConverter = RentalGeekClient.getRetrofitInstance().responseBodyConverter<AuthFailureResponse>(AuthFailureResponse::class.java, arrayOf(a))
val authFailureResponse = errorConverter.convert(response.errorBody())
}
}
override fun onFailure(call: Call<AuthResponse>, t: Throwable) {
}
})
errorBody values should set APIError object in Retrofit. So that, you can use the below code structure.
public class APIErrorUtils {
public static APIError parseError(Response<?> response) {
Converter<ResponseBody, APIError> converter = API.getClient().responseBodyConverter(APIError.class, new Annotation[0]);
APIError error;
try {
error = converter.convert(response.errorBody());
Log.d("SERVICELOG", "****************************************************");
Log.d("SERVICELOG", "***** SERVICE LOG");
Log.d("SERVICELOG", "***** TIMESTAMP: " + String.valueOf(error.getTimestamp()));
Log.d("SERVICELOG", "***** STATUS: " + String.valueOf(error.getStatus()));
Log.d("SERVICELOG", "***** ERROR: " + error.getError());
Log.d("SERVICELOG", "***** MESSAGE: " + error.getMessage());
Log.d("SERVICELOG", "***** PATH: " + error.getPath());
Log.d("SERVICELOG", "****************************************************");
} catch (IOException e) {
return new APIError();
}
return error;
}
}
APIError error = APIErrorUtils.parseError(response);
if (error.getStatus() == 400) {
....
}
val error = JSONObject(callApi.errorBody()?.string() as String)
CustomResult.OnError(CustomNotFoundError(userMessage = error["userMessage"] as String))
open class CustomError (
val traceId: String? = null,
val errorCode: String? = null,
val systemMessage: String? = null,
val userMessage: String? = null,
val cause: Throwable? = null
)
open class ErrorThrowable(
private val traceId: String? = null,
private val errorCode: String? = null,
private val systemMessage: String? = null,
private val userMessage: String? = null,
override val cause: Throwable? = null
) : Throwable(userMessage, cause) {
fun toError(): CustomError = CustomError(traceId, errorCode, systemMessage, userMessage, cause)
}
class NetworkError(traceId: String? = null, errorCode: String? = null, systemMessage: String? = null, userMessage: String? = null, cause: Throwable? = null):
CustomError(traceId, errorCode, systemMessage, userMessage?: "Usted no tiene conexión a internet, active los datos", cause)
class HttpError(traceId: String? = null, errorCode: String? = null, systemMessage: String? = null, userMessage: String? = null, cause: Throwable? = null):
CustomError(traceId, errorCode, systemMessage, userMessage, cause)
class UnknownError(traceId: String? = null, errorCode: String? = null, systemMessage: String? = null, userMessage: String? = null, cause: Throwable? = null):
CustomError(traceId, errorCode, systemMessage, userMessage?: "Unknown error", cause)
class CustomNotFoundError(traceId: String? = null, errorCode: String? = null, systemMessage: String? = null, userMessage: String? = null, cause: Throwable? = null):
CustomError(traceId, errorCode, systemMessage, userMessage?: "Data not found", cause)`
Error body handling in kotlin Android
catch (cause: Throwable) {
when (cause) {
is HttpException -> {
try {
val YourErrorResponseClassObj = Gson().fromJson(cause.response()?.errorBody()?.charStream(), YourErrorResponseClass::class.java)
} catch (e: Exception) {
}
}
else -> {
//Other errors like Network ...
}
}
}
very simple. and this save my life ever
public static void displayApiResponseErrorBody(Response<?> response)
{
InputStream i = response.errorBody().byteStream();
BufferedReader r = new BufferedReader(new InputStreamReader(i));
StringBuilder errorResult = new StringBuilder();
String line;
try {
while ((line = r.readLine()) != null)
{
errorResult.append(line).append('\n');
}
Log.d("API_RESPONSE_ERROR_BODY",String.valueOf(errorResult));
System.out.println(errorResult);
} catch (IOException e) {
e.printStackTrace();
}
}
In case of retrofit error Response, You can get body using error.getResponse(), Here is the example.
#Override
public void failure(RetrofitError error){
if(error.getResponse().getStatus()==201){
LogUtil.INSTANCE.debug("Success : " + error.toString());
callback.success(error.getResponse().getBody);
}else{
LogUtil.INSTANCE.debug("failure: " + error.toString());
callback.failure(error);
}
}
In Kotlin I solved it creating a custom ResponseBody generic extension function function that converts the response body to a JSONObject. then you can use gson to customize the error response body with your custom Error Data Class.
inline fun <reified T> ResponseBody.getErrorObject(): T {
val gson = Gson()
val jsonObject = JSONObject(charStream().readText())
return gson.fromJson(jsonObject.toString(), T::class.java)
}
You can then customer the error response to your custom class. For this I'm using an example
data class LoginError(
val error: Error,
val message: String,
val success: Boolean
)
data class Error(
val error: String,
val status: Int
)
then use the extension function this way
val error = state.errorBody.getErrorObject<LoginError>()
the state.errorBody is my error response from retrofit of type ResponseBody
val reader = BufferedReader(response.errorBody()?.source().inputStream().reader())
val content = StringBuilder()
reader.use { readerBuffer ->
var line = readerBuffer.readLine()
while (line != null) {
content.append(line)
line = readerBuffer.readLine()
}
}
Gson().fromJson(content.toString(), ResponseData::class.java)
Does anyone know where to find a little how to on using dbpedia spotlight in java or scala? Or could anyone explain how it's done? I can't find any information on this...
The DBpedia Spotlight wiki pages would be a good place to start.
And I believe the installation page has listed the most popular ways (using a jar, or set up a web service) to use the application.
It includes instructions on using the Java/Scala API with your own installation, or calling the Web Service.
There are some additional data needed to be downloaded to run your own server for full service, good time to make a coffee for yourself.
you need download dbpedia spotlight (jar file) after that u can use next two classes ( author pablomendes ) i only make some change .
public class db extends AnnotationClient {
//private final static String API_URL = "http://jodaiber.dyndns.org:2222/";
private static String API_URL = "http://spotlight.dbpedia.org:80/";
private static double CONFIDENCE = 0.0;
private static int SUPPORT = 0;
private static String powered_by ="non";
private static String spotter ="CoOccurrenceBasedSelector";//"LingPipeSpotter"=Annotate all spots
//AtLeastOneNounSelector"=No verbs and adjs.
//"CoOccurrenceBasedSelector" =No 'common words'
//"NESpotter"=Only Per.,Org.,Loc.
private static String disambiguator ="Default";//Default ;Occurrences=Occurrence-centric;Document=Document-centric
private static String showScores ="yes";
#SuppressWarnings("static-access")
public void configiration(double CONFIDENCE,int SUPPORT,
String powered_by,String spotter,String disambiguator,String showScores){
this.CONFIDENCE=CONFIDENCE;
this.SUPPORT=SUPPORT;
this.powered_by=powered_by;
this.spotter=spotter;
this.disambiguator=disambiguator;
this.showScores=showScores;
}
public List<DBpediaResource> extract(Text text) throws AnnotationException {
LOG.info("Querying API.");
String spotlightResponse;
try {
String Query=API_URL + "rest/annotate/?" +
"confidence=" + CONFIDENCE
+ "&support=" + SUPPORT
+ "&spotter=" + spotter
+ "&disambiguator=" + disambiguator
+ "&showScores=" + showScores
+ "&powered_by=" + powered_by
+ "&text=" + URLEncoder.encode(text.text(), "utf-8");
LOG.info(Query);
GetMethod getMethod = new GetMethod(Query);
getMethod.addRequestHeader(new Header("Accept", "application/json"));
spotlightResponse = request(getMethod);
} catch (UnsupportedEncodingException e) {
throw new AnnotationException("Could not encode text.", e);
}
assert spotlightResponse != null;
JSONObject resultJSON = null;
JSONArray entities = null;
try {
resultJSON = new JSONObject(spotlightResponse);
entities = resultJSON.getJSONArray("Resources");
} catch (JSONException e) {
//throw new AnnotationException("Received invalid response from DBpedia Spotlight API.");
}
LinkedList<DBpediaResource> resources = new LinkedList<DBpediaResource>();
if(entities!=null)
for(int i = 0; i < entities.length(); i++) {
try {
JSONObject entity = entities.getJSONObject(i);
resources.add(
new DBpediaResource(entity.getString("#URI"),
Integer.parseInt(entity.getString("#support"))));
} catch (JSONException e) {
LOG.error("JSON exception "+e);
}
}
return resources;
}
}
second class
/**
* #author pablomendes
*/
public abstract class AnnotationClient {
public Logger LOG = Logger.getLogger(this.getClass());
private List<String> RES = new ArrayList<String>();
// Create an instance of HttpClient.
private static HttpClient client = new HttpClient();
public List<String> getResu(){
return RES;
}
public String request(HttpMethod method) throws AnnotationException {
String response = null;
// Provide custom retry handler is necessary
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
try {
// Execute the method.
int statusCode = client.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
LOG.error("Method failed: " + method.getStatusLine());
}
// Read the response body.
byte[] responseBody = method.getResponseBody(); //TODO Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
// Deal with the response.
// Use caution: ensure correct character encoding and is not binary data
response = new String(responseBody);
} catch (HttpException e) {
LOG.error("Fatal protocol violation: " + e.getMessage());
throw new AnnotationException("Protocol error executing HTTP request.",e);
} catch (IOException e) {
LOG.error("Fatal transport error: " + e.getMessage());
LOG.error(method.getQueryString());
throw new AnnotationException("Transport error executing HTTP request.",e);
} finally {
// Release the connection.
method.releaseConnection();
}
return response;
}
protected static String readFileAsString(String filePath) throws java.io.IOException{
return readFileAsString(new File(filePath));
}
protected static String readFileAsString(File file) throws IOException {
byte[] buffer = new byte[(int) file.length()];
#SuppressWarnings("resource")
BufferedInputStream f = new BufferedInputStream(new FileInputStream(file));
f.read(buffer);
return new String(buffer);
}
static abstract class LineParser {
public abstract String parse(String s) throws ParseException;
static class ManualDatasetLineParser extends LineParser {
public String parse(String s) throws ParseException {
return s.trim();
}
}
static class OccTSVLineParser extends LineParser {
public String parse(String s) throws ParseException {
String result = s;
try {
result = s.trim().split("\t")[3];
} catch (ArrayIndexOutOfBoundsException e) {
throw new ParseException(e.getMessage(), 3);
}
return result;
}
}
}
public void saveExtractedEntitiesSet(String Question, LineParser parser, int restartFrom) throws Exception {
String text = Question;
int i=0;
//int correct =0 ; int error = 0;int sum = 0;
for (String snippet: text.split("\n")) {
String s = parser.parse(snippet);
if (s!= null && !s.equals("")) {
i++;
if (i<restartFrom) continue;
List<DBpediaResource> entities = new ArrayList<DBpediaResource>();
try {
entities = extract(new Text(snippet.replaceAll("\\s+"," ")));
System.out.println(entities.get(0).getFullUri());
} catch (AnnotationException e) {
// error++;
LOG.error(e);
e.printStackTrace();
}
for (DBpediaResource e: entities) {
RES.add(e.uri());
}
}
}
}
public abstract List<DBpediaResource> extract(Text text) throws AnnotationException;
public void evaluate(String Question) throws Exception {
evaluateManual(Question,0);
}
public void evaluateManual(String Question, int restartFrom) throws Exception {
saveExtractedEntitiesSet(Question,new LineParser.ManualDatasetLineParser(), restartFrom);
}
}
main()
public static void main(String[] args) throws Exception {
String Question ="Is the Amazon river longer than the Nile River?";
db c = new db ();
c.configiration(0.0, 0, "non", "CoOccurrenceBasedSelector", "Default", "yes");
System.out.println("resource : "+c.getResu());
}
I just add one little fix for your answer.
Your code is running, if you add the evaluate method call:
public static void main(String[] args) throws Exception {
String question = "Is the Amazon river longer than the Nile River?";
db c = new db ();
c.configiration(0.0, 0, "non", "CoOccurrenceBasedSelector", "Default", "yes");
c.evaluate(question);
System.out.println("resource : "+c.getResu());
}
Lamine
In the request method of the second class (AnnotationClient) in Adel's answer, the author Pablo Mendes hasn't finished
TODO Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
which is an annoying warning that needs to be removed by replacing
byte[] responseBody = method.getResponseBody(); //TODO Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
// Deal with the response.
// Use caution: ensure correct character encoding and is not binary data
response = new String(responseBody);
with
Reader in = new InputStreamReader(method.getResponseBodyAsStream(), "UTF-8");
StringWriter writer = new StringWriter();
org.apache.commons.io.IOUtils.copy(in, writer);
response = writer.toString();