Develop Reference

Encountering this error while running test cases -ERROR N/A (Null Pointer Exception)

#Test
public void testBatchFailClientBatchSyncCallIllegalArgumentExceptions() throws Exception {
Map<String, String> singletonMap = Collections.singletonMap(ACCEPT_STRING_ID, defaultLocalizationMap.get(ACCEPT_STRING_ID));
StringRequest[] requests = stringRequestFactory.createRequests(singletonMap);
when(lmsClient.batchSyncCall(requests)).thenThrow(new IllegalArgumentException());
List<Backend.Response> responses = callLms(new StringRequest[] {requests[0]});
Assert.assertNotNull(responses);
assertEquals(EntityDescriptors.ERROR_V1, responses.get(0).entityDescriptor());
assertEquals(Http.Status.SERVICE_UNAVAILABLE, responses.get(0).status());
}
#Test
public void testBatchFailClientBatchSyncCallIOException() throws Exception {
Map<String, String> singletonMap = Collections.singletonMap(ACCEPT_STRING_ID, defaultLocalizationMap.get(ACCEPT_STRING_ID));
StringRequest[] requests = stringRequestFactory.createRequests(singletonMap);
when(lmsClient.batchSyncCall(requests)).thenThrow(new IOException());
List<Backend.Response> responses = callLms(new StringRequest[] {requests[0]});
Assert.assertNotNull(responses);
assertEquals(EntityDescriptors.ERROR_V1, responses.get(0).entityDescriptor());
assertEquals(Http.Status.SERVICE_UNAVAILABLE, responses.get(0).status());
}
Source Code -
#Override
public List<Backend.Response> handleRequests(BackendRequestContext context, List<Backend.Request> requests, Metrics metrics) {
StringRequest[] stringRequests = new StringRequest[requests.size()];
final String language = context.locale().toLanguageTag().replace("-", "_");
for (int i = 0; i < requests.size(); i++) {
final Backend.Request request = requests.get(i);
final String id = request.requiredPathParam(STRING_ID_PATH_PARAM);
final Optional<String> marketplaceDisplayName = request.queryParam(MARKETPLACE_NAME_QUERY_PARAM);
final Optional<String> stage = request.queryParam(STAGE_QUERY_PARAM);
final StringRequest stringRequest = new StringRequest(id);
stringRequest.setLanguage(language);
marketplaceDisplayName.ifPresent(stringRequest::setMarketplaceName);
stage.map(Stage::getStage).ifPresent(stringRequest::setStage);
stringRequests[i] = stringRequest;
}
StringResultBatch batchResult = invokeBatchSync(stringRequests);
return IntStream.of(requests.size()).mapToObj(i -> {
final Backend.Request request = requests.get(i);
try {
return transform(request, batchResult.get(i), language);
} catch (IOException e) {
LOGGER.error("", e);
return Backend.Response.builder()
.withRequest(request)
.withEntityDescriptor(EntityDescriptors.ERROR_V1)
.withStatus(Http.Status.SERVICE_UNAVAILABLE)
.withBody(ErrorResponses.ServerError.serviceUnavailable(ErrorResponse.InternalInfo.builder()
.withMessage("Error retrieving ["
+ request.requiredPathParam(STRING_ID_PATH_PARAM)
+ "]")
.build())
.tokens())
.build();
}
}
).collect(Collectors.toList());
}
private StringResultBatch invokeBatchSync(StringRequest[] stringRequests) {
try {
// LMS Client has an async batch call,
// but it returns a proprietary class (StringResultBatchFuture) which eventually wraps a BSFFutureReply.
// Neither of which provide access to anything like a Java-standard Future.
return client.batchSyncCall(stringRequests);
} catch (IllegalArgumentException | IOException e) {
//
return null;
}
}
I have two test cases here for the source file. I'm getting the Error N/A. It says null pointer exception. Can someone please review this and help me with this. It will be really appreciated. Thank you in advance
P.S - The source file takes input request as string and performs string translation and returns us that string.

TupleTag not found in DoFn

I have a DoFn that is supposed to split input into two separate PCollections. The pipeline builds and runs up until it is time to output in the DoFn, and then I get the following exception:
"java.lang.IllegalArgumentException: Unknown output tag Tag<edu.mayo.mcc.cdh.pipeline.PubsubToAvro$PubsubMessageToArchiveDoFn$2.<init>:219#2587af97b4865538>
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:216)...
If I declare the TupleTags I'm using in the ParDo, I get that error, but if I declare them outside of the ParDo I get a syntax error saying the OutputReceiver can't find the tags. Below is the apply and the ParDo/DoFn:
PCollectionTuple results = (messages.apply("Map to Archive", ParDo.of(new PubsubMessageToArchiveDoFn()).withOutputTags(noTag, TupleTagList.of(medaPcollection))));
PCollection<AvroPubsubMessageRecord> medaPcollectionTransformed = results.get(medaPcollection);
PCollection<AvroPubsubMessageRecord> noTagPcollectionTransformed = results.get(noTag);
static class PubsubMessageToArchiveDoFn extends DoFn<PubsubMessage, AvroPubsubMessageRecord> {
final TupleTag<AvroPubsubMessageRecord> medaPcollection = new TupleTag<AvroPubsubMessageRecord>(){};
final TupleTag<AvroPubsubMessageRecord> noTag = new TupleTag<AvroPubsubMessageRecord>(){};
#ProcessElement
public void processElement(ProcessContext context, MultiOutputReceiver out) {
String appCode;
PubsubMessage message = context.element();
String msgStr = new String(message.getPayload(), StandardCharsets.UTF_8);
try {
JSONObject jsonObject = new JSONObject(msgStr);
LOGGER.info("json: {}", jsonObject);
appCode = jsonObject.getString("app_code");
LOGGER.info(appCode);
if(appCode == "MEDA"){
LOGGER.info("Made it to MEDA tag");
out.get(medaPcollection).output(new AvroPubsubMessageRecord(
message.getPayload(), message.getAttributeMap(), context.timestamp().getMillis()));
} else {
LOGGER.info("Made it to default tag");
out.get(noTag).output(new AvroPubsubMessageRecord(
message.getPayload(), message.getAttributeMap(), context.timestamp().getMillis()));
}
} catch (Exception e) {
LOGGER.info("Error Processing Message: {}\n{}", msgStr, e);
}
}
}

Can you try without MultiOutputReceiver out parameter in the processElement method ?
Outputs are then returned with context.output with passing element and corresponding TupleTag.
Your example only with context :
static class PubsubMessageToArchiveDoFn extends DoFn<PubsubMessage, AvroPubsubMessageRecord> {
final TupleTag<AvroPubsubMessageRecord> medaPcollection = new TupleTag<AvroPubsubMessageRecord>(){};
final TupleTag<AvroPubsubMessageRecord> noTag = new TupleTag<AvroPubsubMessageRecord>(){};
#ProcessElement
public void processElement(ProcessContext context) {
String appCode;
PubsubMessage message = context.element();
String msgStr = new String(message.getPayload(), StandardCharsets.UTF_8);
try {
JSONObject jsonObject = new JSONObject(msgStr);
LOGGER.info("json: {}", jsonObject);
appCode = jsonObject.getString("app_code");
LOGGER.info(appCode);
if(appCode == "MEDA"){
LOGGER.info("Made it to MEDA tag");
context.output(medaPcollection, new AvroPubsubMessageRecord(
message.getPayload(), message.getAttributeMap(), context.timestamp().getMillis()));
} else {
LOGGER.info("Made it to default tag");
context.output(noTag, new AvroPubsubMessageRecord(
message.getPayload(), message.getAttributeMap(), context.timestamp().getMillis()));
}
} catch (Exception e) {
LOGGER.info("Error Processing Message: {}\n{}", msgStr, e);
}
}
I also show you an example that works for me :
public class WordCountFn extends DoFn<String, Integer> {
private final TupleTag<Integer> outputTag = new TupleTag<Integer>() {};
private final TupleTag<Failure> failuresTag = new TupleTag<Failure>() {};
#ProcessElement
public void processElement(ProcessContext ctx) {
try {
// Could throw ArithmeticException.
final String word = ctx.element();
ctx.output(1 / word.length());
} catch (Throwable throwable) {
final Failure failure = Failure.from("step", ctx.element(), throwable);
ctx.output(failuresTag, failure);
}
}
public TupleTag<Integer> getOutputTag() {
return outputTag;
}
public TupleTag<Failure> getFailuresTag() {
return failuresTag;
}
}
In my first output (good case), no need to pass the TupleTag ctx.output(1 / word.length());
For my second output (failure case), I pass the Failure tag with the corresponding element.

I was able to get around this by making my ParDo an anonymous function instead of a class. I put the whole function inline and had no problem finding the output tags after I did that. Thanks for the suggestions!

how to remove a query parameter from a query string

I am using UriBuilder to remove a parameter from a URI:
public static URI removeParameterFromURI(URI uri, String param) {
UriBuilder uriBuilder = UriBuilder.fromUri(uri);
return uriBuilder.replaceQueryParam(param, "").build();
}
public static String removeParameterFromURIString(String uriString, String param) {
try {
URI uri = removeParameterFromURI(new URI(uriString), param);
return uri.toString();
} catch (URISyntaxException e) {
throw new RuntimeException(e);
}
}
The above sort of works and modifies:
http://a.b.c/d/e/f?foo=1&bar=2&zar=3
… into:
http://a.b.c/d/e/f?bar=&foo=1&zar=3
But it has the following issues:
It messes up the order of the parameters. I know that the order is not relevant but it still bothers me.
it doesn't fully remove the parameter, it just sets its value to the empty string. I would prefer is the parameter is completely removed from the query string.
Is there some standard or commonly used library that can achieve the above neatly without having to parse and hack the query string myself?

In Android, without import any library.
I write a util method inspired by this answerReplace query parameters in Uri.Builder in Android?(Replace query parameters in Uri.Builder in Android?)
Hope can help you. Code below:
public static Uri removeUriParameter(Uri uri, String key) {
final Set<String> params = uri.getQueryParameterNames();
final Uri.Builder newUri = uri.buildUpon().clearQuery();
for (String param : params) {
if (!param.equals(key)) {
newUri.appendQueryParameter(param, uri.getQueryParameter(param));
}
}
return newUri.build();
}

Using httpclient URIBuilder would be much cleaner if you can.
public String removeQueryParameter(String url, String parameterName) throws URISyntaxException {
URIBuilder uriBuilder = new URIBuilder(url);
List<NameValuePair> queryParameters = uriBuilder.getQueryParams();
for (Iterator<NameValuePair> queryParameterItr = queryParameters.iterator(); queryParameterItr.hasNext();) {
NameValuePair queryParameter = queryParameterItr.next();
if (queryParameter.getName().equals(parameterName)) {
queryParameterItr.remove();
}
}
uriBuilder.setParameters(queryParameters);
return uriBuilder.build().toString();
}

If you are on Android and want to remove all query parameters, you can use
Uri uriWithoutQuery = Uri.parse(urlWithQuery).buildUpon().clearQuery().build();

Using streams and URIBuilder from httpclient it would look like this
public String removeQueryParameter(String url, String parameterName) throws URISyntaxException {
URIBuilder uriBuilder = new URIBuilder(url);
List<NameValuePair> queryParameters = uriBuilder.getQueryParams()
.stream()
.filter(p -> !p.getName().equals(parameterName))
.collect(Collectors.toList());
if (queryParameters.isEmpty()) {
uriBuilder.removeQuery();
} else {
uriBuilder.setParameters(queryParameters);
}
return uriBuilder.build().toString();
}

To fully remove the parameter, you can use
public static URI removeParameterFromURI(URI uri, String param) {
UriBuilder uriBuilder = UriBuilder.fromUri(uri);
return uriBuilder.replaceQueryParam(param, (Object[]) null).build();
}

The following piece of code worked for me:
Code:
import java.util.Arrays;
import java.util.stream.Collectors;
public class RemoveURL {
public static void main(String[] args) {
final String remove = "password";
final String url = "http://testdomainxyz.com?username=john&password=cena&password1=cena";
System.out.println(url);
System.out.println(RemoveURL.removeParameterFromURL(url, remove));
}
public static String removeParameterFromURL(final String url, final String remove) {
final String[] urlArr = url.split("\\?");
final String params = Arrays.asList(urlArr[1].split("&")).stream()
.filter(item -> !item.split("=")[0].equalsIgnoreCase(remove)).collect(Collectors.joining("&"));
return String.join("?", urlArr[0], params);
}
}
Output
http://testdomainxyz.com?username=john&password=cena&password1=cena
http://testdomainxyz.com?username=john&password1=cena

Based on the suggestion by JB Nizzet, this is what I ended up doing (I added some extra logic to be able to assert whether I expect the parameter to be present, and if so, how many times):
public static URI removeParameterFromURI(URI uri, String parameter, boolean assertAtLeastOneIsFound, Integer assertHowManyAreExpected) {
Assert.assertFalse("it makes no sense to expect 0 or less", (assertHowManyAreExpected!=null) && (assertHowManyAreExpected<=0) );
Assert.assertFalse("it makes no sense to not assert that at least one is found and at the same time assert a definite expected number", (!assertAtLeastOneIsFound) && (assertHowManyAreExpected!=null) );
String queryString = uri.getQuery();
if (queryString==null)
return uri;
Map<String, List<String>> params = parseQuery(queryString);
Map<String, List<String>> paramsModified = new LinkedHashMap<>();
boolean found = false;
for (String key: params.keySet()) {
if (!key.equals(parameter))
Assert.assertNull(paramsModified.put(key, params.get(key)));
else {
found = true;
if (assertHowManyAreExpected!=null) {
Assert.assertEquals((long) assertHowManyAreExpected, params.get(key).size());
}
}
}
if (assertAtLeastOneIsFound)
Assert.assertTrue(found);
UriBuilder uriBuilder = UriBuilder.fromUri(uri)
.replaceQuery("");
for (String key: paramsModified.keySet()) {
List<String> values = paramsModified.get(key);
uriBuilder = uriBuilder.queryParam(key, (Object[]) values.toArray(new String[values.size()]));
}
return uriBuilder.build();
}
public static String removeParameterFromURI(String uri, String parameter, boolean assertAtLeastOneIsFound, Integer assertHowManyAreExpected) {
try {
return removeParameterFromURI(new URI(uri), parameter, assertAtLeastOneIsFound, assertHowManyAreExpected).toString();
} catch (URISyntaxException e) {
throw new RuntimeException(e);
}
}
private static Map<String, List<String>> parseQuery(String queryString) {
try {
final Map<String, List<String>> query_pairs = new LinkedHashMap<String, List<String>>();
final String[] pairs = queryString.split("&");
for (String pair : pairs) {
final int idx = pair.indexOf("=");
final String key = idx > 0 ? URLDecoder.decode(pair.substring(0, idx), StandardCharsets.UTF_8.name()) : pair;
if (!query_pairs.containsKey(key)) {
query_pairs.put(key, new ArrayList<String>());
}
final String value = idx > 0 && pair.length() > idx + 1 ? URLDecoder.decode(pair.substring(idx + 1), StandardCharsets.UTF_8.name()) : null;
query_pairs.get(key).add(value);
}
return query_pairs;
} catch (UnsupportedEncodingException e) {
throw new RuntimeException(e);
}
}

You can use simpler method from Collection based on #Flips solution:
public String removeQueryParameter(String url, String parameterName) throws URISyntaxException {
URIBuilder uriBuilder = new URIBuilder(url);
List<NameValuePair> queryParameters = uriBuilder.getQueryParams();
queryParameters.removeIf(param ->
param.getName().equals(parameterName));
uriBuilder.setParameters(queryParameters);
return uriBuilder.build().toString();
}

I am not sure if there is some library to help, but I would just split the string on "?" and take the second half and split it on "&". Then I would rebuild the string accordingly.
public static void main(String[] args) {
// TODO code application logic here
System.out.println("original: http://a.b.c/d/e/f?foo=1&bar=2&zar=3");
System.out.println("new : " + fixString("http://a.b.c/d/e/f?foo=1&bar=2&zar=3"));
}
static String fixString(String original)
{
String[] processing = original.split("\\?");
String[] processing2ndHalf = processing[1].split("&");
return processing[0] + "?" + processing2ndHalf[1] + "&" + processing2ndHalf[0] + "&" + processing2ndHalf[2];
}
Output:
To remove a paramater just remove it from the return string.

public static String removeQueryParameter(String url, List<String> removeNames) {
try {
Map<String, String> queryMap = new HashMap<>();
Uri uri = Uri.parse(url);
Set<String> queryParameterNames = uri.getQueryParameterNames();
for (String queryParameterName : queryParameterNames) {
if (TextUtils.isEmpty(queryParameterName)
||TextUtils.isEmpty(uri.getQueryParameter(queryParameterName))
|| removeNames.contains(queryParameterName)) {
continue;
}
queryMap.put(queryParameterName, uri.getQueryParameter(queryParameterName));
}
// remove all params
Uri.Builder uriBuilder = uri.buildUpon().clearQuery();
for (String name : queryMap.keySet()) {
uriBuilder.appendQueryParameter(name, queryMap.get(name));
}
return uriBuilder.build().toString();
} catch (Exception e) {
return url;
}
}

#TTKatrina's answer worked for me, but I need to remove query param from fragment too. So extended that for fragment and came up with this.
fun Uri.removeQueryParam(key: String): Uri {
//Create new Uri builder with no query params.
val builder = buildUpon().clearQuery()
//Add all query params excluding the key we don't want back to the new Uri.
queryParameterNames.filter { it != key }
.onEach { builder.appendQueryParameter(it, getQueryParameter(it)) }
//If query param is in fragment, remove from it.
val fragmentUri = fragment?.toUri()
if (fragmentUri != null) {
builder.encodedFragment(fragmentUri.removeQueryParam(key).toString())
}
//Now this Uri doesn't have the query param for [key]
return builder.build()
}

java.lang.IllegalArgumentException: Object would be serialized to `null`: Android

I have recently been setting up mobile apps to work with my meteor server. As a part of this I have to pass the meteor web app data from android. Unfortunately I have been receiving a error that tells me that the java object I am passing "would be serialized to null". How do I prevent this?
JSONObject json = new JSONObject();
try{
json.put("Foo", "1");
json.put("Blah", 0);
}catch (JSONException e){
}
Object[] object = new Object[1];
object[0] = json;
System.out.println(object + ", " + object[0] + ", " + object[0].toString());
mMeteor.call("xxx", object, new ResultListener() {
#Override
public void onSuccess(String result) {
}
#Override
public void onError(String error, String reason, String details) {
}
});
}
#Override
public void onError(String error, String reason, String details) {
}
});
Android/Meteor interface Library function
public void callWithSeed(final String methodName, final String randomSeed, final Object[] params, final ResultListener listener) {
// create a new unique ID for this request
final String callId = uniqueID();
// save a reference to the listener to be executed later
if (listener != null) {
mListeners.put(callId, listener);
}
// send the request
final Map<String, Object> data = new HashMap<String, Object>();
data.put(Protocol.Field.MESSAGE, Protocol.Message.METHOD);
data.put(Protocol.Field.METHOD, methodName);
data.put(Protocol.Field.ID, callId);
if (params != null) {
data.put(Protocol.Field.PARAMS, params);
}
if (randomSeed != null) {
data.put(Protocol.Field.RANDOM_SEED, randomSeed);
}
send(data);
}

I was having this same issue, my first error was passing a CharSequence instead a String as a parameter (your Object[]), and my other error was passing an Object[] as another parameter (I solved this by sending a String instead, like : String.valueOf(your_object_list)) Dont forget to handle this on your server side, you will receive a String instead of an Object.

Convert the JSONArray to List & JSONObject to HashMap and then pass those instead of the raw JSONObject or JSONArray.
You can write a recursive function for the conversion in case of nested JSONObject and JSONArray or can use GSON library for the conversion.
For more details about the conversion, this SO post may be helpful.

Java sends a JSON string in POST request with escaped double quotes [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How can I stop HTTP from escaping quotes?
I am creating a JSONObject and sending JSON string to a server in a POST request body.
public String toJson() {
JSONObject filter = new JSONObject();
try {
filter.put("gender", gender.getCode());
filter.put("feature_id", productCategory);
} catch (JSONException e) {
e.printStackTrace();
}
JSONObject filterObject = new JSONObject();
try {
filterObject.put("filter", filter);
} catch (JSONException e) {
e.printStackTrace();
}
return filterObject.toString();
}
So I'm creating a request:
private IJsonExecutorInterface requestExecutorForRelativePathAndParams(String path, WebParams params) throws UnsupportedEncodingException {
HttpPost postRequest = new HttpPost(rootUrl + path);
if(params != null) {
postRequest.setHeader("content-type", params.getContentType());
postRequest.setEntity(params.getFormEntity());
}
// Blah blah
return executor;
}
public IJsonExecutorInterface getProducts(ProductFilter filter, int offset, int limit) throws UnsupportedEncodingException {
WebParams webParams = new WebParams();
webParams.addPair("filter", filter.toJson());
webParams.addPair("offset", String.format("%d", offset));
webParams.addPair("limit", String.format("%d", limit));
return requestExecutorForRelativePathAndParams("products", webParams);
}
// WebParams class
public class WebParams {
private ArrayList<NameValuePair> params;
private String contentType = "application/x-www-form-urlencoded";
public WebParams() {
params = new ArrayList<NameValuePair>();
}
public void addPair(String name, String value) {
params.add(new BasicNameValuePair(name, value));
}
public String getContentType() {
return contentType;
}
public HttpEntity getFormEntity() throws UnsupportedEncodingException {
return new UrlEncodedFormEntity(params);
}
}
I see it in debugger: it's ok.
But on my server I getting something like this:
Array
(
[filter] => {\"gender\":\"w\",\"feature_id\":\"41_7459\"}
[offset] => 0
[limit] => 18
)
The quotes ARE escaped.
I don't want to replace something on the server. replace("\\\"", "\"") in Java doesn't affect on the string.

Looks like your using a UrlEncodedFormEntity, which, according to the docs is 'An entity composed of a list of url-encoded pairs' ([http://developer.android.com/reference/org/apache/http/client/entity/UrlEncodedFormEntity.html]). I've never used this, but it doesn't sound like its what you want, as you are sending data in the post body, not through URL parameters.
I've used the StringEntity class before to send json data via post, although it only encodes a string, not name/value pairs, so you've got to do a bit more work to put the string in a format you want to deal with on your server:
public class WebParams {
private ArrayList<NameValuePair> params;
private String contentType = "application/x-www-form-urlencoded";
public WebParams() {
params = new ArrayList<NameValuePair>();
}
public void addPair(String name, String value) {
params.add(new BasicNameValuePair(name, value));
}
public String getContentType() {
return contentType;
}
public HttpEntity getFormEntity() throws UnsupportedEncodingException {
//TODO: Build a string in what ever format you want.
// This will include the gender & feature_id fields as well as the json
StringBuilder b = new StringBuilder();
for(NameValuePair nvp : params) {
builder.append(nvp.getName()).append('=').append(nvp.getValue()).append(',');
}
//Now that we have a string to send to the server, get your entity!
StringEntity entity = new StringEntity(b.toString());
entity.setContentEncoding(new BasicHeader(HTTP.CONTENT_TYPE, "application/json"));
return entity;
}
}

Is there a problem to use simple quotes instead of double quotes? Because I think it would solve your problem.