Develop Reference

Can you use Lightcouch to query documents by filter?

I am using lightcouch for my spring boot application and I need to be able to query my CouchDb database for documents based on a provided filter. Since these filters can always be different, I can't make use of a preset view. I am looking for something that would work in a similar way of the normal find like below:
public List<MyEntity> getEntities(MyFilter myFilter)
{
return dbClient.find(resourceFilter, MyEntity.class);
}
myFilter would be a Map object which I would use to query documents based on certain values provided in this map. Is it possible? Is there an approach to achieve what I want? Thanks

LightCouch provides a method for querying CouchDB using mango selectors.
See CouchDbClientBase.java
/**
* Find documents using a declarative JSON querying syntax.
* #param <T> The class type.
* #param jsonQuery The JSON query string.
* #param classOfT The class of type T.
* #return The result of the query as a {#code List<T> }
* #throws CouchDbException If the query failed to execute or the request is invalid.
*/
public <T> List<T> findDocs(String jsonQuery, Class<T> classOfT) { ...

LightCouch internal API allows for a user-defined raw HTTP request to execute against a database. This can be done through the CouchDbClient#executeRequest method.
I'm not using LightCouch in my Java project but Apache HTTPClient together with GSON. Below example assumes CouchDB being installed on your local computer and both, user and password to be "admin". It could be easily adapted to use CouchDbClient#executeRequest.
The mangoSelector parameter in the find method must comply with the CouchDB selector syntax.
public class CouchDBAccess {
private static final String BASE_URL = "http://localhost:5984/";
private static final Gson GSON = new GsonBuilder().create();
private final Header[] httpHeaders;
public CouchDBAccess() {
this.httpHeaders = new Header[] { //
new BasicHeader("Accept", "application/json"), //
new BasicHeader("Content-type", "application/json"), //
new BasicHeader("Authorization", "Basic " + Base64.getEncoder().encodeToString("admin:admin".getBytes())) //
};
}
FindResult find(String dbName, String mangoSelector) throws IOException {
try (CloseableHttpClient client = HttpClientBuilder.create().build()) {
HttpPost httpPost = new HttpPost(BASE_URL + dbName + "/_find");
httpPost.setHeaders(httpHeaders);
httpPost.setEntity(new StringEntity(mangoSelector, ContentType.APPLICATION_JSON));
HttpResponse response = client.execute(httpPost);
int statusCode = response.getStatusLine().getStatusCode();
if (statusCode == HttpStatus.SC_OK) {
return GSON.fromJson(extractContent(response), FindResult.class);
} else {
// handle invalid response
}
}
}
private String extractContent(HttpResponse response) throws IOException {
StringWriter writer = new StringWriter();
IOUtils.copy(response.getEntity().getContent(), writer, defaultCharset());
return writer.toString();
}
}
class FindResult {
MyEntity[] docs;
}
A corresponding jUnit test method could look as follows:
#Test
public void testFind() throws IOException {
String mangoSelector = "{\"selector\": {\"Host\": \"local drive\"}}";
FindResult findResult = couchDBAccess.find("data_1", mangoSelector);
assertEquals(100, findResult.docs.length); // or whatever you expect
}

How to call Akka Http POST in loop (1000-10000 times)?

I am learning Akka with Java. I have written a simple program with two actors.
My first actor ActorA is called with list containing 1000 strings. ActorA loops through the list and calls ActorB for each element.
ActorB makes a Http POST call to external service using the String parameter received from ActorA.
I am expecting that ActorB will successfully make 1000 Http POST calls and will receive equal number of responses. However ActorB is able to make POST request randomly between 80-120 times then it stops making POST calls.
I tried providing a custom dispatcher as HTTP POST call is a blocking operation but still no luck!!
Refer to code and configuration given below.
public class ActorA extends AbstractActor {
static public Props props() {
return Props.create(ActorA.class);
}
static public class IdWrapper {
List<String> ids;
public IdWrapper(List<String> ids) {
this.ids = ids;
}
}
#Override
public Receive createReceive() {
return receiveBuilder()
.match(IdWrapper.class, this::process)
.build();
}
private void process(IdWrapper msg) {
msg.ids.forEach(id -> {
context().actorSelection("actorB").tell(new MessageForB(id), ActorRef.noSender());
}
);
}
}
public class ActorB extends AbstractActor {
final Http http = Http.get(getContext().system());
final Materializer materializer = ActorMaterializer.create(context());
public static Props props() {
return Props.create(ActorB.class);
}
static public class MessageForB implements Serializable {
String id;
public MessageForB(String id) {
this.id = id;
}
}
#Override
public Receive createReceive() {
return receiveBuilder()
.match(MessageForB.class, this::process)
.build();
}
private void process(MessageForB messageForB) {
ExecutionContext ec = getContext().getSystem().dispatchers().lookup("my-blocking-dispatcher");
/**
* Get id from request
*/
String reqId = messageForB.id;
/**
* Prepare request
*/
XmlRequest requestEntity = getRequest(Stream.of(reqId).collect(Collectors.toList()));
String requestAsString = null;
try {
/**
* Create and configure JAXBMarshaller.
*/
JAXBContext jaxbContext = JAXBContext.newInstance(XmlRequest.class);
Marshaller jaxbMarshaller = jaxbContext.createMarshaller();
jaxbMarshaller.setProperty(Marshaller.JAXB_FRAGMENT, Boolean.TRUE);
/**
* Convert request entity to string before making POST request.
*/
StringWriter sw = new StringWriter();
jaxbMarshaller.marshal(requestEntity, sw);
requestAsString = sw.toString();
} catch (JAXBException e) {
e.printStackTrace();
}
/**
* Create RequestEntity from request string.
*/
RequestEntity entity = HttpEntities.create(
MediaTypes.APPLICATION_XML.toContentType(HttpCharsets.ISO_8859_1),
requestAsString);
/**
* Create Http POST with necessary headers and call
*/
final CompletionStage<HttpResponse> responseFuture =
http.singleRequest(HttpRequest.POST("http://{hostname}:{port}/path")
.withEntity(entity));
responseFuture
.thenCompose(httpResponse -> {
/**
* Convert response into String
**/
final CompletionStage<String> res = Unmarshaller.entityToString().unmarshal
(httpResponse.entity(), ec, materializer);
/**
* Consume response bytes
**/
httpResponse.entity().getDataBytes().runWith(Sink.ignore(), materializer);
return res;
})
.thenAccept(s -> {
try {
/**
* Deserialize string to DTO.
*/
MyResponse MyResponse = getMyResponse(s);
// further processing..
} catch (JAXBException e) {
e.printStackTrace();
}
});
}
private XmlRequest getRequest(List<String> identifiers){
XmlRequest request = new XmlRequest();
// Business logic to create req entity
return request;
}
private MyResponse getMyResponse(String s) throws JAXBException {
JAXBContext jaxbContext = JAXBContext.newInstance
(MyResponse.class);
javax.xml.bind.Unmarshaller jaxbUnmarshaller = jaxbContext
.createUnmarshaller();
StringReader reader = new StringReader(s);
return (MyResponse)
jaxbUnmarshaller.unmarshal(reader);
}
}
my-blocking-dispatcher {
type = Dispatcher
executor = "thread-pool-executor"
thread-pool-executor {
core-pool-size-min = 5
core-pool-size-max = 20
}
throughput = 1
}
Where can I improve or correct my code so that ActorB will successfully be able to make Http POST calls for all the items sent by ActorA ?

As i see you have used http.singleReques.
According to the akka-http docs
For these cases Akka HTTP offers the Http().singleRequest(...) method, which simply turns an HttpRequest instance into Future[HttpResponse]. Internally the request is dispatched across the (cached) host connection pool for the request’s effective URI.
http.singleRequest uses connection pool to handle requests so you need to increase number of connections in connection pool from akka http config.
in host-connection-pool section with this defaults:
host-connection-pool {
max-connections = 4
min-connections = 0
max-retries = 5
max-open-requests = 32
pipelining-limit = 1
idle-timeout = 30 s
}
Solution 2 :
using http.outgoingConnection
according to the akka-http docs it will be create an specific connection per request. So you can handle 1000 connections in parallel without connection pool.
With the connection-level API you open a new HTTP connection to a target endpoint by materializing a Flow returned by the Http().outgoingConnection(...) method. Here is an example:
def run(req:String): Unit ={
val apiBaseUrl = "example.com" //without protocol
val path = "/api/update"
val body = HttpEntity(ContentTypes.`application/json`,req.getBytes)
val request = HttpRequest(HttpMethods.POST, path,entity = body)
val connectionFlow = Http().outgoingConnection(apiBaseUrl)
val result = Source.single(request).via(connectionFlow).runWith(Sink.head)
result.onComplete{
case Success(value) =>
println(value)
case Failure(e)=>
e.printStackTrace()
}
}

What's the problem of my Request parameter Interceptor in Spring-boot Restful service?

The original question is here:
How to resolve URI encoding problem in spring-boot?. And following one of the suggestions, I am trying to come up with a solution with an Interceptor, but still has some issue.
I need to be able to handle some special characters in URL, for instance "%" and my spring controller is below:
#Controller
#EnableAutoConfiguration
public class QueryController {
private static final Logger LOGGER = LoggerFactory.getLogger(QueryController.class);
#Autowired
QueryService jnService;
#RequestMapping(value="/extract", method = RequestMethod.GET)
#SuppressWarnings("unchecked")
#ResponseBody
public ExtractionResponse extract(#RequestParam(value = "extractionInput") String input) {
// LOGGER.info("input: " + input);
JSONObject inputObject = JSON.parseObject(input);
InputInfo inputInfo = new InputInfo();
JSONObject object = (JSONObject) inputObject.get(InputInfo.INPUT_INFO);
String inputText = object.getString(InputInfo.INPUT_TEXT);
inputInfo.setInputText(inputText);
return jnService.getExtraction(inputInfo);
}
}
Following suggestions I want to write an interceptor to encode the URL before the request is sent to the controller, and my interceptor looks like (not complete yet):
public class ParameterInterceptor extends HandlerInterceptorAdapter {
private static final Logger LOGGER = LoggerFactory.getLogger(ParameterInterceptor.class);
#Override
public boolean preHandle(HttpServletRequest request,
HttpServletResponse reponse,
Object handler) throws Exception {
Enumeration<?> e = request.getParameterNames();
LOGGER.info("Request URL::" + request.getRequestURL().toString());
StringBuffer sb = new StringBuffer();
if (e != null) {
sb.append("?");
}
while (e.hasMoreElements()) {
String curr = (String) e.nextElement();
sb.append(curr + "=");
sb.append(request.getParameter(curr));
}
LOGGER.info("Parameter: " + sb.toString());
return true;
}
}
I tested a URL in my browser:
http://localhost:8090/extract?extractionInput={"inputInfo":{"inputText":"5.00%"}}
Due to the % sign, I received the error:
[log] - Character decoding failed. Parameter [extractionInput] with value [{"inputInfo":{"inputText":"5.0022:%225.00%%22}}] has been ignored. Note that the name and value quoted here may be corrupted due to the failed decoding.
When I test the interceptor, "request.getRequestURL()" gives the expected result:
http://localhost:8090/extract
However, "request.getParameterNames()" always get an empty Elumentation object. Why doesn't it get the parameters? What I hope is to first encode the parameter value:
"inputText":"5.00%"
'inputText' is a field of the object InputInfo in the json format. So how to get the request parameters to solve the problem?

MD5 hashcode using spring for http request

i want to generate unique md5 for every http request that will hit REST API.
So far i have just used String requestParameters but actual httpRequest will have many other things.
How can i achieve this ?
public final class MD5Generator {
public static String getMd5HashCode(String requestParameters) {
return DigestUtils.md5DigestAsHex(requestParameters.getBytes());
}
}
My Controller
#RequestMapping(value = { "/dummy" }, method = RequestMethod.GET)
public String processOperation(HttpServletRequest request) {
serviceLayer = new ServiceLayer(request);
return "wait operation is executing";
}
Service layer
private String httpRequestToString() {
String request = "";
Enumeration<String> requestParameters = httpRequest.getParameterNames();
while (requestParameters.hasMoreElements()) {
request += String.valueOf(requestParameters.nextElement());
}
if (!request.equalsIgnoreCase(""))
return request;
else {
throw new HTTPException(200);
}
}
private String getMD5hash() {
return MD5Generator.getMd5HashCode(httpRequestToString());
}

Do you see any issues with generating an UUID for every request and use that instead?
For example, you could generate the UUID and attach it to the request object if you need it during the request life-cycle:
String uuid = UUID.randomUUID().toString();
request.setAttribute("request-id", uuid);

You can combine request time (System.currentTimeMillis()) and remote address from HttpServletRequest. However, if you're expecting high loads, multiple requests may arrive from a particular client in the same millisecond. To overcome this situation, you may add a global atomic counter to your String combination.
Once you generate an MD5 key, you can set it in ThreadLocal to reach afterwards.

You can do this but in future maybe. I search and not found automated way to achieve this
#GetMapping("/user/{{md5(us)}}")

Retrieve a list of a given user's tweets using Twitter API 1.1 and Retrofit

I'm trying to obtain a list of a user's tweets and I've run into some trouble when trying to authenticate my call to the API. I currently get a 401 when executing the code below:
public interface TwitterApi {
String API_URL = "https://api.twitter.com/1.1";
String CONSUMER_KEY = "<CONSUMER KEY GOES HERE>";
String CONSUMER_SECRET = "<CONSUMER SECRET GOES HERE>";
String ACCESS_TOKEN = "<ACCESS TOKEN GOES HERE>";
String ACCESS_TOKEN_SECRET = "<ACCESS TOKEN SECRET GOES HERE>";
#GET("/statuses/user_timeline.json")
List<Tweet> fetchUserTimeline(
#Query("count") final int count,
#Query("screen_name") final String screenName);
}
The following throws a 401 Authorisation error when calling fetchUserTimeline()
RetrofitHttpOAuthConsumer consumer = new RetrofitHttpOAuthConsumer(TwitterApi.CONSUMER_KEY, TwitterApi.CONSUMER_SECRET);
consumer.setTokenWithSecret(TwitterApi.ACCESS_TOKEN, TwitterApi.ACCESS_TOKEN_SECRET);
RestAdapter restAdapter = new RestAdapter.Builder()
.setEndpoint(TwitterApi.API_URL)
.setClient(new SigningOkClient(consumer))
.build();
TwitterApi twitterApi = restAdapter.create(TwitterApi.class)
tweets = twitterApi.fetchUserTimeline(2, screenName);
I've also included the relevant code from the signpost-retrofit plugin:
public class SigningOkClient extends OkClient {
private final RetrofitHttpOAuthConsumer mOAuthConsumer;
public SigningOkClient(RetrofitHttpOAuthConsumer consumer) {
mOAuthConsumer = consumer;
}
public SigningOkClient(OkHttpClient client, RetrofitHttpOAuthConsumer consumer) {
super(client);
mOAuthConsumer = consumer;
}
#Override
public Response execute(Request request) throws IOException {
Request requestToSend = request;
try {
HttpRequestAdapter signedAdapter = (HttpRequestAdapter) mOAuthConsumer.sign(request);
requestToSend = (Request) signedAdapter.unwrap();
} catch (OAuthMessageSignerException | OAuthExpectationFailedException | OAuthCommunicationException e) {
// Fail to sign, ignore
e.printStackTrace();
}
return super.execute(requestToSend);
}
}
The signpost-retrofit plugin can be found here: https://github.com/pakerfeldt/signpost-retrofit
public class RetrofitHttpOAuthConsumer extends AbstractOAuthConsumer {
private static final long serialVersionUID = 1L;
public RetrofitHttpOAuthConsumer(String consumerKey, String consumerSecret) {
super(consumerKey, consumerSecret);
}
#Override
protected HttpRequest wrap(Object request) {
if (!(request instanceof retrofit.client.Request)) {
throw new IllegalArgumentException("This consumer expects requests of type " + retrofit.client.Request.class.getCanonicalName());
}
return new HttpRequestAdapter((Request) request);
}
}
Any help here would be great. The solution doesn't have to include the use of signpost but I do want to use Retrofit. I also do not want to show the user an 'Authenticate with Twitter' screen in a WebView - I simply want to display a handful of relevant tweets as part of a detail view.

Are you certain the signpost-retrofit project works for twitter oauth? I've used twitter4j successfully in the past - and if you don't want the full library you can use their code for reference. twitter4j