Develop Reference

AWS ElasticSearch/OpenSearch not connecting from java

Im using a simple example from amazon aws site to connect to opensearch index.
This is the example source https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html#request-signing-java.
The health status of my node is yellow and its open
yellow open my-index
The error message
Exception in thread "main" java.net.ConnectException
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:943)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1256)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1231)
at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:587)
at com.amazonaws.lambda.demo.AWSElasticsearchServiceClient.main(AWSElasticsearchServiceClient.java:41)
Caused by: java.net.ConnectException
at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Unknown Source) ```
private static String region = "us-west-1";
private static String domainEndpoint = "<my-index...amazon.com>"; // e.g. https://search-mydomain.us-west-1.es.amazonaws.com
private static String index = "my-index";
private static String type = "_doc";
private static String id = "1";
static final AWSCredentialsProvider credentialsProvider = new DefaultAWSCredentialsProviderChain();
``` public static void main(String[] args) throws IOException {
RestHighLevelClient searchClient = searchClient(serviceName, region);
// Create the document as a hash map
Map<String, Object> document = new HashMap<>();
document.put("title", "Walk the Line");
document.put("director", "James Mangold");
document.put("year", "2005");
// Form the indexing request, send it, and print the response
IndexRequest request = new IndexRequest(index, type, id).source(document);
IndexResponse response = searchClient.index(request, RequestOptions.DEFAULT);
System.out.println(response.toString());
}
// Adds the interceptor to the OpenSearch REST client
public static RestHighLevelClient searchClient(String serviceName, String region) {
AWS4Signer signer = new AWS4Signer();
signer.setServiceName(serviceName);
signer.setRegionName(region);
HttpRequestInterceptor interceptor = new AWSRequestSigningApacheInterceptor(serviceName, signer, credentialsProvider);
return new RestHighLevelClient(RestClient.builder(HttpHost.create(domainEndpoint)).setHttpClientConfigCallback(hacb -> hacb.addInterceptorLast(interceptor)));
}

Try this example. I tried the same and it did work well for me. I did not bother doing anything in regards to the cert as I had followed AWS demo examples to create the domain.
Hopefully this is what you are looking for...

POST API request timeout issue

I'm trying to make a POST API call to our external API via the API PROXY. I'm facing some timeout issues while making a API call. I used below restTemplate to make a API request. But at the same time I've implemented the retry for the template in case of any timeouts. I can avoid this issue by retrying that api request again.But I would like to find out the root cause for that to completely resolve the issue.Can anyone help me with this
StackTrace:
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://localhost:8080/test": Operation timed out (Read failed); nested exception is java.net.SocketException: Operation timed out (Read failed)
private static final int HTTP_CLIENT_RETRY_COUNT = 3;
private static final int MAXIMUM_TOTAL_CONNECTION = 10;
private static final int MAXIMUM_CONNECTION_PER_ROUTE = 5;
private static final int CONNECTION_VALIDATE_AFTER_INACTIVITY_MS = 10 * 1000;
public static RestTemplate createRestTemplate(int connectionTimeoutMs, int readTimeoutMs, ObjectMapper objectMapper) {
HttpClientBuilder clientBuilder = HttpClients.custom();
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
// Set the maximum number of total open connections.
connectionManager.setMaxTotal(MAXIMUM_TOTAL_CONNECTION);
// Set the maximum number of concurrent connections per route, which is 2 by default.
connectionManager.setDefaultMaxPerRoute(MAXIMUM_CONNECTION_PER_ROUTE);
connectionManager.setValidateAfterInactivity(CONNECTION_VALIDATE_AFTER_INACTIVITY_MS);
clientBuilder.setConnectionManager(connectionManager);
clientBuilder.setRetryHandler(new DefaultHttpRequestRetryHandler(HTTP_CLIENT_RETRY_COUNT, true, new ArrayList<>()) {
#Override
public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
LOGGER.info("Retry request, execution count: {}, exception: {}", executionCount, exception);
return super.retryRequest(exception, executionCount, context);
}
});
HttpComponentsClientHttpRequestFactory httpRequestFactory = new HttpComponentsClientHttpRequestFactory(clientBuilder.build());
httpRequestFactory.setConnectTimeout(connectionTimeoutMs);
httpRequestFactory.setConnectionRequestTimeout(readTimeoutMs);
httpRequestFactory.setReadTimeout(readTimeoutMs);
RestTemplate restTemplate = new RestTemplate(httpRequestFactory);
List<ClientHttpRequestInterceptor> interceptors = new ArrayList<ClientHttpRequestInterceptor>();
interceptors.add(new LoggingRequestInterceptor());
restTemplate.setInterceptors(interceptors);
restTemplate.setRequestFactory(new BufferingClientHttpRequestFactory(httpRequestFactory));
MappingJackson2HttpMessageConverter messageConverter = restTemplate.getMessageConverters().stream().filter(MappingJackson2HttpMessageConverter.class::isInstance)
.map(MappingJackson2HttpMessageConverter.class::cast).findFirst().orElseThrow(() -> new RuntimeException("MappingJackson2HttpMessageConverter not found"));
messageConverter.setObjectMapper(objectMapper);
restTemplate.getMessageConverters().stream().filter(StringHttpMessageConverter.class::isInstance).map(StringHttpMessageConverter.class::cast).forEach(a -> {
a.setWriteAcceptCharset(false);
a.setDefaultCharset(StandardCharsets.UTF_8);
});
return restTemplate;
}

Reactive Spring Boot API wrapping Elasticsearch's async bulk indexing

I am developing prototype for a new project. The idea is to provide a Reactive Spring Boot microservice to bulk index documents in Elasticsearch. Elasticsearch provides a High Level Rest Client which provides an Async method to bulk process indexing requests. Async delivers callbacks using listeners are mentioned here. The callbacks receive index responses (per requests) in batches. I am trying to send this response back to the client as Flux. I have come up with something based on this blog post.
Controller
#RestController
public class AppController {
#SuppressWarnings("unchecked")
#RequestMapping(value = "/test3", method = RequestMethod.GET)
public Flux<String> index3() {
ElasticAdapter es = new ElasticAdapter();
JSONObject json = new JSONObject();
json.put("TestDoc", "Stack123");
Flux<String> fluxResponse = es.bulkIndex(json);
return fluxResponse;
}
ElasticAdapter
#Component
class ElasticAdapter {
String indexName = "test2";
private final RestHighLevelClient client;
private final ObjectMapper mapper;
private int processed = 1;
Flux<String> bulkIndex(JSONObject doc) {
return bulkIndexDoc(doc)
.doOnError(e -> System.out.print("Unable to index {}" + doc+ e));
}
private Flux<String> bulkIndexDoc(JSONObject doc) {
return Flux.create(sink -> {
try {
doBulkIndex(doc, bulkListenerToSink(sink));
} catch (JsonProcessingException e) {
sink.error(e);
}
});
}
private void doBulkIndex(JSONObject doc, BulkProcessor.Listener listener) throws JsonProcessingException {
System.out.println("Going to submit index request");
BiConsumer<BulkRequest, ActionListener<BulkResponse>> bulkConsumer =
(request, bulkListener) ->
client.bulkAsync(request, RequestOptions.DEFAULT, bulkListener);
BulkProcessor.Builder builder =
BulkProcessor.builder(bulkConsumer, listener);
builder.setBulkActions(10);
BulkProcessor bulkProcessor = builder.build();
// Submitting 5,000 index requests ( repeating same JSON)
for (int i = 0; i < 5000; i++) {
IndexRequest indexRequest = new IndexRequest(indexName, "person", i+1+"");
String json = doc.toJSONString();
indexRequest.source(json, XContentType.JSON);
bulkProcessor.add(indexRequest);
}
System.out.println("Submitted all docs
}
private BulkProcessor.Listener bulkListenerToSink(FluxSink<String> sink) {
return new BulkProcessor.Listener() {
#Override
public void beforeBulk(long executionId, BulkRequest request) {
}
#SuppressWarnings("unchecked")
#Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
for (BulkItemResponse bulkItemResponse : response) {
JSONObject json = new JSONObject();
json.put("id", bulkItemResponse.getResponse().getId());
json.put("status", bulkItemResponse.getResponse().getResult
sink.next(json.toJSONString());
processed++;
}
if(processed >= 5000) {
sink.complete();
}
}
#Override
public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
failure.printStackTrace();
sink.error(failure);
}
};
}
public ElasticAdapter() {
// Logic to initialize Elasticsearch Rest Client
}
}
I used FluxSink to create the Flux of Responses to send back to the Client. At this point, I have no idea whether this correct or not.
My expectation is that the calling client should receive the responses in batches of 10 ( because bulk processor processess it in batches of 10 - builder.setBulkActions(10); ). I tried to consume the endpoint using Spring Webflix Client. But unable to work it out. This is what I tried
WebClient
public class FluxClient {
public static void main(String[] args) {
WebClient client = WebClient.create("http://localhost:8080");
Flux<String> responseFlux = client.get()
.uri("/test3")
.retrieve()
.bodyToFlux(String.class);
responseFlux.subscribe(System.out::println);
}
}
Nothing is printing on console as I expected. I tried to use System.out.println(responseFlux.blockFirst());. It prints all the responses as a single batch at the end and not in batches at .
If my approach is correct, what is the correct way to consume it? For the solution in my mind, this client will reside is another Webapp.
Notes: My understanding of Reactor API is limited. The version of elasticsearch used is 6.8.

So made the following changes to your code.
In ElasticAdapter,
public Flux<Object> bulkIndex(JSONObject doc) {
return bulkIndexDoc(doc)
.subscribeOn(Schedulers.elastic(), true)
.doOnError(e -> System.out.print("Unable to index {}" + doc+ e));
}
Invoked subscribeOn(Scheduler, requestOnSeparateThread) on the Flux, Got to know about it from, https://github.com/spring-projects/spring-framework/issues/21507
In FluxClient,
Flux<String> responseFlux = client.get()
.uri("/test3")
.headers(httpHeaders -> {
httpHeaders.set("Accept", "text/event-stream");
})
.retrieve()
.bodyToFlux(String.class);
responseFlux.delayElements(Duration.ofSeconds(1)).subscribe(System.out::println);
Added "Accept" header as "text/event-stream" and delayed Flux elements.
With the above changes, was able to get the response in real time from the server.

HTTP response 411: Length Required when communicating with

I am trying to send a soap request and keep getting HTTP response 411 error because of larger size of soap request. In most of the cases soap request length is more that 8k.
ERROR MESSAGE
2020-02-27 08:26:09,618 WARNING [100] [org.apache.cxf.phase.PhaseInterceptorChain] (my-thread-1) Interceptor for {http://example.com}CreationService#{http://cxf.apache.org/jaxws/dispatch}Invoke has thrown exception, unwinding now: org.apache.cxf.interceptor.Fault: Could not send Message.
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:514) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:423) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:324) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:277) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.endpoint.ClientImpl.invokeWrapped(ClientImpl.java:312) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.jaxws.DispatchImpl.invoke(DispatchImpl.java:327) [cxf-rt-frontend-jaxws-3.1.6.jar:3.1.6]
at org.apache.cxf.jaxws.DispatchImpl.invoke(DispatchImpl.java:246) [cxf-rt-frontend-jaxws-3.1.6.jar:3.1.6]
...
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response '411: Length Required' when communicating with http://192.100.110.17:8504/example/services/CreationREQ
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1600) [cxf-rt-transports-http-3.1.6.jar:3.1.6]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1607) [cxf-rt-transports-http-3.1.6.jar:3.1.6]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1551) [cxf-rt-transports-http-3.1.6.jar:3.1.6]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348) [cxf-rt-transports-http-3.1.6.jar:3.1.6]
at org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:56) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:216) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651) [cxf-rt-transports-http-3.1.6.jar:3.1.6]
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) [cxf-core-3.1.6.jar:3.1.6]
... 84 more
2020-02-27 08:26:09,621 ERROR [100] [org.jboss.as.ejb3.invocation] (my-thread-1) JBAS014134: EJB invocation failed on DaoFacade component for method public abstract void com.example.addon.core.dao.facade.DaoFacadeInterface.invokeExternalService(com.example.db.models.Synchronizable) throws com.example.addon.addon.SOAPException: javax.ejb.EJBException: javax.xml.ws.WebServiceException: Could not send Message.
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_79]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_79]
Caused by: javax.xml.ws.WebServiceException: Could not send Message.
at org.apache.cxf.jaxws.DispatchImpl.mapException(DispatchImpl.java:272) [cxf-rt-frontend-jaxws-3.1.6.jar:3.1.6]
at org.apache.cxf.jaxws.DispatchImpl.invoke(DispatchImpl.java:334) [cxf-rt-frontend-jaxws-3.1.6.jar:3.1.6]
at org.apache.cxf.jaxws.DispatchImpl.invoke(DispatchImpl.java:246) [cxf-rt-frontend-jaxws-3.1.6.jar:3.1.6]
...
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:280) [jboss-as-ejb3-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
... 40 more
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response '411: Length Required' when communicating with http://192.100.110.17:8504/example/services/CreationREQ
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1600) [cxf-rt-transports-http-3.1.6.jar:3.1.6]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1607) [cxf-rt-transports-http-3.1.6.jar:3.1.6]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1551) [cxf-rt-transports-http-3.1.6.jar:3.1.6]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348) [cxf-rt-transports-http-3.1.6.jar:3.1.6]
at org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:56) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:216) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651) [cxf-rt-transports-http-3.1.6.jar:3.1.6]
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:514) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:423) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:324) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:277) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.endpoint.ClientImpl.invokeWrapped(ClientImpl.java:312) [cxf-core-3.1.6.jar:3.1.6]
at org.apache.cxf.jaxws.DispatchImpl.invoke(DispatchImpl.java:327) [cxf-rt-frontend-jaxws-3.1.6.jar:3.1.6]
... 77 more
Here is my soap client code looks like:
import static java.lang.Boolean.FALSE;
import static java.lang.Boolean.TRUE;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.Reader;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.security.GeneralSecurityException;
import javax.xml.namespace.QName;
import javax.xml.soap.MessageFactory;
import javax.xml.soap.SOAPException;
import javax.xml.soap.SOAPMessage;
import javax.xml.soap.SOAPPart;
import javax.xml.transform.stream.StreamSource;
import javax.xml.ws.BindingProvider;
import javax.xml.ws.Dispatch;
import javax.xml.ws.Service;
import javax.xml.ws.soap.SOAPBinding;
import org.apache.cxf.endpoint.Client;
import org.apache.cxf.jaxws.DispatchImpl;
import org.apache.cxf.transport.http.HTTPConduit;
import org.apache.cxf.transports.http.configuration.HTTPClientPolicy;
import org.apache.log4j.Logger;
public class SOAPClient {
private static final String CXF_RESPONSE_CODE = "org.apache.cxf.message.Message.RESPONSE_CODE";
private static final String JAXRS_RESPONSE_CODE = "javax.xml.ws.http.response.code";
private static Logger logger = Logger.getLogger(SOAPClient.class);
/**
* Nested class only accessible after {#link SOAPClient#builder()} method call
* <p>
* It provides a Fluent Interface to make the client code more readable.
*/
public static class Builder {
private String endpoint;
private String namespace;
private String serviceName;
private Boolean basicAuthentication = FALSE;
private String username;
private String password;
private Long connectionTimeout;
private Long receiveTimeout;
private String portName;
private String soapAction;
private StreamSource soapRequest;
private OutputStream soapResponse;
private Dispatch<SOAPMessage> dispatch;
private SOAPMessage requestSOAPMessage;
private SOAPMessage returnedSOAPMessage;
public Builder endpoint(String endpoint) {
this.endpoint = endpoint;
return this;
}
public Builder namespace(String namespace) {
this.namespace = namespace;
return this;
}
public Builder serviceName(String serviceName) {
this.serviceName = serviceName;
return this;
}
public Builder portName(String portName) {
this.portName = portName;
return this;
}
public Builder soapAction(String soapAction) {
this.soapAction = soapAction;
return this;
}
public Builder basicAuthentication(Boolean basicAuthentication) {
this.basicAuthentication = basicAuthentication;
return this;
}
public Builder username(String username) {
this.username = username;
return this;
}
public Builder password(String password) {
this.password = password;
return this;
}
public Builder connectionTimeout(Long connectionTimeout) {
this.connectionTimeout = connectionTimeout;
return this;
}
public Builder receiveTimeout(Long receiveTimeout) {
this.receiveTimeout = receiveTimeout;
return this;
}
public Builder soapRequest(File soapRequest) throws IOException {
if (soapRequest == null)
throw new IllegalStateException("soapRequest not set");
InputStream is = Files.newInputStream(soapRequest.toPath());
this.soapRequest = new StreamSource(is, StandardCharsets.UTF_8.name());
return this;
}
public Builder soapRequest(InputStream soapRequest) {
this.soapRequest = new StreamSource(soapRequest);
return this;
}
public Builder soapRequest(Reader soapRequest) {
this.soapRequest = new StreamSource(soapRequest);
return this;
}
public int execute(File soapResponse) throws IOException, SOAPException, GeneralSecurityException {
if (soapResponse == null)
throw new IllegalStateException("soapResponse not set");
this.soapResponse = Files.newOutputStream(soapResponse.toPath());
return execute();
}
public int execute(OutputStream soapResponse) throws IOException, SOAPException, GeneralSecurityException {
if (soapResponse == null)
throw new IllegalStateException("soapResponse not set");
this.soapResponse = soapResponse;
return execute();
}
private int execute() throws SOAPException, IOException, GeneralSecurityException {
createDispatch();
addBasicAuthentication();
addSoapAction();
addTimeouts();
setSOAPMessage();
returnedSOAPMessage = dispatch.invoke(requestSOAPMessage);
returnedSOAPMessage.writeTo(soapResponse);
return Integer.parseInt(getResponseCode());
}
private String getResponseCode() {
Object responseCode = dispatch.getResponseContext().get(CXF_RESPONSE_CODE);
if (responseCode != null)
return responseCode.toString();
else {
responseCode = dispatch.getResponseContext().get(JAXRS_RESPONSE_CODE);
if (responseCode != null)
return responseCode.toString();
else
return "-1";
}
}
private void createDispatch() {
if (endpoint == null || endpoint.isEmpty())
throw new IllegalStateException("endpoint not set");
QName serviceQName = new QName(namespace, serviceName);
logger.debug("Creating the Service QName, " + serviceQName);
// Add a separate name space for method if required
QName portQName = new QName(namespace, portName);
logger.debug("Creating port QName, " + portQName);
Service serviceRef = Service.create(serviceQName);
serviceRef.addPort(portQName, SOAPBinding.SOAP11HTTP_BINDING, endpoint);
dispatch = serviceRef.createDispatch(portQName, SOAPMessage.class, Service.Mode.MESSAGE);
}
private void addBasicAuthentication() throws GeneralSecurityException {
if (basicAuthentication)
setBasicAuthentication();
}
private void setBasicAuthentication() throws GeneralSecurityException {
if (username == null || username.isEmpty())
throw new IllegalStateException("username not set with BasicAuthentication");
if (password == null || password.isEmpty())
throw new IllegalStateException("password not set with BasicAuthentication");
dispatch.getRequestContext().put(BindingProvider.USERNAME_PROPERTY, username);
dispatch.getRequestContext().put(BindingProvider.PASSWORD_PROPERTY, JcodificaLib.decrypt(password));
}
private void addSoapAction() {
if (soapAction != null)
setSoapAction();
}
private void setSoapAction() {
logger.debug("SoapAction:" + soapAction);
dispatch.getRequestContext().put(BindingProvider.SOAPACTION_USE_PROPERTY, TRUE);
dispatch.getRequestContext().put(BindingProvider.SOAPACTION_URI_PROPERTY, soapAction);
}
private void addTimeouts() {
if (receiveTimeout != null) {
setReceiveTimeout();
}
if (connectionTimeout != null) {
setConnectionTimeout();
}
}
private void setConnectionTimeout() {
logger.debug("connectionTimeout:" + connectionTimeout);
dispatch.getRequestContext().put("javax.xml.ws.client.connectionTimeout", connectionTimeout);
}
private void setReceiveTimeout() {
logger.debug("receiveTimeout:" + receiveTimeout);
dispatch.getRequestContext().put("javax.xml.ws.client.receiveTimeout", receiveTimeout);
}
private void setSOAPMessage() throws SOAPException {
MessageFactory messageFactory = MessageFactory.newInstance();
requestSOAPMessage = messageFactory.createMessage();
SOAPPart soapPart = requestSOAPMessage.getSOAPPart();
soapPart.setContent(soapRequest);
}
}
public static SOAPClient.Builder builder() {
return new SOAPClient.Builder();
}
}
I tried fixing the issue by enabling chunking and setting chunking threshold to 8192 as below, but I am getting the same error message:
private void setHttpClientPolicies() {
final Client client = ((DispatchImpl<SOAPMessage>) dispatch).getClient();
final HTTPConduit httpConduit = (HTTPConduit) client.getConduit();
final HTTPClientPolicy httpClientPolicy = httpConduit.getClient();
httpConduit.setClient(httpClientPolicy);
httpClientPolicy.setAllowChunking(TRUE);
httpClientPolicy.setChunkingThreshold(8192);
}
Any clue how to fix the issue ?

I suspect you may try to set the content length in the header. IE, the size of of your request using Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing.
I would guess that if you correctly identified the size of the request the server might not choke on it. Worth a try.
Content-Length: SIZE OF REQUIRED DATA IN BYTES \n\n
It may require accessing some lower level portions of your request framework.
References:
https://www.rfc-editor.org/rfc/rfc7230#section-3.3.2
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Length

I would try the contrary of what you did.
I suspect the server is not supporting chunking. Therefore, you should configure CXF to disable (and not enable!) chunking, like the doc suggests it:
If you are getting strange errors (generally not soap faults, but
other HTTP type errors) when trying to interact with a service, try
turning off chunking to see if that helps.

SOAP request by org.apache.http.client.HttpClient(4.1)
You have to specify data length.
req_xml.length()
// SOAP request(xml) read-in
File req_xml = new File("test/xml/request.xml");
// SOAP request send
HttpPost post = new HttpPost("http://localhost:8080/test/api/");
post.setEntity(new InputStreamEntity(new FileInputStream(req_xml), **req_xml.length()**));
post.setHeader("Content-type", "text/xml; charset=UTF-8");
post.setHeader("SOAPAction", "");
HttpClient client = new DefaultHttpClient();
HttpResponse response = client.execute(post);
// SOAP response(xml) get
String res_xml = EntityUtils.toString(response.getEntity());
Check that working code just supply xml and change web path.
You have whole builder code where I don't understand are you doing post/get.By Default everything is http get.
createDispatch();
addBasicAuthentication();
addSoapAction();
addTimeouts();
setSOAPMessage();
returnedSOAPMessage = dispatch.invoke(requestSOAPMessage);
returnedSOAPMessage.writeTo(soapResponse);
return Integer.parseInt(getResponseCode());

Answering my own question.
The answers posted here are pointing correctly that by disabling chunking or adding Content-Length the issue can be resolved. But my challenge was that I wasn't able to disable the chunking as it can be seen in my question description.
So I was trying to add Content-Length to http header and as you can see I'm not using any Apache library to write my soap client code so it was difficult to figure out how to add the Content-Length to the HTTP Header. Lastly I found a way to do so. The code snippet can be found below:
private void setSOAPMessage() throws SOAPException {
MessageFactory messageFactory = MessageFactory.newInstance();
requestSOAPMessage = messageFactory.createMessage();
SOAPPart soapPart = requestSOAPMessage.getSOAPPart();
soapPart.setContent(soapRequest);
// Added Content-Length to HTTP Header
Map<String, List<String>> requestHeaderMap = new HashMap<String, List<String>>();
requestHeaderMap.put("Content-Length", Collections.singletonList(String.valueOf(soapRequestXMLSize)));
dispatch.getRequestContext().put(MessageContext.HTTP_REQUEST_HEADERS, requestHeaderMap);
}
Here the soapRequestXMLSize was calculated from the actual soap request xml which is stored as String in my case.
Integer soapRequestSize = soapRequestXML.length();

Retrieve a list of a given user's tweets using Twitter API 1.1 and Retrofit

I'm trying to obtain a list of a user's tweets and I've run into some trouble when trying to authenticate my call to the API. I currently get a 401 when executing the code below:
public interface TwitterApi {
String API_URL = "https://api.twitter.com/1.1";
String CONSUMER_KEY = "<CONSUMER KEY GOES HERE>";
String CONSUMER_SECRET = "<CONSUMER SECRET GOES HERE>";
String ACCESS_TOKEN = "<ACCESS TOKEN GOES HERE>";
String ACCESS_TOKEN_SECRET = "<ACCESS TOKEN SECRET GOES HERE>";
#GET("/statuses/user_timeline.json")
List<Tweet> fetchUserTimeline(
#Query("count") final int count,
#Query("screen_name") final String screenName);
}
The following throws a 401 Authorisation error when calling fetchUserTimeline()
RetrofitHttpOAuthConsumer consumer = new RetrofitHttpOAuthConsumer(TwitterApi.CONSUMER_KEY, TwitterApi.CONSUMER_SECRET);
consumer.setTokenWithSecret(TwitterApi.ACCESS_TOKEN, TwitterApi.ACCESS_TOKEN_SECRET);
RestAdapter restAdapter = new RestAdapter.Builder()
.setEndpoint(TwitterApi.API_URL)
.setClient(new SigningOkClient(consumer))
.build();
TwitterApi twitterApi = restAdapter.create(TwitterApi.class)
tweets = twitterApi.fetchUserTimeline(2, screenName);
I've also included the relevant code from the signpost-retrofit plugin:
public class SigningOkClient extends OkClient {
private final RetrofitHttpOAuthConsumer mOAuthConsumer;
public SigningOkClient(RetrofitHttpOAuthConsumer consumer) {
mOAuthConsumer = consumer;
}
public SigningOkClient(OkHttpClient client, RetrofitHttpOAuthConsumer consumer) {
super(client);
mOAuthConsumer = consumer;
}
#Override
public Response execute(Request request) throws IOException {
Request requestToSend = request;
try {
HttpRequestAdapter signedAdapter = (HttpRequestAdapter) mOAuthConsumer.sign(request);
requestToSend = (Request) signedAdapter.unwrap();
} catch (OAuthMessageSignerException | OAuthExpectationFailedException | OAuthCommunicationException e) {
// Fail to sign, ignore
e.printStackTrace();
}
return super.execute(requestToSend);
}
}
The signpost-retrofit plugin can be found here: https://github.com/pakerfeldt/signpost-retrofit
public class RetrofitHttpOAuthConsumer extends AbstractOAuthConsumer {
private static final long serialVersionUID = 1L;
public RetrofitHttpOAuthConsumer(String consumerKey, String consumerSecret) {
super(consumerKey, consumerSecret);
}
#Override
protected HttpRequest wrap(Object request) {
if (!(request instanceof retrofit.client.Request)) {
throw new IllegalArgumentException("This consumer expects requests of type " + retrofit.client.Request.class.getCanonicalName());
}
return new HttpRequestAdapter((Request) request);
}
}
Any help here would be great. The solution doesn't have to include the use of signpost but I do want to use Retrofit. I also do not want to show the user an 'Authenticate with Twitter' screen in a WebView - I simply want to display a handful of relevant tweets as part of a detail view.

Are you certain the signpost-retrofit project works for twitter oauth? I've used twitter4j successfully in the past - and if you don't want the full library you can use their code for reference. twitter4j