I want to index a large batch of index requests to 60 different indices(about 1000 million pieces,indices' name are like boss-log-yyyy-MM-dd).
Here is my java code:
List<IndexRequest> indexRequesList = bossMockDataService.indexRequestGenerator(batch); //generate random mock data.
indexRequesList.forEach(indexRequest -> {
bulkProcessor.add(indexRequest);
});
try {
return bulkProcessor.awaitClose(30L, TimeUnit.SECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
return false;
}
And here is my bulk processor:
#Override
public BulkProcessor initESbulkProcessor(RestHighLevelClient client) {
Settings settings = Settings.EMPTY;
ThreadPool threadPool = new ThreadPool(settings); // I have no Idea what the 'settings' object is.
BulkProcessor.Listener listener = new BulkProcessor.Listener() {
#Override
public void beforeBulk(long executionId, BulkRequest request) {
int numberOfActions = request.numberOfActions();
logger.debug("Executing bulk [{}] with {} requests", executionId, numberOfActions);
}
#Override
public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
if (response.hasFailures()) {
logger.warn("Bulk [{}] executed with failures", executionId);
} else {
logger.debug("Bulk [{}] completed in {} milliseconds", executionId, response.getTook().getMillis());
}
}
#Override
public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
logger.error("Failed to execute bulk", failure);
}
};
BulkProcessor.Builder builder = new BulkProcessor.Builder(client::bulkAsync, listener, threadPool);
BulkProcessor bulkProcessor = builder.build();
builder.setBulkActions(2000);
builder.setBulkSize(new ByteSizeValue(10L, ByteSizeUnit.MB));
builder.setConcurrentRequests(10);
builder.setFlushInterval(TimeValue.timeValueSeconds(10L));
builder.setBackoffPolicy(BackoffPolicy.constantBackoff(TimeValue.timeValueSeconds(1L), 3));
logger.info("bulk processor build complete!");
return bulkProcessor;
}
It seems some data index request failed.The log is like:
2017-11-15 16:42:24 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [96] executed with failures
2017-11-15 16:42:24 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [97] executed with failures
2017-11-15 16:42:24 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [98] executed with failures
2017-11-15 16:42:24 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [99] executed with failures
2017-11-15 16:42:24 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [100] executed with failures
2017-11-15 16:42:24 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [101] executed with failures
2017-11-15 16:42:24 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [102] executed with failures
2017-11-15 16:42:24 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [103] executed with failures
2017-11-15 16:42:25 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [104] executed with failures
2017-11-15 16:42:25 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [105] executed with failures
2017-11-15 16:42:25 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [106] executed with failures
2017-11-15 16:42:25 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [107] executed with failures
2017-11-15 16:42:25 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [108] executed with failures
2017-11-15 16:42:25 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [109] executed with failures
2017-11-15 16:42:25 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [110] executed with failures
2017-11-15 16:42:26 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [111] executed with failures
2017-11-15 16:42:26 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [112] executed with failures
2017-11-15 16:42:26 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [113] executed with failures
2017-11-15 16:42:26 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [114] executed with failures
2017-11-15 16:42:26 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [115] executed with failures
2017-11-15 16:42:26 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [116] executed with failures
2017-11-15 16:42:26 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [117] executed with failures
2017-11-15 16:42:27 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [118] executed with failures
2017-11-15 16:42:27 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [119] executed with failures
2017-11-15 16:42:27 [I/O dispatcher 1] WARN BossMockDataServiceImpl:220 - Bulk [120] executed with failures
My situation:
New to ES.
1 million pieces of data per day need to be stored and queried.
Yet I have not configured my shard/replica/node/cluster,they are default now.
When I use MySQL,I create a new partition for a day's data.
The limit for MySQL is 60 days.The IO operation is getting more and more slow.
My goal:
Easy to maintain my ES.
Fast query,update and aggregation.
Store 60+ days data.
No tolerance for bulk request failure.(Seems some data lost above)
What I wonder is :
Is it a good idea to create new index for a day's data ? Or I put them in one index?
Why the bulk processor call back is failure ? How to fix it ?
Is it necessary to create more cluster/node ? Where should I begin?
Related
I am new to reactive programming and I want to implement spring-webclient on our existing project.
For simplicity, I created a pseudo code based from my original code. Here I am sending SMS to each provider. If condition is met then it will not proceed to next provider.
public List<Sms> send(List<Sms> smsRequests) {
return Flux.fromIterable(smsRequests)
.flatMap(smsRequest -> {
return Flux.fromIterable(smsRequest.getProviders())
.concatMap(smsProvider -> this.invokeApiUsingWebClient(smsProvider, smsRequest))
.filter(providerResponse -> providerResponse.getErrorStatus() == null)
.map(ProviderResponse::toSms)
.next(); // emits cancel(). Resulting to connection being closed.
})
.toStream()
.collect(Collectors.toList());
}
My problem is that this flow calls cancel() every time next() is invoke. Resulting to poor performance by not reusing the connection on the webclient threads.
See below logs.
TRACE 15112 [reactor-http-nio-1] [ExchangeFunctions] - [5e2f49c] Response 200 OK...
INFO 15112 [reactor-http-nio-1] [1] - onNext(...)
INFO 15112 [reactor-http-nio-1] [1] - cancel()
DEBUG 15112 [reactor-http-nio-1] [ExchangeFunctions] - [5e2f49c] Cancel signal (to close connection)
INFO 15112 [reactor-http-nio-1] [1] - onComplete()
TRACE 15112 [reactor-http-nio-2] [ExchangeFunctions] - [5e2f49c] Response 200 OK...
INFO 15112 [reactor-http-nio-2] [2] - onNext(...)
INFO 15112 [reactor-http-nio-2] [2] - cancel()
DEBUG 15112 [reactor-http-nio-2] [ExchangeFunctions] - [5e2f49c] Cancel signal (to close connection)
INFO 15112 [reactor-http-nio-2] [2] - onComplete()
Is there a possible way to refactor above code without emitting the cancel() method?
I want to use CloseableHttpAsyncClient to send a http request in Async mode. But the request is not send out. When enable comment HttpResponse response = future.get(). It works. But I'd like to know why I need future.get() even I didn't care about the response.
Code is here
public class CloseableHttpAsyncClientTest {
#Test
public void whenUseHttpAsyncClient_thenCorrect() throws Exception {
CloseableHttpAsyncClient client = HttpAsyncClients.createDefault();
client.start();
HttpDelete request = new HttpDelete("http://www.bing.com");
Future<HttpResponse> future = client.execute(request, null);
// The delete request will send out if we remove comment here. We just want to send out delete http
// request but not care about the response
// HttpResponse response = future.get();
client.close();
}
}
The console like this
20:28:48.930 [main] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 1] start execution
20:28:48.941 [main] DEBUG org.apache.http.client.protocol.RequestAddCookies - CookieSpec selected: default
20:28:48.950 [main] DEBUG org.apache.http.client.protocol.RequestAuthCache - Auth cache not set in the context
20:28:48.951 [main] DEBUG org.apache.http.impl.nio.client.InternalHttpAsyncClient - [exchange: 1] Request connection for {}->http://www.bing.com:80
20:28:48.953 [main] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection request: [route: {}->http://www.bing.com:80][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
20:28:48.976 [main] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection manager is shutting down
20:28:49.003 [main] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection manager shut down
After I enable the comments. Like this. It works.
public class CloseableHttpAsyncClientTest {
#Test
public void whenUseHttpAsyncClient_thenCorrect() throws Exception {
CloseableHttpAsyncClient client = HttpAsyncClients.createDefault();
client.start();
HttpDelete request = new HttpDelete("http://www.bing.com");
Future<HttpResponse> future = client.execute(request, null);
// The delete request will send out if we remove comment here. We just want to send out delete http
// request but not care about the response
HttpResponse response = future.get();
client.close();
}
}
From the log, we can see. The http delete has been send out.
20:39:15.998 [main] DEBUG org.apache.http.impl.nio.client.MainClientExec - [exchange: 1] start execution
......
......
20:39:16.137 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> DELETE / HTTP/1.1
20:39:16.137 [I/O dispatcher 1] DEBUG org.apache.http.headers - http-outgoing-0 >> Host: www.bing.com
......
......
20:39:16.142 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "DELETE / HTTP/1.1[\r][\n]"
20:39:16.142 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Host: www.bing.com[\r][\n]"
20:39:16.142 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
20:39:16.142 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "User-Agent: Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_202)[\r][\n]"
20:39:16.143 [I/O dispatcher 1] DEBUG org.apache.http.wire - http-outgoing-0 >> "[\r][\n]"
20:39:16.143 [I/O dispatcher 1] DEBUG org.apache.http.impl.nio.client.InternalIODispatch - http-outgoing-0 [ACTIVE] Request ready
20:39:16.143 [I/O dispatcher 1] DEBUG
......
20:39:16.209 [main] DEBUG org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager - Connection manager shut down
Future<HttpResponse> future = client.execute(request, null);
client.execute will send out http request in Async way. It will take another thread to complete the http request.
HttpResponse response = future.get();
future.get() will block the main thread until client.execute complete and fill the response into future. Without this, the client will be close before http request send out. Cos http request send out by one thread, but client close be closed by main thread. So, main thread need be blocked until future.get() get the result by another thread.
My AWS Java Client is throwing
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541) ~[na:1.8.0_60]
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553) ~[na:1.8.0_60]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71) ~[na:1.8.0_60]
at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:159) ~[httpcore-4.3.3.jar:4.3.3]
My code is
public void save(String name, byte[] file) {
ObjectMetadata metaData = new ObjectMetadata();
String streamMD5 = new String(Base64.encodeBase64(file));
metaData.setContentMD5(streamMD5);
metaData.setContentLength(file.length);
InputStream stream = new ByteArrayInputStream(file);
try {
PutObjectRequest put = new PutObjectRequest(
configuration.getBucketName(), name, stream, metaData);
s3client.putObject(put);
} finally {
IOUtils.closeQuietly(stream);
}
}
The s3client is a spring bean and is not garbage collected before the stream has finished uploading. I've tried without specifying MD5 and/or content length but still has the same exception thrown.
Logging through AWS library shows:
10:09:15.540 [http-nio-8080-exec-1] DEBUG o.a.h.c.protocol.RequestAddCookies - CookieSpec selected: best-match
10:09:15.540 [http-nio-8080-exec-1] DEBUG o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
10:09:15.540 [http-nio-8080-exec-1] DEBUG o.a.h.c.p.RequestProxyAuthentication - Proxy auth state: UNCHALLENGED
10:09:15.540 [http-nio-8080-exec-1] DEBUG c.a.http.impl.client.SdkHttpClient - Attempt 1 to execute request
10:09:15.540 [http-nio-8080-exec-1] DEBUG o.a.h.i.conn.DefaultClientConnection - Sending request: PUT /documeent.pdf HTTP/1.1
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "PUT /document.pdf HTTP/1.1[\r][\n]"
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "Host: bucket.s3.amazonaws.com[\r][\n]"
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "Authorization: AWS 123445667788=[\r][\n]"
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "User-Agent: aws-sdk-java/1.10.21 Linux/3.13.0-65-generic Java_HotSpot(TM)_Server_VM/25.60-b23/1.8.0_60[\r][\n]"
10:09:15.540 [http-nio-8080-exec-1] DEBUG org.apache.http.wire - >> "Date: Tue, 06 Oct 2015 09:09:15 GMT[\r][\n]"
10:09:15.663 [http-nio-8080-exec-1] DEBUG com.amazonaws.internal.SdkSSLSocket - closing bucket.s3.amazonaws.com/12.34.56.78:443
10:09:15.665 [http-nio-8080-exec-1] DEBUG o.a.h.i.conn.DefaultClientConnection - I/O error closing connection
I've checked that the file size (3.2M) does not exceed the max file siez for this bucket.
Get/List requests work fine and I can copy files into the S3 bucket using the s3 client tools.
Does anyone know of anything else I should check?
Thanks.
This is the Logback stacktrace of an exception:
15:32:04.773 [localhost:9080] [UNKNOWN] [Unauthenticated] [0AE48E3EEF47ED3210CC45C3E20D273D] DEBUG org.apache.http.wire - http-outgoing-4 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
15:32:04.773 [localhost:9080] [UNKNOWN] [Unauthenticated] [0AE48E3EEF47ED3210CC45C3E20D273D] DEBUG org.apache.http.wire - http-outgoing-4 << "[\r][\n]"
15:32:04.773 [localhost:9080] [UNKNOWN] [Unauthenticated] [0AE48E3EEF47ED3210CC45C3E20D273D] DEBUG org.apache.http.wire - http-outgoing-4 << "10e[\r][\n]"
15:32:04.773 [localhost:9080] [UNKNOWN] [Unauthenticated] [0AE48E3EEF47ED3210CC45C3E20D273D] DEBUG org.apache.http.wire - http-outgoing-4 << "<env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope"><env:Body><env:Fault><env:Code><env:Value>env:Sender</env:Value></env:Code><env:Reason><env:Text xml:lang="en-US">Internal Error (from client)</env:Text></env:Reason></env:Fault></env:Body></env:Envelope>[\r][\n]"
15:32:04.773 [localhost:9080] [UNKNOWN] [Unauthenticated] [0AE48E3EEF47ED3210CC45C3E20D273D] DEBUG org.apache.http.wire - http-outgoing-4 << "0[\r][\n]"
My question is: How to stop Logback logging exceptions with a time prefix on each line? Is it a xml change or a change to the Java code?
EDIT: Clarification
I'm happy with the time prefix in general. The issue here is that some information is coming back and it is being split across multiple lines, and each line is prefixed with the time, which makes interpreting the original information even harder. What I want is the time on the first line, and the rest of the stack trace on the remaining lines.
This is the error I'm trying to see:
<env:Text xml:lang="en-US">Internal Error (from client)</env:Text>
I use xfire to develop a web-service client in my Spring 3.x project, here are my code below
WebServiceClient wClient = new WebServiceClient();
WebServiceSoap soap = wClient.getWebServiceSoap(Commons.VD_INFO_WSDL);
com.webservice.object.GetVDInfo input = new com.webservice.object.GetVDInfo();
input.setPASSWORD(Commons.VD_INFO_PASSWORD);
com.webservice.object.GetVDInfoResponse response = soap.getVDInfo(input);
System.out.println(response==null ? "response=null" : "response not null,");
When I run these on the server, I can see the wsdl was get on my Eclipse console, but the "response" object is null....
Here is the final part log below, Can anyone give me a hint or suggestion?
If there is not enough information, plz let me know....
thx so much...
DEBUG: httpclient.wire.content - << " <wsdl:service name="ToServiceImplService">[\r][\n]"
DEBUG: httpclient.wire.content - << "[\n]"
DEBUG: httpclient.wire.content - << " <wsdl:port binding="impl:ToServiceImplSoapBinding" name="ToServiceImpl">[\r][\n]"
DEBUG: httpclient.wire.content - << "[\n]"
DEBUG: httpclient.wire.content - << " <wsdlsoap:address location="http://xx.xx.10.77/TO_SERVICE/services/ToServiceImpl"/>[\r][\n]"
DEBUG: httpclient.wire.content - << "[\n]"
DEBUG: httpclient.wire.content - << " </wsdl:port>[\r][\n]"
DEBUG: httpclient.wire.content - << "[\n]"
DEBUG: httpclient.wire.content - << " </wsdl:service>[\r][\n]"
DEBUG: httpclient.wire.content - << "[\n]"
DEBUG: httpclient.wire.content - << "</wsdl:definitions>[\r][\n]"
DEBUG: org.apache.commons.httpclient.HttpMethodBase - Resorting to protocol version default close connection policy
DEBUG: org.apache.commons.httpclient.HttpMethodBase - Should NOT close connection, using HTTP/1.1
DEBUG: org.apache.commons.httpclient.HttpConnection - Releasing connection back to connection manager.
DEBUG: org.apache.commons.httpclient.MultiThreadedHttpConnectionManager - Freeing connection, hostConfig=HostConfiguration[host=http://61.60.10.77]
DEBUG: org.apache.commons.httpclient.util.IdleConnectionHandler - Adding connection at: 1404713700948
DEBUG: org.apache.commons.httpclient.MultiThreadedHttpConnectionManager - Notifying no-one, there are no waiting threads
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - Invoking phase pre-dispatch
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - Invoking handler org.codehaus.xfire.client.CorrelatorHandler in phase pre-dispatch
DEBUG: org.codehaus.xfire.client.Client - Correlating context with ID 14047137002771-1400602077
DEBUG: org.codehaus.xfire.client.Client - Found correlated context with ID 14047137002771-1400602077
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - adding handler org.codehaus.xfire.client.ClientReceiveHandler#37e80c87 to phase service
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - Invoking phase dispatch
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - Invoking handler org.codehaus.xfire.handler.LocateBindingHandler in phase dispatch
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - Invoking handler org.codehaus.xfire.soap.handler.SoapBodyHandler in phase dispatch
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - Invoking handler org.codehaus.xfire.soap.handler.SoapActionInHandler in phase dispatch
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - Invoking phase policy
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - Invoking phase user
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - Invoking phase pre-invoke
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - Invoking handler org.codehaus.xfire.soap.handler.ValidateHeadersHandler in phase pre-invoke
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - Invoking phase service
DEBUG: org.codehaus.xfire.handler.HandlerPipeline - Invoking handler org.codehaus.xfire.client.ClientReceiveHandler in phase service
DEBUG: org.codehaus.xfire.client.XFireProxy - Result [null]
response=null