Elasticsearch CouchDB river bogus chunk size - java
Sorry, this is a kind of related to another post. Once my CouchDB gets a fair amount of documents in it, ES starts throwing errors in the log and doesn't index newer files:
[2013-08-19 17:55:08,379][WARN ][river.couchdb ] [Morning Star] [couchdb][portal_production] failed to read from _changes, throttling....
java.io.IOException: Bogus chunk size
at sun.net.www.http.ChunkedInputStream.processRaw(ChunkedInputStream.java:319)
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:572)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:609)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:696)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3052)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:154)
at java.io.BufferedReader.readLine(BufferedReader.java:317)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at org.elasticsearch.river.couchdb.CouchdbRiver$Slurper.run(CouchdbRiver.java:477)
at java.lang.Thread.run(Thread.java:724)
[2013-08-19 17:55:13,392][WARN ][river.couchdb ] [Morning Star] [couchdb][portal_production] failed to read from _changes, throttling....
What's the dealio?
Edit - river status
$ curl http://localhost:9200/_river/portal_production/_status?pretty=true
{
"_index" : "_river",
"_type" : "portal_production",
"_id" : "_status",
"_version" : 2,
"exists" : true, "_source" : {"ok":true,"node":{"id":"EVxlLNZ9SrSXYOLS0YBw7w","name":"Shadow Slasher","transport_address":"inet[/192.168.1.106:9300]"}}
}
Edit - River sequence data
Seems pretty low!
curl -X GET http://localhost:9200/_river/portal_production/_seq?pretty=true
{
"_index" : "_river",
"_type" : "portal_production",
"_id" : "_seq",
"_version" : 1,
"exists" : true, "_source" : {"couchdb":{"last_seq":"4"}}
}
My _changes is much bigger, by the way:
curl -X GET http://localhost:5984/portal_production/_changes?limit=5
{"results":[
{"seq":4,"id":"Ifilter-1","changes":[{"rev":"4-d9c8e771bc345d1182fbe7c2d63f5d00"}]},
{"seq":7,"id":"Document-2","changes":[{"rev":"1-42f52115c4a5321328be07c490932b61"}]},
{"seq":10,"id":"Document-4","changes":[{"rev":"1-42f52115c4a5321328be07c490932b61"}]},
{"seq":13,"id":"Document-6","changes":[{"rev":"1-42f52115c4a5321328be07c490932b61"}]},
{"seq":16,"id":"Document-8","changes":[{"rev":"1-42f52115c4a5321328be07c490932b61"}]},
...
{"seq":208657,"id":"Document-11295","changes":[{"rev":"8-37cb48660d28bef854b2c31132bc9635"}]},
{"seq":208661,"id":"Document-11297","changes":[{"rev":"6-daf5c5d557d0fa30b2b08be26582a33c"}]},
{"seq":208665,"id":"Document-11299","changes":[{"rev":"6-22e57345c2ee5c7aee8b7d664606b874"}]},
{"seq":208669,"id":"Document-11301","changes":[{"rev":"6-06deee0c3c6705238a8b07e400b2414b"}]},
{"seq":208673,"id":"Document-11303","changes":[{"rev":"6-86fc60dd8c1d415d42a25a23eb975121"}]},
{"seq":208677,"id":"Document-11305","changes":[{"rev":"6-6d51a577fdc9013abf64ec4ffbf9eeee"}]},
{"seq":208683,"id":"Document-11307","changes":[{"rev":"6-726a7835ce390094b9b9e0a91aeb11f0"}]},
{"seq":208684,"id":"Document-11286","changes":[{"rev":"9-747e63e0304a974cc7db7ff84ae80697"}]}
],
"last_seq":208684}
Edit - Couchdb log
This seems bad:
[Thu, 22 Aug 2013 02:49:37 GMT] [info] [<0.340.0>] 127.0.0.1 - - 'GET' /portal_production/_changes?feed=continuous&include_docs=true&heartbeat=10000&since=4 500
[Thu, 22 Aug 2013 02:49:42 GMT] [info] [<0.348.0>] 127.0.0.1 - - 'GET' /portal_production/_changes?feed=continuous&include_docs=true&heartbeat=10000&since=4 200
[Thu, 22 Aug 2013 02:49:42 GMT] [error] [<0.348.0>] Uncaught error in HTTP request: {exit,{ucs,{bad_utf8_character_code}}}
[Thu, 22 Aug 2013 02:49:42 GMT] [info] [<0.348.0>] Stacktrace: [{xmerl_ucs,from_utf8,1},
{mochijson2,json_encode_string,2},
{mochijson2,'-json_encode_proplist/2-fun-0-',3},
{lists,foldl,3},
{mochijson2,json_encode_proplist,2},
{mochijson2,'-json_encode_proplist/2-fun-0-',3},
{lists,foldl,3},
{mochijson2,json_encode_proplist,2}]
So I kept deleting documents one by one, and retrying the _changes query with include_doc=true. But I never got to the bottom of it. Reading through some other related issues, imports done on text from Microsoft documents can have some of the funky characters. We're doing something like that, so we dumped the database, and are filtering non-UTF8 characters out. A little painful, but we had too many documents to find where the issue was. So far, no errors on the Elasticsearch side (well, some timeouts, but that is probably another thread).
Are you indexing office documents?
You could use the attachment plugin.
I have a branch not already merged that index couchdb attachments. If you want to test it, I will be happy to get feedback!
Related
Cannot publish json message on Kafka topic using ZeroCode
I am trying to create a test framework using ZeroCode for Kafka. The product I am trying to test is based on micro-services and Kafka. All I am trying to do is to connect to my topic and publish a message to it, at the moment. But when I run the test case I get an error saying 'Exception during operation:produce
Stacktrace
-------------------------- BDD: Scenario:Produce a message to kafka topic - vanilla -------------------------
27 Mar 2020 10:43:21,531 INFO [main] runner.ZeroCodeMultiStepsScenarioRunnerImpl |
### Executing Scenario -->> Count No: 0
27 Mar 2020 10:43:21,531 INFO [main] runner.ZeroCodeMultiStepsScenarioRunnerImpl |
### Executing Step -->> Count No: 0
---------------------------------------------------------
kafka.bootstrap.servers - <myKafkaBootstrapServer>
---------------------------------------------------------
27 Mar 2020 10:43:21,681 INFO [main] client.BasicKafkaClient | <myKafkaBootstrapServer>, topicName:executions.enriched, operation:produce, requestJson:{"recordType":"JSON","records":[{"value":"EquityExecution"}]}
27 Mar 2020 10:43:21,683 ERROR [main] client.BasicKafkaClient | Exception during operation:produce, topicName:executions.enriched, error:null
java.lang.RuntimeException: java.lang.NullPointerException
at org.jsmart.zerocode.core.kafka.client.BasicKafkaClient.execute(BasicKafkaClient.java:50)
at org.jsmart.zerocode.core.engine.executor.JsonServiceExecutorImpl.executeKafkaService(JsonServiceExecutorImpl.java:102)
at org.jsmart.zerocode.core.runner.ZeroCodeMultiStepsScenarioRunnerImpl.runScenario(ZeroCodeMultiStepsScenarioRunnerImpl.java:190)
at org.jsmart.zerocode.core.runner.ZeroCodeUnitRunner.runLeafJsonTest(ZeroCodeUnitRunner.java:198)
I am using .properties file to give broker and SSL credentials. Then send a test JSON. If publishing is successful then I plan to consume from a certain topic and assert on the values - thereby performing an integration test on the service.
Please help me resolving this as I cannot find any meaningful information online as to how to fix this. Much appreciated!
My .properties file look something like this:
security.properties=SSL
ssl.keystore.password=<myPassword>
ssl.keystore.location=<myLocation>
kafka.bootstrap.servers=<myServer>
My JSON file (Test Scenario, null key is a valid input to my topic) looks something like this:
{
"scenarioName": "Produce a message to kafka topic - vanilla",
"steps": [
{
"name": "produce_step",
"url": "kafka-topic:my.topic",
"operation": "produce",
"request": {
"records":[
{
"value": "My test value"
}
]
},
"assertions": {
"status" : "Ok",
}
}
]
}
Everything looking ok except the below. You should fix these first, then it will work fine.
kafka.bootstrap.servers=<myServer> should go into the Kafka broker properties which your #TargetEnv("kafka_servers/kafka_test_server.properties") is pointing to.
The producer.properties should not have kafka.bootstrap.servers=... redundant entry.
And properties file as below:
kafka_test_server.properties
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
# kafka bootstrap servers comma separated
# e.g. localhost:9092,host2:9093
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
kafka.bootstrap.servers=localhost:9092
kafka.producer.properties=producer.properties
kafka.consumer.properties=consumer.properties
That's it.
#TargetEnv("kafka_servers/kafka_test_server.properties")
#RunWith(ZeroCodeUnitRunner.class)
public class KafkaProduceTest {
#Test
#JsonTestCase("kafka/produce/test_kafka_produce.json")
public void testProduce() throws Exception {
}
}
There is a working example KafkaProduceTest in GitHub Kafka HelloWorld project you can clone and run locally.
Sunspot Solr on Ruby: "Exception writing document id to the index, java.lang.OutOfMemoryError"
We have a Ruby on Rails application running for some time now, but since a couple of days we're just getting a HTTP500 Error while trying to update any documents. The system is running Ruby 1.9.3 with unicorn 4.8.2 and nginx 1.2.1 and a sunspot_solr 2.1.1 gem for the search index (besides some others).
The Production log tells me that every time someone tries to create or update a document the solr server throws a bunch of errors. The most prominent I think is this one:
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:164)\n\t... 30 more\nCaused by: java.lang.OutOfMemoryError: Java heap space\n\
Unfortunately nothing changes if i edit config/sunstpot.yml to include "max_memory: 1G":
production:
solr:
hostname: localhost
port: 8080
log_level: WARNING
min_memory: 32M
max_memory: 1G
path: /solr-4.10.2/default #production #ollection1 #production
Here is the complete production.log entry:
[a086bc878daa0abd367379daade96682] Completed 500 Internal Server Error in 195ms
[a086bc878daa0abd367379daade96682]
RSolr::Error::Http (RSolr::Error::Http - 500 Internal Server Error
Error: {"responseHeader":{"status":500,"QTime":2},"error":{"msg":"Exception writing document id Neuigkeit 393 to the index; possible analysis error.","trace":"org.apache.solr.common.SolrException: Exception writing document id Neuigkeit 393 to the index; possible analysis error.\n\
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:168)\n\
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)\n\tat
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:926)\n\tat
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1080)\n\tat
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:692)\n\tat
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)\n\
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:247)\n\tat
org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174)\n\
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:99)\n\tat
org.apache.solr.haa086bc878daa0abd367379daade96682] Completed 500 Internal Server Error in 195ms
[a086bc878daa0abd367379daade96682]
RSolr::Error::Http (RSolr::Error::Http - 500 Internal Server Error
Error: {"responseHeader":{"status":500,"QTime":2},"error":{"msg":"Exception writing document id Neuigkeit 393 to the index; possible analysis error.","trace":"org.apache.solr.common.SolrException: Exception writing document indler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)\n\
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\
org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)\n\
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)\n\
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)\n\
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)\n\
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)\n\
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)\n\
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)\n\
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)\n\
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)\n\
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)\n\
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)\n\
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)\n\
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)\n\
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1003)\n\
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)\n\
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)\n\
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\
java.lang.Thread.run(Thread.java:745)\nCaused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed\n\
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:698)\n\
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:712)\n\
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1507)\n\
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:240)\n\
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:164)\n\t... 30 more\nCaused by: java.lang.OutOfMemoryError: Java heap space\n\
org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.<init>(FreqProxTermsWriterPerField.java:207)\n\
org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.newInstance(FreqProxTermsWriterPerField.java:230)\n\
org.apache.lucene.index.ParallelPostingsArray.grow(ParallelPostingsArray.java:48)\n\
org.apache.lucene.index.TermsHashPerField$PostingsBytesStartArray.grow(TermsHashPerField.java:252)\n\
org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:292)\n\
org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:151)\n\
org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:659)\n\
org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:359)\n\
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:318)\n\
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)\n\
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:454)\n\
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1511)\n\
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:240)\n\
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:164)\n\
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)\n\
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)\n\
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:926)\n\
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1080)\n\
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:692)\n\
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)\n\
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:247)\n\
org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174)\n\
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:99)\n\
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)\n\
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\
org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)\n\
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)\n\
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)\n\
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)\n\
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)\n\
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)\n\
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)\n","code":500}}
URI: http://localhost:8080/solr-4.10.2/default/update?wt=json
EDIT: I'm an idiot. The solr server isn't contained in the unicorn service. Restarting it via 'service tomcat7 restart' loaded the updated config/sunstpot.yml and solved the problem.
MongoDB MongoException$Network
We seem to be out of ideas on how to continue troubleshooting this issue. Suddenly we see the exception listed below being hit every few minutes.
This is a 4 Mongo Shard Setup. The 4 MongoS servers are proxied using 3 HAProxies. We aren't having any visible network issues between our application and shards. The mongo logs for the shards, config servers or mongos don't show anything out of the ordinary. The haproxies seem to be doing their job just fine. Any thoughts and leads on this would be most appreciated!
{"#timestamp":"2016-02-10T05:10:23.780+00:00","#version":1,"message":"unable to process event [ Request Id: [ eb8c702c-b3e7-4605-99a7-c3dcb5a076a9 ] - Event: id [ 49e0ae8d-f16e-448c-9b28-a4af59aa2eb0 ] messageType [ bulk ] operationType: [ create ] xpoint [ 10254910941235296908401 ] ]","logger_name":"com.xyz.event.RabbitMQMessageProcessor","thread_name":"SimpleAsyncTaskExecutor-1","level":"WARN","level_value":30000,"stack_trace":"java.io.EOFException: null\n\tat org.bson.io.Bits.readFully(Bits.java:50) ~[mongo-java-driver-2.12.5.jar:na]\n\tat org.bson.io.Bits.readFully(Bits.java:35)\n\tat org.bson.io.Bits.readFully(Bits.java:30)\n\tat com.mongodb.Response.(Response.java:42)\n\tat com.mongodb.DBPort$1.execute(DBPort.java:141)\n\tat com.mongodb.DBPort$1.execute(DBPort.java:135)\n\tat com.mongodb.DBPort.doOperation(DBPort.java:164)\n\tat com.mongodb.DBPort.call(DBPort.java:135)\n\tat c.m.DBTCPConnector.innerCall(DBTCPConnector.java:289)\n\t... 56 common frames omitted\nWrapped by: c.m.MongoException$Network: Read operation to server prod_mongos.internal.xyz.com:27017 failed on database xyz\n\tat c.m.DBTCPConnector.innerCall(DBTCPConnector.java:297) ~[mongo-java-driver-2.12.5.jar:na]\n\tat c.m.DBTCPConnector.call(DBTCPConnector.java:268)\n\tat c.m.DBCollectionImpl.find(DBCollectionImpl.java:84)\n\tat c.m.DBCollectionImpl.find(DBCollectionImpl.java:66)\n\tat c.m.DBCollection.findOne(DBCollection.java:869)\n\tat c.m.DBCollection.findOne(DBCollection.java:843)\n\tat c.m.DBCollection.findOne(DBCollection.java:789)\n\tat o.s.d.m.c.MongoTemplate$FindOneCallback.doInCollection(MongoTemplate.java:2013) ~[spring-data-mongodb-1.6.3.RELEASE.jar:na]\n\tat o.s.d.m.c.MongoTemplate$FindOneCallback.doInCollection(MongoTemplate.java:1997)\n\tat o.s.d.m.c.MongoTemplate.executeFindOneInternal(MongoTemplate.java:1772)\n\t... 47 common frames omitted\nWrapped by: o.s.d.DataAccessResourceFailureException: Read operation to server prod_mongos.internal.xyz.com:27017 failed on database xyz; nested exception is com.mongodb.MongoException$Network: Read operation to server prod_mongos.internal.xyz.com:27017 failed on database xyz\n\tat o.s.d.m.c.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:59) ~[spring-data-mongodb-1.6.3.RELEASE.jar:na]\n\tat o.s.d.m.c.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:1946)\n\tat o.s.d.m.c.MongoTemplate.executeFindOneInternal(MongoTemplate.java:1776)\n\tat o.s.d.m.c.MongoTemplate.doFindOne(MongoTemplate.j...","HOSTNAME":"prod-node-09","requestId":"eb8c702c-b3e7-4605-99a7-c3dcb5a076a9","WHAT":"ProcessBulkDiscoveredXYZEvent","host":"172.30.31.155:44243","type":"cloud_service","tags":["_grokparsefailure"]}
Failed to restore snapshot - IndexShardRestoreFailedException file not found in elasticsearch
I am using 3 node cluster setup with the elasticsearch 1.3.1, i have 17 indices each one is having min 0.5 M (1Gi) documents and 1.4 M (3 Gi) max. now i would like to try the snapshot and restore process in my cluster. i used the following REST calls to do the same...
To create a repository:
curl -XPUT 'http://host.name:9200/_snapshot/es_snapshot_repo' -d '{
"type": "fs",
"settings": {
"location": "/data/es_snapshot_bkup_repo/es_snapshot_repo"
}
}'
Verified the repository:
curl -XGET 'http://host.name:9200/_snapshot/es_snapshot_repo?pretty' the response is
{
"es_snapshot_repo" : {
"type" : "fs",
"settings" : {
"location" : "/data/es_snapshot_bkup_repo/es_snapshot_repo"
}
}
}
done the SNAPSHOT using
curl -XPUT "http://host.name:9200/_snapshot/es_snapshot_repo/snap_001" -d '{
"indices": "index_01",
"ignore_unavailable": "true",
"include_global_state": false,
"wait_for_completion": true
}'
the response is
{
"accepted": true
}
then I am trying to restore the snapshot by the request
curl -XPOST "http://host.name:9200/_snapshot/es_snapshot_repo/snap_001/_restore" -d '{
"indices": "index_01",
"ignore_unavailable": "true",
"include_global_state": false,
"rename_pattern": "index_01",
"rename_replacement": "index_01_bk",
"include_aliases": false
}'
ISSUE:
As I informed I have 3 nodes. the index which I am trying to take snapshot & restore is has 6 shards and 2 replicas.
Most of the shards and its replicas are restored properly, but sometimes 1, sometimes 2 primary shards and its replicas restoring is not happen. those primary shards are in the INITIALIZING state. I allow the cluster to relocate them for more than an hour but the shards are not relocating to the correct node... I got the following exception in my node.
the restore process trying to place the shard in the other 2 nodes... but it can't possible...
[2014-08-27 07:10:35,492][DEBUG][cluster.service ] [node_01] processing [
shard-failed (
[snap_001][4],
node[r4UoA7vJREmQfh6lz634NA],
[P],
restoring[es_snapshot_repo:snap_001],
s[INITIALIZING]),
reason [Failed to start shard,
message [IndexShardGatewayRecoveryException[[snap_001][4] failed recovery];
nested: IndexShardRestoreFailedException[[snap_001][4] restore failed];
nested: IndexShardRestoreFailedException[[snap_001][4] failed to restore snapshot [snap_001]];
nested: IndexShardRestoreFailedException[[snap_001][4] failed to read shard snapshot file];
nested: FileNotFoundException[/data/es_snapshot_bkup_repo/es_snapshot_repo/indices/index_01/4/snapshot-snap_001 (No such file or directory)]; ]]]:
done applying updated cluster_state (version: 56391)
Could anyone help me to overcome this issue and please correct me if I done any mistake in these process...
FYI I am using master node to pass the curl request
We need to provide a shared file system location which should be access by all the elasticsearch nodes with read & write permission.
Javamail Pop3 Session timeout?
I'm trying to use javamail to download a mailbox of 1000 messages of hotmail. The problem is after an hour or so, I get a Pop Session timeout exception, and I can't fetch messages anymore. C: TOP 3210 0 S: +OK 1444 byte(s) X-Message-Delivery: Vj0zLjQuMDt1cz0wO2k9MDtsPTA7YT0x X-Message-Status: n:0 X-SID-PRA: Super Wall <apps+ocdlfcez#facebookmail.com> X-SID-Result: Pass X-Message-Info: R00BdL5giqoqgO8FeGWl8Lch6n3is6BT1wNitKPj0Jb+fghk1p9MsC+MFGyB2nflerotq/xZ5r8LiguM+3GjEOSj3umkoXeU Received: from mx-out.facebook.com ([204.15.20.140]) by bay0-mc7-f15.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.2668); Tue, 8 Apr 2008 15:14:55 -0700 Received: from api.facebook.com (intlb01-mip1.sctm.tfbnw.net [10.1.240.6]) by mx-out.facebook.com [email018.sctm.facebook.com] (8.13.6/8.13.6) with ESMTP id m38MEtOg030239 for <xgameprogrammer#hotmail.com>; Tue, 8 Apr 2008 15:14:55 -0700 X-Facebook: from zuckmail ([168.143.164.188]) by api.facebook.com with HTTP (ZuckMail); Date: Tue, 8 Apr 2008 15:14:55 -0700 To: Ahmed Saleh <xgameprogrammer#hotmail.com> From: Super Wall <apps+ocdlfcez#facebookmail.com> Reply-to: Facebook <apps+ocdlfcez#facebookmail.com> Subject: You just received a new photo from Lejla Boric Message-ID: <03df95f3306af0a88432e7fcca22f7ac#api.facebook.com> X-Priority: 3 X-Mailer: ZuckMail [version 1.00] X-Facebook-Notify: platform_email Errors-To: apps+ocdlfcez#facebookmail.com MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="b1_03df95f3306af0a88432e7fcca22f7ac" Return-Path: apps+ocdlfcez#facebookmail.com X-OriginalArrivalTime: 08 Apr 2008 22:14:55.0926 (UTC) FILETIME=[F9D6A560:01C899C5] . C: RETR 3210 S: -ERR POP3 session timed out javax.mail.MessagingException: No inputstream from datasource; SentDate : Wed Apr 09 01:14:55 AST 2008 nested exception is:
As I said in my comment, this is just a simple case of the session timing out. Like anything, the mail server will have a time limit to prevent excessive access, which is how would describe downloading emails for over an hour! There is nothing wrong with your code, your just asking too much of hotmail / the mail server! Though, just some friendly advice: the next time you ask a question here on SO, please don't post irrelevant information like you have done here. In cases like yours, it is helpful to see atleast a snippet of the code that is responsible for that part of the application that is functioning incorrectly and any exceptions - we'd also need to see the line(s) of code mentioned in the stack trace... However, a start would be to actually ask a question instead of this stating a fact, like you have done here!