I have java application based on seedstack framework, it uses internally glassfish server.
I generate a xls file and export it. If the file generation completes in 60 second, it works fine. If the file generation takes more than 60 sec it error out. How to resolve this issue ? Is this related to proxy_read_timeout ? how to set this key to increase the time ? Thanks for help.
Log
{ "#timestamp": "2023-02-10T05:35:31.671Z", "#version": "1", "message": "An I/O error has occurred while writing a response message entity to the container output stream.", "logger_name": "org.glassfish.jersey.server.ServerRuntime$Responder", "thread_name": "XNIO-1 task-4", "level": "ERROR", "level_value": 40000, "stack_trace": "org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Broken pipe\n\tat org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:91)\n\tat org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)\n\tat org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135)\n\tat
Im having trouble using Thingsboard platform (IoT) when simulating 7.5K devices sending data to the platform. I have the following error in the logs as soon as I start sending data (over MQTT):
2020-08-01 01:17:06,946 [ForkJoinPool-12-worker-0] ERROR c.g.c.u.concurrent.AggregateFuture - Got more than one input Future failure. Logging failures after the first
java.lang.IllegalStateException: Deque full
at java.util.concurrent.LinkedBlockingDeque.addLast(LinkedBlockingDeque.java:335)
at java.util.concurrent.LinkedBlockingDeque.add(LinkedBlockingDeque.java:633)
at org.thingsboard.server.dao.util.AbstractBufferedRateExecutor.submit(AbstractBufferedRateExecutor.java:109)
at org.thingsboard.server.dao.nosql.CassandraAbstractDao.executeAsync(CassandraAbstractDao.java:93)
at org.thingsboard.server.dao.nosql.CassandraAbstractDao.executeAsyncWrite(CassandraAbstractDao.java:76)
at org.thingsboard.server.dao.timeseries.CassandraBaseTimeseriesDao.savePartition(CassandraBaseTimeseriesDao.java:434)
at org.thingsboard.server.dao.timeseries.BaseTimeseriesService.saveAndRegisterFutures(BaseTimeseriesService.java:153)
at org.thingsboard.server.dao.timeseries.BaseTimeseriesService.save(BaseTimeseriesService.java:144)
at org.thingsboard.server.service.telemetry.DefaultTelemetrySubscriptionService.saveAndNotify(DefaultTelemetrySubscriptionService.java:124)
at org.thingsboard.rule.engine.telemetry.TbMsgTimeseriesNode.onMsg(TbMsgTimeseriesNode.java:89)
at org.thingsboard.server.actors.ruleChain.RuleNodeActorMessageProcessor.onRuleChainToRuleNodeMsg(RuleNodeActorMessageProcessor.java:107)
at org.thingsboard.server.actors.ruleChain.RuleNodeActor.onRuleChainToRuleNodeMsg(RuleNodeActor.java:97)
at org.thingsboard.server.actors.ruleChain.RuleNodeActor.doProcess(RuleNodeActor.java:60)
at org.thingsboard.server.actors.service.ContextAwareActor.process(ContextAwareActor.java:45)
at org.thingsboard.server.actors.TbActorMailbox.processMailbox(TbActorMailbox.java:121)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
I have try to google and see the reason behind it, but i havent found anything.
While simulating with 5K devices, this error came like 3 times each day (over a 4 day period), but it stopped showing that error eventually. However, when increasing the number of devices, the error is constant. Im using Kafka as the broker, but I dont see any Kafka related error. I just want to know why the error appears, is it related to memory, or any other limit?
Thanks in advance
Francisco P
I'm developing a MS with Kotlin and Micronaut which access a Firestore database. When I run this MS locally I can make it work with 128M because it's very simple just read and write data to Firestore, and not big amounts of data, really small data like this:
{
"project": "DUMMY",
"columns": [
{
"name": "TODO",
"taskStatus": "TODO"
},
{
"name": "IN_PROGRESS",
"taskStatus": "IN_PROGRESS"
},
{
"name": "DONE",
"taskStatus": "DONE"
}
],
"tasks": {}
}
I'm running this in App Engine Standard in a F1 instance (256 MB 600 MHz) with this properties in my app.yaml
runtime: java11
instance_class: F1 # 256 MB 600 MHz
entrypoint: java -Xmx200m -jar MY_JAR.jar
service: data-connector
env_variables:
JAVA_TOOL_OPTIONS: "-Xmx230m"
GAE_MEMORY_MB: 128M
automatic_scaling:
max_instances: 1
max_idle_instances: 1
I know all that properties for handling memory are not necessary but I was desperate trying to make this work and just tried a lot of solutions because my first error message was:
Exceeded soft memory limit of 256 MB with 263 MB after servicing 1 requests total. Consider setting a larger instance class in app.yaml.
The error below is not fixed with the properties in the app.yaml, but now everytime I make a call to return that JSON I get this error
2020-04-10 12:09:15.953 CEST
While handling this request, the process that handled this request was found to be using too much memory and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application or may be using an instance with insufficient memory. Consider setting a larger instance class in app.yaml.
It always last longer in the first request, I think due to some Firestore configuration, but the thing is that I cannot make that work, always getting the same error.
Do you have any idea what I could be doing wrong or what I need to fix this?
TL;DR The problem was I tried to used a very small instance for a simple application, but even with that I needed more memory.
Ok, a friend helped me with this. I was using a very small instance and even when I didn't get the error of memory limit it was a memory problem.
Updating my instance to a F2 (512 MB 1.2 GHz) solved the problem and testing my app with siege resulted in a very nice performance:
Transactions: 5012 hits
Availability: 100.00 %
Elapsed time: 59.47 secs
Data transferred: 0.45 MB
Response time: 0.30 secs
Transaction rate: 84.28 trans/sec
Throughput: 0.01 MB/sec
Concurrency: 24.95
Successful transactions: 3946
Failed transactions: 0
Longest transaction: 1.08
Shortest transaction: 0.09
My sysops friends tells me that this instances are more for python scripting code and things like that, not JVM REST servers.
We have a Ruby on Rails application running for some time now, but since a couple of days we're just getting a HTTP500 Error while trying to update any documents. The system is running Ruby 1.9.3 with unicorn 4.8.2 and nginx 1.2.1 and a sunspot_solr 2.1.1 gem for the search index (besides some others).
The Production log tells me that every time someone tries to create or update a document the solr server throws a bunch of errors. The most prominent I think is this one:
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:164)\n\t... 30 more\nCaused by: java.lang.OutOfMemoryError: Java heap space\n\
Unfortunately nothing changes if i edit config/sunstpot.yml to include "max_memory: 1G":
production:
solr:
hostname: localhost
port: 8080
log_level: WARNING
min_memory: 32M
max_memory: 1G
path: /solr-4.10.2/default #production #ollection1 #production
Here is the complete production.log entry:
[a086bc878daa0abd367379daade96682] Completed 500 Internal Server Error in 195ms
[a086bc878daa0abd367379daade96682]
RSolr::Error::Http (RSolr::Error::Http - 500 Internal Server Error
Error: {"responseHeader":{"status":500,"QTime":2},"error":{"msg":"Exception writing document id Neuigkeit 393 to the index; possible analysis error.","trace":"org.apache.solr.common.SolrException: Exception writing document id Neuigkeit 393 to the index; possible analysis error.\n\
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:168)\n\
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)\n\tat
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)\n\tat
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:926)\n\tat
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1080)\n\tat
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:692)\n\tat
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)\n\
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:247)\n\tat
org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174)\n\
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:99)\n\tat
org.apache.solr.haa086bc878daa0abd367379daade96682] Completed 500 Internal Server Error in 195ms
[a086bc878daa0abd367379daade96682]
RSolr::Error::Http (RSolr::Error::Http - 500 Internal Server Error
Error: {"responseHeader":{"status":500,"QTime":2},"error":{"msg":"Exception writing document id Neuigkeit 393 to the index; possible analysis error.","trace":"org.apache.solr.common.SolrException: Exception writing document indler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)\n\
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\
org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)\n\
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)\n\
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)\n\
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)\n\
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)\n\
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)\n\
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)\n\
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)\n\
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)\n\
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)\n\
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)\n\
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)\n\
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)\n\
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1003)\n\
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)\n\
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)\n\
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\
java.lang.Thread.run(Thread.java:745)\nCaused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed\n\
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:698)\n\
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:712)\n\
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1507)\n\
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:240)\n\
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:164)\n\t... 30 more\nCaused by: java.lang.OutOfMemoryError: Java heap space\n\
org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.<init>(FreqProxTermsWriterPerField.java:207)\n\
org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.newInstance(FreqProxTermsWriterPerField.java:230)\n\
org.apache.lucene.index.ParallelPostingsArray.grow(ParallelPostingsArray.java:48)\n\
org.apache.lucene.index.TermsHashPerField$PostingsBytesStartArray.grow(TermsHashPerField.java:252)\n\
org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:292)\n\
org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:151)\n\
org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:659)\n\
org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:359)\n\
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:318)\n\
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239)\n\
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:454)\n\
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1511)\n\
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:240)\n\
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:164)\n\
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)\n\
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)\n\
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:926)\n\
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1080)\n\
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:692)\n\
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)\n\
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:247)\n\
org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174)\n\
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:99)\n\
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)\n\
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)\n\
org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)\n\
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)\n\
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)\n\
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)\n\
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)\n\
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)\n\
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)\n","code":500}}
URI: http://localhost:8080/solr-4.10.2/default/update?wt=json
EDIT: I'm an idiot. The solr server isn't contained in the unicorn service. Restarting it via 'service tomcat7 restart' loaded the updated config/sunstpot.yml and solved the problem.
I have installed Docker Toolbox for Mac OSX and running several containers inside. First two I created were with Cassandra and were running fine. After that I've created 2 Debian containers, connected to bash through docker terminal with the purpose to install Oracle JDK8.
At the point when I was about to extract java from the tarball - I've got a ton of "Cannot write: No space left on device" error messages during the execution of "tar" command.
I've checked the space:
$ docker ps -s
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE
9d8029e21918 debian:latest "/bin/bash" 54 minutes ago Up 54 minutes deb-2 620.5 MB (virtual 744 MB)
49c7a0e37475 debian:latest "/bin/bash" 55 minutes ago Up 55 minutes deb-1 620 MB (virtual 743.5 MB)
66a17af83ca3 cassandra "/docker-entrypoint.s" 4 hours ago Up 4 hours 7000-7001/tcp, 7199/tcp, 9042/tcp, 9160/tcp node-2 40.16 MB (virtual 412.6 MB)
After seeing that output I noticed that one of my nodes with cassandra is missing. In went to check to Kitematic and found out that it is in the DOWN state and I can't start it: "Cannot write node . No space left on device" - error message shown for this attempt.
Are there any limits that Docker has to run the containers?
When I remove all my cassandra ones and leave just a couple of Debian - java is able to be extracted from the tar. So the issue is definitely in some Docker settings related to sizing.
What is the correct way to resolve the issue with space limits here?
UPDATE.
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
cassandra latest 13ea610e5c2b 11 hours ago 374.8 MB
debian jessie 23cb15b0fcec 2 weeks ago 125.1 MB
debian latest 23cb15b0fcec 2 weeks ago 125.1 MB
The output of df -hi
$ df -hi
Filesystem Inodes IUsed IFree IUse% Mounted on
none 251K 38K 214K 15% /
tmpfs 251K 18 251K 1% /dev
tmpfs 251K 12 251K 1% /sys/fs/cgroup
tmpfs 251K 38K 214K 15% /etc/hosts
shm 251K 1 251K 1% /dev/shm
`df -h
Filesystem Size Used Avail Use% Mounted on
none 1.8G 1.8G 0 100%
/ tmpfs 1002M 0 1002M 0%
/dev tmpfs 1002M 0 1002M 0%
/sys/fs/cgroup tmpfs 1.8G 1.8G 0 100%
/etc/hosts shm 64M 0 64M 0% /dev/shm`
Appreciate help.
I have resolved this issue in docker somehow.
By default the memory for the docker is set to be 2048M by default.
First step I performed is stopping my docker machine:
$ docker-machine stop default
Then I went to the $HOME/.docker/machine/machines/default/config.json file and changed the "Memory" setting to be higher, i.e. 4096.
{
"ConfigVersion": 3,
"Driver": {
"VBoxManager": {},
"IPAddress": "192.168.99.102",
"MachineName": "default",
"SSHUser": "docker",
"SSHPort": 59177,
"SSHKeyPath": "/Users/lenok/.docker/machine/machines/default/id_rsa",
"StorePath": "/Users/lenok/.docker/machine",
"SwarmMaster": false,
"SwarmHost": "tcp://0.0.0.0:3376",
"SwarmDiscovery": "",
"CPU": 1,
"Memory": 4096,
"DiskSize": 204800,
"Boot2DockerURL": "",
"Boot2DockerImportVM": "",
"HostDNSResolver": false,
"HostOnlyCIDR": "192.168.99.1/24",
"HostOnlyNicType": "82540EM",
"HostOnlyPromiscMode": "deny",
"NoShare": false,
"DNSProxy": false
},
"DriverName": "virtualbox",
"HostOptions": {
"Driver": "",
"Memory": 0,
"Disk": 0,
"EngineOptions": {
"ArbitraryFlags": [],
"Dns": null,
"GraphDir": "",
"Env": [],
"Ipv6": false,
"InsecureRegistry": [],
"Labels": [],
"LogLevel": "",
"StorageDriver": "",
"SelinuxEnabled": false,
"TlsVerify": true,
"RegistryMirror": [],
"InstallURL": "https://get.docker.com"
},
"SwarmOptions": {
"IsSwarm": false,
"Address": "",
"Discovery": "",
"Master": false,
"Host": "tcp://0.0.0.0:3376",
"Image": "swarm:latest",
"Strategy": "spread",
"Heartbeat": 0,
"Overcommit": 0,
"ArbitraryFlags": [],
"config.json" [noeol] 75L, 2560C
"Overcommit": 0,
"ArbitraryFlags": [],
"Env": null
},
"AuthOptions": {
"CertDir": "/Users/lenok/.docker/machine/certs",
"CaCertPath": "/Users/lenok/.docker/machine/certs/ca.pem",
"CaPrivateKeyPath": "/Users/lenok/.docker/machine/certs/ca-key.pem",
"CaCertRemotePath": "",
"ServerCertPath": "/Users/lenok/.docker/machine/machines/default/server.pem",
"ServerKeyPath": "/Users/lenok/.docker/machine/machines/default/server-key.pem",
"ClientKeyPath": "/Users/lenok/.docker/machine/certs/key.pem",
"ServerCertRemotePath": "",
"ServerKeyRemotePath": "",
"ClientCertPath": "/Users/lenok/.docker/machine/certs/cert.pem",
"ServerCertSANs": [],
"StorePath": "/Users/lenok/.docker/machine/machines/default"
}
},
"Name": "default"
}
Finally, started my docker machine again:
$ docker-machine start default
Issue 18869 refers to a docker-machine memory allocation problem.
This can be tested on the fly with
vboxmanage controlvm default 4096
Since drivers/virtualbox/virtualbox.go#L344-L352 reloads the settings from HOME/.docker/machine/machines/default/config.json, it is best to record that new value in that file (as mentioned in this answer).
That "No space left on device" was seen in docker/machine issue 2285, where the vmdk image created is a dynamically allocated/grow at run-time (default), creating a smaller on-disk foot-print initially, therefore even when creating a ~20GiB vm, with --virtualbox-disk-size 20000 requires on about ~200MiB of free space on-disk to start with.
And the default memory is quite low.
Make sure you that don't have :
any more exited container that you could remove:
docker rm -v $(docker ps --filter status=exited -q 2>/dev/null) 2>/dev/null
any dangling images
docker rmi $(docker images --filter dangling=true -q 2>/dev/null) 2>/dev/null
(Those are the result of rebuild which makes intermediate images unused)
See also "How to remove old and unused Docker images"
Then make sure you don't have an inode exhaustion problem, as in issue 10613.
Check df -hi (with i for inodes)
connected to bash through docker terminal with the purpose to install Oracle JDK8.
Try instead to specify the installation in a Dockerfile and build an image with the JDK installed.