I have an IgniteCache<String, byte[]>
The values in this cache can be small all the way to several hundred megabytes.
From the documentation I've read, there is no hard limit on the value of a cache entry. Which is reassuring.
But are there any gotchas for my use-case that I need to worry about?
For example, any configuration settings I need to set specifically for this situation?
I don't recommend having entries larger than a few MB. It may cause all sorts of network or memory issues.
If you have to, you need to try with largest possible values and then adjust settings until you are good. One thing I can think off, setting TcpCommunicationSpi.ackSendThreshold to some small value such as 4 (default 32) to avoid these large messages sitting in queues. But there's probably a lot of things to tune.
Internally, Ignite uses byte buffers to serialize data, and those buffers are limited by Integer.MAX_VALUE, so 2GiB is the maximum cache entry size.
Some Ignite APIs process data in batches, it is a good idea to reduce batch/page sizes when dealing with large entries:
Query.pageSize (applies to Scan, SQL, Continuous queries) - default is 1024
IgniteDataStreamer.perNodeBufferSize, perThreadBufferSize - defaults are 512 and 4096
Related
Consider the task of fetching a limited amount of data from a (w.r.t memory) infinite source, e.g. I need to fetch counts from an huge table consisting of billions and more entries, labeled by timestamp and foreign_key within a finite but possibly huge time window. But for later consumption I only need maximally gridSize values.
Note:
The backing DB is a MariaDb and we use spring data jpa to connect to the database.
One stream based approach is:
int stepSize = (int) (numberOfCountsInWindow / gridSize) + 1;
StreamUtils
.zipWithIndex(countStream)
.filter(countIndexed -> countIndexed.getIndex() % stepSize == 0)
...
<intermediate steps>
...
.collect(Collectors.toList())
I've tried plenty of others, like:
a custom Collector, that uses an AtromicLong to decide if the next value is added to the final list or just ignored,
Flux.defere( () -> ), to use the Flux back-pressure management
Java 11
plenty of MySQL options like "useCursorFetch=true" in combination with a finite prefetch size as #QueryHint()
...
However, all of what I've tried leads to GC overhead limit exceeded (with the heap size limited to 2048, which is the default for our applications).
The reason is, the cpu is totally busy garbage collecting, while the memory consumption is just filling up until finally the application crashes.
What I actually expected (or at least hoped for), was the stream to "realize" the filtering and just continue with the actual computing, at best using 100% cpu, and the garbage collector to easily remove the unused objects, since they are filtered out anyway and don't even need to be parsed (was hoping for the laziness of the stream to help here). However, this doesn't seem to work as I was hoping for.
Comments or suggestions are very welcome, since I'd like to (at least) understand (at best solve) rather then just accept the situation.
EDIT: Alternatives to MariaDB (maybe Cassandra?) are also welcome.
Below is default configuration in hazelcast.xml,
<jobtracker name="default">
<max-thread-size>0</max-thread-size>
<!-- Queue size 0 means number of partitions * 2 -->
<queue-size>0</queue-size>
<retry-count>0</retry-count>
<chunk-size>1000</chunk-size>
<communicate-stats>true</communicate-stats>
<topology-changed-strategy>CANCEL_RUNNING_OPERATION</topology-changed-strategy>
</jobtracker>
How to update this configuration to get better performance for map reducing in java application???
The values you normally want to optimize are chunk-size and communicate-stats. First property is heavily depending on the way your mr job works and needs some trial and error, best is to keep reducers busy all the time (so depending on the reducing operation either bigger chunk size for heavy ops or smaller chunks for light operations). The communicate-stats deactivates transmission of statistical information which is normally not being used anyways.
I need to get rough estimation of memory usage of my Infinispan cache ( which is implemented using version 5.3.0) - ( For learning purposes )
Since there is no easy way to do this, I came up with following procedure.
Add cache listener to listen cache put/remove events and log the size of inserted entry using jamm library which uses java.lang.instrument.Instrumentation.getObjectSize. But i'm little skeptical about it, whether it returns right memory usage for cache. Am I doing this measurement correctly ? Am I missing something here or do I need consider more factors to do this ?
If you don't need real-time information, the easiest way to find the actual memory usage is to take a heap dump and look at the cache objects' retained sizes (e.g. with Eclipse MAT).
Your method is going to ignore the overhead of Infinispan's internal structures. Normally, the per-entry overhead should be somewhere around 150 bytes, but sometimes it can be quite big - e.g. when you enable eviction and Infinispan allocates structures based on the configured size (https://issues.jboss.org/browse/ISPN-4126).
My Java application deals with large binary data files using memory mapped file (MappedByteBuffer, FileChannel and RandomAccessFile). It often needs to grow the binary file - my current approach is to re-map the file with a larger region.
It works, however there are two problems
Grow takes more and more time as the file becomes larger.
If grow is conducted very rapidly (E.G. in a while(true) loop), JVM will hang forever after the re-map operation is done for about 30,000+ times.
What are the alternative approaches, and what is the best way to do this?
Also I cannot figure out why the second problem occurs. Please also suggest your opinion on that problem.
Thank you!
Current code for growing a file, if it helps:
(set! data (.map ^FileChannel data-fc FileChannel$MapMode/READ_WRITE
0 (+ (.limit ^MappedByteBuffer data) (+ DOC-HDR room))))
You probably want to grow your file in larger chunks. Use a doubling each time you remap, like a dynamic array, so that the cost for growing is an amortized constant.
I don't know why the remap hangs after 30,000 times, that seems odd. But you should be able to get away with a lot less than 30,000 remaps if you use the scheme I suggest.
The JVM doesn't clean up memory mappings even if you call the cleaner explicitly. Thank you #EJP for the correction.
If you create 32,000 of these they could be all in existence at once. BTW: I suspect you might be hitting some 15-bit limit.
The only solution for this is; don't create so many mapping. You can map an entire disk 4 TB disk with less than 4K mapping.
I wouldn't create a mapping less than 16 to 128 MB if you know the usage will grow and I would consider up to 1 GB per mapping. The reason you can do this with little cost is that the main memory and disk space will not be allocated until you actually use the pages. i.e. the main memory usage grows 4 KB at a time.
The only reason I wouldn't create a 2 GB mapping is Java doesn't support this due to an Integer.MAX_VALUE size limit :( If you have 2 GB or more you have to create multiple mappings.
Unless you can afford an exponential growth on the file like doubling, or any other constant multiplier, you need to consider whether you really need a MappedByteBuffer at all, considering their limitations (unable to grow the file, no GC, etc). I personally would either be reviewing the problem or else using a RandomAccessFile in "rw" mode, probably with a virtual-array layer over the top of it.
I have an application that has to insert about 13 million rows of about 10 average length strings into an embedded HSQLDB. I've been tweaking things (batch size, single threaded/multithreaded, cached/non-cached tables, MVCC transactions, log_size/no logs, regular calls to checkpoint, ...) and it still takes 7 hours on a 16 core, 12 GB machine.
I chose HSQLDB because I figured I might have a substantial performance gain if I put all of those cores to good use but I'm seriously starting to doubt my decision.
Can anyone show me the silver bullet?
With CACHED tables, disk IO is taking most of the time. There is no need for multiple threads because you are inserting into the same table. One thing that noticably improves performance is the reuse of a single parameterized PreparedStatment, setting the parameters for each row insert.
On your machine, you can improve IO significantly by using a large NIO limit for memory-mapped IO. For example SET FILES NIO SIZE 8192. A 64 bit JVM is required for larger sizes to have an effect.
http://hsqldb.org/doc/2.0/guide/management-chapt.html
To reduce IO for the duration of the bulk insert use SET FILES LOG FALSE and do not perform a checkpoint until the end of the insert. The details are discussed here:
http://hsqldb.org/doc/2.0/guide/deployment-chapt.html#dec_bulk_operations
UPDATE: An insert test with 16 million rows below resulted in a 1.9 GigaByte .data file and took just a few minutes on an average 2 core processor and 7200 RPM disk. The key is large NIO allocation.
connection time -- 47
complete setup time -- 78 ms
insert time for 16384000 rows -- 384610 ms -- 42598 tps
shutdown time -- 38109
check what your application is doing. First things would be to look at resource utilization in taskmanager (or OS specific comparable) and visualvm.
Good candidates for causing bad performance:
disk IO
garbage collector
H2Database may give you slightly better performance than HSQLDB (while maintaining syntax compatibility).
In any case, you might want to try using a higher delay for syncing to disk to reduce random access disk I/O. (ie. SET WRITE_DELAY <num>)
Hopefully you're doing bulk INSERT statements, rather than a single insert per row. If not, do that if possible.
Depending on your application requirements, you might be better off with a key-value store than an RDBMS. (Do you regularly need to insert 1.3*10^7 entries?)
Your main limiting factor is going to be random access operations to disk. I highly doubt that anything you're doing will be CPU-bound. (Take a look at top, then compare it to iotop!)
With so many records, maybe you could consider switching to a NoSQL DB. It depends on the nature/format of the data you need to store, of course.