I'm one of the developers of the Hawk model indexing tool. Our tool indexes XMI models into graphs in order to speed up later queries, and it needs to toggle back and forth between "batch insert" and "transactional update" modes. "batch insert" is used the first time we notice a new file in a directory, and from then on we use "transactional update" mode to keep its graph in sync.
Our recently added OrientDB 2.1.4 backend uses the getTx()/getNoTx() methods in OrientGraphFactory to get the appropriate OrientGraph/OrientGraphNoTx instances. However, we aren't getting very good throughput when compared to Neo4j. Indexing set0.xmi takes 90s when placing the WAL in a Linux ramdisk with OrientDB, while it takes 22s with our Neo4j backend in the same conditions (machine + OS + JDK). We're using these additional settings to try and reduce times:
Increased WAL cache size to 10000
Disable sync on page flush
Save only dirty objects
Use massive insert intent
Disable transactional log
Disable MVCC
Disable validation
Use lightweight edges when possible
We've thought of disabling the WAL when entering "batch insert" mode, but there doesn't seem to be an easy way to toggle that on and off. It appears it can only be set once at program startup and that's it. We've tried explicitly closing the underlying storage so the USE_WAL flag will be read once more while reopening the storage, but that only results in NullPointerExceptions and other random errors.
Any pointers on how we could toggle the WAL, or improve performance beyond that would be greatly appreciated.
Update: We've switched to using the raw document API and marking dirty nodes/edges ourselves and we're now hitting 55 seconds, but the WAL problem still persists. Also tried 2.2.0-beta, but it actually took longer.
We solved this ourselves. Leaving this in case it might help someone :-). We've hit 30 seconds after many internal improvements in our backend (still using the raw doc API) and switching to OrientDB 2.0.15, and we found the way to toggle the Write Ahead Log ourselves. This works for us (db is our ODatabaseDocumentTx instance):
private void reopenWithWALSetTo(final boolean useWAL) {
db.getStorage().close(true, false);
db.close();
OGlobalConfiguration.USE_WAL.setValue(useWAL);
db = new ODatabaseDocumentTx(dbURL);
db.open("admin", "admin");
}
I was being silly and thought I had to close the DB first and then the storage, but it turns out that wasn't the case :-). It's necessary to use the two-arg version of the ODatabaseDocumentTx#close method, as the no-arg version basically does nothing for the OAbstractPaginatedStorage implementation used through plocal:// URLs.
Related
Background : in Java I'm memory mapping a file (shared).
I'm writing some value at the address 0 of that file. I understand the corresponding PAGE in the PAGE CACHE is flagged as DIRTY and will be written later depending on the dirty_ratio and the like settings.
So far so good.
But I'm wondering what is happening when writing once more at the address 0 while the kernel is writing back the dirty page to the file. Is my process blocked somehow waiting for the writeback to be completed?
It may be. It is only necessary when the device-level I/O requests include a checksum alongside the written data. Otherwise, the first write may be torn, but it can then be corrected by the second write.
As always, carefully consider your safety against power-failure, kernel crashes etc.
The waiting is allegedly avoided in btrfs. (Also, by happenstance, in the legacy ext3 filesystem. But not ext4 or ext2).
This looks like it is a bit of a moving target. The above (as far as I could tell) describes the first optimization of this "stable page write" code, following the complaints when it was first introduced. The commit description mentions several possibilities for future changes.
bdi: allow block devices to say that they require stable page writes
mm: only enforce stable page writes if the backing device requires it
Does my device currently use "stable page writes"?
There is a sysfs attribute you can look at, called stable_pages_required
This question is to do with FatWire Content Server version 7.6.
I have a FatWire template which goes through some assets and returns some markup. It takes about 2 minutes to complete. The result gets cached successfully in FatWire ContentServer cache and I can see it in the CacheManager tool. The ExpiryTime on the template is 10 years in the future. After a short while (usually 1-2 mins), the ExpiryTime changes to a past date (1980-02-01 01:01:01) and hence is expired. The item then disappears from the cache.
Has anyone experienced this before? It is only happening with this template. Any ideas as to the reason the item expires after first being cached successfully?
If you are using old-style page cache implementation (SystemPageCache, SystemItemCache table), then there may I'd suggest to enable some debug, to see if a particular page/element or event is running after this, to cause the change to the table:
enable these loggers (restart afterwards):
com.futuretense.cs.db=DEBUG
com.futuretense.cs.request=DEBUG
com.futuretense.cs.event=DEBUG
com.futuretense.cs=DEBUG
Tail futuretense.txt/sites.log, and reproduce the issue - You should be able to see the point where the new page let is cached with future expiration, and then something subsequent changes it. That may tell you whether it occurred as a result of a system event, or another page request. In case this is occurring on a clustered environment, you would need to set the same logging & tail on the other cluster nodes to spot whether the change is occurring from those.
If you are using new-style page cache ("InCache", cs-cache.xml etc), then it may be that another node is unexpectedly interacting with this node. You could temporarily isolate this node from a cache cluster, by adjusting the multicast settings in cs-cache.xml - e.g. timeToLive=0 will prevent any other nodes on different physical servers from seeing this one.
I have a java application that uses Lucene (latest version, 5.2.1 as of this writing) in "near realtime" mode; it has one network connection to receive requests to index documents, and another connection for search requests.
I'm testing with a corpus of pretty large documents (several megabytes of plain text) and several versions of each field with different analyzers. One of them being a phonetic analyzer with the Beider-Morse filter, the indexing of some documents can take quite a bit of time (over a minute in some cases). Most of this time is spent in the call to IndexWriter.addDocument(doc);
My problem is that while a document is being indexed, searches get blocked, and they aren't processed until the indexing operation finishes. Having the search blocked for more than a couple seconds is unacceptable.
Before each search, I do the following:
DirectoryReader newReader = DirectoryReader.openIfChanged(reader, writer, false);
if (newReader != null)
{
reader = newReader;
searcher = new IndexSearcher(reader);
}
I guess this is what causes the problem. However, is the only way to get the most recent changes when I do a search. I'd like to maintain this behaviour in general, but if the search would block I wouldn't mind to use a slightly old version of the index.
Is there any way to fix this?
Among other options, consider having always an IndexWriter open and perform "commits" to it as you need.
Then you should ask for index readers to it (not to the directory) and refresh them as needed. Or simply use a SearcherManager that will not only refresh searchers for you, but also will maintain a pool of readers and will manage references to them, in order to avoid reopening if the index contents haven't change.
See more here.
I'm making a server with Java that will provide chat services for flash clients. The server will store data about each user on a .txt file somewhere on the server. For example when a user logs in, information about this user is requested to the DatabaseManger class. It will then search through the database and return the information. The point is that when allot of people log in a short amount of time the server is doing allot of checks again and again.
The idea that I want to implement is that a connection class does something like this:
String userData = DatabaseManager.getUserData(this.username);
The DatabaseManager then doesn't search immediately, it stores this request in an array of requests, then in a fixed interval it goes through the database 1 time and returns data to the clients that requested this. This way when 15 people log in in a second it wont go through all the information 15 times. How to implement this?
You use a real DBMS like everyone else on the planet. I'm eager to hear a reason why someone wouldn't choose a DB for this application. I can't think of anything that would prevent it. Back in the day, RDBMS were ungainly, expensive, complicated beasts. Today, they're as readily available as tabloids at the checkout counter.
There are few excuses to not to a DB nowadays, and arguably there are more excuses to use the DB than the file system for most any application.
As above I'd recommend using an existing database solution like HSQLDB, you'd be far better off in the long run doing things this way rather than hacking your own solution together.
If you really want to do this anyway, have a look at the ScheduledExecutorService. You can then fire off a request to the executor service with a delay, and in that delay listen for more data and add it to the query.
I've setup a Wicket + Hibernate + Spring Web application that involves gathering some data (having some files generated and returned), storing this in a database, creating some images and displaying all this on a webpage.
This all works fine for short runs, but sometimes gathering the data (which involves some remote number crunching) takes too long (20+ minutes) and times out. I've tried to resolve this using two approaches, but both of them show some problems.
The first approach was using AjaxLazyLoadPanels and just doing everything within the getLazyLoadComponent. This worked fine for the short runs, but for the 20+ minute runs the LazyLoadComponents would not load (nice oxymoron there) due to timeouts.
The second approach involved creating an intermediate Fragment with an added AjaxSelfUpdatingTimerBehavior with a duration set to 10 seconds, that polled for the files that are created in the number crunching. This seems to make the tasks run in the background without problems, but fails when the returned data needs to be stored in the database. I'm using the Open Session in View pattern, but maybe this fails when attempting to store data after 20 minutes?? (Solution could lie in resolving this..).
Due to the above problems I'm now reading up on alternate approaches to handle these long running tasks and came across:
org.apache.wicket.util.time.Task
org.apache.wicket.util.watch.ModificationWatcher
I'm now wondering if either of these might be better suited to solve the time-out problems I'm having in both running the tasks and storing the data in the database afterwards, or if anyone has any other solutions that might help in this situation.
I'd really like to know if a new approach is viable before I spend another day implementing something that might turn out not to work after all.
Best regards,
Tim
I know we have had success in using a Panel with an attached AjaxSelfUpdatingTimerBehavior. The task and the results piece are separated from the view logic, but are made accessible for the view via a service you create. The service implementation we have used is then responsible for starting a TheadPool or ExectutorService for running the individual tasks. The service can provide a way to monitor the progress/status of the particular job/call that is taking place. Once it is complete it should also make the data available for the view. Injection of a SessionFactory into the service implementation (or injected DAO) should be sufficient to create the HibernateSession outside of a WebSession.