This is my source code of Main.java. It was grabbed from neo4j-apoc-1.0 examples. The goal of modification to store 1M records of 2 nodes and 1 relation:
package javaapplication2;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.Node;
import org.neo4j.graphdb.RelationshipType;
import org.neo4j.graphdb.Transaction;
import org.neo4j.kernel.EmbeddedGraphDatabase;
public class Main
{
private static final String DB_PATH = "neo4j-store-1M";
private static final String NAME_KEY = "name";
private static enum ExampleRelationshipTypes implements RelationshipType
{
EXAMPLE
}
public static void main(String[] args)
{
GraphDatabaseService graphDb = null;
try
{
System.out.println( "Init database..." );
graphDb = new EmbeddedGraphDatabase( DB_PATH );
registerShutdownHook( graphDb );
System.out.println( "Start of creating database..." );
int valIndex = 0;
for(int i=0; i<1000; ++i)
{
for(int j=0; j<1000; ++j)
{
Transaction tx = graphDb.beginTx();
try
{
Node firstNode = graphDb.createNode();
firstNode.setProperty( NAME_KEY, "Hello" + valIndex );
Node secondNode = graphDb.createNode();
secondNode.setProperty( NAME_KEY, "World" + valIndex );
firstNode.createRelationshipTo(
secondNode, ExampleRelationshipTypes.EXAMPLE );
tx.success();
++valIndex;
}
finally
{
tx.finish();
}
}
}
System.out.println("Ok, client processing finished!");
}
finally
{
System.out.println( "Shutting down database ..." );
graphDb.shutdown();
}
}
private static void registerShutdownHook( final GraphDatabaseService graphDb )
{
// Registers a shutdown hook for the Neo4j instance so that it
// shuts down nicely when the VM exits (even if you "Ctrl-C" the
// running example before it's completed)
Runtime.getRuntime().addShutdownHook( new Thread()
{
#Override
public void run()
{
graphDb.shutdown();
}
} );
}
}
After a few iterations (around 150K) I got error message:
"java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at org.neo4j.kernel.impl.nioneo.store.PlainPersistenceWindow.(PlainPersistenceWindow.java:30)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.allocateNewWindow(PersistenceWindowPool.java:534)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.refreshBricks(PersistenceWindowPool.java:430)
at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:122)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:459)
at org.neo4j.kernel.impl.nioneo.store.AbstractDynamicStore.updateRecord(AbstractDynamicStore.java:240)
at org.neo4j.kernel.impl.nioneo.store.PropertyStore.updateRecord(PropertyStore.java:209)
at org.neo4j.kernel.impl.nioneo.xa.Command$PropertyCommand.execute(Command.java:513)
at org.neo4j.kernel.impl.nioneo.xa.NeoTransaction.doCommit(NeoTransaction.java:443)
at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:316)
at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:399)
at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64)
at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:514)
at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:571)
at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:543)
at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:102)
at org.neo4j.kernel.EmbeddedGraphDbImpl$TransactionImpl.finish(EmbeddedGraphDbImpl.java:329)
at javaapplication2.Main.main(Main.java:62)
28.05.2010 9:52:14 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool logWarn
WARNING: [neo4j-store-1M\neostore.propertystore.db.strings] Unable to allocate direct buffer"
Guys! Help me plzzz, what I did wrong, how can I repair it? Tested on platform Windows XP 32bit SP3. Maybe solution within creation custom configuration?
thnx 4 every advice!
this is a configuration issue on Windows, where Neo4j cannot use memory mapped buffers. Instead, a Java Buffer on the heap is created. In 1.0 this buffer was 470MB per default, which is more than the default heap for the Windows JVM. You have two options:
Switch to APOC 1.1-SNAPSHOT instead of 1.0 in your pom.xml which has an autoconfiguration, assigning max 50% of the available JVM heap to Neo4j
Adjust the JVM heap to more (e.g. 512Mb) by running Java with
java -Xmx512m ....
You can even insert that under JVM arguments in the Run Configurations in Eclipse
Let us know if this helps!
Also, doing a full transaction for every node pair is going to take a long time. Try opening a transaction in the first loop and do commits only every 1000 node pairs?
/peter
Related
My Java program (see below) sometimes crashes with a java.nio.file.AccessDeniedException in a java.nio.File.move() method execution.
I could not understand why this exception is thrown and I have no bypass for now.
Here is an example of the exception :
java.nio.file.AccessDeniedException: C:\PROJECTS\PROJECT0\CHANGES -> C:\PROJECTS\PROJECT0\GEN70\CHANGES
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:95)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:109)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:399)
at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:299)
at java.nio.file.Files.move(Files.java:1406)
at com.ibm.cldt.engine.tool.TestMove.generate(TestMove.java:75)
at com.ibm.cldt.engine.tool.TestMove.createAndUseProject(TestMove.java:42)
at com.ibm.cldt.engine.tool.TestMove.main(TestMove.java:25)
Here the problem is detected on "GEN70" of "PROJECT0", but, it varies. For example, here is another run :
java.nio.file.AccessDeniedException: C:\PROJECTS\PROJECT2\CHANGES -> C:\PROJECTS\PROJECT2\GEN33\CHANGES
Note : before running the program, you have to delete the directory C:/PROJECTS if you have one.
What can I do to prevent my program from throwing this exception ?
I run this code on Windows 10 Enterprise, and an IBM JRE 1.8.
java version "1.8.0"
Java(TM) SE Runtime Environment (build pwa6480sr4fp5-20170421_01(SR4 FP5))
IBM J9 VM (build 2.8, JRE 1.8.0 Windows 10 amd64-64 Compressed References 20170419_344392 (JIT enabled, AOT enabled)
J9VM - R28_20170419_1004_B344392
JIT - tr.r14.java_20170419_344392
GC - R28_20170419_1004_B344392_CMPRSS
J9CL - 20170419_344392)
JCL - 20170420_01 based on Oracle jdk8u131-b11
Here is the code. You can run it as Java standalone application. Before launching, check that you do not have a C:/PROJECTS directory.
I will be surprised if the program execution ends without exception on your machine. If that is the case, please retry ...
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardOpenOption;
public class TestMove
{
private static final String PROJECTS_ROOT = "C:/PROJECTS";
private static final int NUMBER_OF_PROJECTS = 10;
private static final int NUMBER_OF_GENERATIONS = 100;
private static final int NUMBER_OF_CHANGES = 10;
public static void main( String[] args )
{
try
{
for ( int project = 0; project < NUMBER_OF_PROJECTS; ++project )
{
createAndUseProject( "PROJECT"+project );
}
}
catch ( IOException ioe )
{
ioe.printStackTrace();
}
}
private static void createAndUseProject( String projectName ) throws IOException
{
Path projectRoot = Paths.get( PROJECTS_ROOT, projectName );
Files.createDirectories( projectRoot );
for ( int generation = 0; generation < NUMBER_OF_GENERATIONS; ++generation )
{
addNewChanges( projectRoot );
generate( projectRoot, generation );
}
}
private static final StandardOpenOption[] CREATE_APPEND =
new StandardOpenOption[] { StandardOpenOption.CREATE, StandardOpenOption.APPEND };
private static void addNewChanges( Path projectRoot ) throws IOException
{
Path changesDir = projectRoot.resolve( "CHANGES" );
Files.createDirectory( changesDir );
String newLine = System.lineSeparator();
Path changesLogFile = changesDir.resolve( "changes.log" );
try ( BufferedWriter changesWriter = Files.newBufferedWriter( changesLogFile, CREATE_APPEND ) )
{
for ( int change = 0; change < NUMBER_OF_CHANGES; ++change )
{
changesWriter.append( "This is my change number "+ change ).append( newLine );
}
}
}
private static void generate( Path projectRoot, int generation ) throws IOException
{
Path generationDir = projectRoot.resolve( "GEN"+generation );
Files.createDirectory( generationDir );
Path projectChangesDir = projectRoot.resolve( "CHANGES" );
Path generationChangesDir = generationDir.resolve( "CHANGES" );
// Here is the problem : AccessDeniedException is thrown ... sometimes.
Files.move( projectChangesDir, generationChangesDir );
Path changesLogFile = generationChangesDir.resolve( "changes.log" );
try ( BufferedReader changesReader = Files.newBufferedReader( changesLogFile ) )
{
for ( String change = changesReader.readLine(); change != null; change = changesReader.readLine() )
computeChange( change );
}
}
private static void computeChange( String change )
{
// Do whatever needed ...
}
}
What can I do to prevent my program from throwing this exception ?
COMPLEMENTS
From the first answers, I downloaded the Oracle JDK 1.8.0_221 from Oracle website. Then, I used javac and java commands to compile and run my program from a CMD window.
Here is the transcript:
Microsoft Windows [Version 10.0.18362.356]
(c) 2019 Microsoft Corporation. All rights reserved.
C:\tmp\Java>dir
Volume in drive C is Windows
Volume Serial Number is 8A56-3036
Directory of C:\tmp\Java
09/24/2019 06:57 PM <DIR> .
09/24/2019 06:57 PM <DIR> ..
09/24/2019 06:54 PM 2,678 TestMove.java
1 File(s) 2,678 bytes
2 Dir(s) 353,415,393,280 bytes free
C:\tmp\Java>"C:\Program Files\Java\jdk1.8.0_221\bin\javac" TestMove.java
C:\tmp\Java>"C:\Program Files\Java\jdk1.8.0_221\bin\java" TestMove
java.nio.file.AccessDeniedException: C:\PROJECTS\PROJECT0\CHANGES -> C:\PROJECTS\PROJECT0\GEN97\CHANGES
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at TestMove.generate(TestMove.java:73)
at TestMove.createAndUseProject(TestMove.java:40)
at TestMove.main(TestMove.java:23)
C:\tmp\Java>
Same problem with a standard up-to-date JVM, an no eclipse. I feel bad ;-) ...
UPDATE :
I have found this bypass. It works well, but I do not feel good with it in my app in production.
I have replaced those two lines :
// Here is the problem : AccessDeniedException is thrown ... sometimes.
Files.move( projectChangesDir, generationChangesDir );
With this code:
while ( true )
{
try
{
Files.move( projectChangesDir, generationChangesDir );
break;
}
catch ( IOException ioe ) { ++failures; }
}
It works suprisingly well and makes it possible for my program to run until its normal end. But ... well ... not so satisfactory. At the end failures counter is around 10, sometimes less, sometimes more, for a total of 1000 attempts (10 projects x 100 generations).
I know this question is rather old, and I don't have a clear answer, but a suspicion.
I'm having they same problem sometimes, but only when trying to move executable files (or folders containing such), and it happens more often when my computer is busy doing other stuff.
My suspicion is that an (corporate level) antivirus software is the culprit. Sometimes it isn't fast enough to scan the file and still has a lock on it when your program tries to move it.
I also didn't find a nice solution, and use a practically identical workaround to the one you are using. Disabling the antivirus isn't an option here, because even if I get an exception from my company, we also have to ensure the software works for our costumers without the need disable antivirus.
AccessDeniedException suggests there is a permissions problem. How are you executing your program, and does the user have permissions to create the c:/projects directory, and write to it?
One option would be to try running your code as an admin user. This post should help with that: https://superuser.com/questions/42537/is-there-any-sudo-command-for-windows
For now, as far as I know, only Aaron (https://stackoverflow.com/users/1678362/aaron) did try the sample Java code I provided in my question (thank you Aaron). He did not reproduce the issue.
I wonder what could be the difference between our two setups. Maybe the fact that my Windows 10 is a Professional one ?
Also, I would be happy if others can try the small code sample on their machine ... especially with various setups, including Windows 10 Professional.
I am using Nifi to create data flow pipeline where I use Infinispan a a cache server But when I use executescript with Groovy script , it goes on infinite loop and open many socket connections. I have tried to close the same but still it opens many connections and then it throws
java.net.SocketException: No buffer space available (maximum connections reached?): connect
By following below link I changed the registry
https://support.pitneybowes.com/VFP06_KnowledgeWithSidebarTroubleshoot?id=kA280000000PEE1CAO&popup=false&lang=en_US
Then checked the open connections with netstat -n I opens 65534 because of the above settings.
Below is the groovy script to read from Infinispan cache
import org.infinispan.client.hotrod.RemoteCache;
import org.infinispan.client.hotrod.RemoteCacheManager;
import org.infinispan.client.hotrod.configuration.ConfigurationBuilder;
import org.apache.commons.io.IOUtils;
import java.nio.charset.StandardCharsets;
def cacheName = "mycache"
def configuration = new ConfigurationBuilder()
.addServer().host("localhost").port(11322).build();
def cacheManager = new RemoteCacheManager(configuration)
RemoteCache cacheA = cacheManager.getCache(cacheName)
flowFile = session.get()
if(!flowFile) return
key = flowFile.getAttribute('key')
id = flowFile.getAttribute('id')
jsonFromCache = cacheA.get(key + "_" + id);
if(cacheA != null) {
cacheA.stop()
}
if(cacheManager != null) {
cacheManager.stop()
}
flowFile = session.write(flowFile, {outputStream ->
outputStream.write(jsonFromCache.getBytes(StandardCharsets.UTF_8))
} as OutputStreamCallback)
session.transfer(flowFile, REL_SUCCESS)
You are opening connection to cache before getting file from session.
So, you are opening connection and at the following line just exits script without closing it:
if(!flowFile) return
Another point:
you can use ExecuteGroovyScript processor. Then processor start & stop could be managed. Example you can find here: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-groovyx-nar/1.9.2/org.apache.nifi.processors.groovyx.ExecuteGroovyScript/additionalDetails.html
import org.apache.nifi.processor.ProcessContext
import java.util.concurrent.atomic.AtomicLong
class Const{
static Date startTime = null;
static AtomicLong triggerCount = null;
}
static onStart(ProcessContext context){
Const.startTime = new Date()
Const.triggerCount = new AtomicLong(0)
println "onStart $context ${Const.startTime}"
}
static onStop(ProcessContext context){
def alive = (System.currentTimeMillis() - Const.startTime.getTime()) / 1000
println "onStop $context executed ${ Const.triggerCount } times during ${ alive } seconds"
}
def flowFile = session.get()
if(!flowFile)return
flowFile.'trigger.count' = Const.triggerCount.incrementAndGet()
REL_SUCCESS << flowFile
I need to get the free available disk space for all disks in system, or all partitions, I don't mind that. (I dont have to use Sigar, but I am using it already on the project for some other processes, so I can use it for this as well)
I am using Sigar API and got this
public double getFreeHdd() throws SigarException{
FileSystemUsage f= sigar.getFileSystemUsage("/");
return ( f.getAvail());
}
But this only gives me the system partition (root), how can i get a list of all partition and loop them to get their free space?
I tried this
FileSystemView fsv = FileSystemView.getFileSystemView();
File[] roots = fsv.getRoots();
for (int i = 0; i < roots.length; i++) {
System.out.println("Root: " + roots[i]);
}
But it only returns the root dir
Root: /
Thanks
Edit
it seems that I could use
FileSystem[] fslist = sigar.getFileSystemList();
But the results i am getting do not match the ones i get from the terminal. On the other hand on this system I am working on, i have 3 disks with a total 12 partitions, so i might be loosing something there. Will try it on some other system in case i can make something useful out of the results.
We use SIGAR extensively for cross-platform monitoring. This is the code we use to get the file system list:
/**
* #return a list of directory path names of file systems that are local or network - not removable media
*/
public static Set<String> getLocalOrNetworkFileSystemDirectoryNames() {
Set<String> ret = new HashSet<String>();
try {
FileSystem[] fileSystemList = getSigarProxy().getFileSystemList();
for (FileSystem fs : fileSystemList) {
if ((fs.getType() == FileSystem.TYPE_LOCAL_DISK) || (fs.getType() == FileSystem.TYPE_NETWORK)) {
ret.add(fs.getDirName());
}
}
}
catch (SigarException e) {
// log or rethrow as appropriate
}
return ret;
}
You can then use that as the input to other SIGAR methods:
FileSystemUsage usageStats = getSigarProxy().getFileSystemUsage(fileSystemDirectoryPath);
The getSigarProxy() is just a convenience base method:
// The Humidor handles thread safety for a single instance of a Sigar object
static final private SigarProxy sigarProxy = Humidor.getInstance().getSigar();
static final protected SigarProxy getSigarProxy() {
return sigarProxy;
}
You can use java.nio.file.FileSystems to get a list of java.nio.file.FileStorages and then see the usable/available space. Per instance (assuming that you are using Java 7+):
import java.io.IOException;
import java.nio.file.FileStore;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.util.function.Consumer;
public static void main(String[] args) {
FileSystem fs = FileSystems.getDefault();
fs.getFileStores().forEach(new Consumer<FileStore>() {
#Override
public void accept(FileStore store) {
try {
System.out.println(store.getTotalSpace());
System.out.println(store.getUsableSpace());
} catch (IOException e) {
e.printStackTrace();
}
}
});
}
Also, keep in mind that FileStore.getUsableSpace() returns the size in bytes. See the docs for more information.
In one of my projects I have concurrent write access to one single file within one JRE and want to handle that by first writing to a temporary file and afterwards moving that temp file to the target using an atomic move. I don't care about the order of the write access or such, all I need to guarantee is that any given time the single file is usable. I'm already aware of Files.move and such, my problem is that I had a look at at least one implementation for that method and it raised some doubts about if implementations really guarantee atomic moves. Please look at the following code:
Files.move on GrepCode for OpenJDK
1342 FileSystemProvider provider = provider(source);
1343 if (provider(target) == provider) {
1344 // same provider
1345 provider.move(source, target, options);
1346 } else {
1347 // different providers
1348 CopyMoveHelper.moveToForeignTarget(source, target, options);
1349 }
The problem is that the option ATOMIC_MOVE is not considered in all cases, but the location of the source and target path is the only thing that matters in the first place. That's not what I want and how I understand the documentation:
If the move cannot be performed as an atomic file system operation then
AtomicMoveNotSupportedException is thrown. This can arise, for example, when the target
location is on a different FileStore and would require that the file be copied, or target
location is associated with a different provider to this object.
The above code clearly violates that documentation because it falls back to a copy-delete-strategy without recognizing ATOMIC_MOVE at all. An exception would be perfectly OK in my case, because with that a hoster of our service could change his setup to use only one filesystem which supports atomic moves, as that's what we expect in the system requirements anyway. What I don't want to deal with is things silently failing just because an implementation uses a copy-delete-strategy which may result in data corruption in the target file. So, from my understanding it is simply not safe to rely on Files.move for atomic operations, because it doesn't always fail if those are not supported, but implementations may fall back to a copy-delete-strategy.
Is such behaviour a bug in the implementation and needs to get filed or does the documentation allow such behaviour and I'm understanding it wrong? Does it make any difference at all if I now already know that such maybe broken implementations are used out there? I would need to synchronize the write access on my own in that case...
You are looking at the wrong place. When the file system providers are not the same, the operation will be delegated to moveToForeignTarget as you have seen within the code snippet you’ve posted. The method moveToForeignTarget however will use the method convertMoveToCopyOptions (note the speaking name…) for getting the necessary copy options for the translated operation. And convertMoveToCopyOptions will throw an AtomicMoveNotSupportedException if it encounters the ATOMIC_MOVE option as there is no way to convert that move option to a valid copy option.
So there’s no reason to worry and in general it’s recommended to avoid hasty conclusion from seeing just less than ten lines of code (especially when not having tried a single test)…
The standard Java library does not provide a way to perform an atomic move in all cases.
Files.move() does not guarantee atomic move. You can pass ATOMIC_MOVE as an option, but if the move cannot be performed as an atomic operation, AtomicMoveNotSupportedException is thrown (this is the case when target location is on a different FileStore and would require that the file be copied).
You have to implement it yourself if you really need that. One solution can be to catch AtomicMoveNotSupportedException and then do this: Try to move the file without the ATOMIC_MOVE option but catch exceptions and remove the target if error occured during the copy.
I came across similar problem to be solved:
One process frequently updates file via 'save to tempfile -> move tempfile to final file' using Files.move(tmp, out, ATOMIC_MOVE, REPLACE_EXISTING);
Another one or more processes read that file - completely, all-at-once, and closes immediatelly. File is rather small - less than 50k.
And it just does not work reliably, at least on windows. Under heavy load reader occasionally gets NoSuchFileException - this means Files.move is not that ATOMIC even on the same file system :(
My env: Windows 10 + java 11.0.12
Here is the code to play with:
import org.junit.Test;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.ByteBuffer;
import java.nio.channels.ByteChannel;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Set;
import static java.nio.charset.StandardCharsets.UTF_8;
import static java.nio.file.StandardCopyOption.ATOMIC_MOVE;
import static java.nio.file.StandardCopyOption.REPLACE_EXISTING;
import static java.util.Locale.US;
public class SomeTest {
static int nWrite = 0;
static int nRead = 0;
static int cErrors = 0;
static boolean writeFinished;
static boolean useFileChannels = true;
static String filePath = "c:/temp/test.out";
#Test
public void testParallelFileAccess() throws Exception {
new Writer().start();
new Reader().start();
while( !writeFinished ) {
Thread.sleep(10);
}
System.out.println("cErrors: " + cErrors);
}
static class Writer extends Thread {
public Writer() {
setDaemon(true);
}
#Override
public void run() {
File outFile = new File("c:/temp/test.out");
File outFileTmp = new File(filePath + "tmp");
byte[] bytes = "test".getBytes(UTF_8);
for( nWrite = 1; nWrite <= 100000; nWrite++ ) {
if( (nWrite % 1000) == 0 )
System.out.println("nWrite: " + nWrite + ", cReads: " + nRead);
try( FileOutputStream fos = new FileOutputStream(outFileTmp) ) {
fos.write(bytes);
}
catch( Exception e ) {
logException("write", e);
}
int maxAttemps = 10;
for( int i = 0; i <= maxAttemps; i++ ) {
try {
Files.move(outFileTmp.toPath(), outFile.toPath(), ATOMIC_MOVE, REPLACE_EXISTING);
break;
}
catch( IOException e ) {
try {
Thread.sleep(1);
}
catch( InterruptedException ex ) {
break;
}
if( i == maxAttemps )
logException("move", e);
}
}
}
System.out.println("Write finished ...");
writeFinished = true;
}
}
static class Reader extends Thread {
public Reader() {
setDaemon(true);
}
#Override
public void run() {
File inFile = new File(filePath);
Path inPath = inFile.toPath();
byte[] bytes = new byte[100];
ByteBuffer buffer = ByteBuffer.allocateDirect(100);
try { Thread.sleep(100); } catch( InterruptedException e ) { }
for( nRead = 0; !writeFinished; nRead++ ) {
if( useFileChannels ) {
try ( ByteChannel channel = Files.newByteChannel(inPath, Set.of()) ) {
channel.read(buffer);
}
catch( Exception e ) {
logException("read", e);
}
}
else {
try( InputStream fis = Files.newInputStream(inFile.toPath()) ) {
fis.read(bytes);
}
catch( Exception e ) {
logException("read", e);
}
}
}
}
}
private static void logException(String action, Exception e) {
cErrors++;
System.err.printf(US, "%s: %s - wr=%s, rd=%s:, %s%n", cErrors, action, nWrite, nRead, e);
}
}
I want to be able to execute the .Jar file, and if the heap space isn't set big enough, it should launch a new JVM with the same .Jar file, but set with a bigger heap space, and then close the first JVM and .Jar.
I've tried using the ProcessBuilder, but I can't get it to work.
It has to work cross platform.
-ONi
I have found the solution, and it works cross platform. To restart the JVM from code, use the following. This answer is taken from another question I found after hours of search in here. If you want, you can follow it with an System.exit(0), to terminate the JVM that started the new process, after a call to this method.
public static void startSecondJVM() throws Exception {
String separator = System.getProperty("file.separator");
String classpath = System.getProperty("java.class.path");
String path = System.getProperty("java.home")
+ separator + "bin" + separator + "java";
ProcessBuilder processBuilder =
new ProcessBuilder(path, "-Xmx1024m", "-cp",
classpath,
Main.class.getName());
Process process = processBuilder.start();
}
You can launch java with an initial heap size, and also specify a maximum heap size which will be only be used as required. I'm not sure what you're trying to do but it might emulate the behaviour you want?
java -Xms256m -Xmx1g -jar myapp.jar
In this example you start with 256M, if the app needs more memory it will take it, incrementally, up until 1G.
You might try combining these two sources.
MemoryRecoveryTest.java
Makes attempts to recover from an OutOfMemoryError.
/*License - LGPL
<h3>Recovery from an OutOfMemory Error</h3>
<p>The JavaDocs for Error state, in the first sentence..
<blockquote>"An Error is a subclass of Throwable that indicates
serious problems that a reasonable application should
not try to catch."</blockquote>
<p>This advice has led to the fallacy that an OutOfMemoryError
should not be caught and dealt with.But this demo. shows
that it is quite easy to recover to the point of providing
the user with meaningful information, and advice on how to
proceed.
<p>I aim to make my applications 'unreasonable'.;-)
*/
import java.awt.event.ActionListener;
import java.awt.event.ActionEvent;
import java.awt.event.WindowAdapter;
import java.awt.event.WindowEvent;
import javax.swing.JPanel;
import javax.swing.JLabel;
import javax.swing.JProgressBar;
import javax.swing.JOptionPane;
import javax.swing.JDialog;
import javax.swing.Timer;
import javax.swing.border.EmptyBorder;
import java.util.ArrayList;
/** A demo. showing recovery from an OutOfMemoryError.
Our options once an OOME is encountered are relatively
few, but we can still warn the end user and provide
advice on how to correct the problem.
#author Andrew Thompson */
public class MemoryRecoveryTest {
public static void main(String[] args) {
// reserve a buffer of memory
byte[] buffer = new byte[2^10];
ArrayList<Object> list = new ArrayList<Object>();
final JProgressBar memory = new JProgressBar(
0,
(int)Runtime.getRuntime().totalMemory());
ActionListener listener = new ActionListener() {
#Override
public void actionPerformed(ActionEvent ae) {
memory.setValue(
(int)Runtime.getRuntime().freeMemory() );
}
};
Timer timer = new Timer(500, listener);
timer.start();
JDialog dialog = new JDialog();
dialog.setTitle("Available Memory");
JPanel memoryPanel = new JPanel();
memoryPanel.add(memory);
memoryPanel.setBorder(new EmptyBorder(25,25,25,25));
dialog.add( memoryPanel );
dialog.pack();
dialog.setLocationRelativeTo(null);
dialog.setVisible(true);
dialog.addWindowListener( new WindowAdapter(){
#Override
public void windowClosing(WindowEvent we) {
System.exit(0);
}
} );
// prepare a memory warning panel in advance
JPanel memoryWarning = new JPanel();
memoryWarning.add( new JLabel(
"<HTML><BODY>There is not enough memory to" +
" complete the task!<BR> Use a variant " +
" of the application that assigns more memory.") );
try {
// do our 'memory intensive' task
while(true) {
list.add( new Object() );
}
} catch(OutOfMemoryError oome) {
// provide the VM with some memory 'breathing space'
// by clearing the buffer
buffer = null;
// tell the user what went wrong, and how to fix it
JOptionPane.showMessageDialog(
dialog,
memoryWarning,
"Out of Memory!",
JOptionPane.ERROR_MESSAGE);
}
}
}
IWantToBeBig.java
Ensures a Process is started with a memory size specified.
import java.awt.EventQueue;
import javax.swing.JOptionPane;
import java.io.File;
class IWantToBeBig {
public static void main(String[] args) throws Exception {
if (args.length==0) {
ProcessBuilder pb = new ProcessBuilder(
"java",
"-jar",
"-Xmx512m",
"big.jar",
"anArgument"
);
pb.directory(new File("."));
Process process = pb.start();
process.waitFor();
System.out.println("Exit value: " + process.exitValue());
} else {
Runnable r = new Runnable() {
public void run() {
JOptionPane.showMessageDialog(
null,
"Max Memory: " +
Runtime.getRuntime().maxMemory() +
" bytes.");
}
};
EventQueue.invokeLater(r);
}
}
}
I'd do this kind of work in an outer script file - in pseudo code:
$heap := 128
$ok := true
do {
exitCode = java -Xmx$heapM -jar myApp.jar
if (exitCode = OOME) {
heap += 128
$ok := false
}
while(!$ok)
Catch OOME and exiting with a custom code should always be possible. There's one problem with this approach - if the $heap value exceeds the maximum heap space that is possible for the target system (example: ~1.4GByte on Win32 systems) then it will not terminate.
Note: this is just an answer to the question - usually one would assign a high amount of memory and/or fight the memory leaks - but I don't know the actual requirments/restrictions