Develop Reference

Wiremock withBodyFile OutOfMemoryError for large files

In the following stub request, I use withBodyFile which loads large files (>300MB) from specific locations on local disk using streams:
public void mockGetUrlContent(String url) {
stubFor(get(urlEqualTo(url))
.willReturn(ok()
.withBodyFile(FilenameUtils.getName(url))));
}
Once the stub is being called, wiremock tries to complete a served event and in order to log the response it uses the from method below:
public static LoggedResponse from(Response response) {
return new LoggedResponse(
response.getStatus(),
response.getHeaders() == null || response.getHeaders().all().isEmpty()
? null
: response.getHeaders(),
response.getBody(),
response.getFault());
}
When reached to wiremock's Response getBody method it converts the stream in to byte[] and in that points it blows on OutOfMemoryError (Java heap space).:
public byte[] getBody() {
try (InputStream stream = bodyStreamSource == null ? null : getBodyStream()) {
return stream == null ? null : ByteStreams.toByteArray(stream);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
Any ideas about what am I doing wrong?
Your help is highly appreciated!

WireMock 2.34.0 has a solution to this.
To ensure that logged response bodies are limited to a particular size you need to start WireMock with a new configuration parameter specifying the max size in bytes
When running standalone add the CLI parameter:
--logged-response-body-size-limit=1000000
When running in Java add the following to the configuration builder:
wireMockConfig().maxLoggedResponseSize(1000000)

How to fix "The handle is invalid" when closing a directoryReader?

I am working with Lucene 4.8.0 and I have a DirectoryReader that I can't close. When I try I get the following error :
Caused by: java.lang.RuntimeException: java.io.IOException: The handle is invalid
at apache.lucene.util.IOUtils.reThrowUnchecked(IOUtils.java:368)
at apache.lucene.index.SegmentCoreReaders.notifyCoreClosedListeners(SegmentCoreReaders.java:203)
at apache.lucene.index.SegmentCoreReaders.decRef(SegmentCoreReaders.java:183)
at apache.lucene.index.SegmentReader.doClose(SegmentReader.java:258)
at apache.lucene.index.IndexReader.decRef(IndexReader.java:243)
at apache.lucene.index.StandardDirectoryReader.doClose(StandardDirectoryReader.java:364)
at apache.lucene.index.IndexReader.decRef(IndexReader.java:243)
at apache.lucene.index.IndexReader.close(IndexReader.java:479)
My code is as follows :
FSDirectory indexDir = null;
DirectoryReader dirReader = null
try{
indexDir = new SimpleFSDirectory(new File(baseDir + "/index"),null);
dirReader = DirectoryReader.open(indexDir);
//operations
finally {
if (dirReader != null) {
dirReader.close();
}
if (indexDir != null) {
indexDir.close();
}
}
But this crashes at the dirReader.close(). I am new to Lucene, if I missed something well known or obvious I am sorry.

File.createNewFile() randomly fails

I've build a simple test which creates and deletes a file (name does not change) in an infinite loop. The test does run for a couple of seconds (sometimes over 77,000 iterations!) and then fails with this exception:
Exception in thread "main" java.io.IOException: Access is denied
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(Unknown Source)
at DeleteTest.main(DeleteTest.java:11)
Here's the test logic:
final File f = new File(pathname);
while (true) {
final boolean create = f.createNewFile();
if (!create) {
System.out.println("crate failed");
} else {
final boolean delete = f.delete();
if (!delete) {
System.out.println("delete failed");
}
}
}
How is this possible? The delete call does not fail. It would tell. So delete always succeeds but createNewFile fails. This is what MSDN says about win32 api function DeleteFile:
The DeleteFile function marks a file for deletion on close. Therefore,
the file deletion does not occur until the last handle to the file is
closed. Subsequent calls to CreateFile to open the file fail with
ERROR_ACCESS_DENIED.
So createNewFile does not close the file? The openjdk source tells us that the file is closed:
JNIEXPORT jboolean JNICALL
Java_java_io_Win32FileSystem_createFileExclusively(JNIEnv *env, jclass cls,
jstring pathname)
{
jboolean rv = JNI_FALSE;
DWORD a;
WITH_PLATFORM_STRING(env, pathname, path) {
int orv;
int error;
JVM_NativePath((char *)path);
orv = JVM_Open(path, JVM_O_RDWR | JVM_O_CREAT | JVM_O_EXCL, 0666);
if (orv < 0) {
if (orv != JVM_EEXIST) {
error = GetLastError();
// If a directory by the named path already exists,
// return false (behavior of solaris and linux) instead of
// throwing an exception
a = GetFileAttributes(path);
if ((a == INVALID_FILE_ATTRIBUTES) ||
!(a & FILE_ATTRIBUTE_DIRECTORY)) {
SetLastError(error);
JNU_ThrowIOExceptionWithLastError(env, path);
}
}
} else {
JVM_Close(orv);
rv = JNI_TRUE;
}
} END_PLATFORM_STRING(env, path);
return rv;
}
Can anyone explain this behaviour?

I've found an explanation while writing the question. I still posted the question because I wanted to share what I learned.
My application is not the only process on the system accessing files. The Windows Search Index Service for example could open this file because it wants to add it to it's index. Or the windows Explorer if it is updating the view.

This issue reminds me a problem I experienced recently with the File.renameTo() method. It is (was?) due to this bug in the jvm :
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6213298
A weird workaround is to call System.gc() and to retry renaming the file again (and it works...).
Not sure it has a link with your issue, but it may be worth exploring...

Try this:
final File f = new File("file");
while (true) {
final boolean create = f.createNewFile();
if (!create) {
System.out.println("crate failed");
} else {
final boolean delete = f.delete();
try {
Thread.sleep(10);
} catch (InterruptedException e) {
System.out.println("...");
}
if (!delete) {
System.out.println("delete failed");
}
}
}
In this way we ensure that the file is released by the delete before invoking createNewFile.

OFX4J parse exception after it was working just fine - aggregate SONRS

I had a piece of code to read ofx file to retrieve several tags (such as acct number, balance etc.) I am using net.sf.ofx4j
Piece of code:
public void parse(String filename) throws OFXParseException, IOException, SQLException {
AggregateUnmarshaller<ResponseEnvelope> unmarshaller = new AggregateUnmarshaller<ResponseEnvelope>(
ResponseEnvelope.class);
FileInputStream file = null;
try {
file = new FileInputStream(filename);
ResponseEnvelope envelope = unmarshaller.unmarshal(file);
BankingResponseMessageSet messageSet = (BankingResponseMessageSet) envelope.getMessageSet(MessageSetType.banking);
List<BankStatementResponseTransaction> responses = messageSet.getStatementResponses();
for (BankStatementResponseTransaction response : responses) {
BankStatementResponse message = response.getMessage();
String currencyCode = message.getCurrencyCode();
String acct_number = message.getAccount().getAccountNumber();
double av = message.getAvailableBalance().getAmount();
double cur = message.getLedgerBalance().getAmount();
AccountType acct_type = message.getAccount().getAccountType();
}
} catch (OFXParseException e) {
System.out.println("Error: " + e.getMessage());
}
return null;
}
It was working fine until one day it started throwing the following exception:
net.sf.ofx4j.io.AggregateStackContentHandler onElement
INFO: Element INTU.BID is not supported on aggregate SONRS (class net.sf.ofx4j.domain.data.signon.SignonResponse) at index 70.
net.sf.ofx4j.io.AggregateStackContentHandler onElement
INFO: Element INTU.USERID is not supported on aggregate SONRS (class net.sf.ofx4j.domain.data.signon.SignonResponse) at index 70.
Exception in thread "main" java.lang.IllegalStateException: java.io.IOException: Unexpected EOF
Thanks

Hope ofx file format should be changed. because you are requesting (MessageSetType.*banking*). But in your ofx file may be have credit card details.
In early, transaction data is include in .ofx file inside the <BANKMSGSRSV1> tag.
But now transaction data is include in <CREDITCARDMSGSRSV> tag. you need to change the data receiving code.
Hope you can get some help from this. Thank you

NullPointerException using ImageIO.read

I'm getting an NPE while trying to read in an image file, and I can't for the life of me figure out why. Here is my line:
BufferedImage source = ImageIO.read(new File(imgPath));
imgPath is basically guaranteed to be valid and right before it gets here it copies the file from the server. When it hits that line, I get this stack trace:
Exception in thread "Thread-26" java.lang.NullPointerException
at com.ctreber.aclib.image.ico.ICOReader.getICOEntry(ICOReader.java:120)
at com.ctreber.aclib.image.ico.ICOReader.read(ICOReader.java:89)
at javax.imageio.ImageIO.read(ImageIO.java:1400)
at javax.imageio.ImageIO.read(ImageIO.java:1286)
at PrintServer.resizeImage(PrintServer.java:981) <---My function
<Stack of rest of my application here>
Also, this is thrown into my output window:
Can't create ICOFile: Can't read bytes: 2
I have no idea what is going on, especially since the File constructor is succeeding. I can't seem to find anybody who has had a similar problem. Anybody have any ideas? (Java 5 if that makes any difference)

I poked around some more and found that you can specify which ImageReader ImageIO will use and read it in that way. I poked around our codebase and found that we already had a function in place for doing EXACTLY what I was trying to accomplish here. Just for anybody else who runs into a similar issue, here is the crux of the code (some of the crap is defined above, but this should help anybody who tries to do it):
File imageFile = new File(filename);
Iterator<ImageReader> imageReaders = ImageIO.getImageReadersByFormatName("jpeg");
if ( imageReaders.hasNext() ) {
imageReader = (ImageReader)imageReaders.next();
stream = ImageIO.createImageInputStream(imageFile);
imageReader.setInput(stream, true);
ImageReadParam param = imageReader.getDefaultReadParam();
curImage = imageReader.read(0, param);
}
Thanks for the suggestions and help all.

The File constructor will almost certainly succeed, regardless of whether it points to a valid/existing file. At the very least, I'd check whether your underlying file exists via the exists() method.

Also note that ImageIO.read is not thread-safe (it reuses cached ImageReaders which are not thread-safe).
This means you can't easily read multiple files in parallel. To do that, you'll have to deal with ImageReaders yourself.

Have you considered that the file may simply be corrupted, or that ImageIO is trying to read it as the wrong type of file?

Googling for the ICOReader class results in one hit: IconsFactory from jide-common.
Apparently they had the same problem:
// Using ImageIO approach results in exception like this.
// Exception in thread "main" java.lang.NullPointerException
// at com.ctreber.aclib.image.ico.ICOReader.getICOEntry(ICOReader.java:120)
// at com.ctreber.aclib.image.ico.ICOReader.read(ICOReader.java:89)
// at javax.imageio.ImageIO.read(ImageIO.java:1400)
// at javax.imageio.ImageIO.read(ImageIO.java:1322)
// at com.jidesoft.icons.IconsFactory.b(Unknown Source)
// at com.jidesoft.icons.IconsFactory.a(Unknown Source)
// at com.jidesoft.icons.IconsFactory.getImageIcon(Unknown Source)
// at com.jidesoft.plaf.vsnet.VsnetMetalUtils.initComponentDefaults(Unknown Source)
// private static ImageIcon createImageIconWithException(final Class<?> baseClass, final String file) throws IOException {
// try {
// InputStream resource =
// baseClass.getResourceAsStream(file);
// if (resource == null) {
// throw new IOException("File " + file + " not found");
// }
// BufferedInputStream in =
// new BufferedInputStream(resource);
// return new ImageIcon(ImageIO.read(in));
// }
// catch (IOException ioe) {
// throw ioe;
// }
// }
What did they do instead?
private static ImageIcon createImageIconWithException(
final Class<?> baseClass, final String file)
throws IOException {
InputStream resource = baseClass.getResourceAsStream(file);
final byte[][] buffer = new byte[1][];
try {
if (resource == null) {
throw new IOException("File " + file + " not found");
}
BufferedInputStream in = new BufferedInputStream(resource);
ByteArrayOutputStream out = new ByteArrayOutputStream(1024);
buffer[0] = new byte[1024];
int n;
while ((n = in.read(buffer[0])) > 0) {
out.write(buffer[0], 0, n);
}
in.close();
out.flush();
buffer[0] = out.toByteArray();
} catch (IOException ioe) {
throw ioe;
}
if (buffer[0] == null) {
throw new IOException(baseClass.getName() + "/" + file
+ " not found.");
}
if (buffer[0].length == 0) {
throw new IOException("Warning: " + file
+ " is zero-length");
}
return new ImageIcon(Toolkit.getDefaultToolkit().createImage(
buffer[0]));
}
So you might want to try the same approach: read the raw bytes and use Toolkit to create an image from them.

"it's a jpeg but doesn't have a jpeg
extension."
That might be it.
It appears that the library AC.lib-ICO is throwing the NPE. Since this library is intended to read the Microsoft ICO file format, a JPEG might be a problem for it.
Consider explicitly providing the format using an alternative method.