The usual problem in Java is that you have to hack to get a proper unmapping of memory mapped files - see here for the 14year old bug report ;)
But on Android there seems to be 0 solutions in pure Java and just via NDK. Is this true? If yes, any pointers to an open source solution with Android/Java bindings?
There is no hack available under Android.
But there are a few helpers and snippets which make the C-Java binding for mmap files easy/easier:
util-mmap, Apache License 2.0, here is an issue regarding Android support
Using Memory Mapped Files and JNI to communicate between Java and C++ programs or easier with tools like javacpp?
It looks tomcat has implement a helper (jni.MMap) that is able to unmap/delete a mmap file
See the util-mmap in action, really easy:
public class MMapTesting {
public static void main(String[] args) throws IOException {
File file = new File("test");
MMapBuffer buffer = new MMapBuffer(file, 0, 1000, FileChannel.MapMode.READ_WRITE, ByteOrder.BIG_ENDIAN)) {
buffer.memory().intArray(0, 100).set(2, 234);
// calls unmap under the hood
buffer.close();
// here we call unmap automatically at the end of this try-resource block
try (MMapBuffer buffer = new MMapBuffer(file, FileChannel.MapMode.READ_WRITE, ByteOrder.BIG_ENDIAN)) {
System.out.println("length: " + buffer.memory().length());
IntArray arr = buffer.memory().intArray(0, buffer.memory().length() / 8);
// prints 234
System.out.println(arr.get(2));
}
}
}
From the Android Developers website:
A direct byte buffer whose content is a memory-mapped region of a file.
Mapped byte buffers are created via the FileChannel.map method. This class extends the ByteBuffer class with operations that are specific to memory-mapped file regions.
A mapped byte buffer and the file mapping that it represents remain valid until the buffer itself is garbage-collected.
The content of a mapped byte buffer can change at any time, for example if the content of the corresponding region of the mapped file is changed by this program or another. Whether or not such changes occur, and when they occur, is operating-system dependent and therefore unspecified.
As for what I've understood from this text, is that there is no way to unmap the MappedByteBuffer using the Android Java SDK. Only using the NDK, like you said.
Related
I am using the amazing Superpowered library (SuperpoweredAndroidAudioIO) for low-latency recording of audio. While the basic concepts are clear to me, I want to pass the recorded audio (which arrives in a buffer) back to an InputStream in Java (without recording to a file), from which I can then read the recorded audio and process it.
I guess this question could also be more generally asked - how to feed an InputStream in Java from a periodically updated buffer in C++?
Well, the suggestion I received in a comment turned out to be a simple and working solution:
Creation of pipe in C++:
if (pipe(pipefd) == -1) {
__android_log_print(ANDROID_LOG_VERBOSE, "C++", "Error creating pipe");
}
...
Passing file descriptor to Java:
...
return pipefd[0];
...
Then in Java/Android:
private ParcelFileDescriptor.AutoCloseInputStream underlyingStream;
ParcelFileDescriptor pfd = ParcelFileDescriptor.adoptFd(getFD());
underlyingStream = new ParcelFileDescriptor.AutoCloseInputStream(pfd);
Worked well for me, but of course I'm still happy to receive other suggestions.
I'm doing a program that modifies only the metadata (standard and custom) in files Doc, xls, ppt and Vsd, the program works correctly but I wonder if there is a way to do this without loading the entire file into memory:
POIFSFileSystem POIFS = new POIFSFileSystem (new FileInputStream ("file.xls"))
The NPOIFSFileSystem method is faster and consumes less memory but is read only.
I'm using Apache POI 3.9
You could map the desired part to memory and then work on it using java.nio.FileChannel.
In addition to the familiar read, write, and close operations of byte channels, this class defines the following file-specific operations:
Bytes may be read or written at an absolute position in a file in a way that does not affect the channel's current position.
A region of a file may be mapped directly into memory; for large files this is often much more efficient than invoking the usual read or write methods.
At the time of your question, there sadly wasn't a very low memory way to do it. The good news is that as of 2014-04-28 it is possible! (This code should be in 3.11 when that's released, but for now it's too new)
Now that NPOIFS supports writing, including in-place write, what you'll want to do is something like:
// Open the file, and grab the entries for the summary streams
NPOIFSFileSystem poifs = new NPOIFSFileSystem(file, false);
DocumentNode sinfDoc =
(DocumentNode)root.getEntry(SummaryInformation.DEFAULT_STREAM_NAME);
DocumentNode dinfDoc =
(DocumentNode)root.getEntry(DocumentSummaryInformation.DEFAULT_STREAM_NAME);
// Open and parse the metadata
SummaryInformation sinf = (SummaryInformation)PropertySetFactory.create(
new NDocumentInputStream(sinfDoc));
DocumentSummaryInformation dinf = (DocumentSummaryInformation)PropertySetFactory.create(
new NDocumentInputStream(dinfDoc));
// Make some metadata changes
sinf.setAuthor("Changed Author");
sinf.setTitle("Le titre \u00e9tait chang\u00e9");
dinf.setManager("Changed Manager");
// Update the metadata streams in the file
sinf.write(new NDocumentOutputStream(sinfDoc));
dinf.write(new NDocumentOutputStream(dinfDoc));
// Write out our changes
fs.writeFilesystem();
fs.close();
You ought to be able to do all of that in under 20% of the memory of the size of your file, quite possibly less than that for larger files!
(If you want to see more on this, look at the ModifyDocumentSummaryInformation example and the HPSF TestWrite unit test)
I'm currently working on a project that is done in Java, on google appengine. i have above 2000 records
Appengine does not allow files to be stored so any on-disk representation objects cannot be used. Some of these include the File class.
I want to write data and export it to a few csv files, the user to download it.
How may I do this without using any File classes? I'm not very experienced in file handling so I hope you guys can advise me.
Thanks.
Just generate the csv in memory using a StringBuffer and then use StringBuffer.toString().getBytes() to get a byte array which can then be sent to your output stream.
For instance if using a servlet in GAE:
protected void doGet(HttpServletRequest req, HttpServletResponse resp) {
StringBuffer buffer = new StringBuffer();
buffer.append("header1, header2, header3\n");
buffer.append("row1column1, row1column2, row1column3\n");
buffer.append("row2column1, row2column2, row2column3\n");
// Add more CSV data to the buffer
byte[] bytes = buffer.toString().getBytes();
// This will suggest a filename for the browser to use
resp.addHeader("Content-Disposition", "attachment; filename=\"myFile.csv\"");
resp.getOutputStream().write(bytes, 0, bytes.length);
}
More information about GAE Servlets
More information about Content-Disposition
You can store data in memory using byte arrays, stings and streams. For example,
ByteArrayOutputStream csv = new ByteArrayOutputStream();
PrintStream printer = new PrintStream(csv);
printer.println("a;b;c");
printer.println("1;2;3");
printer.close();
csv.close();
Then in your Servlet you can serve your csv.toByteArray() as a stream. Some example is given here: Implementing a simple file download servlet.
You can use OpenCSV library in Google App Engine.
I am using JavaFX media player to play an mp3 file using following code
new MediaPlayer(new Media(FileObject.toURI().toString())).play();
However now I have a requirement that I have the mp3 byte data in memory instead of an File Object. The reason is the mp3 file is encrypted and then shipped along with the program. Hence I need to decrypt the mp3 file in memory or input stream.
I could decrypt the mp3 file to an temporary file in temp directory but this would be a performance overhead and the audio content would be insecure.
From the Media Javadoc
Only HTTP, FILE, and JAR URLs are supported. If the provided URL is invalid then an exception will be thrown. If an asynchronous error occurs, the error property will be set. Listen to this property to be notified of any such errors.
I'm not personally familiar with JavaFX, but that would suggest to me that without resorting to nasty hacks, you're not going to be able to read media directly from memory. Normally for this kind of URI only interface I'd suggest registering a custom UrlStreamHandler and a custom protocol that reads from memory. But assuming that JavaDoc is correct, the JavaFX uses it's own resolution so presumably this will not work.
Given this then I suspect the only way to make this work is to provide access to the in-memory MP3 over HTTP. You could do this using Jetty or any similar embeddable servlet container. Something along the following lines:
1) Start Up Jetty as per Quick Start Guide
2) Register a servlet that looks something like below. This servlet will expose your in-memory data:
public class MagicAccessServlet extends HttpServlet {
private static final Map<String, byte[]> mediaMap = new ConcurrentHashMap();
public static String registerMedia(byte[] media) {
String key = UUID.randomUUID().toString();
mediaMap.put(key, media);
return key;
}
public static deregisterMedia(String key) {
mediaMap.remove(key);
}
public void doGet(HttpServletRequest req, HttpServletResponse resp) {
String key = req.get("key");
byte[] media = mediaMap.get(key);
resp.setContentLength(media.length);
resp.getOutputStream().write(media);
}
}
Then you can access from within your application using an http url. E.g. something like
MagicAccessServlet.registerMedia(decodedMp3);
new MediaPlayer(new Media("http://localhost:<port>/<context>/<servlet>?key=" + key)).play();
Unfortunately as the Media constructor stands I see no easy way to do this other than the temporary file approach. Note that while I agree the performance would have an overhead, if the files aren't too big (most mp3 files generally aren't) then the overhead should be minimal in this sense. And technically, decoding the content to memory also renders it insecure (though admittedly much harder to extract.)
One slightly crazy approach I did think of was to use sockets. You could setup a separate part of your application which decrypts the unencrypted content and then streams the raw mp3 bytes over a certain port on localhost. You could then provide this as a HTTP URI to the Media constructor.
I am very new to Java, and trying to use Mathematica's Java interface to access a file using memory mapping (in hope of a performance improvement).
The Mathematica code I have is (I believe) equivalent to the following Java code (based on this):
import java.io.FileInputStream;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
public class MainClass {
private static final int LENGTH = 8*100;
public static void main(String[] args) throws Exception {
MappedByteBuffer buffer = new FileInputStream("test.bin").getChannel().map(FileChannel.MapMode.READ_ONLY, 0, LENGTH);
buffer.load();
buffer.isLoaded(); // returns false, why?
}
}
I would like to use the array() method on buffer, so I am trying to load the buffers contents into memory first using load(). However, even after load(), isLoaded() returns false, and buffer.array() throws an exception: java.lang.UnsupportedOperationException
at java.nio.ByteBuffer.array(ByteBuffer.java:940).
Why doesn't the buffer load and how can I call the array() method?
My ultimate aim here is to get an array of doubles using asDoubleBuffer().array(). The method getDouble() does work correctly, but I was hoping to get this done in one go for good performance. What am I doing wrong?
As I am doing this from Mathematica, I'll post the actual Mathematica code I used too (equivalent to the above in Java):
Needs["JLink`"]
LoadJavaClass["java.nio.channels.FileChannel$MapMode"]
buffer = JavaNew["java.io.FileInputStream", "test.bin"]#getChannel[]#map[FileChannel$MapMode`READUONLY, 0, 8*100]
buffer#load[]
buffer#isLoaded[] (* returns False *)
According to Javadoc
"The content of a mapped byte buffer can change at any time, for example if the content of the corresponding region of the mapped file is changed by this program or another. Whether or not such changes occur, and when they occur, is operating-system dependent and therefore unspecified.
All or part of a mapped byte buffer may become inaccessible at any time, for example if the mapped file is truncated. An attempt to access an inaccessible region of a mapped byte buffer will not change the buffer's content and will cause an unspecified exception to be thrown either at the time of the access or at some later time. It is therefore strongly recommended that appropriate precautions be taken to avoid the manipulation of a mapped file by this program, or by a concurrently running program, except to read or write the file's content."
To me it seems to many conditions and undesirable misbehavior. Do you need particularly this class?
If you just need to read file contents in fastest way, give a try:
FileChannel fChannel = new FileInputStream(f).getChannel();
byte[] barray = new byte[(int) f.length()];
ByteBuffer bb = ByteBuffer.wrap(barray);
bb.order(ByteOrder.LITTLE_ENDIAN);
fChannel.read(bb);
It works at speed almost equal to disk system test speed.
For double you can use DoubleBuffer (with double[] array if f.length()/4 size) or just call getDouble(int) method of ByteBuffer.
in Java:
final byte[] hb; // Non-null only for heap buffers
so it is not even implemented for MappedByteBuffer but is for HeapByteBuffer.
in Android:
**
* Child class implements this method to realize {#code array()}.
*
* #see #array()
*/
abstract byte[] protectedArray();
and again not in MappedByteBuffer, but for example ByteArrayBuffer does implement the backing array.
#Override byte[] protectedArray() {
if (isReadOnly) {
throw new ReadOnlyBufferException();
}
return backingArray;
}
The point of memory map is to be off heap. A backing array would be on heap.
If you can get the FileChannel open from RandomAccessFile and then call map on the channel, you can also use the bulk get() method on the MappedByteBuffer to read into a byte[]. This copies from off heap, avoiding IO, into heap again.
buffer = fileChannel.map(FileChannel.MapMode.READ_ONLY, 0, fileChannel.size());
byte[] b = new byte[buffer.limit()];
buffer.get(b);