Develop Reference

ParcelFileDescriptor.createPipe in JNI

I'm trying to figure out how I can pass a stream of data within ContentProvider.openFile. The data to be sent is created in JNI. I tried createPipe with a transfer thread but I had a ton of trouble with broken pipes. So I thought I might just pass the 'write' pipe to JNI and write the data directly to it.
Java:
ParcelFileDescriptor[] pipe = ParcelFileDescriptor.createPipe();
boolean result = ImageProcessor.getThumb(fd/*source fd*/, pipe[1].getFd()); //JNI call (formerly returned a byte[])
return pipe[0];
C:
unsigned char* jpeg = NULL;
unsigned long jpegSize = 0;
getThumbnail(env, &jpeg, &jpegSize, rawProcessor); // Populates jpeg thumb, works when converted to byte[] in second segment
FILE* out = fdopen(dest, "wb");
int written = fwrite(jpeg, 1, jpegSize, out);
return TRUE;
When I convert to byte[] everything works fine, just not within a ContentProvider obviously:
jbyteArray thumb = env->NewByteArray(jpegSize);
env->SetByteArrayRegion(thumb, 0, jpegSize, (jbyte *) jpeg);
free(jpeg);
return thumb;
When I debug it gets to fwrite then the stack trace just seems to disappear. Never hits return TRUE or return pipe[0], but also doesn't crash or throw. Very strange...
Has anyone done something similar? Is it sufficient to simply write binary to the "write" pipe? Am I doing anything fundamentally wrong here? Thanks.
Update (after discussion with #pskink)
I tried implementing the PipeDataWriter. I used FileProvider.java as an example.
#Override
public void writeDataToPipe(#NonNull ParcelFileDescriptor output, #NonNull Uri uri, #NonNull String mimeType, #Nullable Bundle opts, #Nullable byte[] args)
{
try (FileOutputStream fout = new FileOutputStream(output.getFileDescriptor()))
{
fout.write(args, 0, args.length);
}
catch (IOException e)
{
Log.e(TAG, "Failed transferring", e);
}
}
byte[] rawData = ImageUtil.getRawThumb(fd.getParcelFileDescriptor().getFd());
return openPipeHelper(Uri.parse("invalid"), "image/jpg", null, rawData, this);
However, I'm getting the same errors I got when I used the transfer thread above:
java.io.IOException: write failed: EBADF (Bad file descriptor)
at libcore.io.IoBridge.write(IoBridge.java:498)
at java.io.FileOutputStream.write(FileOutputStream.java:186)
at
com.anthonymandra.content.MetaProvider.writeDataToPipe(MetaProvider.java:273)
and
java.io.IOException: write failed: EPIPE (Broken pipe)
at libcore.io.IoBridge.write(IoBridge.java:498)
at java.io.FileOutputStream.write(FileOutputStream.java:186)
at
com.anthonymandra.content.MetaProvider.writeDataToPipe(MetaProvider.java:273)
When I stepped through to make sure the data was fine for the images I found that everything loaded fine. It looks to me like this is actually a thread safety issue.

There were actually a bunch of things going wrong that all rolled up into a confusing mess:
I wasn't closing the ParcelFileDescriptor in a finally.
I use Glide for an image cache and it uses two fetchers when you load a Uri, meaning openFile was being called twice per file.
(2) caused endless broken pipe errors.
StrictMode was killing the app because of (1) and I missed it in the flurry of errors from (3).

LibGdx android accessing .txt files and turning them to Strings

This is a libgdx project using gradle if that matters. I have a class that implements screen where I want to draw a string to the display. on Desktop the following code works.
try {
string = readFile("bin/001.txt",Charset.defaultCharset());
} catch (IOException e) {
e.printStackTrace();
}
And then I can use that string in a Label for a stage. However on the android version I use "assets/001.txt" and get the error VFY: unable to resove static method 11542: Ljava/nio/file/Paths;.g
I then researched it and found a suggestion here
FileInputStream fis;
fis = openFileInput("test.txt");
StringBuffer fileContent = new StringBuffer("");
byte[] buffer = new byte[1024];
while ((n = fis.read(buffer)) != -1)
{
fileContent.append(new String(buffer, 0, n));
}
This did not work either because openFileInput is undefined because "Those are methods defined on the Context class" however
"If you are using LibGDX, you need to forego the concept of Activities and Views, since your entire game will now just be a single Activity. To have a main menu when you game starts up doesn't switch the Activity, but just presents a different set of objects to be rendered to the same Activity."
And now I'm lost

To read a file in LibGdx you need to abstract from the underlying. This is because the same application will work in several systems (Desktop, Android, Web, ...).
So to read a file you need to use:
FileHandle file = Gdx.files.internal("myfile.txt");
String text = file.readString();
The path and location of the files can be summarized in the documentation

Problems with facebooks conceal library

I'm having issues with reading decrypted data from conceal. It looks like I can't correctly finish streaming.
I pretend there is some issue with conceal, because of when I switch my proxyStream (just the encryption part) to not run it through conceal, everything works as expected. I'm also assuming that writing is ok, there is no exception whatsoever and I can find the encrypted file on disk.
I'm proxying my data through contentprovider to allow other apps read decrypted data when the user wants it. (sharing,...)
In my content provider I'm using the openFile method to allow contentResolvers read the data
#Override
public ParcelFileDescriptor openFile(Uri uri, String mode) throws FileNotFoundException {
try {
ParcelFileDescriptor[] pipe = ParcelFileDescriptor.createPipe();
String name = uri.getLastPathSegment();
File file = new File(name);
InputStream fileContents = mStorageProxy.getDecryptInputStream(file);
ParcelFileDescriptor.AutoCloseOutputStream stream = new ParcelFileDescriptor.AutoCloseOutputStream(pipe[1]);
PipeThread pipeThread = new PipeThread(fileContents, stream);
pipeThread.start();
return pipe[0];
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
I guess in the Facebook app Facebook android team could be rather using a standard query() method with a byte array sent in MediaStore.MediaColumns() which is not suitable for me because of I'm not only encrypting media files and I also like the approach of streams better.
This is how I'm reading from the Inpustream. It's basically a pipe between two parcelFileDescriptors. The inputstream comes from conceal and it is a FileInputstream wrapped into a BufferedInputStream originaly.
static class PipeThread extends Thread {
InputStream input;
OutputStream out;
PipeThread(InputStream inputStream, OutputStream out) {
this.input=inputStream;
this.out=out;
}
#Override
public void run() {
byte[] buf=new byte[1024];
int len;
try {
while ((len=input.read(buf)) > 0) {
out.write(buf, 0, len);
}
input.close();
out.flush();
out.close();
}
catch (IOException e) {
Log.e(getClass().getSimpleName(),
"Exception transferring file", e);
}
}
}
I've tried other methods how to read the stream, so it really shouldn't be the issue.
Finally here's the exception I'm constantly ending up with. Do you know what could be the issue? It points to native calls, which I got lost in..
Exception transferring file
com.facebook.crypto.cipher.NativeGCMCipherException: decryptFinal
at com.facebook.crypto.cipher.NativeGCMCipher.decryptFinal(NativeGCMCipher.java:108)
at com.facebook.crypto.streams.NativeGCMCipherInputStream.ensureTagValid(NativeGCMCipherInputStream.java:126)
at com.facebook.crypto.streams.NativeGCMCipherInputStream.read(NativeGCMCipherInputStream.java:91)
at com.facebook.crypto.streams.NativeGCMCipherInputStream.read(NativeGCMCipherInputStream.java:76)
EDIT:
It looks like the stream is working ok, but what fails is the last iteration of reading from it. As I'm using buffer it seems like the fact that the buffer is bigger then the amount of remaiming data is causing the issue. I've been looking into sources of conceal and it seems to be ok from this regard there. Couldn't it be failing somewhere in the native layer?
Note: I've managed to get the decrypted file except its final chunk of bytes..So I have for example an incomplete image file (with last few thousands of pixels not being displayed)

From my little experience with conceal, I have noticed that, only the same application that encrypts a file could decrypt it successfully irrespective whether it has the same package or not. Be sure to put this in mind

This was resolved in https://github.com/facebook/conceal/issues/24. For posterity's sake, the problem here is that the author forgot to call close() on the output stream.

Accessing Windows disks directly with Java NIO

I am using a library that uses Java NIO in order to directly map files to memory, but I am having trouble reading disks directly.
I can read the disks directly using FileInputStream with UNC, such as
File disk = new File("\\\\.\\PhysicalDrive0\\");
try (FileInputStream fis = new FileInputStream(disk);
BufferedInputStream bis = new BufferedInputStream(fis)) {
byte[] somebytes = new byte[10];
bis.read(somebytes);
} catch (Exception ex) {
System.out.println("Oh bother");
}
However, I can't extend this to NIO:
File disk = new File("\\\\.\\PhysicalDrive0\\");
Path path = disk.toPath();
try (FileChannel fc = FileChannel.open(path, StandardOpenOption.READ)){
System.out.println("No exceptions! Yay!");
} catch (Exception ex) {
System.out.println("Oh bother");
}
The stacktrace (up to the cause) is:
java.nio.file.FileSystemException: \\.\PhysicalDrive0\: The parameter is incorrect.
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:334)
at hdreader.HDReader.testcode(HDReader.java:147)
I haven't been able to find a solution, though I saw something close on How to access specific raw data on disk from java. The answer by Daniel Alder suggesting the use of GLOBALROOT seems to be relevant, as the answer uses FileChannel in the answer, but I can't seem to find the drive using this pattern. Is there a way to list all devices under GLOBALROOT or something like that?
At the moment I am looking at replacing uses of NIO with straight InputStreams, but I want to avoid this if I can. Firstly, NIO was used for a reason, and secondly, it runs through a lot of code and will require a lot of work. Finally, I'd like to know how to implement something like Daniel's solution so that I can write to devices or use NIO in the future.
So in summary: how can I access drives directly with Java NIO (not InputStreams), and/or is there a way to list all devices accessible through GLOBALROOT so that I might use Daniel Alser's solution?
Summary of Answers:
I have kept the past edits (below) to avoid confusion. With the help of EJP and Apangin I think I have a workable solution. Something like
private void rafMethod(long posn) {
ByteBuffer buffer = ByteBuffer.allocate(512);
buffer.rewind();
try (RandomAccessFile raf = new RandomAccessFile(disk.getPath(), "r");
SeekableByteChannel sbc = raf.getChannel()) {
sbc.read(buffer);
} catch (Exception ex) {
System.out.println("Oh bother: " + ex);
ex.printStackTrace();
}
return buffer;
}
This will work as long as the posn parameter is a multiple of the sector size (set at 512 in this case). Note that this also works with the Channels.newChannel(FileInputStream), which seems to always return a SeekableByteStream in this case and it appears it is safe to cast it to one.
From quick and dirty testing it appears that these methods truly do seek and don't just skip. I searched for a thousand locations at the start of my drive and it read them. I did the same but added an offset of half of the disk size (to search the back of the disk). I found:
Both methods took almost the same time.
Searching the start or the end of the disk did not affect time.
Reducing the range of the addresses did reduce time.
Sorting the addresses did reduce time, but not by much.
This suggests to me that this is truly seeking and not merely reading and skipping (as a stream tends to). The speed is still terrible at this stage and it makes my hard drive sound like a washing machine, but the code was designed for a quick test and has yet to be made pretty. It may still work fine.
Thanks to both EJP and Apangin for the help. Read more in their respective answers.
Edit:
I have since run my code on a Windows 7 machine (I didn't have one originally), and I get a slightly different exception (see below). This was run with admin privileges, and the first piece of code still works under the same conditions.
java.nio.file.FileSystemException: \\.\PhysicalDrive0\: A device attached to the system is not functioning.
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at testapp.TestApp.doStuff(TestApp.java:30)
at testapp.TestApp.main(TestApp.java:24)
Edit 2:
In response to EJP, I have tried:
byte[] bytes = new byte[20];
ByteBuffer bb = ByteBuffer.wrap(bytes);
bb.rewind();
File disk = new File("\\\\.\\PhysicalDrive0\\");
try (FileInputStream fis = new FileInputStream(disk);
ReadableByteChannel rbc = Channels.newChannel(new FileInputStream(disk))) {
System.out.println("Channel created");
int read = rbc.read(bb);
System.out.println("Read " + read + " bytes");
System.out.println("No exceptions! Yay!");
} catch (Exception ex) {
System.out.println("Oh bother: " + ex);
}
When I try this I get the following output:
Channel created
Oh bother: java.io.IOException: The parameter is incorrect
So it appears that I can create a FileChannel or ReadableByteChannel, but I can't use it; that is, the error is simply deferred.

When accessing physical drive without buffering, you can read only complete sectors. This means, if a sector size is 512 bytes, you can read only multiple of 512 bytes. Change your buffer length to 512 or 4096 (whatever your sector size is) and FileChannel will work fine:
ByteBuffer buf = ByteBuffer.allocate(512);
try (RandomAccessFile raf = new RandomAccessFile("\\\\.\\PhysicalDrive0", "r");
FileChannel fc = raf.getChannel()) {
fc.read(buf);
System.out.println("It worked! Read bytes: " + buf.position());
} catch (Exception e) {
e.printStackTrace();
}
See Alignment and File Access Requirements.
Your original FileInputStream code works obviously because of BufferedInputStream which has the default buffer size of 8192. Take it away - and the code will fail with the same exception.

Using NIO your original code only needs to change very slightly.
Path disk = Paths.get("d:\\.");
try (ByteChannel bc = Files.newByteChannel(disk, StandardOpenOption.READ)) {
ByteBuffer buffer = ByteBuffer.allocate(10);
bc.read(buffer);
} catch (Exception e){
e.printStackTrace();
}
Is fine, workable code, but I get an access denied error in both your version and mine.

Run this as administrator. It really does work, as it's only a thin wrapper over java.io:
try (FileInputStream fis = new FileInputStream(disk);
ReadableByteChannel fc = Channels.newChannel(fis))
{
System.out.println("No exceptions! Yay!");
ByteBuffer bb = ByteBuffer.allocate(4096);
int count = fc.read(bb);
System.out.println("read count="+count);
}
catch (Exception ex)
{
System.out.println("Oh bother: "+ex);
ex.printStackTrace();
}
EDIT If you need random access, you're stuck with RandomAccessFile. There's no mapping from that via Channels. But the solution above isn't NIO anyway, just a Java NIO layer over FileInput/OutputStream.

How to implement a Java compiler and DEX converter into an Android app?

While trying to find an answer to Android Jasper Reporting I found out that there are two other questions to be answered therefor, which I been asked to ask as a question, not as an answer ;):
My questions are now: "Is there any compiler to use directly on the device" AND "how to execute such without rooting the device.
If anybody could give me a hint I would really appreciate it...
I looked a little time forward on this approach, and found apps which makes it possible to create APKs directly on an Android device which is NOT rooted:
TerminalIDE - https://play.google.com/store/apps/details?id=com.spartacusrex.spartacuside&hl=de
JavaIDEdroid - http://code.google.com/p/java-ide-droid/
AIDE - https://play.google.com/store/apps/details?id=com.aide.ui&hl=en
Looks like they're using the compiler from eclipse and a ported dex converter. Now I'm trying to figure out how to do the same.
Sure: get the source code and look into it. But while I'm having curious problems to get a connection to the servers and trying to solve it, I follow the plea to ask this question here. Hoping both to help others with it and also getting an answer for myself ;)
I took the org.eclipse.jdt.core_3.7.3.v20120119-1537.jar from the plugin directory of my indigo and tried following code:
org.eclipse.jdt.internal.compiler.batch.Main ecjMain = new org.eclipse.jdt.internal.compiler.batch.Main(new PrintWriter(System.out), new PrintWriter(System.err), false/*noSystemExit*/, null, progress);
System.err.println("compiling...");
ecjMain.compile(new String[] {"-classpath", "/system/framework", storage.getAbsolutePath()+"/Test.java"});
ecjMain.compile(new String[] {storage.getAbsolutePath()+"/Test.java"});
System.err.println("compile succeeded!!!");
Sometimes the Exception was thrown that java.lang.Object could not be found and othertimes it stuck doing nothing while heating up my processor with 100% usage ... ...
At this time i could not figure out what is happening and why. And in cause that i have other work to do this part has to wait a little.

I succeeded after taking inspiration from source of JavaIDEdroid and realizing that I'm dumb (for a time I tried to uses the compiler with the dexified framework classes on the device - which naturtally could not work).
After i succeeded compiling my Test.java with a copy of ADTs android-jar on sdcard I just had to load the classes with the DexClassLoader.
While informing myselft about how to do that I found this nice article Custom Class Loading in Dalvik which inspired me at least to write this piece of code:
File storage = getDir("all41", Context.MODE_PRIVATE);
System.err.println("copying the android.jar from asssets to the internal storage to make it available to the compiler");
BufferedInputStream bis = null;
OutputStream dexWriter = null;
int BUF_SIZE = 8 * 1024;
try {
bis = new BufferedInputStream(getAssets().open("android.jar"));
dexWriter = new BufferedOutputStream(
new FileOutputStream(storage.getAbsolutePath() + "/android.jar"));
byte[] buf = new byte[BUF_SIZE];
int len;
while((len = bis.read(buf, 0, BUF_SIZE)) > 0) {
dexWriter.write(buf, 0, len);
}
dexWriter.close();
bis.close();
} catch (Exception e) {
System.err.println("Error while copying from assets: " + e.getMessage());
e.printStackTrace();
}
System.err.println("instantiating the compiler and compiling the java file");
org.eclipse.jdt.internal.compiler.batch.Main ecjMain = new org.eclipse.jdt.internal.compiler.batch.Main(new PrintWriter(System.out), new PrintWriter(System.err), false/*noSystemExit*/, null);
ecjMain.compile(new String[] {"-classpath", storage.getAbsolutePath()+"/android.jar", Environment.getExternalStorageDirectory().getAbsolutePath() + "/Test.java"});
System.err.println("calling DEX and dexifying the test class");
com.android.dx.command.Main.main(new String[] {"--dex", "--output=" + storage.getAbsolutePath() + "/Test.zip", Environment.getExternalStorageDirectory().getAbsolutePath() + "/./Test.class"});
System.err.println("instantiating DexClassLoader, loading class and invoking toString()");
DexClassLoader cl = new DexClassLoader(storage.getAbsolutePath() + "/Test.zip", storage.getAbsolutePath(), null, getClassLoader());
try {
Class libProviderClazz = cl.loadClass("Test");
Object instance = libProviderClazz.newInstance();
System.err.println(instance.toString());
} catch (Exception e) {
System.err.println("Error while instanciating object: " + e.getMessage());
e.printStackTrace();
}
The Test.java only contains one method:
public String toString() {
return "Hallo Welt!";
}
To get it running you need the jars jdt-compiler-x.x.x.jar (found in plugins directory of eclipse) and dx.jar (found in directory platform-tools/lib of Android SDK)
Not really hard ;) And now I will find out what to change in source of JasperReports to get it work on our beloved Android devices :D