I've created an ARCore Session and attached an OpenGL texture id through the Session#setCameraTextureName method to display my camera data. I'd like to have access to the camera image data bytes displayed on the texture.
ARKit and Tango provide access to the image bytes for each frame but there doesn't seem to be anything that easily provides that in the ARCore API.
Is there any other way I can access the image bytes when using ARCore?
Maybe that could help you I wanted to obtain the camera view in a bitmap form. I have tested on Samsung s8.
int w=1080;
int h = 2220;
int b[]=new int[w*(0+h)];
int bt[]=new int[w*h];
IntBuffer ib = IntBuffer.wrap(b);
ib.position(0);
GLES20.glReadPixels(0, 0, w, h, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, ib);
for(int i=0, k=0; i<h; i++, k++)
{//remember, that OpenGL bitmap is incompatible with Android bitmap
//and so, some correction need.
for(int j=0; j<w; j++)
{
int pix=b[i*w+j];
int pb=(pix>>16)&0xff;
int pr=(pix<<16)&0x00ff0000;
int pix1=(pix&0xff00ff00) | pr | pb;
bt[(h-k-1)*w+j]=pix1;
}
}
sb=Bitmap.createBitmap(bt, w, h, Bitmap.Config.ARGB_8888);
For the time being, your best bet for accessing image data is probably drawing the texture to a renderbuffer and using glReadPixels into a persistent-mapped pixel unpack buffer. Use a fence sync to detect when the glReadPixels is complete.
Another option is to use a compute shader and write directly to a persistent-mapped SSBO. (Disregard persistent-mapped suggestion. I thought EXT_buffer_storage had broader support)
The later is possibly fewer copies (the renderbuffer pixels may still hit DRAM even if you invalidate it after the glReadPixels), but it's also a less-common code path and incurs render/compute changeovers so I don't have intuition about which approach would be more efficient.
As of ARCore v1.1.0, there is an API to access the image bytes for the current frame:
https://developers.google.com/ar/reference/java/com/google/ar/core/Frame.html#acquireCameraImage()
Related
I have copy pasted some code I found on stackoverflow to convert the default camera preview YUV into RGB format and then uploaded it to OpenGL for processing.
That worked fine, the issue is that most of the CPU was busy at converting the YUV images into the RGB format and it turned into the bottle neck.
I want to upload the YUV image into the GPU and then convert it into RGB in a fragment shader.
I took the same Java YUV to RGB function I found which worked on the CPU and tried to make it work on the GPU.
It turned to be quite a little nightmare, since there are several differences on doing calculations on Java and the GPU.
First, the preview image comes in byte[] in Java, but bytes are signed, so there might be negative values.
In addition, the fragment shader normally deals with [0..1] floating values for instead of a byte.
I am sure this is solveable and I almost solved it. But I spent a few hours trying to figure out what I was doing wrong and couldn't make it work.
Bottom line, I ask for someone to just write this shader function and preferably test it. For me it would be a tedious monkey job since I don't really understand why this conversion works the way it is, and I just try to mimic the same function on the GPU.
This is a very similar function to what I used on Java:
Displaying YUV Image in Android
What I did some of the job on the CPU, such as turnning the 1.5*wh bytes YUV format into a wh*YUV, as follows:
static public void decodeYUV420SP(int[] rgba, byte[] yuv420sp, int width,
int height) {
final int frameSize = width * height;
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (int) yuv420sp[yp]+127;
if ((i & 1) == 0) {
v = (int)yuv420sp[uvp++]+127;
u = (int)yuv420sp[uvp++]+127;
}
rgba[yp] = 0xFF000000+(y<<16) | (u<<8) | v;
}
}
}
I added 127 because byte is signed.
I then loaded the rgba into a OpenGL texture and tried to do the rest of the calculation on the GPU.
Any help would be appreaciated...
I used this code from wikipedia to calculate the conversion from YUV to RGB on the GPU:
private static int convertYUVtoRGB(int y, int u, int v) {
int r,g,b;
r = y + (int)1.402f*v;
g = y - (int)(0.344f*u +0.714f*v);
b = y + (int)1.772f*u;
r = r>255? 255 : r<0 ? 0 : r;
g = g>255? 255 : g<0 ? 0 : g;
b = b>255? 255 : b<0 ? 0 : b;
return 0xff000000 | (b<<16) | (g<<8) | r;
}
I converted the floats to 0.0..255.0 and then use the above code.
The part on the CPU was to rearrange the original YUV pixels into a YUV matrix(also shown in wikipdia).
Basically I used the wikipedia code and did the simplest float<->byte conersions to make it work out.
Small mistakes like adding 16 to Y or not adding 128 to U and V would give undesirable results. So you need to take care of it.
But it wasn't a lot of work once I used the wikipedia code as the base.
Converting on CPU sounds easy but I believe question is how to do it on GPU?
I did it recently in my project where I needed to get very fast QR code detection even when camera angle is 45 degrees to surface where code is printed, and it worked with great performance:
(following code is trimmed just to contain key lines, it is assumed that you have both Java and OpenGLES solid understanding)
Create a GL texture that will contain stored Camera image:
int[] txt = new int[1];
GLES20.glGenTextures(1,txt,0);
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,txt[0]);
GLES20.glTextParameterf(... set min filter to GL_LINEAR );
GLES20.glTextParameterf(... set mag filter to GL_LINEAR );
GLES20.glTextParameteri(... set wrap_s to GL_CLAMP_TO_EDGE );
GLES20.glTextParameteri(... set wrap_t to GL_CLAMP_TO_EDGE );
Pay attention that texture type is not GL_TEXTURE_2D. This is important, since only a GL_TEXTURE_EXTERNAL_OES type is supported by SurfaceTexture object, which will be used in the next step.
Setup SurfaceTexture:
SurfaceTexture surfTex = new SurfaceTeture(txt[0]);
surfTex.setOnFrameAvailableListener(this);
Above assumes that 'this' is an object that implements 'onFrameAvailable' function.
public void onFrameAvailable(SurfaceTexture st)
{
surfTexNeedUpdate = true;
// this flag will be read in GL render pipeline
}
Setup camera:
Camera cam = Camera.open();
cam.setPreviewTexture(surfTex);
This Camera API is deprecated if you target Android 5.0, so if you are, you have to use new CameraDevice API.
In your render pipeline, have following block to check if camera has frame available, and update surface texture with it. When surface texture is updated, will fill in GL texture that is linked with it.
if( surfTexNeedUpdate )
{
surfTex.updateTexImage();
surfTexNeedUpdate = false;
}
To bind GL texture which has Camera -> SurfaceTeture link to, just do this in rendering pipe:
GLES20.glBindTexture(GLES20.GL_TEXTURE_EXTERNAL_OS, txt[0]);
Goes without saying, you need to set current active texture.
In your GL shader program which will use above texture in it's fragment part, you must have first line:
#extension GL_OES_EGL_imiage_external : require
Above is a must-have.
Texture uniform must be samplerExternalOES type:
uniform samplerExternalOES u_Texture0;
Reading pixel from it is just like from GL_TEXTURE_2D type, and UV coordinates are in same range (from 0.0 to 1.0):
vec4 px = texture2D(u_Texture0, v_UV);
Once you have your render pipeline ready to render a quad with above texture and shader, just start the camera:
cam.startPreview();
You should see quad on your GL screen with live camera feed. Now you just need to grab the image with glReadPixels:
GLES20.glReadPixels(0,0,width,height,GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, bytes);
Above line assumes that your FBO is RGBA, and that bytes is already initialized byte[] array to proper size, and that width and height are size of your FBO.
And voila! You have captured RGBA pixels from camera instead of converting YUV bytes received in onPreviewFrame callback...
You can also use RGB framebuffer object and avoid alpha if you don't need it.
It is important to note that camera will call onFrameAvailable in it's own thread which is not your GL render pipeline thread, thus you should not perform any GL calls in that function.
In February 2011, Renderscript was first introduced. Since Android 3.0 Honeycomb (API 11), and definitely since Android 4.2 JellyBean (API 17), when ScriptIntrinsicYuvToRGB was added, the easiest and most efficient solution has been to use renderscript for YUV to RGB conversion. I have recently generalized this solution to handle device rotation.
I'm saving a very large PNG (25 MB or so) with Java. The problem is that while it's being generated, it's using 3+ gigabytes of memory, which is not ideal since it severely slows down systems with low memory.
The code I'm working with needs to combine a set of tiled images into a single image; in other words, I have nine images (PNG):
A1 A2 A3
B1 B2 B3
C1 C2 C3
which need to be combined into a single image.
The code I'm using is this:
image = new BufferedImage(width, height, height, BufferedImage.TYPE_INT_ARGB_PRE);
g2d = image.createGraphics();
g2d.setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BICUBIC);
// draw the 9 images on here at their proper positions...
// save image
g2d.dispose();
File file = getOutputFile();
ImageIO.write(image, "png", file);
Is there a way to make and save an image without having the entire image in memory?
Edit:
To draw the images, I'm doing this in a loop:
BufferedImage tile = ImageIO.read(new File("file.png"));
g2d.drawImage(tile, x, y, w, h);
This is being repeated many times (it's usually about 25x25, but sometimes more), so if there is even a small memory leak here, that could be causing the problem.
You can also take a look at this PNGJ library (disclaimer: I coded it), it allows to save a PNG image line by line.
ImageIO.write(image, "png", file); is internally using com.sun.imageio.plugins.png.PNGImageWriter. That method and that writer expect image to be a rendered image but PNG writting is done by 'bands' so you can make a subclass of RenderedImage that generates the requested bands of the composed large image as the writer ask for that bands to the image.
From PNGImageWriter class:
private void encodePass(ImageOutputStream os,
RenderedImage image,
int xOffset, int yOffset,
int xSkip, int ySkip) throws IOException {
// (...)
for (int row = minY + yOffset; row < minY + height; row += ySkip) {
Rectangle rect = new Rectangle(minX, row, width, 1); // <--- *1
Raster ras = image.getData(rect); // <--- *2
*2 I think this is the only place where the writer reads pixels from you image. You should make a getData(rect) method that computes that rect joining 3 bands from 3 images into one.
*1 As you see it reads bands with a height of 1 pixel.
If the things are as I think you should only need to compose 3 images at a time. There would be no need for the other 6 to be in memory.
I know it is not an easy solution but it might help you if you don't find anything easier.
Would using an external tool be an option? I remember using ImageMagick for similar purpose, you would need to save your smaller images first.
It's extremely easy to get the Bitmap data in the NDK when working with Android 2.2, but with 2.1 and lower, the AndroidBitmap_lockPixels function is not available. I've been searching for the past few hours, but nothing has worked.
How can I access the pixel data of a bitmap without using that function?
Create empty bitmap with dimensions of original image and ARGB_8888 format:
int width = src.getWidth();
int height = src.getHeight();
Bitmap dest = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
Copy pixels from source bitmap to the int array:
int[] pixels = new int[width * height];
src.getPixels(pixels, 0, width, 0, 0, width, height);
And set these pixels to destination bitmap:
dest.setPixels(pixels, 0, width, 0, 0, width, height);
Create an IntBuffer in your Java code and pass the array down to your native library:
// this is called from native code
buffer = IntBuffer.allocate(width*height);
return buffer.array();
Use GetIntArrayElements to get an jint* and write to the array:
jint * arr = env->GetIntArrayElements((jintArray)bufferArray, NULL);
Write to the array and when finished, release:
env->ReleaseIntArrayElements((jintArray)bufferArray, arr, 0);
Notify the Java code that the array has been updated and use Canvas.drawBitmap() to draw the IntBuffer:
canvas.drawBitmap(buffer.array(), ....);
To draw to a Bitmap, initialize the canvas with the bitmap
... new Canvas(bitmap)
Someone else just asked the same question - I'll just link to it to avoid duplicating my answer:
Android rendering to live wallpapers
In any event, you probably don't want to copy the bitmap data every time you need to exchange it between Java and JNI code, so if your code is performance sensitive, this may be your only option on Android 2.1 and lower.
I'm working on a project that takes video input from a webcam and displays regions of motion to the user. My "beta" attempt at this project was to use the Java Media Framework to retrieve the webcam feed. Through some utility functions, JMF conveniently returns webcam frames as BufferedImages, which I built a significant amount of framework around to process. However, I soon realized that JMF isn't well supported by Sun/Oracle anymore, and some of the higher webcam resolutions (720p) are not accessible through the JMF interface.
I'd like to continue processing frames as BufferedImages, and use OpenCV (C++) to source the video feed. Using OpenCV's framework alone, I've found that OpenCV does a good job of efficiently returning high-def webcam frames and painting them to screen.
I figured it would be pretty straightforward to feed this data into Java and achieve the same efficiency. I just finished writing the JNI DLL to copy this data into a BufferedImage and return it to Java. However, I'm finding that the amount of data copying I'm doing is really hindering performance. I'm targeting 30 FPS, but it takes roughly 100 msec alone to even copy the data from the char array returned by OpenCV into a Java BufferedImage. Instead, I'm seeing about 2-5 FPS.
When returning a frame capture, OpenCV provides a pointer to a 1D char array. This data needs to be provided to Java, and apparently I don't have the time to copy any of it.
I need a better solution to get these frame captures into a BufferedImage. A few solutions I'm considering, none of which I think are very good (fairly certain they would also perform poorly):
(1) Override BufferedImage, and return pixel data from various BufferedImage methods by making native calls to the DLL. (Instead of doing the array copying at once, I return individual pixels as requested by the calling code). Note that calling code typically needs all pixels in the image to paint the image or process it, so this individual pixel-grab operation would be implemented in a 2D for-loop.
(2) Instruct the BufferedImage to use a java.nio.ByteBuffer to somehow directly access data in the char array returned by OpenCV. Would appreciate any tips as to how this is done.
(3) Do everything in C++ and forget Java. Well well, yes this does sound like the most logical solution, however I will not have time to start this many-month project from scratch.
As of now, my JNI code has been written to return the BufferedImage, however at this point I'm willing to accept the return of a 1D char array and then put it into a BufferedImage.
By the way... the question here is: What is the most efficient way to copy a 1D char array of image data into a BufferedImage?
Provided is the (inefficient) code that I use to source image from OpenCV and copy into BufferedImage:
JNIEXPORT jobject JNICALL Java_graphicanalyzer_ImageFeedOpenCV_getFrame
(JNIEnv * env, jobject jThis, jobject camera)
{
//get the memory address of the CvCapture device, the value of which is encapsulated in the camera jobject
jclass cameraClass = env->FindClass("graphicanalyzer/Camera");
jfieldID fid = env->GetFieldID(cameraClass,"pCvCapture","I");
//get the address of the CvCapture device
int a_pCvCapture = (int)env->GetIntField(camera, fid);
//get a pointer to the CvCapture device
CvCapture *capture = (CvCapture*)a_pCvCapture;
//get a frame from the CvCapture device
IplImage *frame = cvQueryFrame( capture );
//get a handle on the BufferedImage class
jclass bufferedImageClass = env->FindClass("java/awt/image/BufferedImage");
if (bufferedImageClass == NULL)
{
return NULL;
}
//get a handle on the BufferedImage(int width, int height, int imageType) constructor
jmethodID bufferedImageConstructor = env->GetMethodID(bufferedImageClass,"<init>","(III)V");
//get the field ID of BufferedImage.TYPE_INT_RGB
jfieldID imageTypeFieldID = env->GetStaticFieldID(bufferedImageClass,"TYPE_INT_RGB","I");
//get the int value from the BufferedImage.TYPE_INT_RGB field
jint imageTypeIntRGB = env->GetStaticIntField(bufferedImageClass,imageTypeFieldID);
//create a new BufferedImage
jobject ret = env->NewObject(bufferedImageClass, bufferedImageConstructor, (jint)frame->width, (jint)frame->height, imageTypeIntRGB);
//get a handle on the method BufferedImage.getRaster()
jmethodID getWritableRasterID = env->GetMethodID(bufferedImageClass, "getRaster", "()Ljava/awt/image/WritableRaster;");
//call the BufferedImage.getRaster() method
jobject writableRaster = env->CallObjectMethod(ret,getWritableRasterID);
//get a handle on the WritableRaster class
jclass writableRasterClass = env->FindClass("java/awt/image/WritableRaster");
//get a handle on the WritableRaster.setPixel(int x, int y, int[] rgb) method
jmethodID setPixelID = env->GetMethodID(writableRasterClass, "setPixel", "(II[I)V"); //void setPixel(int, int, int[])
//iterate through the frame we got above and set each pixel within the WritableRaster
jintArray rgbArray = env->NewIntArray(3);
jint rgb[3];
char *px;
for (jint x=0; x < frame->width; x++)
{
for (jint y=0; y < frame->height; y++)
{
px = frame->imageData+(frame->widthStep*y+x*frame->nChannels);
rgb[0] = abs(px[2]); // OpenCV returns BGR bit order
rgb[1] = abs(px[1]); // OpenCV returns BGR bit order
rgb[2] = abs(px[0]); // OpenCV returns BGR bit order
//copy jint array into jintArray
env->SetIntArrayRegion(rgbArray,0,3,rgb); //take values in rgb and move to rgbArray
//call setPixel() this is a copy operation
env->CallVoidMethod(writableRaster,setPixelID,x,y,rgbArray);
}
}
return ret; //return the BufferedImage
}
There is another option if you wish to make your code really fast and still use Java. The AWT windowing toolkit has a direct native interface you can use to draw to an AWT surface using C or C++. Thus, there would be no need to copy anything to Java, as you could render directly from the buffer in C or C++. I am not sure of the specifics on how to do this because I have not looked at it in a while, but I know that it is included in the standard JRE distribution. Using this method, you could probably approach the FPS limit of the camera if you wished, rather than struggling to reach 30 FPS.
If you want to research this further I would start here and here.
Happy Programming!
I would construct the RGB int array required by BufferedImage and then use a single call to
void setRGB(int startX, int startY, int w, int h, int[] rgbArray, int offset, int scansize)
to set the entire image data array at once. Or at least, large portions of it.
Without having timed it, I would suspect that it's the per-pixel calls to
env->SetIntArrayRegion(rgbArray,0,3,rgb);
env->CallVoidMethod(writableRaster,setPixelID,x,y,rgbArray);
which are taking the lion's share of the time.
EDIT: It will be likely the method invocations rather than manipulation of memory, per se, that is taking the time. So build data in your JNI code and copy it in blocks or a single hit to the Java image. Once you create and pin a Java int[] you can access it via native pointers. Then one call to setRGB will copy the array into your image.
Note: You do still have to copy the data at least once, but doing all pixels in one hit via 1 function call will be vastly more efficient than doing them individually via 2 x N function calls.
EDIT 2:
Reviewing my JNI code, I have only ever used byte arrays, but the principles are the same for int arrays. Use:
NewIntArray
to create an int array, and
GetIntArrayElements
to pin it and get a pointer, and when you are done,
ReleaseIntArrayElements
to release it, remembering to use the flag to copy data back to Java's memory heap.
Then, you should be able to use your Java int array handle to invoke the setRGB function.
Remember also that this is actually setting RGBA pixels, so 4 channels, including alpha, not just three (the RGB names in Java seem to predate alpha channel, but most of the so-named methods are compatible with a 32 bit value).
As a secondary consideration, if the only difference between the image data array returned by OpenCV and what is required by Java is the BGR vs RGB, then
px = frame->imageData+(frame->widthStep*y+x*frame->nChannels);
rgb[0] = abs(px[2]); // OpenCV returns BGR bit order
rgb[1] = abs(px[1]); // OpenCV returns BGR bit order
rgb[2] = abs(px[0]); // OpenCV returns BGR bit order
is a relatively inefficient way to convert them. Instead you could do something like:
uint32 px = frame->imageData+(frame->widthStep*y+x*frame->nChannels);
javaArray[ofs]=((px&0x00FF0000)>>16)|(px&0x0000FF00)|((px&0x000000FF)<<16);
(note my C code is rusty, so this might not be entirely valid, but it shows what is needed).
Managed to speed up the process using an NIO ByteBuffer.
On the C++ JNI side...
JNIEXPORT jobject JNICALL Java_graphicanalyzer_ImageFeedOpenCV_getFrame
(JNIEnv * env, jobject jThis, jobject camera)
{
//...
IplImage *frame = cvQueryFrame(pCaptureDevice);
jobject byteBuf = env->NewDirectByteBuffer(frame->imageData, frame->imageSize);
return byteBuf;
}
and on the Java side...
void getFrame(Camera cam)
{
ByteBuffer frameData = cam.getFrame(); //NATIVE call
byte[] imgArray = new byte[frame.data.capacity()];
frameData.get(imgArray); //although it seems like an array copy, this call returns very quickly
DataBufferByte frameDataBuf = new DataBufferByte(imgArray,imgArray.length);
//determine image sample model characteristics
int dataType = DataBuffer.TYPE_BYTE;
int width = cam.getFrameWidth();
int height = cam.getFrameHeight();
int pixelStride = cam.getPixelStride();
int scanlineStride = cam.getScanlineStride();
int bandOffsets = new int[] {2,1,0}; //BGR
//create a WritableRaster with the DataBufferByte
PixelInterleavedSampleModel pism = new PixelInterleavedSampleModel
(
dataType,
width,
height,
pixelStride,
scanlineStride,
bandOffsets
);
WritableRaster raster = new ImgFeedWritableRaster( pism, frameDataBuf, new Point(0,0) );
//create the BufferedImage
ColorSpace cs = ColorSpace.getInstance(ColorSpace.CS_sRGB);
ComponentColorModel cm = new ComponentColorModel(cs, false, false, Transparency.OPAQUE, DataBuffer.TYPE_BYTE);
BufferedImage newImg = new BufferedImage(cm,raster,false,null);
handleNewImage(newImg);
}
Using the java.nio.ByteBuffer, I can quickly address the char array returned by the OpenCV code without (apparently) doing much gruesome array copying.
I have some png files that I am applying a color to. The color changes depending on a user selection. I change the color via 3 RGB values set from another method. The png files are a random shape with full transparency outside the shape. I don't want to modify the transparency, only the RGB value. Currently, I'm setting the RGB values pixel by pixel (see code below).
I've come to realize this is incredibly slow and possibly just not efficient enough do in an application. Is there a better way I could do this?
Here is what I am currently doing. You can see that the pixel array is enormous for an image that takes up a decent part of the screen:
public void foo(Component component, ComponentColor compColor, int userColor) {
int h = component.getImages().getHeight();
int w = component.getImages().getWidth();
mBitmap = component.getImages().createScaledBitmap(component.getImages(), w, h, true);
int[] pixels = new int[h * w];
//Get all the pixels from the image
mBitmap[index].getPixels(pixels, 0, w, 0, 0, w, h);
//Modify the pixel array to the color the user selected
pixels = changeColor(compColor, pixels);
//Set the image to use the new pixel array
mBitmap[index].setPixels(pixels, 0, w, 0, 0, w, h);
}
public int[] changeColor(ComponentColor compColor, int[] pixels) {
int red = compColor.getRed();
int green = compColor.getGreen();
int blue = compColor.getBlue();
int alpha;
for (int i=0; i < pixels.length; i++) {
alpha = Color.alpha(pixels[i]);
if (alpha != 0) {
pixels[i] = Color.argb(alpha, red, green, blue);
}
}
return pixels;
}
Have you looked at the functions available in Bitmap? Something like extractAlpha sounds like it might be useful. You an also look at the way functions like that are implemented in Android to see how you could adapt it to your particular case, if it doesn't exactly meet your needs.
The answer that worked for me was a write up Square did here Transparent jpegs
They provide a faster code snippet for doing this exact thing. I tried extractAlpha and it didn't work but Square's solution did. Just modify their solution to instead modify the color bit and not the alpha bit.
i.e.
pixels[x] = (pixels[x] & 0xFF000000) | (color & 0x00FFFFFF);