I've created an ARCore Session and attached an OpenGL texture id through the Session#setCameraTextureName method to display my camera data. I'd like to have access to the camera image data bytes displayed on the texture.
ARKit and Tango provide access to the image bytes for each frame but there doesn't seem to be anything that easily provides that in the ARCore API.
Is there any other way I can access the image bytes when using ARCore?
Maybe that could help you I wanted to obtain the camera view in a bitmap form. I have tested on Samsung s8.
int w=1080;
int h = 2220;
int b[]=new int[w*(0+h)];
int bt[]=new int[w*h];
IntBuffer ib = IntBuffer.wrap(b);
ib.position(0);
GLES20.glReadPixels(0, 0, w, h, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, ib);
for(int i=0, k=0; i<h; i++, k++)
{//remember, that OpenGL bitmap is incompatible with Android bitmap
//and so, some correction need.
for(int j=0; j<w; j++)
{
int pix=b[i*w+j];
int pb=(pix>>16)&0xff;
int pr=(pix<<16)&0x00ff0000;
int pix1=(pix&0xff00ff00) | pr | pb;
bt[(h-k-1)*w+j]=pix1;
}
}
sb=Bitmap.createBitmap(bt, w, h, Bitmap.Config.ARGB_8888);
For the time being, your best bet for accessing image data is probably drawing the texture to a renderbuffer and using glReadPixels into a persistent-mapped pixel unpack buffer. Use a fence sync to detect when the glReadPixels is complete.
Another option is to use a compute shader and write directly to a persistent-mapped SSBO. (Disregard persistent-mapped suggestion. I thought EXT_buffer_storage had broader support)
The later is possibly fewer copies (the renderbuffer pixels may still hit DRAM even if you invalidate it after the glReadPixels), but it's also a less-common code path and incurs render/compute changeovers so I don't have intuition about which approach would be more efficient.
As of ARCore v1.1.0, there is an API to access the image bytes for the current frame:
https://developers.google.com/ar/reference/java/com/google/ar/core/Frame.html#acquireCameraImage()
I am trying to save yuv 420 preview frames obtained in android camera2 to jpeg. The only way I found of doing it was to convert the yuv420 to nv21, construct a yuvimage and then use the compresstojpeg method to get the jpeg. In order to convert from yuv420 to jpeg I am using the logic below
Image.Plane Y = img.getPlanes()[0];
Image.Plane U = img.getPlanes()[2];
Image.Plane V = img.getPlanes()[1];
int Yb = Y.getBuffer().remaining();
int Ub = U.getBuffer().remaining();
int Vb = V.getBuffer().remaining();
byte[] data = new byte[Yb + Ub + Vb];
Y.getBuffer().get(data, 0, Yb);
U.getBuffer().get(data, Yb, Ub);
V.getBuffer().get(data, Yb + Ub, Vb);
YuvImage yuvImage = new YuvImage(data, ImageFormat.NV21,
mPreviewSize.getWidth(), mPreviewSize.getHeight(), null);
ByteArrayOutputStream out = new ByteArrayOutputStream();
yuvImage.compressToJpeg(new Rect(0, 0,
mPreviewSize.getWidth(), mPreviewSize.getHeight()),
100, out);
However this results in getting green images for certain resolutions 144 x 176 , 176x144, 352x288, 480x360, 1280x960. Is the logic for converting to nv21 correct? What other way can I use for converting to jpeg from yuv420.
Is there any Java/Android api for this?
No, this isn't correct - you're not paying attention to the Plane's row stride or pixel stride.
You have to parse those, and make sure that your output buffer actually matches the input expectations of YuvImage's NV21 input, which assumes row stride = width, and interleaved V/U planes.
The code you have will only work if the input Image U/V planes are actually interleaved (in which case you're adding twice the UV data you need, but the first copy happens to be right layout...), and if width==row stride. Whether width==row stride depends on the resolution; usually the stride has to be a multiple of 16 pixels or something similar due to hardware restrictions. So for resolutions that aren't a multiple of 16, for example, your code wouldn't work.
Please fix both issues - paying attention to row and pixel stride; otherwise you might make it work on your device by accident, and still have it broken on devices with different parameters for strides.
Edit:
Some sample C++ code that does this kind of conversion can be found in the Android AOSP camera service code: CallbackProcessor::convertFromFlexibleYuv.
Mappings for reference:
previewFormat of HAL_PIXEL_FORMAT_YCrCb_420_SP is NV21.
src.data is plane 0
src.dataCb is is plane 1
src.dataCr is plane 2
src.stride is plane 0 rowStride
src.chromaStride is plane 1 and 2 rowStride
src.chromaStep is plane 1 and2 pixelStride
plane 0 pixel stride is 1
I am currently working on a program to help photographers with the creation of timelapses.
It calculates an decline or rise in brightness over a series of images. So the change in Exposure and iso for example dont affect the overall decline in brightness.
For this i use a simple Swing-based Interface which displays the first and last image. Under them are sliders to adjust the Brightness of the image.
This is applied via a direct manipulation of the BufferedImages underlying DataBuffer.
Mostly this works but i encountered some images which seem to have kind of a problem.
Do you have an idea why this is happening?
public BufferedImage getImage(float mult){
BufferedImage retim;
retim = new BufferedImage(img.getWidth(), img.getHeight(), img.getType());
Graphics g = retim.getGraphics();
g.drawImage(img, 0, 0, null);
g.dispose();
DataBufferByte db = (DataBufferByte) retim.getRaster().getDataBuffer();
byte[] bts = db.getData();
for(int i=0;i<bts.length;i++){
float n = bts[i]*mult;
if(n > 255){
bts[i]= (byte) 255;
}else{
bts[i] = (byte) n;
}
}
return retim;
}
This is the method which takes an float and multiplies every pixel in the image with it. (And some code to prevent the byte values from overflowing).
This is the unwanted behaviour (on the left) and the expected on the right.
Your problem is this line, and it occurs due to the fact that Java bytes are signed (in the range [-128...127]):
float n = bts[i] * mult;
After the multiplication, your n variable may be negative, thus causing the overflow to occur.
To fix it, use a bit mask to get the value as an unsigned integer (in the range [0...255]), before multiplying with the constant:
float n = (bts[i] & 0xff) * mult;
A better fix yet, is probably to use the RescaleOp, which is built to do brightness adjustments on BufferedImages.
Something like:
public BufferedImage getImage(float mult) {
return new RescaleOp(mult, 0, null).filter(img, null);
}
This is due to the capping of the value in certain bytes in the image.
For example (assuming RGB simple colour space):
The pixel starts at (125,255,0), if you multiply by factor 2.0, the result is (255,255,0). This is a different hue than the original.
This is also why the strange results only occur on pixels that already have high brightness to start with.
This link may help with better algorithm for adjusting brightness.
You could also refer to this related question.
The context of the question is OpenGL ES 2.0 in the Android environment. I have a texture. No problem to display or use it.
Is there a method to know its width and height and other info (like internal format) simply starting from its binding id?
I need to save texture to bitmap without knowing the texture size.
Not in ES 2.0. It's actually kind of surprising that the functionality is not there. You can get the size of a renderbuffer, but not the size of a texture, which seems inconsistent.
The only thing available are the values you can get with glGetTexParameteriv(), which are the FILTER and WRAP parameters for the texture.
It's still not in ES 3.0 either. Only in ES 3.1, glGetTexLevelParameteriv() was added, which gives you access to all the values you're looking for. For example to get the width and height of the currently bound texture:
int[] texDims = new int[2];
GLES31.glGetTexLevelParameteriv(GLES31.GL_TEXTURE_2D, 0, GL_TEXTURE_WIDTH, texDims, 0);
GLES31.glGetTexLevelParameteriv(GLES31.GL_TEXTURE_2D, 0, GL_TEXTURE_HEIGHT, texDims, 1);
As #Reto Koradi said there is no way to do it but you can store the width and height of a texture when you are loading it from android context before you bind it in OpenGL.
AssetManager am = context.getAssets();
InputStream is = null;
try {
is = am.open(name);
} catch (IOException e) {
e.printStackTrace();
}
final Bitmap bitmap = BitmapFactory.decodeStream(is);
int width = bitmap.getWidth();
int height = bitmap.getHeight();
// here is you bind your texture in openGL
I'll suggest a hack for doing this. Use ESSL's textureSize function. To access its result from the CPU side you're going to have to pass the texture as an uniform to a shader, and output the texture size as the r & g components of your shader output. Apply this shader to an 1x1px primitive drawn to a 1x1px FBO, then readback the drawn value from the GPU with glReadPixels.
You'll have to be careful with rounding, clamping and FBO formats. You may need a 16-bit integer FBO format.
How can I have that functionality in my game through which the players can change their hairstyle, look, style of clothes, etc., and so whenever they wear a different item of clothing their avatar is updated with it.
Should I:
Have my designer create all possible combinations of armor, hairstyles, and faces as sprites (this could be a lot of work).
When the player chooses what they should look like during their introduction to the game, my code would automatically create this sprite, and all possible combinations of headgear/armor with that sprite. Then each time they select some different armor, the sprite for that armor/look combination is loaded.
Is it possible to have a character's sprite divided into components, like face, shirt, jeans, shoes, and have the pixel dimensions of each of these. Then whenever the player changes his helmet, for example, we use the pixel dimensions to put the helmet image in place of where its face image would normally be. (I'm using Java to build this game)
Is this not possible in 2D and I should use 3D for this?
Any other method?
Please advise.
One major factor to consider is animation. If a character has armour with shoulder pads, those shoulderpads may need to move with his torso. Likewise, if he's wearing boots, those have to follow the same cycles as hid bare feet would.
Essentially what you need for your designers is a Sprite Sheet that lets your artists see all possible frames of animation for your base character. You then have them create custom hairstyles, boots, armour, etc. based on those sheets. Yes, its a lot of work, but in most cases, the elements will require a minimal amount of redrawing; boots are about the only thing I could see really taking a lot of work to re-create since they change over multiple frames of animation. Be rutheless with your sprites, try to cut down the required number as much as possible.
After you've amassed a library of elements you can start cheating. Recycle the same hair style and adjust its colour either in Photoshop or directly in the game with sliders in your character creator.
The last step, to ensure good performance in-game, would be to flatten all the different elements' sprite sheets into a single sprite sheet that is then split up and stored in sprite buffers.
3D will not be necessary for this, but the painter algorithm that is common in the 3D world might IMHO save you some work:
The painter algorithm works by drawing the most distant objects first, then overdrawing with objects closer to the camera. In your case, it would boild down to generating the buffer for your sprite, drawing it onto the buffer, finding the next dependant sprite-part (i.e. armour or whatnot), drawing that, finding the next dependant sprite-part (i.e. a special sign that's on the armour), and so on. When there are no more dependant parts, you paint the full generated sprite on to the display the user sees.
The combinated parts should have an alpha channel (RGBA instead of RGB) so that you will only combine parts that have an alpha value set to a value of your choice. If you cannot do that for whatever reason, just stick with one RGB combination that you will treat as transparent.
Using 3D might make combining the parts easier for you, and you'd not even have to use an offscreen buffer or write the pixel combinating code. The flip-side is that you need to learn a little 3D if you don't know it already. :-)
Edit to answer comment:
The combination part would work somewhat like this (in C++, Java will be pretty similar - please note that I did not run the code below through a compiler):
//
// #param dependant_textures is a vector of textures where
// texture n+1 depends on texture n.
// #param combimed_tex is the output of all textures combined
void Sprite::combineTextures (vector<Texture> const& dependant_textures,
Texture& combined_tex) {
vector< Texture >::iterator iter = dependant_textures.begin();
combined_tex = *iter;
if (dependant_textures.size() > 1)
for (iter++; iter != dependant_textures.end(); iter++) {
Texture& current_tex = *iter;
// Go through each pixel, painting:
for (unsigned char pixel_index = 0;
pixel_index < current_tex.numPixels(); pixel_index++) {
// Assuming that Texture had a method to export the raw pixel data
// as an array of chars - to illustrate, check Alpha value:
int const BYTESPERPIXEL = 4; // RGBA
if (!current_tex.getRawData()[pixel_index * BYTESPERPIXEL + 3])
for (int copied_bytes = 0; copied_bytes < 3; copied_bytes++)
{
int index = pixel_index * BYTESPERPIXEL + copied_bytes;
combined_tex.getRawData()[index] =
current_tex.getRawData()[index];
}
}
}
}
To answer your question for a 3D solution, you would simply draw rectangles with their respective textures (that would have an alpha channel) over each other. You would set the system up to display in an orthogonal mode (for OpenGL: gluOrtho2D()).
I'd go with the procedural generation solution (#2). As long as there isn't a limiting amount of sprites to be generated, such that the generation takes too long. Maybe do the generation when each item is acquired, to lower the load.
Since I was asked in comments to supply a 3D way aswell, here is some, that is an excerpt of some code I wrote quite some time ago. It's OpenGL and C++.
Each sprite would be asked to draw itself. Using the Adapter pattern, I would combine sprites - i.e. there would be sprites that would hold two or more sprites that had a (0,0) relative position and one sprite with a real position having all those "sub-"sprites.
void Sprite::display (void) const
{
glBindTexture(GL_TEXTURE_2D, tex_id_);
Display::drawTranspRect(model_->getPosition().x + draw_dimensions_[0] / 2.0f,
model_->getPosition().y + draw_dimensions_[1] / 2.0f,
draw_dimensions_[0] / 2.0f, draw_dimensions_[1] / 2.0f);
}
void Display::drawTranspRect (float x, float y, float x_len, float y_len)
{
glPushMatrix();
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glColor4f(1.0, 1.0, 1.0, 1.0);
glBegin(GL_QUADS);
glTexCoord2f(0.0f, 0.0f); glVertex3f(x - x_len, y - y_len, Z);
glTexCoord2f(1.0f, 0.0f); glVertex3f(x + x_len, y - y_len, Z);
glTexCoord2f(1.0f, 1.0f); glVertex3f(x + x_len, y + y_len, Z);
glTexCoord2f(0.0f, 1.0f); glVertex3f(x - x_len, y + y_len, Z);
glEnd();
glDisable(GL_BLEND);
glPopMatrix();
}
The tex_id_ is an integral value that identifies which texture is used to OpenGL. The relevant parts of the texture manager are these. The texture manager actually emulates an alpha channel by checking to see if the color read is pure white (RGB of (ff,ff,ff)) - the loadFile code operates on 24 bits per pixel BMP files:
TextureManager::texture_id
TextureManager::createNewTexture (Texture const& tex) {
texture_id id;
glGenTextures(1, &id);
glBindTexture(GL_TEXTURE_2D, id);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexImage2D(GL_TEXTURE_2D, 0, 4, tex.width_, tex.height_, 0,
GL_BGRA_EXT, GL_UNSIGNED_BYTE, tex.texture_);
return id;
}
void TextureManager::loadImage (FILE* f, Texture& dest) const {
fseek(f, 18, SEEK_SET);
signed int compression_method;
unsigned int const HEADER_SIZE = 54;
fread(&dest.width_, sizeof(unsigned int), 1, f);
fread(&dest.height_, sizeof(unsigned int), 1, f);
fseek(f, 28, SEEK_SET);
fread(&dest.bpp_, sizeof (unsigned short), 1, f);
fseek(f, 30, SEEK_SET);
fread(&compression_method, sizeof(unsigned int), 1, f);
// We add 4 channels, because we will manually set an alpha channel
// for the color white.
dest.size_ = dest.width_ * dest.height_ * dest.bpp_/8 * 4;
dest.texture_ = new unsigned char[dest.size_];
unsigned char* buffer = new unsigned char[3 * dest.size_ / 4];
// Slurp in whole file and replace all white colors with green
// values and an alpha value of 0:
fseek(f, HEADER_SIZE, SEEK_SET);
fread (buffer, sizeof(unsigned char), 3 * dest.size_ / 4, f);
for (unsigned int count = 0; count < dest.width_ * dest.height_; count++) {
dest.texture_[0+count*4] = buffer[0+count*3];
dest.texture_[1+count*4] = buffer[1+count*3];
dest.texture_[2+count*4] = buffer[2+count*3];
dest.texture_[3+count*4] = 0xff;
if (dest.texture_[0+count*4] == 0xff &&
dest.texture_[1+count*4] == 0xff &&
dest.texture_[2+count*4] == 0xff) {
dest.texture_[0+count*4] = 0x00;
dest.texture_[1+count*4] = 0xff;
dest.texture_[2+count*4] = 0x00;
dest.texture_[3+count*4] = 0x00;
dest.uses_alpha_ = true;
}
}
delete[] buffer;
}
This was actually a small Jump'nRun that I developed occasionally in my spare time. It used gluOrtho2D() mode aswell, btw. If you leave means to contact you, I will send you the source if you want.
Older 2d games such as Diablo and Ultima Online use a sprite compositing technique to do this. You could search for art from those kind of older 2d isometric games to see how they did it.