Summary:
Our app depends on a high detection speed of facial landmarks (= like eyes open or closed). Thus I developed an algorythm that takes the position of the face from the last frame and crops the image from the next frame. This works perfectly and the Face Detector only has to process a quarter of the image.
But it does not increase the detection speed. Does anybody know why?
Edit: All that my algorythm is doing is croping the image based on the information from the last image. But it does not perform the ImageRecognition itself. We are using Mobile Vision from Google.
important Code snippets:
This snippped is executed before passing the bitmap to the Face Detector. It takes the face position from the previous frame and only passes this part of the image:
Bitmap bitmapReturn = Bitmap.createBitmap(bitmap, topLeftX, topLeftY, width, height);
This snippet is executed after the frame is processed by the Face Detector. It porviedes the location of the image for the next frame:
float spotY = getSpotY(face);
float spotX = getRatioX(face);
int moveX = (int) (((float)bitMapScanWidth / 2) - spotX) ;
int moveY = (int) (((float)bitMapScanHeight / 2) - spotY);
moveValues(moveX, moveY);
There are some further code snippets that make sure the image values topLeftX and topLeftY don't reach values beyond the bitmap size and others that make sure the face has the same size on the image.
But as said before. The algorythm works fine, but doesn't lead to anymore speed. I can't figure out why, because it should massively reduce the required computation time. Can anybody explain me why this is not the case? Do I have to adjust something? Or is there another way, to increase speed in my algorythm?
Note that when I compared the speed between the two versions (With the algorythm that crops the image and without it) both versions actually calculated through the required functions to crop the image. The only difference was that one of them actually used the values to crop the image and the other one just calculated them in the background. This means, that the computation required for my algorythm was not the reason for the missing speed improvement.
If you are building your own algorithm for facial recognition, you can try to change the actual algorithm and use an architecture that is suitable for mobile devices. like MobilNetSSD or such, also you can try and change how you compile your algorithm and deploy it on mobile because both those techniques can boost the performance beyond what a simple cropping function can do.
further, if you don't have any problem sharing the actual algorithm you are using I will do my best to see why cropping doesn't work for your specific case.
Related
I am using Java with OpenCV Library to detect Face,Eyes and Mouth using Laptop Camera.
What I have done so far:
Capture Video Frames using VideoCapture object.
Detect Face using Haar-Cascades.
Divide the Face region into Top Region and Bottom Region.
Search for Eyes inside Top region.
Search for Mouth inside Bottom region.
Problem I am facing:
At first Video is running normally and suddenly it becomes slower.
Main Questions:
Do Higher Cameras' Resolutions work better for Haar-Cascades?
Do I have to capture Video Frames in a certain scale? for example (100px X100px)?
Do Haar-Cascades work better in Gray-scale Images?
Does different lighting conditions make difference?
What does the method detectMultiScale(params) exactly do?
If I want to go for further analysis for Eye Blinking, Eye Closure Duration, Mouth Yawning, Head Nodding and Head Orientation to Detect Fatigue (Drowsiness) By Using Support Vector Machine, any advices?
Your help is appreciated!
The following article, would give you an overview of the things going under the hood, I would highly recommend to read the article.
Do Higher Cameras' Resolutions work better for Haar-Cascades?
Not necessarily, the cascade.detectMultiScale has params to adjust for various input width, height scenarios, like minSize and maxSize, These are optional params However, But you can tweak these to get robust predictions if you have control over the input image size. If you set the minSize to smaller value and ignore maxSize then it will work for smaller and high res images as well, but the performance would suffer. Also if you imagine now, How come there is no differnce between High-res and low-res images then you should consider that the cascade.detectMultiScale internally scales the images to lower resolutions for performance boost, that is why defining the maxSize and minSize is important to avoid any unnecessary iterations.
Do I have to capture Video Frames in a certain scale? for example
(100px X100px)
This mainly depends upon the params you pass to the cascade.detectMultiScale. Personally I guess that 100 x 100 would be too small for smaller face detection in the frame as some features would be completely lost while resizing the frame to smaller dimensions, and the cascade.detectMultiScale is highly dependent upon the gradients or features in the input image.
But if the input frame only has face as a major part, and there are no other smaller faces dangling behind then you may use 100 X 100. I have tested some sample faces of size 100 x 100 and it worked pretty well. And if this is not the case then 300 - 400 px width should work good. However you would need to tune the params in order to achieve accuracy.
Do Haar-Cascades work better in Gray-scale Images?
They work only in gray-scale images.
In the article, if you read the first part, you will come to know that it face detection is comprised of detecting many binary patterns in the image, This basically comes from the ViolaJones, paper which is the basic of this algorithm.
Does different lighting conditions make difference?
May be in some cases, largely Haar-features are lighting invariant.
If you are considering different lighting conditions as taking images under green or red light, then it may not affect the detection, The haar-features (since dependent on gray-scale) are independent of the RGB color of input image. The detection mainly depends upon the gradients/features in the input image. So as far as there are enough gradient differences in the input image such as eye-brow has lower intensity than fore-head, etc. it will work fine.
But consider a case when input image has back-light or very low ambient light, In that case it may be possible that some prominent features are not found, which may result in face not detected.
What does the method detectMultiScale(params) exactly do?
I guess, if you have read the article, by this time, then you must be knowing it well.
If I want to go for further analysis for Eye Blinking, Eye Closure
Duration, Mouth Yawning, Head Nodding and Head Orientation to Detect
Fatigue (Drowsiness) By Using Support Vector Machine, any advices?
No, I won't suggest you to perform these type of gesture detection with SVM, as it would be extremely slow to run 10 different cascades to conclude current facial state, However I would recommend you to use some Facial Landmark Detection Framework, such as Dlib, You may search for some other frameworks as well, because the model size of dlib is nearly 100MB and it may not suit your needs i f you want to port it to mobile device. So the key is ** Facial Landmark Detection **, once you get the full face labelled, you can draw conclusions like if the mouth if open or the eyes are blinking, and it works in Real-time, so your video processing won't suffer much.
I am new to libgdx and this question might be obvious since they skip it in every tutorial.
But say I set a camera up like this:
cam = new OrthographicCamera(100, 100);
This means I will now be working with my own units instead of pixels. So how do I know what size to make an image? Say for example I want an image to fill the width of the camera and half of the height. How would I do this? Do I make the image 100x50px? that makes no sense to me.
You say you are working with your own units when you define your camera, yet you are still thinking with pixels when you ask whether you should make your image 100x50px.
Since you are working with your own units, I would assume that they are not completely detached from original pixel units, meaning that everything should now be measured by your units including the size of the images.
If you can calculate what 1 unit of your represents pixel-wise, you can then determine the scale to which you can scale all of your images.
Then you can say that your image should be 100x50units in size, you don't need to make the image to fit this condition, you just need to adjust its scale so that it corresponds with your defined unit measurement.
If you are using SpriteBatch to draw your images, you might find that a couple of the defined draw overloads documented in the API can be given scale for both X and Y and could prove to be useful in this scenario.
I am looking for the simplest (and still non-problematic) way to resize a BufferedImage in Java.
In some answer to a question, the user coobird suggested the following solution, in his words (very slightly changed by me):
**
The Graphics object has a method to draw an Image while also performing a resize operation:
Graphics.drawImage(Image, int, int, int, int, ImageObserver)
method can be used to specify the location along with the size of the image when drawing.
So, we could use a piece of code like this:
BufferedImage originalImage = // .. created somehow
BufferedImage newImage = new BufferedImage(SMALL_SIZE, SMALL_SIZE, BufferedImage.TYPE_INT_RGB);
Graphics g = newImage.createGraphics();
g.drawImage(originalImage, 0, 0, SMALL_SIZE, SMALL_SIZE, null);
g.dispose();
This will take originalImage and draw it on the newImage with the width and height of SMALL_SIZE.
**
This solution seems rather simple. I have two questions about it:
Will it also work (using the exact same code), if I want to resize an image to a larger size, not only a smaller one?
Are there any problems with this solution?
If there is a better way to do this, please suggest it.
Thanks
The major problem with single step scaling is they don't generally produce quality output, as they focus on taking the original and squeezing into a smaller space, usually by dropping out a lot of pixel information (different algorithms do different things, so I'm generalizing)
Will drawGraphics scale up and down, yes, will it do it efficiently or produce a quality output? These will come down to implementation, generally speaking, most of the scaling algorithms used by default are focused on speed. You can effect these in a little way, but generally, unless you're scaling over a small range, the quality generally suffers (from my experience).
You can take a look at The Perils of Image.getScaledInstance() for more details and discussions on the topic.
Generally, what is generally recommend is to either use a dedicated library, like imgscalr, which, from the ten minutes I've played with it, does a pretty good job or perform a stepped scale.
A stepped scale basically steps the image up or down by the power of 2 until it reaches it's desired size. Remember, scaling up is nothing more then taking a pixel and enlarging it a little, so quality will always be an issue if you scale up to a very large size.
For example...
Quality of Image after resize very low -- Java
Scale the ImageIcon automatically to label size
Java: JPanel background not scaling
Remember, any scaling is generally an expensive operation (based on the original and target size of the image), so it is generally best to try and do those operations out side of the paint process and in the background where possible.
There is also the question whether you want to maintain the aspect ratio of the image? Based on you example, the image would be scaled in a square manner (stretched to meet to the requirements of the target size), this is generally not desired. You can pass -1 to either the width or height parameter and the underlying algorithm will maintain the aspect ratio of the original image or you could simply take control and make more determinations over whether you want to fill or fit the image to a target area, for example...
Java: maintaining aspect ratio of JPanel background image
In general, I avoid using drawImage or getScaledInstance most of the time (if your scaling only over a small range or want to do a low quality, fast scale, these can work) and rely more on things like fit/fill a target area and stepped scaling. The reason for using my own methods simply comes down to not always being allowed to use outside libraries. Nice not to have to re-invent the wheel where you can
It will enlarge the original if you set the parameters so. But: you should use some smart algorithm which preserves edges because simply enlarging an image will make it blurry and will result in worse perceived quality.
No problems. Theoretically this can even be hardware-accelerated on certain platforms.
I am using Libgdx to code an android game and as you may know, many screen resolutions cause some problems if done incorrectly. So I am trying to use this DP unit rather than pixels.
However, I have this method here:
public static float pixelToDP(float dp){
return dp * Gdx.graphics.getDensity();
}
the Gdx.graphics.getDensity() method actually gets the SCALE, so it's already done for me.
Now the problem, libgdx is cross platform which is good for testing. When I launch this on my S4 which has a resolution of 1920x1080 with a dpi of a whopping 480, opposed to my terrible and overpriced laptop which has 1366x768 # 92dpi it is placed exactly where I want it. On desktop it is way off, a good few hundred pixels on the X and Y axis.
Is this due to the fact my screen dpi is measured #92dpi, the resolution is a lot lower and the actual game is not fullscreen on the desktop?
Here is the code for drawing the object:
table.setPosition(MathHelper.pixelToDP(150), MathHelper.pixelToDP(200));
In order to get it perfect on desktop I have to do:
table.setPosition(MathHelper.pixelToDP(480), MathHelper.pixelToDP(700));
Which is not even visible on my phone, since the scale is actually 3x, which puts it a good 200 pixels off the screen on the Y axis.
Is there a way around this? Or am I basically going to have to deal with doing platform checks and different blocks of code?
Possible solution:
So I changed my dp conversion method, if I was to do 100 * 0.5 it would return a new value of 50 but in reality I want the orignal value of 100 + 100 * 0.5.
Not sure if this is a proper fix or not but regardless by table is drew in the exact same place on both laptop and phone:
public static float pixelToDP(float dp){
if(Gdx.graphics.getDensity() < 1)
return dp + dp * Gdx.graphics.getDensity();
return dp * Gdx.graphics.getDensity();
Is this just a cheap fix or is this pretty much how it should be done?
Usage of density independent pixels implies that the physical size of the table on all screens should be same. Since your laptop screen is (physically) much bigger, you would see the table to be lot smaller than expected.
I would suggest an alternative approach of placing objects in fractions of size. e.g. 30% of width or 45% of height.
To implement this, just assume a stage resolution and place objects as you like then change viewport in resize method such that you get full view.
Hope it helps.
For more,
https://code.google.com/p/libgdx-users/wiki/AspectRatio
The best approach for this is to manipulate the density based on the execution target.
So what I usually do is to store the density in a field in a singleton, and set it based on the scenario:
public class Game {
public static float density;
public static initDensity(){
if (GDX.app.getTarget() == 0){
density = 2.0f;
}else {
density = GDX.graphics.getDensity();
}
}
public float toPixel(float dip){
return dip * density;
}
}
with this approach you can "simulate" a more dense screen then you actually have, and by using properties in your run config like -Ddensity=2 and System.getPropery("density") you can vary the screens you like to simulate.
One approach is having a fixed viewport size. Create your camera for example 1366x768 and place all your objects using that coordinate. Then the camera will fill the whole screen of every other resolution.
cam = new OrthographicCamera(1366, 768);
try seeing few tutorials....I personally think it is best to deal with pixels and using the camera will help you a lot, check this link once
getting different screen resolutions
I'm currently developing my first Android app and am having some issues rendering images. The image itself is great quality to begin with, but upon rendering it the quality drastically lowers. Edges become jagged and it just looks poorly done. Everyone I've showed it to thus far has almost immediately noticed it, without any prompting about it. [start on left, end on right:]
I'm trying everything I am aware of and every tip I've been able to find by looking around online, but nothing seems to fix it.
Currently, I get the image as a Bitmap and scale it:
Bitmap holeImage = BitmapFactory.decodeResource(res, R.drawable.hole_image);
Bitmap holeImageBMP = Bitmap.createScaledBitmap(holeImage, width, height, true);
Once I have the image, I create a Paint, set a few smoothing attributes to true, and then draw it on the canvas:
Paint smoothingPaint = new Paint();
smoothingPaint.setAntiAlias(true);
smoothingPaint.setFilterBitmap(true);
smoothingPaint.setDither(true);
canvas.drawBitmap(holeImageBMP, 0, 0, smoothingPaint);
Yet, as you can obviously see above, the image quality drastically decreases. I've seen plenty of images being rendered beautifully and I'm honestly just not sure what's going on so any advice would be great!
Other notes: I'm using a SurfaceView method to handle the drawing, similar in nature to the LunarLander example given in the SDK.
Thanks again!
If you aren't restricted to much less colors than the original picture has (Does Android have 256 color modes?), I'd suggest to disable dithering, if you zoom into your picture, it does have a visible effect that perhaps destroys a smooth look.
I think in your case, dithering infers with anti-aliasing by destroying the additional colors that anti-aliasing needs for a smooth look. A quick color count on your pictures (left one about 850, right one about 140) confirms this.
That is probably related to converting images from one format to another. Also, android screens vary from device to device. Try to use another device and it might look better... Almost for sure it will have a different tone.
Try to read this great article on this problem (and banding and dithering) and consider adapting the image you created for it to work better in android devices: http://www.curious-creature.org/2010/12/08/bitmap-quality-banding-and-dithering/