Develop Reference

Kinect for Processing error when using multiple cameras

I'm trying to run two Kinect V1 cameras simultaneously in Processing 3. I have gotten to a solution that is not sustainable, and am I trying to make something more stable/reliable.
At the moment, whenever I try to run both cameras simultaneously on a single sketch, I am hit with the error
"Could not claim interface on camera: -3
Failed to open camera subdevice or it is not disabled.Failed to open motor subddevice or it is not disabled.Failed to open audio subdevice or it is not disabled.There are no kinects, returning null"
One camera opens, the other does not. It is not always consistent which camera opens, which leads me to believe there's something tripping over permissions after the objects are created, or when the second object is initialized.
My code is as such
import SimpleOpenNI.*;
import org.openkinect.freenect.*;
import org.openkinect.freenect2.*;
import org.openkinect.processing.*;
import org.openkinect.tests.*;
//Imported libraries
//Some might be unnecessary but I don't have time to check
//Better safe than sorry, maybe I'll delete later
Kinect kinect;
Kinect kinect2;
PImage depthImage;
PImage depthImage2;
//Set depth threshold
float minDepth = 996;
float maxDepth = 2493;
float iWidth1 = 0;
float iHeight1 = 0;
float iWidth2 = 0;
float iHeight2 = 0;
//Double check for the number of devices, mostly for troubleshooting
int numDevices = 0;
//control which device is being controlled (in case I want device control)
int deviceIndex = 0;
void setup() {
//set Arbitrary size
size(640, 360);
//Set up window to resize, need to figure out how to keep things centered
surface.setResizable(true);
//not necessary, but good for window management. Window label
surface.setTitle("KINECT 1");
//get number of devices, print to console
numDevices = Kinect.countDevices();
println("number of V1 Kinects = "+numDevices);
//set up depth for the first kinect tracking
kinect = new Kinect(this);
kinect.initDepth();
//Blank Image
depthImage = new PImage(kinect.width, kinect.height);
//set up second window
String [] args = {"2 Frame Test"};
SecondApplet sa = new SecondApplet();
PApplet.runSketch(args, sa);
}
//Draw first window's Kinect Threshold
void draw () {
if ((width/1.7778) < height) {
iWidth1 = width;
iHeight1 = width/1.7778;
} else {
iWidth1 = height*1.7778;
iHeight1 = height;
}
//Raw Image
image(kinect.getDepthImage(), 0, 0, iWidth1, iHeight1);
//Threshold Equation
int[] rawDepth = kinect.getRawDepth();
for (int i=0; i < rawDepth.length; i++) {
if (rawDepth[i] >= minDepth && rawDepth[i] <= maxDepth) {
depthImage.pixels[i] = color(255);
} else {
depthImage.pixels[i] = color(1);
}
}
}
public class SecondApplet extends PApplet {
public void settings() {
//arbitrary size
size(640, 360);
kinect2 = new Kinect(this);
kinect2.initDepth();
//Blank Image
depthImage2 = new PImage(kinect2.width, kinect2.height);
}
void draw () {
if ((width/1.7778) < height) {
iWidth2 = width;
iHeight2 = width/1.7778;
} else {
iWidth2 = height*1.7778;
iHeight2 = height;
}
image(kinect2.getDepthImage(), 0, 0, iWidth2, iHeight2);
surface.setResizable(true);
surface.setTitle("KINECT 2");
int[] rawDepth2 = kinect2.getRawDepth();
for (int i=0; i < rawDepth2.length; i++) {
if (rawDepth2[i] >= minDepth && rawDepth2[i] <= maxDepth) {
depthImage2.pixels[i] = color(255);
} else {
depthImage2.pixels[i] = color(1);
}
}
}
}
Curiously, the code returns a confirmation that there are two kinect devices connected in the console. For some reason, it cannot access both at the same time.
I'm not a very experienced code, so this code might look amateur. Open to feedback on other parts, but really just looking to solve this problem.
This code returns the error pasted above when there are two Kinect V1's connected to the computer.
Running Mac OS11.6.8 on an Intel MacBook Pro
Using Daniel Schiffman's OpenKinect for Processing as a starting point for the code.
I've run a successful iteration of this code with a slimmed down version of Daniel Schiffman's Depth Threshold example.
import org.openkinect.freenect.*;
import org.openkinect.processing.*;
Kinect kinect;
// Depth image
PImage depthImg;
// Which pixels do we care about?
// These thresholds can also be found with a variaty of methods
float minDepth = 996;
float maxDepth = 2493;
// What is the kinect's angle
float angle;
void setup() {
size(1280, 480);
kinect = new Kinect(this);
kinect.initDepth();
angle = kinect.getTilt();
// Blank image
depthImg = new PImage(kinect.width, kinect.height);
}
void draw() {
// Draw the raw image
image(kinect.getDepthImage(), 0, 0);
// Calibration
//minDepth = map(mouseX,0,width, 0, 4500);
//maxDepth = map(mouseY,0,height, 0, 4500);
// Threshold the depth image
int[] rawDepth = kinect.getRawDepth();
for (int i=0; i < rawDepth.length; i++) {
if (rawDepth[i] >= minDepth && rawDepth[i] <= maxDepth) {
depthImg.pixels[i] = color(255);
} else {
depthImg.pixels[i] = color(0);
}
}
// Draw the thresholded image
depthImg.updatePixels();
image(depthImg, kinect.width, 0);
//Comment for Calibration
fill(0);
text("TILT: " + angle, 10, 20);
text("THRESHOLD: [" + minDepth + ", " + maxDepth + "]", 10, 36);
//Calibration Text
//fill(255);
//textSize(32);
//text(minDepth + " " + maxDepth, 10, 64);
}
Using this code, I was able to get both cameras operating using the following process:
Connect a single Kinect v1 to the computer
Open and run the above code
Duplicate the sketch file
Connect the second Kinect V1 to the computer
Open and run the duplicated sketch of the same code
This worked for my purposes and remained stable for an extended period of time. However, this isn't a sustainable solution if anyone other than me wants to utilize this program.
Any help with this problem would be greatly appreciated

Multiple viewports in javafx application

I am attempting to create a multi-user, multi-screen application within JavaFX, and I am having trouble with the multi-screen part.
Think an FPS with couch co-op: the screen splits evenly depending on how many people are connected locally. Every different view is looking in a different direction, and at a different place, but at the same 'world'.
I learned the hard way (confirmed in a comment here) that each node can only appear in the active scene graph once, so, for instance, I cannot have the same node spread across multiple distinct panes (which is conceptually ideal). And that's where I'm not sure where to go next.
Looking at other similar technologies like OpenGL, (example) most have the ability to create another viewport for the application, but JavaFX does not seem to have this.
Some things I have ruled out as unreasonable/impossible (correct me if I'm wrong):
Using shapes to create a clip mask for a pane (Can only use one mask per node)
Having a complete deep copy of each node for each view (too expensive, nodes moving constantly)
Having x number of users each have their own set of nodes and have one update loop update every node in every view (too expensive, too many nodes for scene graph, too much)
How would I go about creating multiple views of the same set of nodes, while still maintaining individual user control, and changing persistence/moving nodes, between every different view?
Thanks.

Thanks to the people in the comments for the solution. I ended up creating a background model for each view to mirror, and then creating a new set of nodes per view that has the relevant properties bound to the background model.
The update loop then has to only update the one background model, and the copies all update automatically. Each node copy has a reference to the model node that it is mimicking, so when a user inputs a change for a node, the model node is changed, which changes the copy node.
The solution is not too elegant and I will have to look more into multithreading (multitasking?) with Tasks (here) and Platform.runLater() (here) functions of JavaFX to increase functionality.
Here is a quick example of what I accomplished:
Main.java
public class Main extends Application {
private static Group root = new Group();
private static Scene initialScene = new Scene(root, Color.BLACK);
private static final int NUM_OF_CLIENTS = 8;
private static long updateSpeed = 20_666_666L;
private static double deltaTime;
private static double counter = 0;
#Override
public void start(Stage primaryStage) {
primaryStage.setFullScreen(true);
primaryStage.setScene(initialScene);
primaryStage.show();
initModel();
initModelViews();
startUpdates();
}
private void initModel() {
for (int i = 0; i < NUM_OF_CLIENTS; i++) {
Model.add(new UpdateObject());
}
}
private void initModelViews() {
//Correctly positioning the views
int xPanes = (NUM_OF_CLIENTS / 4.0 > 1.0) ? 4 : NUM_OF_CLIENTS;
int yPanes = (NUM_OF_CLIENTS / 4) + ((NUM_OF_CLIENTS % 4 > 0) ? 1 : 0);
for (int i = 0; i < NUM_OF_CLIENTS; i++) {
Pane clientView = new Pane(copyModelNodes());
clientView.setBackground(new Background(new BackgroundFill(Color.color(Math.random(), Math.random(), Math.random()), CornerRadii.EMPTY, Insets.EMPTY)));
System.out.println(clientView.getChildren());
clientView.relocate((i % 4) * (Main.initialScene.getWidth() / xPanes), (i / 4) * (Main.initialScene.getHeight() / yPanes)) ;
clientView.setPrefSize((Main.initialScene.getWidth() / xPanes), (Main.initialScene.getHeight() / yPanes));
root.getChildren().add(clientView);
}
}
private Node[] copyModelNodes() {
ObservableList<UpdateObject> model = Model.getModel();
Node[] modelCopy = new Node[model.size()];
for (int i = 0; i < model.size(); i++) {
ImageView testNode = new ImageView();
testNode.setImage(model.get(i).getImage());
testNode.layoutXProperty().bind(model.get(i).layoutXProperty());
testNode.layoutYProperty().bind(model.get(i).layoutYProperty());
testNode.rotateProperty().bind(model.get(i).rotateProperty());
modelCopy[i] = testNode;
}
return modelCopy;
}
private void startUpdates() {
AnimationTimer mainLoop = new AnimationTimer() {
private long lastUpdate = 0;
#Override
public void handle(long frameTime) {
//Time difference from last frame
deltaTime = 0.00000001 * (frameTime - lastUpdate);
if (deltaTime <= 0.1 || deltaTime >= 1.0)
deltaTime = 0.00000001 * updateSpeed;
if (frameTime - lastUpdate >= updateSpeed) {
update();
lastUpdate = frameTime;
}
}
};
mainLoop.start();
}
private void update() {
counter += 0.1;
if (counter > 10.0) {
counter = 0;
}
for (UpdateObject objectToUpdate : Model.getModel()) {
objectToUpdate.setLayoutX(objectToUpdate.getLayoutX() + 0.02 * counter * deltaTime);
objectToUpdate.setLayoutY(objectToUpdate.getLayoutY() + 0.02 * counter * deltaTime);
objectToUpdate.setRotate(objectToUpdate.getRotate() + 5);
}
}
}
UpdateObject.java
class UpdateObject extends ImageView {
private static Random random = new Random();
private static Image testImage = new Image("duckTest.png");
UpdateObject() {
this.setImage(testImage);
this.setLayoutX(random.nextInt(50));
this.setLayoutY(random.nextInt(50));
this.setRotate(random.nextInt(360));
}
}
Model.java
class Model {
private static ObservableList<UpdateObject> modelList = FXCollections.observableArrayList();
static void add(UpdateObject objectToAdd) {
modelList.add(objectToAdd);
}
static ObservableList<UpdateObject> getModel() {
return modelList;
}
}
Test image used

bouncing balls never stay calm on the ground

I try to create a very simple physics engine for my study (processing used for a interactive installation).
The target is to have a ground covered with balls that you can throw around with gestures (based on Kinect information).
Therefor I need to do some basic physic simulation like bouncing and thats what I started with. So there are just balls falling down and bouncing. I simulated the air resistance with a simple 0.995f multiplication on the speed if the ball moves up. Works nice and looks realistic. The main problem is, that the balls never stay calm on the ground. Instead they start to tremble on the ground. That means there is a movement of 1 or 2 pixels up and down.
How can I prevent that without implementing some "borders" on which I set the position directly to the bottom and the speed to zero?
My applet:
public class BubblePhysicApplet extends PApplet {
public static int width = 640;
public static int height = 480;
long lastTime = -1;
Bubble[] mBubbles = new Bubble[10];
Random mRandom = new Random();
public void setup() {
// size(width, height, OPENGL);
size(width, height, P2D);
for (int i = 0; i < mBubbles.length; i++) {
mBubbles[i] = new Bubble(mRandom.nextInt(width), mRandom.nextInt(height), 50);
}
lastTime = System.currentTimeMillis();
}
public void draw() {
background(0);
long tmp = System.currentTimeMillis();
long elapsed = tmp - lastTime;
for (Bubble bubble : mBubbles) {
bubble.animate(elapsed);
bubble.draw(this);
}
lastTime = System.currentTimeMillis();
}
}
The ball/bubble:
public class Bubble {
float mX;
float mY;
float mSize;
float mSpeedX = 0;
float mSpeedY = 0;
public Bubble(int x, int y, int size) {
mX = x;
mY = y;
mSize = size;
}
public void draw(PApplet applet) {
applet.stroke(255);
applet.noFill();
applet.ellipseMode(PApplet.CENTER);
applet.ellipse(mX, mY, mSize, mSize);
}
public void animate(long elapsed) {
updateSpeedY(elapsed);
if (mSpeedX != 0 || mSpeedY != 0) {
checkBorders();
}
}
private void checkBorders() {
if (mY > BubblePhysicApplet.height - mSize / 2) {
mY = 2 * BubblePhysicApplet.height - (mY + mSize);
mSpeedY = -mSpeedY;
}
if (mX > BubblePhysicApplet.width) {
mX = BubblePhysicApplet.width - (mX - BubblePhysicApplet.width);
mSpeedX = -mSpeedX;
}
}
private void updateSpeedX() {
}
private void updateSpeedY(long elapsed) {
mSpeedY += (elapsed / 1000f) * 9.81f;
if (mSpeedY < 0) {
mSpeedY *= 0.95f;
}
mY += mSpeedY;
}
}

It's not only air resistance that slows the ball down, but the fact that it's not perfectly elastic as this line suggests: mSpeedY = -mSpeedY;
The ball absorbs energy when it squishes against the floor before it bounces back, so it doesn't bounce as high. Try a real super ball. I seem to remember it only bounces 80% as high or so. You might try:
mSpeedY = - (0.8 * mSpeedY);

you have to fix your check borders method, read this answer I just gave a complete formulas needed for realistic physical simulation. and it's also more realistic if you move objects using hist method (p = v*dt + 1/2*adtdt)

The problem is that in updateSpeedY we have mSpeedY += (elapsed / 1000f) * 9.81f; even when there is a collision. That said collision is detected later in checkBorders where the speed is flipped mSpeedY = -mSpeedY;. The problem is that if the ball is hitting the floor with a speed near 0, it bounces with a speed of 0 + (elapsed / 1000f) * 9.81f;!!
You have to rethink your code.

in the same fashion you used a friction factor for the air, you can also include a friction factor for the contact with the ground, and which even higher values, so at each contact, it starts to lose eneger rapidly and finally stops

Getting a NullPointerException at seemingly random intervals, not sure why

I'm running an example from a Kinect library for Processing (http://www.shiffman.net/2010/11/14/kinect-and-processing/) and sometimes get a NullPointerException pointing to this line:
int rawDepth = depth[offset];
The depth array is created in this line:
int[] depth = kinect.getRawDepth();
I'm not exactly sure what a NullPointerException is, and much googling hasn't really helped. It seems odd to me that the code compiles 70% of the time and returns the error unpredictably. Could the hardware itself be affecting it?
Here's the whole example if it helps:
// Daniel Shiffman
// Kinect Point Cloud example
// http://www.shiffman.net
// https://github.com/shiffman/libfreenect/tree/master/wrappers/java/processing
import org.openkinect.*;
import org.openkinect.processing.*;
// Kinect Library object
Kinect kinect;
float a = 0;
// Size of kinect image
int w = 640;
int h = 480;
// We'll use a lookup table so that we don't have to repeat the math over and over
float[] depthLookUp = new float[2048];
void setup() {
size(800,600,P3D);
kinect = new Kinect(this);
kinect.start();
kinect.enableDepth(true);
// We don't need the grayscale image in this example
// so this makes it more efficient
kinect.processDepthImage(false);
// Lookup table for all possible depth values (0 - 2047)
for (int i = 0; i < depthLookUp.length; i++) {
depthLookUp[i] = rawDepthToMeters(i);
}
}
void draw() {
background(0);
fill(255);
textMode(SCREEN);
text("Kinect FR: " + (int)kinect.getDepthFPS() + "\nProcessing FR: " + (int)frameRate,10,16);
// Get the raw depth as array of integers
int[] depth = kinect.getRawDepth();
// We're just going to calculate and draw every 4th pixel (equivalent of 160x120)
int skip = 4;
// Translate and rotate
translate(width/2,height/2,-50);
rotateY(a);
for(int x=0; x<w; x+=skip) {
for(int y=0; y<h; y+=skip) {
int offset = x+y*w;
// Convert kinect data to world xyz coordinate
int rawDepth = depth[offset];
PVector v = depthToWorld(x,y,rawDepth);
stroke(255);
pushMatrix();
// Scale up by 200
float factor = 200;
translate(v.x*factor,v.y*factor,factor-v.z*factor);
// Draw a point
point(0,0);
popMatrix();
}
}
// Rotate
a += 0.015f;
}
// These functions come from: http://graphics.stanford.edu/~mdfisher/Kinect.html
float rawDepthToMeters(int depthValue) {
if (depthValue < 2047) {
return (float)(1.0 / ((double)(depthValue) * -0.0030711016 + 3.3309495161));
}
return 0.0f;
}
PVector depthToWorld(int x, int y, int depthValue) {
final double fx_d = 1.0 / 5.9421434211923247e+02;
final double fy_d = 1.0 / 5.9104053696870778e+02;
final double cx_d = 3.3930780975300314e+02;
final double cy_d = 2.4273913761751615e+02;
PVector result = new PVector();
double depth = depthLookUp[depthValue];//rawDepthToMeters(depthValue);
result.x = (float)((x - cx_d) * depth * fx_d);
result.y = (float)((y - cy_d) * depth * fy_d);
result.z = (float)(depth);
return result;
}
void stop() {
kinect.quit();
super.stop();
}
And here are the errors:
processing.app.debug.RunnerException: NullPointerException
at processing.app.Sketch.placeException(Sketch.java:1543)
at processing.app.debug.Runner.findException(Runner.java:583)
at processing.app.debug.Runner.reportException(Runner.java:558)
at processing.app.debug.Runner.exception(Runner.java:498)
at processing.app.debug.EventThread.exceptionEvent(EventThread.java:367)
at processing.app.debug.EventThread.handleEvent(EventThread.java:255)
at processing.app.debug.EventThread.run(EventThread.java:89)
Exception in thread "Animation Thread" java.lang.NullPointerException
at org.openkinect.processing.Kinect.enableDepth(Kinect.java:70)
at PointCloud.setup(PointCloud.java:48)
at processing.core.PApplet.handleDraw(PApplet.java:1583)
at processing.core.PApplet.run(PApplet.java:1503)
at java.lang.Thread.run(Thread.java:637)

You are getting a NullPointerException since the value of the depth array is null. You can see from the source code of the Kinect class, there is a chance of a null value being returned by the getRawDepth() method. It is likely that there is no image being displayed at the time.
The code can be found at:
https://github.com/shiffman/libfreenect/blob/master/wrappers/java/processing/KinectProcessing/src/org/openkinect/processing/Kinect.java
Your code should check if the depth array is null before trying to process it. For example...
int[] depth = kinect.getRawDepth();
if (depth == null) {
// do something here where you handle there being no image
} else {
// We're just going to calculate and draw every 4th pixel (equivalent of 160x120)
int skip = 4;
// Translate and rotate
translate(width/2,height/2,-50);
rotateY(a);
for(int x=0; x<w; x+=skip) {
for(int y=0; y<h; y+=skip) {
int offset = x+y*w;
// Convert kinect data to world xyz coordinate
int rawDepth = depth[offset];
PVector v = depthToWorld(x,y,rawDepth);
stroke(255);
pushMatrix();
// Scale up by 200
float factor = 200;
translate(v.x*factor,v.y*factor,factor-v.z*factor);
// Draw a point
point(0,0);
popMatrix();
}
}
// Rotate
a += 0.015f;
}

I would suggest using a Java Debugger so that you can see the state of the variables at the time the exception is thrown. Some people also like to use log statements to output the values of the variables at different points in the application.
You can then trace the problem back to a point where one of the values is not populated with a non-null value.

The null pointer is happening when offset > kinect.getRawDepth();
You have a lot of code here, I'm not going to look at it all. Why can you assume that offset is < kinect.getRawDepth()?
Edit:
On second though, #Asaph's comment is probably right.

Null Pointer exception happens when depth[offset] does not exist or has not been allocated. Check when depth[offset] is undefined and that is the cause of the nullpointer exception.
Check when kinect.getRawDepth(); is greater than offset.

euclidean algorithm for image comparison

i am going to develop an application for image comparison on java. For this i have choosen euclidean algorithm. This application involves with 2 images.
1. Actual image
2. Part of the actual image.
Algorithm should compare the part of the image with actual image. If the part is existed in actual image, it should return one value as matching success.
Can anyone give me the algorithmic steps? code on java will be appreciated..!

Here is a relatively simple idea, with some parts left out intentionally, since the question smells like homework.
public static boolean contains(Image large, Image small) {
final int largeWidth = large.getWidth(), largeHeight = large.getHeight();
final int smallWidth = small.getWidth(), smallHeight = small.getHeight();
if (smallWidth > largeWidth || smallHeight > largeHeight) {
return false;
}
for (int x = 0; x < largeWidth - smallWidth; x++) {
for (int y = 0; y < largeHeight - smallHeight; y++) {
if (subImageEquals(large, x, y, small)) {
return true;
}
}
}
return false;
}
private static boolean subImageEquals(Image large, int x, int y, Image small) {
// TODO: checks whether all pixels starting at (x, y) match
// those of the small image.
}