Develop Reference

Polynomial Regression values generated too far from the coordinates

As per the the below code for Polynomial Regression coefficients value, when I calculate the regression value at any x point. Value obtained is way more away from the equivalent y coordinate (specially for the below coordinates). Can anyone explain why the difference is so high, can this be minimized or any flaw in understanding. The current requirement is not a difference of more 150 at every point.
import numpy as np
x=[0,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100]
y=[0,885,3517,5935,8137,11897,10125,13455,14797,15925,16837,17535,18017,18285,18328,18914,19432,19879,20249,20539,20746]
z=np.polyfit(x,y,3)
print(z)
I have also tried various various codes available in java, but the coefficient values are same every where for this data. Please help with the understanding.For example
0.019168 * N^3 + -5.540901 * N^2 + 579.846493 * N + -1119.339450
N equals 5 Value equals 1643.76649Y value 885
N equals 10 Value equals 4144.20338Y value 3517
N equals 100; Value=20624.29985 Y value 20746

The polynomial fit performs as expected. There is no error here, just a great deviation in your data. You might want to rescale your data though. If you add the parameter full=True to np.polyfit, you will receive additional information, including the residuals which essentially is the sum of the square fit errors. See this other SO post for more details.
import matplotlib.pyplot as plt
import numpy as np
x = [0,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100]
y = [0,885,3517,5935,8137,11897,10125,13455,14797,15925,16837,17535,18017,18285,18328,18914,19432,19879,20249,20539,20746]
m = max(y)
y = [p/m for p in y] # rescaled y such that max(y)=1, and dimensionless
z, residuals, rank, sing_vals, cond_thres = np.polyfit(x,y,3,full=True)
print("Z: ",z) # [ 9.23914285e-07 -2.67082878e-04 2.79497972e-02 -5.39544708e-02]
print("resi:", residuals) # 0.02188 : quite decent, depending on WHAT you're measuring ..
Z = [z[3] + q*z[2] + q*q*z[1] + q*q*q*z[0] for q in x]
fig = plt.figure()
ax = fig.add_subplot(111)
ax.scatter(x,y)
ax.plot(x,Z,'r')
plt.show()

After I reviewed the answer of #Magnus, I reduced the limits used for the data in a 3rd order polynomial. As you can see, the points within my crudely drawn red circle cannot both lie on a smooth line with the nearby data. While I could fit smooth lines such as a Hill sigmoidal equation through the data, the data variance (noise) itself appears to be the limiting factor in achieving a peak absolute error of 150 with this data set.

cubic interpolation in java

For a Java program I'm making I need to interpolate 4 points, to calculate the y-value of a 5th point, at given x-value.
Say I have the following points:
p1(0, 0)
p2(1, 2)
p3(2, 4)
p4(3, 3)
// The x-values will always be 0, 1, 2, and 3
Now I want to interpolate these points to find what the y-value should be for given x, say x=1.2 (this x value will always be between point 2 and point 3)
Can anyone help me to make a Java method for finding the y coordinate of this fifth point?

Generally given two points (x0,y0) and (x1,y1), for x0 < x < x1, by interpolation,
y = y0 + (x - x0)/(x1 - x0) * (y1 - y0)
Here,
y = 2 + (x-1) * 2

There are an infinite number of lines that can go through all four of your points.
For example, you could assume that at each point the slope is zero and then draw simple s shaped curves between them, or you could assume that the slope is calculated from the neighboring points (with some special decisions about the ends), or you could assume that the slope at the points is 2, etc.
And it doesn't stop there, you could effectively draw a curve that will encapsulate your points and any other point you could imagine while still meeting the requirement of being continuous (as you call, smooth).
I think you need to start off with a formula you wish to fit against the points, and then choose the technique you use to minimize the error of the fit. Without you making more decisions, it is very likely that it will be very hard to give more direction.

xy position at a certain z depth

In processing (java dialect) there are the methods screenX, screenY (and screenZ but we skip that for now).
Let's say i have a object at xyz = 50, 100, 500. Then with screenX and screenY you can now where they will apear on the canvas.
float x = screenX(50, 100, 500);
float y = screenY(50, 100, 500);
here is the reference:
http://processing.org/reference/screenX_.html
What i'm interested in is like a inverse method.
For example, i want a sphere to apear on the canvas on x=175 and y=100. The sphere should have a z of 700. Then what would the actual x and y position be at z=700 to make it apear on the canvas at 175,100?
So the method would be float unscreenX(float x, float y, float z) and it would return the x value.
My math / programming skills is not so advanced (let's call it bad) (i'm more a designer) so i'm looking for some help. I all ready asked at the processing board but often there are more people here with deeper knowledge about matrixes etc.
The normal screenX method from processing can be found here:
https://github.com/processing/processing/blob/master/core/src/processing/opengl/PGraphicsOpenGL.java
public float screenX(float x, float y, float z) {
return screenXImpl(x, y, z);
}
protected float screenXImpl(float x, float y, float z) {
float ax =
modelview.m00*x + modelview.m01*y + modelview.m02*z + modelview.m03;
float ay =
modelview.m10*x + modelview.m11*y + modelview.m12*z + modelview.m13;
float az =
modelview.m20*x + modelview.m21*y + modelview.m22*z + modelview.m23;
float aw =
modelview.m30*x + modelview.m31*y + modelview.m32*z + modelview.m33;
return screenXImpl(ax, ay, az, aw);
}
protected float screenXImpl(float x, float y, float z, float w) {
float ox =
projection.m00*x + projection.m01*y + projection.m02*z + projection.m03*w;
float ow =
projection.m30*x + projection.m31*y + projection.m32*z + projection.m33*w;
if (nonZero(ow)) {
ox /= ow;
}
float sx = width * (1 + ox) / 2.0f;
return sx;
}
Of corse there is also for the y and for the z (i don't understand the z but let's ignore that).
I thought this might give some insight in how to inverse it.
modelview and projection is a 3d matrix, the code is here:
https://github.com/processing/processing/blob/master/core/src/processing/core/PMatrix3D.java
But i guess it's pretty basic and common.
I also made a post on the processing board since you never know. It explains a litle bit different what i want.
http://forum.processing.org/topic/unscreenx-and-unscreeny
For the tags describing this post, i didn't went to specific cause i can imagine a programmer who never worked with java but did work with c++ for example and has experience in matrixes is still able to provide a good answer.
hope someone can help.

I highly recommend you study some linear algebra or matrix math for 3d graphics. It's fun and easy, but a bit longer than a SO answer. I'll try though :) Disclaimer: I have no idea about the API you are using!
It looks like you are returning 3 coordinate for a position (often called a vertex). But you also mention a projection matrix and that function has 4 coordinates. Usually a shader or API will take 4 coordinates for a vertex. x,y,z,w. To get them on screen it does something like this:
xscreen = x/w
yscreen = y/w
zbuffer = z/w
This is useful because you get to pick w. If you are just doing 2d drawing you can just put w=1. But if you are doing 3d and want some perspective effect you want to divide by distance from the camera. And that's what the projection matrix is for. It mainly takes the z of your point where z means distance to camera and puts it into w. It also might scale things around a bit, like field of view.
Looking back at the code you posted this is exactly what the last ScreenXImpl function does.
It applies a projection matrix, which mostly just moves z into w, and then divides by w. At the end it does an extra scale and offset from (-1,1) to (0,widhtinpixels) but we can ignore that.
Now, why am I rambling on about this stuff? All you want to do is to get the x,y,z coordinates for a given xscreen, yscreen, zbuffer, right? Well, the trick is just going backwards. In in order to do that you need to have a firm grasp on going forward :)
There are two problems with going backwards: 1) Do you really know or care for the zbuffer value? 2) Do you know what the projection matrix did?
For 1) Let's say we don't care. There's many possible values for that, so we might just pick one. For 2) You will have to look at what it does. Some projection matrices might just take (x,y,z,w) and output (x,y,z,1). That would be 2d. Or (x,y+z,z,1) which would be isometric. But in perspective it usually does (x,y,1,z). Plus some scaling and so on.
I just noticed your second screenXImpl already passes x,y,z,w to the next stage. That is useful sometimes, but for all practical cases that w will be 1.
At this point I realize that I am terrible at explaining things. :) You really should pick up that linear algebra book, I learned from this one: http://www.amazon.com/Elementary-Linear-Algebra-Howard-Anton but it came with a good lecture, so I don't know how useful it is on it's own.
Anyhow! Let's get more practical. Back to your code: the last function of screenXImpl. We now know that the input w=1 and that ow=~z and ox=~x; The squiggly line here means times some scale plus some offset. And the screen x we have to begin with is ~ox/ow. (+1/2,*width.. that's what squiggly lines are for). And now we are back at 1)... if you want a special oz - pick one now. Otherwise, we can pick any. For rendering it probably makes sense to pick anything in front of the camera and easy to work with. Like 1.
protected float screenXImpl(float x, float y, float z, float w==1) {
float ox = 1*x + 0*y + 0*z + 0*w; // == x
float ow = 0*x + 0*y + 1*z + 0*w; // == z == 1
ox /= ow; // == ox
float sx = width * (1 + ox) / 2.0f;
return sx;
}
WTF? sx = width * (1+ox)/2 ? Why didn't I just say so? Well, all the zeros I put in there are probably not zero. But it's going to end up just as simple. Ones might not be ones. I tried to show the important assumptions you have to make to be able to go back. Now it should be as easy as going back from sx to ox.
That was the hard part! But you still have to go from the last function to the second one. I guess the second to the first is easy. :) That function is doing a linear matrix transform. Which is good for us. It takes an input of four values (x,y,z) and (w=1) implicit and outputs four other values (ax,ay,az,aw). We could figure out how to go back there manually! I had to do that in school.. four unknowns, four equations. You know ax,ay,az,aw... solve for x,y,z and you get w=1 for free! Very possible and a good exercise but also tedious. The good news is that the way those equations are written is called a matrix. (x,y,z,1) * MODELMATRIX = (ax,ay,az,aw). Really convenient because we can find MODELMATRIX^-1. It's called the inverse! Just like 1/2 is the inverse of 2 for multiplying real numbers or -1 is the inverse of 1 for addition. You really should read up on this it's fun and not hard, btw :).
Anyhow, use any standard library to get the inverse of your model matrix. Probably something like modelView.Inverse(). And then do the same function with that and you go backwards. Easy!
Now, why did we not do the same thing with the PROJECTION matrix earlier? Glad you asked! That one takes 4 inputs(x,y,z,w) and spits out only three outputs (screenx,screeny,zbufferz). So without making some assumptions we could not solve it! An intuitive way to look at that is that if you have a 3d point, that you project on a 2d screen, there's going to be a lot of possible solutions. So we have to pick something. And we can not use the convenient matrix inverse function.
Let me know if this was somewhat helpful or not. I have a feeling that it's not, but I had fun writing it! Also google for unproject in processing gives this: http://forum.processing.org/topic/read-3d-positions-gluunproject

You'd need to know the project matrix before you can make this work, which Processing doesn't supply you with. However, we can work it out ourselves by checking the screenX/Y/Z values for for the three vectors (1,0,0), (0,1,0) and (0,0,1). From those we can work out what the plane formula is for our screen (which is technically just a cropped flat 2D surface running through the 3D space). Then, given an (x,y) coordinate on the "screen" surface, and a predetermined z value, we could find the intersection between the normal line through our screen plane, and the plane at z=....
However, this is not what you want to do, because you can simply reset the coordinate system for anything you want to do. Use pushMatrix to "save" your current 3D transforms, resetMatrix to set everything back to "straight", and then draw your sphere based on the fact that your world axes and view axes are aligned. Then when you're done, call popMatrix to restore your earlier world transform and done. Save yourself the headache of implementing the math =)

You can figure this out with simple trigonometry. What you need is, h, the distance of the eye from the center of the canvas, and the cx and cy representing the center of the canvas. For simplicity, assume cx and cy are 0. Note that it is not the distance of your actual eye but the distance of the virtual eye used to construct the perspective on your 3d scene.
Next, given sx and sy, compute the distance to center, b = sqrt(sx * sx + sy * sy)
Now, you have a right-angled triangle with base b and height h. This triangle is formed by the "eye", the center on canvas, and the desired position of the object on the screen: (sx, sy).
This triangle forms the top part of another right-angled triangle formed by the "eye", the center on canvas pushed back by given depth z and the object itself: (x, y).
The ratio of the base and height of the triangles is exactly the same, so it should be trivial to calculate the base of the larger triangle bb given its height hh = z or hh = h + z depending on whether the z value is from the eye or from the canvas. The equation to use is b / h = bb / hh where you know b, h and hh
From there you can easily compute (x, y) because the two bases are at the same angle from the horizontal. I. e. sy / sx = y / x.
The only messy part will be to extract the distance of eye to canvas and the center of the canvas from the 3d setup.

Overview of Transforming 3d point onto your 2d screen
When you have a 3d representation of your object (x,y,z), you want to 'project' this onto your monitor, which is in 2d. To do this, there is a transformation function that takes in your 3d coordinates, and spits out the 2d coordinates. Under the hood, (at least in openGL), the transformation function that takes place is a special matrix. To do the transformation, you take your point (represented by a vector), and do a simple matrix multiplication.
For some nice fancy diagrams, and a derivation (if you are curious, it isn't necessary), check this out: http://www.songho.ca/opengl/gl_projectionmatrix.html
To me, screenXImpl looks like it is doing the matrix multiplication.
Doing the reverse
The reverse transformation is simply the inverse matrix of the original transformation.

Java, Generate a direct path of points between two points

I currently have a bad method for doing this, it just translates the start point by 1/-1 depending on if the x/y coordinate is over or under the current coordinates and adds it to an ArrayList until the start point .equals(end), this is a bad solution because it creates a really bad path that looks like the black path below.
I'm trying to generate a direct path of points between two points (The same kind of line as Graphics.drawLine makes).
I'm guessing I need to use the Math class to get the angle, but I'm not very familiar with the Math API.

One somewhat naive way, is to work out the ratio of the slope, rather than the angle.
Something like:
float ratio = (y2 - y1) / (x2 - x1);
Then:
width = x2 - x1;
for(int i = 0; i < width; i++) {
float x = x1 + i;
float y = y1 + (ratio * i);
points.add(new Point(x,y));
}
If float isn't what you need, you'll need to do some conversion.
You'll also need to handle the special case of x1 == x2, or you'll get divide-by-zero errors.
Using this method, the steeper the line, the fewer points you will generate. If you want the points evenly spaced no matter what the angle, you'll need to break out sin/cos/tan.
Write unit tests for lines in at least the main 8 compass directions, to iron out any glitches.

This is actually more of a math problems than a programming problem. You need to find the vector that goes from start to end and then scale it and add it to start for the number of points that you want.
In pseudo code:
f(start,end,nPoints) -> (path)
delta = (end-start) / nPoints //Find the best difference vector.
current = start //Set the current point to the start.
i = 0
while(|current-end| < epsilon)
current += delta
path[i] = current
i = i + 1
I'm assuming that the points are floating point (due to the division). epslion should be chosen to be a small value (depends on your problem) to check equality (never use != for floating points!).

This is a very simple concept, but you've not been too clear on what exactly you want. By definition, there are an infinite number of points between any two points (and between two of those points, exist an infinite number of points. And so on.)
The simplest solution would be to calculate the formula for the line between point A and point B, and then decide how far apart you want your points to be, and calculate those points using your line formula.
Elementary geometry tells us that the slope between two points in the change in y over the change in x. So m = (y1 - y2)/(x1 - x2), where all of these values are doubles, and x1 and y1 belong to the same point.
Then the formula for the line between two points is y - y1 = m(x - x1). Then all you'd have to do is start plugging in values for, say, x, taking the resultant y, and drawing it.
This is, of course, the idea behind the whole thing. You'd be much better off using vectors.

Check out this previously answered question to calculate the angle. And then use the distance formula to find the distance.
double dist = Math.sqr((x2 - x1)^2 + (y2 - y1)^2); //obviously wont compile but you get the idea.

Best algorithm for comparing a base coordinate to a list of n coordinates and determining the closest m coordinates?

I have some code doing this right now. It works fine with small to medium sized lists, but when I have a list of size n > 5000 then the my algorithm can take almost 1 minute on a mobile device to run. I'm basically comparing a Coordinate object in Java to a list (Vector) of Coordinate objects.
Here's my basic algorithm:
traverse each element in the list nx
if there is less 10 items in the "10 closest" list then add nx to the list
and go to the next element
if the "10 closest" list has 10 items already, then calculate the
distance between nx and the base
Coordinates
if the distance is less than furthest distance in the "10 closest
list" then remove the furthest item
from that list and replace it with nx
I keep looking at this and am trying to find a more efficient way of doing this. It's sort of like a sorting algorithm problem so there must be a better way.
Here is my distance calculation method:
public static double distance(double lat1, double lon1, double lat2, double lon2, char unit) {
double theta = lon1 - lon2;
double dist = Math.sin(deg2rad(lat1)) * Math.sin(deg2rad(lat2)) + Math.cos(deg2rad(lat1)) * Math.cos(deg2rad(lat2)) * Math.cos(deg2rad(theta));
dist = acos(dist);
dist = rad2deg(dist);
dist = dist * 60 * 1.1515;
if (unit == 'K') {
dist = dist * 1.609344;
} else if (unit == 'N') {
dist = dist * 0.8684;
}
return (dist);
}

You could store your coordinates in some space partitioning tree.
Or, for a simpler approach, you could use a two-dimensional array of buckets, and check the closest buckets first, until you found enough nearest neighbors. This only works well if the coordinates are distributed evenly.
Edit: To compare the distances you could precompute 3D coordinates on the sphere and use the square of the Euclidean distance in the comparisons:
dx * dx + dy * dy + dz * dz

Well, maybe it would be faster to do this with arrays. And you could compare the square of the distance instead of the distance, which means that you don't have to work with square roots.
It would be good to have the actual code.

You might be able use something like the approach at this website to restrict the number of points that actually require you to compute the distance.
The website shows how to compute the lat, lon bounding coordinates for a point and a given distance. That is not exactly the same problem that you have, but it could serve as a filter. In your case you are apparently trying to find the 10 (or n) closest points to a given point. You could apply the following algorithm to find the 10 (or n) closest points:
For the first n points, you could go through the full-blown distance
calculation that you have, saving the distance along each point.
Save the overall longest distance. Compute the lat, lon bounding
box as illustrated on the website above.
Continue through the rest of your points.
If any point is outside of the lat, lon bounding box, it cannot be
closer than any of the current 10 closest points. If it is inside
the bounding box, calculate the distance.
Discard the farthest of the previous set of 10 "closest" points.
Recompute the lat, lon bounding box based on the new farthest point.
Repeat until all points processed.
The benefit of this approach is that you might be able to avoid heavy calculations for a large number of your points. Depending on the distribution of your points, you could still suffer from poor performance, such as if the points turn out to be ordered such that they are in decreasing distance from your target points (point[0] is the farthest and point[N] is the closest)).