Develop Reference

What is the fastest way to calculate nearby points from large data set of points

I have a large set of 3d points, (20,000+), scattered throughout a 3d space. I need to identify which points are within a specific arbitrary range of each point in the set. For example, for each point, what is the group of points that is within a range of 10 units. The permutations for this are pretty big. So, what would be the most computationally efficient way to approach this ? (I need to solve this using java only.)

You can use k-d tree, which is basically a k-dimensional binary tree. Range search in k-d tree is very efficient.

Since it's a theorical question without code, I will throw my 2 cents here. If you don't use any geometrical DB like postgis (http://postgis.net/), I will suggest the following with the premise that the points have three coordinates (X, Y, Z).
Make three Arrays containing the id of the point and one of the coordinates. Sort them by coordinates. Then for each points, check if the last and the next are within range. If the two are not, eliminate that point. Make that for each array. You will then have a much less space to compute. Then for each points within the reach of a point, calculate the distance and flag, eliminate the fartest points.
Hope this will help.

You can use a space filling curve and approximate. Treat the points as a binary and interleave it. Then sort the numbers and exploit that the curve visits nearby points first. You can try many curves most likely it depends on the points.

Sounds like you need an R-tree. Or maybe a range tree like a kd-tree, it will return all points in a box, and then you just filter all at the desired distance from your query point.

Using an ArrayList that is preallocated to full size. (Uses a cube shape zone)
public class Point3D {
public int x, y, z;
public static List<Point3D> allWithinRange(List<Point3D> possiblePoints, int x, int y, int z, int inter) {
List<Point3D> list = new ArrayList<Point3D>(possiblePoints.size());
possiblePoints.stream()
.filter(it -> it.x <= x + inter && it.x >= x - inter)
.filter(it -> it.y <= y + inter && it.y >= y - inter)
.filter(it -> it.z <= z + inter && it.z >= z - inter)
.forEach(list::add);
return list;
}
}

How to convert an angle to a vector

I'm making a simple 2d Java game similar to goemetry wars. I'm currently programming the player's shooting.
I have a target point specified by the mouse location. Now I want to add a bit of randomization, so that the shooting angle has a random offset. I am currently converting the point to a vector directly. My idea was to convert the vector to an angle, apply the randomization to that angle, and then, convert the angle back to a vector (given the length of the vector).
I can't figure out how to convert the angle back to a vector. I don't need code, it's basically just a question about the maths.
If there's a better way to randomize the shooting, I would love to hear that too! I can!t apply the randomization to the point because there are other things then the mouse that can set it.

Polar coordinate system
As everybody seems to have just answered in the comments, here goes the answer to your question as it is formulated : you need to use the polar coordinate system.
Let's call your angle a, the angle you want to add to it b, so the modified angle is a+b.
In the polar coordinate system, your point is represented by an angle a = Math.atan2(y, x) and a radius r = sqrt(x*x + y*y). If you just use the angle, you loose some information, which is at which distance the mouse is from your (0,0) point.
Converting back from your polar representation (after the angle has been modified) is now possible : x = r * Math.cos(a+b), y = r * Math.sin(a+b)
Without computing the angle
By using some cool trigonometry formulas, we can do better. We don't need to go to an angle and come back to a vector, we can modify the x and y values of the vector directly, still changing the angle like you want.
Remember that we want to find x'=r cos(a+b) and y'=r sin(a+b). We will obviously the following formulas :
Now let us multiply by r on both sides to get what whe want.
We now recognize x=r cos(a) and y=r sin(a), so we have the final expression :
If you come to think of it, for small values of b (which is what you want), sin(b) is close to 0, and cos(b) is close to 1, so the perturbation is small.
Not only do you reduce the number of trigonometry operations from 3 to 2, but you also have a cos and a sin of small values, which is nice if they are computed from Taylor series, makes convergence very fast. But that's implementation dependent.
NB: An alternative (and more elegant?) way to find the same result (and exact same formulas) is to use a rotation matrix.
I can haz cod
Whoopsie, you said you didn't want code. Well let's do it like this : I don't post it here, but if you want to compare with how I'd do it once you're done coding your implementation, you can see mine in action here : http://ideone.com/zRV4lL

Java - Detect Straight Lines with given Coordinates

ADDED INFO:
I'm using the inside of a square as an arena. On start up, the square spawns in a random position, and rotation, and I can't access any of the squares attributes.
I then have a moving object inside the square, that I'm building AI for, and I want the object to 'learn' where the arena walls are. Every time the object bumps into a wall, I get a touch return, so I know if its hit or not. I'm using this to map the global position of where the object hit the wall and save it ... After 3 hits on the same wall, I want to mathematically 'draw a straight line' down those dots which will represent the arena walls - with this, I can tell my object not to go near these coordinates.
The reason for 3 dots? Well, if the object hit one side of a wall, then was to hit another side of a wall, I will have a line drawn from one side to another, giving false data about where the wall is.
If Java sees three (or more) dots inline, it knows that the object has hit the same wall (either being further up or so).
CONTINUED:
I'm trying to map out lines with given coordinate data. Basically I have an array that holds X and Y coordinates, and I want to be able to mathematically detect if it's they make up a straight line (give or take a few pixels). (The coordinate are a boarder of a square)
For example, the array might be like this:
[x0][y0] - 1,1
[x1][y1] - 2,2
[x2][y2] - 5,5
Which will present a diagonal line of on side of the square, like so:
But sometimes I might get one coordinate of one side of the square, and then another side, all mixed up (and not necessarily on a 90 degree angle either!). So I want to be able to run through the array, and detect what coordinates make a line (or the boarder side of the square), like so:
So right now, I have a 2D array:
private double wallLocations[][] = new double[10][10];
and a while loop that doesn't do the job. I don't really know where to even start with this one:
for(int r = 0; r < wallIndex; r++){
for(int c = 0; c < wallIndex; c++){
int index = 0;
if(wallLocations[r][index] == wallLocations[r][c] && wallLocations[r][index + 1] == wallLocations[r][c] &&
wallLocations[r][index + 2] == wallLocations[r][c]){
System.out.println("***** Wall Here! *****");
index++;
}
}
}
---- UPDATE ----
Heres a better example in what I'm looking for. The red dots represent the coordinates coming in, a line is detected when 3 or more dots line up (if it was 2 dots, then it would detect any and ever dot) ... You notice that this is starting to look like the boarder of a square?

This seems to essentially be a clustering problem, and those can be generally pretty hard. Part of a reason clustering is hard is that there may be more than one applicable mapping.
For instance (please forgive my bad ascii art):
X X X
X X X
X X X X
could be mapped
X---X---X X X X
\ / \ / \
X---X---X or X X X
/ \ / \ \
X---X---X---X X X X X
I've seen uses of the Expectation Maximization algorithm using mixed Gaussian models used for this kind of thing (when there were a lot of points but only a few expected lines) but you generally do have to give that algorithm a definite number of clusters, and while its results are good, it's a pretty slow algorithm requiring possibly many iterations. I'm kinda thinking I've seen something generally faster that's some sort of image processing algorithm but I'd have to do some research.
I'm kinda wondering about something where you find y=mx+b for every pair of points and them sort them over m and b. It might be advantageous to find the angle θ in [0,pi) for each pair instead and sort over the angles instead of m, or maybe beter cluster by cos(2θ)-- the point of that being that the group of lines {y= -0.0001x + 1, y =1, and y=0.0001x + 1} are very similar, the group of lines {y= -10000x + 10, x = 0, and y=10000x - 10} are also very similar, but cos(2θ) should put them as far apart as possible, as any two pairs between each group should be nearly perpendicular.
Also note, in my example, b doesn't matter much for the lines nearly perpendicular to the x axis, so "b" might not be so useful for direct clustering.
I guess, perhaps, there may be some utilizable measure of "distance" between two lines, I'm just not sure what it would be. Two lines that are nearly parallel that converge "on screen" (where the points generally are) might ought to be considered "closer" than if they converge a trillion units a way from the screen--or should they? Purely speaking, three lines can never pairwise be considered closer to one another if none of them are parallel (If they're on a plane, they'll all meet somewhere), but intuitively, if we have two lines that are generally one inch apart in the area we're concerned with, we'd pick that pair as closer over two identically pointed lines that are a mile apart in the area of concern. That makes me think maybe the area between the lines,as bound by our area* ought to be used as a metric.
Sorry, I'm not sure how useful all that brainstorming might be, but it might put a different light on things.
Edit: You know what, a better answer might possibly be found by researching this:
http://en.wikipedia.org/wiki/Hough_transform
Edit II & III:
Ok,... the situation you've just described is a lot simpler and less generic (although, to be honest, I think I misread your initial query to be more generic than it really was).
You've got 4 candidate walls. Let your AI bounce around until it finds three co-linear points. That should be a simple test of combinations. Assign those three points a wall. Depending upon what other points you have, you actually might be able to determine or at least estimate the other three walls (assuming it is a square). If you have 5 points with 3 on separate walls, you should be able to calculate the distance between walls, and therefore the likely position of the 4th wall. To test if the other two points are on separate walls, make sure they're pair-wise not co-linear with a line perpendicular or parallel to the line defined by your wall, or if they are on a line parallel, test to see if the distance between them is less than the distance between the wall and them (if that's the case, they're on the wall opposite of the first candidate wall). Given that they are on separate walls, either one is facing the first found wall, or they're on walls perpendicular to that wall. Either way you can find the lines defining the walls with a little tricky geometry.
(and actually, to determine the dimensions, I don't think you need to even test to see that you have 3 co-linear points... I think you just need to test to see that you've made two turns... which takes 4 points minimum but likely more if you're unlucky. two of the points would have to be determinable to be on a different wall from the other two, which means really big bounces!)
There's a bit of math involved, and I'm a bit too tired to explain further tonight, and I don't know how much of the geometry of points around a square you want to take advantage of, because you wouldn't be able to use those properties in a more general case, so I'll leave it at that, and maybe also remove some of my other previous brainstorm cruft later.

If you have two points you can calculate the slope of the connecting line with Math.atan2.

Euclidean distance to calculate if something is near something

currently i have using a framework and it has a function called distance2D, and it has this description:
Calculate the Euclidean distance
between two points (considering a
point as a vector object). Disregards
the Z component of the vectors and is
thus a little faster.
and this is how i use it
if(g.getCenterPointGlobal().distance2D(target.getCenterPointGlobal()) > 1)
System.out.println("Near");
i have totally no idea what a Euclidean distance is, i am thinking that it can be used to calculate how far 2 points are? because i am trying to compare distance between 2 objects and if they are near within a certain range i want to do something. how would i be able to use this?

Euclidean distance is the distance between 2 points as if you were using a ruler. I don't know what are the dimensions of your Euclidean space, but be careful because the function you are using just takes in consideration the first two dimensions (x,y). Thus if you have a space with 3 dimensions(x,y,z) it will only use the first two(x,y of x,y,z) to calculate the distance. This may give a wrong result.
For what I understood, if you want to trigger some action when two points are within some range you should make:
<!-- language: lang-java -->
if(g.getCenterPointGlobal().distance2D(target.getCenterPointGlobal()) < RANGE)
System.out.println("Near");

The Euclidean distance is calculated tracing a straight line between two points and measuring as the hypotenuse of a imaginary isosceles triangle between the two lines and a complementary point. This measure is scalar, so it's a good metric for your calculations.

Euclidean geometry is a coordinate system in which space is flat, not curved. You don't need to care about non-Euclidean geometry unless for example you're dealing with coordinates mapped onto a sphere, such as calculating the shortest travel distance between two places on Earth.
I imagine this function will basically use Pythagoras' theorem to calculate the distance between the two objects. However, as the description says, it disregards the Z component. In otherwords, it will only give a correct answer if both points have the same Z value (aka "depth").

If you wish to compare distances and save time, use not the distance itself, but its square: (x1-x2)^2 + (y1-y2)^2. Don't take sqrt. So, your distances will work exactly as euclidian ones, but quickly.

Mapping Pixels to Data

I've written some basic graphing software in Clojure/Java using drawLine() on the graphics context of a modified JPanel. The plotting itself is working nicely, but I've come to an impasse while trying to converting a clicked pixel to the nearest data point.
I have a simple bijection between the list of all pixels that mark end points of my lines and my actual raw data. What I need is a surjection from all the pixels (say, 1200x600 px2) of my graph window to the pixels in my pixel list, giving me a trivial mapping from that to my actual data points.
e.g.
<x,y>(px) ----> <~x,~y>(pixel points) ----> <x,y>(data)
This is the situation as I'm imagining it now:
A pixel is clicked in the main graph window, and the MouseListener catches that event and gives me the <x,y> coordinates of the action.
That information is passed to a function that returns a predicate which determines whether or not a value passed to it is "good enough", and filter though the list with that pred, and take the first value it okays.
Possibly, instead of a predicate, it returns a function which is passed the list of the pixel-points, and returns a list of tuples (x index) which indicate how good the point is with the magnitude of x, and where that point is with index. I'd do this with both the x points and the y points. I then filter though that and find the one with the max x, and take that one to be the point which is most likely to be the one the user meant.
Are these reasonable solutions to this problem? It seems that the solution which involves confidence ratings (distance from pix-pt, perhaps) may be too processor heavy, and a bit memory heavy if I'm holding all the points in memory again. The other solution, using just the predicate, doesn't seem like it'd always be accurate.
This is a solved problem, as other graphing libraries have shown, but it's hard to find information about it other than in the source of some of these programs, and there's got to be a better way then to dig through the thousands of lines of Java to find this out.
I'm looking for better solutions, or just general pointers and advice on the ones I've offered, if possible.

So I'm guessing something like JFreeChart just wasn't cutting it for your app? If you haven't gone down that road yet, I'd suggest checking it out before attempting to roll your own.
Anyway, if you're looking for the nearest point to a mouse event, getting the point with the minimum Euclidean distance (if it's below some threshold) and presenting that will give the most predictable behavior for the user. The downside is that Euclidean distance is relatively slow for large data sets. You can use tricks like ignoring the square root or BSP trees to speed it up a bit. But if those optimizations are even necessary really depends on how many data points you're working with. Profile a somewhat naive solution in a typical case before going into optimization mode.

I think your approach is decent. This basically only requires one iteration through your data array, a little simple maths and no allocations at each step so should be very fast.
It's probably as good as you are going to get unless you start using some form of spatial partitioning scheme like a quadtree, which would only really make sense if your data array is very large.
Some Clojure code which may help:
(defn squared-distance [x y point]
(let [dx (- x (.x point))
dy (- y (.y point))]
(+ (* dx dx) (* dy dy))))
(defn closest
([x y points]
(let [v (first points)]
(closest x y (rest points) (squared-distance x y v) v)))
([x y points bestdist best]
(if (empty? points)
best
(let [v (first points)
dist (squared-distance x y v)]
(if (< dist bestdist)
(recur x y (rest points) dist v)
(recur x y (rest points) bestdist best))))))