Develop Reference

How to cheaply deal with multiple ranges (finding a maximum)

I have an amount of ranges, each with a weight. Every point on the total range is scored by the sum of the weights of all the ranges the point falls into. I'd like to be able to cheaply find the total value of points, and would like to be able to find a maximum. Ideally, it would also be able to find the maximum for a set of (equidistantly) spaced points.
Unfortunately, I'm heavily limited by performance, and am struggling to find a good algorithm for this.
The only two decent solutions I could find are:
- Bruteforce it by sampling a bunch of points. For each: check every range whether it fits, find the total value, then check if it's better than the best so far. Decent point samples can be found by taking the boundaries of the ranges.
- Create a set of buckets. Iterate through all the ranges, adding a value to all the buckets that fit within the range. Then iterate through all the buckets to find the best one
Neither are fast enough for my liking (they have been tested), and the latter isn't continuous so has accuracy problems.
I'd be okay with getting a slightly inaccurate response as long as the performance is way better.
What adds a bit of extra complexity to my particular case is that I'm actually dealing with angles, so the environment is modular. The ranges can't be ordered, and I need to ensure that a range going from 340 degrees to 20 degrees contains both a point at 350 and at 10 degrees.
The angle-ranges I'm dealing with can't exceed 180 beyond degrees and only very rarely are above 90.
The amount of ranges generally isn't very high (1-30), but I need to do this calculation a lot.
The language is Java if it matters.

Make a list (array) of angle intervals. If interval finish value less than start value (20<340), add 360 to the finish (340, 380)
Make a list of pair (angle, +weight for start point or -weight for finish point).
Concatenate list with its copy to provide circular intersection. (It is possible to copy only part of list)
Sort them by angle (use +/- as secondary key in case of tie: - before +)
Make CurrWeight=0
Walk through the list, adding +/weight field to CurrWeight. Check for max value.
(Such approach works for linear lists, I tried to modify it for circular ones, perhaps I might miss some caveats)

here, instead of the term 'edges', i should have better used the term 'boundaries', because it referes to interval boundaries
import java.util.ArrayList;
import java.util.Iterator;
import java.util.SortedSet;
import java.util.TreeSet;
public class Main {
ArrayList<Interval> intervals;
public static void main(String args[]) {
Main main = new Main();
main.intervals = new ArrayList<Interval>();
Interval i1 = new Interval(10, 30, 1);
Interval i2= new Interval(20, 40, 1);
Interval i3= new Interval(50, 60, 1);
Interval i4= new Interval(0, 70, 1);
main.intervals.add(i1);
main.intervals.add(i2);
main.intervals.add(i3);
main.intervals.add(i4);
Interval winningInterval = main.processIntervals(main.intervals);
System.out.println("winning interval="+winningInterval);
}
public Interval processIntervals(ArrayList<Interval> intervals)
{
SortedSet<Integer> intervalEdges = new TreeSet<Integer>();
for(int i = 0;i<intervals.size();i++)
{
Interval currentInterval = intervals.get(i);
intervalEdges.add(currentInterval.a);
intervalEdges.add(currentInterval.b);
}
System.out.println(intervalEdges);
//edges stores the same data as intervalEdges, but for convenience, it is a list
ArrayList<Integer> edges = new ArrayList<Integer>(intervalEdges);
ArrayList<Interval> intersectionIntervals = new ArrayList<Interval>();
for(int i=0; i<edges.size()-1;i++)
{
Interval newInterval = new Interval(edges.get(i), edges.get(i+1), 0);
int score = 0; //the sum of the values of the overlapping intervals
for(int j=0; j<intervals.size();j++)
{
if(newInterval.isIncludedInInterval(intervals.get(j)))
score = score+ intervals.get(j).val;
}
newInterval.val = score;
intersectionIntervals.add(newInterval);
}
System.out.println(intersectionIntervals);
int maxValue=0; //the maximum value of an interval
Interval x = new Interval(-1,-1,0);//that interval with the maximum value
for(int i=0; i<intersectionIntervals.size();i++)
{
if(intersectionIntervals.get(i).val > maxValue)
{
maxValue=intersectionIntervals.get(i).val;
x=intersectionIntervals.get(i);
}
}
return x;
}
}
class Interval
{
public int a, b, val;
public Interval(int a, int b, int val) {
super();
this.a = a;
this.b = b;
this.val = val;
}
#Override
public String toString() {
return "Interval [a=" + a + ", b=" + b + ", val=" + val + "]";
}
boolean isIncludedInInterval(Interval y)
{
//returns true if current interval is included in interval y
return this.a>=y.a && this.b<= y.b;
}
}
gives the output
[0, 10, 20, 30, 40, 50, 60, 70]
[Interval [a=0, b=10, val=1], Interval [a=10, b=20, val=2], Interval [a=20, b=30, val=3], Interval [a=30, b=40, val=2], Interval [a=40, b=50, val=1], Interval [a=50, b=60, val=2], Interval [a=60, b=70, val=1]]
winning interval=Interval [a=20, b=30, val=3]
This solves the case when the intervals are straight line intervals, and not angular intervals. I will come back with modifications to take into account the fact that x=x+360.

Find the top N most popular elements

I have a List of TrackDay objects for a runner going around a track field on different days. Each pair of start/finish times signal a single lap run by the runner. We are guaranteed that there is a matching start/finish date (in the order in which they appear in the appropriate lists) :
TrackDay() {
List<DateTime> startTimes
List<DateTime> finishTimes
}
I would like to find the top N days (lets say 3) that runner ran the most. This translates to finding the N longest total start/finish times per TrackDay object. The naive way would be to do the following:
for (TrackDay td : listOftrackDays) {
// loop through each start/finish lists and find out the finish-start time for each pair.
// Add the delta times (finish-start) up for each pair of start/finish objects.
// Create a map to store the time for each TrackDay
// sort the map and get the first N entries
}
Is there a better, more clean/efficient way to do the above?

The problem you're trying to solve is well-known as Selection algorithm, in particular - Quick select. While sorting in general works good, for large collections it would be better to consider this approach, since it will give you linear time instead of N*log(N).

This solution should be linear time. I have assumed that startTimes and finishTimes support random access. I don't know what API your DateTime is part of, so have used java.time.LocalDateTime.
public List<TrackDay> findTop(List<TrackDay> trackDays, int limit) {
limit = Math.min(limit, trackDays.size());
List<Duration> durations = new ArrayList<>(Collections.nCopies(limit, Duration.ZERO));
List<TrackDay> result = new ArrayList<>(Collections.nCopies(limit, null));
int lastIndex = limit - 1;
for (TrackDay trackDay : trackDays) {
Duration duration = Duration.ZERO;
for (int i = 0, n = trackDay.startTimes.size(); i < n; i++) {
duration = duration.plus(Duration.between(trackDay.startTimes.get(i), trackDay.finishTimes.get(i)));
}
Integer destinationIndex = null;
for (int i = lastIndex; i >= 0; i--) {
if (durations.get(i).compareTo(duration) >= 0) {
break;
}
destinationIndex = i;
}
if (destinationIndex != null) {
durations.remove(lastIndex);
result.remove(lastIndex);
durations.add(destinationIndex, duration);
result.add(destinationIndex, trackDay);
}
}
return result;
}

How can I improve my 2 sum algorithm for a range of numbers using a hash table?

I have developed an algorithm to solve the 2 sum problem using a hash table although its performance is dreadful for huge inputs.
My goal is to find all distinct numbers x,y where -10000<= x+y <=10000. By the way, is the performance of my code O(n*m) where n the size of input and m the number of keys on the map?
Here is my code:
import com.google.common.base.Stopwatch;
import java.util.Scanner;
import java.util.HashMap;
import java.util.ArrayList;
import static com.google.common.collect.Lists.newArrayList;
public class TwoSum {
private HashMap<Long, Long> map;
private ArrayList<Long> Ts;
private long result = 0L;
public TwoSum() {
Ts = newArrayList();
for(long i = -10000; i < 10001; i++){
Ts.add(i);
}
Scanner scan = new Scanner(System.in);
map = new HashMap<>();
while (scan.hasNextLong()) {
long a = scan.nextLong();
if (!map.containsKey(a)) {
map.put(a, a);
}
}
}
private long count(){
//long c = 0L;
for (Long T : Ts) {
long t = T;
for (Long x : map.values()) {
long y = t - x;
if (map.containsValue(y) && y != x) {
result++;
}
//System.out.println(c++);
}
}
return result / 2;
}
public static void main(String [] args) {
TwoSum s = new TwoSum();
Stopwatch stopwatch = Stopwatch.createStarted();
System.out.println(s.count());
stopwatch.stop();
System.out.println("time:" + stopwatch);
}
}
sample input:
-7590801
-3823598
-5316263
-2616332
-7575597
-621530
-7469475
1084712
-7780489
-5425286
3971489
-57444
1371995
-5401074
2383653
1752912
7455615
3060706
613097
-1073084
7759843
7267574
-7483155
-2935176
-5128057
-7881398
-637647
-2607636
-3214997
-8253218
2980789
168608
3759759
-5639246
555129
-4489068
44019
2275782
-3506307
-8031288
-213609
-4524262
-1502015
-1040324
3258235
32686
1047621
-3376656
7601567
-7051390
6633993
-6245148
4994051
-4259178
856589
6047000
1785511
4449514
-1177519
4972172
8274315
7725694
-4923179
5076288
-876369
-7663790
1613721
4472116
-4587501
3194726
6195357
-3364248
-113737
6260410
1974241
3174620
3510171
7289166
4532581
-6650736
-3782721
7007010
6007081
-7661180
-1372125
-5967818
516909
-7625800
-2700089
-7676790
-2991247
2283308
1614251
-4619234
2741749
567264
4190927
5307122
-5810503
-6665772
output: 6

The gist of your algorithm can be rewritten in pseudocode as:
for all integers t from -10k to 10k,
for all map keys x,
if t - x in map, and t is not 2*x,
count ++
return count / 2
You can easily improve this a bit:
for all integers t from -10k to 10k,
for the lower half of keys x in ascending order such that t is not 2*x
if t - x in map,
count ++
This makes it go twice as fast (you no longer double-count). However, you need to sort your inputs to ensure map keys in ascending order. You can add them into a TreeSet and then move it into a LinkedHashSet. Using Sets is better than Maps if you do not care about the values, and all the information is in the keys.
Running time is still O(inputs * range), since you have two nested loops, one with range iterations and the other with half your input. This is a fundamental shortcoming of the algorithm, and no amount of optimization will fix it.

The question is an assignment from Algorithms: Design and Analysis
- an online course offered by Stanford University and taught by Prof. Tim Roughgarden. I happen to be taking the same course.
The usual solution for looking up t - i in a hash table is O(n) for a single t, but doing that 20001 * 1000000 times results in roughly 20 billion lookups!
A better solution is to create a sorted set xs from the input file, and ∀i ∈ xs, find all numbers from xs in the range [-10000 - i, 10000 - i]. Since a sorted set, by definition, doesn't have duplicates, so we don't need to worry about any number in the range being equal to i. There's one gotcha though, which is really unclear in the problem statement. It is not only sufficient to find unique (x, y) ∀ x, y ∈ xs, but also that their sum is unique. Obviously, 2 unique numbers may produce equal sums (e.g. 2 + 4 = 1 + 5 = 6). Thus, we need to keep track of the sums too.
Lastly, we can stop once we go past 5000, since there can't be any more numbers to the right that add up to less than 10000.
Here's a Scala solution:
def twoSumCount(xs: SortedSet[Long]): Int = {
xs
.foldLeft(collection.mutable.Set.empty[Long]) { (sums, i) =>
if (i < TenThou / 2) {
xs
// using from makes it slower
.range(-TenThou - i, TenThou - i + 1)
.map(_ + i)
// using diff makes it slower
.withFilter(y => !sums.contains(y))
// adding individual elements is faster than using
// diff/filter/filterNot and adding all using ++=
.foreach(sums.add)
}
sums
}
.size
}
Benchmark:
cores: 8
hostname: ***
name: OpenJDK 64-Bit Server VM
osArch: x86_64
osName: Mac OS X
vendor: Azul Systems, Inc.
version: 11.0.1+13-LTS
Parameters(file -> 2sum): 116.069441 ms

Best choice for in memory data structure for IP address filter in Java

I have file that is CIDR format like this 192.168.1.0/24 and it is converted into this two column strucutre
3232236030 3232235777
Each string IP address convertion happens with this code:
String subnet = "192.168.1.0/24";
SubnetUtils utils = new SubnetUtils(subnet);
Inet4Address a = (Inet4Address) InetAddress.getByName(utils.getInfo().getHighAddress());
long high = bytesToLong(a.getAddress());
Inet4Address b = (Inet4Address) InetAddress.getByName(utils.getInfo().getLowAddress());
long low = bytesToLong(b.getAddress());
private static long bytesToLong(byte[] address) {
long ipnum = 0;
for (int i = 0; i < 4; ++i) {
long y = address[i];
if (y < 0) {
y += 256;
}
ipnum += y << ((3 - i) * 8);
}
return ipnum;
}
Consider that there are over 5 million entries of (low high : 3232236030 3232235777).
Also there will be intersects so the IP can originate from multiple ranges. Just the first one is more than OK.
The data is read only.
What would be the fastest way to find the range the ipToBefiltered belongs to? The structure will be entirely in memory so no database lookups.
UPDATE:
I found this Peerblock project (it has over million download so I'm thinking it must have some fast algorithms):
http://code.google.com/p/peerblock/source/browse/trunk/src/pbfilter/filter_wfp.c
Does anyone know what technique is the project using for creating the list of ranges and than searching them?

When it comes down to it I just need to know if the IP is present in any of the 5M ranges.
I would consider an n-ary tree, where n=256, and work from the dotted address rather than the converted integer.
The top level would be an array of 256 objects. A null entry means "No" there is no range that contains the address, so given your example 192.168.1.0/24 array[192] would contain an object, but array[100] might be null because no range was defined for any 100.x.x.x/n
The stored object contains a (reference to) another array[256] and a range specifier, only one of the two would be set, so 192.0.0.0/8 would end up with a range specifier indicating all addresses within that range are to be filtered. This would allow for things like 192.255.0.0/10 where the first 10 bits of the address are significant 1100 0000 11xx xxxx -- otherwise you need to check the next octet in the 2nd level array.
Initially coalescing overlapping ranges, if any, into larger ranges... e.g. 3 .. 10 and 7 .. 16 becomes 3 .. 16 ... allows this, since you don't need to associate a given IP with which range defined it.
This should require no more than 8 comparisons. Each octet is initially used directly as an index, followed by a compare for null, a compare for terminal-node (is it a range or a pointer to the next tree level)
Worst case memory consumption is theoretically 4 GB (256 ^ 4) if every IP address was in a filtering range, but of course that would coalesce into a single range so actually would be only 1 range object. A more realistic worst-case would probably be more like (256 ^ 3) or 16.7 MB. Real world usage would probably have the majority of array[256] nodes at each level empty.
This is essentially similar to Huffman / prefix coding. The shortest distinct prefix can terminate as soon as an answer (a range) is found, so often you would have averages of < 4 compares.

I would use a sorted array of int (the base address) and another array the same size (the end address). This would use 5M * 8 = 40 MB. The first IP is the base and the second IP is the last address in range. You would need to remove intersections.
To find if an address is filtered to a binary search O(log N) and if not an exact match, check it is less than (or equal to) the upper bound.

I found this binary chop algorithm in Vuze (aka azureus) project:
public IpRange isInRange(long address_long) {
checkRebuild();
if (mergedRanges.length == 0) {
return (null);
}
// assisted binary chop
int bottom = 0;
int top = mergedRanges.length - 1;
int current = -1;
while (top >= 0 && bottom < mergedRanges.length && bottom <= top) {
current = (bottom + top) / 2;
IpRange e = mergedRanges[current];
long this_start = e.getStartIpLong();
long this_end = e.getMergedEndLong();
if (address_long == this_start) {
break;
} else if (address_long > this_start) {
if (address_long <= this_end) {
break;
}
// lies to the right of this entry
bottom = current + 1;
} else if (address_long == this_end) {
break;
} else {
// < this_end
if (address_long >= this_start) {
break;
}
top = current - 1;
}
}
if (top >= 0 && bottom < mergedRanges.length && bottom <= top) {
IpRange e = mergedRanges[current];
if (address_long <= e.getEndIpLong()) {
return (e);
}
IpRange[] merged = e.getMergedEntries();
if (merged == null) {
//inconsistent merged details - no entries
return (null);
}
for (IpRange me : merged) {
if (me.getStartIpLong() <= address_long && me.getEndIpLong() >= address_long) {
return (me);
}
}
}
return (null);
}
Seems to be performing pretty well. If you know about something faster please let me know.

If you just have a CIDR address (or a list of them) and you want to check if some ipAddress is in the range of that CIDR (or list of CIDR's), just define a Set of SubnetUtils objects.
Unless you are filtering a very large N addresses, this is all String comparison and will execute extremely fast. You dont need to build a binary tree based on the higher/lower order bits and all of that complicated Jazz.
String subnet = "192.168.1.0/24";
SubnetUtils utils = new SubnetUtils(subnet);
//...
//for each subnet, create a SubnetUtils object
Set<SubnetUtils> subnets = getAllSubnets();
//...
Use a Guava Predicate to filter the ipAddresses that are not in the range of your set of subnets:
Set<String> ipAddresses = getIpAddressesToFilter();
Set<String> ipAddressesInRange =
Sets.filter(ipAddresses, filterIpsBySubnet(subnets))
Predicate<String> filterIpsBySubnet(final Set<SubnetUtils> subnets){
return new Predicate<String>() {
#Override
public boolean apply(String ipAddress) {
for (SubnetUtils subnet : subnets) {
if (subnet.getInfo().isInRange(ipAddress)) {
return true;
}
}
return false;
}
};
}
Now if the IP is in any of the Subnets, you have a nice simple filter and you dont have to build a data structure that you will have to unit test. If this is not performant enough, then go to optimization. Don't prematurely optimize :)

Here is the beginning of an answer, I'll come back when I get more freetime
Setup:
Sort the ranges by the starting number.
Since these are IP Addresses, I assume that none of the ranges overlap. If there are overlaps, you should probably run the list merging ranges and trimming unnecessary ranges (ex. if you have a range 1 - 10, you can trim the range 5 - 7).
To merge or trim do this (assume range a immediately precedes range b):
If b.end < a.end then range b is a subset of range a and you can remove range b.
If b.start < b.end and b.end > a.end then you can merge range a and b. Set a.end = b.end then remove range b.

Compute the different ways to make (money) change from $167.37?

This was an interview question:
Given an amount, say $167.37 find all the possible ways of generating the change for this amount using the denominations available in the currency?
Anyone who could think of a space and time efficient algorithm and supporting code, please share.
Here is the code that i wrote (working) . I am trying to find the running time of this, any help is appreciated
import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.Map;
public class change_generation {
/**
* #param args
*/
public static void generatechange(float amount,LinkedList<Float> denominations,HashMap<Float,Integer> useddenominations)
{
if(amount<0)
return;
if(amount==0)
{
Iterator<Float> it = useddenominations.keySet().iterator();
while(it.hasNext())
{
Float val = it.next();
System.out.println(val +" :: "+useddenominations.get(val));
}
System.out.println("**************************************");
return;
}
for(Float denom : denominations)
{
if(amount-denom < 0)
continue;
if(useddenominations.get(denom)== null)
useddenominations.put(denom, 0);
useddenominations.put(denom, useddenominations.get(denom)+1);
generatechange(amount-denom, denominations, useddenominations);
useddenominations.put(denom, useddenominations.get(denom)-1);
}
}
public static void main(String[] args) {
// TODO Auto-generated method stub
float amount = 2.0f;
float nikle=0.5f;
float dollar=1.0f;
float ddollar=2.0f;
LinkedList<Float> denominations = new LinkedList<Float>();
denominations.add(ddollar);
denominations.add(dollar);
denominations.add(nikle);
HashMap<Float,Integer> useddenominations = new HashMap<Float,Integer>();
generatechange(amount, denominations, useddenominations);
}
}

EDIT
This is a specific example of the combination / subset problem, answered here.
Finding all possible combinations of numbers to reach a given sum
--- I am retaining my answer below (as it was usefull to someone), however, admittedly, it is not a direct answer to this question ---
ORIGINAL ANSWER
The most common solution is dynamic programming :
First, you find the simplest way to make change of 1, then you use that solution to make change for 2, 3, 4, 5, 6, etc.... At each iteration, you "check" if you can go "backwards" and decrease the amount of coins in your answer. For example, up to "4" you must add pennies. But, once you get to "5", you can remove all pennies, and your solution has only one coin required : the nickel. But then, until 9, you again must add pennies, etc etc etc.
However, the dynamic programming methodology is not gauranteed to be fast.
Alternatively, you can use a greedy method, where you continually pick the largest coin possible. This is extremely fast , but doesnt always give you an optimal solution. However, if your coins are 1 5 10 and 25 , Greedy works perfectly, and is much faster then the linear programming method.

Memoization (kind of) is your friend here. A simple implementation in C:
unsigned int findRes(int n)
{
//Setup array, etc.
//Only one way to make zero... no coins.
results[0] = 1;
for(i=0; i<number_of_coins; i++)
{
for(j=coins[i]; j<=n; j++)
{
results[j] += results[j - coins[i]];
}
}
return results[n];
}
So, what we're really doing here is saying:
1) Our only possible way to make 0 coins is 0 (this is our base case)
2) If we are trying to calculate value m, then let's check each coin k. As long as k <= m, we can use that coin k in a solution
3) Well, if we can use k in a solution, then couldn't we just take the solution for (m-k) and add it to our current total?

I'd try to model this in real life.
If you were at the till and you knew you had to find $167.37 you would probably initially consider $200 as the "simplest" tender, being just two notes. Then, if I had it, I may consider $170, i.e. $100, $50 and $20 (three notes). See where I am going?
More formally, try to over-tender with the minimum number of notes/coins. This would be much easier to enumerate than the full set of possibilities.

Don't use floats, even tiniest inaccuracies will destroy your algorithm.
Go from biggest to lowest coin/banknote. For every possible amount call the function recursively. When there are no more coins left pay the rest in ones and print the solution. This is how it looks in pseudo-C:
#define N 14
int coinValue[N]={20000,10000,5000,2000,1000,500,200,100,50,20,10,5,2,1};
int coinCount[N];
void f(int toSpend, int i)
{
if(coinValue[i]>1)
{
for(coinCount[i]=0;coinCount[i]*coinValue[i]<=toSpend;coinCount[i]++)
{
f(toSpend-coinCount[i]*coinValue[i],i+1);
}
}
else
{
coinCount[i]=toSpend;
print(coinCount);
}
}

import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.Map;
public class change_generation {
static int jj=1;
public static void generatechange(float amount,LinkedList<Float> denominations,
HashMap<Float,Integer> useddenominations) {
if(amount<0)
return;
if(amount==0) {
Iterator<Float> it = useddenominations.keySet().iterator();
while(it.hasNext()) {
Float val = it.next();
System.out.println(val +" :: "+useddenominations.get(val));
}
System.out.println("**************************************");
return;
}
for(Float denom : denominations) {
if(amount-denom < 0)
continue;
if(useddenominations.get(denom)== null)
useddenominations.put(denom, 0);
useddenominations.put(denom, useddenominations.get(denom)+1);
generatechange(amount-denom, denominations, useddenominations);
useddenominations.put(denom, useddenominations.get(denom)-1);
}
}
public static void main(String[] args) {
float amount = 2.0f;
float nikle=0.25f;
float dollar=1.0f;
float ddollar=2.0f;
LinkedList<Float> denominations = new LinkedList<Float>();
denominations.add(ddollar);
denominations.add(dollar);
denominations.add(nikle);
HashMap<Float,Integer> useddenominations = new HashMap<Float,Integer>();
generatechange(amount, denominations, useddenominations);
}
}