I am working on a sharding problem.
Imagine I have 10 lists.
Each list has a series of items that are independently sorted.
I want to get the Nth item as if all the lists were sorted together in one large list.
Do I need to sort the lists overall to get an item at a particular index?
I solved a similar but not equivalent problem where there is:
10 lists
Each list represents a range of items that are after the previous list.
here's the code to iterate through all the indexes of the lists:
/* code to iterate through all items in order
* threads refers to one of the lists */
int sizes[] = new int[threads.size()];
for (int i = 0 ; i < threads.size(); i++) {
sizes[i] = threads.get(i).data2.size();
}
int n = 0;
int thread = 0;
int size = threads.size();
int offset = 0;
long iterationStart = System.nanoTime();
while (thread < size) {
// System.out.println(String.format("%d %d", thread, offset + threads.get(thread).data.get(n)));
int current = offset + threads.get(thread).data.get(n);
n = n + 1;
if (n == sizes[thread]) {
offset += sizes[thread];
thread++;
n = 0;
}
}
long iterationEnd = System.nanoTime();
long iterationTime = iterationEnd - iterationStart;
Here's the code to lookup an item by index.
int lookupKey = 329131;
int current = lookupKey;
int currentThread = 0;
int total = 0;
while (current >= 0 && currentThread <= size - 1) {
int next = current - sizes[currentThread];
if (next >= 0) {
total += sizes[currentThread];
current -= sizes[currentThread];
currentThread++;
} else {
break;
}
}
long lookupEnd = System.nanoTime();
long lookupTime = lookupEnd - lookupStart;
System.out.println(String.format("%d %d",
currentThread,
total + threads.get(currentThread).data.get(current)));
I'm hoping there's some property of sorted collections that I can use to retrieve the Nth item in an overall sorted lists.
What I have in effect is multiple partial orders.
I have some other code that does a N way merge between multiple sorted lists. Is the fastest option to run this in a loop up to lookupIndex?
int size1 = threads.size();
int[] positions = new int[size1];
Arrays.fill(positions, 0);
PriorityQueue<Tuple> pq = new PriorityQueue<>(new Comparator<Tuple>() {
#Override
public int compare(Tuple o1, Tuple o2) {
return o1.value.compareTo(o2.value);
}
});
long startOrderedIteration = System.nanoTime();
for (ShardedTotalRandomOrder thread : threads) {
for (int i = 0; i < 10; i++) {
// System.out.println(thread.data2.get(i));
pq.add(thread.data2.get(i));
}
}
List<Integer> overall = new ArrayList<>();
while (!pq.isEmpty()) {
Tuple poll = pq.poll();
ArrayList<Tuple> data2 = threads.get(poll.thread).data2;
if (positions[poll.thread] < data2.size()) {
Tuple nextValue = data2.get(positions[poll.thread]++);
pq.offer(nextValue);
}
overall.add(poll.value);
// System.out.println(String.format("%d %d", poll.thread, poll.value));
}
System.out.println(overall);
long endOrderedIteration = System.nanoTime();
long orderedIterationTime = endOrderedIteration - startOrderedIteration;
You don't need to resort them. Since each list is already sorted you can merge them as follows. This uses a single method to merge two lists based on their relative values. Then it returns that list and feeds it back into the method to merge it with the next list.
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
public class Merging {
public static void main(String[] args) {
List<Integer> list1 = List.of(5,10,15,20,25,30,35,40,45,50);
List<Integer> list2 = List.of(2,4,6,8,10);
List<Integer> list3 = List.of(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20);
int nth = 10;
List<List<Integer>> lists = List.of(list1,list2,list3);
List<Integer> merged = lists.get(0);
for (int i = 1; i < lists.size(); i++) {
merged = mergeLists(merged, lists.get(i));
}
System.out.println(merged.get(nth));
}
prints
7
This works with any type that implements the Comparable interface.
It will loop until one list is exhausted or until both indices exceed the combined list size.
Once either list is finished, the other can be appended via the sublist.
public static <T extends Comparable<? super T>> List<T> mergeLists(List<T> list1, List<T> list2) {
List<T> merged = new ArrayList<>();
int i1 = 0;
int i2 = 0;
while (i1 + i2 < list1.size() + list2.size()) {
if (i1 >= list1.size()) {
merged.addAll(list2.subList(i2,list2.size()));
break;
}
if (i2 >= list2.size()) {
merged.addAll(list1.subList(i1,list1.size()));
break;
}
if(list1.get(i1).compareTo(list2.get(i2)) <= 0) {
merged.add(list1.get(i1++));
} else {
merged.add(list2.get(i2++));
}
}
return merged;
}
}
Here is a relatively efficient (linear with respect to the number of lists) algorithm that leverages some of the power of streams, but avoids a full list merge.
EDIT: To address shortcomings such as array length checking, array destruction, and readability I have improved this example. For better comparison, I have used the same integer test data as the other answer.
This virtual queue backed by the (presumably) immutable array will not mutate or otherwise
public class VirtualQueue<T> {
private List<T> list;
private int index=0;
public VirtualQueue(List<T> list) { this.list = list; }
public boolean hasMore() { return index < list.size(); }
public T pop() { return list.get(index++); }
public T peek() { return list.get(index);}
}
(I suspect that there is an easier way to do this with standard collections)
List<Integer> list1 = List.of(5,10,15,20,25,30,35,40,45,50);
List<Integer> list2 = List.of(2,4,6,8,10);
List<Integer> list3 = List.of(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20);
List<VirtualQueue<Integer>> listList = List.of(
new VirtualQueue<>(list1),
new VirtualQueue<>(list2),
new VirtualQueue<>(list3));
int n=10;
var value = IntStream.range(0,n)
.mapToObj(i -> listList.stream()
.filter(VirtualQueue::hasMore)
.min(Comparator.comparing(l -> l.peek()))
.get().pop())
.skip(n-1).findFirst().get();
//value is now the nth item in a hypothetical merged list.
Assuming that you have k sorted Lists, and you need to obtain the n from the aggregated List (but the merged list itself don't needed), then this problem can be solved in O(n * log k) time, and by using O(k) additional space.
Note:
If the code below looks too much involved, here's the rationale behind that. This solution is more performant than straightforward comparison of elements from each list which can be observed in this and this answers, which time complexity O(n * k) (opposed to O(n * log k)). A moderate additional complexity is a cost of the performance gain, and note that it's still maintainable.
In case you would need to materialize the merged sorted list (solution below is not doing this), you can simply combine the lists together and make use of the built-in Timsort algorithm implementation via List.sort(). Timsort is very good at spotting the sorted runs, therefore sorting a list that consists of sorted chunks would have a linear time complexity.
To address the problem in O(n * log k) time, we can maintain a PriorityQueue which would always have a size of k or less (therefore enqueue/dequeue operation would have a cost of O(log k)). At the beginning, the Queue should be initialized by adding the very first element from each List.
Then we need to perform n iteration (to find the target element). At each iteration step the Head element of the Queue should be removed, and the next element originated from the same list should be added to the Queue (i.e. if let's say the 7th element from the 3rd List appear to be the Head of the Queue, then after removing it, we need to enqueue the 8th element from the 3rd List).
In order to be able to track from which List each element is coming from and what was its index in the List, we can define a custom type:
public class ElementWrapper<V extends Comparable<V>> implements Comparable<ElementWrapper<V>> {
private V value;
private int listIndex;
private int elementIndex;
// all-args constructor, getters
#Override
public int compareTo(ElementWrapper<V> o) {
return value.compareTo(o.getValue());
}
}
And here's how this algorithm for finding the n-th element can be implemented. As I've said, the time complexity is O(n * log k), since we need n iteration step with each step having a cost of O(log k). Additional memory required only to maintain a Queue of k element.
public static <T extends Comparable<T>> T getNElement(List<List<T>> lists, int n) {
Queue<ElementWrapper<T>> queue = initializeWithFirstElements(lists);
T result = null;
int count = 1;
while (!queue.isEmpty()) {
ElementWrapper<T> current = queue.remove();
if (count == n) { // target index was reached
result = current.getValue();
break;
}
count++;
if (hasNext(current, lists)) {
addNext(current, lists, queue);
}
}
return result;
}
public static <T extends Comparable<T>> Queue<ElementWrapper<T>>
initializeWithFirstElements(List<List<T>> lists) {
Queue<ElementWrapper<T>> queue = new PriorityQueue<>();
for (int i = 0; i < lists.size(); i++) {
if (lists.get(i).isEmpty()) continue;
queue.add(new ElementWrapper<>(lists.get(i).get(0), i, 0));
}
return queue;
}
public static <T extends Comparable<T>> boolean
hasNext(ElementWrapper<T> current, List<List<T>> lists) {
return current.getElementIndex() + 1 < lists.get(current.getListIndex()).size();
}
public static <T extends Comparable<T>> void
addNext(ElementWrapper<T> current, List<List<T>> lists,
Queue<ElementWrapper<T>> queue) {
ElementWrapper<T> next = new ElementWrapper<>(
lists.get(current.getListIndex()).get(current.getElementIndex() + 1),
current.getListIndex(),
current.getElementIndex() + 1
);
queue.add(next);
}
Usage example:
public static void main(String[] args) {
List<List<Integer>> input =
List.of(List.of(1, 3), List.of(),
List.of(2, 6, 7), List.of(10), List.of(4, 5, 8, 9)
);
System.out.println(getNElement(input, 1));
System.out.println(getNElement(input, 3));
System.out.println(getNElement(input, 9));
}
Output:
1 // 1st
3 // 3rd
9 // 9th
Note: depending how do you want the n-th element to be indexed the count variable in the getNElement() method should be initialized accordingly, i.e. with 1 if you want to use 1-based indexes, and with 0 if you want n to be 0-based.
I have a program where I am creating an arrayList. One method randomly removes an item from the list. I also have a method that resizes the array if a certain amount of elements are added to make space for more elements to be added if wanted.
I am receiving a NullPointerExceptionError when the lists are resized and then the method tries to remove one of the null spaces in the newly resized array.
Without using a for loop and keeping a constant time, I'm trying to find a way to make my int value of choice just the number of non null elements in the array. I hope that makes sense - can anyone help?
remove method:
public T remove() {
if (isEmpty()) {
return null;
}
Random rand = new Random();
int choice = rand.nextInt(size);
size--;
elements[choice] = elements[size];
elements[size] = null;
return elements[choice];
}
This question boils down to "Given an array with null elements scattered throughout, how can I return the nth non-null element?" The answer is iterate through the array, which doesn't uphold the constant time constraint your're looking for.
But wait, how can array lists work in that case?!?
Here is how java.util.ArrayList does it:
#Override
public E get(int location) {
if (0 <= location && location < (lastIndex - firstIndex)) {
return (E) array[firstIndex + location];
}
throw new IndexOutOfBoundsException(Messages.getString("luni.0A",
location, lastIndex - firstIndex));
}
Note that there are no null checks, the Arraylist always maintains its elements in a contiguous block so gets are super fast. So some magic must happen when removing elements so there won't be nulls scattered about...
#Override
public E remove(int location) {
E result;
int localLastIndex = lastIndex;
int size = localLastIndex - firstIndex;
if (0 <= location && location < size) {
if (location == size - 1) {
result = (E) array[--localLastIndex];
array[localLastIndex] = null;
} else if (location == 0) {
result = (E) array[firstIndex];
array[firstIndex++] = null;
} else {
int elementIndex = firstIndex + location;
result = (E) array[elementIndex];
if (location < size / 2) {
System.arraycopy(array, firstIndex, array, firstIndex + 1,
location);
array[firstIndex++] = null;
} else {
System.arraycopy(array, elementIndex + 1, array,
elementIndex, size - location - 1);
array[--localLastIndex] = null;
}
}
if (firstIndex == localLastIndex) {
firstIndex = localLastIndex = 0;
}
} else {
throw new IndexOutOfBoundsException(Messages.getString("luni.0A",
location, localLastIndex - firstIndex));
}
lastIndex = localLastIndex;
modCount++;
return result;
}
There is the key! When removing an element the JDK impl doesn't leave a gap, nor does it move all the elements over to close the gap. It calls a native method that copies a chunk of array around as an atomic operation: https://docs.oracle.com/javase/7/docs/api/java/lang/System.html#arraycopy(java.lang.Object,%20int,%20java.lang.Object,%20int,%20int)
Is there an Iterator to loop over data structure in cycles?
Let's say there is an array:
int[] arr = {-1,5,7,-1,-1,-1}
I want to find index of first non -1 value from this array and starting to search from the random position (idx = random.nextInt(arr.length)). For example idx = 4;
So first check if arr[4] == -1, then if arr[5] == -1 and so on. If the end of the array reached then start from 0 position and continue until non -1 found. It is guaranteed that there will be at least one value not equal to -1 in the array.
This can be done so:
int idx = -1;
for (int i = random.nextInt(arr.length); ; i++) {
if (i == arr.length) {
/** start over */
i = 0;
}
if (-1 != arr[i]) {
idx = i;
break;
}
}
Or so:
int idx = -1;
int i = random.nextInt(arr.length);
do {
if (-1 != arr[i]) {
idx = i;
}
i == arr.length ? i=0 : i++;
} while (-1 == idx);
Is there an Iterator, that supports cycling (call next() , if the end of array reached then automatically start from 0)?
Limitations: 1) efficiency is not considered; 2) standard Java API is preferred.
in java API there is no such api which satisfy your problem but you can made it by your own.
what you can do is use List to create LinkedList. to solve your problem.
you can extend List to your class (CircularLinkedList extends List) & then override method hasNext() & getNext() thats all you need.
I don't think there are any iterators that let you know the index of the element as you call next(), so you'd have to keep track of the current index separately. You might be able to build up a "wrap-around" iterator using Guava's Iterators.concat (or some other third-party class) to concatenate an iterator over the trailing part of the array with an iterator over the leading part. However, I think the code is likely to be more complex than a simple for loop or two.
I believe there is no such circular Iterator that will automatically go to the beginning of the array once the end has been reached. I have created one below (not tested, and design is flawed), which requires an entirely new class of code, and is much longer than your short for/while loops.
public class MyCircularIterator<E> implements Iterator<E> {
private List<E> list;
private int pos;
public MyCircularIterator(List<E> list) {
this(list, 0);
}
public MyCircularIterator(List<E> list, int start) {
this.list = list;
pos = start;
}
public boolean hasNext() {
if(list.get(pos) != -1) return false;
return true;
}
public E next() {
if(hasNext()) {
E obj = list.get(pos);
pos = (pos + 1) % list.size();
return obj;
}
}
public void remove() {
list.remove(this.nextIndex);
}
}
// Queue.java
// demonstrates queue
// to run this program: C>java QueueApp
class Queue
{
private int maxSize;
private long[] queArray;
private int front;
private int rear;
private int nItems;
public Queue(int s) // constructor
{
maxSize = s;
queArray = new long[maxSize];
front = 0;
rear = -1;
nItems = 0;
}
public void insert(long j)
{
if(rear == maxSize-1)
rear = -1;
queArray[++rear] = j;
nItems++;
}
public long remove()
{
long temp = queArray[front++];
if(front == maxSize)
front = 0;
nItems--;
return temp;
}
public long peekFront()
{
return queArray[front];
}
public boolean isEmpty() // true if queue is empty
{
return (nItems==0);
}
public boolean isFull() // true if queue is full
{
return (nItems==maxSize);
}
public int size() // number of items in queue
{
return nItems;
}
public void display()
{ int startFront = front;
for (int j = front ;j <nItems; j++ )
{
System.out.println(queArray[j]);
if (j == nItems-1 )
{ j=0;
System.out.println(queArray[j]);
}
if (j==startFront-1)
return;
}
}
}
class QueueApp
{
public static void main(String[] args)
{
Queue theQueue = new Queue(5); // queue holds 5 items
theQueue.insert(10); // insert 4 items
theQueue.insert(20);
theQueue.insert(30);
theQueue.insert(40);
theQueue.remove(); // remove 3 items
theQueue.remove(); // (10, 20, 30)
theQueue.remove();
theQueue.insert(50); // insert 4 more items
theQueue.insert(60); // (wraps around)
theQueue.insert(70);
theQueue.insert(80);
theQueue.display();
while( !theQueue.isEmpty() ) // remove and display
{ // all items
long n = theQueue.remove(); // (40, 50, 60, 70, 80)
System.out.print(n);
System.out.print(" ");
}
System.out.println("");
} // end main()
} // end class QueueApp
Okay, this is the basic, out of the book, queue code. I am attempting to create a display method that will show the queue in order, from front to back. (This is an assignment, i know this is not practical....) If i run the program as is, it will display the queue in order from front to rear(at least that is what i believe i did). The problem i am having is if i change the nItems, it ceases to work. For example if you add the line of code, theQueue.remove(); right above the call to the display, the method ceases to work, i know it is because the front is now = to 4, instead of 3,and it will not enter the for method which needs the front to be < nItems, 4<4 is not true so the for loop does not initiate.
Simply use something like:
public void display() {
for (int i = 0; i < nItems; i++) {
System.out.println(queArray[(front + i) % maxSize]);
}
}
In my opinion you're using too many variables which you don't need. You only need the Queue size and its item count.
public Queue(int s) {
size = s;
queArray = new long[s];
nItems = 0;
}
public void insert(long j) {
if(nItems < size) {
queArray[nItems] = j;
nItems++;
}
}
public long remove() {
if(nItems > 0) {
long temp = queArray[nItems];
nItems--;
return temp;
}
}
public void display() {
for(int j = 0; j < nItems; j++) {
System.out.println(queArray[j]);
}
}
So what's happening right now is that j is the position of the element in your array, which is different from the number of elements that you've printed so far.
You need to either use a different index to count how many elements you printed or check whether you're at the end by comparing j to rear.
When the queue is full (rear == maxSize - 1) and you do a insert, it will replace the first
item, so i think the line nItems++ should not be incremented when the queue is already full.
Edit: Avoid modulus operations when you don't need them, they consume a lot of cpu.
The backing store for your queue is :
private long[] queArray;
Why don't you instead use :
private List<Long> queArray
and let List worry about the resizing effort after add/remove operations. Your current queue implementation needs to know exactly how many elements are going into the queue on construction. That's pretty inconvenient for clients using this API.
You can instantiate the queArray as :
queArray = new ArrayList<Long>();
in your constructor. Once you really understand that code, you can then move onto worrying about the re-sizing logic yourself.
Following up my previous bit-twiddling question, now I'm looking to trim down the method that uses that one (though, an unbuffered version, since the life of the array is only this object). This is the iterator method for a power set of some long base; the actual contents of the set isn't stored - it would be a memory hog and individual members are only interesting when one wants to iterate over the set - so the members are generated by the iterator.
However, those members need to be returned in order by size first (instead of lexicographical order) - i.e., "sorted" by bit count.
Below is my (working) first cut; any suggestions from optimization addicts? This one I'm pretty sure has more obvious cuts.
public Iterator<Long> iterator() { return new Iterator<Long>(){
private boolean hasNext = true;
#Override public boolean hasNext() { return hasNext; }
private final long[] split = BitTwiddling.decompose(base);
int size = 0;
private long next = 0;
private long lastOfSize = 0;
private long firstOfSize = 0;
int[] positions = new int[split.length];
#Override
public Long next() {
long result = next;
if (next == lastOfSize) {
if (size == split.length) {
hasNext = false;
return result;
}
next = (firstOfSize |= split[size]);
lastOfSize |= split[split.length - ++size];
for(int i=0; i<size; i++) positions[i] = i;
} else {
if (positions[size-1] == split.length-1) {
int index = size-1;
int ref = split.length - 1;
while (positions[index] == ref) { index--; next ^= split[ref--]; }
next ^= split[positions[index]++];
next |= split[positions[index++]];
do {
next |= split[positions[index] = positions[index-1]+1];
} while (++index < size);
} else {
next ^= split[positions[size-1]++];
next |= split[positions[size-1]];
}
}
return result;
}
If I understand the question correctly, you want to Compute the lexicographically next bit permutation