public static ArrayList<Integer> duplicates(int[] arr) {
ArrayList<Integer> doubles = new ArrayList<Integer>();
boolean isEmpty = true;
for(int i = 0; i<arr.length; i++) {
for (int j = i+1; j< arr.length; j++) {
if( arr[i] == arr[j] && !doubles.contains(arr[i]) ){
doubles.add(arr[i]);
isEmpty = false;
break;
}
}
}
if(isEmpty) doubles.add(-1);
Collections.sort(doubles);
return doubles;
}
public static void main(String[] args) {
System.out.println( ( duplicates( new int[]{1,2,3,4,4,4} ) ) ); // Return: [4]
}
I made this function in Java which returns multiples of an input int array or returns a -1 if the input array is empty or when there are no multiples.
It works, but there is probably a way to make it faster.
Are there any good practices to make functions more efficient and faster in general?
There are, in broad strokes, 2 completely unrelated performance improvements you can make:
Reduce algorithmic complexity. This is a highly mathematical concept.
Reduce actual performance characteristics - literally, just make it run faster and/or use less memory (often, 'use less memory' and 'goes faster' go hand in hand).
The first is easy enough, but can be misleading: You can write an algorithm that does the same job in an algorithmically less complex way which nevertheless actually runs slower.
The second is also tricky: Your eyeballs and brain cannot do the job. The engineers that write the JVM itself are on record as stating that in general they have no idea how fast any given code actually runs. That's because the JVM is way too complicated: It has so many complicated avenues for optimizing how fast stuff runs (not just complicated in the code that powers such things, also complicated in how they work. For example, hotspot kicks in eventually, and uses the characteristics of previous runs to determine how best to rewrite a given method into finely tuned machine code, and the hardware you run it on also matters rather a lot).
This leads to the following easy conclusions:
Don't do anything unless there is an actual performance issue.
You really want a profiler report that actually indicates which code is 'relevant'. Generally, for any given java app, literally 1% of all of your lines of code is responsible for 99% of the load. There is just no point at all optimizing anything, except that 1%. A profiler report is useful in finding the 1% that requires the attention. Java ships with a profiler and there are commercial offerings as well.
If you want to micro-benchmark (time a specific slice of code against specific inputs), that's really difficult too, with many pitfalls. There's really only one way to do it right: Use the Java Microbenchmark Harness.
Whilst you can decide to focus on algorithmic complexity, you may still want a profiler report or JMH run because algorithmic complexity is all about 'Eventually, i.e. with large enough inputs, the algorithmic complexity overcomes any other performance aspect'. The trick is: Are your inputs large enough to hit that 'eventually' space?
For this specific algorithm, given that I have no idea what reasonable inputs might be, you're going to have to do the work on setting up JMH and or profiler runs. However, as far as algorithmic complexity goes:
That doubles.contains call has O(N) algorithmic complexity: The amount of time that call takes is linear relative to how large your inputs are.
You can get O(1) algorithmic complexity if you use a HashSet instead.
From the point of view of just plain performance, generally an ArrayList's performance and memory load vs. an int[] is quite large.
This gives 2 alternate obvious strategies to optimize this code:
Replace the ArrayList<Integer> with an int[].
Replace the ArrayList<integer> with a HashSet<Integer> instead.
You can't really combine these two, not without spending a heck of a long time handrolling a primitive int array backed hashbucket implementation. Fortunately, someone did the work for you: Eclipse Collections has a primitive int hashset implementation.
Theoretically it's hard to imagine how replacing this with IntHashSet can be slower. However, I can't go on record and promise you that it'll be any faster: I can imagine if your input is an int array with a few million ints in there, IntHashSet is probably going to be many orders of magnitude faster. But you really need test data and a profiler report and/or a JMH run or we're all just guessing, which is a bad idea, given that the JVM is such a complex beast.
So, if you're serious about optimizing this:
Write a bunch of test cases.
Write a wrapper around this code so you can run those tests in a JMH setup.
Replace the code with IntHashSet and compare that vs. the above in your JMH harness.
If that really improves things and the performance now fits your needs, great. You're done.
If not, you may have to re-evaluate where and how you use this code, or if there's anything else you can do to optimize things.
It works, but there is probably a way to make it faster.
I think you will find this approach significantly faster. I omitted the sort from both methods just to check. This does not discuss general optimizations as rzwitserloot's excellent answer already does that.
The two main problems with your method are:
you are using a nested loop which is essentially is an O(N*N) problem.
and you use contains on a list which must do a linear search each time to find the value.
A better way is to use a HashSet which works very close to O(1) lookup time (relatively speaking and depending on the set threshold values).
The idea is as follows.
Create two sets, one for the result and one for what's been seen.
iterate over the array
try to add the value to the seen set, if it returns true, that means a duplicate is not in the seen set so it is ignored.
if it returns false, a duplicate does exist in the seen set so it is added to the duplicate set.
Note the use of the bang ! to invert the above conditions.
once the loop is finished, return the duplicates in a list as required.
public static List<Integer> duplicatesSet(int[] arr) {
Set<Integer> seen = new HashSet<>();
Set<Integer> duplicates = new HashSet<>();
for (int v : arr) {
if (!seen.add(v)) {
duplicates.add(v);
}
}
return duplicates.isEmpty()
? new ArrayList<>(List.of(-1))
: new ArrayList<>(duplicates);
}
The sort is easily added back in. That will take additional computing time but that was not the real problem.
To test this I generated a list of random values and put them in an array. The following generates an array of 1,000,000 ints between 1 and 1000 inclusive.
Random r = new Random();
int[] val = r.ints(1_000_000, 1, 1001).toArray();
Related
I am trying to convert a very long int[] with length of 1,000,000 to Integer[] so that I can sort it with a custom comparator (which sorts elements based on the length of their corresponding lists in a defined Map<Integer, List<Integer>>).
I have done the following:
private static Integer[] convert(int[] arr) {
Integer[] ans = new Integer[arr.length];
for (int i = 0; i < arr.length; i++) {
ans[i] = arr[i];
}
return ans;
}
It works well for me but I have also come across
Integer[] ans = Arrays.stream(intArray).boxed().toArray( Integer[]::new );
and
Integer[] ans = IntStream.of(intArray).boxed().toArray( Integer[]::new );
Is there any of them that is significantly faster than the rest? Or is there any other approach that is fast enough to shorten the run-time?
Is there any of them that is significantly faster than the rest?
You realize the question you're asking is akin to:
"I have 500,000 screws to screw into place. Unfortunately, I can't be bothered to go out and buy a screwdriver. I do have a clawhammer and an old shoe. Should I use the clawhammer to bash these things into place, or is the shoe a better option?"
The answer is clearly: Uh, neither. Go get a screwdriver, please.
To put it differently, if the 'cost' of converting to Integer[] first is 1000 points of cost in some arbitrary unit, then the difference in the options you listed is probably between 0.01 and 0.05 points - i.e. dwarfed so much, it's irrelevant. Thus, the direct answer to your question? It just does not matter.
You have 2 options:
Performance is completely irrelevant. In which case this is fine, and there's absolutely no point to actually answering this question.
You care about performance quite a bit. In which case, this Integer[] plan needs to be off the table.
Assuming you might be intrigued by option 2, you have various options.
The easiest one is to enjoy the extensive java ecosystem. Someone's been here before and made an excellent class just for this purpose. It abstracts the concept of an int array and gives you all sorts of useful methods, including sorting, and the team that made it is extremely concerned about performance, so they put in the many, many, many personweeks it takes to do proper performance analysis (between hotspot, pipelining CPUs, and today's complex OSes, it's much harder than you might think!).
Thus, I present you: IntArrayList. It has a .sortThis() method, as well as a .sortThis(IntComparator c) method, which you can use for sorting purposes.
There are a few others out there, searching the web for 'java primitive collections' should find them all, if for some reason the excellent eclipse collections project isn't to your liking (NB: You don't need eclipse-the-IDE to use it. It's a general purpose library that so happens to be maintained by the eclipse team).
If you must handroll it, searching the web for how to implement quicksort in java is not hard, thus, you can easily write your own 'sort this int array for me' code. Not that I would reinvent that particular wheel. Just pointing out that it's not too difficult if you must.
This is regarding identifying time complexity of a java program. If i've iterations like for or while etc, we can identify the complexity. But if i use java API to do some task, if it is internally iterating, i think we should include that as well. If so, how to do that.
Example :
String someString = null;
for(int i=0;i<someLength;i++){
someString.contains("something");// Here i think internal iteration will happen, likewise how to identify time complexity
}
Thanks,
Aditya
Internal operations in the Java APIs have their own time complexity based on their implementation. For example the contains method of the String variable runs with linear complexity, where the dependency is based on the length of your someString variable.
In short - you should check how inner operations work and take them into consideration when calculating complexity.
Particularly for your code the time complexity is something like O(N*K), where N is the number of iterations of your loop (someLength) and K is the length of your someString variable.
You are correct in that the internal iterations will add to your complexity. However, except in a fairly small number of cases, the complexity of API methods is not well documented. Many collection operations come with an upper bound requirement for all implementations, but even in such cases there is no guarantee that the actual code doesn't have lower complexity than required. For cases like String.contains() an educated guess is almost certain to be correct, but again there is no guarantee.
Your best bet for a consistent metric is to look at the source code for the particular API implementation you are using and attempt to figure out the complexity from that. Another good approach would be to run benchmarks on the methods you care about with a wide range of input sizes and types and simply estimate the complexity from the shape of the resulting graph. The latter approach will probably yield better results for cases where the code is too complex to analyze directly.
I need to efficiently find the ratio of (intersection size / union size) for pairs of Lists of strings. The lists are small (mostly about 3 to 10 items), but I have a huge number of them (~300K) and have to do this on every pair, so I need this actual computation to be as efficient as possible. The strings themselves are short unicode strings -- averaging around 5-10 unicode characters.
The accepted answer here Efficiently compute Intersection of two Sets in Java? looked extremely helpful but (likely because my sets are small (?)) I haven't gotten much improvement by using the approach suggested in the accepted answer.
Here's what I have so far:
protected double uuEdgeWeight(UVertex u1, UVertex u2) {
Set<String> u1Tokens = new HashSet<String>(u1.getTokenlist());
List<String> u2Tokens = u2.getTokenlist();
int intersection = 0;
int union = u1Tokens.size();
for (String s:u2Tokens) {
if (u1Tokens.contains(s)) {
intersection++;
} else {
union++;
}
}
return ((double) intersection / union);
My question is, is there anything I can do to improve this, given that I'm working with Strings which may be more time consuming to check equality than other data types.
I think because I'm comparing multiple u2's against the same u1, I could get some improvement by doing the cloning of u2 into a HashSet outside of the loop (which isn't shown -- meaning I'd pass in the HashSet instead of the object from which I could pull the list and then clone into a set)
Anything else I can do to squeak out even a small improvement here?
Thanks in advance!
Update
I've updated the numeric specifics of my problem above. Also, due to the nature of the data, most (90%?) of the intersections are going to be empty. My initial attempt at this used the clone the set and then retainAll the items in the other set approach to find the intersection, and then shortcuts out before doing the clone and addAll to find the union. That was about as efficient as the code posted above, presumably because of the trade of between it being a slower algorithm overall versus being able to shortcut out a lot of the time. So, I'm thinking about ways to take advantage of the infrequency of overlapping sets, and would appreciate any suggestions in that regard.
Thanks in advance!
You would get a large improvement by moving the HashSet outside of the loop.
If the HashSet really has only got a few entries in it then you are probably actually just as fast to use an Array - since traversing an array is much simpler/faster. I'm not sure where the threshold would lie but I'd measure both - and be sure that you do the measurements correctly. (i.e. warm up loops before timed loops, etc).
One thing to try might be using a sorted array for the things to compare against. Scan until you go past current and you can immediately abort the search. That will improve processor branch prediction and reduce the number of comparisons a bit.
If you want to optimize for this function (not sure if it actually works in your context) you could assign each unique String an Int value, when the String is added to the UVertex set that Int as a bit in a BitSet.
This function should then become a set.or(otherset) and a set.and(otherset). Depending on the number of unique Strings that could be efficient.
I am having trouble in calculating the time complexity of program shown below. It is a simple program to generate valid parentheses such as "((()))" "(()())" etc. However, I don't really know how to estimate time complexity for this kind of problems.
It will be appreciated if you can share some techniques you find useful here. It will be the best if you can analyze the program I linked as an example : )
My aim :
Estimate time complexity for nontrivial program. Typically a recursive program which has some pruning.
I am looking for a fast estimate solution, not a rigorous mathematical proving.
Thank you in advance.
The code in question:
public ArrayList<String> generateParenthesis(int n) {
ArrayList<String> res = new ArrayList<String>();
String oneSolu = "";
Generate(n, n, res, oneSolu);
return res;
}
private void Generate(int l, int r, ArrayList<String> res, String oneSolu) {
if (l==0 && r==0) {
res.add(oneSolu);
return ;
}
//add left
if (l > 0) {
String t = oneSolu;
t += "(";
Generate(l-1, r, res, t);
}
if (r>l) {
String t = oneSolu;
t += ")";
Generate(l, r-1, res, t);
}
}
I have to admit, your particular use case seems particularly tough, so don't be too hard on yourself.
Estimate time complexity for nontrivial program. Typically a recursive
program which has some pruning.
I am looking for a fast estimate solution, not a rigorous mathematical
proving.
I can give you my normal thought process when I'm analyzing runtimes. It won't be terribly helpful for this particular case, but can certainly be helpful in the general case (if you run into issues analyzing other programs later on).
I can't give any guarantees about not using rigorous math though; I tend to default to it if I want to be really sure of a bound. For loose bounds, the stuff is generally simple enough to where it's not a big issue though.
There's two main things that I generally try to think about first.
1) Can I at least write down the recurrence?
Some recurrences are familiar to a large number of people (like T(n) = T(n-1) + T(n-2)), while some have been studied pretty extensively (like anything solvable with the master method). If a program falls under this category, consider yourself pretty lucky.
In your particular case, the recurrence seems to be something like
T(L,R) = T(L-1,R) + T(L, R-1) if R > L
T(L,R) = T(L-1,R) otherwise, with base case
T(0,R) = R
Not the greatest start.
2) Analyzing how many times a particular function is called with specific arguments
This one is generally more useful in dynamic programming, where past results are stored to save computation, but is also another tool in the belt. That being said, this isn't an option if you can't compute how many times the function is called with specific arguments.
In this case though, this approach gets heavy on the math. The basic problem is that the number of times Generate() is called with a specific l and r depends entirely on the possible values of oneSolu. (The ArrayList is an accumulator, so that's not a worry)
In our case, we happen to know how long the string is (since the first call had l = r = n and each recursive call decreased exactly one of the two by 1), and we can also show that
For every value of oneSolu passed in, we can guarantee that every prefix has more (s than )s.
Every such string of this specific length is covered.
I'm pretty sure that value can be found, but 1) the math will get ugly very quickly and 2) even if you got that far, you then have to wrap it around a double summation and evaluate that too. Not practical, and even getting this far dealt with way more math than you wanted.
Now for the really coarse way to get an upper bound. This is the "quick" way, but it doesn't take into account any sort of pruning, so it can be pretty useless if you want a tight bound. It's already been posted, but I'll add it on anyway so that this answer sums up everything by itself.
3) Multiply the maximum depth by the max branching factor.
As already pointed out by #VikramBhat, you've got a branching factor of 2, and a maximum depth of 2n, so you're looking at a (very very) loose bound of 22n = 4n total nodes, and as pointed out by #KarolyHorvath in the comments, the work per node is going to be linear, so that gives us a O(n4n) running time.
The number of valid parenthesis generated with n-pairs is nth catalan number which is defined as 2nCn/(n+1) but if u need more simplified bound then it is O(4^N) . More generally any recursive function is upper bounded by its max branching factor and depth as O(b^d) if work done at each level is O(1) so in this case depth = 2N and branching factor is approximately 2 hence T(n) = 2^(2N)=4^N.
Using Java (if it matters)
I am running MergeSort and QuickSort one after the other and comparing the run times of both, on my computer when sorting 10,000,000 values I am finding that The run times when MergeSort is run then QuickSort
MergeSort = 1.6s (approx)
QuickSort = 0.3s (approx)
When running Quicksort first then MergeSort for the same input size of 10,000,000 I get
MergeSort = 0.6s
QuickSort = 1.2s
I'm assuming this might have something to do with Memory Allocation but I'm not sure how you would explain it
-EDIT- Before running both routines I am creating two seperate arrays a[ ] and b[ ] of a file randomintegers.dat which contains 10,000,000 random values. MergeSort sorts array a[ ], QuickSort sorts array b[ ]. (i.e. both arrays are seperate
Another alternative is that you are using the output of one as the input of the next. QuickSort is more sensitive to input "sortedness" (in a positive way) than MergeSort.
Most likely your benchmarking code is not up to standard because benchmarking on the JVM is notoriously tricky due to the distance between Java code you write and the code actually executed by the JVM. If you want really solid results, you should use Google Caliper or Oracle jmh to leverage many years of professional benchmarking. Lacking that, follow at least these guidelines:
create two arrays: one master array, which holds the random data, and the other which will be passed to the sorting methods. Copy the master into the working array each time;
run each kind of sort repeatedly and display the timing for each pass;
observe the times reported. You should witness a drop in the sort time between the first and second run, possibly even further runs. You must run as many times until you see the times have stabilized;
MergeSort, unlike QuickSort, uses dynamic memory allocation. Therefore it will cause the Garbage Collector to steal a portion of the overall time. Control for this with explicit garbage collection, which is once again difficult to achieve. My recommendation for a simple approach that I found workable:
for (int i = 0; i < 3; i++) { System.gc(); Thread.sleep(500); }
Further than this there are GC tuning paramters, heap sizing, etc. in order to minimize GC while MergeSort is running. On the other hand, GC during MergeSort is a fact of life and it will influence your real-life performance.
Finally, general practice would suggest that QuickSort should be the first choice as long as you don't mind the fact that it is unstable. This property makes it useless when sorting objects on several criteria (first by X, then by Y).