here is a part of my code where I am getting error:
long p=1000000000-size-1;
long j;
for(j=p;j>p-k;j--)
{
sum2=sum2+sum[j];
}
System.out.print(sum2);
I think it is because I am using a long variable to define the size of array.
How could I deal with this error
Or if I am wrong please tell me how could I declare an array containing 10^10 elements.
Java does not provide a way to create arrays with size bigger than Integer.MAX_VALUE (which is about 2*10^9). Also are you sure you have enough memory to store such array? If you want to store array of 10^10 int values, you will need at least 40 Gb of RAM.
Change both the variables to int. If the values don't fit into an int the code won't work anyway.
I think it is because I am using a long variable to define the size of array. How could I deal with this error
No it isn't, because you aren't. You can't.
Aside from the the maximum array concerns and availability of enough virtual memory:
You have an integer literal 1000000000, but what you actually want is a long literal 1000000000L.
Related
I'm trying to create a byte array whose size is of type long. For example, think of it as:
long x = _________;
byte[] b = new byte[x];
Apparently you can only specify an int for the size of a byte array.
Before anyone asks why I would need a byte array so large, I'll say I need to encapsulate data of message formats that I am not writing, and one of these message types has a length of an unsigned int (long in Java).
Is there a way to create this byte array?
I am thinking if there's no way around it, I can create a byte array output stream and keep feeding it bytes, but I don't know if there's any restriction on a size of a byte array...
(It is probably a bit late for the OP, but it might still be useful for others)
Unfortunately Java does not support arrays with more than 231−1 elements. The maximum consumption is 2 GiB of space for a byte[] array, or 16 GiB of space for a long[] array.
While it is probably not applicable in this case, if the array is going to be sparse, you might be able to get away with using an associative data structure like a Map to match each used offset to the appropriate value. In addition, Trove provides an more memory-efficient implementation for storing primitive values than standard Java collections.
If the array is not sparse and you really, really do need the whole blob in memory, you will probably have to use a two-dimensional structure, e.g. with a Map matching offsets modulo 1024 to the proper 1024-byte array. This approach might be be more memory efficient even for sparse arrays, since adjacent filled cells can share the same Map entry.
A byte[] with size of the maximum 32-bit signed integer would require 2GB of contiguous address space. You shouldn't try to create such an array. Otherwise, if the size is not really that large (and it's just a larger type), you could safely cast it to an int and use it to create the array.
You should probably be using a stream to read your data in and another to write it out. If you are gong to need access to data later on in the file, save it. If you need access to something you haven't ran into yet, you need a two-pass system where you run through once and store the "stuff you'll need for the second pass, then run through again".
Compilers work this way.
The only case for loading in the entire array at once is if you have to repeatedly randomly access many locations throughout the array. If this is the case, I suggest you load it into multiple byte arrays all stored in a single container class.
The container class would have an array of byte arrays, but from outside all the accesses would seem contiguous. You would just ask for byte 49874329128714391837 and your class would divide your Long by the size of each byte array to calculate which array to access, then use the remainder to determine the byte.
It could also have methods to store and retrieve "Chunks" that could span byte-array boundaries that would require creating a temporary copy--but the cost of creating a few temporary arrays would be more than made up for by the fact that you don't have a locked 2gb space allocated which I think could just destroy your performance.
Edit: ps. If you really need the random access and can't use streams then implementing a containing class is a Very Good Idea. It will let you change the implementation on the fly from a single byte array to a group of byte arrays to a file-based system without any change to the rest of your code.
It's not of immediate help but creating arrays with larger sizes (via longs) is a proposed language change for Java 7. Check out the Project Coin proposals for more info
One way to "store" the array is to write it to a file and then access it (if you need to access it like an array) using a RandomAccessFile. The api for that file uses long as an index into file instead of int. It will be slower, but much less hard on the memory.
This is when you can't extract what you need during the initial input scan.
Suppose there is an array declaration:
int [] a = new int[3];
I'm reading a book that says the reason we need to explicitly create arrays at runtime is because we cannot know how much space to reserve for the array at compile time. So, in this case above don't we know we need to reserve 3 times the size of int of the underlying java platform ?
So, in this case above don't we know we need to reserve 3 times the size of int of the underlying java platform ?
Well actually slightly more than that - because arrays are objects with a class reference, as well as the length.
But yes, in this case you happen to know the size in advance. But Java also lets you write:
int[] a = new int[getLengthFromMethod()];
... in which case no, the amount of memory won't be known at compile time. This flexibility makes it considerably simpler to work with arrays than if you had to know the size at compile time. (Imagine trying to make ArrayList work otherwise, for example.)
But bear in mind that memory allocation is very dynamic in Java in general - it's effectively only really the size of stack frames which is known in advance, and then only in relative terms (as the size of a reference can vary between platforms). All1 objects are allocated on the heap and references are used to keep track of them.
1 Okay, aside from smart JVMs which are sometimes able to allocate inline after performing escape analysis.
In that book it probably says that there are scenarios in which you don't know at compile time how large an array will be (e.g.: when you ask a user for a set of numbers you don't know how many numbers he will insert).
In your example (int [] a = new int[3]) you obviously know the size of the array at compile time; it's 3.
Yes. Your correct.
3 * sizeOf(int) + array overhead.
You are reserving that much memory and then handing a pointer to the first position to your variable a. Then a can figure out how to index the array based on the location and size of what is being stored.
In your case the size of the array is 3.The number inside the square brackets is the size.CHeck this weeks tutorial about arrays on www.codejava.co.uk
Are you sure that book you are reading is about Java?
Unlike C/C++, Java does not have that dilemma - arrays are always allocated at runtime, even if you know their size during compile time.
I am just beginning to learn Java and heard that getting the size of an array is O(1). Why is it not O(n)?
Mustn't the java virtual machine count the number of items in the array?
No, the number of items in the array is fixed when you create it.
Mustn't the java virtual machine count the number of items in the array?
That is not necessary. The JVM representation for an array has a special field that gives the length of the array instance, and a special bytecode for accessing it. When it needs the length, the JVM uses these.
(It is also true that an array's length is fixed when the array has been created ... but that doesn't directly answer the question. Hypothetically, the JVM could still count elements. However, I find it hard to see how the JVM would decide where / when to stop counting elements. How would it know when the bits it is looking at don't represent a valid value of the array base-type?)
As mentioned above, it's a special bytecode function:
http://en.wikipedia.org/wiki/Java_bytecode_instruction_listings
arraylength be arrayref → length get the length of an array
I'm trying to create a byte array whose size is of type long. For example, think of it as:
long x = _________;
byte[] b = new byte[x];
Apparently you can only specify an int for the size of a byte array.
Before anyone asks why I would need a byte array so large, I'll say I need to encapsulate data of message formats that I am not writing, and one of these message types has a length of an unsigned int (long in Java).
Is there a way to create this byte array?
I am thinking if there's no way around it, I can create a byte array output stream and keep feeding it bytes, but I don't know if there's any restriction on a size of a byte array...
(It is probably a bit late for the OP, but it might still be useful for others)
Unfortunately Java does not support arrays with more than 231−1 elements. The maximum consumption is 2 GiB of space for a byte[] array, or 16 GiB of space for a long[] array.
While it is probably not applicable in this case, if the array is going to be sparse, you might be able to get away with using an associative data structure like a Map to match each used offset to the appropriate value. In addition, Trove provides an more memory-efficient implementation for storing primitive values than standard Java collections.
If the array is not sparse and you really, really do need the whole blob in memory, you will probably have to use a two-dimensional structure, e.g. with a Map matching offsets modulo 1024 to the proper 1024-byte array. This approach might be be more memory efficient even for sparse arrays, since adjacent filled cells can share the same Map entry.
A byte[] with size of the maximum 32-bit signed integer would require 2GB of contiguous address space. You shouldn't try to create such an array. Otherwise, if the size is not really that large (and it's just a larger type), you could safely cast it to an int and use it to create the array.
You should probably be using a stream to read your data in and another to write it out. If you are gong to need access to data later on in the file, save it. If you need access to something you haven't ran into yet, you need a two-pass system where you run through once and store the "stuff you'll need for the second pass, then run through again".
Compilers work this way.
The only case for loading in the entire array at once is if you have to repeatedly randomly access many locations throughout the array. If this is the case, I suggest you load it into multiple byte arrays all stored in a single container class.
The container class would have an array of byte arrays, but from outside all the accesses would seem contiguous. You would just ask for byte 49874329128714391837 and your class would divide your Long by the size of each byte array to calculate which array to access, then use the remainder to determine the byte.
It could also have methods to store and retrieve "Chunks" that could span byte-array boundaries that would require creating a temporary copy--but the cost of creating a few temporary arrays would be more than made up for by the fact that you don't have a locked 2gb space allocated which I think could just destroy your performance.
Edit: ps. If you really need the random access and can't use streams then implementing a containing class is a Very Good Idea. It will let you change the implementation on the fly from a single byte array to a group of byte arrays to a file-based system without any change to the rest of your code.
It's not of immediate help but creating arrays with larger sizes (via longs) is a proposed language change for Java 7. Check out the Project Coin proposals for more info
One way to "store" the array is to write it to a file and then access it (if you need to access it like an array) using a RandomAccessFile. The api for that file uses long as an index into file instead of int. It will be slower, but much less hard on the memory.
This is when you can't extract what you need during the initial input scan.
When I try to make a very large boolean array using Java, such as:
boolean[] isPrime1 = new boolean[600851475144];
I get a possible loss of precision error?
Is it too big?
To store 600 billion bits, you need an absolute minimum address space of 75 gigabytes! Good luck with that!
Even worse, the Java spec doesn't specify that a boolean array will use a single bit of memory for each element - it could (and in some cases does) use more.
In any case, I recognise that number from Project Euler #3. If it needs that much memory, you're doing it wrong...
Consider using a BitSet.
Since you're attempting to solve Euler-problem #3 the wrong way, here's a hint: You're supposed to find all the prime factors of a number, not all the prime numbers below a certain limit.
BTW: This particular Euler-problem can be solved using a very small amount of RAM.
An array index is an int, not a long, so your "array" is too big to fit into an array. One of the java Collection classses might be more suited. Never mind - Collection.size() returns an int as well, so Collection can't store more than Integer.MAX_VALUE items either.
Um... that would be about 70GB worth of booleans. Not gonna work. No way.
The problem is you are using a long value vs. an int value for the size of the array. Java does not support array lengths longer that the maximum value of an int. Java is treating your length as a long because the size you specified exceeds the maximum value for an int but fits within a long. Hence it must convert the length back to an int to create an array. The conversion from long -> int is producing the warning you are seeing
You can use an array of longs, encapsulated in a class that would handle all the operations on the array. Something like your own implementation of BitSet.
Why not just store the values in a file, and then seek to the position in the file and pull up the right value. Like others have stated, that's 70GB of data. In most cases, you wouldn't even be able to hold that in memory. If you're going to store it to a file, you could even look at individual bits when storing and retrieving the data using bitwise operators to save on storage space.
Also, since the number of primes decreases with the size of the numbers, it's probably better just to store the prime numbers themselves in the file, in order, and then do a binary search for the number to see if it is one of the primes.
What values do you have in the array?
For a such large number I guess it's going to be a sparse array so maybe it would be best to use a Map/List and just allocate space and store a value for a 1 value for a bit. Or for a 0 value if most of your values will be 1.
Apache ActiveMQ has a datastructure called BitArrayBin. This is used to find out whether a message is duplicated. A message ID is a combination of producer ID and sequence ID.
Each producer will have a BitArrayBin to track its sequence IDs. Once it finds out the BitArrayBin for the given producer, it sets the sequence ID which is a long value to the BitArrayBin.
oldValue = bitArrayBin.setBit(sequenceId, true)
if (oldVlaue) {
"message is duplicated"
}
The method returns the old Value.
If y is the long index, it is used to derive at a bin index and an offset into it.
y = bin index * 64 + offset
BitArrayBin is nothing but a holder for many bins where the size can be defined during its construction. Each bin contains a long variable to store the bits so so it can store up to 64 boolean values.
Bit masking is used to set the bit, and then, get it's value.
This class doesn't have much documentation. You need to go through its source code to know the internals.