Develop Reference

What type of console println do competitive programmers use in Java? [duplicate]

I'm doing something like this:
for (int i = 0; i < 100000; i++) {
System.out.println( i );
}
Basically, I compute an integer and output a string about 10K-100K times and then need to write the result to system.out, each result separated by a newline.
What's the fastest way to achieve this?

Thank you for the suggestions. I created a test program to compare them:
import java.io.BufferedOutputStream;
import java.io.BufferedWriter;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.lang.StringBuilder;
public class systemouttest {
public static void main(String[] args) throws Exception {
long starttime = System.currentTimeMillis();
for (int i = 0; i < 100000; i++) {
System.out.println( i );
}
long printlntime = System.currentTimeMillis();
StringBuilder sb = new StringBuilder();
for (int i = 0; i < 100000; i++) {
sb.append( i + "\n" );
}
System.out.print(sb.toString());
long stringbuildertime = System.currentTimeMillis();
OutputStream out = new BufferedOutputStream ( System.out );
for (int i = 0; i < 100000; i++) {
out.write((i + "\n").getBytes());
}
out.flush();
long bufferedoutputtime = System.currentTimeMillis();
BufferedWriter log = new BufferedWriter(new OutputStreamWriter(System.out));
for (int i = 0; i < 100000; i++) {
log.write(i + "\n");
}
log.flush();
long bufferedwritertime = System.currentTimeMillis();
System.out.println( "System.out.println: " + (printlntime - starttime) );
System.out.println( "StringBuilder: " + (stringbuildertime - printlntime) );
System.out.println( "BufferedoutputStream: " + (bufferedoutputtime - stringbuildertime) );
System.out.println( "BufferedWriter: " + (bufferedwritertime - bufferedoutputtime) );
}
}
Results:
Environment1
System.out.println: 482
StringBuilder: 210
BufferedoutputStream: 86
BufferedWriter: 202
Environment2
System.out.println: 1763
StringBuilder: 45
BufferedoutputStream: 76
BufferedWriter: 34
The suggestions all performed better than System.out.println. BufferedOutputStream seems to be the safest choice as it performed well in both test environments. BufferedWriter maybe faster though.
Please post further suggestions if anyone has some ideas. I'm sure someone can make it go faster :)

For large amount of data,System.out.println might be inefficient as it does not
do very good buffering. In that case, you can use a BufferedOutputStream or a BufferedWriter.

Keep in mind that I/O operations are very slow compared to in-memory processing (e.g. parsing of Integer).
So, I would propose you to create the whole string 'in advance' and then print it out only once (of course if its possible):
StringBuilder sb = new StringBuilder();
for(int i = 0 ; i < 100000; i++) { sb.append(i).append("\n");}
String printMe = sb.toString();
System.out.println(printMe);
There are various techniques like buffering the the level of output stream you're using, but I assume that you prefer to stay with the most basic System.out.println
Hope this helps

This includes fast input and output method as well
import java.io.*;
public class templa{
static class FastReader
{
BufferedReader br;
StringTokenizer st;
public FastReader()
{
br = new BufferedReader(new
InputStreamReader(System.in));
}
String next()
{
while (st == null || !st.hasMoreElements())
{
try
{
st = new StringTokenizer(br.readLine());
}
catch (IOException e)
{
e.printStackTrace();
}
}
return st.nextToken();
}
int nextInt()
{
return Integer.parseInt(next());
}
long nextLong()
{
return Long.parseLong(next());
}
double nextDouble()
{
return Double.parseDouble(next());
}
String nextLine()
{
String str = "";
try
{
str = br.readLine();
}
catch (IOException e)
{
e.printStackTrace();
}
return str;
}
}
public static void main(String...args) throws Exception {
OutputStream outputStream =System.out;
PrintWriter out =new PrintWriter(outputStream);
FastReader in =new FastReader();
int testcase = in.nextInt();
while(testcase-- >0){
//in object works same as Scanner Object but much faster
//out.println() works faster than System.out.println()
//Write your code here
}
out.close();
}
}

The slowest part of writing to System.out is the time taken to display what you are writing. i.e. for every line you write the computer has to turn the information into pixels using a font and scroll a whole line. This is much more work than whatever you are likely to be doing to display the text.
You can speed up writing to the console by
writing less (usually the best idea)
writing to a file instead (This can be 5-10x faster)

How to append java output to text file?

The program I have below works perfectly when I print it out in the Java console but when I try to append the program to be put into a text file, it only prints 1/5 of the student's averages into the appending text file.
Bobby, average = 93
I want it to be printing all 5 student's averages as so
Agnes, average = 76
Bufford, average = 91
Julie, average = 94
Alice, average = 39
Bobby, average = 93
Thanks in advance.
import java.util.*;
import java.io.*;
public class StudentAverage {
public static void main(String[]args) throws IOException {
Scanner scanner = new Scanner(new
File("D:\\School\\StudentGrades.txt"));
while (scanner.hasNextLine()) {
Scanner scanners = new Scanner(scanner.nextLine());
String name = scanners.next();
double total = 0;
int num = 0;
while (scanners.hasNextInt()) {
total += scanners.nextInt();
num++;
}
PrintStream output = new PrintStream (("D:\\School\\WriteStudentAverages.txt"));
output.print(name + ", average = " + (Math.round(total/num)));
output.flush();
}
scanner.close();
}
}

PrintStream output = new PrintStream (("D:\\School\\WriteStudentAverages.txt"));
every time it gets to this line, it deletes the file ,opens a new file and adds only the current line. write this line before the loop, and in the loop just leave your other code as is.

To append your output/text file you have to use a different way of writing it.
I also suggest using try-with-resource blocks to avoid memeory leaks.
try (OutputStream output = new FileOutputStream("myFile.txt", true);
Writer writer = new OutputStreamWriter(output)) {
writer.write(name + ", average = " + (Math.round(total / num)) + '\n');
}
You don't have to flush/close them manually

There are a few more glitches in your code. Since you need to manage both the Scanner as well as the PrintWriter with a try/resource-construct, I'd prefer to keep the input- and output-routines separate and read the relevant contents of your file temporarily into memory.
Here is an idea:
LinkedHashMap<String, Integer> students = new LinkedHashMap<>();
// Input routine
try (Scanner scanner = new Scanner(new File("D:\\School\\StudentGrades.txt"))) {
while (scanner.hasNextLine()) {
try (Scanner scanners = new Scanner(scanner.nextLine())) {
String name = scanners.next();
int total = 0;
int num = 0;
while (scanners.hasNextInt()) {
total += scanners.nextInt();
num++;
}
students.put(name, (int) Math.round((double)total/num));
}
}
}
catch (FileNotFoundException e) {
// do something smart
}
// Output routine
try (PrintStream output = new PrintStream ("D:\\School\\WriteStudentAverages.txt")) {
for (Entry<String, Integer> student : students.entrySet()) {
output.println(student.getKey() + ", average = " + Integer.toString(student.getValue()));
}
}
catch (FileNotFoundException e) {
// do something smart
}
The above also allows you to get rid of that nasty throws IOException in the signature of your main()-method. Instead, there are now two beautiful skeletons of an exception handler (the two catch blocks) where you can put some logic that triggers in case
the input file has not been found / is not readable (first catch)
the output file isn't writable / cannot be created (second catch)

How to read values in a text document and then make then parse them to ints in JAVA

I'm trying to read lines of a text document and take the average of the numbers. My plan was to first read ALL the data in the text file. Then split the string up into a String array, then parse each index into an int array. This is my code up to the point of reading the document.
My Text Doc:
3, 7, 24, 66,
10, 50, 20, 40,
100, 20, 69, 911,
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
public class Testing
{
public static void main(String[] args) throws IOException
{
try
{
String path;
path = "TestNumbers.txt";
File f = new File(path);
FileReader re = new FileReader(f);
BufferedReader in = new BufferedReader(re);
String line = "";
String store = "";
while((line = in.readLine()) != null)
{
store = store + line;
}
System.out.println(store);
}
catch (FileNotFoundException e)
{
e.printStackTrace();
}
}
}
Output: 3, 7, 24, 66, 10, 50, 20, 40,100, 20, 69, 911,
In a perfect world I want the values to be separated by either "," or " ".
I tried modifying the store = store + line + " "; but this failed because it would iterate the space even when readLine() was on a blank line.
I can traverse the arrays to parse the int and take the average, but setting up this string to be split is stumping me. I tried String.replace(), String.trim(), and another one that failed me. This isn't homework, I'm in highschool AP CS and this is my own independent study.
Thanks everyone for the help, you all showed plenty of ways to do it. Ended up going with .replace, and made the " " into "". Then just split via the commas. I do want to try out that regex thing though. Thanks everyone again.

The other two solutions may be already what you need, however here is a more idiomatic approach which handles edge cases properly:
{
String path = "TestNumbers.txt";
try (BufferedReader reader = new BufferedReader(new FileReader(path))) {
StringBuilder builder = new StringBuilder();
String line;
while ((line = reader.readLine()) != null)
builder.append(line);
String[] split = builder.toString().split("\\D+");
int[] numbers = new int[split.length];
for (int i = 0; i < split.length; i++ )
numbers[i] = Integer.parseInt(split[i]);
// 'numbers' now stores every digit segment there is in your string.
} catch(IOException e) {
e.printStackTrace();
}
}
Notable:
Declare BufferedReader as Closeable in a try-with-resources scope.
Do not concatenate Strings in a loop, use StringBuilder instead.
Split to \D+ to remove all non-digits, obtaining digit segments as elements in the final array instead.

Scary Wombat mentioned how to handle blank lines.
You could do a .replace to swap all " " chars for "," chars. If you expect both like ", " in some cases you could even .replace those into just "," first. These are basically tricks.
You could look at String.split(regex) since the regex could be a pattern like "[, ]+" to take any number of spaces and/or commas between numbers as a separator.
Here's a quick demo I tested (skipping the file input, since you seem to have that figured out):
public class tmp {
public static void main(String[] args) {
String input = "1,2, 3, 4\n5, 6, 7\n\n,8"; //what we would read from file
input = input.replace("\n"," "); //simulate removing newlines, ignore this
String[] items = input.split("[, ]+"); //regex split
for(String one : items) {
System.out.println(one);
}
}
}

Something like this?
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
public class Testing
{
public static void main(String[] args) throws IOException
{
try
{
//open a file
String path;
path = "TestNumbers.txt";
File f = new File(path);
FileReader re = new FileReader(f);
BufferedReader in = new BufferedReader(re);
String line = ""; //reads a line
String store = ""; //combines all the lines
while((line = in.readLine()) != null) //for each line
{
if(line.length() > 0){ //only add to store if the line contains something
store = store + line + " ";
}
}
//create the string array
String[] finalString = store.split(", ");
//and the integer array to hold the numbers
int [] intArray = new int [finalString.length];
//parse the string array and put each index to the integer array
for (int i = 0; i < finalString.length; i++){
intArray[i] = Integer.parseInt(finalString[i]);
}
//simply calculating the average
//summing up
int sum = 0;
for (int i = 0; i < intArray.length; i++){
sum = sum + intArray[i];
}
//printing the average
System.out.println("Average: " + sum / intArray.length);
}
catch (FileNotFoundException e)
{
e.printStackTrace();
}
}
}

Replacing text from a file in Java

I've been trying to create a simple program, in java, that replaces some words into a file. To change the text into the file I created, I create a String and set it to the file text:
Path path = Paths.get("somePath/someFile.someExtension");
Charset charset = StandardCharsets.UTF_8;
String s = new String(Files.readAllBytes(path), charset);
EDIT: To save to the file with s, I used Files.write(path, s.getBytes(charset));.
Then I change the String with commands like s.replaceAll("A", "B"). But now, I'm stuck. I want to make a more complicated then just replacing "A" with "B". I'll try to explain has best I can:
I need to find at the file if wall someNumber someNumer someNumber is in there, and if there are three arguments (someNumber someNumber someNumber), then get the value of "someNumber" at the center. For example:
If the commands are:
wall 200 500 100
wall 200 500 100
Then I want to get the argument from the center (in the first case 500 and in the second 500), and store it into a variable, then delete it from the String. After that, on the top of these commands (in the example wall 200 500 100 wall 200 500 100), I want to write:
usemtl texture
ceil (someNumber that we stored, in the case, 500)
Note that if the arguments wall wall don't have any kind of separation (for example #other wall), then the someNumber at the center will be equal (500 and 500 are equal). So, the command below will just appear per group (if wall wall wall... isn't separed with #other wall, for example).
Other example, this would be the file before / after:
Before:
wall 100 300 50
wall 100 300 100
wall 100 300 400
After:
usemtl texture
ceil 300
wall 100 50
wall 100 100
wall 100 400
So, how can I make this replace?
Please answer! I have no idea how!
EDIT: question to #Roan, owner of most of this code:
Now, after the answers #Roan code transformed into:
package com.fileConverter.main;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import javax.swing.JFileChooser;
public class FileReplace extends JFileChooser {
private static final long serialVersionUID = -254322941935132675L;
private static FileReplace chooser = new FileReplace();
public static void main(String[] args) {
chooser.showDialog(chooser, "Open");
}
public void cancelSelection() {
System.exit(0);
}
public void approveSelection() {
super.approveSelection();
System.out.println("starting...");
// The path were your file is
String path = chooser.getSelectedFile().getAbsolutePath();
File file = new File(path);
// try to create an inputstream from the file
FileInputStream fis = null;
try {
fis = new FileInputStream(file);
} catch (FileNotFoundException e) {
// If we are here the file is not found
e.printStackTrace();
}
// make it a buffered reader
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(fis));
// to store the current line
String line;
// array to store the different words
String[] words;
// create a second temporally file that will replace the original file
File file2 = new File(chooser.getSelectedFile().getParentFile()
+ "$$$$$$$$$$$$$$$.tmp");
try {
file.createNewFile();
} catch (IOException e1) {
e1.printStackTrace();
}
// and create the streams
FileOutputStream file2Os = null;
try {
file2Os = new FileOutputStream(file2);
} catch (FileNotFoundException e1) {
e1.printStackTrace();
}
PrintWriter writer = new PrintWriter(file2Os);
try {
System.out.println("replacing code...");
writer.println("mtllib textures.mtl");
// loop through all lines and
while ((line = bufferedReader.readLine()) != null) {
line = line
.replace("//", "#")
.replace("(", "wall")
.replace(")", "\n")
.replace("{", "")
.replace("}", "")
.replace("# brush from cube",
"room cube" + countWords(line, "cube"))
.replace(" NULL 0 0 0 1 1 0 0 0", "")
.replace("\"classname\"", "")
.replace("\"worldspawn\"", "");
// get all the diffent terms
words = line.split(" ");
// see if there are 4 terms in there: wall x x x
// and if the first term equals wall28
// and if the middle number is the number you want to delete
// if not just copy the line over
if (words.length == 4 && words[0].contains("wall")) {
double doubleVal = Double.parseDouble(words[2]);
int val = (int) doubleVal;
// now modify the line by removing the middel number
String newLine = words[0] + " " + words[1] + " " + words[3];
String valInsert = null;
if (val >= 0)
valInsert = "\n" + "usemtl texture" + "\n" + "ceil "
+ val;
else if (val < 0)
valInsert = "\n" + "usemtl texture" + "\n" + "floor "
+ val;
// write this to the new file
writer.println(valInsert);
writer.println(newLine);
} else {
// copy the old line
writer.println(line);
}
}
} catch (IOException e) {
e.printStackTrace();
}
// close our resources
writer.close();
try {
bufferedReader.close();
} catch (IOException e) {
e.printStackTrace();
}
// now we rename the temp file and replace the old file
// with the new file with the new content
file.delete();
file2.renameTo(file);
System.out.println("done!");
}
public int countWords(String string, String word) {
int lastIndex = 0;
int count = 0;
while (lastIndex != -1) {
lastIndex = string.indexOf(word, lastIndex);
if (lastIndex != -1) {
count++;
lastIndex += word.length();
}
}
return count;
}
}
The problem is that this part doesn't make any replaces:
if (words.length == 4 && words[0].contains("wall")) {
double doubleVal = Double.parseDouble(words[2]);
int val = (int) doubleVal;
// now modify the line by removing the middel number
String newLine = words[0] + " " + words[1] + " " + words[3];
String valInsert = null;
if (val >= 0)
valInsert = "\n" + "usemtl texture" + "\n" + "ceil "
+ val;
else if (val < 0)
valInsert = "\n" + "usemtl texture" + "\n" + "floor "
+ val;
// write this to the new file
writer.println(valInsert);
writer.println(newLine);
}
How can I fix it? Other thing, this part is suppose to create a number that grows after checking how many times cube is wrote, but it doesn't works too :(
.replace("# brush from cube", "room cube" + countWords(line, "cube"))
The countWords method:
public int countWords(String string, String word) {
int lastIndex = 0;
int count = 0;
while (lastIndex != -1) {
lastIndex = string.indexOf(word, lastIndex);
if (lastIndex != -1) {
count++;
lastIndex += word.length();
}
}
return count;
}
Many Thanks

Ok I'm very unsure if I understood this correctly.
This is my interpretation of wat your question is:
You have a file with lines that say: wall [number] [number] [number]
Now you want to check if there there are 3 numbers and then delete the middle number if it's the number you're searching for.
So I would do this like this:
Befor you run the programm you'll need a folder on your C: drive with the name "text" and inside that folder you'll need a file called text.txt with you format in it so: for example:
wall 123 300 320
If you change the value of number you can specify wich number the middle number has to be in oder for it to be deleted.
public class FileReplace {
public static void main(String[] args){
//The path were your file is
String path = "C:\\text\\text.txt";
File file = new File(path);
//The number you want to delete
int number = 300;
//try to create an inputstream from the file
FileInputStream fis = null;
try {
fis = new FileInputStream(file);
} catch (FileNotFoundException e) {
//If we are here the file is not found
e.printStackTrace();
}
//make it a buffered reader
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(fis));
//to store the current line
String line;
//array to store the different words
String[] words;
//create a second temporally file that will replace the original file
File file2 = new File("C:\\text\\$$$$$$$$$$.tmp");
try {
file.createNewFile();
} catch (IOException e1) {
e1.printStackTrace();
}
//and create the streams
FileOutputStream fos = null;
try {
fos = new FileOutputStream(file2);
} catch (FileNotFoundException e1) {
e1.printStackTrace();
}
PrintWriter writer = new PrintWriter(fos);
try {
//loop through all lines and
while((line = bufferedReader.readLine()) != null){
//get all the diffent terms
words = line.split(" ");
//see if there are 4 terms in there: wall x x x
//and if the first term equals wall
//and if the middle number is the number you want to delete
//if not just copy the line over
if(words.length == 4 && words[0].equals("wall") && words[2].equals(String.valueOf(number))){
//now modify the line by removing the middel number
String newLine = words[0] + " " + words[1] + " " + words[3];
//write this to the new file
writer.println(newLine);
}else{
//copy the old line
writer.println(line);
}
}
} catch (IOException e) {
e.printStackTrace();
}
//close our resources
writer.close();
try {
bufferedReader.close();
} catch (IOException e) {
e.printStackTrace();
}
//now we rename the temp file and replace the old file
//with the new file with the new content
file.delete();
file2.renameTo(file);
}
}
If you have ay questions about this code feel free to ask them.
Oh and also you might need to run this with administrator right as it uses files.
Hope this helps.

To analyse a string and see if it matches ("wall" number number number), you can use a REGEX expression: see the doc here.
To use the regex expression, just apply .matches() on your String variable and it'll return true or false depending on if the format is verified.
If the format is verified, then just use the SubString function, specify the start and end index so you get the middle number.
To take it out, you could do the opposite. SubString the start (everything until middle number), then SubString the end (everything after the middle number), and then create a new string using those 2.

A simple solution without using (explicitly) regex is to split the String using token (in your case it's a white space.
line = "wall 100 300 50";
String[] words = line.split("\\s+");
You can then get the words[2] convert to an int etc. Then you can write back to a new file (or the same if you have read all file contents).
Regex are more powerful but to me a bit more intimidating so you can pick whatever matches your needs.

You could use this to count the number of occurances of a word in a string:Try 1:
public static int countWords(String string, String word) {
//get all individual words
String[] terms = string.split(" ");
//this variable counts how many times word occurs
int count = 0;
//a count variable for the loop
int counter = 0;
//loop through all the words and if we see a word that equals word we add one to the count variable
while(counter < terms.length){
//check if the term equals the word
if(terms[counter].equals(word)){
//the term matched add one to the count variable
count++;
}
counter++;
}
//return the number of occurrences
return count;
}
Try 2:
public static String countWords(String string, String word) {
//get all individual words
String[] terms = string.split(" ");
//this variable counts how many times word occurs
int count = 0;
//a count variable for the loop
int counter = 0;
StringBuffer sb = new StringBuffer();
sb.append("1");
//loop trough all the words and if we see a word that equals word we add one to the count variable
while(counter < terms.length){
//check if the term equals the word
if(terms[counter].equals(word)){
//the term matched add one to the count variable
count++;
sb.append(" " + word + (count + 1));
}
counter++;
}
//return the number of occurrences
return sb.toString();
}<br><br>
Try 3: you need to have a static variable in your code called lastVar:
static int lastVar = 0;
public static String countWords(String string, String word) {
//get all individual words
String[] terms = string.split(" ");
//this variable counts how many times word occurs
int count = 0;
//a count variable for the loop
int counter = 0;
StringBuffer sb = new StringBuffer();
sb.append("1");
//loop trough all the words and if we see a word that equals word we add one to the count variable
while(counter < terms.length){
//check if the term equals the word
if(terms[counter].equals(word)){
//the term matched add one to the count variable
count++;
sb.append(" " + word + (count + 1 + lastVar));
}
counter++;
}
lastVar += count + 1;
//return the number of occurrences
return sb.toString();
}
That should work.
Hope this helps :D.

To reformat your cube lines you could use:
Try 1:
// loop through all lines and
while ((line = bufferedReader.readLine()) != null) {
if(line.contains("// brush from cube")){
line = line.replace("// brush from cube ", "").replace("(", "").replace(")", "");
String[] arguments = line.split("\\s+");
line = "cube" + Cube + " usemtl texture ceil " + arguments[2] + " wall " + arguments[1] + " " + arguments[3] + " usemtl texture floor " + arguments[5] + " wall " + arguments[4] + " " + arguments[6];
Cube++;
}
line = line
.replace("//", "#")
.replace("(", "wall")
.replace(")", "\n")
.replace("{", "")
.replace("}", "")
.replace(" NULL 0 0 0 1 1 0 0 0", "")
.replace("\"classname\"", "")
.replace("\"worldspawn\"", "");
try 2:
// loop through all lines and
while ((line = bufferedReader.readLine()) != null) {
if(line.contains("// brush from cube")){
line = line + bufferedReader.readLine() + " " + bufferedReader.readLine();
line = line.replace("// brush from cube ", "").replace("(", "").replace(")", "");
String[] arguments = line.split("\\s+");
line = "cube" + Cube + " usemtl texture ceil " + arguments[2] + " wall " + arguments[1] + " " + arguments[3] + " usemtl texture floor " + arguments[5] + " wall " + arguments[4] + " " + arguments[6];
Cube++;
}
line = line
.replace("//", "#")
.replace("(", "wall")
.replace(")", "\n")
.replace("{", "")
.replace("}", "")
.replace(" NULL 0 0 0 1 1 0 0 0", "")
.replace("\"classname\"", "")
.replace("\"worldspawn\"", "");
P.S. I've only posted the important part. You should be able to see where in the code this goes. Also you need to have a static int called cube somewhere in you code like:
static int Cube = 1;
That should be it if it doesn't work let me know! :D

Java bmp from Databbase

I need to create a BMP (bitmap) image from a database using Java. The problem is that I have huge sets of integers ranging from 10 to 100.
I would like to represent the whole database as a bmp. The amount of data 10000x10000 per table (and growing) exceeds the amount of data I can handle with int arrays.
Is there a way to write the BMP directly to the hard drive, pixel by pixel, so I don't run out of memory?

A file would work (I definitely woudln't do a per pixel call, you'll be waiting hours for the result). You just need a buffer. Break the application apart along the lines of ->
int[] buffer = new int[BUFFER_SIZE];
ResultSet data = ....; //Forward paging result set
while(true)
{
for(int i = 0; i < BUFFER_SIZE; i++)
{
//Read result set into buffer
}
//write buffer to cache (HEAP/File whatever)
if(resultSetDone)
break;
}
Read the documentation on your database driver, but any major database is going to optimize your ResultSet object so you can use a cursor and not worry about memory.
All that being said... an int[10000][10000] isn't why you're running out of memory. Its probably what you're doing with those values and your algorithm. Example:
public class Test
{
public static void main(String... args)
{
int[][] ints = new int[10000][];
System.out.println(System.currentTimeMillis() + " Start");
for(int i = 0; i < 10000; i++)
{
ints[i] = new int[10000];
for(int j = 0; j < 10000; j++)
ints[i][j] = i*j % Integer.MAX_VALUE / 2;
System.out.print(i);
}
System.out.println();
System.out.println(Integer.valueOf(ints[500][999]) + " <- value");
System.out.println(System.currentTimeMillis() + " Stop");
}
}
Output ->
1344554718676 Start
//not even listing this
249750 <- value
1344554719322 Stop
Edit--Or if I misinterpreted your question try this ->
http://www.java2s.com/Code/Java/Database-SQL-JDBC/LoadimagefromDerbydatabase.htm
I see... well take a look around, I'm rusty but this seems to be a way to do it. I'd double check my buffering...
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
public class Test
{
public static void main(String... args)
{
// 2 ^ 24 bytes, streams can be bigger, but this works...
int size = Double.valueOf((Math.floor((Math.pow(2.0, 24.0))))).intValue();
byte[] bytes = new byte[size];
for(int i = 0; i < size; i++)
bytes[i] = (byte) (i % 255);
ByteArrayInputStream stream = new ByteArrayInputStream(bytes);
File file = new File("test.io"); //kill the hard disk
//Crappy error handling, you'd actually want to catch exceptions and recover
BufferedInputStream in = new BufferedInputStream(stream);
BufferedOutputStream out = null;
byte[] buffer = new byte[1024 * 8];
try
{
//You do need to check the buffer as it will have crap in it on the last read
out = new BufferedOutputStream(new FileOutputStream(file));
while(in.available() > 0)
{
int total = in.read(buffer);
out.write(buffer, 0, total);
}
}
catch (IOException e)
{
e.printStackTrace();
}
finally
{
if(out != null)
try
{
out.flush();
out.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
System.out.println(System.currentTimeMillis() + " Start");
System.out.println();
System.out.println(Integer.valueOf(bytes[bytes.length - 1]) + " <- value");
System.out.println("File size is-> " + file.length());
System.out.println(System.currentTimeMillis() + " Stop");
}
}

You could save it as a file, which is conceptually just a sequence of bytes.