I'm only asking this as a last resort. I'm stumped.
I have written a small app which performs very simple image processing. It is made using JDK 1.8, written in Netbeans 8.0.1 and is running on Debian Linux.
The application captures a large number of individual frames at a certain framerate, set by the user, by calling the 'streamer' webcam program via a process builder. Once it has begun capturing, it begins to translate the frames into RGB values and checks whether or not any pixels are above a user defined threshold. If no pixels exceed this threshold, it simply deletes the frame. If any pixels do exceed it, it moves the frame to a different folder for user inspection. This all works fine. It keeps up with even relatively high framerates and selects appropriate frames as expected. However, when the number of frames processed reaches around 1500 (or fewer for lower framerates), the program is freezing. I've tried issuing the commands to streamer manually at the command line, and it seems perfectly capable of producing as many as required, so I have to assume the issue is with my coding. The images are only small (320x240). Am I somehow maxxing out the available memory (I am getting no errors, just freezing).
The purpose of this program is to detect cosmic ray impacts on a CMOS sensor, part of a friend's dissertation. If I can't get this working reliably, the poor kid's going to have to go through the footage manually!
The code is attached below. Apologies for the length of the code, but as all of it is fairly crucial, I didn't want to omit anything.
package cosmicraysiii;
import java.io.File;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.swing.JOptionPane;
public class CosmicRaysIII {
public static int dark = 0;//Dark current determined by averaging
public static int tol = 0;//Tolerance set by user
public static int frames = 0;//Total number of frames set by user
public static int runs = 0;
public static int rate = 0;//Framerate set by user
public static int time = 0;//Total time calculated
public static String zeros = "";
public static int ready = 0;
public static void main(String[] args) throws IOException, InterruptedException {
//Get directory ID from user
String id = JOptionPane.showInputDialog("Enter a folder name for detections (no ., in name):");
//Get dark current
Dark d = new Dark();
dark = d.getCurrent();
//Get tolerance from user, will be added to dark current.
String t = JOptionPane.showInputDialog("Dark Current = " + dark + "/255\n"
+ "Enter a tolerance (integer values only).\n "
+ "This will be added to the dark current:");
tol = Integer.parseInt(t) + dark;
//Get number of frames from user
String fs = JOptionPane.showInputDialog("Enter the total number of frames required (Mulitples of 500 only):");
frames = Integer.parseInt(fs);
runs = frames / 500;
//Get framerate from user
String r = JOptionPane.showInputDialog("Enter the framerate required:");
rate = Integer.parseInt(r);
//Determine duration
time = (int) Math.round(frames / rate);
//Provide summary for user and request permission to continue
int secs = time % 60;
int mins = (time - secs) / 60;
int hrs = (mins - (mins % 60)) / 60;
if (hrs >= 1) {
mins = mins % 60;
}
String theMessage = "The following parameters have been set:\n"
+ "Tolerance (including dark current): " + tol + "\n"
+ "Frames: " + frames + "\n"
+ "Frame rate: " + rate + " fps\n"
+ "Total capture time: " + time + " sec\n"
+ " " + hrs + " h " + mins + " m " + secs + " s\n"
+ "\n"
+ "Would you like to proceed?";
int result = JOptionPane.showConfirmDialog(null, theMessage, "Continue?", JOptionPane.OK_CANCEL_OPTION);
if (result == 2) {
System.exit(0);
}
//Create directory for data acquisition
ProcessBuilder pb1 = new ProcessBuilder("mkdir", "data");
pb1.start();
//Establish array of filenames
String[] filenames = new String[frames];
//Fill filenames array with filenames
//Taking into consideration that the filename
//will have a varying number of zeros appended
//before the frame number, dependent on the
//order of the frame number
for (int i = 0; i < frames; i++) {
if (i < 10) {
zeros = "00000000";
} else if (i >= 10 && i < 100) {
zeros = "0000000";
} else if (i >= 100 && i < 1000) {
zeros = "000000";
} else if (i >= 1000 && i < 10000) {
zeros = "00000";
} else if (i >= 10000 && i < 100000) {
zeros = "0000";
} else if (i >= 100000 && i < 1000000) {
zeros = "000";
} else if (i >= 1000000 && i < 10000000) {
zeros = "00";
} else if (i >= 10000000 && i < 100000000) {
zeros = "0";
} else {
zeros = "";
}
filenames[i] = "./data/frame" + zeros + i + ".ppm";
}
//Begin data acquisition
new Thread(new Runnable() {
public void run() {
try {
//Capture images
ProcessBuilder pb2 = new ProcessBuilder("streamer", "-t", Integer.toString(frames), "-r", Integer.toString(rate), "-p", "0", "-o", "./data/frame000000000.ppm");
Process p = pb2.start();
p.waitFor();
ready = 1;
} catch (IOException | InterruptedException ex) {
Logger.getLogger(CosmicRaysIII.class.getName()).log(Level.SEVERE, null, ex);
}
}
}).start();
//Sleep to allow some image capture to prevent thread disordering
Thread.sleep(3000);
//Check array size
System.out.println("Array size: " + filenames.length);
//Conduct image analysis
new Thread(new Runnable() {
public void run() {
int done = 0;
int donea = 0;
while (ready == 0) {
for (int i = 0; i < frames; i++) {
File f = new File(filenames[i]);
if (f.exists() && !filenames[i].equals("")) {//Check file still exists
try {
//Perform analysis steps
Analysis a = new Analysis();
//STEP 1: Convert file from P6 to P3
String newfile = a.convert(filenames[i], zeros, i);
//STEP 2: Read file
a.read(newfile, tol, i, id);
filenames[i] = "";
done++;
} catch (IOException ex) {
Logger.getLogger(CosmicRaysIII.class.getName()).log(Level.SEVERE, null, ex);
}
}
if (done > donea) {
System.out.println(done + " files processed");
donea = done;
}
}
}
}
}).start();
}
}
Then the Analyse.java class is as follows:
package cosmicraysiii;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
public class Analysis {
public String convert(String ofile, String zeros, int number) throws IOException {
//Create new file name
String nfile = "./proc/frame" + zeros + number + ".ppm";
//Ensure process directory exists
ProcessBuilder mkdir = new ProcessBuilder("mkdir", "proc");
mkdir.start();
//Convert file to P3 PPM (RGB format) and move to process folder
ProcessBuilder convert = new ProcessBuilder("convert", ofile, "-compress", "none", nfile);
convert.start();
//Delete original file
ProcessBuilder del = new ProcessBuilder("sudo", "rm", ofile);
del.start();
//Return new filename
return nfile;
}
public void read(String filename, int tol, int ix, String id) throws FileNotFoundException, IOException {
int move = 0;
//Make directory for hits
ProcessBuilder mkdir = new ProcessBuilder("mkdir", "hits" + id);
mkdir.start();
//Open reader to read file
File f = new File(filename);
if (f.exists()) {
BufferedReader br = new BufferedReader(new FileReader(filename));
String line;
//To eliminate header
int x = 0;
//Iterate through text to find abnormal pixels
while ((line = br.readLine()) != null) {
x++;
String[] pixrgb = line.split("\\ ");
//Iterate through pixels on each line
for (int i = 0; i < pixrgb.length; i++) {
if (x >= 4) {//Eliminate header
//Check each pixel value
try {
int pixval = Integer.parseInt(pixrgb[i]);
if (pixval > tol) {
move = 1;
break;
}
} catch (NumberFormatException ne) {
}
}
}
}
if (move == 1) {
//Move file to hits folder
ProcessBuilder pb3 = new ProcessBuilder("sudo", "cp", filename, "./hits" + id + "/detection" + ix + ".ppm");
pb3.start();
//Delete original file
ProcessBuilder del = new ProcessBuilder("sudo", "rm", filename);
del.start();
}
}
//Delete original file
ProcessBuilder del = new ProcessBuilder("sudo", "rm", filename);
del.start();
}
}
I appreciate this is quite a lengthly chunk of code to be posting. Really appreciate any help that can be given.
G
OK I have managed to solve this by completely overhauling the analysis process. Firstly, rather than converting the image into a P3 .ppm file, I now examine the pixels directly from the image using BufferedReader. Then, I stopped looping through the file list repeatedly. Instead, the loop which calls Analysis() just runs through the list of filenames once. If it encounters a file which does not yet exist, it does Thread.sleep(500) and then tries again. I have now successfully run a batch of 50,000 frames without incident, and there is now much less of a drain on memory thanks to the improved process. Just thought I should place this answer up here in case anyone comes across it. I may post code if anyone wants it.
Related
I have generating data of users with auto-increment ID, then write it to file following these rules:
Name the file in following structure (FileCounter)_(StartID)_(EndID)
Maximum 1000 records per file
If don't have enough 1000 records to write, wait maximum 10s, if any added, write it all to file otherwise, write the remain list to file (not enough 1000), if nothing to write after wait, create empty file with naming (FileCounter)_0_0
My approach is using 2 thread, 1 thread to generate data then push it to the queue, 1 thread to take from the queue add to a list then write the list to the file.
//Generate function
public void generatedata() {
int capacity = 1678;
synchronized(users) {
for(int index = 0; index <capacity; index++) {
users.add(generateUser());
// notify to read thread
users.notifyAll();
}
}
//Write function
public void writeToFile(ArrayList<User> u) {
String fileName ="";
if(!u.isEmpty()) {
String filename = "" + (++FileCounter) + "_"+ u.get(0).getId() + "_" +
u.get(u.size() - 1).getId() + ".txt";
try {
FileWriter writer = new FileWriter(filename, true);
for (User x : u) {
System.out.println(x.toString());
writer.write(x.getId() + " | " + x.getFormatedDate() + " | " +
x.getSex() + " | " + x.getPhoneNum().getPhoneNumber() + " | " +
x.getPhoneNum().getProvider() + "\r\n");
}
writer.close();
}
catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
else {
try {
fileName = ""+(++FileCounter) +"_0_0.txt";
File f = new File(fileName);
f.createNewFile();
} catch (IOException ex) {
Logger.getLogger(UsersManager.class.getName()).log(Level.SEVERE,
null, ex);
}
}
}
//Read function
public ArrayList<User> ReadFromQueue(ArrayList<User> u) {
while(true) {
try {
int size = users.size();
if(users.isEmpty() && u.size() < 1000) {
users.wait(10000);
if(isChanged(size)) {
System.out.println("Size changed here");
u.add(users.take());
}
else return u;
}
if(u.size() == 1000) {
System.out.println("Check the size is 1000");
return u;
}
u.add(users.take());
} catch (InterruptedException ex) {
Logger.getLogger(UsersManager.class.getName()).log(Level.SEVERE,
null, ex);
}
}
It work fine when I run 1 thread to generate data, 1 thread to read then write data to file but when I use 2++ thread for each generate thread of write thread, There are 1 problems :
The list written in the file still has 1000 records as expected but not sequential at all, it only ascending order.
My output is like:
1_2_1999.txt
2_1_2000.txt
3_2001_3000.txt
My expected output is like:
1_1_1000.txt
2_1001_2000.txt
....
Thanks in advance!
using the thread approach is best for when you do not want to control the amount per file. but since you have a constraint of 1000 records, it's probably easier to use a counter;
public class DataReaderWriter(){
//keeps track of where you left off at, which row in source data.
static int currentRowInSourceData = 0;
public static void main(String[] args){
List<ContactRecord> contacts = getMoreData();
writeRecords(contacts);
}
writeRecords(List<ContactRecord> contacts){
int maxRecords = currentRowInSourceData+1000;
for(int i = currentRowInSourceData;i<maxRecords;i++){
ContactRecord c = contacts.get(i);
writeToFile(c);
currentRowInSourceData++;
}
}
I had a project where I needed to create 90 second previews from larger MP4 files. What I did was to have multiple threads start up with access to a shared Queue of file names. Each thread consumes work from the Queue by using queue.poll().
Here is the Constructor:
public Worker(Queue<String> queue, String conferenceYear, CountDownLatch startSignal, CountDownLatch doneSignal) {
this.queue = queue;
this.startSignal = startSignal;
this.doneSignal = doneSignal;
}
Then, as I said above, I keep polling for data:
public void run() {
while (!queue.isEmpty()) {
String fileName = queue.poll() + ".mp4";
File f = new File("/home/ubuntu/preview_" + fileName);
if (fileName != null && !f.exists()) {
System.out.println("Processing File " + fileName + "....");
I started these threads in another class called WorkLoad:
public static void main(String[] args) {
long startTime = System.currentTimeMillis();
BlockingQueue<String> filesToDownload = new LinkedBlockingDeque<String>(1024);
BlockingQueue<String> filesToPreview = new LinkedBlockingDeque<String>(1024);
BlockingQueue<String> filesToUpload = new LinkedBlockingDeque<String>(1024);
for (int x = 0; x < NUMBER_OF_THREADS; x++) {
workers[x] = new Thread(new Worker(filesToPreview, currentYear, startSignal, doneSignal));
workers[x].start();
}
In your specific case, you could provide each thread its own file name, or a handle on a file. If you want the file names and entries in a chronological sequence, then just start 2 threads, 1 for acquiring data and placing on a queue, with a barrier/limit of 1000 records, and the other thread as a consumer.
the original code creates multiple threads. I am able to create 90 second snippets from over 1000 MP4 videos in about 30 minutes.
Here I am creating a thread per processor, I usually end up with at least 4 threads on my AWS EC2 instance:
/**
* Here we can find out how many cores we have.
* Then make the number of threads NUMBER_OF_THREADS = the number of cores.
*/
NUMBER_OF_THREADS = Runtime.getRuntime().availableProcessors();
System.out.println("Thread Count: "+NUMBER_OF_THREADS);
for (int x = 0; x < NUMBER_OF_THREADS; x++) {
workers[x] = new Thread(new MyClass(param1, param2));
workers[x].start();
}
I have completed a java assignment. The target is to build a java app using eclipse IDE to add.userAccount(String id, String name, String email) in to a file.
I am curious about optimization. I put that code into a loops (such as:
for(int times = 0, time < 10000; time++)
add.userAccount(String id, String name, String email);)
but it takes alot of time (5 minutes). I use FileInputStream and FileOutputStream. (to input and output) Is anyone here suggest me the way to optimize this code?
Here is my code in Main test, i create a test case with 10000 times:
String path = "/home/thinhnv/user.txt";
Account account = new Account(path);
for (int i = 0; i < 10000; i++) {
Random rd = new Random();
int rd1 = rd.nextInt(10);
int rd2 = rd.nextInt(10);
int rd3 = rd.nextInt(10);
int rd4 = rd.nextInt(10);
String ten= "" + rd1 + rd2;
String maTK = "Thinh" + rd1 + rd2 + rd3 + rd4;
String sdt = "" + rd1 + rd2 + rd1 + rd2 + rd1 + rd2 + rd1 + rd2 + rd1 + rd2;
String pass = sdt;
User user = new User(maTK, pass, ten, sdt);
account.add(user);
}
and here is my code for more details:
public void add(User user) {// user i create in main
// TODO Auto-generated method stub
try {
getUsersInforFromFile();// this method i open a file, read
//data from this and take it in to List<User> users private property and// //end of this method i close input file
} catch (IOException e) {
// TODO Auto-generated catch block
System.out.println(" Error: getUsersInforFromFile: " + e.getMessage());
}
if (this.users.size() == 0) {
writeUserToFile(user);
System.out.println("signup successfully!");
return;
}
if (this.users.indexOf(user) == -1) {
writeUserToFile(user);// this method i open a file and write a data at append mode and then close this file
System.out.println("signup successfully!");
} else {
System.out.println("Account" + user.getMaTK() + " is already exist! Sign up failed");
}
}
Lastly, i use only methods : fineIn.read(), and fileOut.write() byte by byte. Is this make my programme run more slowly?
You can reduce number of write operations by creating large of chunks of data together, Opening and closing file stream takes time so try reduce that as well.
import java.io.*;
import java.util.Random;
public class LargeDataset {
public static void main(String[] args) throws Exception {
File file = new File("src/Salary.txt");
if (file.exists()) {
System.out.print("Sorry this file already exists.");
System.exit(0);
}
String firstName = "";
String lastName = "";
String rank = "";
double salaryRange = 0.0;
for (int i = 1; i <= 1000; i++) {
try (PrintWriter output = new PrintWriter(file))
{
firstName = "FirstName" + i;
lastName = "LastName" + i;
rank = generateRandomRank();
if (rank == "assistant")
salaryRange = generateSalary(50000.00, 80000.00);
else if (rank == "associate")
salaryRange = generateSalary(60000.00, 110000.00);
else
salaryRange = generateSalary(75000.00, 130000.00);
output.printf("%s %s %s $%.2f", firstName, lastName, rank, salaryRange);
output.println();
}
}
}
public static String generateRandomRank() {
String[] rank = {"assistant", "associate", "full"};
Random random1 = new Random();
return rank[random1.nextInt(3)];
}
public static double generateSalary(double minSalary, double maxSalary) {
double randomSalary = minSalary + Math.random() * (maxSalary - minSalary);
return randomSalary;
}
}
Hi everyone. I have a program that generates 1000 lines of text and saves it into a file named Salary. The format of each line is: firstNamei, lastNamei, a random rank, and a random salary that is suited to the rank. However when I run this program it only outputs the 1000th line of the loop. I noticed however, when I don't put the PrintWriter in the try statement and close it after the loop by myself, it runs fine and generates all 1000 lines. Why is it only generating the last line based on how it is right now though?
You should open your PrintWriter once, and then write to it many times from your loop, not the other way around:
try (PrintWriter output = new PrintWriter(file)) {
for (int i = 1; i <= 1000; i++) {
firstName = "FirstName" + i;
lastName = "LastName" + i;
rank = generateRandomRank();
if (rank == "assistant")
salaryRange = generateSalary(50000.00, 80000.00);
else if (rank == "associate")
salaryRange = generateSalary(60000.00, 110000.00);
else
salaryRange = generateSalary(75000.00, 130000.00);
output.printf("%s %s %s $%.2f", firstName, lastName, rank, salaryRange);
output.println();
}
}
You should use the above pattern instead of what you have. If you want an exact fix to your current code, then you may try opening the PrintWriter in append mode:
for (int i=1; i <= 1000; i++) {
try (PrintWriter output = new PrintWriter(new FileOutputStream(file, true)) {
// same logic
}
}
This should also work, because now, even though you create a new PrintWriter for each iteration of the loop (inefficient), you open the underlying file in append mode, so each new line should get written properly.
Every time that you are iterating through your 1000 you are creating a new file
for (int i = 1; i <= 1000; i++) {
try (PrintWriter output = new PrintWriter(file))
...
}
move it before the loop
try (PrintWriter output = new PrintWriter(file)) {
for (int i = 1; i <= 1000; i++) {
}
}
So I'm working on a project for school. I need to record the data from my arrays into Excel and I'm having some trouble. This is part of my experiment class.
public static void exp(Params params) {
Scanner s = new Scanner(System.in);
String temp;
for (temp = ""; temp.isEmpty(); temp = s.nextLine()) {
System.out.println("Enter a directory and filename that you want the results to be saved under.");
System.out.println("(If no directory is specified, results will be in same folder as the jar.)");
}
params.setFileName(temp);
s.close();
System.out.println("Executing...");
long timeArray[] = new long[params.getFinalSize() / params.getIncrement()];
int counter = 0;
SortFacade facade = new SortFacade();
for (int n = params.getInitialSize(); n <= params.getFinalSize(); n += params.getIncrement()) {
System.out.println((new StringBuilder("Array of size: ")).append(n).toString());
long tempTime = 0L;
for (int j = 0; j < params.getNumTrials(); j++) {
System.out.println((new StringBuilder("Trial #")).append(j + 1).toString());
params.generateArrays(n, params.getTypeList());
tempTime += facade.sort(params);
}
timeArray[counter] = tempTime / (long) params.getNumTrials();
counter++;
}
params.setTimeArray(timeArray);
System.out.println("Times for each array size(ms): " + Arrays.toString(timeArray));
System.out.println("Writing to File...");
System.out.println("Complete.");
}
}
Start with Filewriter
Make sure what you write is comma-delimeted and saved as a .csv
Also, take a look at this existing question.
I made a java agent that generates an xml code containing texts per slide from powerpoint files attached in lotus notes documents. However, when it tries a 66 MB attachment the error is thrown.
Here's the full code in the agent
import java.io.*;
import org.apache.poi.hslf.HSLFSlideShow;
import org.apache.poi.hslf.model.Slide;
import org.apache.poi.hslf.model.TextRun;
import org.apache.poi.hslf.usermodel.SlideShow;
import org.apache.poi.poifs.filesystem.POIFSFileSystem;
import org.apache.poi.xslf.usermodel.XMLSlideShow;
import org.apache.poi.xslf.usermodel.XSLFShape;
import org.apache.poi.xslf.usermodel.XSLFSlide;
import org.apache.poi.xslf.usermodel.XSLFTextParagraph;
import org.apache.poi.xslf.usermodel.XSLFTextShape;
import lotus.domino.*;
public class JavaAgent extends AgentBase {
public void NotesMain() {
System.out.println("Start");
try {
Session session = getSession();
AgentContext agentContext = session.getAgentContext();
DocumentCollection dc = agentContext.getUnprocessedDocuments();
Document doc = dc.getFirstDocument();
Database db = agentContext.getCurrentDatabase();
Document profdoc = db.getProfileDocument("f.SomeProfileDoc", "");
String siteurl = profdoc.getItemValueString("SomeField");
//System.out.println(siteurl);
//ArrayList listFilenames = new ArrayList();
//int indexnum = 0;
String extension = "";
String buildXMLStr = "";
InputStream fis = null;
POIFSFileSystem fs = null;
while (doc != null) {
RichTextItem body = (RichTextItem)doc.getFirstItem("Attachments");
if(body != null) {
RichTextNavigator rtnav = body.createNavigator();
String filename = "";
if (rtnav.findFirstElement(RichTextItem.RTELEM_TYPE_FILEATTACHMENT)) {
buildXMLStr = "<data>";
do {
EmbeddedObject att = (EmbeddedObject) rtnav.getElement();
filename = att.getSource();
int z = filename.lastIndexOf('.');
if (z > 0) {
extension = filename.substring(z+1);
}
if(extension.equalsIgnoreCase("ppt") || extension.equalsIgnoreCase("pptx")) {
buildXMLStr += "\n<sfile>";
buildXMLStr += "<sfextension>" + extension+ "</sfextension>";
buildXMLStr += "\n<surl>" + siteurl + "/ContentDocs/_" + doc.getUniversalID() + "/$FILE/" + filename + "</surl>";
try {
if(extension.equalsIgnoreCase("ppt")) { // If file is a 2003 powerpoint file
System.out.println("File has ppt extension");
fs = new POIFSFileSystem(att.getInputStream());
HSLFSlideShow show = new HSLFSlideShow(fs);
SlideShow ss = new SlideShow(show);
Slide[] slides = ss.getSlides();
for (int x = 0; x < slides.length; x++) {
buildXMLStr += "\n<slide>";
buildXMLStr += "\n<snum>" + (x + 1) + "</snum>";
if (slides[x].getTitle() == null) {
//System.out.println("Slide " + (x + 1) + " | " + "No Title");
buildXMLStr += "\n<stitle>No Title</stitle>";
} else {
//System.out.println("Slide " + (x + 1) + " | " + slides[x].getTitle());
buildXMLStr += "\n<stitle>" + slides[x].getTitle() + "</stitle>";
}
buildXMLStr += "\n<stexts>";
TextRun[] runs = slides[x].getTextRuns();
for (int i = 0; i < runs.length; i++) {
TextRun run = runs[i];
buildXMLStr += " " + run.getText().replaceAll("[^\\w\\s-]", " ").replaceAll("(?m)^\\s+$", "");
}
buildXMLStr += "\n</stexts>";
buildXMLStr += "\n</slide>";
}
att.getInputStream().close();
}
if(extension.equalsIgnoreCase("pptx")) { // If file is 2007+ powerpoint file
System.out.println("File has pptx extension");
try {
XMLSlideShow pptxshow = new XMLSlideShow(att.getInputStream());
XSLFSlide[] slides = pptxshow.getSlides();
for (int x = 0; x < slides.length; x++) {
buildXMLStr += "\n<slide>";
buildXMLStr += "\n<snum>" + (x + 1) + "</snum>";
if (slides[x].getTitle() == null) {
// System.out.println("No Title");
buildXMLStr += "\n<stitle>No Title</stitle>";
} else {
// System.out.println((x + 1) + slides[x].getTitle());
buildXMLStr += "\n<stitle>" + slides[x].getTitle() + "</stitle>";
}
buildXMLStr += "\n<stexts>";
for (XSLFShape shape : slides[x]) {
if (shape instanceof XSLFTextShape) {
XSLFTextShape txShape = (XSLFTextShape) shape;
for (XSLFTextParagraph xslfParagraph : txShape.getTextParagraphs()) {
buildXMLStr += " " + xslfParagraph.getText().replaceAll("[^\\w\\s-]", " ").replaceAll("(?m)^\\s+$", "");
}
}
}
buildXMLStr += "\n</stexts>";
buildXMLStr += "\n</slide>";
}
att.getInputStream().close();
} catch(Exception epptx) {
epptx.printStackTrace();
}
}
} catch(Exception estart) {
System.out.println("IOE Exception");
estart.printStackTrace();
}
//System.out.println(filename);
buildXMLStr += "\n</sfile>";
buildXMLStr += "\n</data>";
System.out.println(buildXMLStr);
}
//indexnum++;
} while (rtnav.findNextElement());
}
}
else {
System.out.println("There are no attachments on the current document");
}
System.out.println("Next Document");
doc = dc.getNextDocument();
}
} catch(Exception e) {
System.out.println("General Exception");
e.printStackTrace();
}
System.out.println("Finished scanning reports");
}
}
Two things I would suggest:
Use a NPOIFSFileSystem instead of a POIFSFileSystem. It uses NIO and should consume less memory.
Don't concatenate your output into strings. If you have to keep it in memory, use a StringBuilder, otherwise immediately write it to an output stream.
If all else fails, increase the memory allocated to your JVM.
As Richard Schwartz points out you can increase the amount of memory available to the agent manager by setting JavaMaxHeapSize to something larger than the default value (which is a max of 64 MB as detailed here).
(Please notice that JavaMaxHeapSize does not impact the HTTP JVM. To set the amount of memory available to the HTTP task you need to set HTTPJVMMaxHeapSize).
“java.lang.OutOfMemoryError: Java heap space” error is be triggered when you try to add more data into the heap space area in memory, but the size of this data is larger than the JVM can accommodate in the Java heap space.
Java applications are allowed to use only limited amount of memory. This limit is specified during the application startup. To make things more complex, the Java memory is separated to two different regions. These regions are called heap space and permgen
Size of those regions is set during the JVM launch by specifying JVM parameters such as -Xmx and -XX:MaxPermSize. If you do not explicitly set the sizes, platform-specific defaults will be used.
In many cases, the easiest way to get rid of the java.lang.OutOfMemoryError: Java heap space error if you just increase the heap space, specified in the -Xmx parameter, similar to the following
java -Xmx1024m com.yourcompany.YourClass
First of all - you should decide how much memory you can give when starting your program (if you don't have enough memory(RAM), no point running the program if you expect the attachments to be big (66mb shouldn't be an issue). If you have to read the whole document in memory and the size is more than memory allocated to your program, then you can't do anything. It will run out of memory.
But - if you can process part of the document and then remove it from memory, then you can load the next part and continue processing.