Develop Reference

get Horizontal Text with PDFBox [duplicate]

I am using PDFBox to do a simple extraction of words from a PDF file. Then it inserts those words to a table in database. From what I have tested, a 90 degrees clockwise rotated text in PDF will gives gibberish result when I tried to extract the words.
For example, database in the file will yield atabase and also database itself as two different words. Obviously, atabase does not exist in the PDF file.
I tried converting the original file to be rotated upright and do the extraction and it works perfectly as expected. I understand this could be a limitation of the PDFBox itself.
So, in the case of someone trying to index a rotated PDF file, is there a way to tackle this?
Code snippet ( just for reference) :
String lines[] = text.split("\\r?\\n");
for (String line : lines) {
String[] words = line.split(" ");
System.out.println("Line: " + line);
preparedStatement = con1.prepareStatement(sql);
int i=0;
for (String word : words) {
// check if one or more special characters at end of string then remove OR
// check special characters in beginning of the string then remove
// insert every word directly to table db
word = word.replaceAll("([\\W]+$)|(^[\\W]+)", "");
preparedStatement.setString(1, path1);
preparedStatement.setString(2, word);
System.out.println("Token: " +word);
preparedStatement.executeUpdate();
}
}
preparedStatement.close();
}

This is the PDFBox ExtractText command line utility, which can detect rotations since 2.0.13 (PDFBOX-4371). (That release had a bug with type 3 fonts, which was fixed (PDFBOX-4390) in the repository and in this code, and is in 2.0.14). Later code may have been improved since then. The current 2.0.* source can be found here.
To extract text from rotated files, use the "rotationMagic" setting. This setting first detects the angle of every glyph, collects these angles (AngleCollector), and in a second pass it does an extraction for every angle while discarding the rest (FilteredTextStripper). The order of extraction is by angle, which may or may not make sense if there are several different angles in a page.
The PDF is modified while extracting, so don't use this on documents you are saving.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.pdfbox.tools;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.pdfbox.cos.COSArray;
import org.apache.pdfbox.cos.COSName;
import org.apache.pdfbox.io.IOUtils;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.pdmodel.PDDocumentCatalog;
import org.apache.pdfbox.pdmodel.PDDocumentNameDictionary;
import org.apache.pdfbox.pdmodel.PDEmbeddedFilesNameTreeNode;
import org.apache.pdfbox.pdmodel.PDPage;
import org.apache.pdfbox.pdmodel.PDPageContentStream;
import org.apache.pdfbox.pdmodel.common.filespecification.PDComplexFileSpecification;
import org.apache.pdfbox.pdmodel.common.filespecification.PDEmbeddedFile;
import org.apache.pdfbox.pdmodel.encryption.AccessPermission;
import org.apache.pdfbox.text.PDFTextStripper;
import org.apache.pdfbox.text.TextPosition;
import org.apache.pdfbox.util.Matrix;
/**
* This is the main program that simply parses the pdf document and transforms it
* into text.
*
* #author Ben Litchfield
* #author Tilman Hausherr
*/
public final class ExtractText
{
private static final Log LOG = LogFactory.getLog(ExtractText.class);
private static final String PASSWORD = "-password";
private static final String ENCODING = "-encoding";
private static final String CONSOLE = "-console";
private static final String START_PAGE = "-startPage";
private static final String END_PAGE = "-endPage";
private static final String SORT = "-sort";
private static final String IGNORE_BEADS = "-ignoreBeads";
private static final String DEBUG = "-debug";
private static final String HTML = "-html";
private static final String ALWAYSNEXT = "-alwaysNext";
private static final String ROTATION_MAGIC = "-rotationMagic";
private static final String STD_ENCODING = "UTF-8";
/*
* debug flag
*/
private boolean debug = false;
/**
* private constructor.
*/
private ExtractText()
{
//static class
}
/**
* Infamous main method.
*
* #param args Command line arguments, should be one and a reference to a file.
*
* #throws IOException if there is an error reading the document or extracting the text.
*/
public static void main( String[] args ) throws IOException
{
// suppress the Dock icon on OS X
System.setProperty("apple.awt.UIElement", "true");
ExtractText extractor = new ExtractText();
extractor.startExtraction(args);
}
/**
* Starts the text extraction.
*
* #param args the commandline arguments.
* #throws IOException if there is an error reading the document or extracting the text.
*/
public void startExtraction( String[] args ) throws IOException
{
boolean toConsole = false;
boolean toHTML = false;
boolean sort = false;
boolean separateBeads = true;
boolean alwaysNext = false;
boolean rotationMagic = false;
String password = "";
String encoding = STD_ENCODING;
String pdfFile = null;
String outputFile = null;
// Defaults to text files
String ext = ".txt";
int startPage = 1;
int endPage = Integer.MAX_VALUE;
for( int i=0; i<args.length; i++ )
{
if( args[i].equals( PASSWORD ) )
{
i++;
if( i >= args.length )
{
usage();
}
password = args[i];
}
else if( args[i].equals( ENCODING ) )
{
i++;
if( i >= args.length )
{
usage();
}
encoding = args[i];
}
else if( args[i].equals( START_PAGE ) )
{
i++;
if( i >= args.length )
{
usage();
}
startPage = Integer.parseInt( args[i] );
}
else if( args[i].equals( HTML ) )
{
toHTML = true;
ext = ".html";
}
else if( args[i].equals( SORT ) )
{
sort = true;
}
else if( args[i].equals( IGNORE_BEADS ) )
{
separateBeads = false;
}
else if (args[i].equals(ALWAYSNEXT))
{
alwaysNext = true;
}
else if (args[i].equals(ROTATION_MAGIC))
{
rotationMagic = true;
}
else if( args[i].equals( DEBUG ) )
{
debug = true;
}
else if( args[i].equals( END_PAGE ) )
{
i++;
if( i >= args.length )
{
usage();
}
endPage = Integer.parseInt( args[i] );
}
else if( args[i].equals( CONSOLE ) )
{
toConsole = true;
}
else
{
if( pdfFile == null )
{
pdfFile = args[i];
}
else
{
outputFile = args[i];
}
}
}
if( pdfFile == null )
{
usage();
}
else
{
Writer output = null;
PDDocument document = null;
try
{
long startTime = startProcessing("Loading PDF "+pdfFile);
if( outputFile == null && pdfFile.length() >4 )
{
outputFile = new File( pdfFile.substring( 0, pdfFile.length() -4 ) + ext ).getAbsolutePath();
}
document = PDDocument.load(new File( pdfFile ), password);
AccessPermission ap = document.getCurrentAccessPermission();
if( ! ap.canExtractContent() )
{
throw new IOException( "You do not have permission to extract text" );
}
stopProcessing("Time for loading: ", startTime);
if( toConsole )
{
output = new OutputStreamWriter( System.out, encoding );
}
else
{
if (toHTML && !STD_ENCODING.equals(encoding))
{
encoding = STD_ENCODING;
System.out.println("The encoding parameter is ignored when writing html output.");
}
output = new OutputStreamWriter( new FileOutputStream( outputFile ), encoding );
}
startTime = startProcessing("Starting text extraction");
if (debug)
{
System.err.println("Writing to " + outputFile);
}
PDFTextStripper stripper;
if(toHTML)
{
// HTML stripper can't work page by page because of startDocument() callback
stripper = new PDFText2HTML();
stripper.setSortByPosition(sort);
stripper.setShouldSeparateByBeads(separateBeads);
stripper.setStartPage(startPage);
stripper.setEndPage(endPage);
// Extract text for main document:
stripper.writeText(document, output);
}
else
{
if (rotationMagic)
{
stripper = new FilteredTextStripper();
}
else
{
stripper = new PDFTextStripper();
}
stripper.setSortByPosition(sort);
stripper.setShouldSeparateByBeads(separateBeads);
// Extract text for main document:
extractPages(startPage, Math.min(endPage, document.getNumberOfPages()),
stripper, document, output, rotationMagic, alwaysNext);
}
// ... also for any embedded PDFs:
PDDocumentCatalog catalog = document.getDocumentCatalog();
PDDocumentNameDictionary names = catalog.getNames();
if (names != null)
{
PDEmbeddedFilesNameTreeNode embeddedFiles = names.getEmbeddedFiles();
if (embeddedFiles != null)
{
Map<String, PDComplexFileSpecification> embeddedFileNames = embeddedFiles.getNames();
if (embeddedFileNames != null)
{
for (Map.Entry<String, PDComplexFileSpecification> ent : embeddedFileNames.entrySet())
{
if (debug)
{
System.err.println("Processing embedded file " + ent.getKey() + ":");
}
PDComplexFileSpecification spec = ent.getValue();
PDEmbeddedFile file = spec.getEmbeddedFile();
if (file != null && "application/pdf".equals(file.getSubtype()))
{
if (debug)
{
System.err.println(" is PDF (size=" + file.getSize() + ")");
}
InputStream fis = file.createInputStream();
PDDocument subDoc = null;
try
{
subDoc = PDDocument.load(fis);
if (toHTML)
{
// will not really work because of HTML header + footer
stripper.writeText( subDoc, output );
}
else
{
extractPages(1, subDoc.getNumberOfPages(),
stripper, subDoc, output, rotationMagic, alwaysNext);
}
}
finally
{
fis.close();
IOUtils.closeQuietly(subDoc);
}
}
}
}
}
}
stopProcessing("Time for extraction: ", startTime);
}
finally
{
IOUtils.closeQuietly(output);
IOUtils.closeQuietly(document);
}
}
}
private void extractPages(int startPage, int endPage,
PDFTextStripper stripper, PDDocument document, Writer output,
boolean rotationMagic, boolean alwaysNext) throws IOException
{
for (int p = startPage; p <= endPage; ++p)
{
stripper.setStartPage(p);
stripper.setEndPage(p);
try
{
if (rotationMagic)
{
PDPage page = document.getPage(p - 1);
int rotation = page.getRotation();
page.setRotation(0);
AngleCollector angleCollector = new AngleCollector();
angleCollector.setStartPage(p);
angleCollector.setEndPage(p);
angleCollector.writeText(document, new NullWriter());
// rotation magic
for (int angle : angleCollector.getAngles())
{
// prepend a transformation
// (we could skip these parts for angle 0, but it doesn't matter much)
PDPageContentStream cs = new PDPageContentStream(document, page,
PDPageContentStream.AppendMode.PREPEND, false);
cs.transform(Matrix.getRotateInstance(-Math.toRadians(angle), 0, 0));
cs.close();
stripper.writeText(document, output);
// remove prepended transformation
((COSArray) page.getCOSObject().getItem(COSName.CONTENTS)).remove(0);
}
page.setRotation(rotation);
}
else
{
stripper.writeText(document, output);
}
}
catch (IOException ex)
{
if (!alwaysNext)
{
throw ex;
}
LOG.error("Failed to process page " + p, ex);
}
}
}
private long startProcessing(String message)
{
if (debug)
{
System.err.println(message);
}
return System.currentTimeMillis();
}
private void stopProcessing(String message, long startTime)
{
if (debug)
{
long stopTime = System.currentTimeMillis();
float elapsedTime = ((float)(stopTime - startTime))/1000;
System.err.println(message + elapsedTime + " seconds");
}
}
/**
* This will print the usage requirements and exit.
*/
private static void usage()
{
String message = "Usage: java -jar pdfbox-app-x.y.z.jar ExtractText [options] <inputfile> [output-text-file]\n"
+ "\nOptions:\n"
+ " -password <password> : Password to decrypt document\n"
+ " -encoding <output encoding> : UTF-8 (default) or ISO-8859-1, UTF-16BE,\n"
+ " UTF-16LE, etc.\n"
+ " -console : Send text to console instead of file\n"
+ " -html : Output in HTML format instead of raw text\n"
+ " -sort : Sort the text before writing\n"
+ " -ignoreBeads : Disables the separation by beads\n"
+ " -debug : Enables debug output about the time consumption\n"
+ " of every stage\n"
+ " -alwaysNext : Process next page (if applicable) despite\n"
+ " IOException (ignored when -html)\n"
+ " -rotationMagic : Analyze each page for rotated/skewed text,\n"
+ " rotate to 0° and extract separately\n"
+ " (slower, and ignored when -html)\n"
+ " -startPage <number> : The first page to start extraction (1 based)\n"
+ " -endPage <number> : The last page to extract (1 based, inclusive)\n"
+ " <inputfile> : The PDF document to use\n"
+ " [output-text-file] : The file to write the text to";
System.err.println(message);
System.exit( 1 );
}
}
/**
* Collect all angles while doing text extraction. Angles are in degrees and rounded to the closest
* integer (to avoid slight differences from floating point arithmethic resulting in similarly
* angled glyphs being treated separately). This class must be constructed for each page so that the
* angle set is initialized.
*/
class AngleCollector extends PDFTextStripper
{
private final Set<Integer> angles = new TreeSet<Integer>();
AngleCollector() throws IOException
{
}
Set<Integer> getAngles()
{
return angles;
}
#Override
protected void processTextPosition(TextPosition text)
{
Matrix m = text.getTextMatrix();
m.concatenate(text.getFont().getFontMatrix());
int angle = (int) Math.round(Math.toDegrees(Math.atan2(m.getShearY(), m.getScaleY())));
angle = (angle + 360) % 360;
angles.add(angle);
}
}
/**
* TextStripper that only processes glyphs that have angle 0.
*/
class FilteredTextStripper extends PDFTextStripper
{
FilteredTextStripper() throws IOException
{
}
#Override
protected void processTextPosition(TextPosition text)
{
Matrix m = text.getTextMatrix();
m.concatenate(text.getFont().getFontMatrix());
int angle = (int) Math.round(Math.toDegrees(Math.atan2(m.getShearY(), m.getScaleY())));
if (angle == 0)
{
super.processTextPosition(text);
}
}
}
/**
* Dummy output.
*/
class NullWriter extends Writer
{
#Override
public void write(char[] cbuf, int off, int len) throws IOException
{
// do nothing
}
#Override
public void flush() throws IOException
{
// do nothing
}
#Override
public void close() throws IOException
{
// do nothing
}
}

java- rotated file extraction?

I am using PDFBox to do a simple extraction of words from a PDF file. Then it inserts those words to a table in database. From what I have tested, a 90 degrees clockwise rotated text in PDF will gives gibberish result when I tried to extract the words.
For example, database in the file will yield atabase and also database itself as two different words. Obviously, atabase does not exist in the PDF file.
I tried converting the original file to be rotated upright and do the extraction and it works perfectly as expected. I understand this could be a limitation of the PDFBox itself.
So, in the case of someone trying to index a rotated PDF file, is there a way to tackle this?
Code snippet ( just for reference) :
String lines[] = text.split("\\r?\\n");
for (String line : lines) {
String[] words = line.split(" ");
System.out.println("Line: " + line);
preparedStatement = con1.prepareStatement(sql);
int i=0;
for (String word : words) {
// check if one or more special characters at end of string then remove OR
// check special characters in beginning of the string then remove
// insert every word directly to table db
word = word.replaceAll("([\\W]+$)|(^[\\W]+)", "");
preparedStatement.setString(1, path1);
preparedStatement.setString(2, word);
System.out.println("Token: " +word);
preparedStatement.executeUpdate();
}
}
preparedStatement.close();
}

This is the PDFBox ExtractText command line utility, which can detect rotations since 2.0.13 (PDFBOX-4371). (That release had a bug with type 3 fonts, which was fixed (PDFBOX-4390) in the repository and in this code, and is in 2.0.14). Later code may have been improved since then. The current 2.0.* source can be found here.
To extract text from rotated files, use the "rotationMagic" setting. This setting first detects the angle of every glyph, collects these angles (AngleCollector), and in a second pass it does an extraction for every angle while discarding the rest (FilteredTextStripper). The order of extraction is by angle, which may or may not make sense if there are several different angles in a page.
The PDF is modified while extracting, so don't use this on documents you are saving.
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.pdfbox.tools;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.pdfbox.cos.COSArray;
import org.apache.pdfbox.cos.COSName;
import org.apache.pdfbox.io.IOUtils;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.pdmodel.PDDocumentCatalog;
import org.apache.pdfbox.pdmodel.PDDocumentNameDictionary;
import org.apache.pdfbox.pdmodel.PDEmbeddedFilesNameTreeNode;
import org.apache.pdfbox.pdmodel.PDPage;
import org.apache.pdfbox.pdmodel.PDPageContentStream;
import org.apache.pdfbox.pdmodel.common.filespecification.PDComplexFileSpecification;
import org.apache.pdfbox.pdmodel.common.filespecification.PDEmbeddedFile;
import org.apache.pdfbox.pdmodel.encryption.AccessPermission;
import org.apache.pdfbox.text.PDFTextStripper;
import org.apache.pdfbox.text.TextPosition;
import org.apache.pdfbox.util.Matrix;
/**
* This is the main program that simply parses the pdf document and transforms it
* into text.
*
* #author Ben Litchfield
* #author Tilman Hausherr
*/
public final class ExtractText
{
private static final Log LOG = LogFactory.getLog(ExtractText.class);
private static final String PASSWORD = "-password";
private static final String ENCODING = "-encoding";
private static final String CONSOLE = "-console";
private static final String START_PAGE = "-startPage";
private static final String END_PAGE = "-endPage";
private static final String SORT = "-sort";
private static final String IGNORE_BEADS = "-ignoreBeads";
private static final String DEBUG = "-debug";
private static final String HTML = "-html";
private static final String ALWAYSNEXT = "-alwaysNext";
private static final String ROTATION_MAGIC = "-rotationMagic";
private static final String STD_ENCODING = "UTF-8";
/*
* debug flag
*/
private boolean debug = false;
/**
* private constructor.
*/
private ExtractText()
{
//static class
}
/**
* Infamous main method.
*
* #param args Command line arguments, should be one and a reference to a file.
*
* #throws IOException if there is an error reading the document or extracting the text.
*/
public static void main( String[] args ) throws IOException
{
// suppress the Dock icon on OS X
System.setProperty("apple.awt.UIElement", "true");
ExtractText extractor = new ExtractText();
extractor.startExtraction(args);
}
/**
* Starts the text extraction.
*
* #param args the commandline arguments.
* #throws IOException if there is an error reading the document or extracting the text.
*/
public void startExtraction( String[] args ) throws IOException
{
boolean toConsole = false;
boolean toHTML = false;
boolean sort = false;
boolean separateBeads = true;
boolean alwaysNext = false;
boolean rotationMagic = false;
String password = "";
String encoding = STD_ENCODING;
String pdfFile = null;
String outputFile = null;
// Defaults to text files
String ext = ".txt";
int startPage = 1;
int endPage = Integer.MAX_VALUE;
for( int i=0; i<args.length; i++ )
{
if( args[i].equals( PASSWORD ) )
{
i++;
if( i >= args.length )
{
usage();
}
password = args[i];
}
else if( args[i].equals( ENCODING ) )
{
i++;
if( i >= args.length )
{
usage();
}
encoding = args[i];
}
else if( args[i].equals( START_PAGE ) )
{
i++;
if( i >= args.length )
{
usage();
}
startPage = Integer.parseInt( args[i] );
}
else if( args[i].equals( HTML ) )
{
toHTML = true;
ext = ".html";
}
else if( args[i].equals( SORT ) )
{
sort = true;
}
else if( args[i].equals( IGNORE_BEADS ) )
{
separateBeads = false;
}
else if (args[i].equals(ALWAYSNEXT))
{
alwaysNext = true;
}
else if (args[i].equals(ROTATION_MAGIC))
{
rotationMagic = true;
}
else if( args[i].equals( DEBUG ) )
{
debug = true;
}
else if( args[i].equals( END_PAGE ) )
{
i++;
if( i >= args.length )
{
usage();
}
endPage = Integer.parseInt( args[i] );
}
else if( args[i].equals( CONSOLE ) )
{
toConsole = true;
}
else
{
if( pdfFile == null )
{
pdfFile = args[i];
}
else
{
outputFile = args[i];
}
}
}
if( pdfFile == null )
{
usage();
}
else
{
Writer output = null;
PDDocument document = null;
try
{
long startTime = startProcessing("Loading PDF "+pdfFile);
if( outputFile == null && pdfFile.length() >4 )
{
outputFile = new File( pdfFile.substring( 0, pdfFile.length() -4 ) + ext ).getAbsolutePath();
}
document = PDDocument.load(new File( pdfFile ), password);
AccessPermission ap = document.getCurrentAccessPermission();
if( ! ap.canExtractContent() )
{
throw new IOException( "You do not have permission to extract text" );
}
stopProcessing("Time for loading: ", startTime);
if( toConsole )
{
output = new OutputStreamWriter( System.out, encoding );
}
else
{
if (toHTML && !STD_ENCODING.equals(encoding))
{
encoding = STD_ENCODING;
System.out.println("The encoding parameter is ignored when writing html output.");
}
output = new OutputStreamWriter( new FileOutputStream( outputFile ), encoding );
}
startTime = startProcessing("Starting text extraction");
if (debug)
{
System.err.println("Writing to " + outputFile);
}
PDFTextStripper stripper;
if(toHTML)
{
// HTML stripper can't work page by page because of startDocument() callback
stripper = new PDFText2HTML();
stripper.setSortByPosition(sort);
stripper.setShouldSeparateByBeads(separateBeads);
stripper.setStartPage(startPage);
stripper.setEndPage(endPage);
// Extract text for main document:
stripper.writeText(document, output);
}
else
{
if (rotationMagic)
{
stripper = new FilteredTextStripper();
}
else
{
stripper = new PDFTextStripper();
}
stripper.setSortByPosition(sort);
stripper.setShouldSeparateByBeads(separateBeads);
// Extract text for main document:
extractPages(startPage, Math.min(endPage, document.getNumberOfPages()),
stripper, document, output, rotationMagic, alwaysNext);
}
// ... also for any embedded PDFs:
PDDocumentCatalog catalog = document.getDocumentCatalog();
PDDocumentNameDictionary names = catalog.getNames();
if (names != null)
{
PDEmbeddedFilesNameTreeNode embeddedFiles = names.getEmbeddedFiles();
if (embeddedFiles != null)
{
Map<String, PDComplexFileSpecification> embeddedFileNames = embeddedFiles.getNames();
if (embeddedFileNames != null)
{
for (Map.Entry<String, PDComplexFileSpecification> ent : embeddedFileNames.entrySet())
{
if (debug)
{
System.err.println("Processing embedded file " + ent.getKey() + ":");
}
PDComplexFileSpecification spec = ent.getValue();
PDEmbeddedFile file = spec.getEmbeddedFile();
if (file != null && "application/pdf".equals(file.getSubtype()))
{
if (debug)
{
System.err.println(" is PDF (size=" + file.getSize() + ")");
}
InputStream fis = file.createInputStream();
PDDocument subDoc = null;
try
{
subDoc = PDDocument.load(fis);
if (toHTML)
{
// will not really work because of HTML header + footer
stripper.writeText( subDoc, output );
}
else
{
extractPages(1, subDoc.getNumberOfPages(),
stripper, subDoc, output, rotationMagic, alwaysNext);
}
}
finally
{
fis.close();
IOUtils.closeQuietly(subDoc);
}
}
}
}
}
}
stopProcessing("Time for extraction: ", startTime);
}
finally
{
IOUtils.closeQuietly(output);
IOUtils.closeQuietly(document);
}
}
}
private void extractPages(int startPage, int endPage,
PDFTextStripper stripper, PDDocument document, Writer output,
boolean rotationMagic, boolean alwaysNext) throws IOException
{
for (int p = startPage; p <= endPage; ++p)
{
stripper.setStartPage(p);
stripper.setEndPage(p);
try
{
if (rotationMagic)
{
PDPage page = document.getPage(p - 1);
int rotation = page.getRotation();
page.setRotation(0);
AngleCollector angleCollector = new AngleCollector();
angleCollector.setStartPage(p);
angleCollector.setEndPage(p);
angleCollector.writeText(document, new NullWriter());
// rotation magic
for (int angle : angleCollector.getAngles())
{
// prepend a transformation
// (we could skip these parts for angle 0, but it doesn't matter much)
PDPageContentStream cs = new PDPageContentStream(document, page,
PDPageContentStream.AppendMode.PREPEND, false);
cs.transform(Matrix.getRotateInstance(-Math.toRadians(angle), 0, 0));
cs.close();
stripper.writeText(document, output);
// remove prepended transformation
((COSArray) page.getCOSObject().getItem(COSName.CONTENTS)).remove(0);
}
page.setRotation(rotation);
}
else
{
stripper.writeText(document, output);
}
}
catch (IOException ex)
{
if (!alwaysNext)
{
throw ex;
}
LOG.error("Failed to process page " + p, ex);
}
}
}
private long startProcessing(String message)
{
if (debug)
{
System.err.println(message);
}
return System.currentTimeMillis();
}
private void stopProcessing(String message, long startTime)
{
if (debug)
{
long stopTime = System.currentTimeMillis();
float elapsedTime = ((float)(stopTime - startTime))/1000;
System.err.println(message + elapsedTime + " seconds");
}
}
/**
* This will print the usage requirements and exit.
*/
private static void usage()
{
String message = "Usage: java -jar pdfbox-app-x.y.z.jar ExtractText [options] <inputfile> [output-text-file]\n"
+ "\nOptions:\n"
+ " -password <password> : Password to decrypt document\n"
+ " -encoding <output encoding> : UTF-8 (default) or ISO-8859-1, UTF-16BE,\n"
+ " UTF-16LE, etc.\n"
+ " -console : Send text to console instead of file\n"
+ " -html : Output in HTML format instead of raw text\n"
+ " -sort : Sort the text before writing\n"
+ " -ignoreBeads : Disables the separation by beads\n"
+ " -debug : Enables debug output about the time consumption\n"
+ " of every stage\n"
+ " -alwaysNext : Process next page (if applicable) despite\n"
+ " IOException (ignored when -html)\n"
+ " -rotationMagic : Analyze each page for rotated/skewed text,\n"
+ " rotate to 0° and extract separately\n"
+ " (slower, and ignored when -html)\n"
+ " -startPage <number> : The first page to start extraction (1 based)\n"
+ " -endPage <number> : The last page to extract (1 based, inclusive)\n"
+ " <inputfile> : The PDF document to use\n"
+ " [output-text-file] : The file to write the text to";
System.err.println(message);
System.exit( 1 );
}
}
/**
* Collect all angles while doing text extraction. Angles are in degrees and rounded to the closest
* integer (to avoid slight differences from floating point arithmethic resulting in similarly
* angled glyphs being treated separately). This class must be constructed for each page so that the
* angle set is initialized.
*/
class AngleCollector extends PDFTextStripper
{
private final Set<Integer> angles = new TreeSet<Integer>();
AngleCollector() throws IOException
{
}
Set<Integer> getAngles()
{
return angles;
}
#Override
protected void processTextPosition(TextPosition text)
{
Matrix m = text.getTextMatrix();
m.concatenate(text.getFont().getFontMatrix());
int angle = (int) Math.round(Math.toDegrees(Math.atan2(m.getShearY(), m.getScaleY())));
angle = (angle + 360) % 360;
angles.add(angle);
}
}
/**
* TextStripper that only processes glyphs that have angle 0.
*/
class FilteredTextStripper extends PDFTextStripper
{
FilteredTextStripper() throws IOException
{
}
#Override
protected void processTextPosition(TextPosition text)
{
Matrix m = text.getTextMatrix();
m.concatenate(text.getFont().getFontMatrix());
int angle = (int) Math.round(Math.toDegrees(Math.atan2(m.getShearY(), m.getScaleY())));
if (angle == 0)
{
super.processTextPosition(text);
}
}
}
/**
* Dummy output.
*/
class NullWriter extends Writer
{
#Override
public void write(char[] cbuf, int off, int len) throws IOException
{
// do nothing
}
#Override
public void flush() throws IOException
{
// do nothing
}
#Override
public void close() throws IOException
{
// do nothing
}
}

Why does Java app crash after processing x number of files?

I'm only asking this as a last resort. I'm stumped.
I have written a small app which performs very simple image processing. It is made using JDK 1.8, written in Netbeans 8.0.1 and is running on Debian Linux.
The application captures a large number of individual frames at a certain framerate, set by the user, by calling the 'streamer' webcam program via a process builder. Once it has begun capturing, it begins to translate the frames into RGB values and checks whether or not any pixels are above a user defined threshold. If no pixels exceed this threshold, it simply deletes the frame. If any pixels do exceed it, it moves the frame to a different folder for user inspection. This all works fine. It keeps up with even relatively high framerates and selects appropriate frames as expected. However, when the number of frames processed reaches around 1500 (or fewer for lower framerates), the program is freezing. I've tried issuing the commands to streamer manually at the command line, and it seems perfectly capable of producing as many as required, so I have to assume the issue is with my coding. The images are only small (320x240). Am I somehow maxxing out the available memory (I am getting no errors, just freezing).
The purpose of this program is to detect cosmic ray impacts on a CMOS sensor, part of a friend's dissertation. If I can't get this working reliably, the poor kid's going to have to go through the footage manually!
The code is attached below. Apologies for the length of the code, but as all of it is fairly crucial, I didn't want to omit anything.
package cosmicraysiii;
import java.io.File;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.swing.JOptionPane;
public class CosmicRaysIII {
public static int dark = 0;//Dark current determined by averaging
public static int tol = 0;//Tolerance set by user
public static int frames = 0;//Total number of frames set by user
public static int runs = 0;
public static int rate = 0;//Framerate set by user
public static int time = 0;//Total time calculated
public static String zeros = "";
public static int ready = 0;
public static void main(String[] args) throws IOException, InterruptedException {
//Get directory ID from user
String id = JOptionPane.showInputDialog("Enter a folder name for detections (no ., in name):");
//Get dark current
Dark d = new Dark();
dark = d.getCurrent();
//Get tolerance from user, will be added to dark current.
String t = JOptionPane.showInputDialog("Dark Current = " + dark + "/255\n"
+ "Enter a tolerance (integer values only).\n "
+ "This will be added to the dark current:");
tol = Integer.parseInt(t) + dark;
//Get number of frames from user
String fs = JOptionPane.showInputDialog("Enter the total number of frames required (Mulitples of 500 only):");
frames = Integer.parseInt(fs);
runs = frames / 500;
//Get framerate from user
String r = JOptionPane.showInputDialog("Enter the framerate required:");
rate = Integer.parseInt(r);
//Determine duration
time = (int) Math.round(frames / rate);
//Provide summary for user and request permission to continue
int secs = time % 60;
int mins = (time - secs) / 60;
int hrs = (mins - (mins % 60)) / 60;
if (hrs >= 1) {
mins = mins % 60;
}
String theMessage = "The following parameters have been set:\n"
+ "Tolerance (including dark current): " + tol + "\n"
+ "Frames: " + frames + "\n"
+ "Frame rate: " + rate + " fps\n"
+ "Total capture time: " + time + " sec\n"
+ " " + hrs + " h " + mins + " m " + secs + " s\n"
+ "\n"
+ "Would you like to proceed?";
int result = JOptionPane.showConfirmDialog(null, theMessage, "Continue?", JOptionPane.OK_CANCEL_OPTION);
if (result == 2) {
System.exit(0);
}
//Create directory for data acquisition
ProcessBuilder pb1 = new ProcessBuilder("mkdir", "data");
pb1.start();
//Establish array of filenames
String[] filenames = new String[frames];
//Fill filenames array with filenames
//Taking into consideration that the filename
//will have a varying number of zeros appended
//before the frame number, dependent on the
//order of the frame number
for (int i = 0; i < frames; i++) {
if (i < 10) {
zeros = "00000000";
} else if (i >= 10 && i < 100) {
zeros = "0000000";
} else if (i >= 100 && i < 1000) {
zeros = "000000";
} else if (i >= 1000 && i < 10000) {
zeros = "00000";
} else if (i >= 10000 && i < 100000) {
zeros = "0000";
} else if (i >= 100000 && i < 1000000) {
zeros = "000";
} else if (i >= 1000000 && i < 10000000) {
zeros = "00";
} else if (i >= 10000000 && i < 100000000) {
zeros = "0";
} else {
zeros = "";
}
filenames[i] = "./data/frame" + zeros + i + ".ppm";
}
//Begin data acquisition
new Thread(new Runnable() {
public void run() {
try {
//Capture images
ProcessBuilder pb2 = new ProcessBuilder("streamer", "-t", Integer.toString(frames), "-r", Integer.toString(rate), "-p", "0", "-o", "./data/frame000000000.ppm");
Process p = pb2.start();
p.waitFor();
ready = 1;
} catch (IOException | InterruptedException ex) {
Logger.getLogger(CosmicRaysIII.class.getName()).log(Level.SEVERE, null, ex);
}
}
}).start();
//Sleep to allow some image capture to prevent thread disordering
Thread.sleep(3000);
//Check array size
System.out.println("Array size: " + filenames.length);
//Conduct image analysis
new Thread(new Runnable() {
public void run() {
int done = 0;
int donea = 0;
while (ready == 0) {
for (int i = 0; i < frames; i++) {
File f = new File(filenames[i]);
if (f.exists() && !filenames[i].equals("")) {//Check file still exists
try {
//Perform analysis steps
Analysis a = new Analysis();
//STEP 1: Convert file from P6 to P3
String newfile = a.convert(filenames[i], zeros, i);
//STEP 2: Read file
a.read(newfile, tol, i, id);
filenames[i] = "";
done++;
} catch (IOException ex) {
Logger.getLogger(CosmicRaysIII.class.getName()).log(Level.SEVERE, null, ex);
}
}
if (done > donea) {
System.out.println(done + " files processed");
donea = done;
}
}
}
}
}).start();
}
}
Then the Analyse.java class is as follows:
package cosmicraysiii;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
public class Analysis {
public String convert(String ofile, String zeros, int number) throws IOException {
//Create new file name
String nfile = "./proc/frame" + zeros + number + ".ppm";
//Ensure process directory exists
ProcessBuilder mkdir = new ProcessBuilder("mkdir", "proc");
mkdir.start();
//Convert file to P3 PPM (RGB format) and move to process folder
ProcessBuilder convert = new ProcessBuilder("convert", ofile, "-compress", "none", nfile);
convert.start();
//Delete original file
ProcessBuilder del = new ProcessBuilder("sudo", "rm", ofile);
del.start();
//Return new filename
return nfile;
}
public void read(String filename, int tol, int ix, String id) throws FileNotFoundException, IOException {
int move = 0;
//Make directory for hits
ProcessBuilder mkdir = new ProcessBuilder("mkdir", "hits" + id);
mkdir.start();
//Open reader to read file
File f = new File(filename);
if (f.exists()) {
BufferedReader br = new BufferedReader(new FileReader(filename));
String line;
//To eliminate header
int x = 0;
//Iterate through text to find abnormal pixels
while ((line = br.readLine()) != null) {
x++;
String[] pixrgb = line.split("\\ ");
//Iterate through pixels on each line
for (int i = 0; i < pixrgb.length; i++) {
if (x >= 4) {//Eliminate header
//Check each pixel value
try {
int pixval = Integer.parseInt(pixrgb[i]);
if (pixval > tol) {
move = 1;
break;
}
} catch (NumberFormatException ne) {
}
}
}
}
if (move == 1) {
//Move file to hits folder
ProcessBuilder pb3 = new ProcessBuilder("sudo", "cp", filename, "./hits" + id + "/detection" + ix + ".ppm");
pb3.start();
//Delete original file
ProcessBuilder del = new ProcessBuilder("sudo", "rm", filename);
del.start();
}
}
//Delete original file
ProcessBuilder del = new ProcessBuilder("sudo", "rm", filename);
del.start();
}
}
I appreciate this is quite a lengthly chunk of code to be posting. Really appreciate any help that can be given.
G

OK I have managed to solve this by completely overhauling the analysis process. Firstly, rather than converting the image into a P3 .ppm file, I now examine the pixels directly from the image using BufferedReader. Then, I stopped looping through the file list repeatedly. Instead, the loop which calls Analysis() just runs through the list of filenames once. If it encounters a file which does not yet exist, it does Thread.sleep(500) and then tries again. I have now successfully run a batch of 50,000 frames without incident, and there is now much less of a drain on memory thanks to the improved process. Just thought I should place this answer up here in case anyone comes across it. I may post code if anyone wants it.

Exception in thread "main" java.lang.RuntimeException: Stub , viewed all thread but of no use

This is a face recognition class which read the faces from data folder but when i run this code i came up with the error given below. All the Related Threads couldn't solve my problem. any help will be appreciated.
public class FaceRecognition {
/** the logger */
private static final Logger LOGGER = Logger.getLogger(FaceRecognition.class);
//JavaLoggingClassName.loginfo();
/** the number of training faces */
private int nTrainFaces = 0;
private int width = 320;
private int height = 240;
/** the training face image array */
IplImage[] trainingFaceImgArr = null;//IplImage.create(width, height, IPL_DEPTH_8U, 4);
/** the test face image array */
IplImage[] testFaceImgArr= null;
//IplImage image = IplImage.create(width, height, IPL_DEPTH_8U, 4);
// Bitmap mBitmap;
//IplImage image = IplImage.create(width, height, IPL_DEPTH_8U, 4);
//private Bitmap mBitmap;
/** the person number array **/
CvMat personNumTruthMat;
/** the number of persons **/
int nPersons;
/** the person names */
final List personNames = new ArrayList<>();
/** the number of eigenvalues */
int nEigens = 0;
/** eigenvectors */
IplImage[] eigenVectArr;
/** eigenvalues */
CvMat eigenValMat;
/** the average image */
IplImage pAvgTrainImg;
/** the projected training faces */
CvMat projectedTrainFaceMat;
/** Constructs a new FaceRecognition instance. */
public FaceRecognition() {
}
//JavaLoggingClassName.loginfo()
/** Trains from the data in the given training text index file, and store the trained data into the file 'data/facedata.xml'.
*
* #param trainingFileName the given training text index file
*/
public void learn(final String trainingFileName) {
int i;
// load training data
LOGGER.info("===========================================");
//IplImage
LOGGER.info("Loading the training images in " + trainingFileName);
//IplImage image = IplImage.create(width, height, IPL_DEPTH_8U, 4);
//mBitmap.copyPixelsFromBuffer(image.getByteBuffer());
try
{
trainingFaceImgArr /*mBitmap*/ = loadFaceImgArray(trainingFileName);
} catch (Exception e)
{
Log.i("ERROR", "ERROR in Code: " + e.toString());
e.printStackTrace();
}
nTrainFaces = trainingFaceImgArr.length;
LOGGER.info("Got " + nTrainFaces + " training images");
if (nTrainFaces < 3) {
LOGGER.error("Need 3 or more training faces\n"
+ "Input file contains only " + nTrainFaces);
return;
}
LOGGER.info("created projectedTrainFaceMat with " + nTrainFaces + " (nTrainFaces) rows and " + nEigens + " (nEigens) columns");
if (nTrainFaces < 5) {
LOGGER.info("projectedTrainFaceMat contents:\n" + oneChannelCvMatToString(projectedTrainFaceMat));
}
/* #param szFileTest the index file of test images
} catch (IOException ex) {
throw new RuntimeException(ex);
}
LOGGER.info("Data loaded from '" + filename + "': (" + nFaces + " images of " + nPersons + " people).");
final StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append("People: ");
if (nPersons > 0) {
stringBuilder.append("<").append(personNames.get(0)).append(">");
}
for (i = 1; i < nPersons && i < personNames.size(); i++) {
stringBuilder.append(", <").append(personNames.get(i)).append(">");
}
LOGGER.info(stringBuilder.toString());
return faceImgArr;
}
/** Does the Principal Component Analysis, finding the average image and the eigenfaces that represent any image in the given dataset. */
private void doPCA() {
int i;
CvTermCriteria calcLimit;
CvSize faceImgSize = new CvSize();
// set the number of eigenvalues to use
nEigens = nTrainFaces - 1;
LOGGER.info("allocating images for principal component analysis, using " + nEigens + (nEigens == 1 ? " eigenvalue" : " eigenvalues"));
// allocate the eigenvector images
faceImgSize.width(trainingFaceImgArr[0].width());
faceImgSize.height(trainingFaceImgArr[0].height());
eigenVectArr = new IplImage[nEigens];
for (i = 0; i < nEigens; i++) {
eigenVectArr[i] = cvCreateImage(
faceImgSize, // size
IPL_DEPTH_32F, // depth
1); // channels
}
// allocate the eigenvalue array
eigenValMat = cvCreateMat(
1, // rows
nEigens, // cols
CV_32FC1); // type, 32-bit float, 1 channel
// allocate the averaged image
pAvgTrainImg = cvCreateImage(
faceImgSize, // size
IPL_DEPTH_32F, // depth
1); // channels
// set the PCA termination criterion
calcLimit = cvTermCriteria(
CV_TERMCRIT_ITER, // type
nEigens, // max_iter
1); // epsilon
LOGGER.info("computing average image, eigenvalues and eigenvectors");
// compute average image, eigenvalues, and eigenvectors
cvCalcEigenObjects(
nTrainFaces, // nObjects
new PointerPointer(trainingFaceImgArr), // input
new PointerPointer(eigenVectArr), // output
CV_EIGOBJ_NO_CALLBACK, // ioFlags
0, // ioBufSize
null, // userData
calcLimit,
pAvgTrainImg, // avg
eigenValMat.data_fl()); // eigVals
LOGGER.info("normalizing the eigenvectors");
cvNormalize(
eigenValMat, // src (CvArr)
eigenValMat, // dst (CvArr)
1, // a
0, // b
CV_L1, // norm_type
null); // mask
}
/** Stores the training data to the file 'data/facedata.xml'. */
private void storeTrainingData() {
CvFileStorage fileStorage;
int i;
LOGGER.info("writing data/facedata.xml");
// create a file-storage interface
fileStorage = cvOpenFileStorage(
"data/facedata.xml", // filename
null, // memstorage
CV_STORAGE_WRITE, // flags
null); // encoding
// Store the person names. Added by Shervin.
cvWriteInt(
fileStorage, // fs
"nPersons", // name
nPersons); // value
for (i = 0; i < nPersons; i++) {
String varname = "personName_" + (i + 1);
String personame=(String)personNames.get(i);
cvWriteString(
fileStorage, // fs
varname, // name
personame, // string
0); // quote
}
// store all the data
cvWriteInt(
fileStorage, // fs
"nEigens", // name
nEigens); // value
cvWriteInt(
fileStorage, // fs
"nTrainFaces", // name
nTrainFaces); // value
cvWrite(
fileStorage, // fs
"trainPersonNumMat", // name
personNumTruthMat, // value
cvAttrList()); // attributes
cvWrite(
fileStorage, // fs
"eigenValMat", // name
eigenValMat, // value
cvAttrList()); // attributes
cvWrite(
fileStorage, // fs
"projectedTrainFaceMat", // name
projectedTrainFaceMat,
cvAttrList()); // value
cvWrite(fileStorage, // fs
"avgTrainImg", // name
pAvgTrainImg, // value
cvAttrList()); // attributes
for (i = 0; i < nEigens; i++) {
String varname = "eigenVect_" + i;
cvWrite(
fileStorage, // fs
varname, // name
eigenVectArr[i], // value
cvAttrList()); // attributes
}
// release the file-storage interface
cvReleaseFileStorage(fileStorage);
}
/** Opens the training data from the file 'data/facedata.xml'.
*
* #param pTrainPersonNumMat
* #return the person numbers during training, or null if not successful
*/
private CvMat loadTrainingData() {
LOGGER.info("loading training data");
CvMat pTrainPersonNumMat = null; // the person numbers during training
CvFileStorage fileStorage;
int i;
// create a file-storage interface
fileStorage = cvOpenFileStorage(
"data/facedata.xml", // filename
null, // memstorage
CV_STORAGE_READ, // flags
null); // encoding
if (fileStorage == null) {
LOGGER.error("Can't open training database file 'data/facedata.xml'.");
return null;
}
// Load the person names.
personNames.clear(); // Make sure it starts as empty.
nPersons = cvReadIntByName(
fileStorage, // fs
null, // map
"nPersons", // name
0); // default_value
if (nPersons == 0) {
LOGGER.error("No people found in the training database 'data/facedata.xml'.");
return null;
} else {
LOGGER.info(nPersons + " persons read from the training database");
}
// Load each person's name.
for (i = 0; i < nPersons; i++) {
String sPersonName;
String varname = "personName_" + (i + 1);
sPersonName = cvReadStringByName(
fileStorage, // fs
null, // map
varname,
"");
personNames.add(sPersonName);
}
LOGGER.info("person names: " + personNames);
// Load the data
nEigens = cvReadIntByName(
fileStorage, // fs
null, // map
"nEigens",
0); // default_value
nTrainFaces = cvReadIntByName(
fileStorage,
null, // map
"nTrainFaces",
0); // default_value
Pointer pointer = cvReadByName(
fileStorage, // fs
null, // map
"trainPersonNumMat", // name
cvAttrList()); // attributes
pTrainPersonNumMat = new CvMat(pointer);
pointer = cvReadByName(
fileStorage, // fs
null, // map
"eigenValMat", // nmae
cvAttrList()); // attributes
eigenValMat = new CvMat(pointer);
pointer = cvReadByName(
fileStorage, // fs
null, // map
"projectedTrainFaceMat", // name
cvAttrList()); // attributes
projectedTrainFaceMat = new CvMat(pointer);
pointer = cvReadByName(
fileStorage,
null, // map
"avgTrainImg",
cvAttrList()); // attributes
pAvgTrainImg = new IplImage(pointer);
eigenVectArr = new IplImage[nTrainFaces];
for (i = 0; i < nEigens; i++) {
String varname = "eigenVect_" + i;
pointer = cvReadByName(
fileStorage,
null, // map
varname,
cvAttrList()); // attributes
eigenVectArr[i] = new IplImage(pointer);
}
// release the file-storage interface
cvReleaseFileStorage(fileStorage);
LOGGER.info("Training data loaded (" + nTrainFaces + " training images of " + nPersons + " people)");
final StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append("People: ");
if (nPersons > 0) {
stringBuilder.append("<").append(personNames.get(0)).append(">");
}
for (i = 1; i < nPersons; i++) {
stringBuilder.append(", <").append(personNames.get(i)).append(">");
}
LOGGER.info(stringBuilder.toString());
return pTrainPersonNumMat;
}
/** Saves all the eigenvectors as images, so that they can be checked. */
private void storeEigenfaceImages() {
// Store the average image to a file
LOGGER.info("Saving the image of the average face as 'data/out_averageImage.bmp'");
cvSaveImage("data/out_averageImage.bmp", pAvgTrainImg);
// Create a large image made of many eigenface images.
// Must also convert each eigenface image to a normal 8-bit UCHAR image instead of a 32-bit float image.
LOGGER.info("Saving the " + nEigens + " eigenvector images as 'data/out_eigenfaces.bmp'");
if (nEigens > 0) {
// Put all the eigenfaces next to each other.
int COLUMNS = 8; // Put upto 8 images on a row.
int nCols = Math.min(nEigens, COLUMNS);
int nRows = 1 + (nEigens / COLUMNS); // Put the rest on new rows.
int w = eigenVectArr[0].width();
int h = eigenVectArr[0].height();
CvSize size = cvSize(nCols * w, nRows * h);
final IplImage bigImg = cvCreateImage(
size,
IPL_DEPTH_8U, // depth, 8-bit Greyscale UCHAR image
1); // channels
for (int i = 0; i < nEigens; i++) {
// Get the eigenface image.
IplImage byteImg = convertFloatImageToUcharImage(eigenVectArr[i]);
// Paste it into the correct position.
int x = w * (i % COLUMNS);
int y = h * (i / COLUMNS);
CvRect ROI = cvRect(x, y, w, h);
cvSetImageROI(
bigImg, // image
ROI); // rect
cvCopy(
byteImg, // src
bigImg, // dst
null); // mask
cvResetImageROI(bigImg);
cvReleaseImage(byteImg);
}
cvSaveImage(
"data/out_eigenfaces.bmp", // filename
bigImg); // image
cvReleaseImage(bigImg);
}
}
/** Converts the given float image to an unsigned character image.
*
* #param srcImg the given float image
* #return the unsigned character image
*/
private IplImage convertFloatImageToUcharImage(IplImage srcImg) {
IplImage dstImg;
if ((srcImg != null) && (srcImg.width() > 0 && srcImg.height() > 0)) {
// Spread the 32bit floating point pixels to fit within 8bit pixel range.
CvPoint minloc = new CvPoint();
CvPoint maxloc = new CvPoint();
double[] minVal = new double[1];
double[] maxVal = new double[1];
cvMinMaxLoc(srcImg, minVal, maxVal, minloc, maxloc, null);
// Deal with NaN and extreme values, since the DFT seems to give some NaN results.
if (minVal[0] < -1e30) {
minVal[0] = -1e30;
}
if (maxVal[0] > 1e30) {
maxVal[0] = 1e30;
}
if (maxVal[0] - minVal[0] == 0.0f) {
maxVal[0] = minVal[0] + 0.001; // remove potential divide by zero errors.
} // Convert the format
dstImg = cvCreateImage(cvSize(srcImg.width(), srcImg.height()), 8, 1);
cvConvertScale(srcImg, dstImg, 255.0 / (maxVal[0] - minVal[0]), -minVal[0] * 255.0 / (maxVal[0] - minVal[0]));
return dstImg;
}
return null;
}
/** Find the most likely person based on a detection. Returns the index, and stores the confidence value into pConfidence.
*
* #param projectedTestFace the projected test face
* #param pConfidencePointer a pointer containing the confidence value
* #param iTestFace the test face index
* #return the index
*/
private int findNearestNeighbor(float projectedTestFace[], FloatPointer pConfidencePointer) {
double leastDistSq = Double.MAX_VALUE;
int i = 0;
int iTrain = 0;
int iNearest = 0;
LOGGER.info("................");
LOGGER.info("find nearest neighbor from " + nTrainFaces + " training faces");
for (iTrain = 0; iTrain < nTrainFaces; iTrain++) {
//LOGGER.info("considering training face " + (iTrain + 1));
double distSq = 0;
for (i = 0; i < nEigens; i++) {
//LOGGER.debug(" projected test face distance from eigenface " + (i + 1) + " is " + projectedTestFace[i]);
float projectedTrainFaceDistance = (float) projectedTrainFaceMat.get(iTrain, i);
float d_i = projectedTestFace[i] - projectedTrainFaceDistance;
distSq += d_i * d_i; // / eigenValMat.data_fl().get(i); // Mahalanobis distance (might give better results than Eucalidean distance)
// if (iTrain < 5) {
// LOGGER.info(" ** projected training face " + (iTrain + 1) + " distance from eigenface " + (i + 1) + " is " + projectedTrainFaceDistance);
// LOGGER.info(" distance between them " + d_i);
// LOGGER.info(" distance squared " + distSq);
// }
}
if (distSq < leastDistSq) {
leastDistSq = distSq;
iNearest = iTrain;
LOGGER.info(" training face " + (iTrain + 1) + " is the new best match, least squared distance: " + leastDistSq);
}
}
// Return the confidence level based on the Euclidean distance,
// so that similar images should give a confidence between 0.5 to 1.0,
// and very different images should give a confidence between 0.0 to 0.5.
float pConfidence = (float) (1.0f - Math.sqrt(leastDistSq / (float) (nTrainFaces * nEigens)) / 255.0f);
pConfidencePointer.put(pConfidence);
LOGGER.info("training face " + (iNearest + 1) + " is the final best match, confidence " + pConfidence);
return iNearest;
}
/** Returns a string representation of the given float array.
*
* #param floatArray the given float array
* #return a string representation of the given float array
*/
private String floatArrayToString(final float[] floatArray) {
final StringBuilder stringBuilder = new StringBuilder();
boolean isFirst = true;
stringBuilder.append('[');
for (int i = 0; i < floatArray.length; i++) {
if (isFirst) {
isFirst = false;
} else {
stringBuilder.append(", ");
}
stringBuilder.append(floatArray[i]);
}
stringBuilder.append(']');
return stringBuilder.toString();
}
/** Returns a string representation of the given float pointer.
*
* #param floatPointer the given float pointer
* #return a string representation of the given float pointer
*/
private String floatPointerToString(final FloatPointer floatPointer) {
final StringBuilder stringBuilder = new StringBuilder();
boolean isFirst = true;
stringBuilder.append('[');
for (int i = 0; i < floatPointer.capacity(); i++) {
if (isFirst) {
isFirst = false;
} else {
stringBuilder.append(", ");
}
stringBuilder.append(floatPointer.get(i));
}
stringBuilder.append(']');
return stringBuilder.toString();
}
/** Returns a string representation of the given one-channel CvMat object.
*
* #param cvMat the given CvMat object
* #return a string representation of the given CvMat object
*/
public String oneChannelCvMatToString(final CvMat cvMat) {
//Preconditions
if (cvMat.channels() != 1) {
throw new RuntimeException("illegal argument - CvMat must have one channel");
}
final int type = cvMat.type();
StringBuilder s = new StringBuilder("[ ");
for (int i = 0; i < cvMat.rows(); i++) {
for (int j = 0; j < cvMat.cols(); j++) {
if (type == CV_32FC1 || type == CV_32SC1) {
s.append(cvMat.get(i, j));
} else {
throw new RuntimeException("illegal argument - CvMat must have one channel and type of float or signed integer");
}
if (j < cvMat.cols() - 1) {
s.append(", ");
}
}
if (i < cvMat.rows() - 1) {
s.append("\n ");
}
}
s.append(" ]");
return s.toString();
}
/** Executes this application.
*
* #param args the command line arguments
*/
public static void main(final String[] args) {
BasicConfigurator.configure();
// PropertyConfigurator.configure(args[0]);
// if(args[0]!=null)
/*{
System.out.println("null index");
}
else continue;
*/
final FaceRecognition faceRecognition = new FaceRecognition();
// main myMain = new main();
// myMain.FaceRecognition();
//faceRecognition.learn("data/some-training-faces.txt");
// faceRecognition.learn("G:\\android_support\\javacv-examples\\OpenCV2_Cookbook\\data\\all10.txt");
faceRecognition.learn("data/all100.txt");
//faceRecognition.recognizeFileList("data/some-test-faces.txt");
// faceRecognition.recognizeFileList("G:\\android_support\\javacv-examples\\OpenCV2_Cookbook\\data\\lower3.txt");
faceRecognition.recognizeFileList("data/lower3.txt");
}
}
Error
Exception in thread "main" java.lang.RuntimeException: Stub!
at android.util.Log.i(Log.java:9)
at FaceRecognition.learn(FaceRecognition.java:126)
at FaceRecognition.main(FaceRecognition.java:846)

It seems that you try to test android classes on the host computer. Perhaps using JUnit. That does not work. There are two options open:
If possible move the functionality into library so that no android packages are used. For example use java.util.logging instead of android.util.Log.
Use android instrumentation tests (google it).
And if you use Maven: remember that the android library is <scope>provided</scope>.

Thumbnail of a PDF page (Java) [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking for code must demonstrate a minimal understanding of the problem being solved. Include attempted solutions, why they didn't work, and the expected results. See also: Stack Overflow question checklist
Closed 9 years ago.
Improve this question
How can I generate a thumbnail image of pages in a PDF document, using Java?

I think http://pdfbox.apache.org/ will do what you're looking for since you can create an image from a page and then scale the image
From their example code -
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.pdfbox;
import java.awt.HeadlessException;
import java.awt.Toolkit;
import java.awt.image.BufferedImage;
import javax.imageio.ImageIO;
import java.util.List;
import org.apache.pdfbox.exceptions.InvalidPasswordException;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.pdmodel.PDPage;
import org.apache.pdfbox.pdmodel.common.PDRectangle;
import org.apache.pdfbox.util.PDFImageWriter;
/**
* Convert a PDF document to an image.
*
* #author Ben Litchfield
* #version $Revision: 1.6 $
*/
public class PDFToImage
{
private static final String PASSWORD = "-password";
private static final String START_PAGE = "-startPage";
private static final String END_PAGE = "-endPage";
private static final String IMAGE_FORMAT = "-imageType";
private static final String OUTPUT_PREFIX = "-outputPrefix";
private static final String COLOR = "-color";
private static final String RESOLUTION = "-resolution";
private static final String CROPBOX = "-cropbox";
/**
* private constructor.
*/
private PDFToImage()
{
//static class
}
/**
* Infamous main method.
*
* #param args Command line arguments, should be one and a reference to a file.
*
* #throws Exception If there is an error parsing the document.
*/
public static void main( String[] args ) throws Exception
{
String password = "";
String pdfFile = null;
String outputPrefix = null;
String imageFormat = "jpg";
int startPage = 1;
int endPage = Integer.MAX_VALUE;
String color = "rgb";
int resolution;
float cropBoxLowerLeftX = 0;
float cropBoxLowerLeftY = 0;
float cropBoxUpperRightX = 0;
float cropBoxUpperRightY = 0;
try
{
resolution = Toolkit.getDefaultToolkit().getScreenResolution();
}
catch( HeadlessException e )
{
resolution = 96;
}
for( int i = 0; i < args.length; i++ )
{
if( args[i].equals( PASSWORD ) )
{
i++;
if( i >= args.length )
{
usage();
}
password = args[i];
}
else if( args[i].equals( START_PAGE ) )
{
i++;
if( i >= args.length )
{
usage();
}
startPage = Integer.parseInt( args[i] );
}
else if( args[i].equals( END_PAGE ) )
{
i++;
if( i >= args.length )
{
usage();
}
endPage = Integer.parseInt( args[i] );
}
else if( args[i].equals( IMAGE_FORMAT ) )
{
i++;
imageFormat = args[i];
}
else if( args[i].equals( OUTPUT_PREFIX ) )
{
i++;
outputPrefix = args[i];
}
else if( args[i].equals( COLOR ) )
{
i++;
color = args[i];
}
else if( args[i].equals( RESOLUTION ) )
{
i++;
resolution = Integer.parseInt(args[i]);
}
else if( args[i].equals( CROPBOX ) )
{
i++;
cropBoxLowerLeftX = Float.valueOf(args[i]).floatValue();
i++;
cropBoxLowerLeftY = Float.valueOf(args[i]).floatValue();
i++;
cropBoxUpperRightX = Float.valueOf(args[i]).floatValue();
i++;
cropBoxUpperRightY = Float.valueOf(args[i]).floatValue();
}
else
{
if( pdfFile == null )
{
pdfFile = args[i];
}
}
}
if( pdfFile == null )
{
usage();
}
else
{
if(outputPrefix == null)
{
outputPrefix = pdfFile.substring( 0, pdfFile.lastIndexOf( '.' ));
}
PDDocument document = null;
try
{
document = PDDocument.load( pdfFile );
//document.print();
if( document.isEncrypted() )
{
try
{
document.decrypt( password );
}
catch( InvalidPasswordException e )
{
if( args.length == 4 )//they supplied the wrong password
{
System.err.println( "Error: The supplied password is incorrect." );
System.exit( 2 );
}
else
{
//they didn't supply a password and the default of "" was wrong.
System.err.println( "Error: The document is encrypted." );
usage();
}
}
}
int imageType = 24;
if ("bilevel".equalsIgnoreCase(color))
{
imageType = BufferedImage.TYPE_BYTE_BINARY;
}
else if ("indexed".equalsIgnoreCase(color))
{
imageType = BufferedImage.TYPE_BYTE_INDEXED;
}
else if ("gray".equalsIgnoreCase(color))
{
imageType = BufferedImage.TYPE_BYTE_GRAY;
}
else if ("rgb".equalsIgnoreCase(color))
{
imageType = BufferedImage.TYPE_INT_RGB;
}
else if ("rgba".equalsIgnoreCase(color))
{
imageType = BufferedImage.TYPE_INT_ARGB;
}
else
{
System.err.println( "Error: the number of bits per pixel must be 1, 8 or 24." );
System.exit( 2 );
}
//si une cropBox a ete specifier, appeler la methode de modification de cropbox
//changeCropBoxes(PDDocument document,float a, float b, float c,float d)
if ( cropBoxLowerLeftX!=0 || cropBoxLowerLeftY!=0 || cropBoxUpperRightX!=0 || cropBoxUpperRightY!=0 )
{
changeCropBoxes(document,cropBoxLowerLeftX, cropBoxLowerLeftY, cropBoxUpperRightX, cropBoxUpperRightY);
}
//Make the call
PDFImageWriter imageWriter = new PDFImageWriter();
boolean success = imageWriter.writeImage(document, imageFormat, password,
startPage, endPage, outputPrefix, imageType, resolution);
if (!success)
{
System.err.println( "Error: no writer found for image format '"
+ imageFormat + "'" );
System.exit(1);
}
}
catch (Exception e)
{
System.err.println(e);
}
finally
{
if( document != null )
{
document.close();
}
}
}
}
/**
* This will print the usage requirements and exit.
*/
private static void usage()
{
System.err.println( "Usage: java org.apache.pdfbox.PDFToImage [OPTIONS] <PDF file>\n" +
" -password <password> Password to decrypt document\n" +
" -imageType <image type> (" + getImageFormats() + ")\n" +
" -outputPrefix <output prefix> Filename prefix for image files\n" +
" -startPage <number> The first page to start extraction(1 based)\n" +
" -endPage <number> The last page to extract(inclusive)\n" +
" -color <string> The color depth (valid: bilevel, indexed, gray, rgb, rgba)\n" +
" -resolution <number> The bitmap resolution in dpi\n" +
" -cropbox <number> <number> <number> <number> The page area to export\n" +
" <PDF file> The PDF document to use\n"
);
System.exit( 1 );
}
private static String getImageFormats()
{
StringBuffer retval = new StringBuffer();
String[] formats = ImageIO.getReaderFormatNames();
for( int i = 0; i < formats.length; i++ )
{
retval.append( formats[i] );
if( i + 1 < formats.length )
{
retval.append( "," );
}
}
return retval.toString();
}
private static void changeCropBoxes(PDDocument document,float a, float b, float c,float d)
{
List pages = document.getDocumentCatalog().getAllPages();
for( int i = 0; i < pages.size(); i++ )
{
System.out.println("resizing page");
PDPage page = (PDPage)pages.get( i );
PDRectangle rectangle = new PDRectangle();
rectangle.setLowerLeftX(a);
rectangle.setLowerLeftY(b);
rectangle.setUpperRightX(c);
rectangle.setUpperRightY(d);
page.setMediaBox(rectangle);
page.setCropBox(rectangle);
}
}
}

You could also have a look at JPedal (details at http://www.jpedal.org/pdf_thumbnail.php)

IcePdf is the best that I've seen (that's free) for reading pdfs. JPedal is awesome, but not free.
If you're going to be generating images from pdfs that the general public can send you, I assure you (from experience) that you'll get pdfs that will crash the JVM. (ie: If they're many-layered pdfs with all vector graphics). This pdf is an example that will crash many libraries (but is a perfectly valid PDF without anything funny like Javascript, etc).
We've gone down the route of trying to use a multitude of libraries and eventually resorting to delegating the work of creating a thumbnail to ImageMagick, which is a highly optimized C program for image manipulation.

This post is not only pdf, but also many other file type like office, image, text....