Develop Reference

PDF file encode to base64 take more time if 100k documents are to be encode

Am trying to encode pdf documents to base64, If it is less in number ( like 2000 documents) its working nicely. But am having 100k plus doucments to be encode.
Its take more time to encode all those files. Is there any better approach to encode large data set.?
Please find my current approach
String filepath=doc.getPath().concat(doc.getFilename());
file = new File(filepath);
if(file.exists() && !file.isDirectory()) {
try {
FileInputStream fileInputStreamReader = new FileInputStream(file);
byte[] bytes = new byte[(int) file.length()];
fileInputStreamReader.read(bytes);
encodedfile = new String(Base64.getEncoder().encodeToString(bytes));
fileInputStreamReader.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}

Try this:
Figure out how many files you need to encode.
int files = Files.list(Paths.get(directory)).count();
Split them up into a reasonable amount that a thread can handle in java. I.E) If you have 100k files to encode. Split it into 1000 lists of 1000, something like that.
int currentIndex = 0;
for (File file : filesInDir) {
if (fileMap.get(currentIndex).size() >= cap)
currentIndex++;
fileMap.get(currentIndex).add(file);
}
/** Its going to take a little more effort than this, but its the idea im trying to show you*/
Execute each worker thread one after another if the computers resources are available.
for (Integer key : fileMap.keySet()) {
new WorkerThread(fileMap.get(key)).start();
}
You can check the current resources available with:
public boolean areResourcesAvailable() {
return imNotThatNice();
}
/**
* Gets the resource utility instance
*
* #return the current instance of the resource utility
*/
private static OperatingSystemMXBean getInstance() {
if (ResourceUtil.instance == null) {
ResourceUtil.instance = ManagementFactory.getOperatingSystemMXBean();
}
return ResourceUtil.instance;
}

OpenNLP model builder addon doesnt continue

I'm using the model builder addon for OpenNLP to create a better NER model.
According to this post, I have used the code posted by markg :
public class ModelBuilderAddonUse {
private static List<String> getSentencesFromSomewhere() throws Exception
{
List<String> list = new ArrayList<String>();
BufferedReader reader = new BufferedReader(new FileReader("D:\\Work\\workspaces\\default\\UpdateModel\\documentrequirements.docx"));
String line;
while ((line = reader.readLine()) != null)
{
list.add(line);
}
reader.close();
return list;
}
public static void main(String[] args) throws Exception {
/**
* establish a file to put sentences in
*/
File sentences = new File("D:\\Work\\workspaces\\default\\UpdateModel\\sentences.text");
/**
* establish a file to put your NER hits in (the ones you want to keep based
* on prob)
*/
File knownEntities = new File("D:\\Work\\workspaces\\default\\UpdateModel\\knownentities.txt");
/**
* establish a BLACKLIST file to put your bad NER hits in (also can be based
* on prob)
*/
File blacklistedentities = new File("D:\\Work\\workspaces\\default\\UpdateModel\\blentities.txt");
/**
* establish a file to write your annotated sentences to
*/
File annotatedSentences = new File("D:\\Work\\workspaces\\default\\UpdateModel\\annotatedSentences.txt");
/**
* establish a file to write your model to
*/
File theModel = new File("D:\\Work\\workspaces\\default\\UpdateModel\\nl-ner-person.bin");
//------------create a bunch of file writers to write your results and sentences to a file
FileWriter sentenceWriter = new FileWriter(sentences, true);
FileWriter blacklistWriter = new FileWriter(blacklistedentities, true);
FileWriter knownEntityWriter = new FileWriter(knownEntities, true);
//set some thresholds to decide where to write hits, you don't have to use these at all...
double keeperThresh = .95;
double blacklistThresh = .7;
/**
* Load your model as normal
*/
TokenNameFinderModel personModel = new TokenNameFinderModel(new File("D:\\Work\\workspaces\\default\\UpdateModel\\nl-ner-person.bin"));
NameFinderME personFinder = new NameFinderME(personModel);
/**
* do your normal NER on the sentences you have
*/
for (String s : getSentencesFromSomewhere()) {
sentenceWriter.write(s.trim() + "\n");
sentenceWriter.flush();
String[] tokens = s.split(" ");//better to use a tokenizer really
Span[] find = personFinder.find(tokens);
double[] probs = personFinder.probs();
String[] names = Span.spansToStrings(find, tokens);
for (int i = 0; i < names.length; i++) {
//YOU PROBABLY HAVE BETTER HEURISTICS THAN THIS TO MAKE SURE YOU GET GOOD HITS OUT OF THE DEFAULT MODEL
if (probs[i] > keeperThresh) {
knownEntityWriter.write(names[i].trim() + "\n");
}
if (probs[i] < blacklistThresh) {
blacklistWriter.write(names[i].trim() + "\n");
}
}
personFinder.clearAdaptiveData();
blacklistWriter.flush();
knownEntityWriter.flush();
}
//flush and close all the writers
knownEntityWriter.flush();
knownEntityWriter.close();
sentenceWriter.flush();
sentenceWriter.close();
blacklistWriter.flush();
blacklistWriter.close();
/**
* THIS IS WHERE THE ADDON IS GOING TO USE THE FILES (AS IS) TO CREATE A NEW MODEL. YOU SHOULD NOT HAVE TO RUN THE FIRST PART AGAIN AFTER THIS RUNS, JUST NOW PLAY WITH THE
* KNOWN ENTITIES AND BLACKLIST FILES AND RUN THE METHOD BELOW AGAIN UNTIL YOU GET SOME DECENT RESULTS (A DECENT MODEL OUT OF IT).
*/
DefaultModelBuilderUtil.generateModel(sentences, knownEntities, blacklistedentities, theModel, annotatedSentences, "person", 3);
}
}
It also runs, but my output quits at :
annotated sentences: 1862
knowns: 58
Building Model using 1862 annotations
reading training data...
But in the example in the post it should go futher like this :
Indexing events using cutoff of 5
Computing event counts... done. 561755 events
Indexing... done.
Sorting and merging events... done. Reduced 561755 events to 127362.
Done indexing.
Incorporating indexed data for training...
done.
Number of Event Tokens: 127362
Number of Outcomes: 3
Number of Predicates: 106490
...done.
Can anyone help me to fix this problem, so I can generate a model?
I have searched realy a lot but cant find any good documutation about it.
Would really appreciat it, thanks.

Correct the path to your training data file like this:
File sentences = new File("D:/Work/workspaces/default/UpdateModel/sentences.text");
instead of
File sentences = new File("D:\\Work\\workspaces\\default\\UpdateModel\\sentences.text");
Update
This is how is used, by adding the files to the project folder. Try it like this -
File sentences = new File("src/training/resources/CreateModel/sentences.txt");
Check my respository for reference on Github
This should help.

How to save txt on custom path

I have a code that saves a textfile, but the path of that textfile and the name are hardcoded.
How can i make that the user select his/her own path and the name of the file?
int[][] sudokuNumbers = {{1, 2, 3}, {4, 5, 6}, {7, 8, 9}};
try (
PrintStream output = new PrintStream(new File("C:\\Users\\David\\Desktop\\Proyecto4.3\\output.txt"));) {
for (int i = 0; i < sudokuNumbers.length; i++) {
String s= "";
for (int j = 0; j < sudokuNumbers[i].length; j++) {
s+= "|" + sudokuNumbers[i][j] + "|";
}
output.println(s);
}
output.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
}

The task is relatively easy to carry out however you will need a couple methods to assist you. If a User is to supply a custom path (specific for them-self) then you will need to ensure that the path will be created so that it exists before the file is created and written to. Another thing to consider is that there may also be a requirement of Permissions to be granted to the application so as to carry out the task within the local file system (if this is where the file is to be created) or wherever else.
Obviously you will need to provide a means for the User to supply a path and file name. The best way to do this in my opinion is with a Folder Chooser dialog. The reason I say this is because it's not nearly as prone to mistakes (spelling, system format, etc.) as it would be if the User completely typed in the desired path and file name. Basically, the less control you give the User to the local file system the better off your application will be. Creating a folder (directory) based on the User Name within a preset file system location and utilizing a User name and date/time for file names is relatively common and can be all auto-generated, this eliminates the possibility of any file system errors. If possible, your application should maintain full control of such a task but understandably this can not always be the case for specific applications where User interaction with the file system is a requirement due to the fact that it's sole functionality is based on such interaction. For example, a File System Explorer where creating, cutting, copying, pasting, deleting, and editing of files and folders is its allowable purpose.
To demonstrate what I'm talking about I have quickly created a small Java runnable you can play with to see how the provided custom methods work. The code is well commented so that you can understand what is going on. Unfortunately all the commenting makes the code file look large when in fact it isn't if you were to remove them.
For simplicity, when we run the provided little console application it first asks for a User's Name. This Users Name will serve two specific purposes.
1) It will be used to create a specific Folder for that particular User only.
2) When a file is saved the file name will start with the User's name and then be followed with date/time data and finally contain the
".txt" file name extension.
Again, just for example purposes, as soon as a User enters his/her name into the console a Folders Selection dialog window will be displayed so that a main folder location can be selected where Users sub-folders will be created in and ultimately the Users data files will be saved under his/her folder name.
The file data is automatically saved once the home folder is selected and the console indicates of success once completed. Our little console application then simply ends.
Here is the Java runnable:
package choosefolder;
import java.awt.Component;
import java.awt.Container;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.PrintStream;
import java.time.LocalDateTime;
import java.util.Scanner;
import javax.swing.JComboBox;
import javax.swing.JFileChooser;
import javax.swing.JLabel;
public class ChooseFolder {
// Declare & initialize required String variables.
// These are global to the entire Class.
private static String userName = "";
private static String savePath = "";
public static void main(String[] args) {
//Setup a Scanner to get User input from console.
Scanner input = new Scanner(System.in);
// Ask for and get the Users Name and convert it to lowercase
// so as to maintain nameing consistency. No sense having "John Doe"
// one day and "john Doe" the next. This would consider 2 different
// Users since the second name has a lowercase "j" in "John" and
// therefore create two separate folder. Converting all supplied names
// to lowercase eliminates this problem.
while (userName.equals("")) {
System.out.println("\nPlease Enter User Name:");
userName = input.nextLine().toLowerCase();
}
// Open a Folders selection dialog window and allow for
// selection of a home folder where all Users saved files
// will be saved. Once we get the selected folder we then
// add the Users name to the path and create it.
savePath = openFoldersDialog("", "Select Home Folder To Save User Folders & File In...") + "\\" + userName;
// Create a special path just for the Users Name (ie: C:\My Files\John Doe\).
if (!createPath(savePath)) {
// If false was returnd from the createPath() method
// then there was an error while creating the path.
// Read console output to get the Error.
System.out.println("\n\u001B[31mError creating Users Save Path! Exiting...\u001B[39;49m");
System.exit(0);
}
// Get the local date & time so we can add it to
// our file namewhen its created.
LocalDateTime dateTime = LocalDateTime.now();
// Create the new file path & file name. This will be the path
// and file name where our Users data will be saved. We take the
// Users path e created and then add the file name to the end. The
// file name consists of the supplied User's name, the date/time and
// the .txt file name extension. Unwanted characters are also removed
// from the Date/Time. So, our path may look something like:
// C:\My Files\John Doe-20160419T155758107.txt
String filePathAndName = savePath + "\\" + userName + "-" +
dateTime.toString().replace("-", "").
replace(":", "").replace(".", "") + ".txt";
// Here is your code. This is the data saved to file....
int[][] sudokuNumbers = {{1, 2, 3}, {4, 5, 6}, {7, 8, 9}};
try (
// Notice the filePathAndName variable which we declared and initialized above.
PrintStream output = new PrintStream(new File(filePathAndName));) {
for (int i = 0; i < sudokuNumbers.length; i++) {
String s= "";
for (int j = 0; j < sudokuNumbers[i].length; j++) {
s+= "|" + sudokuNumbers[i][j] + "|";
}
output.println(s);
}
output.close();
}
catch (FileNotFoundException e) {
System.out.println(e.getMessage());
}
finally {
// Notify in console of success.
System.out.println("\n\u001B[34mFile Saved SUCCESSFULLY in:\n\u001B[39;49m" + filePathAndName);
}
}
/**
* Opens a Folder (directory) selection dialog window. All parameters for this method are optional but if you supply
* something to the last parameter then you must supply something to the parameters preceding it even if it's a null
* string ("") (see usage examples below)
*
* Example Usage:
*
* openFolderDialog(); //Allowed
* openFolderDialog("C:\\"); //Allowed
* openFolderDialog("", "Select The Folder You Need"); //Allowed
* openFolderDialog("", "", "Select Folder"); //Allowed
* openFolderDialog("", "", ""); //Allowed
* openFolderDialog( , "Your Folders, "Select Folder"); //Not Allowed
* openFolderDialog( , , "Select Folder"); //Not Allowed
* openFolderDialog("C:\\", "Your Folders, ); //Not Allowed
*
* I think you get the idea here.
*
* #param dialogOptions OPTIONAL - 3 String Type Parameters
* startPath - String - Default is the running applications' home folder. The path
* supplied here tells the dialog where to start displaying folders from.
* If you just want to use default but would like to supply something to
* the other optional parameters then just supply a null string ("") here.
*
* dialogTitle - String - Dialog Window Title - Default is "Select Folder...". You can
* supply whatever title you like for the dialog window.
*
* openButtonCaption - String - The Select Button Caption - Default is "Select". You can supply
* whatever you like for a button caption and it can be more than one word
* but do keep in mind that the button increases in length to accommodate the
* the supplied string.
*
* #return The selected Folder (directory) full path as string. Returns Null is nothing was selected.
*/
public static String openFoldersDialog(String... dialogOptions) {
String startPath = "";
try {
startPath = new File(".").getCanonicalPath();
} catch (IOException ex) {}
String dialogTitle = "Select Folder...";
String openButtonCaption = "Select";
if (dialogOptions.length != 0) {
if (dialogOptions.length >= 1) {
if (!dialogOptions[0].isEmpty()) { startPath = dialogOptions[0]; }
}
if (dialogOptions.length >= 2) {
if (!dialogOptions[1].isEmpty()) { dialogTitle = dialogOptions[1]; }
}
if (dialogOptions.length == 3) {
if (!dialogOptions[2].isEmpty()) { openButtonCaption = dialogOptions[2]; }
}
}
File sp = new File(startPath);
JFileChooser fc = new JFileChooser(sp);
//Remove the "Files Type:" combo box and label from dialog.
removeJFCFilesTypeComponent(fc);
fc.setDialogTitle(dialogTitle);
fc.setAcceptAllFileFilterUsed(false);
fc.setRequestFocusEnabled(true);
fc.setFileSelectionMode(JFileChooser.DIRECTORIES_ONLY);
int userSelection = fc.showDialog(null, openButtonCaption); //.showSaveDialog(null);
if (userSelection == JFileChooser.APPROVE_OPTION) {
String f = fc.getSelectedFile().getPath();
return f;
}
return null;
}
/**
* Removes the Filter ComboBox from the supplied JFileChooser object.
* This helps make the JFileChooser dialog window look more like a
* folder selection dialog.
* #param filechooser The object name of the JFileChooser object.
*/
public static void removeJFCFilesTypeComponent(Container filechooser) {
Component[] components = filechooser.getComponents();
for (Component component : components) {
if (component instanceof JComboBox) {
Object sel = ((JComboBox) component).getSelectedItem();
if (sel.toString().contains("AcceptAllFileFilter")) {
component.setVisible(false);
// OR
//con.remove(component);
}
}
if (component instanceof JLabel) {
String text = ((JLabel) component).getText();
if (text.equals("Files of Type:")) {
component.setVisible(false);
// OR
//con.remove(component);
}
}
if (component instanceof Container) {
removeJFCFilesTypeComponent((Container) component);
}
}
}
/**
* Creates the supplied path into the local file system if it doesn't already exist. You must
* make sure permissions are properly in place before using this method otherwise the path may
* not be created within the file system.
* #param pathString The path to create (must not include a file name). Will not create path if it already exists.
* #return Boolean True if successful and Boolean False if not.
*/
public static boolean createPath(String pathString) {
if (pathString.isEmpty()) { return false; }
if (doesFolderExist(pathString)) { return true; }
try {
File foldersToMake = new File(pathString);
foldersToMake.mkdirs();
return true;
}
catch (Exception ex) {
System.out.println("*** CreatePath() - Exception Encountered:\n*** " + ex.getMessage());
return false;
}
}
/**
* This method simply return boolean true or false if the supplied path exists.
* #param folderPathString (String) The full path to check. No File Name is allowed
* to be in this path.
* #return
*/
public static boolean doesFolderExist(String folderPathString) {
boolean tf = false;
try {
if (folderPathString.isEmpty() || "".equals(folderPathString)) { return tf; }
File f = new File(folderPathString);
if(f.exists() && f.isDirectory()) { tf = true; }
}
catch (Exception e) {
System.out.println("*** DoesFolderExist() - Exception Encountered:\n*** " + e.getMessage());
tf = false;
}
return tf;
}
}
I hope this helps you (and others).

Search Box for Jpanel

I am in the middle of creating an app that allows users to apply for job positions and upload their CVs. I`m currently stuck on trying to make a search box for the admin to be able to search for Keywords. The app will than look through all the CVs and if it finds such keywords it will show up a list of Cvs that contain the keyword. I am fairly new to Gui design and app creation so not sure how to go about doing it. I wish to have it done via java and am using the Eclipse Window builder to help me design it. Any help will be greatly appreciated, hints, advice anything. Thank You.

Well, this not right design approach as real time search of words in all files of given folder will be slow and not sustainable in long run. Ideally you should have indexed all CV's for keywords. The search should run on index and then get the associated CV for that index ( think of indexes similar to tags). There are many options for indexing - simples DB indexing or using Apache Lucene or follow these steps to create a index using Maps and refer this index for search.
Create a map Map<String, List<File>> for keeping the association of
keywords to files
iterate through all files, and for each word in
each file, add that file to the list corresponding to that word in
your index map
here is the java code which will work for you but I would still suggest to change your design approach and use indexes.
File dir = new File("Folder for CV's");
if(dir.exists())
{
Pattern p = Pattern.compile("Java");
ArrayList<String> list = new ArrayList<String>(); // list of CV's
for(File f : dir.listFiles())
{
if(!f.isFile()) continue;
try
{
FileInputStream fis = new FileInputStream(f);
byte[] data = new byte[fis.available()];
fis.read(data);
String text = new String(data);
Matcher m = p.matcher(text);
if(m.find())
{
list.add(f.getName()); // add file to found-keyword list.
}
fis.close();
}
catch(Exception e)
{
System.out.print("\n\t Error processing file : "+f.getName());
}
}
System.out.print("\n\t List : "+list); // list of files containing keyword.
} // IF directory exists then only process.
else
{
System.out.print("\n Directory doesn't exist.");
}
Here you get the files list to show now for "Java". As I said use indexes :)

Thanks for taking your time to look into my problem.
I have actually come up with a solution of my own. It is probably very amateur like but it works for me.
JButton btnSearch = new JButton("Search");
btnSearch.addActionListener(new ActionListener()
{
public void actionPerformed(ActionEvent arg0)
{
list.clear();
String s = SearchBox.getText();
int i = 0,present = 0;
int id;
try
{
Class.forName(driver).newInstance();
Connection conn = DriverManager.getConnection(url+dbName,userName,password);
Statement st = conn.createStatement();
ResultSet res = st.executeQuery("SELECT * FROM javaapp.test");
while(res.next())
{
i = 0;
present = 0;
while(i < 9)
{
String out = res.getString(search[i]);
if(out.toLowerCase().contains(s.toLowerCase()))
{
present = 1;
break;
}
i++;
}
if(tglbtnNormalshortlist.isSelected())
{
if(present == 1 && res.getInt("Shortlist") == 1)
{
id = res.getInt("Candidate");
String print = res.getString("Name");
list.addElement(print+" "+id);
}
}
else
{
if(present == 1 && res.getInt("Shortlist") == 0)
{
id = res.getInt("Candidate");
String print = res.getString("Name");
list.addElement(print+" "+id);
}
}
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
});

Why different outcomes with file-io on Windows and Linux machines when appending to file?

I have a class that appends new configuration to existing one by reading a uploaded file. The problem is it runs fine on windows, but not so on Linux - I am using Servlet for receiving the file. The new configuration must begin one a new line and must not have empty line(s) anywhere. Following is the code.
public class ConfigGen {
public static void process(File configFile, File uploadedFile)
throws IOException {
synchronized (configFile) {
if (shouldAppend(configFile, uploadedFile)) {
StringBuilder builder = readFile(uploadedFile);
StringBuilder orig = readFile(configFile);
FileWriter writer = new FileWriter(configFile, true);
// If the content ends with a new line character then remove the
// new line from content to be appended
if (orig.toString().endsWith(
System.getProperty("line.separator"))) {
writer.append(builder.substring(1));
} else {
writer.append(builder);
}
writer.flush();
writer.close();
}
}
}
/**
* #param configFile
* #param uploadedFile
* #return true if config change is required, false otherwise
* #throws IOException
*/
private static final boolean shouldAppend(File configFile, File uploadedFile)
throws IOException {
synchronized (configFile) {
String originalConfig = readFile(configFile).toString().trim();
String newAddition = readFile(uploadedFile).toString().trim();
String newSecGroup = newAddition.substring(0,
newAddition.indexOf(":"));
return !originalConfig.contains(newSecGroup);
}
}
private static final StringBuilder readFile(File file) throws IOException {
BufferedReader br = new BufferedReader(new FileReader(file));
String str = "";
StringBuilder builder = new StringBuilder();
while ((str = br.readLine()) != null) {
builder.append(System.getProperty("line.separator") + str);
}
br.close();
return builder;
}
}
The file to which new configurations is being appended will at many time be edited by hand using vim or similar editor. The behavior I am getting is - on linux machine.
If the file has been edited using vim, then the new text is appended after an additional new line, thereafter any subsequent updates appears from the next line - until it is edited using vim.
For example: Let say I have the following content in file - initially written using vim.
sg-e696a08c:
classes:
tomcat6:
myappmodule:
endpoint: "10.0.2.11"
port: "1443"
And the new file posted with the following content
sg-fde2a89d:
classes:
tomcat7:
servmod:
db: "xyz.com"
port: "1336"
The expected content of the configuration file is.
sg-e696a08c:
classes:
tomcat6:
myappmodule:
endpoint: "10.0.2.11"
port: "1443"
sg-fde2a89d:
classes:
tomcat7:
servmod:
db: "xyz.com"
port: "1336"
But instead I get
sg-e696a08c:
classes:
tomcat6:
myappmodule:
endpoint: "10.0.2.11"
port: "1443"
sg-fde2a89d:
classes:
tomcat7:
servmod:
db: "xyz.com"
port: "1336"
Notice there is an empty line. Any subsequent file that is posted will not result in any empty new line, until the file is edited using vim. I am not sure why I have getting this kind of behavior. The results of windows are fine. Because of this behavior the generated configuration become invalid. On Windows I am using Notepad++ to edit the files.
Is it that vim adds some special character to the end of file that I might be missing or something else ? What should I do to fix this, and Why this behaviour ?

You are prepending a newline unnecessarily in readFile():
//builder.append(System.getProperty("line.separator") + str);
builder.append(str).append(System.getProperty("line.separator"));
and the commented code (from process) seems superflous.
//if (orig.toString().endsWith(
// System.getProperty("line.separator"))) {
// writer.append(builder.substring(1));
//} else {
// writer.append(builder);
//}
// just use this:
writer.append(builder);
Apart from that it may be worth mentioning that a file written in Vim ends with a newline if fileformat=unix and with a carriage return+newline if fileformat=dos. However, that should be of no consequence in this case since you use trim().