Develop Reference

Fine-tuning method listFiles

Can anyone help in tuning this method? When I log the "files" - it only takes around 5 seconds. But takes more than 10 minutes before returning the "fileInfo"
// fileSystem is HDFS
// dateNow = java.util.Date
// basePath = new Path("/")
// filePattern = "*.sf"
private Map<String, Long> listFiles(final Date dateNow, final Path basePath,
final String filePattern) throws IOException {
RemoteIterator<LocatedFileStatus> files = fileSystem.listFiles(basePath, true);
_LOG.info("files=" + files);
// map containing <filename, filesize>
Map<String, Long> fileInfo = new HashMap<String, Long>();
String regex = RegexUtil.convertGlobToRegex(filePattern);
Pattern pattern = Pattern.compile(regex);
if (files != null) {
while (files.hasNext()) {
LocatedFileStatus file = files.next();
Path filePath = file.getPath();
// Get only the files with created date = current date
if (DateUtils.truncate(new Date(file.getModificationTime()),
java.util.Calendar.DAY_OF_MONTH).equals(dateNow)) {
if (pattern.matcher(filePath.getName()).matches()) {
fileInfo.put(file.getPath().getName(), file.getLen());
}
}
}
}
_LOG.info("fileInfo =" + fileInfo);
return fileInfo;
}

You Said
When I log the "files" - it only takes around 5 seconds
RemoteIterator<LocatedFileStatus> files = fileSystem.listFiles(basePath, true);
Yes. Because this part of the code only checks the File present at that path (eg.:- no.Of Files,size) Status not looking into the file what and how much data it Contains.
Now if you look into this part of code
while (files.hasNext()) {
LocatedFileStatus file = files.next();
Path filePath = file.getPath();
// Get only the files with created date = current date
if (DateUtils.truncate(new Date(file.getModificationTime()),
java.util.Calendar.DAY_OF_MONTH).equals(dateNow)) {
if (pattern.matcher(filePath.getName()).matches()) {
fileInfo.put(file.getPath().getName(), file.getLen());
}
}
}
then you analyze that it Iterate throughout the Content of all the files in List. So, Definitely It will take more time than the previous one. This files may contains a number of files with different size of Content.
So, Iterating into each file content will definitely took more time. It also depends upon the size of the files this directory Contains. The more large your file the more time would took this loop.

Use listStatus with a PathFinder. This does much of the work on the server-side, and accumulated.

How to upload and save an attachment via XPages Java Bean

I get how you can use Expression Language to bind XPages controls to a Java Bean. Then it accesses the setters and getters automatically.
But how do you handle a file attachment?
What does that look like? I'd like to be able to I guess bind the file upload control to the bean. Save the attachment to "whatever" doc... whether it's the current or external document.. the bean should be able to handle that logic.
I guess I don't know how to get that file attachment into the in memory bean to be able to do anything with it like saving to a document.
any advice would be appreciated.
Update: This is a similar question to this: How to store uploaded file to local file system using xPages upload control?
But in that question the user wants to save to local disc. I'm looking to save to a document.
Thanks!

You need to create a getter and setter in the bean using the com.ibm.xsp.component.UIFileuploadEx.UploadedFile class:
private UploadedFile uploadedFile;
public UploadedFile getFileUpload() {
return uploadedFile;
}
public void setFileUpload( UploadedFile to ) {
this.uploadedFile = to;
}
In the function that processes the bean data (e.g. a save function) you can check if a file was uploaded by checking if the object is null. If it's not null, a file was uploaded.
To process that uploaded file, first get an instance of a com.ibm.xsp.http.IUploadedFile object using the getServerFile() method. That object has a getServerFile() method that returns a File object for the uploaded file. The problem with that object is that it has a cryptic name (probably to deal with multiple people uploading files with the same name at the same time). The original file name can be retrieved using the getClientFileName() method of the IUploadedFile class.
What I then tend to do is to rename the cryptic file to its original file name, process it (embed it in a rich text field or do something else with it) and then rename it back to its original (cryptic) name. This last step is important because only then the file is cleaned up (deleted) after the code is finished.
Here's the sample code for the steps above:
import java.io.File;
import com.ibm.xsp.component.UIFileuploadEx.UploadedFile;
import com.ibm.xsp.http.IUploadedFile;
import lotus.domino.Database;
import lotus.domino.Document;
import lotus.domino.RichTextItem;
import com.ibm.xsp.extlib.util.ExtLibUtil; //only used here to get the current db
public void saveMyBean() {
if (uploadedFile != null ) {
//get the uploaded file
IUploadedFile iUploadedFile = uploadedFile.getUploadedFile();
//get the server file (with a cryptic filename)
File serverFile = iUploadedFile.getServerFile();
//get the original filename
String fileName = iUploadedFile.getClientFileName();
File correctedFile = new File( serverFile.getParentFile().getAbsolutePath() + File.separator + fileName );
//rename the file to its original name
boolean success = serverFile.renameTo(correctedFile);
if (success) {
//do whatever you want here with correctedFile
//example of how to embed it in a document:
Database dbCurrent = ExtLibUtil.getCurrentDatabase();
Document doc = dbCurrent.createDocument();
RichTextItem rtFiles = doc.createRichTextItem("files");
rtFiles.embedObject(lotus.domino.EmbeddedObject.EMBED_ATTACHMENT, "", correctedFile.getAbsolutePath(), null);
doc.save();
rtFiles.recycle();
doc.recycle();
//if we're done: rename it back to the original filename, so it gets cleaned up by the server
correctedFile.renameTo( iUploadedFile.getServerFile() );
}
}
}

I have code that processes an uploaded file in Java. The file is uploaded with the normal fileUpload control and then I call the following Java code from a button (that does a full refresh - so that the document including the uploaded file is saved). In the Java code you can do whatever checks you want (filename, filesize etc.):
public void importFile() {
facesContext = FacesContext.getCurrentInstance();
ExternalContext externalContext = facesContext.getExternalContext();
// get a handle an the uploaded file
HttpServletRequest request = (HttpServletRequest) externalContext.getRequest();
String fileUploadID = JSFUtil.findComponent("uploadFile").getClientId(FacesContext.getCurrentInstance());
UploadedFile uploadedFile = ((UploadedFile) request.getParameterMap().get(fileUploadID));
if (uploadedFile == null) {
facesContext.addMessage("messages1", new FacesMessage(FacesMessage.SEVERITY_ERROR, "No file uploaded. Use the file upload button to upload a file.", ""));
return;
}
File file = uploadedFile.getServerFile();
String fileName = uploadedFile.getClientFileName();
// Check that filename ends with .txt
if (!fileName.endsWith(".txt")) {
facesContext.addMessage("messages1", new FacesMessage(FacesMessage.SEVERITY_ERROR, "Error in uploaded file. The file must end with .txt", ""));
return;
}
try {
// Open the file
BufferedReader br;
br = new BufferedReader(new FileReader(file));
String strLine;
// Read File Line By Line
while ((strLine = br.readLine()) != null) {
// do stuff with the contents of the file
}
// Close the input stream
br.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
facesContext.addMessage("messages1", new FacesMessage(FacesMessage.SEVERITY_ERROR, "Error in uploaded file. Please check format of file and try again", ""));
return;
}
facesContext.addMessage("messages1", new FacesMessage(FacesMessage.SEVERITY_INFO, "File successfully uploaded", ""));
}
With a handle on the file object you can store the file in other documents using embedObject.

A better way to convert `File[]` to `List<String>`

I'm trying to list all folders in sdcard via ListAdapter. But I'm getting runtime exceptions # item.add(file.getName() + "/");. Is there a better way to convert File[] to List<String>?
private List <String> item = null;
String extState = Environment.getExternalStorageState();
if (!extState.equals(Environment.MEDIA_MOUNTED)) {
....
} else {
File sd = Environment.getExternalStorageDirectory();
FileFilter filterDirectoriesOnly = new FileFilter() {
public boolean accept(File file) {
return file.isDirectory();
}
};
File[] sdDirectories = sd.listFiles(filterDirectoriesOnly);
for (int i = 0; i < sdDirectories.length; i++) {
File file = sdDirectories[i];
item.add(file.getName() + "/");
}
ArrayAdapter <String> fileList = new ArrayAdapter <String>(this, R.layout.row, item);
setListAdapter(fileList);
}

You can make it a little shorter by using an enhanced for loop:
File[] sdDirectories = sd.listFiles(filterDirectoriesOnly);
for (File file : sdDirectories)
{
item.add(file.getName() + "/");
}
There's a method to get file names directly, but they won't have your trailing /, and you have to be able to filter by file name:
String[] fileNames = sd.list(fileNameFilter);
List<String> fileNamesList = Arrays.asList(fileNames);

You are getting runtime Exception bacause
private List item = null;
is never initialised..
so first initialise then add..
What ever you are doing seems just right.. except that initialization..

Initialize your list (hence the Exception). But apart from that, looks pretty much alright to me. I mean, it is atleast what I would do. The only thing I would recommend here is that you don't re-instantiate the variables File sd and FileFilter filterdirectoriesOnly within the function, because I assume you would be calling it again and again as the user clicks on a directory and then loads results for a new directory. So the two objects I mentioned are the same but being re-initialised every time you make a call. You could just move it outside.

Share uploaded file among servlets

Currently, I am having one jsp file, some java beans classes and two servlets.
The first servlet is responsible to upload a file and print out the context of it.
The second servlet is responsible for fetching the java beans code, execute it and print the result on jsp. However this concludes to duplicate code in servlets. Duplicated code is actually that the file need to be re-uploaded in order to call beans:
FileItemFactory factory = new DiskFileItemFactory();
ServletFileUpload upload = new ServletFileUpload(factory);
Iterator<FileItem> iterator = upload.parseRequest(request).iterator();
File uploadedFile = null;
String dirPath = "C:\\fileuploads";
while (iterator.hasNext()) {
FileItem item = iterator.next();
if (!item.isFormField()) {
String fileNameWithExt = item.getName();
File filePath = new File(dirPath);
if (!filePath.exists()) {
filePath.mkdirs();
}
uploadedFile = new File(dirPath + "/" + fileNameWithExt);
item.write(uploadedFile);
} else {
String otherFieldName = item.getFieldName();
String otherFieldValue = item.getString();
}
}
FileInputStream fstream = new FileInputStream(uploadedFile);
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
Next there is code which connects the servlet with the java beans. This works but my question is what is the best way to share this uploaded file? If I can store the file path in a variable and call it from the first servlet to second with no duplicated code.
Thanks in advance.
P.S I ve read this question as well, Share uploaded file between servlets in session , but i didnt really manage to do it.

If I can store the file path in a variable and call it from the first servlet to second with no
duplicated code.
So you would be just getting the file path and from second servlet you would be reading file again.
session.setAttribute("filePath",yourCalculatedFilePath);
and retrieve it from different servlet using
session.getAttribute("filePath");
You can just set the filePath in session attribute and you can access it across the session. but putting whole file into session isn't a good idea just imagine a user puts a file of size 1MB and there are 1000 users online at a time (just example) it would cost 1GB of server's memory.

Commons File Upload Not Working In Servlet

I have a servlet which is meant to handle the upload of a very large file. I am trying to use commons fileupload to handle it. Currently, the file I am attempting to upload is 287MB.
I set up the FileItemFactory and ServletFileUpload, then set a very large max file size on the ServletFileUpload.
Unfortunately, when I attempt to create a FileItemIterator, nothing happens. The form is set with the correct action, multipart encoding, and for the POST method.
Can anyone assist? doPost() of the servlet is posted below:
#Override
protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
// ensure that the form is multipart encoded since we are uploading a file
if (!ServletFileUpload.isMultipartContent(req)) {
//throw new FileUploadException("Request was not multipart");
log.debug("Request was not multipart. Returning from call");
}
// create a list to hold all of the files
List<File> fileList = new ArrayList<File>();
try {
// setup the factories and file upload stuff
FileItemFactory factory = new DiskFileItemFactory();
ServletFileUpload upload = new ServletFileUpload(factory);
upload.setFileSizeMax(999999999);
// create a file item iterator to cycle through all of the files in the req. There SHOULD only be one, though
FileItemIterator iterator = upload.getItemIterator(req);
// iterate through the file items and create a file item stream to output the file
while (iterator.hasNext()) {
// get the file item stream from the iterator
FileItemStream fileItemStream = iterator.next();
// Use the Special InputStream type, passing it the stream and the length of the file
InputStream inputStream = new UploadProgressInputStream(fileItemStream.openStream(), req.getContentLength());
// create a File from the file name
String fileName = fileItemStream.getName(); // this only returns the filename, not the full path
File file = new File(tempDirectory, fileName);
// add the file to the list
fileList.add(file);
// Use commons-io Streams to copy from the inputstrea to a brand-new file
Streams.copy(inputStream, new FileOutputStream(file), true);
// close the inputstream
inputStream.close();
}
} catch (FileUploadException e) {
e.printStackTrace();
}
// now that we've save the file, we can process it.
if (fileList.size() == 0) {
log.debug("No File in the file list. returning.");
return;
}
for (File file : fileList) {
String fileName = file.getName();
BufferedReader reader = new BufferedReader(new FileReader(fileName));
String line = reader.readLine();
List<Feature> featureList = new ArrayList<Feature>(); // arraylist may not be the best choice since I don't know how many features I'm importing
while (!line.isEmpty()) {
String[] splitLine = line.split("|");
Feature feature = new Feature();
feature.setId(Integer.parseInt(splitLine[0]));
feature.setName(splitLine[1]);
feature.setFeatureClass(splitLine[2]);
feature.setLat(Double.parseDouble(splitLine[9]));
feature.setLng(Double.parseDouble(splitLine[10]));
featureList.add(feature);
line = reader.readLine();
}
file.delete(); // todo: check this to ensure it won't blow up the code since we're iterating in a for each
reader.close(); // todo: need this in a finally block somewhere to ensure this always happens.
try {
featureService.persistList(featureList);
} catch (ServiceException e) {
log.debug("Caught Service Exception in FeatureUploadService.", e);
}
}
}

It was an incredibly stupid problem. I left the name attribute off of the FileUpload entry in the GWT UiBinder. Thanks for all of the help from everyone.

Are the only request parameters available File items? Because you may want to put in a check:
if (!fileItemStream.isFormField()){
// then process as file
otherwise you'll get errors. On the surface of things your code looks fine: no errors in the Tomcat logs?

You need to add enctype='multipart/form-data' in html form