Does anyone know how to see the progress (in percent) of an upload in a multipart upload in Amazon S3?
I would do it like this:
MultipleFileUpload transfer = transferManager.uploadDirectory(mybucket, null, new File(localSourceDataFilesPath), false);
// blocks the thread until the upload is completed
showTransferProgress(transfer);
Then in showTransferProgress, I would create a block the upload using a sleep, and do the math every X seconds:
private void showTransferProgress(MultipleFileUpload xfer) {
while (!xfer.isDone()) {
// some logic to wait so you don't do the math every second like a Thread.sleep
TransferProgress progress = xfer.getProgress();
long bytesTransferred = progress.getBytesTransferred();
long total = progress.getTotalBytesToTransfer();
Double percentDone = progress.getPercentTransferred();
LOG.debug("S3 xml upload progress...{}%", percentDone.intValue());
LOG.debug("{} bytes transferred to S3 out of {}", bytesTransferred, total);
}
// print the final state of the transfer.
TransferState xferState = xfer.getState();
LOG.debug("Final transfer state: " + xferState);
}
this line is what you are looking for:
Double percentDone = progress.getPercentTransferred();
Hi guys here is is my final version
private void awsHoldUntilCompletedAndShowTransferProgress(Upload upload) throws InterruptedException, AmazonClientException {
TransferProgress tProgress = upload.getProgress();
long totalSize = tProgress.getTotalBytesToTransfer();
long bPrevious = 0;
int timerSec = Math.toIntExact(Math.round(Math.sqrt(totalSize / 1024 / 1024) / 4));// calculate based on file size
if (timerSec > 60) {// no longer than 60 second per loop
timerSec = 60;
}
while (!upload.isDone()) {
long bTransferred = tProgress.getBytesTransferred();
String strMbps = Double.valueOf((((bTransferred - bPrevious) / timerSec) / 1024) / 1024).toString() + " MBps";
String strTransfered = bTransferred + " bytes";
if (bTransferred > 1024) {
strTransfered = Double.valueOf((bTransferred / 1024) / 1024).toString() + " MB";
}
log.info("Upload progress: "
+ strTransfered
+ " / "
+ FileUtils.byteCountToDisplaySize(totalSize) + " - "
+ Math.round(tProgress.getPercentTransferred()) + "% "
+ strMbps);
bPrevious = bTransferred;
TimeUnit.SECONDS.sleep(timerSec);
}
Transfer.TransferState transferState = upload.getState();
log.info("Final transfer state: " + transferState);
if (transferState == Transfer.TransferState.Failed) {
throw upload.waitForException();
}
}
and here is where I call the code above from
..stuff...
TransferManager tm = TransferManagerBuilder
.standard()
.withS3Client(s3Client)
.build();
LocalDateTime uploadStartedAt = LocalDateTime.now();
log.info("Starting to upload " + FileUtils.byteCountToDisplaySize(fileSize));
Upload up = tm.upload(bucketName, file.getName(), file);
awsHoldUntilCompletedAndShowTransferProgress(up);
log.info("Time consumed: " + DurationFormatUtils.formatDuration(Duration.between(uploadStartedAt, LocalDateTime.now()).toMillis(), "dd HH:mm:ss"));
this works, but may be a better way to do this
TransferManager tm = TransferManagerBuilder
.standard()
.withS3Client(s3Client)
.build();
PutObjectRequest request = new PutObjectRequest(bucketName, file.getName(), file);
ProgressListener progressListener = new ProgressListener() {
#Override
public void progressChanged(com.amazonaws.event.ProgressEvent progressEvent) {
bytesUploaded += progressEvent.getBytesTransferred();// add counter
if (bytesUploaded > byteTrigger) {
if ((bytesUploaded + sizeRatio) < fileSize) {
byteTrigger = bytesUploaded + sizeRatio ;
} else {
byteTrigger = bytesUploaded + (sizeRatio / 6);// increase precision approaching the end
}
String percent = new DecimalFormat("###.##").format(bytesUploaded * 100.0 / fileSize);
log.info("Uploaded: " + FileUtils.byteCountToDisplaySize(bytesUploaded) + " - " + percent + "%");
}
}
};
request.setGeneralProgressListener(progressListener);
Upload upload = tm.upload(request);
log.info("starting upload");
upload.waitForUploadResult();
log.info("Upload completed");
I am trying to write the results of my program in a file but i do not know why its does not write nothing.
I have done a program and it creates a file but when I open the file, it is empty. What i have done wrong?
MongoClient mongoClient;
DB db;
mongoClient = new MongoClient("localhost", 27017);
db = mongoClient.getDB("behaviourDB_areas");
DBCollection cEvent = db.getCollection("events_Searching");
File file = new File ("C:\\Users\\Nikos\\Documents\\Apotelesmata\\file1.txt");
file.getParentFile().mkdirs();
PrintWriter writer = new PrintWriter (file);
BasicDBObject orderBy = new BasicDBObject();
orderBy.put("timeStamp",1);
DBCursor cursorEvents = null;
BasicDBObject searchQuery = new BasicDBObject();
searchQuery.put("user_id", "55b20db905f333defea9827f");
cursorEvents = cEvent.find(searchQuery).sort(orderBy);
int count = 1;
int start = 1;
String timeStartOld = null;
while (cursorEvents.hasNext()) {
DBObject documentInEventCollection = cursorEvents.next();
if("pageLoad".equals(documentInEventCollection.get("type"))){
writer.println("URL(" + count + "): " + documentInEventCollection.get("url").toString());
//System.out.println("time-start(" + start + "): " + documentInEventCollection.get("timeStamp").toString());
count++;
start++;
try {
String timeStart = (documentInEventCollection.get("timeStamp").toString());
if(timeStartOld==null){
timeStartOld = timeStart;
continue;
}
SimpleDateFormat format = new SimpleDateFormat("yyyy-dd-MM;HH:mm:ss");
Date d1 = null;
Date d2 = null;
d1 = format.parse(timeStartOld);
d2 = format.parse(timeStart);
//in milliseconds
long diff = d2.getTime() - d1.getTime();
long diffSeconds = diff / 1000 % 60;
long diffMinutes = diff / (60 * 1000) % 60;
long diffHours = diff / (60 * 60 * 1000) % 24;
long diffDays = diff / (24 * 60 * 60 * 1000);
writer.println(diffDays + " days, ");
writer.println(diffHours + " hours, ");
writer.println(diffMinutes + " minutes, ");
writer.println(diffSeconds + " seconds.");
timeStartOld = timeStart;
} catch (Exception e) {
e.printStackTrace();
}
}
writer.close();
}
mongoClient.close();
Thank you for your answer.
Try if the below statement is getting printed in console.
System.out.println("time-start(" + start + "): " + documentInEventCollection.get("timeStamp").toString());
If not, execution is not entering the below if statement. Try some conditions where the if gets true.
if("pageLoad".equals(documentInEventCollection.get("type"))){ ...
I'm only asking this as a last resort. I'm stumped.
I have written a small app which performs very simple image processing. It is made using JDK 1.8, written in Netbeans 8.0.1 and is running on Debian Linux.
The application captures a large number of individual frames at a certain framerate, set by the user, by calling the 'streamer' webcam program via a process builder. Once it has begun capturing, it begins to translate the frames into RGB values and checks whether or not any pixels are above a user defined threshold. If no pixels exceed this threshold, it simply deletes the frame. If any pixels do exceed it, it moves the frame to a different folder for user inspection. This all works fine. It keeps up with even relatively high framerates and selects appropriate frames as expected. However, when the number of frames processed reaches around 1500 (or fewer for lower framerates), the program is freezing. I've tried issuing the commands to streamer manually at the command line, and it seems perfectly capable of producing as many as required, so I have to assume the issue is with my coding. The images are only small (320x240). Am I somehow maxxing out the available memory (I am getting no errors, just freezing).
The purpose of this program is to detect cosmic ray impacts on a CMOS sensor, part of a friend's dissertation. If I can't get this working reliably, the poor kid's going to have to go through the footage manually!
The code is attached below. Apologies for the length of the code, but as all of it is fairly crucial, I didn't want to omit anything.
package cosmicraysiii;
import java.io.File;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.swing.JOptionPane;
public class CosmicRaysIII {
public static int dark = 0;//Dark current determined by averaging
public static int tol = 0;//Tolerance set by user
public static int frames = 0;//Total number of frames set by user
public static int runs = 0;
public static int rate = 0;//Framerate set by user
public static int time = 0;//Total time calculated
public static String zeros = "";
public static int ready = 0;
public static void main(String[] args) throws IOException, InterruptedException {
//Get directory ID from user
String id = JOptionPane.showInputDialog("Enter a folder name for detections (no ., in name):");
//Get dark current
Dark d = new Dark();
dark = d.getCurrent();
//Get tolerance from user, will be added to dark current.
String t = JOptionPane.showInputDialog("Dark Current = " + dark + "/255\n"
+ "Enter a tolerance (integer values only).\n "
+ "This will be added to the dark current:");
tol = Integer.parseInt(t) + dark;
//Get number of frames from user
String fs = JOptionPane.showInputDialog("Enter the total number of frames required (Mulitples of 500 only):");
frames = Integer.parseInt(fs);
runs = frames / 500;
//Get framerate from user
String r = JOptionPane.showInputDialog("Enter the framerate required:");
rate = Integer.parseInt(r);
//Determine duration
time = (int) Math.round(frames / rate);
//Provide summary for user and request permission to continue
int secs = time % 60;
int mins = (time - secs) / 60;
int hrs = (mins - (mins % 60)) / 60;
if (hrs >= 1) {
mins = mins % 60;
}
String theMessage = "The following parameters have been set:\n"
+ "Tolerance (including dark current): " + tol + "\n"
+ "Frames: " + frames + "\n"
+ "Frame rate: " + rate + " fps\n"
+ "Total capture time: " + time + " sec\n"
+ " " + hrs + " h " + mins + " m " + secs + " s\n"
+ "\n"
+ "Would you like to proceed?";
int result = JOptionPane.showConfirmDialog(null, theMessage, "Continue?", JOptionPane.OK_CANCEL_OPTION);
if (result == 2) {
System.exit(0);
}
//Create directory for data acquisition
ProcessBuilder pb1 = new ProcessBuilder("mkdir", "data");
pb1.start();
//Establish array of filenames
String[] filenames = new String[frames];
//Fill filenames array with filenames
//Taking into consideration that the filename
//will have a varying number of zeros appended
//before the frame number, dependent on the
//order of the frame number
for (int i = 0; i < frames; i++) {
if (i < 10) {
zeros = "00000000";
} else if (i >= 10 && i < 100) {
zeros = "0000000";
} else if (i >= 100 && i < 1000) {
zeros = "000000";
} else if (i >= 1000 && i < 10000) {
zeros = "00000";
} else if (i >= 10000 && i < 100000) {
zeros = "0000";
} else if (i >= 100000 && i < 1000000) {
zeros = "000";
} else if (i >= 1000000 && i < 10000000) {
zeros = "00";
} else if (i >= 10000000 && i < 100000000) {
zeros = "0";
} else {
zeros = "";
}
filenames[i] = "./data/frame" + zeros + i + ".ppm";
}
//Begin data acquisition
new Thread(new Runnable() {
public void run() {
try {
//Capture images
ProcessBuilder pb2 = new ProcessBuilder("streamer", "-t", Integer.toString(frames), "-r", Integer.toString(rate), "-p", "0", "-o", "./data/frame000000000.ppm");
Process p = pb2.start();
p.waitFor();
ready = 1;
} catch (IOException | InterruptedException ex) {
Logger.getLogger(CosmicRaysIII.class.getName()).log(Level.SEVERE, null, ex);
}
}
}).start();
//Sleep to allow some image capture to prevent thread disordering
Thread.sleep(3000);
//Check array size
System.out.println("Array size: " + filenames.length);
//Conduct image analysis
new Thread(new Runnable() {
public void run() {
int done = 0;
int donea = 0;
while (ready == 0) {
for (int i = 0; i < frames; i++) {
File f = new File(filenames[i]);
if (f.exists() && !filenames[i].equals("")) {//Check file still exists
try {
//Perform analysis steps
Analysis a = new Analysis();
//STEP 1: Convert file from P6 to P3
String newfile = a.convert(filenames[i], zeros, i);
//STEP 2: Read file
a.read(newfile, tol, i, id);
filenames[i] = "";
done++;
} catch (IOException ex) {
Logger.getLogger(CosmicRaysIII.class.getName()).log(Level.SEVERE, null, ex);
}
}
if (done > donea) {
System.out.println(done + " files processed");
donea = done;
}
}
}
}
}).start();
}
}
Then the Analyse.java class is as follows:
package cosmicraysiii;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
public class Analysis {
public String convert(String ofile, String zeros, int number) throws IOException {
//Create new file name
String nfile = "./proc/frame" + zeros + number + ".ppm";
//Ensure process directory exists
ProcessBuilder mkdir = new ProcessBuilder("mkdir", "proc");
mkdir.start();
//Convert file to P3 PPM (RGB format) and move to process folder
ProcessBuilder convert = new ProcessBuilder("convert", ofile, "-compress", "none", nfile);
convert.start();
//Delete original file
ProcessBuilder del = new ProcessBuilder("sudo", "rm", ofile);
del.start();
//Return new filename
return nfile;
}
public void read(String filename, int tol, int ix, String id) throws FileNotFoundException, IOException {
int move = 0;
//Make directory for hits
ProcessBuilder mkdir = new ProcessBuilder("mkdir", "hits" + id);
mkdir.start();
//Open reader to read file
File f = new File(filename);
if (f.exists()) {
BufferedReader br = new BufferedReader(new FileReader(filename));
String line;
//To eliminate header
int x = 0;
//Iterate through text to find abnormal pixels
while ((line = br.readLine()) != null) {
x++;
String[] pixrgb = line.split("\\ ");
//Iterate through pixels on each line
for (int i = 0; i < pixrgb.length; i++) {
if (x >= 4) {//Eliminate header
//Check each pixel value
try {
int pixval = Integer.parseInt(pixrgb[i]);
if (pixval > tol) {
move = 1;
break;
}
} catch (NumberFormatException ne) {
}
}
}
}
if (move == 1) {
//Move file to hits folder
ProcessBuilder pb3 = new ProcessBuilder("sudo", "cp", filename, "./hits" + id + "/detection" + ix + ".ppm");
pb3.start();
//Delete original file
ProcessBuilder del = new ProcessBuilder("sudo", "rm", filename);
del.start();
}
}
//Delete original file
ProcessBuilder del = new ProcessBuilder("sudo", "rm", filename);
del.start();
}
}
I appreciate this is quite a lengthly chunk of code to be posting. Really appreciate any help that can be given.
G
OK I have managed to solve this by completely overhauling the analysis process. Firstly, rather than converting the image into a P3 .ppm file, I now examine the pixels directly from the image using BufferedReader. Then, I stopped looping through the file list repeatedly. Instead, the loop which calls Analysis() just runs through the list of filenames once. If it encounters a file which does not yet exist, it does Thread.sleep(500) and then tries again. I have now successfully run a batch of 50,000 frames without incident, and there is now much less of a drain on memory thanks to the improved process. Just thought I should place this answer up here in case anyone comes across it. I may post code if anyone wants it.
I need to display lots of files with filename and icon in my program.
Therefor I am extracting the icons from the files, but it takes too long.
I have tried 2 different methods to extract the icons, but both are really slow (in my case REALLY slow, because I get the files from a networkdrive).
Here is an example, where I extract the icons and count the number of icons (do nothing with the files/icons)
public class Main {
public static void main(String[] args) {
File folder = new File("C:\\Windows\\System32\\");
File[] list = folder.listFiles();
for(int i = 0; i< 3; i++) {
long startTime = System.currentTimeMillis();
System.out.println("Method 1: " + getIconNumber1(list)+ " Icons");
long stopTime = System.currentTimeMillis();
long elapsedTime = stopTime - startTime;
System.out.println("Finished Method 1 in " + (float) elapsedTime / 1000 + "sec");
long startTime2 = System.currentTimeMillis();
System.out.println("Method 2: " + getIconNumber2(list)+ " Icons");
long stopTime2 = System.currentTimeMillis();
long elapsedTime2 = stopTime2 - startTime2;
System.out.println("Finished Method 2 in " + (float) elapsedTime2 / 1000 + "sec");
System.out.println("-----------------");
}
}
private static int getIconNumber1(File[] list) {
int nr = 0;
for(File f : list) {
try {
ShellFolder sf = ShellFolder.getShellFolder(f);
ImageIcon icon = new ImageIcon(sf.getIcon(true));
nr++;
} catch (Exception e) {
e.printStackTrace();
}
}
return nr;
}
private static int getIconNumber2(File[] list) {
int nr = 0;
for(File f : list){
FileSystemView view = FileSystemView.getFileSystemView();
Icon icon = view.getSystemIcon(f);
nr++;
}
return nr;
}
}
Is there a faster way to do this?
Thank you in advance for your help. I am developing a java based tool that is preforming some database work. I have a very simple problem. For some reason the time reported to complete the task is incorrect.
public static void makeDatabaseThreaded() throws IOException, InterruptedException {
final long startTime = System.nanoTime();
ArrayList<String> tablesMade = new ArrayList<>();
File rootDirectory = root;
String[] files = rootDirectory.list();
double percentDone = 0;
double numOfTablesMade = 0;
double numberOfTables = 62.0;
DatabaseBuilderThread lastThread = null;
for (int i = 0; i <= files.length - 1; i++) {
if (!files[i].contains(".csv")) {
continue;
}
File file = new File(rootDirectory + File.separator + files[i]);
String tableName = getTableNameFromFile(file);
if (!tablesMade.contains(tableName)) {
tablesMade.add(tableName);
DatabaseBuilderThread thread = new DatabaseBuilderThread(i, file);
lastThread = thread;
thread.start();
threadsRunning++;
numOfTablesMade++;
percentDone = (int) (100.0 * (numOfTablesMade) / (numberOfTables));
while (threadsRunning > 10) {
Thread.sleep(1000);
}
System.out.println(percentDone + "% done. Making Table For File: " + file.getName());
}
}
//Make Sure all threads are done
lastThread.join();
final long endTime = System.nanoTime();
final long duration = endTime - startTime;
Time time = new Time(duration);
System.out.println("Done Making The Database. It took " + time.toString());
}
The program reports that it worked about twice as long at it truly did for the cases that I ran.
Thanks
System.nanoTime() returns time values in nanoseconds. Time() takes a value in milliseconds as a parameter. This would throw your time value off by a factor of 10^-6.
Time takes milliseconds as a constructor parameter, where as nanoTime() gives you nanoseconds precision, could that be the problem?
discussion here: System.currentTimeMillis vs System.nanoTime