Develop Reference

How to append batches of stream data to one CSV file using Java NIO Package

I am reading data from API response in batches of bytes which is of Content-Type = text/CSV and using Java's NIO package to transfer bytes between two Channels. I want to write API responses in batches to the same File (APPEND). With below code append doesn't seem to work correctly, it's more of overriding results.
And once all the data is written then I also want to print the number of total lines in CSV.
Version - Java 8
private void downloadFile_NIO(String encoded_token) throws Exception
{
long start_Range = 0;
long end_Range = 5000;
long batch_size = 5000;
long totalsize = 1080612174;
long counter = 0;
//Path path = Paths.get(FILE_NAME);
//long lines = 0;
while (start_Range <= totalsize)
{
URL url = new URL(FILE_URL);
HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
connection.setRequestProperty("Authorization", "Bearer " + encoded_token );
connection.setDoOutput(true);
connection.setRequestMethod("GET");
ReadableByteChannel readableByteChannel = Channels.newChannel(connection.getInputStream());
FileOutputStream fileOutputStream=new FileOutputStream(new File(FILE_NAME),true);
fileOutputStream.getChannel().transferFrom(readableByteChannel, 0, Long.MAX_VALUE);
// lines = Files.lines(path).count();
// System.out.println("lines->" + lines);
System.out.println();
fileOutputStream.close();
readableByteChannel.close();
counter = counter + 1;
if (counter < 2)
{
start_Range = start_Range + batch_size + 1;
end_Range = end_Range + batch_size;
}
else
{
start_Range = start_Range + batch_size;
end_Range = end_Range + batch_size;
}
}
}

Unable to write to a file till some specific line

I am trying to split files from one file to 4 different files. So I am dividing the file by some "x" value and wanna write the file till that value and from there to next file continues till the file contents ends.
I am checking some x value in the file using buffer reader and checking with the content is equal to the x value and do the splitting.
Splitting is coming but in some another way, like it's reading the file and writing till the line number which is "x". But I need all the lines till that "x" value is present in the file.
I have a time in the file like start time hh:mm:ss and I am checking this with the hh:mm:ss with my x value and do the splitting like below
// inputs to the below method
// filePath = "//somepath";
// splitlen = 30;
// name ="somename"; */
public void split(String FilePath, long splitlen, String name) {
long leninfile = 0, leng = 0;
int count = 1, data;
try {
File filename = new File(FilePath);
InputStream infile = new BufferedInputStream(new FileInputStream(filename));
data = infile.read();
BufferedReader br = new BufferedReader(new InputStreamReader(infile));
while (data != -1) {
filename = new File("/Users//Documents/mysrt/" + count + ".srt");
OutputStream outfile = new BufferedOutputStream(new FileOutputStream(filename));
String strLine = br.readLine();
String[] atoms = strLine.split(" --> ");
if (atoms.length == 1) {
// outfile.write(Integer.parseInt(strLine + "\n"));
}
else {
String startTS = atoms[0];
String endTS = atoms[1];
System.out.println(startTS + "\n");
System.out.println(endTS + "\n");
String startTime = startTS.replace(",", ".");
String endTime = endTS.replace(",", ".");
System.out.println("startTime" + "\n" + startTime);
System.out.println("endTime" + "\n" + endTime);
String [] arrOfStr = endTime.split(":");
System.out.println("=====arrOfStr=====");
int x = Integer.parseInt(arrOfStr[1]);
System.out.println(arrOfStr[1]);
System.out.println("===x repeat==");
System.out.println(x);
System.out.println("===splitlen repeat==");
System.out.println(splitlen);
System.out.println(data);
System.out.println(br.readLine());
System.out.println(br.read());
while (data != -1 && x < splitlen) {
outfile.write(br.readLine().getBytes());
data = infile.read();
x++;
}
System.out.println("===== out of while x =====");
System.out.println(br.readLine());
System.out.println(x);
leninfile += leng;
leng = 0;
outfile.close();
firstPage = false;
firstPage = true;
count++;
splitlen = splitlen + 30;
System.out.println("=====splitlen after=====" +splitlen);
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
I am incrementing the time with some number to read the next lines in file and with into another file.
Here splitlen is 30 , so it's writing the data till 30 lines in a new file. Then it's incrementing splitlen+30 i.e 60. But, it's reading next 60 lines and writing into next file.
But I need to check this splitlen with the time provided in the content of file and I should split that line.
Please suggest me where I am doing wrong. If you provide snippet it will be appreciated.
Thanks.

I think this is what you want
public void split(String filePath, long splitLen, String name) {
File fileSource = new File(filePath);
int count = 0;
boolean endOfFile = false;
String lineSeparator = System.getProperty("line.separator");
int hour = 0; // an accumulator for hours
int min = 0; // an accumulator for minutes
int sec = (int) splitLen; // an accumulator for seconds
int _hour = 0; // hours from the file
int _min = 0; // minutes from the file
int _sec = 0; // seconds from the file
try ( // try with resources to close files automatically
FileReader frSource = new FileReader(fileSource);
BufferedReader buffSource = new BufferedReader(frSource);
) {
String strIn = null;
while(!endOfFile) {
File fileOut = new File("f:\\test\\mysrt\\" + count + ".srt");
try ( // try with resources to close files automatically
FileWriter fwOut = new FileWriter(fileOut);
) {
if (strIn != null) {
// write out the last line read to the new file
fwOut.write(strIn + lineSeparator);
}
for (int i = 0; i < splitLen; i++) {
strIn = buffSource.readLine();
if (strIn == null) {
endOfFile = true; // stop the while loop
break; // exit the for loop
}
if (strIn.indexOf("-->") > 0) {
String endTime = strIn.split("-->")[1];
_hour = extractHours(endTime); // get the hours from the file
_min = extractMinutes(endTime); // get the minutes from the file
_sec = extractSeconds(endTime); // get the seconds from the file
if (_hour >= hour && _min >= min && _sec >= sec) { // if the file time is greater than our accumulators
sec += splitLen; // increment our accumulator seconds
if (sec >= 60) { // if accumulator seconds is greater than 59, we need to convert it to minutes and seconds
min += sec / 60;
sec = sec % 60;
}
if (min >= 60) { if accumulator minutes is greater than 59, we need to convert it to hours and minutes
hour += min / 60;
min = min % 60;
}
break; // break out of the for loop, which cause the file to be completed and a new file started.
}
}
fwOut.write(strIn + lineSeparator); // write out to the new file
}
fwOut.flush();
}
count++;
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
private int extractMinutes(String time) {
// You need to implement this, I don't know the format of your time
return 0;
}
private int extractSeconds(String time) {
// You need to implement this, I don't know the format of your time
return 0;
}

The problem with your code is that the timestamp you're looking at is in HH:MM:ss but with the splitlen and x variables you are only working with minutes.
So you need to keep track of both hours and minutes, maybe this could be done with some DateTime class but here is a simple int solution
//somewhere at the top
int hour = 0;
int minutes = 30;
//where you today increase splitlen
minutes += 30;
if (minutes == 60) {
hour++;
minutes = 0;
}
//parse also hours
int y = Integer.parseInt(arrOfStr[0]);
int x = Integer.parseInt(arrOfStr[1]);
//you need to rewrite this to compare x and y against hour and minutes
while (data != -1 && x < splitlen) {
So now you will not be looking for 30, 60, 90,... minutes but instead 00:30, 01:00, 01:30 and so on. Of course you must also be prepared to handle the situation where there is no entry for a whole minute unless of course you already do so.
checkTime is of course a a key method here and it might be a good idea to make the last hour and minute when the file was split into class members but they could of course also be sent as parameters from split().
Update
Here is a simplified version of the split method to give an example on how to solve this, it is not complete but should be a good starting point for solving the issue. I try to make use of how a .str file is constructed and make use of the logic explained above for determining when to open a new output file.
public void split(String filepath, long splitlen, String name) {
int count = 1;
try {
File filename = new File(filepath);
InputStream infile = new BufferedInputStream(new FileInputStream(filename));
BufferedReader br = new BufferedReader(new InputStreamReader(infile));
FileWriter outfile = createOutputFile(count);
boolean isEndOfFile = false;
while (!isEndOfFile) {
String line = null;
int i = 1;
while ((line = br.readLine()) != null) {
outfile.write(line);
if (line.trim().isEmpty()) { //last line of group
i = 1;
continue;
}
if (i == 2) { //Timestamp row
String[] split = line.split("-->");
if (checkTime(split)) {
count++;
outfile.flush();
outfile.close();
outfile = createOutputFile(count);
}
}
i++;
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
private FileWriter createOutputFile(int index) {
//Create new outputfile and writer
return null;
}
private boolean checkTime(String[] arr) {
//use start or end time in arr to check if an even half or full hour has been passed
return true;
}

Is this a net time for transporting a file?

I made an app to calculate file transporting time.
Here is my code:
for (File srcfile : fList) {
srcFileName = srcfile.getName();
sizeInMB = (srcfile.length() / (1024*1024));
srcFilePath = srcPath + "/" + srcFileName;
try
{
fis = new FileInputStream(srcFilePath);
tmp = fis.available();
bufDynmc = new byte[tmp];
while (true)
{
readStartTime = System.currentTimeMillis();
bytesRead = fis.read(bufDynmc);
readEndTime = System.currentTimeMillis();
readTimeSum += (readEndTime - readStartTime);
if (bytesRead > 0)
{
if(rsltCode == RESULT_OK)
{
treeUri = rsltData.getData();
DocumentFile pickedDir = DocumentFile.fromTreeUri(this, treeUri);
DocumentFile newFile = pickedDir.createFile("image/jpg", srcFileName);
fos = getContentResolver().openOutputStream(newFile.getUri());
writeStartTime = System.currentTimeMillis();
fos.write(bufDynmc, 0, bytesRead);
writeEndTIme = System.currentTimeMillis();
writeTimeSum += (writeEndTIme - writeStartTime);
}
continue;
} else break;
}
sizeSum += sizeInMB;
fis.close(); fos.close();
} catch (Exception e) {e.printStackTrace();}
rescanGallery(treeUri);
}
What I want to measure is how much time has been spend from source directory to RAM ,
and from RAM to destination directory.
Am I measuring the reading time and writing time with right method?
I'm not sure cuz this program gives me a time which is too fast.

Android - how to calculate the time needed to download a file?

In my Android application, I am using WIFI link speed to get the speed of the WIFI
and get the length content of a file before downloading the file
and then I am trying to get the esitmated time of download before downloading the file
but the time I get is incorrect I don't know why !
that is my code to estimate the time before downloading
URL u = new URL(url);
HttpURLConnection c = (HttpURLConnection) u.openConnection();
c.setRequestMethod("GET");
c.connect();
contentLength = Long.parseLong(c.getHeaderField("Content-Length"));
System.out.println("content"+contentLength);
float contentLength_float=contentLength/(float)(1000*1000);//migabyte
float speed=((float)(mActivity.speed_wifi()))/(float)8;//convert mbps(migabit) to migabyte ps
float sec=contentLength_float/speed;//get the sec from m/m/s
and function wifi speed ()
public int speed_wifi()
{
WifiManager mainWifi;
mainWifi = (WifiManager) getSystemService(Context.WIFI_SERVICE);
WifiInfo wifiInfo = mainWifi.getConnectionInfo();
int speed=0;
if(wifiInfo.getBSSID()!=null)
{
speed=wifiInfo.getLinkSpeed();
}
return speed;
}

The wifi link speed you get by using that function is the maximum speed that can be achieved by the wifi in the phone, it is not the actual speed.
There is no way of determining the wifi speed before the download starts.
What you can do is that, start showing the estimated time as the download is started based on the current download speed. For this -
find out how much data is downloaded in a small chunk of time like 2 sec which will be current_speed = data_downloaded/time (time can be 2 sec or anything you want)
Now the estimated time will be file_size/current_speed.
So in this way you can start showing the estimated time just 1 or 2 seconds after the download is started.

use AysnTask
InputStream input = null;
OutputStream output = null;
HttpURLConnection connection = null;
try {
URL url = new URL(sUrl[0]);
connection = (HttpURLConnection) url.openConnection();
connection.connect();
// expect HTTP 200 OK, so we don't mistakenly save error report
// instead of the file
if (connection.getResponseCode() != HttpURLConnection.HTTP_OK) {
return "Server returned HTTP " + connection.getResponseCode()
+ " " + connection.getResponseMessage();
}
// this will be useful to display download percentage
// might be -1: server did not report the length
int fileLength = connection.getContentLength();
// download the file
input = connection.getInputStream();
output = new FileOutputStream("/sdcard/file_name.extension");
byte data[] = new byte[4096];
long total = 0;
int count;
while ((count = input.read(data)) != -1) {
// allow canceling with back button
if (isCancelled()) {
input.close();
return null;
}
total += count;
// publishing the progress....
if (fileLength > 0) // only if total length is known
publishProgress((int) (total * 100 / fileLength));
output.write(data, 0, count);
}
} catch (Exception e) {
return e.toString();
} finally {
try {
if (output != null)
output.close();
if (input != null)
input.close();
} catch (IOException ignored) {
}
if (connection != null)
connection.disconnect();
}

Java - Xuggle - Best method to get a frame

I'm working with xuggle since one week and I wrote a method to get a
frame by a video but if video is long this method takes too much time:
public static void getFrameBySec(IContainer container, int videoStreamId, IStreamCoder videoCoder, IVideoResampler resampler, double sec)
{
BufferedImage javaImage = new BufferedImage(videoCoder.getWidth(), videoCoder.getHeight(), BufferedImage.TYPE_3BYTE_BGR);
IConverter converter = ConverterFactory.createConverter(javaImage, IPixelFormat.Type.BGR24);
IPacket packet = IPacket.make();
while(container.readNextPacket(packet) >= 0)
{
if (packet.getStreamIndex() == videoStreamId)
{
IVideoPicture picture = IVideoPicture.make(videoCoder.getPixelType(), videoCoder.getWidth(), videoCoder.getHeight());
int offset = 0;
while(offset < packet.getSize())
{
int bytesDecoded = videoCoder.decodeVideo(picture, packet, offset);
if (bytesDecoded < 0)
throw new RuntimeException("got error decoding video");
offset += bytesDecoded;
if (picture.isComplete())
{
IVideoPicture newPic = picture;
if (resampler != null)
{
newPic = IVideoPicture.make(resampler.getOutputPixelFormat(), picture.getWidth(), picture.getHeight());
if (resampler.resample(newPic, picture) < 0)
throw new RuntimeException("could not resample video from");
}
if (newPic.getPixelType() != IPixelFormat.Type.BGR24)
throw new RuntimeException("could not decode video as RGB 32 bit data in");
javaImage = converter.toImage(newPic);
try
{
double seconds = ((double)picture.getPts()) / Global.DEFAULT_PTS_PER_SECOND;
if (seconds >= sec && seconds <= (sec +(Global.DEFAULT_PTS_PER_SECOND )))
{
File file = new File(Config.MULTIMEDIA_PATH, "frame_" + sec + ".png");
ImageIO.write(javaImage, "png", file);
System.out.printf("at elapsed time of %6.3f seconds wrote: %s \n", seconds, file);
return;
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
}
else
{
// This packet isn't part of our video stream, so we just
// silently drop it.
}
}
converter.delete();
}
Do you know a better way to do this?

Well from just reading your code I see some optimizations that can be made.
One you first read through the entire file once, create an index of byteoffsets and seconds. Then the function can lookup of the byteoffset from the seconds given and you can decode the video at that offset and do the rest of your code.
Another option is to use your method, reading through the whole file each time, but instead of calling all that resampler, newPic, and java image converter code, check if the seconds match up first. If they do, then convert the image into a new pic to be displayed.
So
if(picture.isComplete()){
try {
double seconds = ((double)picture.getPts()) / Global.DEFAULT_PTS_PER_SECOND;
if (seconds >= sec && seconds <= (sec +(Global.DEFAULT_PTS_PER_SECOND )))
{
Resample image
Convert Image
Do File stuff
}

Use seekKeyFrame option. You can use this function to seek to any time in the video file (time is in milliseconds).
double timeBase = 0;
int videoStreamId = -1;
private void seekToMs(IContainer container, long timeMs) {
if(videoStreamId == -1) {
for(int i = 0; i < container.getNumStreams(); i++) {
IStream stream = container.getStream(i);
IStreamCoder coder = stream.getStreamCoder();
if (coder.getCodecType() == ICodec.Type.CODEC_TYPE_VIDEO) {
videoStreamId = i;
timeBase = stream.getTimeBase().getDouble();
break;
}
}
}
long seekTo = (long) (timeMs/1000.0/timeBase);
container.seekKeyFrame(videoStreamId, seekTo, IContainer.SEEK_FLAG_BACKWARDS);
}
From there you use your classic while(container.readNextPacket(packet) >= 0) method of getting the images to files.
Notice: It won't seek to exact time but approximate so you'll still need to go through the packets (but of course much less than before).