Develop Reference

Android CTS code about storage throughput calculation

I need to calculate the storage throughput in my Android device, and I found source code about calculating storage sequential RW throughput in Android CTS.
FileUtil.java
public static long getFileSizeExceedingMemory(Context context, int bufferSize) {
long freeDisk = SystemUtil.getFreeDiskSize(context);
long memSize = SystemUtil.getTotalMemory(context);
long diskSizeTarget = (2 * memSize / bufferSize) * bufferSize;
final long minimumDiskSize = (512L * 1024L * 1024L / bufferSize) * bufferSize;
final long reservedDiskSize = (50L * 1024L * 1024L / bufferSize) * bufferSize;
if ( diskSizeTarget < minimumDiskSize ) {
diskSizeTarget = minimumDiskSize;
}
if (diskSizeTarget > freeDisk) {
Log.i(TAG, "Free disk size " + freeDisk + " too small");
return 0;
}
if ((freeDisk - diskSizeTarget) < reservedDiskSize) {
diskSizeTarget -= reservedDiskSize;
}
return diskSizeTarget;
}
This function is about to create a file from RAM to storage(write) and then read back.
I was just wondering:
long diskSizeTarget = (2 * memSize / bufferSize) * bufferSize;
Why they need to prepare a file which is around double ram size?
I've ever tried file size which is a half of ram size (my device ram size=2GB), and the write throughput looks normal but read throughput is too fast(around 200MB/s).
But the result looks fine when I use around 4GB file(double ram size) and 2GB.
(The buffersize parameters are 10MB for read and write)
Here are the read and write codes:
SequentialRWTest.java
public void testSingleSequentialRead() throws Exception {
final long fileSize = FileUtil.getFileSizeExceedingMemory(getContext(), BUFFER_SIZE);
if (fileSize == 0) { // not enough space, give up
return;
}
long start = System.currentTimeMillis();
final File file = FileUtil.createNewFilledFile(getContext(),
DIR_SEQ_RD, fileSize);
long finish = System.currentTimeMillis();
String streamName = "test_single_sequential_read";
DeviceReportLog report = new DeviceReportLog(REPORT_LOG_NAME, streamName);
report.addValue("file_size", fileSize, ResultType.NEUTRAL, ResultUnit.NONE);
report.addValue("write_throughput",
Stat.calcRatePerSec((double)fileSize / 1024 / 1024, finish - start),
ResultType.HIGHER_BETTER, ResultUnit.MBPS);
final int NUMBER_READ = 10;
final byte[] data = new byte[BUFFER_SIZE];
double[] times = MeasureTime.measure(NUMBER_READ, new MeasureRun() {
#Override
public void run(int i) throws IOException {
final FileInputStream in = new FileInputStream(file);
long read = 0;
while (read < fileSize) {
in.read(data);
read += BUFFER_SIZE;
}
in.close();
}
});
double[] mbps = Stat.calcRatePerSecArray((double)fileSize / 1024 / 1024, times);
report.addValues("read_throughput", mbps, ResultType.HIGHER_BETTER, ResultUnit.MBPS);
Stat.StatResult stat = Stat.getStat(mbps);
report.setSummary("read_throughput_average", stat.mAverage, ResultType.HIGHER_BETTER,
ResultUnit.MBPS);
report.submit(getInstrumentation());
}
And createNewFilledFile function in FileUtil.java
public static File createNewFilledFile(Context context, String dirName, long length)
throws IOException {
final int BUFFER_SIZE = 10 * 1024 * 1024;
File file = createNewFile(context, dirName);
FileOutputStream out = new FileOutputStream(file);
byte[] data = generateRandomData(BUFFER_SIZE);
long written = 0;
while (written < length) {
out.write(data);
written += BUFFER_SIZE;
}
out.flush();
out.close();
return file;
}

Calculated download speed is wrong when using TrafficStats class

I want to build android download speed test. To do that I am using TrafficStats class. Problem is that I am getting wrong results. Results are almost the same when I run the test but I put heavy load on my Internet connection before I run test. I download file for 30 seconds and after that (or when file is downloaded) and then calculate bytes using TrafficStats
Does someone knows where is the problem?
This is code that I am using:
#Override
protected String doInBackground(String... urls) {
String downloaded ="";
// String uploaded = "";
try{
long BeforeTime = System.currentTimeMillis();
long TotalTxBeforeTest = TrafficStats.getTotalTxBytes();
long TotalRxBeforeTest = TrafficStats.getTotalRxBytes();
URL url = new URL(urls[0]);
URLConnection connection = new URL(urls[0]).openConnection();
connection.setUseCaches(false);
connection.connect();
InputStream input = connection.getInputStream();
BufferedInputStream bufferedInputStream = new BufferedInputStream(input);
byte[] buffer = new byte[1024];
int n = 0;
long endLoop = BeforeTime+30000;
while(System.currentTimeMillis() < endLoop) {
/* if (bufferedInputStream.read(buffer) != -1){
break;
}*/
}
long TotalTxAfterTest = TrafficStats.getTotalTxBytes();
long TotalRxAfterTest = TrafficStats.getTotalRxBytes();
long AfterTime = System.currentTimeMillis();
double TimeDifference = AfterTime - BeforeTime;
double rxDiff = TotalRxAfterTest - TotalRxBeforeTest;
double txDiff = TotalTxAfterTest - TotalTxBeforeTest;
Log.e(TAG, "Download skinuto. "+ rxDiff);
if((rxDiff != 0) && (txDiff != 0)) {
double rxBPS = (rxDiff / (TimeDifference/1000)); // total rx bytes per second.
double txBPS = (txDiff / (TimeDifference/1000)); // total tx bytes per second.
downloaded = String.valueOf(rxBPS) + "B/s. Total rx = " + rxDiff;
// uploaded = String.valueOf(txBPS) + "B/s. Total tx = " + txDiff;
}
else {
downloaded = "No downloaded bytes.";
}
}
catch(Exception e){
Log.e(TAG, "Error while downloading. "+ e.getMessage());
}
return downloaded;
}

I tried your code - it seems to work fine for me BUT i changed
while(System.currentTimeMillis() < endLoop) {
/* if (bufferedInputStream.read(buffer) != -1) {
break;
}*/
}
to
while(System.currentTimeMillis() < endLoop) {
if (bufferedInputStream.read(buffer) == -1){
break;
}
}
since read returns -1 if the end of the stream is reached.

RandomAccessFile setLength() writes 0 for large gigabyte values

I am attempting to write files using the RandomAccessFile.setLength() method. The problem I'm running into is: it works fine for some file sizes but not others.
Wild guess: I'm not sure if it has something to do with my RAM (16GB)? AFAIK it seems to be failing for large GB values which are multiples of 8.
If anyone knows why this is actually occurring that would be the best answer.
I included a toy program which should demonstrate this behavior. Let me know if you're seeing the same results.
import java.io.*;
public class Debug {
public static void main( String[] args ) {
// create 5 file sizes: 16 KB, 16 MB, 8GB, 16 GB, 30 GB
long KILO = 1024;
long testNum1 = KILO * 16; // 16 KB
long testNum2 = KILO * KILO * 16; // 16 MB
long testNum3 = KILO * KILO * KILO * 8; // 8 GB
long testNum4 = KILO * KILO * KILO * 16; // 16 GB
long testNum5 = KILO * KILO * KILO * 30; // 30 GB
// print the 5 file sizes
System.out.println("testNum1 is " + testNum1 + " bytes");
System.out.println("testNum2 is " + testNum2 + " bytes");
System.out.println("testNum3 is " + testNum3 + " bytes");
System.out.println("testNum4 is " + testNum4 + " bytes");
System.out.println("testNum5 is " + testNum5 + " bytes");
// attempt to write 5 files to disk, using these sizes
RandomAccessFile f = null;
try {
f = new RandomAccessFile("testNum1", "rw"); // <-- PASS
f.setLength( testNum1 );
f = new RandomAccessFile("testNum2", "rw"); // <-- PASS
f.setLength( testNum2 );
f = new RandomAccessFile("testNum3", "rw"); // <-- FAIL
f.setLength( testNum3 );
f = new RandomAccessFile("testNum4", "rw"); // <-- FAIL
f.setLength( testNum4 );
f = new RandomAccessFile("testNum5", "rw"); // <-- PASS
f.setLength( testNum5 );
} catch( Exception e ) {
System.err.println(e);
} finally {
if( f != null ) {
try {
f.close();
} catch( IOException e ) {
e.printStackTrace();
}
}
}
}
}

writing to a file using java.nio

I have to use java.nio to create a file of any desired size by populating it with data. I am reading through a document, but am confused about when I need to flip, put, or write and am getting errors. I have successfully done this program using .io but I am testing to see if .nio will make it run faster.
This is my code so far. args[0] is the size of the file you want to make and args[1] is the name of the file to be written to
public static void main(String[] args) throws IOException
{
nioOutput fp = new nioOutput();
FileOutputStream fos = new FileOutputStream(args[1]);
FileChannel fc = fos.getChannel();
long sizeOfFile = fp.getFileSize(args[1]);
long desiredSizeOfFile = Long.parseLong(args[0]) * 1073741824; //1 Gigabyte = 1073741824 bytes
int byteLength = 1024;
ByteBuffer b = ByteBuffer.allocate(byteLength);
while(sizeOfFile + byteLength < desiredSizeOfFile)
{
// b.put((byte) byteLength);
b.flip();
fc.write(b);
sizeOfFile += byteLength;
}
int diff = (int) (desiredSizeOfFile - sizeOfFile);
sizeOfFile += diff;
fc.write(b, 0, diff);
fos.close();
System.out.println("Finished at " + sizeOfFile / 1073741824 + " Gigabyte(s)");
}
long getFileSize(String fileName)
{
File file = new File(fileName);
if (!file.exists() || !file.isFile())
{
System.out.println("File does not exist");
return -1;
}
return file.length();
}

If all you want to do is pre-extend a file to a given length with nulls, you can do it in three lines and save all that I/O:
RandomAccessFile raf = new RandomAccessFile(file, "rw");
raf.setLength(desiredSizeOfFile);
raf.close();
This will operate several gazzilion times as quickly as what you are trying to do now.

sorry everyone, I figured it out.
while(sizeOfFile + byteLength < desiredSizeOfFile)
{
fc.write(b);
b.rewind();
sizeOfFile += byteLength;
}
int diff = (int) (desiredSizeOfFile - sizeOfFile);
sizeOfFile += diff;
ByteBuffer d = ByteBuffer.allocate(diff);
fc.write(d);
b.rewind();
fos.close();
System.out.println("Finished at " + sizeOfFile / 1073741824 + " Gigabyte(s)");
}

How do I read an NSInputStream while writing it to an NSOutputStream in iOS?

I am porting an Android app to iPhone (more like improving the iPhone app based on the Android version) and I need to split and combine large uncompressed audio files.
Currently, I load all the files into memory and split them and combine them in separate functions. It crashes with 100MB+ files.
This is the new process needed to do it:
I have two recordings (file1 and file2) and a split position where I want file2 to be inserted inside file1.
-create the input streams for file1 and file2 and the output stream for the outputfile.
-rewrite the new CAF header
-read the data from inputStream1 until it reaches the split point and I write all that data to the output file.
and write it to the output stream.
-read all data from inputStream2 and write it to output file.
-read remaining data from inputStream1 and write it to output file.
Here is my Android code for the process:
File file1File = new File(file1);
File file2File = new File(file2);
long file1Length = file1File.length();
long file2Length = file2File.length();
FileInputStream file1ByteStream = new FileInputStream(file1);
FileInputStream file2ByteStream = new FileInputStream(file2);
FileOutputStream outputFileByteStream = new FileOutputStream(outputFile);
// time = fileLength / (Sample Rate * Channels * Bits per sample / 8)
// convert position to number of bytes for this function
long sampleRate = eRecorder.RECORDER_SAMPLERATE;
int channels = 1;
long bitsPerSample = eRecorder.RECORDER_BPP;
long bytePositionLength = (position * (sampleRate * channels * bitsPerSample / 8)) / 1000;
//calculate total data size
int dataSize = 0;
dataSize = (int)(file1Length + file2Length);
WriteWaveFileHeaderForMerge(outputFileByteStream, dataSize,
dataSize + 36,
eRecorder.RECORDER_SAMPLERATE, 1,
2 * eRecorder.RECORDER_SAMPLERATE);
long bytesWritten = 0;
int length = 0;
//set limit for bytes read, and write file1 bytes to outputfile until split position reached
int limit = (int)bytePositionLength;
//read bytes to limit
writeBytesToLimit(file1ByteStream, outputFileByteStream, limit);
file1ByteStream.close();
file2ByteStream.skip(44);//skip wav file header
writeBytesToLimit(file2ByteStream, outputFileByteStream, (int)file2Length);
file2ByteStream.close();
//calculate length of remaining file1 bytes to be written
long file1offset = bytePositionLength;
//reinitialize file1 input stream
file1ByteStream = new FileInputStream(file1);
file1ByteStream.skip(file1offset);
writeBytesToLimit(file1ByteStream, outputFileByteStream, (int)file1Length);
file1ByteStream.close();
outputFileByteStream.close();
And this is my writeBytesToLimit function:
private void writeBytesToLimit(FileInputStream inputStream, FileOutputStream outputStream, int byteLimit) throws IOException
{
int bytesRead = 0;
int chunkSize = 65536;
int length = 0;
byte[] buffer = new byte[chunkSize];
while((length = inputStream.read(buffer)) != -1)
{
bytesRead += length;
if(bytesRead >= byteLimit)
{
int leftoverBytes = byteLimit % chunkSize;
byte[] smallBuffer = new byte[leftoverBytes];
System.arraycopy(buffer, 0, smallBuffer, 0, leftoverBytes);
outputStream.write(smallBuffer);
break;
}
if(length == chunkSize)
outputStream.write(buffer);
else
{
byte[] smallBuffer = new byte[length];
System.arraycopy(buffer, 0, smallBuffer, 0, length);
outputStream.write(smallBuffer);
}
}
}
How do I do this in iOS? Using the same delegate for two NSInputStreams and an NSOutputStream looks like it will get very messy.
Has anyone seen an example of how to do this (and do it clean)?

I ended up using NSFileHandle. For example, this is the first part of what I am doing.
NSData *readData = [[NSData alloc] init];
NSFileHandle *reader1 = [NSFileHandle fileHandleForReadingAtPath:file1Path];
NSFileHandle *writer = [NSFileHandle fileHandleForWritingAtPath:outputFilePath];
//start reading data from file1 to split point and writing it to file
long bytesRead = 0;
while(bytesRead < splitPointInBytes)
{
//read a chunk of data
readData = [reader1 readDataOfLength:chunkSize];
if(readData.length == 0)break;
//trim data if too much was read
if(bytesRead + readData.length > splitPointInBytes)
{
//get difference of read bytes and byte limit
long difference = bytesRead + readData.length - splitPointInBytes;
//trim data
NSMutableData *readDataMutable = [NSMutableData dataWithData:readData];
[readDataMutable setLength:readDataMutable.length - difference];
readData = [NSData dataWithData:readDataMutable];
NSLog(#"Too much data read, trimming");
}
//write data to output file
[writer writeData:readData];
//update byte counter
bytesRead += readData.length;
}
long file1BytesWritten = bytesRead;