This problem I am facing in title is very similar to this question previously raised here (Azure storage: Uploaded files with size zero bytes), but it was for .NET and the context for my Java scenario is that I am uploading small-size CSV files on a daily basis (about less than 5 Kb per file). In addition the API code uses the latest version of Azure API that I am using in contrast against the 2010 used by the other question.
I couldn't figure out where have I missed out, but the other alternative is to do it in File Storage, but of course the blob approach was recommended by a few of my peers.
So far, I have mostly based my code on uploading a file as a block of blob on the sample that was shown in the Azure Samples git [page] (https://github.com/Azure-Samples/storage-blob-java-getting-started/blob/master/src/BlobBasics.java). I have already done the container setup and file renaming steps, which isn't a problem, but after uploading, the size of the file at the blob storage container on my Azure domain shows 0 bytes.
I've tried alternating in converting the file into FileInputStream and upload it as a stream but it still produces the same manner.
fileName=event.getFilename(); //fileName is e.g eod1234.csv
String tempdir = System.getProperty("java.io.tmpdir");
file= new File(tempdir+File.separator+fileName); //
try {
PipedOutputStream pos = new PipedOutputStream();
stream= new PipedInputStream(pos);
buffer = new byte[stream.available()];
stream.read(buffer);
FileInputStream fils = new FileInputStream(file);
int content = 0;
while((content = fils.read()) != -1){
System.out.println((char)content);
}
//Outputstream was written as a test previously but didn't work
OutputStream outStream = new FileOutputStream(file);
outStream.write(buffer);
outStream.close();
// container name is "testing1"
CloudBlockBlob blob = container.getBlockBlobReference(fileName);
if(fileName.length() > 0){
blob.upload(fils,file.length()); //this is testing with fileInputStream
blob.uploadFromFile(fileName); //preferred, just upload from file
}
}
There are no error messages shown, just we know that the file touches the blob storage and shows a size 0 bytes. It's a one-way process by only uploading CSV-format files. At the blob container, it should be showing those uploaded files a size of 1-5 KBs each.
Instead of blob.uploadFromFile(fileName); you should use blob.uploadFromFile(file.getAbsolutePath()); because uploadFromFile method requires absolute path. And you don't need the blob.upload(fils,file.length());.
Refer to Microsoft Docs: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-java#upload-blobs-to-the-container
The Azure team replied to a same query I've put on mail and I have confirmed that the problem was not on the API, but due to the Upload component in Vaadin which has a different behavior than usual (https://vaadin.com/blog/uploads-and-downloads-inputs-and-outputs). Either the CloudBlockBlob or the BlobContainerUrl approach works.
The out-of-the-box Upload component requires manual implementation of the FileOutputStream to a temporary object unlike the usual servlet object that is seen everywhere. Since there was limited time, I used one of their addons, EasyUpload, because it had Viritin UploadFileHandler incorporated into it instead of figuring out how to stream the object from scratch. Had there been more time, I would definitely try out the MultiFileUpload addon, which has additional interesting stuff, in my sandbox workspace.
I had this same problem working with .png (copied from multipart files) files I was doing this:
File file = new File(multipartFile.getOriginalFilename());
and the blobs on Azure were 0bytes but when I changed to this:
File file = new File("C://uploads//"+multipartFile.getOriginalFilename());
it started saving the files properly
Related
I was wondering if there is a way to access a file and it's path from my assets folder in android studio? The reason why I need to access the file and its path is because I am working with a method that REQUIRES the String path for a file, and it must access the file from its String path. However, in android studio I haven't found a way to access the file directly from the String value of its path. I decided to use a workaround and simply read the file from an InputStream and write the file to an OutputStream, but the file is about 170MB, and it is too memory intensive to write the File to an OutputStream. It takes my application about 10:00 Minutes to download the file when I implement that strategy. I have searched all over this website and numerous sources to find a solution (books and documentation) but am unable to find a viable solution. Here is an example of my code:
#Override
public Model doInBackground(String... params){
try {
String filePath = context.getFilesDir() + File.separator + "my_turtle.ttl";
File destinationFile = new File(filePath);
FileOutputStream outputStream = new FileOutputStream(destinationFile);
AssetManager assetManager = context.getAssets();
InputStream inputStream = assetManager.open("sample_3.ttl");
byte[] buffer = new byte[10000000];
int length = 0;
while ((length = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, length);
}
outputStream.close();
inputStream.close();
model = ModelFactory.createDefaultModel();
TDBLoader.loadModel(model, filePath, false);//THIS METHOD REQUIRES THE FILE PATH.
MainActivity.presenter.setModel(model);
}catch(FileNotFoundException e){
e.printStackTrace(System.out);
}
catch(IOException e){
e.printStackTrace(System.out);
}
return model;
}
As you can see the TDBLoader.loadModel() method requires a String for the file URI as the second argument, so it would be convenient to have the ability to access the File directly from my assets folder without utilizing an InputStream. The method takes as an argument (Model model, String url, Boolean showProgress). As I mentioned, the current strategy I am using utilizes too much memory and either crashes the Application entirely, or takes 10 minutes to download the file I need. I am using an AsyncTask to perform this operation, but due to the length of time required to perform the task that kind of defeats the purpose of an AsyncTask in this scenario.
What further complicates things is that I have to use an old version of Apache Jena because I am working with Android Studio and the official version of Apache Jena is not compatible with android studio. So I have to use a port that is 8 years old which doesn't have the updated classes that Apache Jena offers. If I could use the RDFParser class I could pass an InputStream, but that class does not exist in the older version of Apache Jena that I must use.
So I am stuck at this point. The method must utilize the String url path of the file in my assets folder, but I don't know how to access this without writing to a custom file from an InputStream, but writing to the file from the InputStream utilizes too much memory and forces the App to crash. If anyone has a solution I will greatly appreciate it.
Here is an example of my code
new byte[10000000] may fail, as you may not have a single contiguous block of memory that big. Plus, you might not have that much heap space to begin with. Use a smaller number, such as 65536.
It takes my application about 10:00 Minutes to download the file when I implement that strategy
The time will vary by hardware. I would not expect it to be that slow on most devices, but it could be on some.
I was wondering if there is a way to access a file and it's path from my assets folder in android studio?
You are running your app on Android. Android Studio is not running on Android. Assets are not files on the Android device. They are entries in the APK file, which is basically a ZIP archive. In effect, your code is unZIPping 170MB of material and writing it out to a file.
If anyone has a solution I will greatly appreciate it.
Work with some people to port over an updated version of Jena that offers reading RDF from an InputStream.
Or switch to some other RDF library.
Or work with the RDF file format directly.
Or use a smaller RDF file, so the copy takes less time.
Or download the RDF file, if you think that will be preferable to copying over the asset.
Or do the asset-to-file copying in a foreground JobIntentService, updating the progress in its associated Notification, so that the user can do other things on their device while you complete the copy.
We currently have an existing WCF SOAP service that runs fine with a large range of clients. We present a StreamBody as a way to download larger filesets. I have tried virtually every way to attempt to download large files without loading the file completely into memory by the client. I have in every attempt. Essentially, by calling the following, the JAVA client wants to load the complete file into memory. I am looking for suggestions. Below is my latest attempt:
OrderServiceStub stub = getOrderServiceStub();
OrderServiceStub.GetStreamedOrderOutputRequestMessage getStreamedOrderOutputRequestMessage = new OrderServiceStub.GetStreamedOrderOutputRequestMessage();
OrderServiceStub.GetStreamedOrderOutputRequest getStreamedOrderOutputRequest = new OrderServiceStub.GetStreamedOrderOutputRequest();
for (OrderServiceStub.OrderOutput o : orderoutput.getOrderOutput()){
OrderServiceStub.Guid guidOutput = o.getOrderOutputTicket();
String fileName = o.getOrderOutputName();
getStreamedOrderOutputRequest.setOrderOutputTicket(guidOutput);
getStreamedOrderOutputRequestMessage.setGetStreamedOrderOutputRequest(getStreamedOrderOutputRequest);
int bufferSize = 1024;
InputStream is = new BufferedInputStream(stub.getStreamedOrderOutput(getStreamedOrderOutputRequestMessage).getFileData().getStreamBody().getInputStream());
OutputStream os = new FileOutputStream(new File("C:\\temp\\" + fileName));
org.apache.commons.io.IOUtils.copyLarge(is, os);
org.apache.commons.io.IOUtils.closeQuietly(is);
org.apache.commons.io.IOUtils.closeQuietly(os);
}
Instead of copying the whole thing via apache, you might try putting it into a loop, reading your bufferSize, then flushing the output stream. Then continue, doing the same. Hopefully that will help.
I am working on a application in appengine that we want to be able to make the content available for offline users. This means we need to get all the used blobstore files and save them off for the offline user. I am using the server side to do this so that it is only done once, and not for every end user. I am using the task queue to run this process as it can easily time out. Assume all this code is running as a task.
Small collections work fine, but larger collections result in a appengine error 202 and it restarts the task again and again. Here is the sample code that comes from combination of Writing Zip Files to GAE Blobstore and following the advice for large zip files at Google Appengine JAVA - Zip lots of images saving in Blobstore by reopening the channel as needed. Also referenced AppEngine Error Code 202 - Task Queue as the error.
//Set up the zip file that will be saved to the blobstore
AppEngineFile assetFile = fileService.createNewBlobFile("application/zip", assetsZipName);
FileWriteChannel writeChannel = fileService.openWriteChannel(assetFile, true);
ZipOutputStream assetsZip = new ZipOutputStream(new BufferedOutputStream(Channels.newOutputStream(writeChannel)));
HashSet<String> blobsEntries = getAllBlobEntries(); //gets blobs that I need
saveBlobAssetsToZip(blobsEntries);
writeChannel.closeFinally();
.....
private void saveBlobAssetsToZip(blobsEntries) throws IOException {
for (String blobId : blobsEntries) {
/*gets the blobstote key that will result in the blobstore entry - ignore the bsmd as
that is internal to our wrapper for blobstore.*/
BlobKey blobKey = new BlobKey(bsmd.getBlobId());
//gets the blob file as a byte array
byte[] blobData = blobstoreService.fetchData(blobKey, 0, BlobstoreService.MAX_BLOB_FETCH_SIZE-1);
String extension = type of file saved from our metadata (ie .jpg, .png, .pfd)
assetsZip.putNextEntry(new ZipEntry(blobId + "." + extension));
assetsZip.write(blobData);
assetsZip.closeEntry();
assetsZip.flush();
/*I have found that if I don't close the channel and reopen it, I can get a IO exception
because the files in the blobstore are too large, thus the write a file and then close and reopen*/
assetsZip.close();
writeChannel.close();
String assetsPath = assetFile.getFullPath();
assetFile = new AppEngineFile(assetsPath);
writeChannel = fileService.openWriteChannel(assetFile, true);
assetsZip = new ZipOutputStream(new BufferedOutputStream(Channels.newOutputStream(writeChannel)));
}
}
What is the proper way to get this to run on appengine? Again small projects work fine and zip saves, but larger projects with more blob files results in this error.
I bet that the instance is running out of memory. Are you using appstats? It can consume a large amount of memory. If that doesn't work you will probably need to increase the instance size.
I'm trying to access a shared file from my app, e.g //172.24.9.13/c/2012xp.mdb
By doing:
new File("//172.24.9.13/c/2012xp.mdb");
it doesn't work.
I found the jCIFS library, and creating
new SmbFile("//172.24.9.13/c/2012xp.mdb");
it works, but the problem is that I need a java.io.File.
I have also seen that there is no way to mount smb's on android devices without rooting them.
Is there a way to get a java.io.File instance of my shared file?
I don't think you'll be able to read it as java.io.File because it's not one.
If you absolutely need a java.io.File you'll probably have to call smbFile.getInputStream() and copy to a local file, than you use that local file.
Considering it's a .mdb file you're probably wanting to read data from the DataBase, and it might be a huge file and you don't want to copy it over. On that situation, your only option is to setup a server, with an api that replies in JSON and your app will send GET requests to it.
try this code. i'm not sure about .mdb file format. but this works for PDF files.
first create a SmbFileInputStream using your smb file.
in = new SmbFileInputStream(sFile);
Then create a output file in the device storage (SD card)
outputfile = new File(dir, "todevice.pdf");
Then create a FileOutputStream (java IO)
out = new FileOutputStream(outputfile);
Then read input(SMB) and write it in to the output(IO)
while ((read = in.read(buffer)) > 0 )
{
try {
out.write(buffer, 0, read);
} catch (IOException e) {
}
}
Now u have a file on your sd card
I have a database file in res/raw/ folder. I am calling Resources.openRawResource() with the file name as R.raw.FileName and I get an input stream, but I have an another database file in device, so to copy the contents of that db to the device db I use:
BufferedInputStream bi = new BufferedInputStream(is);
and FileOutputStream, but I get an exception that database file is corrupted. How can I proceed?
I try to read the file using File and FileInputStream and the path as /res/raw/fileName, but that also doesn't work.
Yes, you should be able to use openRawResource to copy a binary across from your raw resource folder to the device.
Based on the example code in the API demos (content/ReadAsset), you should be able to use a variation of the following code snippet to read the db file data.
InputStream ins = getResources().openRawResource(R.raw.my_db_file);
ByteArrayOutputStream outputStream=new ByteArrayOutputStream();
int size = 0;
// Read the entire resource into a local byte buffer.
byte[] buffer = new byte[1024];
while((size=ins.read(buffer,0,1024))>=0){
outputStream.write(buffer,0,size);
}
ins.close();
buffer=outputStream.toByteArray();
A copy of your file should now exist in buffer, so you can use a FileOutputStream to save the buffer to a new file.
FileOutputStream fos = new FileOutputStream("mycopy.db");
fos.write(buffer);
fos.close();
InputStream.available has severe limitations and should never be used to determine the length of the content available for streaming.
http://developer.android.com/reference/java/io/FileInputStream.html#available():
"[...]Returns an estimated number of bytes that can be read or skipped without blocking for more input. [...]Note that this method provides such a weak guarantee that it is not very useful in practice."
You have 3 solutions:
Go through the content twice, first just to compute content length, second to actually read the data
Since Android resources are prepared by you, the developer, hardcode its expected length
Put the file in the /asset directory and read it through AssetManager which gives you access to AssetFileDescriptor and its content length methods. This may however give you the UNKNOWN value for length, which isn't that useful.