I create data in the server (gae) and I want to store it in Blobstore. I saw many answers on how to do this giving a BlobStore URL to the client, but there is no client or HTTP request: it's just an asynchronous task.
Then I guess I should use createUploadUrl(), and instead of giving this URL to a client, from my code HTTP Post my data to it via URL Fetch. This looks weird, isn't there another API for this?
Let's say that the files I want in Blobstore are already stored in my GCS default bucket. Can I just tell Blobstore about them using the gcs location "/gs/bucketname/file"? I tried this by
GcsFilename filename = new GcsFilename(bucketName, fileId);
String gcsKey = "/gs/" + bucketName + "/" + filename.getObjectName();
BlobKey blobKey = blobstoreService.createGsBlobKey(gcsKey);
GcsOutputChannel outputChannel = gcsService.createOrReplace(filename, GcsFileOptions.getDefaultInstance());
ObjectOutputStream oout = new ObjectOutputStream(Channels.newOutputStream(outputChannel));
oout.writeObject(myDataObjectToPersist);
oout.close();
// ...at some other point I have checked the file is correctly stored in
// GCS and I can fetch it using /gs/bucket/fileId
// but it doesn't seem to be in Blobstore, so when
InputStream stream = new BlobstoreInputStream(new BlobKey(blobKey.keyString))
// ... this gives a BlobstoreInputStream$BlobstoreIOException: BlobstoreInputStream received an invalid blob key...
Is this something conceptually wrong - like if I use GcsOutputChannel to save it I will not get it from Blobstore even if I create a BlobKey, or is it something that could work but I just did something wrong?
1K thanks
Why would you want to store the file in blobstore as opposed to writing and reading it directly from GCS?
Yes, you can create a BlobKey for a file stored in GCS, and can use the key in some of the blobstore API (such as fetchData and serve) but unfortunately not in all.
Some of the blobstore API (such as BlobstoreInputStream) depends on BlobInfo and that is not created when using the GCS client.
Related
I am building a small REST API service to store and retrieve photos. For that, I am using S3 as following:
public String upload(InputStream uploadedInputStream,
Map<String, String> metadata, String group, String filename) {
TransferManager tm = TransferManagerBuilder.standard()
.withS3Client(amazonS3)
.build();
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentType(metadata.get(Configuration.CONTENT_TYPE_METADATA_KEY));
// TODO: 26/06/20 Add content-type to metadata
String filepath = group + "/" + filename;
s3transferManager.upload(new PutObjectRequest(
configuration.getProperty("aws.s3.bucket"),
filepath,
uploadedInputStream,
objectMetadata)).waitForUploadResult();
return amazonS3.getUrl(configuration.getProperty("aws.s3.bucket"), filepath).toString();
}
url returned by the function looks like https://photos.tarkshala.com.s3.ap-south-1.amazonaws.com/default-group/1593911534320%230. When accessed it shows up like this
When I open it using the object url(https://s3.ap-south-1.amazonaws.com/photos.tarkshala.com/default-group/1593911534320%230) given in AWS S3 console it shows up fine.
Why getUrl method not returning the second url or is there a way to get second method/api that does it?
This is because of recent changes by AWS regarding s3.
When using virtual hosted–style buckets with SSL, the SSL wild-card
certificate only matches buckets that do not contain dots ("."). To
work around this, use HTTP or write your own certificate verification
logic. For more information, see Amazon S3 Path Deprecation Plan.
amazon-s3-path-deprecation-plan-the-rest-of-the-story
Create a bucket without a dot or use the path style URL or you check VirtualHostingCustomURLs.
S3 support two types of URL to access Object.
Virtual hosted style access
https://bucket-name.s3.Region.amazonaws.com/key name
Path-Style Requests
https://s3.Region.amazonaws.com/bucket-name/key name
Important
Buckets created after September 30, 2020, will support only virtual
hosted-style requests. Path-style requests will continue to be
supported for buckets created on or before this date. For more
information, see Amazon S3 Path Deprecation Plan – The Rest of the
Story.
S3 VirtualHosting
In my application I have a string which is a file content. I need to create new file with this content in blobstore. I tryed to use File API like this:
FileService fileService = FileServiceFactory.getFileService();
AppEngineFile file = fileService.createNewBlobFile("text/plain");
FileWriteChannel writeChannel = fileService.openWriteChannel(file, true);
writeChannel.write(ByteBuffer.wrap(content.getBytes()));
writeChannel.closeFinally();
BlobKey blobKey = fileService.getBlobKey(file);
res.sendRedirect("/serve?blob-key=" + blobKey);
But since the File API is deprecated I only get this error:
HTTP ERROR 500
Problem accessing /create_timetable. Reason:
The Files API is disabled. Further information: https://cloud.google.com/appengine/docs/deprecations/files_api
Caused by:
com.google.apphosting.api.ApiProxy$FeatureNotEnabledException: The Files API is disabled. Further information: https://cloud.google.com/appengine/docs/deprecations/files_api
at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.callInternal(ApiProxyLocalImpl.java:515)
at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.call(ApiProxyLocalImpl.java:484)
at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.call(ApiProxyLocalImpl.java:461)
at java.util.concurrent.Executors$PrivilegedCallable$1.run(Executors.java:533)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.concurrent.Executors$PrivilegedCallable.call(Executors.java:530)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
How can I manually create file in blobstore since user only gives me data content as string, not a file itself, so I can't use <form action="blobstoreService.createUploadUrl("/upload") %>" and <input type="file">
Update
As #ozarov suggested I did it using Google Cloude Storage API and wrote the function below. It also returns BlobKey of this file so you can access it using Blobstore API.
private BlobKey saveFile(String gcsBucket, String fileName,
String content) throws IOException {
GcsFilename gcsFileName = new GcsFilename(gcsBucket, fileName);
GcsOutputChannel outputChannel =
gcsService.createOrReplace(gcsFileName, GcsFileOptions.getDefaultInstance());
outputChannel.write(ByteBuffer.wrap(content.getBytes()));
outputChannel.close();
return blobstoreService.createGsBlobKey(
"/gs/" + gcsFileName.getBucketName() + "/" + gcsFileName.getObjectName());
}
You should write to Google Cloud Storage instead.
Files API is deprecated and you are correct that the Blobstore API
does not provide a programmatic way to write directly to it.
Later you can read directly from Google Cloud Storage using its own API or
you can also use the Blobstore API to do so by creating a BlobKey for it.
I have some file upload code that works on appspot.com but not on local Java dev server.
I am uploading to Google Cloud Storage, via:
String bucketName = AppIdentityServiceFactory.getAppIdentityService().getDefaultGcsBucketName();
String uploadUrl = blobstoreService.createUploadUrl(
successUrl,
UploadOptions.Builder.withGoogleStorageBucketName(bucketName)
);
in my upload handler (pointed to by "successUrl"), I try and read the newly uploaded blob from the GCS bucket. This is done like this:
BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
Map<String, List<BlobKey>> uploads = blobstoreService.getUploads(req);
Map<String, List<FileInfo>> files = blobstoreService.getFileInfos(req);
...
// use the FileInfo instance to do fileInfo.getGsObjectName()
...
GcsInputChannel readChannel = gcsService.openPrefetchingReadChannel(gcsFilename, 0, 1024 * 1024);
InputStream is = Channels.newInputStream(readChannel));
up on the actual appengine instance, fileInfo.getGsObjectName() returns a working path like this:
/gs/myapp.appspot.com/L2F..LongIDHere...YmJV
which works and can be read, BUT, on the local dev server, I get something like:
/gs/app_default_bucket/fake-encoded_gs_key:YXB...LongIdHere...6de
which throws FileNotFoundException when tried read using the same code, that works on appEngine :-(
I'm struggling for few days with this problem and you are my last chance solving it.
The Goal:
To upload a bitmap from android client to google app engine and save it in datastore.
I'm using Serialization to transfer object from client to server and vise versa.
Things I have tried:
sending a Bitmap
but i got java.io.NotSerializableException: android.graphics.Bitmap
then I tried create from the Bitmap Blob at the client like this:
ByteArrayOutputStream bos = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, bos);
Byte[] bArray = bos.toByteArray();
Blob blob = new Blob(bArray);
But at the client side I dont have the google app engine classes (for Blob), So I tried to brute force it by extract specific classess but it created new problems.
So I am asking in which dierction I can go from to solve this problem.
Thanks.
Few notes:
Do not use Java serialization to transfer data between JVMs. Java serialization is not standardized and is not guaranteed to be compatible between JVMs (or even between versions).
To send binary data it's best to use HTTP POST and set Content-Type appropriately (e.g. application/octet-stream).
So, to make this work do this:
Create a servlet which handles POST and gets the binary data. Use servletRequest.getInputStream() to get hold of binary data.
Use Blobstore FileService API to save data to blobstore.
On Android side use a http client to make a POST request and add your bitmap's binary data to it. If you need to add some metadata use Http headers.
This might be useful
How to upload and store an image with google app engine (java)
Alternatively, you can try blobstore api
http://code.google.com/appengine/docs/java/blobstore/overview.html
Here is a production tested way:
Use GAE appengine to upload your bitmap to, and serve for future clients.
On the Android code, follow these steps:
Get an Upload URL from GAE
Upload your bitmap to GAE, and get a blobkey back
Later on, use the blobkey to serve the image to your clients.
GAE Servlet code:
getUploadURL:
BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
String url = blobstoreService.createUploadUrl(path_to_your_upload_servlet);
uploadServlet - stores in blobstore, returns the blobkey to the uploader
BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
Map<String, List<BlobKey>> uploads = blobstoreService.getUploads(request);
String fileName = uploads.keySet().iterator().next();
final BlobKey blobKey = uploads.get(fileName).get(0);
response.getWriter().println(blobKey.getKeyString());
Android client code:
String uploadUrl = getUrlAsString(..your getUrl servlet path...)
// Upload to GAE (include apache-mime4j.jar and httpmime.jar in your project for this code)
File file = new File(imageFilePath);
HttpPost postRequest = new HttpPost(uploadUrl);
MultipartEntity entity = new MultipartEntity();
entity.addPart("file", new FileBody(file));
postRequest.setEntity(entity);
HttpResponse httpResponse;
HttpClient httpClient = new DefaultHttpClient();
httpClient.getParams().setBooleanParameter("http.protocol.handle-redirects",false);
httpResponse = httpClient.execute(postRequest);
int status = httpResponse.getStatusLine().getStatusCode();
String blobKey = getInputStreamAsString(httpResponse.getEntity().getContent())
Im currently building an web-app that allows users to upload content via blobstore and to later download it.
However, the servlet that takes care of the download is called BlobServiceServlet
and whenever a user downloads a blob, the filename is changed to "BlobServiceServlet" and the extension is also changed sometimes to .bin. Does anyone know how to fix this problem?
Add a "Content-disposition" header to the response.
See http://en.wikipedia.org/wiki/MIME#Content-Disposition for an example.
E.g., in the handler,
self.response.headers['Content-Disposition'] = 'attachment; filename=foo.doc'
Just an additional info.
This is the code to let the browser know the file size:
BlobInfoFactory blobInfoFactory = new BlobInfoFactory(DatastoreServiceFactory.getDatastoreService());
BlobInfo blobInfo = blobInfoFactory.loadBlobInfo(blobKey);
resp.setContentLength(new Long(blobInfo.getSize()).intValue());
resp.setHeader("content-type", blobInfo.getContentType());
resp.setHeader("content-disposition", "attachment;filename=" + blobInfo.getFilename());
blobstoreService.serve(blobKey, resp);
Note that if you have files with more than 1MB in size, the file size is not sent to browser. GAE reads the blob 1MB at a time and overwrites the file size header in the response.
I found all this information here:
http://www.mail-archive.com/google-appengine#googlegroups.com/msg29314.html