Earlier, I asked a question https://stackoverflow.com/questions/35581090/can-i-use-resumable-upload-for-gae-blobstore-api
about resumable uploading with Blobstire API.
For myself, I decided that it is impossible to implement resumable uploading with Blobstire API.
In this case i am trying to implement Google Cloud Storage with Java Client Library. At the moment I made the download my video file to bucket and serve video. My servlet look like in google example
#Override
public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
GcsOutputChannel outputChannel =
gcsService.createOrReplace(getFileName(req), GcsFileOptions.getDefaultInstance());
copy(req.getInputStream(), Channels.newOutputStream(outputChannel));
}
private GcsFilename getFileName(HttpServletRequest req) {
String[] splits = req.getRequestURI().split("/", 4);
if (!splits[0].equals("") || !splits[1].equals("gcs")) {
throw new IllegalArgumentException("The URL is not formed as expected. " +
"Expecting /gcs/<bucket>/<object>");
}
return new GcsFilename(splits[2], splits[3]);
}
private void copy(InputStream input, OutputStream output) throws IOException {
try {
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead = input.read(buffer);
while (bytesRead != -1) {
output.write(buffer, 0, bytesRead);
bytesRead = input.read(buffer);
}
} finally {
input.close();
output.close();
}
}
Now I need to implement
resumable upload (due to poor internet on mobile devices)
uploading by chunck (due to limitation in size of one request with 32mb)
I realized, that serverside of resumable upload should be organized manually and my backend should be able to give me range of uploaded chunck and allow to continue booting in to OutputChannel.
The documentation for the GcsOutputChannel says:
This class is serializable, this allows for writing part of a file,
serializing the GcsOutputChannel deserializing it, and continuing to
write to the same file. The time for which a serialized instance is
valid is limited and determined by the Google Cloud Storage service
I have not enough experience, so the question may be stupid:
Please somebody tell me how to serialize my GcsOutputChannel? I do not understand where I can save the file containing the serialized object.
By the way, can anyone knows how long Google Cloud Storage service store that serialized object?
You can serialize your GcsOutputChannel using any Java serialization means (typically using ObjectOutputStream). If you run on AE you probably want to save that serialized bytes in the Datastore (as Datastore Blob). See this link for how to convert the serialized object to and from byte array.
Related
I am generating S3 pre signed URLs so that the client(mobile app) can PUT an image directly to S3 instead of going through a service. For my use case the expiry time of the pre signed URL needs to be configured for a longer window (10-20 mins). Therefore, I want to limit the size of file upload to S3 so that any malicious attacker can not upload large files to the S3 bucket. The client will get the URL from a service which has access to the S3 bucket. I am using AWS Java SDK.
I found that this can be done using POST forms for browser uploads but how can I do this using just signed S3 URL PUT?
I was using S3-Signed-URLS the first time and was also concerned about this.
And I think this whole signed Urls stuff is a bit of a pain because you cant put a maximum Object/Upload size limit on them.
I think thats something very important on file-uploads in general, that is just missing..
By not having this option you are forced to handle that problem with the expiry time etc. This gets really messy..
But it seems that you can use S3 Buckets also with normal Post-Requests, what has a content-length parameter in their policy.
So I'll probably exchange my Signed-URLS with POST-Routes in the future.
I think for proper, larger applications this is the way to go.(?)
What might help with your issue:
In the JavaScript SDK there is a method / function that gets you only the meta-data of the an S3-Object (including File Size) without downloading the whole file.
It's called s3.headObject()
I think, after the upload is done, it takes some time for AWS to process that newly uploaded file and then is available in your bucket.
What I did was, I set a timer after each upload to check the file-size and if its bigger 1mb, it will delete the file.
I think for production you wanna log that somewhere in a DB.
My FileNames also include the user-id of who uploaded the file.
That way, you can block an account after a too big upload if you wanted.
This here worked for me in javascript..
function checkS3(key) {
return new Promise((resolve, reject) => {
s3.headObject(headParams, (err, metadata) => {
console.log("setTimeout upload Url");
if (err && ["NotFound", "Forbidden"].indexOf(err.code) > -1) {
// console.log(err);
return reject(err);
//return resolve();
} else if (err) {
const e = Object.assign({}, Errors.SOMETHING_WRONG, { err });
// console.log(e);
// console.log(err);
return reject(e);
}
return resolve(metadata);
});
});
}
I am using AWS S3 bucket for Uploading list of files, I am using MultipleFileUpload and here is my request, while uploading the files if the internet gets disconnected and again came back then uploading process is not getting updated. How can I do so when internet is coming back, it should automatically get uploaded from the last position.
final ObjectMetadataProvider metadataProvider = new ObjectMetadataProvider() {
public void provideObjectMetadata(File file, ObjectMetadata metadata) {
}
};
final MultipleFileUpload multipleFileUpload = transferManager.uploadFileList(HttpUrls.IMAGE_BUCKET_NAME, "photos/mint_original/", myDir_temp, upload_file, metadataProvider);
The TransferManager component in the AWS Android SDK has been deprecated in favor of the TransferUtility component. The TransferUtility component allows you to pause and resume transfers. It also has support for network monitoring and will automatically pause and resume transfers when the network goes down and comes back up. Here is the link to the TransferUtility documentation - https://aws-amplify.github.io/docs/android/storage
I have log files being dropped into a GCS bucket regularly (e.g. gs://my-bucket/log.json) I want to setup a java process to process the files, gzip them, and move them to a separate bucket where I archive files (i.e. move it to gs://archived-logs/my-bucket/log.json.gz)
gsutil cp -z seems to be the only option I can find currently. Has anybody implemented it in a feasible manner using their Java API?
Ok, I think I solved it. Standard streams solution in the ending.
GcsOutputChannel gcsOutputChannel = gcsService.createOrReplace(new GcsFilename("my-bucket", "log.json.gz"),
new GcsFileOptions.Builder().build());
GZIPOutputStream outputStream = new GZIPOutputStream(Channels.newOutputStream(gcsOutputChannel));
GcsInputChannel inputChannel = gcsService
.openReadChannel(new GcsFilename("my-bucket", "log.json"), 10000);
InputStream inStream = Channels.newInputStream(inputChannel);
byte[] byteArr = new byte[10000];
while (inStream.read(byteArr) > 0) {
outputStream.write(byteArr);
}
For the latest streaming writer you can follow the below code. Do note that gcp will automatically decompress the object while serving it to web clients. If it doesnt work then add the following header to accept gzip files, "Accept-Encoding": "gzip".
Credentials credentials = GoogleCredentials.fromStream(Files.newInputStream(Paths.get(environmentConfig.getServiceAccount())));
Storage storage = StorageOptions.newBuilder().setProjectId(environmentConfig.getProjectID()).setCredentials(credentials).build().getService();
try (WriteChannel writer = storage.writer(blobInfo)) {
GZIPOutputStream gzipOutputStream = new
GZIPOutputStream(Channels.newOutputStream(writer));
gzipOutputStream.write(ByteBuffer.wrap(objectData, 0, objectData.length).array());
gzipOutputStream.flush();
gzipOutputStream.close();
} catch (IOException ex) {
log.error("Upload Error: {}", ex.getMessage());
}
I have some problems in my work..
I have stored TTL serial camera images to MicroSD card successfully using Arduino UNO with the help of Adafruit Tutorial (learn.adafruit.com/ttl-serial-camera/overview) but when i m transferring that images through Zigbee transmitter, At the comport (Zigbee receiver) i m receiving random words. And i think its ASCII.
I want to save images receiving from comport to the folder of my PC.
Is it possible?
I have seen in some forums that use the java or python code, but i can't understand how to use it?
Read image data from COM7 port in Java
I guess this is what you are looking for:
import serial
ser = serial.Serial('/dev/tty.usbserial', 9600)
image = ser.read()
with open('/tmp/image', 'wb') as file:
file.write(image)
Works only in Python 3, in Python 2 you need to use io.open. You may need to install serial-modul first if you don't already have it. I'm not familiar with the Arduino-C-dialect you need to send the image over the com-port...
Arduino IDE's Serial Monitor is using the Serial class to communicate over a serial comm. port.
// Receive & send methods from the SerialMonitor class.
private void send(String s) {
..
serial.write(s);
}
public void message(final String s) {
..
textArea.append(s);
}
My suggestion is to reuse that (Java) code, but since the Serial class is designed for text-only communication you would need to encode the image bytes into e.g. Base64 encoding with this library and decode it on the PC.
If the transfer speed is important, and there is an Arduino binary-based serial communication library, you should use that.
UPDATE
You can read raw bytes from the serial port via the mentioned Serial class like this:
...
Serial port = ...;
byte[] buffer = new byte[1024]; // 1KB buffer
OutputStream imageOutput = new FileOutputStream(...);
// Wait for the image.
while (port.available() > 0) {
int numBytes = port.readBytes(buffer);
if (numBytes > 0) {
imageOutput.write(buffer, numBytes);
}
}
imageOutput.flush();
imageOutput.close();
...
I have a java application on Websphere that is using Apache Commons FTPClient to retrieve files from a Windows server via FTP. When I deploy the application to Websphere running in a Windows environment, I am able to retrieve all of the files cleanly. However, when I deploy the same application to Webpshere on Linux, there are cases where I am getting an incomplete or corrupt files. These cases are consistent though, such that the same files will fail every time and give back the same number of bytes (usually just a few bytes less than what I should be getting). I would say that I can read approximately 95% of the files successfully on Linux.
Here's the relevant code...
ftpc = new FTPClient();
// set the timeout to 30 seconds
ftpc.enterLocalPassiveMode();
ftpc.setDefaultTimeout(30000);
ftpc.setDataTimeout(30000);
try
{
String ftpServer = CoreApplication.getProperty("ftp.server");
String ftpUserID = CoreApplication.getProperty("ftp.userid");
String ftpPassword = CoreApplication.getProperty("ftp.password");
log.debug("attempting to connect to ftp server = "+ftpServer);
log.debug("credentials = "+ftpUserID+"/"+ftpPassword);
ftpc.connect(ftpServer);
boolean login = ftpc.login(ftpUserID, ftpPassword);
if (login)
{
log.debug("Login success..."); }
else
{
log.error("Login failed - connecting to FTP server = "+ftpServer+", with credentials "+ftpUserID+"/"+ftpPassword);
throw new Exception("Login failed - connecting to FTP server = "+ftpServer+", with credentials "+ftpUserID+"/"+ftpPassword);
}
is = ftpc.retrieveFileStream(fileName);
ByteArrayOutputStream out = null;
try {
out = new ByteArrayOutputStream();
IOUtils.copy(is, out);
} finally {
IOUtils.closeQuietly(is);
IOUtils.closeQuietly(out);
}
byte[] bytes = out.toByteArray();
log.info("got bytes from input stream - byte[] size is "+ bytes.length);
Any assistance with this would be greatly appreciated.
Thanks.
I have a suspicion that the FTP might be using ASCII rather than binary transfer mode, and mapping what it thinks are Window end-of-line sequences in the files to Unix end-of-lines. For files that are really text, this will work. For files that are really binary, the result will be corruption and a slightly shorter file if the file contains certain sequences of bytes.
See FTPClient.setFileType(...).
FOLLOWUP
... so why this would work on Windows and not Linux remains a mystery for another day.
The mystery is easy to explain. You were FTP'ing files from a Windows machine to a Windows machine, so there was no need to change the end-of-line markers.