I'm trying to upload file to AWS S3 bucket using Lambda server less application written in Java.
I'm hitting the endpoint with file using postman option binary (screenshot attached).
And I'm receiving binary content as string in my endpoint like as follows (screenshot attached).
I'm trying to convert this binary string to byte array and upload to S3 bucket.
I'm getting success response, But when i downloads the file / image it's looks like not an actual file.
Sample code:
#Override
public ServerlessOutput handleRequest(ServerlessInput serverlessInput, Context context) {
ServerlessOutput output = new ServerlessOutput();
String keyName = UUID.randomUUID().toString();
String content = serverlessInput.getBody();
byte[] encoded = this.toBinary(content).getBytes();
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(encoded.length);
metadata.setContentType(PNG_MIME);
s3.putObject(new PutObjectRequest(
ARTICLE_BUCKET_NAME,
keyName,
new ByteArrayInputStream(encoded),
metadata)
);
output.setBody("Successfully inserted article ");
}
private String toBinary(String data) {
byte[] bytes = data.getBytes();
StringBuilder binary = new StringBuilder();
for (byte b : bytes) {
int val = b;
for (int i = 0; i < 8; i++) {
binary.append((val & 128) == 0 ? 0 : 1);
val <<= 1;
}
binary.append(' ');
}
return binary.toString();
}
Related
Others have posted about this without receiving an answer and now I'm experiencing this same issue. Actually it has been going on for 9 months but I'm just now noticing it.
This sequence does not throw an exception and the message at the end is logged:
AmazonS3 s3 = AmazonS3ClientBuilder.defaultClient();
String bucket = "...";
String key = "...";
File f = new File("...");
PutObjectResult r = s3.putObject(bucket, key, f);
String etag = r.getETag();
LOGGER.info("file ... saved with etag = "+etag);
The file is not present in the bucket when I look.
This fails a few dozen times a day out of thousands of files posted. There are 25 active threads using this sequence of code. Is aws-java-sdk thread safe? Other ideas?
This is running on an ec2 instance in the amazon cloud.
Details:
aws-java-sdk-s3-1.11.693.jar
java: 1.8.0_201-b09
ubuntu: 4.4.0-1077-aws
I saw this issue when uploading a 27MB file using AmazonS3.putObject. The command returned a PutObjectResult but there was no S3 object in the expected location and the result metadata (result.getMetadata().getContentLength()) showed 0 bytes. I fixed this by using multi-part upload per this link.
private static final Logger LOGGER = LoggerFactory.getLogger(S3Handler.class);
private static final long MAX_SINGLE_PART_UPLOAD_BYTES = 5 * 1024 * 1024;
private final AmazonS3 amazonS3;
public S3Handler(AmazonS3 amazonS3) {
this.amazonS3 = amazonS3;
}
public void putS3Object(String bucket, String objectKey, File file) {
if (file.length() <= MAX_SINGLE_PART_UPLOAD_BYTES) {
putS3ObjectSinglePart(bucket, objectKey, file);
} else {
putS3ObjectMultiPart(bucket, objectKey, file);
}
}
private void putS3ObjectSinglePart(String bucket, String objectKey, File file) {
PutObjectRequest request = new PutObjectRequest(bucket, objectKey, file);
PutObjectResult result = amazonS3.putObject(request);
long bytesPushed = result.getMetadata().getContentLength();
LOGGER.info("Pushed {} bytes to s3://{}/{}", bytesPushed, bucket, objectKey);
}
private void putS3ObjectMultiPart(String bucket, String objectKey, File file) {
long contentLength = file.length();
long partSize = MAX_SINGLE_PART_UPLOAD_BYTES;
List<PartETag> partETags = new ArrayList<>();
// Initiate the multipart upload.
InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(bucket, objectKey);
InitiateMultipartUploadResult initResponse = amazonS3.initiateMultipartUpload(initRequest);
// Upload the file parts.
long fileOffset = 0;
for (int partNumber = 1; fileOffset < contentLength; ++partNumber) {
// Because the last part could be less than 5 MB, adjust the part size as needed.
partSize = Math.min(partSize, (contentLength - fileOffset));
// Create the request to upload a part.
UploadPartRequest uploadRequest = new UploadPartRequest()
.withBucketName(bucket)
.withKey(objectKey)
.withUploadId(initResponse.getUploadId())
.withPartNumber(partNumber)
.withFileOffset(fileOffset)
.withFile(file)
.withPartSize(partSize);
// Upload the part and add the response's ETag to our list.
UploadPartResult uploadResult = amazonS3.uploadPart(uploadRequest);
LOGGER.info("Uploading part {} of Object s3://{}/{}", partNumber, bucket, objectKey);
partETags.add(uploadResult.getPartETag());
fileOffset += partSize;
}
// Complete the multipart upload.
CompleteMultipartUploadRequest compRequest = new CompleteMultipartUploadRequest(bucket, objectKey, initResponse.getUploadId(), partETags);
amazonS3.completeMultipartUpload(compRequest);
}
I am trying to encode, Image to Base 64 string in AngularJS2
handleFileSelect(evt) {
var files = evt.target.files;
var file = files[0];
if (files && file) {
var reader = new FileReader();
reader.onload = this._handleReaderLoaded.bind(this);
reader.readAsBinaryString(file);
}
}
_handleReaderLoaded(readerEvt) {
var binaryString = readerEvt.target.result;
this.model.UserProfileImageBase64 = btoa(binaryString);
console.log(this.model.UserProfileImage);
}
I am receiving a different string,in compare to encoding it from Java
Base64.encodeToString(getBytesFromBitmap(bitmap),Base64.NO_WRAP);
Any Idea how can we match both Base64 encoding? I had tried same with base64 encoding in angular as well
this.model.UserProfileImageBase64 = Base64.encode(binaryString);
But no dfiference in result.
BtoA and Base64 producing same result and, if I am verifying it online I am getting image as well but I need it in the same format which is generated by Java
I am getting the same result with java and javascript.
With java I did:
byte[] imageBytes = IOUtils.toByteArray(new URL("https://www.w3schools.com/css/paris.jpg"));
String base64 = Base64.getEncoder().encodeToString(imageBytes);
System.out.println(base64);
With javascript I did:
const fileReader: FileReader = new FileReader();
fileReader.readAsDataURL(file);
fileReader.onloadend = () => {
retVal.next(fileReader.result);
};
NOTE: I would suggest not to use readBynaryAsString as it is deprecated and might not work. FileReader
NOTE: That in javascript I wait till the file is uploaded to get the data and I passed to an observable next(...)
changeListener($event) : void {
this.readThis($event.target);
}
readThis(inputValue: any): void {
var file:File = inputValue.files[0];
var myReader:FileReader = new FileReader();
myReader.onloadend = (e) => {
this.image = myReader.result;
}
myReader.readAsDataURL(file);
}
component.html
<input type="file" accept="image/*" (change)="changeListener($event)">
Follow this link Angular 2 encode image to base64
In one of my application i am cropping the image using http://fengyuanchen.github.io/cropper/
The resultant cropped image am getting in the base64 dataURL format, but i required that to be in file object format.
How to convert the dataURL to file either in client side or server side.
Use Blob instead of the deprecated BlobBuilder. The code is very clean and simple. (Manuel Di Iorio's code is deprecated.)
function dataURLtoBlob(dataurl) {
var arr = dataurl.split(','), mime = arr[0].match(/:(.*?);/)[1],
bstr = atob(arr[1]), n = bstr.length, u8arr = new Uint8Array(n);
while(n--){
u8arr[n] = bstr.charCodeAt(n);
}
return new Blob([u8arr], {type:mime});
}
//test:
//var blob = dataURLtoBlob('data:text/plain;base64,YWFhYWFhYQ==');
Data URI scheme
How to convert dataURL to file object in javascript?
function dataURItoBlob(dataURI) {
// convert base64 to raw binary data held in a string
// doesn't handle URLEncoded DataURIs - see SO answer #6850276 for code that does this
var byteString = atob(dataURI.split(',')[1]);
// separate out the mime component
var mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0];
// write the bytes of the string to an ArrayBuffer
var ab = new ArrayBuffer(byteString.length);
var ia = new Uint8Array(ab);
for (var i = 0; i < byteString.length; i++) {
ia[i] = byteString.charCodeAt(i);
}
// write the ArrayBuffer to a blob, and you're done
var bb = new BlobBuilder();
bb.append(ab);
return bb.getBlob(mimeString);
}
Then just append the blob to a new FormData object and post it to your server using ajax:
var blob = dataURItoBlob(someDataUrl);
var fd = new FormData(document.forms[0]);
var xhr = new XMLHttpRequest();
fd.append("myFile", blob);
xhr.open('POST', '/', true);
xhr.send(fd);
Thats my validation for input.
$data = $_POST['thumb'];
$uriPhp = 'data://' . substr($data, 5);
if ( base64_encode(base64_decode($uriPhp))){
$_POST['thumb'] = $uriPhp;
}
for saving I am using : http://www.verot.net/php_class_upload.htm
$foo = new Upload($_POST['thumb']);
if ($foo->uploaded) {
// save uploaded image with a new name
$foo->file_new_name_body = $name;
$foo->image_convert = 'png';
$foo->Process("uploads/");
}
I am trying to update photo using Google apps engine.I have imageurl i convert it into byte array and then encode
it using base64.i got encoded string,now i m trying to update photodata using directory API Reference
https://developers.google.com/admin-sdk/directory/v1/reference/users/photos#resource
after update i got error invalid byteString.I face this problem from yesterday.So Let me know where i did wrong? Below is my code.
import com.google.appengine.repackaged.org.apache.commons.codec.binary.Base64;
above class used for Base64.
URL url = new URL(myImageUrl);
ByteArrayOutputStream bais = new ByteArrayOutputStream();
InputStream is = null;
try {
is = url.openStream ();
byte[] byteChunk = new byte[4096]; // Or whatever size you want to read in at a time.
int n;
while ( (n = is.read(byteChunk)) > 0 ) {
bais.write(byteChunk, 0, n);
}
System.out.println(byteChunk);
byte[] encoded = Base64.encodeBase64(byteChunk);
String ecodedString = new String(encoded, "UTF-8");
ecodedString = ecodedString.replace("/", "_");
ecodedString = ecodedString.replace("+", "-");
ecodedString = ecodedString.replace("=", "*");
System.out.println(ecodedString);
Padding could be the problem, try not replacing "=" with "*". See also:
Converting string to web-safe Base64 format
p.s. repackaged libs are discouraged in app engine; you may use DatatypeConverter.printBase64Binary() instead.
I'm writing a little Java app which implements an http service that receives http post commands from a client.
The class I'm using to implement all of this is HttpHandler and HttpServer in the com.sun.net. package.
Now I'm implementing an handle(HttpExchange exchange) function which handles the request, and I'm having truble reading the post values received by the request because the only access that I have to these values is via HttpExchange.getResponseBody() which is just an output stream.
I'm looking to parse text post values and uploaded files.
I have written classes that process multipart requests for my project Sceye-Fi, an HTTP server that uses the com.sun.net.httpserver classes that come with java 6, to receive photo uploads from an Eye-Fi card.
This can help with file uploads (multipart posts).
For a non-multipart post, you would need to do something like this:
// determine encoding
Headers reqHeaders = exchange.getRequestHeaders();
String contentType = reqHeaders.getFirst("Content-Type");
String encoding = "ISO-8859-1";
if (contentType != null) {
Map<String,String> parms = ValueParser.parse(contentType);
if (parms.containsKey("charset")) {
encoding = parms.get("charset");
}
}
// read the query string from the request body
String qry;
InputStream in = exchange.getRequestBody();
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte buf[] = new byte[4096];
for (int n = in.read(buf); n > 0; n = in.read(buf)) {
out.write(buf, 0, n);
}
qry = new String(out.toByteArray(), encoding);
} finally {
in.close();
}
// parse the query
Map<String,List<String>> parms = new HashMap<String,List<String>>();
String defs[] = qry.split("[&]");
for (String def: defs) {
int ix = def.indexOf('=');
String name;
String value;
if (ix < 0) {
name = URLDecoder.decode(def, encoding);
value = "";
} else {
name = URLDecoder.decode(def.substring(0, ix), encoding);
value = URLDecoder.decode(def.substring(ix+1), encoding);
}
List<String> list = parms.get(name);
if (list == null) {
list = new ArrayList<String>();
parms.put(name, list);
}
list.add(value);
}
An alternative would be using HttpService from HttpCore.
There is a Basic HTTP server example in the documentation