I am trying to delete amazon s3 object using rest API but not getting any success. I have created the URL(signed url) at server side using java and then made XHR request to that URL at client side(i.e. from browser).
Java code that i have used to sign the url:
public static String getSignedURL(String fileName, int fileOwnerId, String versionId){
Date expiration = new Date();
long milliSeconds = expiration.getTime();
milliSeconds += 1000 * 60 * 10; // Add 10 minutes.
long seconds = (milliSeconds)/1000L;
String URL = null;
try {
String encodedFileName = URLEncoder.encode(fileName, "UTF-8").replaceAll("\\+", "%20");
String canonicalizedResource = "/"+AWS_BUCKET_NAME+"/" + fileOwnerId + "/" + encodedFileName;
String stringToSign = "DELETE\n\n\n" + seconds + "\n" + canonicalizedResource +"?versionId="+versionId;
byte[] keyBytes = AWS_SECRET_API_KEY.getBytes();
SecretKeySpec signingKey = new SecretKeySpec(keyBytes, "HmacSHA1");
Mac mac = Mac.getInstance("HmacSHA1");
mac.init(signingKey);
byte[] digest = mac.doFinal(stringToSign.getBytes());
byte[] base64bytes = Base64.encodeBase64(digest);
String signedString = new String(base64bytes, "UTF-8");
String signature = URLEncoder.encode(signedString, "UTF-8");
URL = "https://"+AWS_BUCKET_NAME+".s3.amazonaws.com/" + fileOwnerId +
"/" + encodedFileName +"?versionId="+versionId +"&Expires=" + seconds+"&AWSAccessKeyId=" +
AWS_ACCESS_KEY + "&Signature=" + signature;
} catch (UnsupportedEncodingException ex) {
Logger.getLogger(Utilities.class.getName()).log(Level.SEVERE, null, ex);
} catch (NoSuchAlgorithmException nsae) {
} catch (InvalidKeyException ike) {
}
System.out.println("URL IS :"+URL);
return URL;
}
And at client side:
var xhr = new XMLHttpRequest();
xhr.addEventListener("load", deleteComplete, false);
xhr.open('DELETE', URL_GENERATED_FROM_SERVER, true);
xhr.setRequestHeader ("Access-Control-Allow-Origin", "*");
xhr.send();
Using this code for downloading an object from amazon s3 bucket works fine by replacing 'DELETE' request with 'GET'. But delete is not working. I have searched a lot but there is very less help available for rest API.
Finally, i integrated the aws sdk to delete the object from amazon s3 bucket and it works like lightning. But unable to get help doing it with rest API. So now i have used rest API for uploading and downloading and the sdk for deleting an object.
Related
I was creating a signed URL for uploading an audio file. Url is used in Android apps. It's migration from azure to GCP so can write azure code here:
String cdnUrl = fileUploadConfig.has(containerName) ? fileUploadConfig.getString(containerName)
: Constants.AZURE_STORAGE_URL;
BlobServiceClient client = new BlobServiceClientBuilder().connectionString(storageConnectionString)
.buildClient();
BlobClient blobClient = client.getBlobContainerClient(containerName).getBlobClient(key);
BlobSasPermission blobSasPermission = new BlobSasPermission().setWritePermission(true);
OffsetDateTime expiryTime = OffsetDateTime.now().plusDays(1);
BlobServiceSasSignatureValues values = new BlobServiceSasSignatureValues(expiryTime, blobSasPermission)
.setStartTime(OffsetDateTime.now());
String signature = blobClient.generateSas(values);
Map<String,String> headerMap = new HashMap<String,String>(){{
put("x-ms-blob-type","BlockBlob");
}};
JSONObject res = new JSONObject();
System.out.println(blobClient.getBlobUrl());
cdnUrl += key;
res.put("presignedUrl", blobClient.getBlobUrl() + "?" + signature);
res.put("cdnUrl", cdnUrl);
res.put("header",new JSONObject(headerMap) );
return res.toString();
When I am transforming it into GCP, I am writing the following code:
String cdnUrl;
try {
cdnUrl = KhabriUtils.getSecret("s3-cdn-mapping-" + containerName);
} catch (Exception e) {
// TODO Auto-generated catch block
cdnUrl = Constants.GCP_STORAGE_URL;
e.printStackTrace();
}
Storage storage = StorageOptions.newBuilder().setProjectId(Constants.GCP_PROJECT_ID).build().getService();
// Define resource
BlobInfo blobInfo = BlobInfo.newBuilder(BlobId.of(containerName, key)).build();
// Generate Signed URL
Map<String, String> extensionHeaders = new HashMap<>();
extensionHeaders.put("Content-Type", "application/octet-stream");
URL url =
storage.signUrl(
blobInfo,
15,
TimeUnit.MINUTES,
Storage.SignUrlOption.httpMethod(HttpMethod.PUT),
Storage.SignUrlOption.withExtHeaders(extensionHeaders),
Storage.SignUrlOption.withV4Signature());
JSONObject res = new JSONObject();
cdnUrl += key;
res.put("presignedUrl", url);
res.put("cdnUrl", cdnUrl);
res.put("header", new JSONObject(extensionHeaders));
return res.toString();
It is giving me an error
Server response: code 403, body SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.GOOG4-RSA-SHA256
Tried to see if there is any permission missing.
public static String buildSignAndUrl(long partnerId, String secret, String path){
long timestamp = System.currentTimeMillis() / 1000L;
String base_String = String.format("%s%s%s", partnerId, path, timestamp);
byte[] baseStringBytes;
byte[] partner_key;
BigInteger sign = null;
try{
baseStringBytes = base_String.getBytes(StandardCharsets.UTF_8);
partner_key = secret.getBytes(StandardCharsets.UTF_8);
Mac mac = Mac.getInstance(HmacAlgorithms.HMAC_SHA_256.toString());
SecretKeySpec secret_key = new SecretKeySpec(partner_key, HmacAlgorithms.HMAC_SHA_256.toString());
mac.init(secret_key);
sign = new BigInteger(mac.doFinal(baseStringBytes));
} catch (Exception e){
e.printStackTrace();
}
String url = SANDBOX_HOST + path + String.format("?partner_id=%s×tamp=%s&sign=%s",
partnerId, timestamp, String.format("%032x", sign));
return url;
}
// Why use this url to send a post may cause "Wrong sign"? the timestamp in url and the timestamp to calculate a sign are same;
I am trying to create a signature for SAS token programmatically which should match with the one generated in the azure portal when a user selects few options. However the documentation is so confusing and doesn't guide you to achieve what is needed for the action. I have a file in azure file share and I want to access it by generating SAS token and sign it. I can easily do it by selecting the appropriate options in Azure portal and access the file, but I want to do the exact thing programmatically. I tried by java but the resulted SAS token signature doesn't match. Please help me.
Below is the SAS token when I generate it through Azure Portal:
?sv=2019-12-12&ss=f&srt=o&sp=r&se=2020-11-23T12:20:39Z&st=2020-11-23T04:20:39Z&spr=https&sig=XI%2FlSZSXp54XVwk2G%2F23j%2FjqsrojVqJoAonh6gdaAPk%3D
The corresponding options selected in azure portal for above sas token is as follows as shown in below image
The key used is key1 and is as follows:
l1wxekiJj9IcTw350w5c1MtVfYVP3qcz3zdxzCCp+YVaXqs9faOJfl/Z07AoLDnsnyn+POGjxjcFy3EF9g/r9Q==
What I tried is to generate a signature for the below options(String to Sign) as taken from the generated SAS token by azure portal and using the key1 above.
sv=2019-12-12&ss=f&srt=o&sp=r&se=2020-11-23T12:20:39Z&st=2020-11-23T04:20:39Z&spr=https
Below is my java code which generates the signature for the above string to sign
import org.apache.tomcat.util.codec.binary.Base64;
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.net.URLEncoder;
public class GenerateSAS {
public static void main(String[] args) throws Exception{
Mac sha256_HMAC = null;
String hash = null;
String input="sv=2019-12-12&ss=f&srt=o&sp=r&se=2020-11-23T12:20:39Z&st=2020-11-23T04:20:39Z&spr=https";
input=URLEncoder.encode(input, "UTF-8");
String key="l1wxekiJj9IcTw350w5c1MtVfYVP3qcz3zdxzCCp+YVaXqs9faOJfl/Z07AoLDnsnyn+POGjxjcFy3EF9g/r9Q==";
sha256_HMAC = Mac.getInstance("HmacSHA256");
SecretKeySpec secret_key = new SecretKeySpec(key.getBytes(), "HmacSHA256");
sha256_HMAC.init(secret_key);
hash = new String(Base64.encodeBase64(sha256_HMAC.doFinal(input.getBytes("UTF-8"))));
System.out.println(hash);
}
The above code generates signature which is MK/uu+NlURscoX1dymzipRN/Jb4aXyVzfbIVBz8l02M= and this is not equal to what was generated in the azure portal, which was XI%2FlSZSXp54XVwk2G%2F23j%2FjqsrojVqJoAonh6gdaAPk%3D
Please help me to generate the signature correctly which shall help me append it to SAS token for file object access.
If you want to create sas token to access Azure file service resource, the signature string should be like as below. For more details, please refer to here
StringToSign = accountname + "\n" +
signedpermissions + "\n" +
signedservice + "\n" +
signedresourcetype + "\n" +
signedstart + "\n" +
signedexpiry + "\n" +
signedIP + "\n" +
signedProtocol + "\n" +
signedversion + "\n"
For example
public static void createSasToken(){
String accountName = "accountName";
String key = "accountKey";
String resourceUrl = "https://"+accountName+".file.core.windows.net/fileShare/fileName";
/**
* please note the date formate should be ISO 8601 UTC formats
* for further information, please refer to https://learn.microsoft.com/en-us/rest/api/storageservices/formatting-datetime-values
*/
String start = "startTime";
String expiry = "expiry";
String apiVersion = "2019-12-12";
String stringToSign = accountName + "\n" +
"r\n" +
"f\n" +
"o\n" +
start + "\n" +
expiry + "\n" +
"\n" +
"https\n" +
apiVersion +"\n";
String signature = getHMAC256(key, stringToSign);
try{
String sasToken = "sv=" + azureApiVersion +
"&ss=f" +
"&srt=o" +
"&sp=r" +
"&se=" +URLEncoder.encode(expiry, "UTF-8") +
"&st=" + URLEncoder.encode(start, "UTF-8") +
"&spr=https" +
"&sig=" + URLEncoder.encode(signature, "UTF-8");
System.out.println(resourceUrl+"?"+sasToken);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
}
private static String getHMAC256(String accountKey, String signStr) {
String signature = null;
try {
SecretKeySpec secretKey = new SecretKeySpec(Base64.getDecoder().decode(accountKey), "HmacSHA256");
Mac sha256HMAC = Mac.getInstance("HmacSHA256");
sha256HMAC.init(secretKey);
signature = Base64.getEncoder().encodeToString(sha256HMAC.doFinal(signStr.getBytes("UTF-8")));
} catch (Exception e) {
e.printStackTrace();
}
return signature;
}
I am trying to create a signed URL in my App Engine code and pass it to the client to make the initial POST call to get the resumable upload location, following the following advice from the documentation:
Performing a resumable upload in a region where it wasn't initiated can
cause slow uploads. To avoid this, you can have the initial POST
request constructed and signed by the server, but then give the signed
URL to the client so that the upload is initiated from their location.
However, when a client makes a POST call to the signed URL, this call results in error 401, which tells me that Cloud Storage expects an authorization header.
This is how my signed URL looks like:
https://www.googleapis.com/upload/storage/v1/b/myBucket/o?uploadType=resumable&name=fileName&GoogleAccessId=xxx#appspot.gserviceaccount.com&Expires=1463628862&Signature=IZNU%2B2ApaUlogY9CR%2B4DC09WKkIpeZLeuqzrudyxA0nETH7Wpr4x8aRZTZj%2BGOriCDL8JfnoyAFhY1XqMG7VU3VPliu7LXoDSvuH0Cjaoz5lefNs80HpneAHI7HpAX2Uv2Lr57ZwbXM3EoegGiTcJb3ck51VH0%2FxTWSnqxlJsgQzo9%2BYKVhkHBzFn29k7k76qsGhN91g5CfTtSKtfa%2FiYwCffoB%2BctIdULwKF7yMugNkLKAfqxrWRXVAmrzXUK9njGTOQqfcvAtq4jJ23Aflo6ETmDKghZuZNLqoHs5umEhFRC1eFGd%2Be10RDRON%2F1ysXahWawSx4YOE9bEZXPKNtQ%3D%3D
I use the same code on the App Engine side to generate signed URLs for downloads and they work just fine.
Is this the correct way to assemble the URL for the initial POST call? Is there something else causing this error?
EDIT
Code to generate URL:
long expiration = System.currentTimeMillis()/1000 + 60;
String unsigned = stringToSign(expiration, gsKey, "POST");
String signature = sign(unsigned);
return new StringBuilder("https://www.googleapis.com/upload/storage/v1/b/")
.append(BUCKET)
.append("/o?uploadType=resumable&name=")
.append(gsKey)
.append("&GoogleAccessId=")
.append(identityService.getServiceAccountName())
.append("&Expires=")
.append(expiration)
.append("&Signature=")
.append(URLEncoder.encode(signature, "UTF-8"))
.toString();
private static String stringToSign(final long expiration, String gsKey, String httpVerb) {
String contentType = "";
String contentMD5 = "";
String canonicalizedExtensionHeaders = "";
String canonicalizedResource = "/" + BUCKET + "/" + gsKey;
String stringToSign = httpVerb + "\n" +
contentMD5 + "\n" +
contentType + "\n" +
expiration + "\n" +
canonicalizedExtensionHeaders + "\n" +
canonicalizedResource;
return stringToSign;
}
private static String sign(final String stringToSign) throws UnsupportedEncodingException {
SigningResult signingResult = identityService.signForApp(stringToSign.getBytes());
String encodedSignature = new String(Base64.encodeBase64(signingResult.getSignature(), false), "UTF-8");
return encodedSignature;
}
I'm using Play framework to develop consumer for Instagram real-time API. But still could not perform x-hub-signature verification properly. So, how can we perform Instagram x-hub-signature verification using Java and Play framework?
Here is my current code:
From the Play framework, I obtain the JSON payload using this method:
public static Result receiveInstaData(){
JsonNode json = request().body().asJson();
//obtain the x-hub-signature from the header
//obtain the corresponding client secret
VerificationResult verificationResult =
SubscriptionUtil.verifySubscriptionPostSignature(
clientSecret, json.toString(), xHubSignature);
if(verificationResult.isSuccess()){
//do something
}
}
Then inside the SubscriptionUtil, I perform verification using this following code:
public static VerificationResult verifySubscriptionPostSignature(String clientSecret, String rawJsonData, String xHubSignature) {
SecretKeySpec keySpec;
keySpec = new SecretKeySpec(clientSecret.getBytes("UTF-8"), HMAC_SHA1);
Mac mac;
mac = Mac.getInstance(HMAC_SHA1);
mac.init(keySpec);
byte[] result;
result = mac.doFinal(rawJsonData.getBytes("UTF-8"));
String encodedResult = Hex.encodeHexString(result);
return new VerificationResult(encodedResult.equals(xHubSignature), encodedResult);
}
I created a standalone Python script that copies the instagram-python implementation and both of them produce the same results for the same clientSecret and jsonString. Maybe I should provide with raw binary data instead of String.
If let's say we need a raw binary data for JSON request, then I need to create my custom BodyParser to parse the JSON request to raw binary data[5]
References:
[1-4]http://pastebin.com/g4uuDwzn (SO doesn't allow me to post more than 2 links, so I put all the references here. The links contain the signature verification in Ruby, Python and PHP)
[5]https://groups.google.com/forum/#!msg/play-framework/YMQb6yeDH5o/jU8FD--yVPYJ
[6]My standalone python script:
#! /usr/bin/env python
import sys
import hmac
import hashlib
hc_client_secret = "myclientsecret"
hc_raw_response = "[{\"subscription_id\":\"1\",\"object\":\"user\",\"object_id\":\"1234\",\"changed_aspect\":\"media\",\"time\":1297286541},{\"subscription_id\":\"2\",\"object\":\"tag\",\"object_id\":\"nofilter\",\"changed_aspect\":\"media\",\"time\":1297286541}]"
client_secret = hc_client_secret
raw_response = hc_raw_response
if len(sys.argv) != 3:
print 'Usage verify_signature <client_secret> <raw_response>.\nSince the inputs are invalid, use the hardcoded value instead!'
else:
client_secret = sys.argv[1]
raw_response = sys.argv[2]
print "client_secret = " + client_secret
print "raw_response = " + raw_response
digest = hmac.new(client_secret.encode('utf-8'), msg=raw_response.encode('utf-8'), digestmod=hashlib.sha1).hexdigest()
print digest
Finally I managed to find the solution. For the Controller in Play Framework, we need to use BodyParser.Raw so the we can extract the payload request as raw data, i.e. array of bytes.
Here's the code for the controller in Play Framework:
#BodyParser.Of(BodyParser.Raw.class)
public static Result receiveRawInstaData(){
Map<String, String[]> headers = request().headers();
RawBuffer jsonRaw = request().body().asRaw();
if(jsonRaw == null){
logger.warn("jsonRaw is null. Something is wrong with the payload");
return badRequest("Expecting serializable raw data");
}
String[] xHubSignature = headers.get(InstaSubscriptionUtils.HTTP_HEADER_X_HUB_SIGNATURE);
if(xHubSignature == null){
logger.error("Invalid POST. It does not contain {} in its header", InstaSubscriptionUtils.HTTP_HEADER_X_HUB_SIGNATURE);
return badRequest("You are not Instagram!\n");
}
String json;
byte[] jsonRawBytes;
jsonRawBytes = jsonRaw.asBytes();
json = new String(jsonRawBytes, StandardCharsets.UTF_8);
try {
String clientSecret = InstaSubscriptionUtils.getClientSecret(1);
VerificationResult verificationResult = SubscriptionUtil.verifySubscriptionPostRequestSignature
(clientSecret,jsonRawBytes, xHubSignature[0]);
if(verificationResult.isSuccess()){
logger.debug("Signature matches!. Received signature: {}, calculated signature: {}", xHubSignature[0], verificationResult.getCalculatedSignature());
}else{
logger.error("Signature doesn't match. Received signature: {}, calculated signature: {}", xHubSignature[0], verificationResult.getCalculatedSignature());
return badRequest("Signature does not match!\n");
}
} catch (InstagramException e) {
logger.error("Instagram exception.", e);
return internalServerError("Internal server error. We will attend to this problem ASAP!");
}
logger.debug("Received xHubSignature: {}", xHubSignature[0]);
logger.info("Sucessfully received json data: {}", json);
return ok("OK!");
}
And for the code for method verifySubscriptionPostRequestSignature in SubscriptionUtil
public static VerificationResult verifySubscriptionPostRequestSignature(String clientSecret, byte[] rawJsonData, String xHubSignature) throws InstagramException{
SecretKeySpec keySpec;
keySpec = new SecretKeySpec(clientSecret.getBytes(StandardCharsets.UTF_8), HMAC_SHA1);
Mac mac;
try {
mac = Mac.getInstance(HMAC_SHA1);
mac.init(keySpec);
byte[] result = mac.doFinal(rawJsonData);
String encodedResult = Hex.encodeHexString(result);
return new VerificationResult(encodedResult.equals(xHubSignature), encodedResult);
} catch (NoSuchAlgorithmException e) {
throw new InstagramException("Invalid algorithm name!", e);
} catch (InvalidKeyException e){
throw new InstagramException("Invalid key: " + clientSecret, e);
}
}
I implemented this solution in jInstagram, here is the link to the source code: SubscriptionUtil