How get all log events in amazon via loop? - java

Every amazon API has it's own token which you have to set to next request. But with aws log api I got infinity loop:
public class Some {
public static void main (String[] args) {
final GetLogEventsRequest request = new GetLogEventsRequest()
.withLogGroupName("myGroup")
.withLogStreamName("myStrean");
final AWSLogs awsLogs = AWSLogsClientBuilder.defaultClient();
Collection<OutputLogEvent> result = new ArrayList<>();
GetLogEventsResult response = null;
do {
response = awsLogs.getLogEvents(request);
result.addAll(response.getEvents());
request.withNextToken(response.getNextBackwardToken());
} while (response.getNextBackwardToken() != null);
}
}
From documentation:
nextBackwardToken
The token for the next set of items in the backward direction. The token expires after 24 hours. This token will never be null. If you have reached the end of the stream, it will return the same token you passed in.
So it can not be null like LastEvaluatedKey when you scan dynamodb:
Map<String, AttributeValue> lastKeyEvaluated = null;
do {
ScanRequest scanRequest = new ScanRequest()
.withTableName("ProductCatalog")
.withLimit(10)
.withExclusiveStartKey(lastKeyEvaluated);
ScanResult result = client.scan(scanRequest);
for (Map<String, AttributeValue> item : result.getItems()){
printItem(item);
}
lastKeyEvaluated = result.getLastEvaluatedKey();
} while (lastKeyEvaluated != null);
So and what I should pass to request.withNextToken if we speak about log api??? And if nextBackwardToken (and nextForwardToken too) can not be null - how to detect that I receive the last response from amazon???

The documentation which you've cited is quite straightforward. I think you need something like this:
final AWSLogs awsLogs = AWSLogsClientBuilder.defaultClient();
Collection<OutputLogEvent> result = new ArrayList<>();
String nextToken = null;
GetLogEventsResult response;
do {
GetLogEventsRequest request = new GetLogEventsRequest()
.withLogGroupName("myGroup")
.withLogStreamName("myStrean");
if (nextToken != null) request = request.withNextToken(nextToken);
response = awsLogs.getLogEvents(request);
result.addAll(response.getEvents());
// check if token is the same
if (response.getNextForwardToken().equals(nextToken)) break;
// save new token
nextToken = response.getNextForwardToken();
} while (true);
So, you simply need to create request each time with the new token, until it becomes equals to the old one.

Related

Check if user is logged in aws cognito, JAVA

I am using Java as back end to communicate with AWS Cognito. I am able to login, logout, create users, sign out and other functions. I am also able to verify an access token by following this link:
But I want to verify if a user is logged in or not.
In JAVA, is there a isLoggedin() function that returns a boolean or is there a way to see if the token is revoked? These functions exist for Android and iOS but what about JAVA.
I thought this verifies if the token is active, but it only verifies if the token is in the right format:
// This parses token to JWT.
JWT jwtparser = JWTParser.parse(accessToken);
String JWTissuer = jwtparser.getJWTClaimsSet().getIssuer();
JWSHeader header = (JWSHeader) jwtparser.getHeader();
Object token_use = jwtparser.getJWTClaimsSet().getClaim("token_use");
Object exp = jwtparser.getJWTClaimsSet().getClaim("iat");
Date expirationDate = jwtparser.getJWTClaimsSet().getExpirationTime();
// Read in JSON Key file saved somewhere safe
File file = new File("jwks.json");
String content = FileUtils.readFileToString(file, "utf-8");
JSONObject JsonObjects = new JSONObject(content);
JSONArray keysArray = JsonObjects.getJSONArray("keys");
JSONObject keyString = (JSONObject) keysArray.get(1);
if (header.getKeyID().equals(keyString.get("kid")) && token_use.toString().equals("access") && JWTissuer.equals("https://cognito-idp.us-west-2.amazonaws.com/us-west-2_xxxxxxx")) {
return true;
} else { return false; }
I want to see if a user is logged in. I have not found an appropriate method to do so.
Mahalo
I found a work around:
public boolean isLoggedin(String accessToken) {
GetUserRequest request = new GetUserRequest();
request.withAccessToken(accessToken);
AWSCognitoIdentityProviderClientBuilder builder =
AWSCognitoIdentityProviderClientBuilder.standard();
builder.withRegion("us-west-2");
AWSCognitoIdentityProvider cognitoCreate = builder.build();
GetUserResult result = cognitoCreate.getUser(request);
try {
System.out.println("success");
return true;
} catch (Exception e) {
e.getMessage();
return false;
}
}

Insert row without overwritting data

I end up here a lot from google and since I started trying to make an android app for myself I'm stumped since the v4 API isn't very helpful. I've search on here but I haven't seen an answer that answer my question.
I have a budget spreadsheet. My app is going to grab the data from the sheet and filter it to my current payweek. So I can look at all my current transactions and future ones so I can know how much I have extra to spend. Then I want to just be able to insert a new traction from the app, say if I get gas I want to be able to add that in a couple clicks rather than try to spend a few minutes editing the sheets in the sheet app. Later I plan to customize the formula for the cell I'm putting it in as well as copy it to the cells north and south of it so the math works out.
I used the Android Quickstart to be able to read data easy enough. I've since tweaked that to filter the sheets data to filter my data. I've even gone as far to get it be able to write data to the sheet. The problem is that I can't find any example in java/android in the same style as the quick start to know how to insert a row in the middle of the sheet instead of overwrite the cells. I'm assuming I have to use the INSERTDATAOPTION=INSERT_ROWS somewhere but I can't find anywhere to add it. The documentation suggest I use spreadsheets.values.append but .append never shows up as an option. I only get BatchGet, BatchUpdate, Get, Update.
I'm a beginner and I'm sure my code is clumsy that I've pieced together after hours of googling but I'm sure I'm just missing something easy. Any help would be appreciated.
private List<String> PostDataForApi() throws IOException {
String spreadsheetID = getResources().getString(R.string.my_google_spreadsheet_id);
Integer sheetID = getResources().getInteger(R.integer.my_google_sheet_id);
List<RowData> rowData = new ArrayList<RowData>();
List<CellData> cellData = new ArrayList<CellData>();
String value = "test";
String formula = "=IF(COUNTBLANK(C510) = 2,\"\",Sum(B511+(SUM(C510))))";
String value2 = "999";
CellData cell = new CellData();
cell.setUserEnteredValue(new ExtendedValue().setStringValue(value));
CellData cell2 = new CellData();
cell2.setUserEnteredValue(new ExtendedValue().setFormulaValue(formula));
CellData cell3 = new CellData();
cell3.setUserEnteredValue(new ExtendedValue().setStringValue(value2));
cellData.add(cell);
cellData.add(cell2);
cellData.add(cell3);
rowData.add(new RowData().setValues(cellData));
BatchUpdateSpreadsheetRequest batchRequests = new BatchUpdateSpreadsheetRequest();
BatchUpdateSpreadsheetResponse response;
List<Request> requests = new ArrayList<Request>();
AppendCellsRequest appendCellReq = new AppendCellsRequest();
appendCellReq.setSheetId(sheetID);
appendCellReq.setRows( rowData );
appendCellReq.setFields("userEnteredValue");
requests = new ArrayList<Request>();
requests.add( new Request().setAppendCells(appendCellReq));
batchRequests = new BatchUpdateSpreadsheetRequest();
batchRequests.setRequests( requests );
response = this.mService.spreadsheets().batchUpdate(spreadsheetID, batchRequests).execute();
System.out.println(response.toPrettyString());
return null;
}
I figured it out after some mind numbing throw things against the wall and see what sticks. I had to do it in two steps. The first step will insert a row into your sheet at row 32 so you'll get a blank row 33. Then the second area will insert values into that blank row. I hope this helps someone in the future.
String spreadsheetID = getResources().getString(R.string.my_google_spreadsheet_id);
Integer sheetID = getResources().getInteger(R.integer.my_google_sheet_id);
BatchUpdateSpreadsheetResponse response;
BatchUpdateSpreadsheetRequest batchRequests = new BatchUpdateSpreadsheetRequest();
List<Request> requests = new ArrayList<Request>();
InsertDimensionRequest insertDimensionRequest = new InsertDimensionRequest();
DimensionRange dimRange = new DimensionRange();
dimRange.setStartIndex(32);
dimRange.setEndIndex(33);
dimRange.setSheetId(sheetID);
dimRange.setDimension("ROWS");
insertDimensionRequest.setRange(dimRange);
insertDimensionRequest.setInheritFromBefore(false);
requests.add( new Request().setInsertDimension(insertDimensionRequest));
batchRequests = new BatchUpdateSpreadsheetRequest();
batchRequests.setRequests( requests );
response = this.mService.spreadsheets().batchUpdate(spreadsheetID, batchRequests).execute();
System.out.println(response.toPrettyString());
List<List<Object>> argData = getData(entryTitle, entryValue);
ValueRange vRange = new ValueRange();
vRange.setRange("2016!A33");
vRange.setValues(argData);
List<ValueRange> vList = new ArrayList<>();
vList.add(vRange);
BatchUpdateValuesRequest batchRequest = new BatchUpdateValuesRequest();
batchRequest.setValueInputOption("USER-ENTERED");
batchRequest.setData(vList);
this.mService.spreadsheets().values().batchUpdate(spreadsheetID, batchRequest).execute();
I've been having sooo much unnecessary hustle with the v4 google sheets api, that it was ridiculous. Therefore, I reverted to the gdata (v3) api, much better easier to follow and much better documented also.
Here is the link to setting it up and a few examples https://developers.google.com/google-apps/spreadsheets/
The only thing missing was the authorization process in those notes, which was a hustle, but after some digging I was able to get a authorization code base, as shown below.
public class YourClass {
// Application name
private static final String APPLICATION_NAME = "Your-Application-Name";
// account info and p12
private static final String ACCOUNT_P12_ID = "Get-the-details-developer-console-google";
private static final File P12FILE = new File("D:/path/Drive API Test-bf290e0ee314.p12");
// scopes
private static final List<String> SCOPES = Arrays.asList(
"https://docs.google.com/feeds",
"https://spreadsheets.google.com/feeds");
// Spreadsheet API URL
private static final String SPREADSHEET_URL = "https://spreadsheets.google.com/feeds/spreadsheets/private/full";
private static final URL SPREADSHEET_FEED_URL;
static {
try {
SPREADSHEET_FEED_URL = new URL(SPREADSHEET_URL);
} catch (MalformedURLException e) {
throw new RuntimeException(e);
}
}
// Authorize
private static Credential authorize() throws Exception {
System.out.println("authorize in");
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
JsonFactory jsonFactory = new JacksonFactory();
GoogleCredential credential = new GoogleCredential.Builder()
.setTransport(httpTransport)
.setJsonFactory(jsonFactory)
.setServiceAccountId(ACCOUNT_P12_ID)
.setServiceAccountPrivateKeyFromP12File(P12FILE)
.setServiceAccountScopes(SCOPES)
.build();
boolean ret = credential.refreshToken();
// debug dump
System.out.println("refreshToken:" + ret);
// debug dump
if (credential != null) {
System.out.println("AccessToken:" + credential.getAccessToken());
}
System.out.println("authorize out");
return credential;
}
// Get service
private static SpreadsheetService getService() throws Exception {
System.out.println("service in");
SpreadsheetService service = new SpreadsheetService(APPLICATION_NAME);
service.setProtocolVersion(SpreadsheetService.Versions.V3);
Credential credential = authorize();
service.setOAuth2Credentials(credential);
// debug dump
System.out.println("Schema: " + service.getSchema().toString());
System.out.println("Protocol: " + service.getProtocolVersion().getVersionString());
System.out.println("ServiceVersion: " + service.getServiceVersion());
System.out.println("service out");
return service;
}
From that onward, I was able to perform a number of inserts and appends to the sheets, without any major hustle.

Sending a POST request with multiple keywords as parameters using HTMLUnit

I am sending a POST request using HTMLUnit that sends keywords as parameters. An example of the URL is:
website.com/foo/bar/api?keywords=word1,word2,word3&language=en
The problem is my application is dynamically picking these words and the amount of words can go up to 10 or 20 or even more. How do you append a Set of words as values to a HTTP request. My code at the moment is:
requestSettings = new WebRequest(new URL("website.com/foo/bar/api?"),
HttpMethod.POST);
Iterator<String> itr = list.iterator();
while(itr.hasNext()) {
requestSettings.getRequestParameters()
.add(new NameValuePair("keywords[]", itr.next()));
}
requestSettings.getRequestParameters().add(new NameValuePair("language", "en"));
System.out.println(requestSettings.getUrl().toString());
response = webClient.getPage(requestSettings).getWebResponse();
This code does not return a valid respone. What am I doing wrong here?
Give this a try:
using (var client = new WebClient())
{
var dataObject = new {
KeyWords = "one, two, three"
};
var serializer = new JavaScriptSerializer();
var json = serializer.Serialize(dataObject);
var response = client.UploadString("yourUrl", json);
}

Java: Marshalling using JaxB to XML, how to properly multithread

I am trying to take a very long file of strings and convert it to an XML according to a schema I was given. I used jaxB to create classes from that schema. Since the file is very large I created a thread pool to improve the performance but since then it only processes one line of the file and marshalls it to the XML file, per thread.
Below is my home class where I read from the file. Each line is a record of a transaction, for every new user encountered a list is made to store all of that users transactions and each list is put into a HashMap. I made it a ConcurrentHashMap because multiple threads will work on the map simultaneously, is this the correct thing to do?
After the lists are created a thread is made for each user. Each thread runs the method ProcessCommands below and receives from home the list of transactions for its user.
public class home{
public static File XMLFile = new File("LogFile.xml");
Map<String,List<String>> UserMap= new ConcurrentHashMap<String,List<String>>();
String[] UserNames = new String[5000];
int numberOfUsers = 0;
try{
BufferedReader reader = new BufferedReader(new FileReader("test.txt"));
String line;
while ((line = reader.readLine()) != null)
{
parsed = line.split(",|\\s+");
if(!parsed[2].equals("./testLOG")){
if(Utilities.checkUserExists(parsed[2], UserNames) == false){ //User does not already exist
System.out.println("New User: " + parsed[2]);
UserMap.put(parsed[2],new ArrayList<String>()); //Create list of transactions for new user
UserMap.get(parsed[2]).add(line); //Add First Item to new list
UserNames[numberOfUsers] = parsed[2]; //Add new user
numberOfUsers++;
}
else{ //User Already Existed
UserMap.get(parsed[2]).add(line);
}
}
}
reader.close();
} catch (IOException x) {
System.err.println(x);
}
//get start time
long startTime = new Date().getTime();
tCount = numberOfUsers;
ExecutorService threadPool = Executors.newFixedThreadPool(tCount);
for(int i = 0; i < numberOfUsers; i++){
System.out.println("Starting Thread " + i + " for user " + UserNames[i]);
Runnable worker = new ProcessCommands(UserMap.get(UserNames[i]),UserNames[i], XMLfile);
threadPool.execute(worker);
}
threadPool.shutdown();
while(!threadPool.isTerminated()){
}
System.out.println("Finished all threads");
}
Here is the ProcessCommands class. The thread receives the list for its user and creates a marshaller. From what I unserstand marshalling is not thread safe so it is best to create one for each thread, is this the best way to do that?
When I create the marshallers I know that each from (from each thread) will want to access the created file causing conflicts, I used synchronized, is that correct?
As the thread iterates through it's list, each line calls for a certain case. There are a lot so I just made pseudo-cases for clarity. Each case calls the function below.
public class ProcessCommands implements Runnable{
private static final boolean DEBUG = false;
private List<String> list = null;
private String threadName;
private File XMLfile = null;
public Thread myThread;
public ProcessCommands(List<String> list, String threadName, File XMLfile){
this.list = list;
this.threadName = threadName;
this.XMLfile = XMLfile;
}
public void run(){
Date start = null;
int transactionNumber = 0;
String[] parsed = new String[8];
String[] quoteParsed = null;
String[] universalFormatCommand = new String[9];
String userCommand = null;
Connection connection = null;
Statement stmt = null;
Map<String, UserObject> usersMap = null;
Map<String, Stack<BLO>> buyMap = null;
Map<String, Stack<SLO>> sellMap = null;
Map<String, QLO> stockCodeMap = null;
Map<String, BTO> buyTriggerMap = null;
Map<String, STO> sellTriggerMap = null;
Map<String, USO> usersStocksMap = null;
String SQL = null;
int amountToAdd = 0;
int tempDollars = 0;
UserObject tempUO = null;
BLO tempBLO = null;
SLO tempSLO = null;
Stack<BLO> tempStBLO = null;
Stack<SLO> tempStSLO = null;
BTO tempBTO = null;
STO tempSTO = null;
USO tempUSO = null;
QLO tempQLO = null;
String stockCode = null;
String quoteResponse = null;
int usersDollars = 0;
int dollarAmountToBuy = 0;
int dollarAmountToSell = 0;
int numberOfSharesToBuy = 0;
int numberOfSharesToSell = 0;
int quoteStockInDollars = 0;
int shares = 0;
Iterator<String> itr = null;
int transactionCount = list.size();
System.out.println("Starting "+threadName+" - listSize = "+transactionCount);
//UO dollars, reserved
usersMap = new HashMap<String, UserObject>(3); //userName -> UO
//USO shares
usersStocksMap = new HashMap<String, USO>(); //userName+stockCode -> shares
//BLO code, timestamp, dollarAmountToBuy, stockPriceInDollars
buyMap = new HashMap<String, Stack<BLO>>(); //userName -> Stack<BLO>
//SLO code, timestamp, dollarAmountToSell, stockPriceInDollars
sellMap = new HashMap<String, Stack<SLO>>(); //userName -> Stack<SLO>
//BTO code, timestamp, dollarAmountToBuy, stockPriceInDollars
buyTriggerMap = new ConcurrentHashMap<String, BTO>(); //userName+stockCode -> BTO
//STO code, timestamp, dollarAmountToBuy, stockPriceInDollars
sellTriggerMap = new HashMap<String, STO>(); //userName+stockCode -> STO
//QLO timestamp, stockPriceInDollars
stockCodeMap = new HashMap<String, QLO>(); //stockCode -> QLO
//create user object and initialize stacks
usersMap.put(threadName, new UserObject(0, 0));
buyMap.put(threadName, new Stack<BLO>());
sellMap.put(threadName, new Stack<SLO>());
try {
//Marshaller marshaller = getMarshaller();
synchronized (this){
Marshaller marshaller = init.jc.createMarshaller();
marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
marshaller.setProperty(Marshaller.JAXB_FRAGMENT, true);
marshaller.marshal(LogServer.Root,XMLfile);
marshaller.marshal(LogServer.Root,System.out);
}
} catch (JAXBException M) {
M.printStackTrace();
}
Date timing = new Date();
//universalFormatCommand = new String[8];
parsed = new String[8];
//iterate through workload file
itr = this.list.iterator();
while(itr.hasNext()){
userCommand = (String) itr.next();
itr.remove();
parsed = userCommand.split(",|\\s+");
transactionNumber = Integer.parseInt(parsed[0].replaceAll("\\[", "").replaceAll("\\]", ""));
universalFormatCommand = Utilities.FormatCommand(parsed, parsed[0]);
if(transactionNumber % 100 == 0){
System.out.println(this.threadName + " - " +transactionNumber+ " - "+(new Date().getTime() - timing.getTime())/1000);
}
/*System.out.print("UserCommand " +transactionNumber + ": ");
for(int i = 0;i<8;i++)System.out.print(universalFormatCommand[i]+ " ");
System.out.print("\n");*/
//switch for user command
switch (parsed[1].toLowerCase()) {
case "One"
*Do Stuff"
LogServer.create_Log(universalFormatCommand, transactionNumber, CommandType.ADD);
break;
case "Two"
*Do Stuff"
LogServer.create_Log(universalFormatCommand, transactionNumber, CommandType.ADD);
break;
}
}
}
The function create_Log has multiple cases so as before, for clarity I just left one. The case "QUOTE" only calls one object creation function but other other cases can create multiple objects. The type 'log' is a complex XML type that defines all the other object types so in each call to create_Log I create a log type called Root. The class 'log' generated by JaxB included a function to create a list of objects. The statement:
Root.getUserCommandOrQuoteServerOrAccountTransaction().add(quote_QuoteType);
takes the root element I created, creates a list and adds the newly created object 'quote_QuoteType' to that list. Before I added threading this method successfully created a list of as many objects as I wanted then marshalled them. So I'm pretty positive the bit in class 'LogServer' is not the issue. It is something to do with the marshalling and syncronization in the ProcessCommands class above.
public class LogServer{
public static log Root = new log();
public static QuoteServerType Log_Quote(String[] input, int TransactionNumber){
ObjectFactory factory = new ObjectFactory();
QuoteServerType quoteCall = factory.createQuoteServerType();
**Populate the QuoteServerType object called quoteCall**
return quoteCall;
}
public static void create_Log(String[] input, int TransactionNumber, CommandType Command){
System.out.print("TRANSACTION "+TransactionNumber + " is " + Command + ": ");
for(int i = 0; i<input.length;i++) System.out.print(input[i] + " ");
System.out.print("\n");
switch(input[1]){
case "QUOTE":
System.out.print("QUOTE CASE");
QuoteServerType quote_QuoteType = Log_Quote(input,TransactionNumber);
Root.getUserCommandOrQuoteServerOrAccountTransaction().add(quote_QuoteType);
break;
}
}
So you wrote a lot of code, but have you try if it is actually working? After quick look I doubt it. You should test your code logic part by part not going all the way till the end. It seems you are just staring with Java. I would recommend practice first on simple one threaded applications. Sorry if I sound harsh, but I will try to be constructive as well:
Per convention, the classes names are starts with capital letter, variables by small, you do it other way.
You should make a method in you home (Home) class not a put all your code in the static block.
You are reading the whole file to the memory, you do not process it line by line. After the Home is initialized literary whole content of file will be under UserMap variable. If the file is really large you will run out of the heap memory. If you assume large file than you cannot do it and you have to redisign your app to store somewhere partial results. If your file is smaller than memmory you could keep it like that (but you said it is large).
No need for UserNames, the UserMap.containsKey will do the job
Your thread pools size should be in the range of your cores not number of users as you will get thread trashing (if you have blocking operation in your code make tCount = 2*processors if not keep it as number of processors). Once one ProcessCommand finish, the executor will start another one till you finish all and you will be efficiently using all your processor cores.
DO NOT while(!threadPool.isTerminated()), this line will completely consume one processor as it will be constantly checking, call awaitTermination instead
Your ProcessCommand, has view map variables which will only had one entry cause as you said, each will process data from one user.
The synchronized(this) is Process will not work, as each thread will synchronized on different object (different isntance of process).
I believe creating marshaller is thread safe (check it) so no need to synchronization at all
You save your log (whatever it is) before you did actual processing in of the transactions lists
The marshalling will override content of the file with current state of LogServer.Root. If it is shared bettween your proccsCommand (seems so) what is the point in saving it in each thread. Do it once you are finished.
You dont need itr.remove();
The log class (for the ROOT variable !!!) needs to be thread-safe as all the threads will call the operations on it (so the list inside the log class must be concurrent list etc).
And so on.....
I would recommend, to
Start with simple one thread version that actually works.
Deal with processing line by line, (store reasults for each users in differnt file, you can have cache with transactions for recently used users so not to keep writing all the time to the disk (see guava cache)
Process multithreaded each user transaction to your user log objects (again if it is a lot you have to save them to the disk not keep all in memmory).
Write code that combines logs from diiffernt users to create one (again you may want to do it mutithreaded), though it will be mostly IO operations so not much gain and more tricky to do.
Good luck
override cont

Error with AWS Java SDK doing an upload with metadata

I am trying to upload a file to a S3 container and before doing the upload, I am setting the metadata of the file. The upload fails with an error saying signature doesn't match. Below is the code I am using :
public URL send(File f, HashMap<String,String> metadata, String type) throws Exception {
String path = type+"/"+f.getName();
InitiateMultipartUploadRequest req = new InitiateMultipartUploadRequest(container, secretKey).withKey(path);
req.setCannedACL(CannedAccessControlList.AuthenticatedRead);
if (metadata != null) {
ObjectMetadata objectMetadata = new ObjectMetadata();
Set<String> keys = metadata.keySet();
Iterator<String> i = keys.iterator();
while (i.hasNext()) {
String key = i.next();
objectMetadata.addUserMetadata(key, metadata.get(key));
}
req.setObjectMetadata(objectMetadata);
}
InitiateMultipartUploadResult res = s3client.initiateMultipartUpload(req);
String uploadId = res.getUploadId();
long fileSize = f.length();
//check the size doesn't exceed max limit
if (fileSize > MAX_OBJ_SIZE) {
throw new Exception("Object size exceeds repository limit");
}
long chunkSize = 1024 * 1024 * 16;
int chunks = (int) (fileSize/chunkSize + 2);
List<PartETag> chunkList = new ArrayList<PartETag>();
long pos = 0;
try {
for (int i = 1; i < chunks; i++) {
if ((chunks -i) < 2) {
chunkSize = fileSize - pos;
}
UploadPartRequest upReq = new UploadPartRequest()
.withBucketName(container).withKey(path)
.withUploadId(uploadId).withPartNumber(i)
.withFileOffset(pos).withFile(f)
.withPartSize(chunkSize);
PartETag pTag = null;
// repeat the upload until it succeeds.
boolean repeat;
do {
repeat = false; // reset switch
try {
// Upload part and add response to our list.
pTag = s3client.uploadPart(upReq).getPartETag();
}
catch (Exception ex) {
repeat = true; // repeat
}
} while (repeat);
chunkList.add(pTag);
pos = pos + chunkSize;
}
CompleteMultipartUploadRequest compl = new CompleteMultipartUploadRequest(
container, secretKey, uploadId, chunkList).withKey(path);
CompleteMultipartUploadResult complRes = s3client.completeMultipartUpload(compl);
return new URL(URLDecoder.decode(complRes.getLocation(), "UTF-8"));
}
catch (Exception ex) {
s3client.abortMultipartUpload(new AbortMultipartUploadRequest(container,
secretKey, uploadId));
throw new Exception("File upload error: "+ex.toString());
}
}
Below is the error I am getting :
com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 0805716BBD0662AB, AWS Error Code: SignatureDoesNotMatch, AWS Error Message: The request signature we calculated does not match the signature you provided. Check your key and signing method., S3 Extended Request ID: wNAzUyrLZgWCazZFe3KpMHO0uh0FM5FF7fiwBzN1A2YDEYS5hKZBYh5nWSjIhnhG
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:767)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:414)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:228)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3316)
at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:2401)
at net.timbusproject.storage.awss3.S3Client.send(S3Client.java:134)
Line 134 in S3Client.java where the error is occurring is :
InitiateMultipartUploadResult res = s3client.initiateMultipartUpload(req);
The upload works fine if I am not attaching any metadata. i.e, if I comment the below line, the upload works :
req.setObjectMetadata(objectMetadata);
I am unable to figure out why the request fails when metadata is set. Am I missing any step in the upload process ?
I was able to work around this problem by URL encoding the metadata keys and values.
objectMetadata.addUserMetadata(URLEncoder.encode(key, "UTF-8"), URLEncoder.encode(metadata.get(key),"UTF-8"));
Obviously the metadata seems to have some offending characters which are messing with the AWS calls. This workaround will let upload complete without error and also updates the metadata but the strings remain url encoded, which can be a problem later.

Categories

Resources