My goal is to rapidly make posts to a server from appengine(java). I am attempting to do this using UrlFetchService.fetchAsync. I have been basing my code after this blog post. I have been able to make the request using the code below, however I get some strange behavior:
private void futureRequests() {
URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();
URL url = new URL("https://someserver.com");
FetchOptions fetchOptions = FetchOptions.Builder.withDefaults();
fetchOptions.doNotValidateCertificate();
fetchOptions.setDeadline(60D);
ArrayList<Future<HTTPResponse>> asyncResponses = new ArrayList<Future<HTTPResponse>>();
for (int i = 0; i < postDatas.size(); i++) {
HTTPRequest request = new HTTPRequest(url, HTTPMethod.POST, fetchOptions);
request.setPayload(postDatas.get(i).getBytes(UTF8));
HTTPHeader header = new HTTPHeader("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8");
request.setHeader(header);
header = new HTTPHeader("Content-Length", Integer.toString(postDatas.get(i).getBytes().length));
request.setHeader(header);
header = new HTTPHeader("Authorization", "auth=" + authToken);
request.setHeader(header);
Future<HTTPResponse> responseFuture = fetcher.fetchAsync(request);
asyncResponses.add(responseFuture);
}
for (Future<HTTPResponse> future : asyncResponses) {
HTTPResponse response;
try {
response = future.get();
int responseCode = response.getResponseCode();
resp.getWriter().println("response: " + responseCode);
logger.warning("Response: " + responseCode);
} catch (Exception e) {
}
}
}
The strange behavior is that I get duplicate posts on the server, and according to my appstats page I use 10x-20x more urlFetches than what was added with the code above. Below is my appstats screen:
There are more urlFetch calls that could not fit on the screen. It appears that the requests are still completing in a synchronous fashion(circled items), but there are many urlFetches that appear to go on at the same time. My question is how am I getting all this calls to urlFetch when I only had 14 Future ?? Could the server be giving an error or 503 and urlFetch retrying until it goes through? And how can I be getting 2 posts for each request??
I understand that I could use the task queue to do asyc request, however I am dealing with a relatively low number of request(20-100) and the cold start time of ramping up another instance would probably make this not a good option for my situation. Can anyone explain this behavior or have experience with this?
This was simply a mistake in my code that was causing my app to make more request than I thought..
Related
I am new to java (I have experience with C# though)
Sadly, I inherited a terrible project (the code is terrible) and what I need to accomplish is to import some csv files into Cloud SQL
So there's a WS which runs this task, apparently the dev followed this guide to import data. But it is not working. Here's the code (Essential parts, actually it is longer and more ugly)
InstancesImportRequest requestBody = new InstancesImportRequest();
ImportContext ic = new ImportContext();
ic.setKind("sql#importContext");
ic.setFileType("csv");
ic.setUri(bucketPath);
ic.setDatabase(CLOUD_SQL_DATABASE);
CsvImportOptions csv = new CsvImportOptions();
csv.setTable(tablename);
List<String> list = new ArrayList<String>();
// here there is some code that populates the list with the columns
csv.setColumns(list);
ic.setCsvImportOptions(csv);
requestBody.setImportContext(ic);
SQLAdmin sqlAdminService = createSqlAdminService();
SQLAdmin.Instances.SQLAdminImport request = sqlAdminService.instances().sqladminImport(project, instance, requestBody);
Operation response = request.execute();
System.out.println("Executed : Going to sleep.>"+response.getStatus());
int c = 1;
while(!response.getStatus().equalsIgnoreCase("Done")){
Thread.sleep(10000);
System.out.println("sleeped enough >"+response.getStatus());
c++;
if(c==50){
System.out.println("timeout?");
break;
}
}
public static SQLAdmin createSqlAdminService() throws IOException, GeneralSecurityException {
HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
JsonFactory jsonFactory = JacksonFactory.getDefaultInstance();
GoogleCredential credential = GoogleCredential.getApplicationDefault();
if (credential.createScopedRequired()) {
credential =
credential.createScoped(Arrays.asList("https://www.googleapis.com/auth/cloud-platform"));
}
return new SQLAdmin.Builder(httpTransport, jsonFactory, credential)
.setApplicationName("Google-SQLAdminSample/0.1")
.build();
}
I am not quite sure how response should be treated, it seems it is an async request. Either way, I always get status Pending; it seems it is not even start to executing.
Of course it ends timing out. What is wrong here, why the requests never starts ? I couldn't find any actual example on the internet about using this java sdk to import files, except the link I gave above
Well, the thing is that the response object is static, so it will always return "Pending" as the initial status since it is a string in the object - it is not actually being updated.
To get the actual status, you have to requested it to google using the sdk. I did something like this (it will be better to use a smaller sleep time, and make it grow as you try more times)
SQLAdmin.Instances.SQLAdminImport request = sqlAdminService.instances().sqladminImport(CLOUD_PROJECT, CLOUD_SQL_INSTANCE, requestBody);
// execution of our import request
Operation response = request.execute();
int tried = 0;
Operation statusOperation;
do {
// sleep one minute
Thread.sleep(60000);
// here we are requesting the status of our operation. Name is actually the unique identifier
Get requestStatus = sqlAdminService.operations().get(CLOUD_PROJECT, response.getName());
statusOperation = requestStatus.execute();
tried++;
System.out.println("status is: " + statusOperation.getStatus());
} while(!statusOperation.getStatus().equalsIgnoreCase("DONE") && tried < 10);
if (!statusOperation.getStatus().equalsIgnoreCase("DONE")) {
throw new Exception("import failed: Timeout");
}
I'm developing a game using Unity engine which have to send cookie from Client side C# to server side - Java , and I facing this problem (maybe cross platform problem? I'm not sure)
I write a bunch of code in client side like this
private HttpWebRequest request(){
try{
string url = "http://localhost:8080/...";
var request = (HttpWebRequest)WebRequest.Create(url);
request.Timeout = 15000;
request.KeepAlive = true ;
request.Method= "GET";
CookieContainer cookieContainer = new CookieContainer();
Cookie Authentication = new Cookie("Session" , "09iubasd");
Authentication.Domain = url;
cookieContainer.Add(Authentication);
request.CookieContainer = cookieContainer;
request.Headers.Add("testting", "hascome");
return request;
}catch(System.Exception ex){
Debug.Log("[Exception]" + ex);
throw ex;
}
}
and The server side is writing in Java Spring. I can't retrieve the Cookie data inside the CookieContainer at server-side. Can anyone give me any suggestion or any solution to solve this problem? Or something similar to the CookieContainer in Java. I have googled but seem no way, If this is a silly question then please teach me. Many thanks.
Vince
I just find out the reason why, my cookie domain set wrong way.
Here the new test Code I just fix. Hope this help who have the same problem in the future ( Of cause it must be great if no one face this silly problem )
private HttpWebRequest request(){
try{
System.Uri uri = new System.Uri("http://localhost:8080/...");
var request = (HttpWebRequest)WebRequest.Create(uri);
request.Timeout = 15000;
request.KeepAlive = true ;
request.Method= "GET";
Cookie Authentication = new Cookie("Session" , "09iubasd");
Authentication.Domain = uri.Host;
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(Authentication);
request.Headers.Add("testting", "hascome");
return request;
}catch(System.Exception ex){
Debug.Log("[Exception]" + ex);
throw ex;
}
}
Im executing a http request like this:
DefaultHttpClient httpClient = new DefaultHttpClient();
HttpGet httpRequest = new HttpGet("http://localhost:8080/test");
HttpResponse httpResponse = httpClient.execute(httpRequest);
What is the best way to log out everything about the request/response? I mean like what method was used for request and what url and all the headers and response code and response message body and so on.
Edit:
At the moment i worked out something like this:
private static void logHttpRequest(HttpMessage httpMessage) {
StringBuilder builder = new StringBuilder();
addHeaders(httpMessage, builder);
builder.append(httpMessage.toString() + "\n");
System.out.println("Request:\n" + builder.toString());
}
private static void logHttpResponse(HttpResponse httpResponse) throws IOException {
StringBuilder builder = new StringBuilder();
builder.append(httpResponse.getStatusLine() + "\n");
addHeaders(httpResponse, builder);
builder.append(EntityUtils.toString(httpResponse.getEntity()));
System.out.println("Response:\n" + builder.toString() + "\n");
}
private static void addHeaders(HttpMessage httpMessage, StringBuilder builder) {
Header[] headers = httpMessage.getAllHeaders();
if (headers != null) {
for (Header header : headers) {
builder.append(header.toString() + "\n");
}
}
}
Is there anything else that should be logged? Can anything be done better/more optimal?
I think this primarily depends on context and your requirements. Keep in mind though, that huge amounts of logging is also not smart. For example, you should log exceptions at a high log level, eg. SEVERE or ERROR (depends on the logging framework you're using). However, messages like, "trying to make a GET request to http://soAndSo/whatevz" could do with lower log level of, say DEBUG, so you could switch your code to log DEBUG level messages when something bad happens in production.
Something I have typically seen people do with the response is that they definitely log ERROR if there's no response at all but when there is a response, it's imperative to check the http status code of the response and if it's not what you expect it to be (typically 200/2xx), you log the response body, headers, etc at ERROR level.
When making POST or PUT requests with some body, try and see if it's useful to log the request body and headers if any at, say, INFO or DEBUG level. These things sometimes help in tracing out failure scenarios or lost requests/lost data.
At the consuming end too, your system should ideally have a monitor on the no.of/rate of ERROR level messages otherwise you'll come to know of something very bad only after it happens, which is not favourable.
I want to share some information in google plus wall from my application.and I am trying for moment.insert, But getting 400 error . Can somebody help me
#Override
public JSONObject getGooglePlusAddUseractivities(Object token) {
Token accessToken = (Token) token;
OAuthService service = createOAuthServiceForGooglePlus();
OAuthRequest request = new OAuthRequest(Method.POST,"https://www.googleapis.com/plus/v1/people/me/moments/vault");
request.addQuerystringParameter("alt", "json");
service.signRequest(accessToken, request);
JSONObject object=new JSONObject();
try {
object.put("kind","plus#moment");
object.put("type","http://schemas.google.com/AddActivity");
JSONObject obj1=new JSONObject();
obj1.put("kind", "plus#itemScope");
obj1.put("url","https://plus.google.com/me");
obj1.put("description","Sign up now to claim and redeem your credits while shopping! ");
obj1.put("image","http://invite.png");
obj1.put("contentUrl", "www.abcd.com");
obj1.put("thumbnailUrl", "http://logo1_favicon.png");
object.putOpt("target", obj1);;
}catch(Exception ex) {
ex.printStackTrace();
}
request.addPayload(object.toString());
request.addHeader("Content-Type", "application/json");
System.out.println("request : "+request.getBodyContents());
Response response = request.send();
String responseBody = response.getBody();
JSONObject googleJSON = null;
try {
googleJSON = new JSONObject(responseBody);
}
catch (JSONException e) {
System.out.println("can not create JSON Object");
}
getting 400 error ?? anyone can tell me..... where am wrong ..!!`
It isn't clear from the documentation, but you can't provide both the target.url and most other target metadata. This is currently opened as bug 485 in the issue tracking system - please go there and star the issue to make sure they properly prioritize a fix.
If you remove the target.url value and add a target.id value, it should work.
(As an aside, this does not post in the user's stream, but will post an App Activity in their app moment vault. They must manually share the activity if they choose.)
At this time, it is not possible to programmatically write to a user's Stream. As a developer, you have two options:
Write an AppActivity (formerly known as a Moment), which writes information to Google, but not to a Google+ Stream. These activities are visible at plus.google.com/apps, and will be used by Google in additional ways over time.
Create an Interactive Post Share button, which a user must initiate. However, you can pre-fill both the text of the post and up to 10 intended recipients. The user can make changes if they want and then perform the actual share. You can learn more at https://developers.google.com/+/web/share/interactive or by watching this Google+ Developers Live episode: http://www.youtube.com/watch?v=U4Iw28jWtAY.
Probably this is often problem, but I can't solve it.
I need to automaticallly fill out some fields on the web form, which is given from server.
I use Apache HttpClient to make more easy my life)
By now, one can consider my steps to acheive the aim:
1. I have not certificate to http://trac.edgewall.org/ so I download this software and install locally and at finish I'll have to create NewTicket.
2. I locally use Trac without any SSL(SSL tunnel). (It's not difficult to change my program to be able to use HTTPS).
3. By now, I can authenticate and perform GET request, but I can't perform POST request
4. For instance: I perform GET request to the http://localhost:8000/tracenvir/newticket .
This (~/newticket) page looks as following:
http://s04.radikal.ru/i177/0912/cb/d43971cebc02.png
And as response I have : (part of it)
"input type="text" id="field-summary" name="field_summary" size="70" "
"textarea id="field-description" name="field_description" class="wikitext" rows="10" cols="68"/textarea"'
5. So, I write this:
int status = 0;
int cookLength=0;
Cookie[] cookies = null;
HttpClient client = new HttpClient();
client.getParams().setCookiePolicy(CookiePolicy.BROWSER_COMPATIBILITY);
HttpState initialState = new HttpState();
client.setState(initialState);
//**********//
//**Log in**//
//**********//
GetMethod login = new GetMethod("http://localhost:8000/tracenvir/login");
client.getState().setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials("rauch", "qwert"));
login.setDoAuthentication(true);
System.setProperty("javax.net.ssl.trustStore", "/home/rauch/NetBeansProjects/jssecacerts");
try {
status = client.executeMethod(login);
System.out.println("response code = "+status);
cookies = client.getState().getCookies();
cookLength = cookies.length;
for(int i=0;i (less than) cookLength;i++) {
System.out.println(cookies[i].toString());
}
login.releaseConnection();
} catch(IOException ex) {
ex.printStackTrace();
}
//*********************//
//**Create New Ticket**//
//*********************//
PostMethod post = new PostMethod("http://localhost:8000/tracenvir/newticket");
NameValuePair[] data = {
new NameValuePair("field-summary","second error"),
new NameValuePair("field-descryption","Some stupid descryption..."),
new NameValuePair("field-type","defect"),
new NameValuePair("field-priority","major"),
new NameValuePair("field-version","1.0"),
new NameValuePair("field-owner","moscow server"),
new NameValuePair("submit","create ticket"),
};
//post.setRequestBody(data);
post.addParameters(data);
post.addRequestHeader("Referer","http://localhost:8000/tracenvir/login");
for(int i=0;i (less than) cookLength;i++) {
initialState.addCookie(cookies[i]);
}
client.setState(initialState);
try {
status = client.executeMethod(post);
System.out.println("response code = "+status);
byte[] buf = new byte[10];
int r=0;
BufferedInputStream is = new BufferedInputStream(post.getResponseBodyAsStream());
while((r = is.read(buf)) > 0) {
System.out.write(buf, 0, r);
}
post.releaseConnection();
} catch(IOException ex) {
ex.printStackTrace();
}
}
And I have this:
400 Error: Bad Request
Missing or invalid form token. Do you have cookies enabled?
What's wrong?
As response on GET request I get this:
response code = 200
trac_auth=38144ec2830678183afebf0b14c51721
trac_form_token=e9648f17987551b8f97e1953
Probably I nedd change this:
client.getParams().setCookiePolicy(CookiePolicy.BROWSER_COMPATIBILITY);
http://www.google.com/support/toolbar/bin/answer.py?hl=en&answer=47972
I think above link will help u on this topic. its provide video also.. i hope your query will be solved
http://www.formautofill.com/
This site will give you a software which will provide auto form filler. provided by microsoft.
Ok. This below link is about post method . i think it might help u.
http://www.willmaster.com/library/manage-forms/automatic-form-submission-to-a-cgi-program.php
give me reply. if it doesn't.