Im trying to improve the Java Html Document a little but i'm running into problems with the HttpUrlConntion. One thing is that some servers block a request if the user agent is a Java VM. Another problem is that the HttpUrlConnection does not set the Referrer or Location header field. Since several sites use these fields to verify that the content was accessed from their own site, I'm blocked here as well. As far as I can see the only resolution is to replace the URL handler of the HTTP protocol. Or is there any way to modify the default HTTP Handler?
Open the URL with URL.openConnection. Optionally cast to HttpURLConnection. Call URLConnection.setRequestProperty/addRequestProperty.
The default User-Agent header value is set from the "http.agent" system property. The PlugIn and WebStart allow you to set this property.
If you use Apache HttpClient to manage your programmatic HTTP connectivity you get an extremely useful API which makes creating connections (and optional automatic re-connecting on fail), setting Headers, posts vs gets, handy methods for retrieving the returned content and much much more.
I solved my problem. We can just send the header to application/json and pass the body as a json object. That simply solves the issue.
Related
Apparently, sun.net.www.protocol.http.HttpURLConnection will always append "Java/version" to the UserAgent. Therefore, JSoup Connection.userAgent cannot set the useragent to what you want; the "Java/version" stuff gets appended anyway.
See Set user-agent property in https connection header
Some websites reject requests that contain "Java" anywhere in
the user agent, giving various 4xx and 5xx HTTP errors.
The StackOverflow post referenced above suggests using Apache instead of Sun's HTTP connection class, but this is not an option if I want to use JSoup.
I wonder what the JSoup team thinks of this. Is my description correct? Is it a bug or a feature? Are there any plans to fix it, i.e. to make it possible to set the userAgent to what one wants, without additional appendages?
thanks
JWG
You could use Jsoup.parse(html) where the html String could be fetched using Apache HTTP or any other library of your choice.
Regards,
Allahbaksh
I need to perform an http GET for a REST service and include a body in the GET. Unfortunately, setting #setDoOutput( true ) on the connection forces a POST. Is there anyway to send a GET with a body?
Edit: The body I'm trying to send is JSON.
It is not possible to send content for an HTTP GET using HttpURLConnection. By setting setDoOutput(true) on an HttpURLConnection the verb is forced to be POST.
The documentation for the REST API I was using described a JSON body for the endpoint in question, but URL parameters were accepted.
It might not be possible through HttpUrlConnection, although you might be able to do it through another APIs BUT, if you have to do it that way chances that you are doing something wrong in your architecture are high because it goes against the basic usage of GET Http Method and different problems might arise like:
If you ever try to take advantage of caching, Proxies are not going to look in the GET body to see if the parameters have an impact on the response.
It's not a good implementation based on standard practices and it could cause problems with some browsers / services.
Take a look at this question for more information.
HTTP GET with request body
Hope this helps.
Regards!
I am trying to get JSON (getJSON()) from server that doesn't have support for jsonp implemented. Namely, when adding callback=? to the URL, the server does return the data, but it returns pure JSON without padding.
I understand this is something that must be corrected server-side - there is no way to resolve it in jQuery. Is this correct?
If CORS support is not supported by server as well jsonp, you might try proxy approach in such cases. One example http://www.corsproxy.com/, there should be other proxy alternatives too.
What does it do?
CORS Proxy allows javascript code on your site to access resources on other domains that would normally be blocked due to the same-origin policy.
How does it work?
CORS Proxy takes advantage of Cross-Origin Resource Sharing, which is a feature that was added along with HTML 5. Servers can specify that they want browsers to allow other websites to request resources they host. CORS Proxy is simply an HTTP Proxy that adds a header to responses saying "anyone can request this".
I am writing a Java based Web application, which, in the actual production environment would be front-ended by another application which would set certain HTTP request headers before the request hits my application.
However, in the development environment I do not have the front-ending application, for which I need to create a mock web application that simulates the same behavior. i.e. this mock application should set the request headers and redirect or forward or whatever that I do not know :) to a certain page in my application.
How can I accomplish this?
The following articles may help you:
Adding Header Information to an existing HTTP Request
How to modify request headers in a J2EE web
application.
P.S.
I am sorry I provided only links, that was one of my early answer on SO ))
In case you don't want to modify your code as suggested by #user1979427 you can use a proxy server to modify headers or add headers on the fly.
For example in Apache HTTPD you would add something like below and proxy the
Header add HEADER "HEADERVALUE"
RequestHeader set HEADER "HEADERVALUE"
Refer to HTTPD doc
You should create a AddReqHeaderForFrowardWrapper request wrapper passing the headername and header values. And, override the request header related methods to return your custom header.
You can use Tracer to implement this.
There are frameworks available to support this implementation.
Spring has Sleuth, Zipkin, OpenTracing available.
I find OpenTracing to be easy to use without worrying about dependency conflicts.
Read more about it here: https://opentracing.io/guides/java/
Instead of writing a mock application, I used a browser add-on that allowed me to add custom headers!
For setting header in java, you can use:
request.setHeader(attributeName, attributeValue);
And for redirecting to another page, you can use:
request.sendRedirect(URL);
I have a jquery plugin and I'm using jsonp for crossdomain call to a jsp file.
I want to strict the jsp return values only to specific websites in our database.
To achieve this I need to somehow get the ip or url of the website the jsonp call triggered and not the client/user ip. I've tried the referer value in the http header but this will not work with IE and I guess this is not the best solution either.
How can I securely now who is calling my jsp file with my plugin, from his website?
Thanks in advance.
The simplest answer would be to issue each website a unique key or other identifier that they include in their request. You parse this identifier and flex your response appropriately.
However with a request originating from the client browser, you would have to be careful and would have to evaluate what you mean by how "securely" you need the request to be handled. (since the untrusted client would be making the request it would be a simple task to harvest and reuse such an identifier)...
Referrer (if present) could be used as a double check, but as you pointed out, this is unreliable and coming from an untrusted client computer, this portion of the request could be faked as well.
If we could assume some server side processing by the website owners, you could have them implement a proxy for the jsonp call (which would ensure such a token would never fall into the hands of the browser)... but we'd have to know if such a safeguard would really be worth it or not :)