with the HttpServletRequest object, we can have the getRequestURL, which shows the ressource requested, but in my case I would like to know from where the request comes:
I tried also getRemoteAddr() and getLocalAddr() that prints my local IP, (as I am running glassfish and small webserver that talks to glassfish locally.
but the IP doesn't show the full referer, that should be in my case
http://my.domain.com/wiki/aPage
from my IP I can resolve to http://my.domain.com, yet not the full url
Does this mean I need to send also "wiki/aPage" in the request, or I hope there is a better possibility?
thanks
You can read the Referer header of the request and get the value by using request.getHeader("Referer");
Related
I write a java program like I saw here
How to read the https page content using java?
but for some sites the code does not work.
I got Error Server returned HTTP response code: 403 for URL: https://research.investors.com/stock-quotes/nyse-sailpoint-tech-holdings-sail.htm
It works for
url = "https://maven.apache.org/guides/mini/guide-repository-ssl.html";
Can someone help me ?
403 HTTP status stands for "Forbidden", most likely investors.com can check your request headers and deny the resource.
Try modifying the request headers using an User-Agent that site might accept.
403 Forbidden
The request contained valid data and was understood by the server, but the server is refusing action. This may be due to the user not having the necessary permissions for a resource or needing an account of some sort, or attempting a prohibited action (e.g. creating a duplicate record where only one is allowed). This code is also typically used if the request provided authentication by answering the WWW-Authenticate header field challenge, but the server did not accept that authentication. The request should not be repeated.
So probably website, which you want to scrape, just restricted requests like yours (i mean requests, that was made not from browser).
But you can try Selenium.
OK , I solved.
I use con.setRequestProperty and set "User-Agent", "Accept", "Content-Type", "Accept-Language".
Thank you.
I have an endpoint to be tested using RestAssured. The same endpoint is working fine while opening it in browser/Postman. But, while trying to test the same using RestAssured,
I am getting Operation Timed Out Error.
I had to connect to proxy to make that end point working in browser. used the same proxy in the rest assured also.
Sample Code below:
given().proxy("My_Proxy_URL_HERE",8080).when().get("My_API_URL_Here").then().log().all();
I am getting the response as
"Operation Timed Out" with Status Code 503.
I need your suggestion, what could be the possible issue, how to debug etc. Any suggestion is appreciated. Thanks in advance.
There can be many reasons for this behavior:
The address is just wrong and given there is some load balancer/proxy it can be configured to wait for a certain period of time and then respond with 503 status code.
Note, 503 is not a "request timed out", but "Service Unavailable".
The request url is good, but the request lacks some headers so that the load balancer/proxy won't be able to route the request to the required server.
How to check this? there exist tools that can come handy in this situation:
Check the access logs of the load balancer/proxy and even of your server if its possible - and see the request.
If it doesn't help, try to compare requests coming from rest-assured vs regular request. You can use tools like Burp for example, there are others, or you can even roll your own.
The idea is simple:
Start the "interceptor" on some port of your local computer (say, 9999 for example)
Configure the interceptor to forward all the requests to proxy of your choice (identified by URL - My_Proxy_URL_HERE and port 8080).
Now rest-assured must call localhost:9999 and the request will be intercepted by this tool. You'll be able to inspect its contents - headers, body, http method - everything.
Do the same for browser request and compare.
I have a Tomcat server running Java servlets. I'm trying to make a servlet that returns stored files, given their encrypted IDs.
ID: 100
Encrypted ID: +e4/E5cR/aM=
URL-encoded ID: %2Be4%2FE5cR%2FaM%3D
Resulting URL: http://localhost/file/demo/%2Be4%2FE5cR%2FaM%3D
When I try to follow that link, I don't even get into my servlet's code - the server returns this error: Failed to load resource: the server responded with a status of 400 (Bad Request)
What's wrong with this URL that's making Tomcat reject it before reaching my code? I ran it though a URL-encoder, and I don't see any invalid characters in it.
You have slash "/" encoded in the url. Apache doesn't allow them due to potential atacks. There is setting to allow them:
System.setProperty("org.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH", "true");
or
-Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
See similar post.
You are likely experiencing one of 2 issues:
1) You have not included the port in your URL. Either you have configured the Tomcat port to port 80, in which case the port is not needed, or you need to include the port, which defaults to 8080, for example:
http://localhost:8080/file/demo/%2Be4%2FE5cR%2FaM%3D
2) You are adding the encrypted ID as part of the URL itself, which would have to be mapped to a Servlet/JSP/View of some sort in your URL mappings and is not likely. Tomcat is not going to recognize a unique ID and know a corresponding handler to call to process the mapping. Assuming you intend to call the servlet/JSP/controller that is mapped to '/file/demo', you would more likely want to pass the ID as a request parameter, for example:
http://localhost:8080/file/demo?id=%2Be4%2FE5cR%2FaM%3D
I know that there are a lot of questions regarding this subject, but I still don't get it.
I want to get the current URL from my browser in my program. What do I need to succeed?
The connection has to be made with HTTP Connection? A proxy server would help me more? because i have to filter those URLs. Help me please, I am so confused.
request.getRequestURL();
will get you the URL from HttpServletRequest. More documentation can be found in https://docs.oracle.com/javaee/6/api/javax/servlet/http/HttpServletRequest.html
I was expecting this code to return a 404, however it produces the output :
"Response code is 200"
Would it be possible to learn how to differentiate between existent and non-existent web pages . . . thanks so much,
try
{
// create the HttpURLConnection
URL url = new URL("http://www.thisurldoesnotexist");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
System.out.println("Response code is " + connection.getResponseCode());
}
EDIT: I see you've call openConnection() but not connect() - could that be the problem? I would expect getResponseCode() to actually make the request if it hasn't already, but it's worth just trying that...
That suggests you've possible got some DNS resolver which redirects to a "helper" (spam) page, or something like that.
The easiest way to see exactly what's going on here is to use Wireshark - have that up and capturing traffic (HTTP-only, to make life easier) and then run your code. You should be able to see what's going on that way.
Note that I wouldn't have expected a 404 - because that would involve being able to find a web server to talk to to start with. If you're trying to go to a host which doesn't involve, there shouldn't be an HTTP response at all. I'd expect connect() to throw an exception.
try adding a "connection.connect();" or look at the contents returned...
it could be a dns issue, ie: your dns is being sent to a parking place... for example: freedns does this.
You could:
Resolve the IP from the host of the page
Try to connect to port 80 on the resolved IP using plain sockets
This is a bit low level however and will add complexity since you will need to make a simple GET request through the socket. Then validate the response so you're sure that its actually a HTTP server running on port 80.
NMap might be able to help you here.
Ideally you should be getting this error:
java.net.UnknownHostException: www.thisurldoesnotexist
But it looks like your URL is resolved by you DNS provider.
For instance on my company's network running your code with URI "http://profile/" displays
the employee profile.
Please also check etc.home file if you are on windows to check if any settings have been changed.
Like #spgennard - I think this is most likely a DNS issue.
The URL you have chosen is owned by a DNS speculator.
The URL you have chosen is "parked" by a DNS provider.
Your ISP is messing with your DNS results to send your browser to some search page.
It is also possible that you are accessing the web via a proxy, and the proxy is doing something strange.
The way to diagnose this is to look at the other information in the HTTP responses you are getting, particularly the response body.