Develop Reference

Why I can not read some https pages with java code?

I write a java program like I saw here
How to read the https page content using java?
but for some sites the code does not work.
I got Error Server returned HTTP response code: 403 for URL: https://research.investors.com/stock-quotes/nyse-sailpoint-tech-holdings-sail.htm
It works for
url = "https://maven.apache.org/guides/mini/guide-repository-ssl.html";
Can someone help me ?

403 HTTP status stands for "Forbidden", most likely investors.com can check your request headers and deny the resource.
Try modifying the request headers using an User-Agent that site might accept.

403 Forbidden
The request contained valid data and was understood by the server, but the server is refusing action. This may be due to the user not having the necessary permissions for a resource or needing an account of some sort, or attempting a prohibited action (e.g. creating a duplicate record where only one is allowed). This code is also typically used if the request provided authentication by answering the WWW-Authenticate header field challenge, but the server did not accept that authentication. The request should not be repeated.
So probably website, which you want to scrape, just restricted requests like yours (i mean requests, that was made not from browser).
But you can try Selenium.

OK , I solved.
I use con.setRequestProperty and set "User-Agent", "Accept", "Content-Type", "Accept-Language".
Thank you.

Rest Assured - Getting Operation Timed Out Error while testing using RestAssured. But The same endpoint working fine in browser

I have an endpoint to be tested using RestAssured. The same endpoint is working fine while opening it in browser/Postman. But, while trying to test the same using RestAssured,
I am getting Operation Timed Out Error.
I had to connect to proxy to make that end point working in browser. used the same proxy in the rest assured also.
Sample Code below:
given().proxy("My_Proxy_URL_HERE",8080).when().get("My_API_URL_Here").then().log().all();
I am getting the response as
"Operation Timed Out" with Status Code 503.
I need your suggestion, what could be the possible issue, how to debug etc. Any suggestion is appreciated. Thanks in advance.

There can be many reasons for this behavior:
The address is just wrong and given there is some load balancer/proxy it can be configured to wait for a certain period of time and then respond with 503 status code.
Note, 503 is not a "request timed out", but "Service Unavailable".
The request url is good, but the request lacks some headers so that the load balancer/proxy won't be able to route the request to the required server.
How to check this? there exist tools that can come handy in this situation:
Check the access logs of the load balancer/proxy and even of your server if its possible - and see the request.
If it doesn't help, try to compare requests coming from rest-assured vs regular request. You can use tools like Burp for example, there are others, or you can even roll your own.
The idea is simple:
Start the "interceptor" on some port of your local computer (say, 9999 for example)
Configure the interceptor to forward all the requests to proxy of your choice (identified by URL - My_Proxy_URL_HERE and port 8080).
Now rest-assured must call localhost:9999 and the request will be intercepted by this tool. You'll be able to inspect its contents - headers, body, http method - everything.
Do the same for browser request and compare.

Gateway Timeout Error on Insert 70000 record using Hibernate in Java

My Problem is:
Upload 10000 Record from Excel To my Database. The Excel Sheet has 10000 Rows and 70 to 100 Column . We Store the Value in 6 Mapping Table using Hibernate cascade.
I call the method using ajax method. Due to inserting large amount of data. it return timeout error(502 (Proxy Error) OR 504 (Gateway Error)).
I am using AWS Services. Is any configuration mistake.. please help
Thanks in Advance

How You Might See a 502 Bad Gateway Error
Based on your web server, you might see a different 502 error. These all mean the same thing, it is only their naming conventions that differ. Here are a few examples of what you might see:
“502 Bad Gateway”
“HTTP Error 502 – Bad Gateway”
“502 Service Temporarily Overloaded”
“Error 502”
“502 Proxy Error”
“HTTP 502”
“502 Bad Gateway NGINX”
You can see in greater detail what the error specifically entails by going to your web server’s error log file. All error / diagnostic information is stored in this file making it a valuable resource to check when you need more details about a particular error. You can locate this file in Apache by going to /var/log/apache2/error.log and in Nginx by going to /var/log/nginx/error.log.
How to solve 502 errors?
Check if your FQDN is resolving correctly by using our DNS test tool.
Verify if your server is reachable by using a ping test or trace-route.
Check your firewall logs if you are seeing unusual drops.
Open a support in the KeyCDN dashboard if you can not solve the 502 problem.

Android Volley - Strange Error with HTTP code 401- java.io.IOException: No authentication challenges found

I meet this error when send a request and get back response with code 401:
com.android.volley.NoConnectionError: java.io.IOException: No authentication challenges found
Some people say that:
This error happens beause the server sends a 401 (Unauthorized) but does not give a "WWW-Authenticate" which is a hint for the client what to do next. The "WWW-Authenticate" Header tells the client which kind of authentication is needed (either Basic or Digest). This is usually not very useful in headless http clients, but thats how the standard is defined. The error occurs because the lib tries to parse the "WWW-Authenticate" header but can't.
( android - volley error No authentication challenges found )
But it's quite weird for me because I don't want to use WWW-authenticate things, I just want to get the code 401, but I always get the exception.
How can I bypass this problem? Any suggestion is really appreciated.

I have do some research and come to conclusion that, this is a server issue, that did not follow the convention.
From wiki:
401 Unauthorized (RFC 7235)
Similar to 403 Forbidden, but specifically for use when authentication is required and has failed or has not yet been provided. The response must include a WWW-Authenticate header field containing a challenge applicable to the requested resource. See Basic access authentication and Digest access authentication.
I think the best way to solve this problem is to solve in the server by add the header (something like:
WWW-Authenticate: xBasic realm=""
For me, I cannot change the server, so I have to check the error message to detect that a 401 error:
if (error.getMessage().equalsIgnoreCase("java.io.IOException: No authentication challenges found")){
showLoginError();
}
Not very elegant solution but work for now.

Exception when submitting a form with recaptcha

I have created a web form which is having few input text fields and a recaptcha field. I am using http://www.google.com/recaptcha/api/verify url to verify the recaptcha input values. I've used connection.getInputStream() of java.net.URLConnection to get the responses. But when I submit the form I received below exception from the server at connection.getInputStream() point. I have googled for possible solutions but I was failed to find a suitable answer for the issue. What could be the possible reason for this?
Exception is:
Server returned HTTP response code: 502 for URL: http://www.google.com/recaptcha/api/verify
My application configurations are listed below:
Apache Tomcat 7.0.47
Struts2 framework
Application running OS is Windows 2003 server/32bit
Above url is not blocked by my firewall or gateway as the application worked fine when I was using jboss server. The issue occured when I migrated to Tomcat server. And also this exception doesn't occure frequently as well. Sometime it submit the form successfully but sometime it gives the above exception, couldn't figure out the pattern it occur. Kindly appreciate if anyone could help me on this.