Develop Reference

How to write a Java code to read fields from a website that requires login and uses POST request?

Need some help with fetching some data from a website.
Previously , we had following code in our application and it used to fetch the required data. We just used to read the required fields by forming a URL by passing username , password and search parameter (DEA number). The same URL (with parameters ) could also be hit from browser directly to see the results. It was a simple GET request:
{URL url = new URL(
"http://www.deanumber.com/Websvc/deaWebsvc.asmx/GetQuery?UserName="+getUsername()+"&Password="+getPassword()+"&DEA="
+ deaNumber
+ "&BAC=&BASC=&ExpirationDate=&Company=&Zip=&State=&PI=&MaxRows=");
Document document = parser.parse(url.toExternalForm());
// Ask the document for a list of all <sect1> tags it contains
NodeList sections = document.getElementsByTagName("DEA");
//Followed by a loop code to get each element by using sections.item(index).getFirstChild() etc.
}
Now, the website URL has got changed to following:
https://www.deanumber.com/RelId/33637/ISvars/default/Home.htm
I am able to login to the URL with credentials , go to the search page , enter the DEA number and search. The login page comes as a pop-up once I click 'Login' link on home page. Also, the final result comes as a pop-up. This is a POST request so I am unable to form the complete URL which I could use in my code.
I am not an expert in Web Services , but I think I need a web service URL like the one mentioned in the code above. Not sure how to get that !! Even if I get the URL , I am not sure how to perform the login through Java code and search the DEA number.
Also, it would be great if I could validate the URL manually before using in Java. Let me know if there is any way.
Or, in case there is any alternate approach in Java; kindly suggest.
Thanks in advance.

First of all, the previous approach provided by the website was completely wrong and insecure, because it passes the username and password as querystring parameters in plain text. I think, they would have realized this thing and changed their way of authentication.
Also, it looks like that they have restricted the direct URL based requests from the client applications like yours. For such requests from clients, they have published the web services. Check this link. They also have mentioned the rates for web service request counts.
So, you may need to open a formal communication channel to get authentication and other details to access their web services for this purpose. Depends on what they use for web service client authentication, you may code your client to access the web services.
I hope this helps.

Authorize.net DPM -- perform server side processing in servlet rather than jsp

I'm currently working with a test account on Authorize.net and am utilizing their Direct Post Method form to submit transactions directly to their gateway without additional server-side processing on my end. My application is a basic jsp webapp sitting on top of Apache Tomcat 7.
Per the instructions provided on their Java Quick Start Guide I have set up 3 files to: 1) take in user input, 2) relay the response, and 3) process and display output.
Truth be told, I don't really need to display an output to the user. Instead, I would like to thoroughly process the response that Authorize.net sends me. The sample code they provide explicitly accounts for this in the relay_response.jsp file:
String receiptPageUrl = "http://MERCHANT_HOST/order_receipt.jsp";
...
net.authorize.sim.Result result = net.authorize.sim.Result.createResult(apiLoginId,
MD5HashKey, request.getParameterMap());
// perform Java server side processing...
// ...
// build receipt url buffer
StringBuffer receiptUrlBuffer = new StringBuffer(receiptPageUrl);
...
...
document.location = "<%=receiptUrlBuffer.toString()%>";
However, it looks like they want me to perform the processing in the jsp, while I would rather perform this work on the back end using a Java servlet. I've tried to accomplish this using 2 methods, neither of which work quite as I want.
Attempt 1) I replaced the 'order_receipt.jsp' tag with a url to another jsp, which subsequently submits a form to a servlet, passing all request parameters.
String receiptPageUrl = "http://<my_server's_ip_address>/another.jsp";
The problem with this approach is that in the initial forward from relay_response.jsp all of the parameters are passed via GET and appear in the URL, which I can't allow.
Attempt 2) Rather than forwarding the results to another jsp, I created a form right inside relay_response.jsp and tried to submit the form with the results passed as a request parameter.
<form id='myform' method='post' action="servlet_action" accept-charset='UTF-8'>
<input id='params' type='hidden' name='params' value='<%= paramsMap %>'/>
</form>
<script type="text/javascript">
document.getElementById("myform").submit();
</script>
The problem here is that although the browser displays my relay_response.jsp file, the value of document.location.hostname is test.authorize.net, so it doesn't recognize my action since that resides on my server rather than on authorize.net's server.
Alternatively, I have tried setting the action on the form to be the full url of my server and servlet action:
<form id='myform' method='post' action="http://<my_server's_ip_address>/webapp/servlet_action" accept-charset='UTF-8'>
But I get a warning (at least in Firefox) saying that the data is not being transmitted over a secure connection: "Although this page is encrypted, the information you have entered is to be sent over an unencrypted connection and could easily be read by a third party."
How can I pass the results of the transaction from relay_response.jsp to my Java servlet without exposing the parameters being passed to the user? Should I be using https? And why is document.location.host pointing to authorize.net rather than my relay_response.jsp?
Thanks!

A friend suggested 2 solutions for the initial question I posted, one of which I have verified.
Solution 1:
Simply redirect the initial form to servlet rather than to relay_response.jsp. Then the servlet can redirect to another jsp as apporpriate. I have verified that this works with Authorize.net DPM.
Solution 2:
Inside the scriptlet in relay_response.jsp, make a call to a Java class that actually handles the logic. You don't have to expose or write any Java code inside the scriptlet, but rather just invoke the class and call a few methods. You can pass the response parameter map as the argument to the method. I suppose the class you invoke could even be a proper servlet, though mixing these up might not be good form.

HtmlUnit: Request website from server in a specific language

I am looking for a clean/simple way in HtmlUnit to request a webpage from a server in a specific language.
To do this i have been trying to request "bankofamerica.com" for their homepage in spanish instead of english.
This is what i have done so far:
I tried to set "Accept-Language" header to "es" in the Http request. I did this using:
myWebClient.addRequestHeader("Accept-Language" , "es");
It did not work. I then created a web request with the following code:
URL myUrl = new URL("https://www.bankofamerica.com/");
WebRequest myRequest = new WebRequest(myUrl);
myRequest.setAdditionalHeader("Accept-Language", "es");
HtmlPage aPage = myWebClient.getPage(myRequest);
Since this failed too i printed out the request object for this url , to check if these headers are being set.
[<url="https://www.bankofamerica.com/", GET, EncodingType[name=application/x-www-form-urlencoded], [], {Accept-Language=es, Accept-Encoding=gzip, deflate, Accept=*/*}, null>]
So the server is being requested for a spanish page but in response its sending the homepage in english (the response header has the value of Content-Language set to en-US)
I did find a hack to retrieve the BOA page in spanish. I visited this page and used the chrome developer tool to get the cookie value from the request
header. I used this value to do the following:
myRequest.setAdditionalHeader("Cookie", "TLTSID= ........._LOCALE_COOKIE=es-US; CONTEXT=es_US; INTL_LANG=es_US; LANG_COOKIE=es_US; hp_pf_anon=anon=((ct=+||st=+||fn=+||zc=+||lang=es_US));..........1870903; throttle_value=43");
I am guessing the answer lies somewhere here.
Here lies my next question. If i am writing a script to retrieve 100 different websites in Spanish (ie Assuming they all have their pages in the spanish) . Is there a clean way in HtmlUnit to accomplish this.
(If cookies is indeed a solution then to create them in htmlunit you need to specify the domain name. One would have to then create cookies for each of the 100 sites. As far as i know there is no way in HtmlUnit to do something like:
Cookie langCookie = new Cookie("All Domains","LANG_COOKIE","es_US");
myWebClient.getCookieManager().addCookie(langCookie);)
NOTE: I am using HtmlUnit 2.12 and setting BrowserVersion.CHROME in the webclient
Thanks.

Regarding your first concern the clear/simple(/only?) way of requesting a webpage in a particular language is, as you said, to set the HTTP Accept-Language request header to the locale(s) you want. That is it.
Now the fact that you request a page in a particular language doesn't mean that you will actually get a page in that language. The server has to be set up to process that HTTP header and respond accordingly. Even if a site has a whole section in spanish it doesn't mean that the site is responding to the HTTP header.
A clear example of this is the page you provided. I performed a quick test on it and found that it is clearly not responding accordingly to the Accept-Language I've set (which was es). Hitting the home page using es resulted in getting results in english. However, the page has a link that states En Español which means In Spanish the page does switch to spanish and you get redirected to https://www.bankofamerica.com?request_locale=es_US.
So you might be tempted to think that the page handles the locale by a request parameter. However, that is not (only) the case. Because if you then open the home page again (without the locale parameter) you will see the Spanish version again. That is clearly a proof that they are being stored somewhere else, most likely in the session, which will most likely be handled by cookies.
That can easily be confirmed by opening a private session or clearing the cookies and confirming this behaviour (I've just done that).
I think that explains the mystery of the webpage existing in Spanish but being fetched in English. (Note how most bank webpages do not conform to basic standards such as responding to simple HTTP requests... and they are handling our money!)
Regarding your second question, it would be like asking What is the recipe to not get ill ever?. It just doesn't depend on you. Also note that your first concerned used the word request while your second concern used the word retrieve. I think it should be clear by now that you can only be 100% sure of what you request but not of what you retrieve.
Regarding setting a value in a cookie manually, that is technically possible. However, that is just like adding another parameter in a get request: http://domain.com?login=yes. The parameter will only be processed by the server if it is expecting it. Otherwise, it will be ignored. That is what will happen to the value in your cookie.
Summary: There are standards to follow. You can try to use them but if the one in the other side doesn't then you won't get the results you expect. Your best choice: do your best and follow the standards.

Logging in remote site then testing the form

There's a website www.runescape.com and I'm trying to find a way to login the site and automate entering 2-3 letter words (from a-z0-9) to find a unique list of available usernames. (https://secure.runescape.com/m=displaynames/name.ws)
This is the only page I know of where you can view usernames that are available or taken.
In the form where you change your name--you type in a name and it queries it for you with no form submit needed. In about a second or two, the page will display a lime green "Available" or a red "Not available" in text.
Is there a way in java where I can login to see this page, then enter in values, and read if username(s) are Available or Not available? Can someone help point me in the right direction to do this?
Much appreciation.

By viewing the source code of the web site, I can see that the form is sent via POST to https://secure.runescape.com/m=weblogin/login.ws. You will want to send an HTTP Request to that address with the associated username and password parameters respectively.
There's a Java HTTP Request tutorial on SO Here: Using java.net.URLConnection to fire and handle HTTP requests
After you've sent the HTTP Request to the page, an HTTP Response object should be returned to you that has the body of the entire web page that you're looking for. Most likely, you'll need to use RegEx to scan through the body of text and filter out the data you want e.g. <li>([a-z0-9]{1,20})<\/li>
Handling the HTTP Response object is also available on the link provided.

Is it a good practice to use asynchronous requests in this scenario?

I have a scenario that I have a button in JSP page which sends an email, the request is send to servlet asynchronously using jQuery Ajax and JSON, servlet searches in DB, if the user has an email, it returns the email address and sends an email to it, then forwards to the result page with success or fail of sending the email, but in a case that the user doesn't have an email, it returns false values using JSON to JSP and then a JSP form appears to the user to enter his email.
Is it good practice to use Ajax and I know that not each time there's a return value to the user or send request to servlet using get method which return a parameter in a case that the user doesn't have an email?

Using ajax is in practically all cases very good for User Experience. With ajax, the user will experience instant feedback without the need to face an annoying "flash of content" or a (partially) empty page because the whole HTML response needs to be generated/buffered by the server first. This is really a huge plus of using JS/ajax.
Using JSON is generally favorable above XML, HTML or even plain text. But there is no "best practice" with regard to the ajax data exchange format between client and server. Just pick whatever suits the requirement the best. JSON is perfectly fine for this case. jQuery understands it out-the-box and in Java you have choice of a plethora of easy-to-use JSON parsers.
However, when developing an ajax-enabled webapplication, you really need to take into account that the core functionality does not break when the client has JS disabled. This is called Unobtrusive JavaScript. Most of the searchbots, mobile browsers and textbased browsers don't use JS. You should try to use JS only for Progressive Enhancements. To test this yourself, in Firefox you can use for example the Web Developer Toolbar to easily enable/disable JS support. Run your website with JS disabled and observe if the core functionality is maintained as well.
The best way to achieve this is to start developing the website without any single line of JS code, even without a single onclick, onsubmit, onwhatever attribute on the HTML elements. Once you get the core functionality to work, then you can start adding JS in flavor of a script which executes during document ready and attachs functions to the HTML elements of interest (even here, you should not change the original HTML code!). Let the JS functions fire ajax requests on the same URL or maybe a different one, depending on the requirement. You can in the Servlet distinguish between an ajax and normal request as follows:
if ("XMLHttpRequest".equals(request.getHeader("X-Requested-With"))) {
// Handle ajax request. Return JSON response here.
} else {
// Handle normal request. Return normal HTML response here (by JSP).
}
See also:
Simple calculator in JSP - contains unobtrusive JSP/Servlet/jQuery example

Json is just a data-interchange format. Using Json or not has nothing to do with using asynchronous communication or not... You can do both communication types using Json (or XML, or serialized objects, it doesn't matter).
Now, in your problem, it looks like you just want to use Asynchronous communication to improve the user experience (it will not flick the user's browser). If that's the case, Asynchronous communication is the way to go!

I don't think you need ot use AJAX in this.
The main idea of the ajax is to render server response without postback and in your case you are redirecting page after you get some kind of result.
In my opinion you shoul choose on of these two ways.
1) Use AJAX, send data to servlet and then render response from server wether the mail is sent or not.
2) Submit your form to servlet and sent email and then redirect to jsp with the success/fail result.
Hope it helps.