Develop Reference

How to write a Java code to read fields from a website that requires login and uses POST request?

Need some help with fetching some data from a website.
Previously , we had following code in our application and it used to fetch the required data. We just used to read the required fields by forming a URL by passing username , password and search parameter (DEA number). The same URL (with parameters ) could also be hit from browser directly to see the results. It was a simple GET request:
{URL url = new URL(
"http://www.deanumber.com/Websvc/deaWebsvc.asmx/GetQuery?UserName="+getUsername()+"&Password="+getPassword()+"&DEA="
+ deaNumber
+ "&BAC=&BASC=&ExpirationDate=&Company=&Zip=&State=&PI=&MaxRows=");
Document document = parser.parse(url.toExternalForm());
// Ask the document for a list of all <sect1> tags it contains
NodeList sections = document.getElementsByTagName("DEA");
//Followed by a loop code to get each element by using sections.item(index).getFirstChild() etc.
}
Now, the website URL has got changed to following:
https://www.deanumber.com/RelId/33637/ISvars/default/Home.htm
I am able to login to the URL with credentials , go to the search page , enter the DEA number and search. The login page comes as a pop-up once I click 'Login' link on home page. Also, the final result comes as a pop-up. This is a POST request so I am unable to form the complete URL which I could use in my code.
I am not an expert in Web Services , but I think I need a web service URL like the one mentioned in the code above. Not sure how to get that !! Even if I get the URL , I am not sure how to perform the login through Java code and search the DEA number.
Also, it would be great if I could validate the URL manually before using in Java. Let me know if there is any way.
Or, in case there is any alternate approach in Java; kindly suggest.
Thanks in advance.

First of all, the previous approach provided by the website was completely wrong and insecure, because it passes the username and password as querystring parameters in plain text. I think, they would have realized this thing and changed their way of authentication.
Also, it looks like that they have restricted the direct URL based requests from the client applications like yours. For such requests from clients, they have published the web services. Check this link. They also have mentioned the rates for web service request counts.
So, you may need to open a formal communication channel to get authentication and other details to access their web services for this purpose. Depends on what they use for web service client authentication, you may code your client to access the web services.
I hope this helps.

Identifying a client behind router/firewall

In my web-app I need to register http clients accessing from a local network behind a router.
I started with remoteHost : remotePort combination, but soon enough it became clear, that the port numer gets regenereated upon each connection.
I need to be able to identify the clients on something similar to MAC address, some property that doesn't change. I wanted to use headers[ "X-Forwarded-For" ], but it's not present at all:
[Pragma=no-cache, Cache-Control=no-cache, Host=somhost.com:8822, Upgrade=websocket, Connection=Upgrade, Sec-WebSocket-Key=scnlM7hzjjy3cklJhJciA==, Sec-WebSocket-Extensions=x-webkit-deflate-frame,deflate-frame, Sec-WebSocket-Version=13]
What are the other options to identify clients?

You could use an API key, that is, a unique identifier that the clients send along with each request to identify themselves. Depending on the authentication method you are using, you could consider the standard HTTP Authorization header to send this value:
Authorization: API-Key <value goes here>
Or create a custom HTTP header for this purpose. But be careful with custom headers: proxies might strip them out.

One option is using cookies. As the client accesses the webapp for the first time we could set a cookie on the client side that has a very long expiry date.
During the subsequent user re-logins we can rely on this cookie as cookies get sent to the server.

You can try this bit of PHP to see what the server knows about an incoming http request:
$keys = array_keys($_SERVER);
echo "<table bgcolor='black' cellpadding='1' cellspacing='1'>\n";
echo " <tr bgcolor='yellow'><td><b>Key</b></td><td><b>Value</b></td></tr>\n";
foreach ($keys as $key) {
echo " <tr bgcolor='white'><td>" . $key . "</td><td>" . $_SERVER[$key] . "</td></tr>\n";
}
echo "</table>\n";

Are you identifying the user at the keyboard or the device making the request? Do you need to track these long term or only for the duration of a use session? Do your users connect from multiple devices?
Client side id certificates could work, depending on how the local machines are managed. If they are accessing your app from someplace they've already authenticated, then setting up a single sign on solution could work. Prompting for authentication always works too.

consuming Java web services. Issues with id session

i am having some issues and almost no documentation on internet about id session tokens and Java web services. We have a server (Tomcat providing web services on JAX-WS + SOAP) that is providing an API to query an SQL server.
I have to code a simple Java client that communicates with the server through this web services. The issue is that before being able to use any of its server WSDL methods, the client has to authenticate through a web server method where you send the user/pass, and the server gives back to client a session-id (alfanumeric string).
The rest of the methods do not have any kind of parameter where i can pass the session id, so i suppose it has to be used as a "context". I have found information about how in JAX-WS you can maintain session-id:
https://weblogs.java.net/blog/ramapulavarthi/archive/2006/06/maintaining_ses.html
Hello port = new HelloService().getHelloPort();
((BindingProvider)port).getRequestContext().put(BindingProvider.
SESSION_MAINTAIN_PROPERTY,true);
in my case, if i want to receive the cookie session-id, i have to:
org.tempuri.RUWebService service = new org.tempuri.RUWebService();
org.tempuri.RUWebServiceSoap myport = new org.tempuri.RUWebServiceSoap();
String session-id = myport.Auth(user,pass);
where the session-id is an UUID variable is a String of a hex variable: 8-4-4-4-8 .. that i can change to real hex like:
java.util.UUID uuidFromHyphens = java.util.UUID.fromString("6f34f25e-0b0d-4426-8ece-a8b3f27f4b63");
I tried the following code, where i change "port" variable to "myport", trying to match both examples:
((BindingProvider)myport).getRequestContext().put(BindingProvider.
SESSION_MAINTAIN_PROPERTY,true);
It compiles, but when i consult the web server through some of its methods, i receive a "null", the same as if i do not make the auth procedure. My issue i think is that i don't know what i am doing wrong with cookie session-id auth procedure.
If somebody could help me, i would be grateful.
regards.

Either you pass it as Request Parameter using GET or POST methods
(or)
Put the session id in outgoing HTTP header, like
GET / HTTP/1.0
Accept: text/plain
Accept: text/html
Session-Id:DFF55566_SOMEID
But for both of this to work, your web server must expect the Session Id from client in some format either at server level (or) your application level.

App engine request headers for ip address

I have an app engine application that runs REST web services.
I want to extract the ip address from all requests that are handled by my web services.
from javax.servlet.http.HttpServletRequest i'm trying to extract the ip address checking the "X-Real-IP" , if empty or "unknown" check the first ip in the list of "X-Forwarded-For" header if empty or "unknown" get it from request.getRemoteAddr().
I thought i covered all the cases but i'm still getting ip addresses like 10.x.x.x, or 127.0.0.1 or unknown.
I know that app engine applications are running behind load balancers, and instances are dynamic and i'm certainly omitting a header in the request cuz i can see the original ip address in the logs (from google) .
Edit : all the requests i'm working on are direct request to service (no queue or cron requests).
Any idea of the other headers to check ?
thx .

The answeres of this Question might help you. There are a lot of headers to check for:
private static final String[] HEADERS_TO_TRY = {
"X-Forwarded-For",
"Proxy-Client-IP",
"WL-Proxy-Client-IP",
"HTTP_X_FORWARDED_FOR",
"HTTP_X_FORWARDED",
"HTTP_X_CLUSTER_CLIENT_IP",
"HTTP_CLIENT_IP",
"HTTP_FORWARDED_FOR",
"HTTP_FORWARDED",
"HTTP_VIA",
"REMOTE_ADDR" };

URL not changing on page redirection in Jersey and MongoDB

My scenario is like this:
I'm building a website where I'm posting an ad regarding a topic. So, after the form filling of ad, the request goes to a REST service class as:
http://localhost:8080/cloudproject/postadvaction?title=tution&tag=tution&description=tution+%401000+%2F+month&category=TUTOR&location=indore
Here, the details of ad go in the database which is MongoDB. After all of this is done I'm redirecting to the profile page of user using Viewable model of jersey, where he can see all the ads posted by him. It is done as:
return new Viewable("/profile.jsp");
After this the response is redirected to profile page of the user.
But the problem is that, on redirecting the response to simply profile.jsp, the URL in the address bar has not changed to http://localhost:8080/profile.jsp, instead, it has remained the same as mentioned above. So, when user refreshes the page, the request of same ad post triggers and the whole process is followed again. Since, database is MongoDB, same ad is stored twice in it and same is displayed on the profile page of user with 2 identical ads.
So, how can I redirect to profile page without having the address of servlet in address bar?
Update: The question is related to PRG technique & Duplicate Form Submissions and not to just redirection.

See Post/Redirect/Get
When a web form is submitted to a server through an HTTP POST request, a web user that attempts to refresh the server response in certain user agents can cause the contents of the original HTTP POST request to be resubmitted, possibly causing undesired results, such as a duplicate web purchase.
To avoid this problem, many web developers use the PRG pattern[1] — instead of returning a web page directly, the POST operation returns a redirection command. The HTTP 1.1 specification introduced the HTTP 303 ("See other") response code to ensure that in this situation, the web user's browser can safely refresh the server response without causing the initial HTTP POST request to be resubmitted. However most common commercial applications in use today (new and old alike) still continue to issue HTTP 302 ("Found") responses in these situations.
With Jersey you can use
Response.seeOther(URI) - Create a new ResponseBuilder for a redirection. Used in the redirect-after-POST (aka POST/redirect/GET) pattern.
You just need to change your method signature to return a Response and return the built Response
return Response.seeOther(URI.create(...)).build();
Also stated about the URI parameter
the redirection URI. If a relative URI is supplied it will be converted into an absolute URI by resolving it relative to the base URI of the application (see UriInfo.getBaseUri()).