Develop Reference

Exposing a web site through web services

I know what I am asking is somehow weird. There is a web application (which we don't have access to its source code), and we want to expose a few of its features as web services.
I was thinking to use something like Selenium WebDriver, so I simulate web clicks on the application according to the web service request.
I want to know whether this is a better solution or pattern to do this.
I shall mention that the application is written using Java, Spring MVC (it is not SPA) and Spring Security. And there is a CAS server providing SSO.

There are multiple ways to implement it. In my opinion Selenium/PhantomJS is not the best option as if the web is properly designed, you can interact with it only using the provided HTML or even some API rather than needing all the CSS, and execute the javascript async requests. As your page is not SPA it's quite likely that an "API" already exists in form of GET/POST requests and you might be lucky enough that there's no CSRF protection.
First of all, you need to solve the authentication against the CAS. There are multiple types of authentication in oAuth, but you should get an API token that enables you access to the application. This token should be added in form of HTTP Header or Cookie in every single request. Ideally this token shouldn't expire, otherwise you'll need to implement a re-authentication logic in your app.
Once the authentication part is resolved, you'll need quite a lot of patience, open the target website with the web inspector of your preferred web browser and go to the Network panel and execute the actions that you want to run programmatically. There you'll find your request with all the headers and content and the response.
That's what you need to code. There are plenty of libraries to achieve that in Java. You can have a look at Jsop if you need to parse HTML, but to run plain GET/POST requests, go for RestTemplate (in Spring) or JAX-RS/Jersey 2 Client.
You might consider implementing a cache layer to increase performance if the result of the query is maintained over the time, or you can assume that in, let's say 5 minutes, the response will be the same to the same query.
You can create your app in your favourite language/framework. I'd recommend to start with SpringBoot + MVC + DevTools. That'd contain all you need + Jsoup if you need to parse some HTML. Later on you can add the cache provider if needed.

We do something similar to access web banking on behalf of a user, scrape his account data and obtain a credit score. In most cases, we have managed to reverse-engineer mobile apps and sniff traffic to use undocumented APIs. In others, we have to fall back to web scraping.
You can have two other types of applications to scrape:
Data is essentially the same for any user, like product listings in Amazon
Data is specific to each user, like in a banking app.
In the firs case, you could have your scraper running and populating a local database and use your local data to provide the web service. In the later case, you cannot do that and you need to scrape the site on user's request.
I understand from your explanation that you are in this later case.
When web scraping you can find really difficult web apps:
Some may require you to send data from previous requests to the next
Others render most data on the client with JavaScript
If any of these two is your case, Selenium will make your implementation easier though not performant.
Implementing the first without selenium will require you to do lots of trial an error to get the thing working because you will be simulating the requests and you will need to know what data is expected from the client. Whereas if you use selenium you will be executing the same interactions that you do with the browser and hence sending the expected data.
Implementing the second case requires your scraper to support JavaScript. AFAIK best support is provided by selenium. HtmlUnit claims to provide fair support, and I think JSoup provides no support to JavaScript.
Finally, if your solution takes too much time you can mitigate the problem providing your web service with a notification mechanism, similar to Webhooks or Resthooks:
A client of your web service would make a request for data providing a URI they would like to get notified when the results are ready.
Your service would respond immediatly with an id of the request and start scraping the necessary info in the background.
If you use skinny payload model, when the scraping is done, you store the response in your data store with an id identifying the original request. This response will be exposed as a resource.
You would execute an HTTPPOST on the URI provided by the client. In the body of the request you would add the URI of the response resource.
The client can now GET the response resource and because the request and response have the same id, the client can correlate both.

Selenium isn't a best way to consume webservices. Selenium is preferably an automation tool largely used for testing the applications.
Assuming the services are already developed, the first thing we need to do is authenticate user request.
This can be done by adding a HttpHeader with key as "Authorization" and value as "Basic "+ Base64Encode(username+":"+password)
If the user is valid (Users login credentials match with credentials in server) then generate a unique token, store the token in server by mapping with the user Id and
set the same token in the response header or create a cookie containing token.
By doing this we can avoid validating credentials for the following requests form the same user by just looking for the token in the response header or cookie.
If the services are designed to chcek login every time the "Authorization" header needs to be set in request every time when the request is made.

I think it is a lot of overhead using a webdriver but it depends on what you really want to achieve. With the info you provided I would rather go with a restTemplate implementation sending the appropriate http messages to the existing webapp, wrap it with a nice #service layer and build your web service (rest or soap) on top of it.
The authentication is a matter of configuration, you can pack this in a microservice with #EnableOAuth2Sso and your restTemplate bean, thanks to spring boot, will handle the underlining auth part for you.

May be overkill..... But RPA? http://windowsitpro.com/scripting/review-automation-anywhere-enterprise

Continuing sessions using JSESSIONID

I have a web application which requires username and password authentication to enter.
What I am doing is, authenticate a user from a stand alone Java app, which would do so by making Http request to the server using username and password. Then I would retrieve JSESSIONID cookie from response of server.
Now what I want is to use this JSESSIONID to continue session on browser i.e. to let user navigate pages of my web app which would be opened by my stand alone java app which I use for authentication.
Is this possible? Or is there any other way to do so.

Cookie can be changed using below mentioned methods.
Cookie cookie = new Cookie("JSESSIONID", NEWSESSIONID);
response.addCookie(cookie);
From your application you can send JSESSIONID as parameter while opening browser first time and reset your cookie using above method either in filter or servlet. This will reset your cookie in client side once you send response back. Next request on wards you will be able to access the session created previously.

It's possible but it's not that simple.
Since web applications don't share sessions, what you're looking for is a Single Sign On (SSO) solution, which involves an "Identity Provider" (IdM) that authenticates users for one or more "Service Providers" (SP). In this case, your servlet is the IdM and your web app is an SP.
Depending on your deployment, the following are third-party, open-source SSO libraries that you may be able to use:
Kerberos
PicketLink (for JBOSS)
OpenAM (for Tomcat)
If you don't want to use a third-party library, you may also be able to modify your servlet to be the IdM. Either way, I suggest reading a little about Security Assertion Markup Language (SAML) before deciding on a solution. SAML is a popular method that the above libraries implement.

JAX-WS, Authentication and Authorization - How to?

What is the best way to do authentication and authorization in web services?
I am developing a set of web services, requiring role based access control.
Using metro - SOAP, simple java without EJBs.
I want to Authenticate the user just one time, using username and
password, to be matched against a data base. In the subsequent calls.
I would like to use some kind of session management. Could be some
session id, retrieved to the client at login, to be presented in all
calls.
So Far:
Read authentication using a database - but I want application level validation;
Read application authentication with jax-ws - but i don't want to do the authentication mechanism every time;
I think I can use a SOAP Handler, to intercept all the messages, and do the authorization control in the hander, using some session identifier token, that comes with the message, that can be matched against an identifier saved in the data base, in the login web method.
EDIT:
I still have some questions:
How to know the name of the web method being called?
What kind of token should I use?
How to pass this token between calls?
EDIT 2
Because of #ag112 answer:
I'm using Glassfish.
I use WS-Policy and WS-Security to encrypt and sign the messages. Using Mutual Certificate Authentication. I would like to complement this message level security between applications, with the authentication and authorization for the users also in message level.
I am just developing the services, and I don't know almost nothing the clients, just that they could be created in different languages.
At this point I think the most important thing is to do what ever I need to do to authenticate and authentication the users, I the most easy way to be implemented for the client applications.

#Luis: Here are my inputs.
Well exact solution for your problem depends upon kind of web service clients you expect, do you have control over web service client system, your app server etc.....but assuming you don't have any control over web service client, for you it is just a SOAP message over HTTP transport, here is probable solution.
You can of course performs session management & authentication at message level or transport level. It means either you can have session token and auth token information in SOAP message or you can use standard HTTP Session and HTTP authentication mechanism.
Of course transport level solution is much simpler and industry wide standard in case if transport layer is HTTP. For message level, ws specifications like ws-security can be used.
Your each web service request is simple HTTP GET/POST identified by a unique HTTP URI.
Typically in jax-ws metro environment, WSServlet is one which entry servlet for any web service call and which eventually delegates the call to right service provider implementation class. Since you application is going to be deployed in web server, you can exploit all session and authentication facilities provided by J2ee web container.
Since you are looking for role-based access control, I would use standard <web-resource-collection> in web.xml to specify which role you would like to have in case of particular HTTP URI. You can use standard JAAS login module which can do authentication and populates the JAAS subject with role. If user name/password are provided in SOAP XML, JAAS login module can also search/parse SOAP XML to retrieve those information. JAAS/app server will automatically create auth token and store it as cookie so that each subsequent request need not to go through authentication process again. This is all J2ee standard. You can find plenty of help on internet on this. Please let me know your app server so that I can provide you additional details.
If you still want to use SOAP message level session management, authentication & authorization process, then to provide you more details, may I know more details about your client side.
EDIT1:
Well based on your further inputs, here is my more thoughts:
Message security namely encryption and signature needs to happen each message travels between server and client. where as message authentication- you intend to do once and give a session token/auth token to client for subsequent calls.
Question still remains: if you put a unique session Identifier in SOAP response of first time authentication, do you expect client to parse SOAP response XML and ensure that client should send you session identifier each time in subsequent SOAP requests.
OR
You want to keep session management transparent to client and for client it needs to send username/password token first time and subsequent calls need not require any username/password token. In this case you would need to rely on transport based session management for e.g. HTTP cookies
Now what is the best for you depends upon your use case. Can you tell me what is expected use case flow? how another system (web service client) makes more than one service call to your system? Is another system user driven/some background process? What is exact need that you want only first service call to go through authentication process not subsequent calls?
PS: Glassfish server provides a way of configuring message authentication provider which automatically enables/disables message level authentication.
EDIT2:
I understand you do not want to store user credentials in client app and web service server need those user credentials.
OAuth is open standard protocol which allows site A to access user's private data on site B. Ultimate idea is site A gets auth token which has specific expiry time. So Token containing encrypted from of user credentials or jsession id help you avoid need of re-authentication. You only need to decide where you want to keep token at client app side
You can keep token as cookie if transport is HTTP protocol.
Having said that ofcourse passing user credentials each time seems bit easier and straight forward.

You can also go for OpenEJB.
It used JAAS with WS-Security.
I hope the link is useful.

After all the help, I create this answer to simplify, and summarize all the ideas that was discussed.
The questions has 2 requisites:
Message level security;
One time authentication.
With ag112 help, this is hard to do, or to elegant in any way. So here are to conclusions:
For message level security send the user
credentials every time (place it in SOAP header);
For one time authentication use transport level security, and do a
session management.
I prefer the first one, because the message level was the biggest requisite.

As had no answers, following #unhillbilly advise, I answer my own question, with the progress so far:
How to know the name of the web method
being called;
Using a SOAP handler, finding the name of the first element in the body.
What kind of token should I use;
I decide to use a 128 bits token, representing each session. The Webservices, continue to be session-less, the key is just for authorizations purposes.
How to pass this token between calls.
For the login web method the result has the token, in each subsequent calls, the token is a parameter.
is there a better answer?

Best option for Session management in Java

Best way managing session in Java. I heard that cookies are not reliable option for this as they gets stored into browser and can be accessed later on? Is this correct? If possible please come up with the answers with the coding example.
Which is the best among:
URL Rewriting: Server will add an additional parameter at the end of URL link
Hidden parameter in Form: server will add an additional parameter at every form in HTML
cookie: Server will ask browser to maintain a cookie.

The session management (client identification, cookie handling, saving session scoped data and so on) is basically already done by the appserver itself. You don't need to worry about it at all. You can just set/get Java objects in the session by HttpSession#setAttribute() and #getAttribute(). Only thing what you really need to take care of is the URL rewriting for the case that the client doesn't support cookies. It will then append a jsessionid identifier to the URL. In the JSP you can use the JSTL's c:url for this. In the Servlet you can use HttpServletResponse#encodeURL() for this. This way the server can identify the client by reading the new request URL.
Your new question shall probably be "But how are cookies related to this? How does the server do it all?". Well, the answer is this: if the server receives a request from a client and the server side code (your code) is trying to get the HttpSession by HttpServletRequest#getSession() while there's no one created yet (first request in a fresh session), the server will create a new one itself. The server will generate a long, unique and hard-to-guess ID (the one which you can get by HttpSession#getId()) and set this ID as a value of the cookie with the name jsessionid. Under the hood the server uses HttpServletResponse#addCookie() for this. Finally the server will store all sessions in some kind of Map with the session ID as key and the HttpSession as value.
According to the HTTP cookie spec the client is required to send the same cookies back in the headers of the subsequent request. Under the hood the server will search for the jsessionid cookie by HttpServletRequest#getCookies() and determine its value. This way the server is able to obtain the associated HttpSession and give it back by every call on HttpServletRequest#getSession().
To the point: the only thing which is stored in the client side is the session ID (in flavor of a cookie) and the HttpSession object (including all of its attributes) is stored in the server side (in Java's memory). You don't need to worry about session management youself and you also don't need to worry about the security.
See also:
Authenticating the username, password by using filters in Java (contacting with database)
How to redirect to Login page when Session is expired in Java web application?
How to implement "Stay Logged In" when user login in to the web application

All Java web frameworks support cookies or URL-encoded session IDs. They will chose the correct approach automatically, so there is nothing you need to do. Just request the session object from your container and it will handle the details.
[EDIT] There are two options: Cookies and a special URL. There are problems with both approaches. For example, if you encode the session in an URL, people can try to pass the session on (by putting the URL into a mail, for example). If you want to understand this, read a couple of articles about security and build app servers. Otherwise: Your Java app server will do the right thing for you. Don't think about it.

The cookie just stores the session ID, this ID is useless once the session has expired.

Servlet specification defines the API for accessing/setting session data in standard J2EE application. Also it defines that session data is stored on the server-side and nothing is transferred to the client except the session identifier. There are 2 mechanisms how session id is transferred:
1) request URL e.g. jessionid=....
2) cookie
Mechanism is determined automatically based on client capabilities.
EDIT. There is no best option, there is servlet specification that defines the way.

Http is a stateless, client-side pull only protocol.
To implement a stateful conversation over it, Java EE Web Server need to hide some information (which is sessionid) in client-side and the mechanism it can use should follow HTTP and HTML spec.
There are three ways to accomplish this goal:
URL Rewriting: Server will add an additional parameter at the end of URL link.
Hidden parameter in Form: server will add an additional parameter at every form in HTML.
cookie: Server will ask browser to maintain a cookie.
Basically, modern web server will have a "filter" to choose which way to use automatically.
So if Server detected that browser already turn off cookie support, it will switch to other ways.

2 important questions:
Which web technology are you using? JSF, Struts, SpringMVC or just plain servlets/JSPs.
Servlets/JSPs already give you the session support you need. JSP Example: Hello, <%= session.getAttribute( "theName" ) %>
I really don't think you have something to worry about cookies, since the data is stored safely in the server and handeling the cookie is done automaticlly.
Is your application installed on a single server?
If YES than you have no problem, use the servlet session option.
if NO than you gotta find another way to do this. Like using a sticky session, or maybe parse the entire session object in the requests/responses as a field. This option indeed requires you to take security measures.

Sending a HTTP POST request

To implement single sign off, i would like the user to get logged out of application B additionally when ever the user clicks logout on application A. Is it possible to implement this using some form of a POST request to application B? i.e. when the user clicks on logout:
Generate existing POST request to logout of application A
Generate additional POST request to logout of application B as well.

The cleanest way to do this is to check if your SSO provider has a single-sign-off feature.
Coding this up and deploying it would make your overall IT solution a bit brittle.
Another suggestion is to take this up with your (Enterprise) architect as SSO is usually an enterprise initiative and point her to (very cogent) arguments in this post : http://lists.danga.com/pipermail/yadis/2005-July/001085.html

Yes, how you do it depends on the programming language you are using.
For example under ASP.Net you'd use System.Net.HttpWebRequest within the handling of the Logout event of application A to make a logout request to application B
If you can post what language you're working in I can give a proper example

Depending on the implementation of your authentication system, probably you can/need to send the POST using JavaScript instead of from server-side.

Without specific information, it's hard to give a specific answer, but as you're refering to POST, I'll assume a browser is involved.
POSTs (without using Javascript or similar) occur when a form is submitted. As the form can have only one action, it can only target one server-side page.
One solution is to simply have Application A forward sign-out credentials to Application B once one action is received, which allows for more opportunities to check returns.
If, however, you're set on POST'ing to different pages, see this tutorial for one iframe-related hack - http://www.codeproject.com/KB/scripting/multiact.aspx

If your login session is stored by a cookie, and there are nothing else you need to supply to log out of application B, clearing the cookie in javascript will usually destroy the session and sign the user out.

How about making it a cookie based authentication? A same cookie authenticates a user for various applications (in your case 2 different application.) Once a user sign off from one application (app A), invalidates a cookie (by expiry date) so that whenever a user sends a POST request to rest of the application (app B) the request is not processed. A Servlet that traces each POST request to validate the cookie is required for each application.