Develop Reference

Exposing a web site through web services

I know what I am asking is somehow weird. There is a web application (which we don't have access to its source code), and we want to expose a few of its features as web services.
I was thinking to use something like Selenium WebDriver, so I simulate web clicks on the application according to the web service request.
I want to know whether this is a better solution or pattern to do this.
I shall mention that the application is written using Java, Spring MVC (it is not SPA) and Spring Security. And there is a CAS server providing SSO.

There are multiple ways to implement it. In my opinion Selenium/PhantomJS is not the best option as if the web is properly designed, you can interact with it only using the provided HTML or even some API rather than needing all the CSS, and execute the javascript async requests. As your page is not SPA it's quite likely that an "API" already exists in form of GET/POST requests and you might be lucky enough that there's no CSRF protection.
First of all, you need to solve the authentication against the CAS. There are multiple types of authentication in oAuth, but you should get an API token that enables you access to the application. This token should be added in form of HTTP Header or Cookie in every single request. Ideally this token shouldn't expire, otherwise you'll need to implement a re-authentication logic in your app.
Once the authentication part is resolved, you'll need quite a lot of patience, open the target website with the web inspector of your preferred web browser and go to the Network panel and execute the actions that you want to run programmatically. There you'll find your request with all the headers and content and the response.
That's what you need to code. There are plenty of libraries to achieve that in Java. You can have a look at Jsop if you need to parse HTML, but to run plain GET/POST requests, go for RestTemplate (in Spring) or JAX-RS/Jersey 2 Client.
You might consider implementing a cache layer to increase performance if the result of the query is maintained over the time, or you can assume that in, let's say 5 minutes, the response will be the same to the same query.
You can create your app in your favourite language/framework. I'd recommend to start with SpringBoot + MVC + DevTools. That'd contain all you need + Jsoup if you need to parse some HTML. Later on you can add the cache provider if needed.

We do something similar to access web banking on behalf of a user, scrape his account data and obtain a credit score. In most cases, we have managed to reverse-engineer mobile apps and sniff traffic to use undocumented APIs. In others, we have to fall back to web scraping.
You can have two other types of applications to scrape:
Data is essentially the same for any user, like product listings in Amazon
Data is specific to each user, like in a banking app.
In the firs case, you could have your scraper running and populating a local database and use your local data to provide the web service. In the later case, you cannot do that and you need to scrape the site on user's request.
I understand from your explanation that you are in this later case.
When web scraping you can find really difficult web apps:
Some may require you to send data from previous requests to the next
Others render most data on the client with JavaScript
If any of these two is your case, Selenium will make your implementation easier though not performant.
Implementing the first without selenium will require you to do lots of trial an error to get the thing working because you will be simulating the requests and you will need to know what data is expected from the client. Whereas if you use selenium you will be executing the same interactions that you do with the browser and hence sending the expected data.
Implementing the second case requires your scraper to support JavaScript. AFAIK best support is provided by selenium. HtmlUnit claims to provide fair support, and I think JSoup provides no support to JavaScript.
Finally, if your solution takes too much time you can mitigate the problem providing your web service with a notification mechanism, similar to Webhooks or Resthooks:
A client of your web service would make a request for data providing a URI they would like to get notified when the results are ready.
Your service would respond immediatly with an id of the request and start scraping the necessary info in the background.
If you use skinny payload model, when the scraping is done, you store the response in your data store with an id identifying the original request. This response will be exposed as a resource.
You would execute an HTTPPOST on the URI provided by the client. In the body of the request you would add the URI of the response resource.
The client can now GET the response resource and because the request and response have the same id, the client can correlate both.

Selenium isn't a best way to consume webservices. Selenium is preferably an automation tool largely used for testing the applications.
Assuming the services are already developed, the first thing we need to do is authenticate user request.
This can be done by adding a HttpHeader with key as "Authorization" and value as "Basic "+ Base64Encode(username+":"+password)
If the user is valid (Users login credentials match with credentials in server) then generate a unique token, store the token in server by mapping with the user Id and
set the same token in the response header or create a cookie containing token.
By doing this we can avoid validating credentials for the following requests form the same user by just looking for the token in the response header or cookie.
If the services are designed to chcek login every time the "Authorization" header needs to be set in request every time when the request is made.

I think it is a lot of overhead using a webdriver but it depends on what you really want to achieve. With the info you provided I would rather go with a restTemplate implementation sending the appropriate http messages to the existing webapp, wrap it with a nice #service layer and build your web service (rest or soap) on top of it.
The authentication is a matter of configuration, you can pack this in a microservice with #EnableOAuth2Sso and your restTemplate bean, thanks to spring boot, will handle the underlining auth part for you.

May be overkill..... But RPA? http://windowsitpro.com/scripting/review-automation-anywhere-enterprise

Designing backend for Mobile app?

I have web application where typical flow is
HTML form submission > MVC Controller(spring) > Services(Fetch data)
Ultimately most of the controllers methods returns html data
Now I need to develop mobile app also. Per mine understanding I need to create new Rest Controllers to return the JSON data.
Mobile app will connect to rest controller whereas web app will continue to connect to web controller already in place. Is that correct ?

This will depend on how your controllers are written.
If your controller actions resemble REST like actions you could use spring mvc content negotiation to return html or json depending on the request.

How to do a SPA using servlets and jsp?

I'm trying to do a single page application using servlets and jsp pages.
For the moment I have the first page, which is simple to do: a servlet that forwards to the corresponding jsp.
How should the implementation look when navigating to the second page?
I guess it should be an ajax call, the servlet would populate the necessary data, but how to display the second page jsp?

JSP is a server side ui technology. A Servlet listens to specific urls and redirects to JSPs pages. The JSP is compiled to a class (another servlet in fact), invoked (data will be added and inline scripts will run) and the output, whith is HTML, is send to the client (browser). To get to a different page its neccessary to query the server (servlet) for another url, resulting in another html page.
To create an SPA you need a client side technology like JavaScript. Your query the server for a single html page. The page, made of HTML and JavaScript, for example, (could even be the output of a single JSP, dont get confused) is send to the client (browser) and the JS is run. This is nomaly backed up by a framework like AngularJS, EmberJS or Backbone. Once the page is set up, the links within the page are anchors (http://example.com/#/mySecondPage), so clicking them will invoke the framework again (Ajax, querying the server for new data), but will stay on the same page. Some contents of the page might then be replaced by new content.

If it is a true SPA then you would just have a single JSP and handle all your functionality (after your initial page load) using Ajax.
Have you looked at using a client-side framework such as Angular to help you with this?
Depending on how rich your SPA is, you could either use the same servlet or multiple servlets to serve each page.
Unless you are doing this for a course or tutorial and have some constraints on how to achieve it, you will very probably save yourself a lot of time if you couple something like Angular with a server-side framework like Spring instead of coding servlets. As a suggestion have a look at Spring with Angular.

In SPA the browser only loads the document once (or a few, once per sub-application), and further communication to the server is done usually via AJAX or Websockets.
I recommend you to model your application as a thin server architecture, that is, a client application running in the browser (HTML, CSS, Javascript) consuming a web service API provided by the server.
The following are some points worth knowing;
Client-side:
Only presentation logic
Represent state by URL hash. This enables bookmarking, hyperlinking and browsing history. Your client app should listen to changes in the URL hash and act in consequence. This technique is called "routing" and it is implemented by all Javascript frameworks.
Client application is packaged server-side such it can be downloaded in a single request (in .html, .jsp, servlet, .jsp + multiple .jspf, ...)
Consumes services provided by the server via AJAX or Websockets
Server:
Offers client application to download
Provides a clean, stateless API to be consumed by the client application, better returning JSON (data) than HTML (presentation logic) (Why is it a bad practice to return generated HTML instead of JSON? Or is it?)
Use a REST or JSON-RPC frameworks to create the API. There is a lot of debate on what to choose (see here or here). In my opinion the only advantage of REST over RPC is that since REST has become a "de facto" standard its interoperability is higher, so my recommendation for SPA applications is using JSON-RPC, because your code is the only client of the API.
There are lots of alternatives for both client and server frameworks.
Javascript: AngularJS, EmberJS or Backbone,...
REST: Spring, Jersey, Restlet,..
JSON-RPC: https://en.wikipedia.org/wiki/JSON-RPC#Implementations
Regarding JSON-RPC, you might want to take a look to Brutusin-RPC, a JEE microframework I have created :)

If you are using an Ajax request, then you need to tell the browser that redirect to the second page. Example:
response.sendRedirect("second_page.jsp");
In your servlet, you need to differentiate a request to the first page, from a request that need to be redirected to the second page. You can use parameters, or session values, for example.
if (request.getParameter("page2") != null) {
response.sendRedirect("second_page.jsp");
} else {
.... // include here the normal logic of your Servlet for page 1
}
Then, you can invoke your servlet with or wihout the parameter page2, to go to page 1 (without parameter), or page 2 (with parameter).

Handling Post Requests

yesterday i started brainstorm for a project of mine and I'm not sure if its the correct approach.
On my website I'm having an (kind of an order form) which sends a post to a target URL, which works with a simple curl php script. The target is an external service (where I have no access no rights, nothing). I only know that I will get a POST with further processed data back from the service, which I have to save into my DB.
in steps:
Users fills out the (order) form and posts data to an external url on my website.
data gets externally handled and after finishing that resents a post.
read incoming post data.
save data into DB.
Success page on my website.
My thoughts were to handle the incoming data with a servlet (spring maven project) but I'm not sure if this is a correct approach. Is there a better why to do this. Or is the first step with the php scripts wrong. thx for any help.

A simplest workflow could be
1. Forward the initial (Order form with values) request to a servlet
2. Invoke a post request using java to an external url inside this servlet (Using Apache http client, or libraries such as HTMLUnit)
3. Once you get the incoming response in your servlet, you can update your database.
If you are using spring, the controller could forward initial request to a business class which will handle this post processing and delegate the database update to respective DAO.

There are a number of suitable ways to handle this, and the decision is largely a matter of preference and what you're familiar with. Spring can handle this sort of job quite well.
Note: Maven is a build system for Java and some other JVM languages. I recommend using it, but it's not part of Spring; what you're probably looking for is Spring MVC.

JSP backend for ExtJS

This question is about a proper architecture using JSP as a controller for ExtJS.
I am fairly new to server side development but I am pretty familiar with ExtJS 4 and getting better with Java and SQL daily.
I am trying to create a JSP controller to write the data from stores in ExtJS. I have MSSQL database and Tomcat running on the server.
I successfully created a JSP (sqlData.jsp) that reads from the database and returns JSON data. I pass a query name to this JSP, it then looks up what the query is from a "query" table (columns: [query_id],[query_name],[query]). It then runs the query and returns the data in a JSON format - this is working fine to get data into ExtJS from a database.
To use this backend set-up I usually configure the store like this:
var store = Ext.create('Ext.data.Store', {
model: 'aModel',
proxy: {
type: 'ajax',
url: 'sqlData.jsp?queryName=aQueryName',
reader: 'json'
},
autoLoad: true
});
Somehow, I need this sqlData.jsp to also handle a store.save() call from the ExtJS framework. Which means the JSP needs to receive a POST request and then do an update based on a pile of JSON data (ExtJS sends read request as GET and write methods like store.save() are POST).
My plan was to add something in the Java to recognize whether it is a POST or GET request. Then, if it is a POST request, I would send it to a different Java method in the JSP to parse the JSON and write it to the database.
Of course I would have to change my "query" table to have another column for update/insert statements linked to the same queryName (i.e.: [query_id],[query_name],[select_query],[update_query]).
Does this backend implementation make any sense?
Anyone else use JSP and ExtJS to achieve this smoother?
I noticed that there is an api config option I can set in my proxy to specify different URLs for the different operations (READ, WRITE, DELETE, etc). Should I make a separate JSP and direct all write requests using this config instead?
Would it be wiser to add a writer: 'json' config on the proxy so that it parses before POSTING? I figured I would have to parse it in the JSP either way so I didn't think I should.
Any pointers will be much appreciated.

since your backend is Java, I would really recommend using Spring 3.0 MVC to code your backend.
JSP is not a good option for the stuff you are doing because:
the functions you write in there are not unit testable.
the functions you write in there are not reusable.
the code you write in JSP are functional in nature, not object oriented, you can't inject the services you need into your JSP.
Spring 3.0 MVC has really good synergy with ExtJS 4, namely the RESTful URL's and content negotiation.
This example shows how to integrate the two things together. http://java.dzone.com/articles/extjs-4-file-upload-spring-mvc

I would skip jsp and just go directly to servlets. i.e. implement the logic in the servlets for both returning json, and handling things like POST, PUT, etc.....
jsps are meant to be views. But in your case, your view layer is its own application running in the client. You only need the data.
The Servlet API puts allows you to handle requests, get the http method, and stream data to the response.

My advice is go with an MVC server side framework. My favorite is Grails which lets you work with JSON objects directly for both input and output. Its also super simple to with grails to read and write data to the database.