User enters the data in the forms by choosing values from the master data drop downs in web layer. Data is populated in the dropdowns based on some logic from the master data tables and we know for sure that its a valid master data as far as web layer is concern. Now my question is, should be check the validity of the master data in service layer again because we want to use service layer for mulitple interfaces (Web User Interface, Web Services, Unit Test Cases etc.) or we should validate the master data in respective interfaces only.
In general we should validate incoming data. Right now, we kow that the data, that is sent to the service layer is valid, just because we trust the web frontend. But you already mentioned that the service layer is not restricted to receiving input from that single trusted source.
And as soon as it provides a 'public interface', it should expect receiving invalid data.
I usually stick to the simple paradigm: a sender or data source should always send in an absolutely correct format and a receiver or data sink should always be prepared to receive invalid data. Nice example for this: web servers and browsers.
Related
I am trying to create a data analytics service that can be used by REST client. REST request needs to send a ANSI SQL queries. Challenge is client/UI team wants to create their own Tableau like drag and drop style user interface where User can see parts of their data and build reports with it. So to bridge the gap between such UI and Analytics service, someone needs to convert form data into SQL Statements (we ruled out json based descriptive rest payload as it doesn't scale for full fledge sql). Datasource Metadata (tables, columns, datatypes etc) is available via REST api to UI so UI can send them back with their API request to some another service which can do SQL transformation. This service can also access API that describe relationships between tables/datasources (join, start-schema etc). so such service has enough information on what kind of SQL queries to form based on user selection. It doesn't need to optimize the query as it will be done by backend data analytics service. Is this seems like a good approach? I wonder UI visualization tool connecting via JDBC to datasources taking same approach. Any other suggestions are welcome.
Also, I can see such middleware service can be a generic framework for development of analytics or cubing visualization tool.
I know what I am asking is somehow weird. There is a web application (which we don't have access to its source code), and we want to expose a few of its features as web services.
I was thinking to use something like Selenium WebDriver, so I simulate web clicks on the application according to the web service request.
I want to know whether this is a better solution or pattern to do this.
I shall mention that the application is written using Java, Spring MVC (it is not SPA) and Spring Security. And there is a CAS server providing SSO.
There are multiple ways to implement it. In my opinion Selenium/PhantomJS is not the best option as if the web is properly designed, you can interact with it only using the provided HTML or even some API rather than needing all the CSS, and execute the javascript async requests. As your page is not SPA it's quite likely that an "API" already exists in form of GET/POST requests and you might be lucky enough that there's no CSRF protection.
First of all, you need to solve the authentication against the CAS. There are multiple types of authentication in oAuth, but you should get an API token that enables you access to the application. This token should be added in form of HTTP Header or Cookie in every single request. Ideally this token shouldn't expire, otherwise you'll need to implement a re-authentication logic in your app.
Once the authentication part is resolved, you'll need quite a lot of patience, open the target website with the web inspector of your preferred web browser and go to the Network panel and execute the actions that you want to run programmatically. There you'll find your request with all the headers and content and the response.
That's what you need to code. There are plenty of libraries to achieve that in Java. You can have a look at Jsop if you need to parse HTML, but to run plain GET/POST requests, go for RestTemplate (in Spring) or JAX-RS/Jersey 2 Client.
You might consider implementing a cache layer to increase performance if the result of the query is maintained over the time, or you can assume that in, let's say 5 minutes, the response will be the same to the same query.
You can create your app in your favourite language/framework. I'd recommend to start with SpringBoot + MVC + DevTools. That'd contain all you need + Jsoup if you need to parse some HTML. Later on you can add the cache provider if needed.
We do something similar to access web banking on behalf of a user, scrape his account data and obtain a credit score. In most cases, we have managed to reverse-engineer mobile apps and sniff traffic to use undocumented APIs. In others, we have to fall back to web scraping.
You can have two other types of applications to scrape:
Data is essentially the same for any user, like product listings in Amazon
Data is specific to each user, like in a banking app.
In the firs case, you could have your scraper running and populating a local database and use your local data to provide the web service. In the later case, you cannot do that and you need to scrape the site on user's request.
I understand from your explanation that you are in this later case.
When web scraping you can find really difficult web apps:
Some may require you to send data from previous requests to the next
Others render most data on the client with JavaScript
If any of these two is your case, Selenium will make your implementation easier though not performant.
Implementing the first without selenium will require you to do lots of trial an error to get the thing working because you will be simulating the requests and you will need to know what data is expected from the client. Whereas if you use selenium you will be executing the same interactions that you do with the browser and hence sending the expected data.
Implementing the second case requires your scraper to support JavaScript. AFAIK best support is provided by selenium. HtmlUnit claims to provide fair support, and I think JSoup provides no support to JavaScript.
Finally, if your solution takes too much time you can mitigate the problem providing your web service with a notification mechanism, similar to Webhooks or Resthooks:
A client of your web service would make a request for data providing a URI they would like to get notified when the results are ready.
Your service would respond immediatly with an id of the request and start scraping the necessary info in the background.
If you use skinny payload model, when the scraping is done, you store the response in your data store with an id identifying the original request. This response will be exposed as a resource.
You would execute an HTTPPOST on the URI provided by the client. In the body of the request you would add the URI of the response resource.
The client can now GET the response resource and because the request and response have the same id, the client can correlate both.
Selenium isn't a best way to consume webservices. Selenium is preferably an automation tool largely used for testing the applications.
Assuming the services are already developed, the first thing we need to do is authenticate user request.
This can be done by adding a HttpHeader with key as "Authorization" and value as "Basic "+ Base64Encode(username+":"+password)
If the user is valid (Users login credentials match with credentials in server) then generate a unique token, store the token in server by mapping with the user Id and
set the same token in the response header or create a cookie containing token.
By doing this we can avoid validating credentials for the following requests form the same user by just looking for the token in the response header or cookie.
If the services are designed to chcek login every time the "Authorization" header needs to be set in request every time when the request is made.
I think it is a lot of overhead using a webdriver but it depends on what you really want to achieve. With the info you provided I would rather go with a restTemplate implementation sending the appropriate http messages to the existing webapp, wrap it with a nice #service layer and build your web service (rest or soap) on top of it.
The authentication is a matter of configuration, you can pack this in a microservice with #EnableOAuth2Sso and your restTemplate bean, thanks to spring boot, will handle the underlining auth part for you.
May be overkill..... But RPA? http://windowsitpro.com/scripting/review-automation-anywhere-enterprise
We are now incorporating Restful Web Services in our System and as a result, both Browser and Mobile Clients are able to access our System available in single form (Clients access same code; No different backend code for different clients) successfully via Restful Web Service (i.e.)
REQUEST:
Technically, Restful Web Service API has been built above our existing system (Specifically built above Controller of MVC). Mobile Clients (be it Android (or) iOS (or) Windows) and Browser Client (Web forms) make uses the Restful API built, for ACCESSING our system and transmitting DATA to our system.
RESPONSE:
So, once we get data inside our system (Controller), it gets processed by moving back and forth from DAO and business layer and finally the processed data lands in Controller for it to be get returned to respective client. So, at this place, currently we determine the type of client from where the request is made (Mobile App or Browser) and
RETURN only processed DATA
IF
Client is 'Mobile App' (or)
RETURN both processed DATA + Name of Web View
ELSE
Client is 'Browser'
Note1: We are returning javax.ws.rs.core.Response object in both the cases.
Note2: processed DATA can be PlainText (or) XML (or) JSON (or) String
TO CLARIFY:
Now our question is, whether can we eliminate this IF-ELSE condition for returning the stuff (i.e.) is there a way to have only one RETURN statement returning generic stuff, irrespective of Client to which the stuff is returned? If so, then how does different Clients (in our case, Mobile App and Browser) interpretes the returned stuff and renders view to us?
Recently I been reading about SOA, the service oriented architecture is based on the idea of exposing web services.
This Architecture also tells you that the services you create have to be reusable, in other words you have to make universal services.
Now, I have to develop web services for one of my applications, I will use Jersey to develop the web services. I would like to develop web services as I've been reading.
For example, in the login I need to expose a web service. As we can see in the image, the login WebService will receive the id_user and it will response with a status and the password if it exits.
The problem is that this web services is not a reusable web service, because it only works for logIn and if, for example, I wanted to check an user's name I would need to create another webservice that will bring me the user's name.
Following what I already now, I change the login web service as follows
Now, if I need to logIn, I will use this webServices and just use the password, if I want to load his information I also will use this same web service and use what I need.
With this new web service I just can search using the user_id, so I also change the webService as follow
I edited the request, now you will need to send the option 1 if you want look for user_id or 2 if you want look for user_name.
I'm wondering if this last change was correct?
I also have another table called bitacora and I need to make a webService for insert and update.
At first I insert data with default values, because the table has foreign key.
For example I will fill my table like follow (1,1,1,0,0,0,0,0,0,0,0) where 0 is the default value for foreign key.
After that I have to update that table changing just a few columns (1,1,1,0,0,1,1,1,0,0,0)
After, I will complete updating my table (1,1,1,1,1,1,1,1,1,1,1).
How can I make a good webService for that need?
As I see, I will need to make 1 web service, and send him in the request the information that I want to update, and an option so I can choose in my web services what to do. (make update 1 or make update 2 or make update 3).
But I think I'm lost in this one.
I am developer of internet payment gateway solution in Java, and sometimes I wonder if I could/should have done something better.
Currently system works as follows:
System is receiving requests in form of html POST request to, lets say, confirm.xhtml
There is some filter mappings for confirm.xhtml. In doFilter method there are some validations of request. ie. is merchant active, are required parameters present in POST parameters and so on.
If request is valid, html page is displayed. This is very simple html page which consists of one form with fields like card number, card exp. date, customer name, surname, etc.
When customer fills the form, message for authorizing host is created and sent via tcp socket connection (proprietary protocol).
While system waits response from authorizing host, loading page is displayed to customer.
When response from host is received, customer is redirected back to merchant site. This is html POST request with parameters like response message, approval code, and some other data relevant to specific implementation.
This is very general description of current system, but I would like to hear how would You do it (or already did it). Specifically, would you use some framework or would that be overkill because this is only one html page? ORM or plain JDBC? What about session management, security, client side or server side validations ... ?
I am aware that this question is very general, but I want to hear how would you approach to designing of such a system from the ground up.
Frameworks are supposed to be a tool to help you. Why reinvent the wheel? I don't think it would be an overkill depending on how you use it, but it's really up to you as the developer whether you want to use it or not, depending on your project.
What about session management, security, client side or server side validations ... ? I would use Enterprise JavaBeans to facilitate handling those.