I am developer of internet payment gateway solution in Java, and sometimes I wonder if I could/should have done something better.
Currently system works as follows:
System is receiving requests in form of html POST request to, lets say, confirm.xhtml
There is some filter mappings for confirm.xhtml. In doFilter method there are some validations of request. ie. is merchant active, are required parameters present in POST parameters and so on.
If request is valid, html page is displayed. This is very simple html page which consists of one form with fields like card number, card exp. date, customer name, surname, etc.
When customer fills the form, message for authorizing host is created and sent via tcp socket connection (proprietary protocol).
While system waits response from authorizing host, loading page is displayed to customer.
When response from host is received, customer is redirected back to merchant site. This is html POST request with parameters like response message, approval code, and some other data relevant to specific implementation.
This is very general description of current system, but I would like to hear how would You do it (or already did it). Specifically, would you use some framework or would that be overkill because this is only one html page? ORM or plain JDBC? What about session management, security, client side or server side validations ... ?
I am aware that this question is very general, but I want to hear how would you approach to designing of such a system from the ground up.
Frameworks are supposed to be a tool to help you. Why reinvent the wheel? I don't think it would be an overkill depending on how you use it, but it's really up to you as the developer whether you want to use it or not, depending on your project.
What about session management, security, client side or server side validations ... ? I would use Enterprise JavaBeans to facilitate handling those.
Related
I know what I am asking is somehow weird. There is a web application (which we don't have access to its source code), and we want to expose a few of its features as web services.
I was thinking to use something like Selenium WebDriver, so I simulate web clicks on the application according to the web service request.
I want to know whether this is a better solution or pattern to do this.
I shall mention that the application is written using Java, Spring MVC (it is not SPA) and Spring Security. And there is a CAS server providing SSO.
There are multiple ways to implement it. In my opinion Selenium/PhantomJS is not the best option as if the web is properly designed, you can interact with it only using the provided HTML or even some API rather than needing all the CSS, and execute the javascript async requests. As your page is not SPA it's quite likely that an "API" already exists in form of GET/POST requests and you might be lucky enough that there's no CSRF protection.
First of all, you need to solve the authentication against the CAS. There are multiple types of authentication in oAuth, but you should get an API token that enables you access to the application. This token should be added in form of HTTP Header or Cookie in every single request. Ideally this token shouldn't expire, otherwise you'll need to implement a re-authentication logic in your app.
Once the authentication part is resolved, you'll need quite a lot of patience, open the target website with the web inspector of your preferred web browser and go to the Network panel and execute the actions that you want to run programmatically. There you'll find your request with all the headers and content and the response.
That's what you need to code. There are plenty of libraries to achieve that in Java. You can have a look at Jsop if you need to parse HTML, but to run plain GET/POST requests, go for RestTemplate (in Spring) or JAX-RS/Jersey 2 Client.
You might consider implementing a cache layer to increase performance if the result of the query is maintained over the time, or you can assume that in, let's say 5 minutes, the response will be the same to the same query.
You can create your app in your favourite language/framework. I'd recommend to start with SpringBoot + MVC + DevTools. That'd contain all you need + Jsoup if you need to parse some HTML. Later on you can add the cache provider if needed.
We do something similar to access web banking on behalf of a user, scrape his account data and obtain a credit score. In most cases, we have managed to reverse-engineer mobile apps and sniff traffic to use undocumented APIs. In others, we have to fall back to web scraping.
You can have two other types of applications to scrape:
Data is essentially the same for any user, like product listings in Amazon
Data is specific to each user, like in a banking app.
In the firs case, you could have your scraper running and populating a local database and use your local data to provide the web service. In the later case, you cannot do that and you need to scrape the site on user's request.
I understand from your explanation that you are in this later case.
When web scraping you can find really difficult web apps:
Some may require you to send data from previous requests to the next
Others render most data on the client with JavaScript
If any of these two is your case, Selenium will make your implementation easier though not performant.
Implementing the first without selenium will require you to do lots of trial an error to get the thing working because you will be simulating the requests and you will need to know what data is expected from the client. Whereas if you use selenium you will be executing the same interactions that you do with the browser and hence sending the expected data.
Implementing the second case requires your scraper to support JavaScript. AFAIK best support is provided by selenium. HtmlUnit claims to provide fair support, and I think JSoup provides no support to JavaScript.
Finally, if your solution takes too much time you can mitigate the problem providing your web service with a notification mechanism, similar to Webhooks or Resthooks:
A client of your web service would make a request for data providing a URI they would like to get notified when the results are ready.
Your service would respond immediatly with an id of the request and start scraping the necessary info in the background.
If you use skinny payload model, when the scraping is done, you store the response in your data store with an id identifying the original request. This response will be exposed as a resource.
You would execute an HTTPPOST on the URI provided by the client. In the body of the request you would add the URI of the response resource.
The client can now GET the response resource and because the request and response have the same id, the client can correlate both.
Selenium isn't a best way to consume webservices. Selenium is preferably an automation tool largely used for testing the applications.
Assuming the services are already developed, the first thing we need to do is authenticate user request.
This can be done by adding a HttpHeader with key as "Authorization" and value as "Basic "+ Base64Encode(username+":"+password)
If the user is valid (Users login credentials match with credentials in server) then generate a unique token, store the token in server by mapping with the user Id and
set the same token in the response header or create a cookie containing token.
By doing this we can avoid validating credentials for the following requests form the same user by just looking for the token in the response header or cookie.
If the services are designed to chcek login every time the "Authorization" header needs to be set in request every time when the request is made.
I think it is a lot of overhead using a webdriver but it depends on what you really want to achieve. With the info you provided I would rather go with a restTemplate implementation sending the appropriate http messages to the existing webapp, wrap it with a nice #service layer and build your web service (rest or soap) on top of it.
The authentication is a matter of configuration, you can pack this in a microservice with #EnableOAuth2Sso and your restTemplate bean, thanks to spring boot, will handle the underlining auth part for you.
May be overkill..... But RPA? http://windowsitpro.com/scripting/review-automation-anywhere-enterprise
My current system architecture for a web application looks like above. Essentially, its just one code base, that is being deployed in different contexts, so for instance, app1.localhost.com, app2.localhost.com.
These are my current challenges -
I need query my webservices using ajax calls from user browser. But, somehow I need to tell the webservices, that the request is coming from user1 for app1 or user2 or app2. Accordingly, the webservice can go query the right schema in the database and return the results.
My webservices need to be stateless.
I cannot have the user tamper with the request from the browser.
Is this a flawed model? Are my expectations out of standard way of doing this? What is the best way to realize this architecture?
Let's go through this step by step.
So your webservice needs to know the application and the user.
Easy: just include it in the request as a parameter or as part of the url.
Same goes for some information possibly provided by the user.
The challenge comes with the requirement that the user must not be able to change the request.
This is typically achived by cryptographic signing the request (or the sensitive parts). I kind of assume that replay attacks are also an issue.
Create a certificate for the apps.
On the application server create a nonce.
Sign the nonce, the application name and the user name using the certificate.
Include nonce, servername, username and signature in the request used for the ajax call.
Check nonce, servername, username and signature match in the webservice
Check also that the nonce wasn't used before.
Checking the nonce does require some state, but just a list of nonces. If you make the nonces increasing you can even discard any nonces much smaller than the last one you received, limiting the amount of state even more.
You should check if something like this exists as a ready made protocol, because it is always a bad idea to create your own security relevant protocols.
I'm building an app to let users export data from a university system. Currently, they can log in and see the data in HTML, but I would like to let people download it as CSV.
I have an app where users supply their username and password. I would like to log in to the university system and HTML scrape the resulting page. How can I do this?
I'm building a GWT app. I could either do this in Java-transliterated-JS on the client, or Java on the server.
Update: Selenium might be nice, but it looks like overkill.
You're going to have to do this from the server unless the domains are the same. You'd need to determine what the POST transaction used by the other server for the login step looks like - parameter names etc. Then you'd perform that operation and do whatever you want with what comes back. If you need to see multiple pages, you need to maintain the appropriate session cookie too so that the server knows you're still logged in on the subsequent HTTP requests.
If you have to hit another site to validate the credentials, then I'm not so sure that people should feel comfortable providing those credentials to you. That is, if you don't have rights to check the credentials directly, why are you trustworthy to receive them? I know sometimes people need to integrate with a system they don't own, so this is just a question.
First, this has to be done server-side because of the limitations on client scripting due to the same origin policy.
The typical way of handling the "screen scraping" you mention is to treat the web page as if it was an XML service. First, examine the source code of the page, then using an internet/HTTP stack, craft a POST to the correct URL and read the response using a standard XML library. It will take some ingenuity to come up with a good way to dig into the XML to find the piece you need that will be as insulated as possible from changes to the page. Keep in mind that your system can break any time that the owners of the site change their page.
Sometimes, you can't just send the POST but have to request the blank page initially in order to get hidden form values that need to be returned in the POST. You'll have to experiment to find out what it requires.
Additionally, you probably have to handle cookies as well, since they usually are an integral part of the web site's authentication and session management (though you might get lucky that the session doesn't matter between the initial POST and the first response).
Last, you may be unlucky enough that the site uses javascript to do part of the authentication work, which may require additional digging to understand how the credentials are posted to the site.
There are other potential barriers such as the site checking to see that the referrer is their own site, possible use of SSL (HTTPS) and so on.
I'm pretty sure that the protection against cross-site scripting in web browsers will mean that you can't log in to the university's app using javascript running in the web browser. So the part of your program that fetches data from the university will need to run on your server. Once you have the data, you can process it either on your server or in javascript in the browser, but I think it would be easier to do it on the server.
See http://en.wikipedia.org/wiki/Same_origin_policy
I'm not too sure about GWT, but in general, you would take the form data submitted by the user, check it against a database of username and hashed passwords. If the database checks out, set a session cookie that says the user is logged in.
In your pages, check if the session cookie say the user is logged in. If not, redirect to login page, otherwise allow them to view the pagfe.
I'm running a transactional web site where client pay for goods. As of now, I'm using my payment processor (pp) company web page to handle the payment, i.e. the client is redirected to the PP server and redirected back to my site when the payment done. A big disadvantage is that I can't customize the PP page, or very little.
Now, I want to handle the payment directly on my web site. I just need to post all the payment data to my payment processor an it returns the response. I already know that I need to be PCI compliant. My website is done using AJAX calling RESTeasy WebServices and a Java server.
Now my question is, should I validate every fields on the client side, using javascript, or swing everything on the server and do the job there? Should I POST to the payment processor directly using a FORM on the client or using an HttpUrlConnection from the server? What is the benchmark on the market for doing this? I found a lot of PHP example but unfortunatly a don't have time to learn PHP.
Thanks
Whether you want to POST data directly to the payment processor or collect the credit card information depends on your payment processor and if you want to store some of the information you collect on the credit card page (don't store the credit card information; store only the last four digits if you need to.)
Posting the form directly to the payment processor is the simplest way. It probably also means that you don't even need to be PCI level 4 compliant, because none of the credit card information ever passes through your system. See here:
http://www.pcicomplianceguide.org/pcifaqs.php#2
On the other hand, if you want to store the billing address or the last four digits of the credit card number - or to insert this data into a receipt that you want to send by e-mail - you will need to write your own server-side code that collects the information from the credit card page and POSTs the data to the payment processor using an HttpUrlConnection. If your payment processor offers an API, you could also use that.
In either case, you should make sure that your payment page can only be viewed over SSL. In a Java web application, you can use a transport-guarantee entry in web.xml to do this:
<security-constraint>
<user-data-constraint>
<transport-guarantee>CONFIDENTIAL</transport-guarantee>
</user-data-constraint>
<web-resource-collection>
<url-pattern>your_payment_page.jsp</url-pattern>
</web-resource-collection>
</security-constraint>
This will make sure that even if a user accesses your payment page on plain HTTP, the application server will redirect to HTTPS.
This page (from a payment processor I've dealt with) has some more information, from a processor point of view:
http://wiki.usaepay.com/developer/transactionapi
As pointed out, you cannot validate only on the client because people can take your client code and modify it. In addition, you'll need to provide account information, which you wouldn't want to be visible to users. Your server must validate and process the request.
This is a fairly complex task. You need to ensure the page is accessed through SSL (HTTPS), and you need to securely access your credit card processor.
You can perform some basic validation from your server-side code. Check out http://www.blackbeltcoder.com/Articles/ecommerce/validating-credit-card-numbers.
Beyond that, it's really up to your processor and the API they make available to you. You'll need to write code that conforms to their specification.
Thanks to all of you for your answers. Finally, it's far more complicated, and it's a too big responsibility to take care of collecting the payment data myself just for the purpose of web page design. I will still modify my payment page, informing the users that they will be redirected on the payment processor web page to complete the payment. Kind of an hybrid of what I have now. This is safer for them and for me.
Thanks again.
This question is more towards Design and Architecture and I want to know SO Readers think on my scenario.
I have a requirement where in my Application should provide other application interface when the user logs in to my application.
For example, lets say my application is www.gmail.com and other application is www.stackoverflow.com so what am trying to accomplish is that when the user log's in gmail account he should see his home page of stackoverflow and a particular questions.
From technology point of view, we have to use Java and so am not sure of what design and architecture consideration would go in to implement the requirement.
One Approach, am thinking on is that when the user logs in to gmail than I will populate the request object with all the login credential parameters for stackoverflow website and also question_id which would be passed in as parameter and then on Stackoverflow side, I would parse the request object and authenticate the user credentials and depending upon request parameter, I would render the question_id which I received from request.
I want to know what would be best approach and issues encountered in designing such an system.
Edit
After seeing all the answer, I would like to add little update to my question. What I am looking for is to get the feel of issues and challenges what I would have to face while trying to accomplish my task, also I am using Java and am not sure how can I accomplish my goal using Java as we do not have something like OLE which we have in Microsoft Technology stack to achieve the task.
Hope I am making some sense here.
I can think of three ways you could solve this.
Implement single sing-on. You log-in to all enterprise applications, and once logged all of them use the same authentication credentials (I think this is the best option. you don't need a full-fledge SSO, at least for these two application you could use the same credential validation mechanism)
You could also do what your are proposing creating the authentication credential for the user (i.e a cookie) and then do a redirect. Keep in mind that both application will need to be in the same sub-domain in order to work.
As mentioned before, you could also expose through your application the data/services you want to consume from the other application.
In my company we have what we call "Graphical Services", which are managed by a central server which also do credential validation, if the credentials are right it display a user interface for the user (generally in a Pop-up or an iframe).
Hope it helps.
You can't definitely do that at client side or java script as it will lead to cross site scripting issues. Or you can use iframes (which isdeprecated).
The other way of doing it would be to have your own interface/UI for the application and use only the service layer from your back end (java/j2ee in your case) which you may end up duplicating all the front end again (on the positive side, you will get your own branding of the site).
Regarding credentialing all most all the sites now used "OAuth" or similar and it should not be that difficult for authorizing
If both applications are web-based in-house applications, you could write a master login component, independent of either application, that will perform the user authentication, load any useful data it can at login time, and send the user's browser to the correct URL, making sure to pass any relevant information to the target app (as part of the forwarding request or behind the scenes in some distributed shared memory). Just a thought.