Im working on integration with some rest API and i need to make calls to their URLS to receive the data.
Im just wondering if its possible to use a REST web-service which will be mapped to that certain URL instead of the local one and later on I will write the client side that will be mapped to these calls.
for example:
#Path("/URL")
public class MessageRestService {
#GET
#Path("/{param}")
public Response printMessage(#PathParam("param") String msg) {
String result = "Restful example : " + msg;
return Response.status(200).entity(result).build();
}
}
I cant make straight API calls from client side for example using AngularJs because i get this error:
Response to preflight request doesn't pass access control check: No 'Access- Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:63342' is therefore not allowed access. The response had HTTP status code 400.
I did find code samples for straight API calls to URLS from java, but it looks messy especially when you have to create it for a lot of API calls:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
public class Connection {
public static void main(String[] args) {
try {
URL url = new URL("INSERT URL HERE");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "application/json");
String messageToPost = "POST";
OutputStream os = conn.getOutputStream();
os.write(input.getBytes());
os.flush();
conn.connect();
BufferedReader br = new BufferedReader(new InputStreamReader(
(conn.getInputStream())));
String output;
System.out.println("Output from Server .... \n");
while ((output = br.readLine()) != null) {
System.out.println(output);
}
conn.disconnect();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
You are facing a same origin policy issue.
This is because your client-side (web browser) application is fetched from Server-A, while it tries to interact with data on Server-B.
Server-A is wherever you application is fetched from (before it is displayed to the user on their web browser).
Server-B is localhost, where your mock service is deployed to
For security reasons, by default, only code originating from Server-B can talk to Server-B (over-simplifying a little bit). This is meant to prevent malicious code from Server-A to hijack a legal application from Server-B and trick it into manipulating data on Server-B, behind the user's back.
To overcome this, if a legal application from Server-A needs to talk to Server-B, Server-B must explicitly allow it. For this you need to to implement CORS (Cross Origin Resource Sharing) - Try googling this, you will find plenty of resources that explain how to do it. https://www.html5rocks.com/en/tutorials/cors/ is also a great starting point.
However, as your Server-B/localhost service is just a mock service used during development and test, if your application is simple enough, you may get away with the mock service simply adding the following HTTP headers to all its responses:
Access-Control-Allow-Origin:*
Access-Control-Allow-Headers:Keep-Alive,User-Agent,Content-Type,Accept [enhance with whatever you use in you app]
As an alternative solution (during dev/tests only!) you may try forcing the web browser to disregard the same origin policy (eg: --disable-web-security for Chrome) - but this is dangerous if you do not pay attention to use separate instances of the web browser for your tests and for you regular web browsing.
Related
I want to open a website in web browser. I know it is easy but i want to do it in different way ...
It is like proxy server .I have made a java code that will get content(source code) of webpage and when browser request localhost on particular port number this code writes source code in browser. But instead of getting web page I am getting source code of webpage in browser and also i want to make a request from java code as a illusion of browser means server should feel that that request is made from a browser and not from java console.
import java.net.*;
import java.io.*;
public class URLConnectionReader {
public static void main(String args[]) throws Exception{
URL ul = null;
HttpURLConnection ulc = null;
ServerSocket server = null;
Socket client = null;
DataInputStream in = null;
DataOutputStream out = null;
String c = null;
server = new ServerSocket(9898);
System.out.println("Server is waiting for clients on port no 9898....");
while(client == null){
client = server.accept();
}
System.out.println("Connected.....");
out = new DataOutputStream(client.getOutputStream());
ul = new URL("http://www.google.com");
ulc = (HttpURLConnection)ul.openConnection();
in = new DataInputStream(ulc.getInputStream());
while((c = in.readLine())!=null){
out.writeBytes(c);
}
in.close();
out.close();
client.close();
}
}
Loading web pages is not quite as simple as you probably think. Both the browser and the server use a protocol called HTTP. In simple terms, the browser sends a request consisting of a request line, headers and sometimes data, and the server responds with a response line, headers and data. Most web pages also have related resources that need to be loaded for displaying the page (such as images, stylesheets and scripts), and each resource is loaded through a separate request.
Your program only accepts one request, completely ignores the details of the request, and then loads a fixed web page and sends it as the response. The way you are loading the web page (with a URL), you are only getting the data part of the response (the page source); the response line and the headers are missing. The headers are very important as one of them (named "Content-Type") specifies what kind of resource it is - web page, image or something else. Without it, browsers usually assume the data is plain text and display it accordingly.
So if you want your experiment to work better, you need to make sure you send a complete and valid HTTP response to the browser. You can probably reconstruct the response line and headers from the HttpURLConnection object. Or you can use sockets directly to load the web page.
A better solution would be to use a java web server (such as Jetty) in which you'd run a servlet that loads the remote page using an HTTP client library (such as Apache HttpComponents) and does the necessary processing of addresses and headers. But.. small steps :)
I'm trying to fetch a CSV-formatted webpage to use as a rudimentary database. The test page is at http://prog.bhstudios.org/bhmi/database/get, and browsers open it no problem. However, when I run the following code, Java throws a 403 error:
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Main
{
static
{
Logger.getGlobal().setLevel(Level.ALL);
}
/**
* #param args the command line arguments
*/
public static void main(String[] args) throws IOException
{
InputStream is = null;
try
{
System.out.println("Starting...");
URL url = new URL("http://prog.bhstudios.org/prog/bhmi/database/get/");
URLConnection urlc = url.openConnection();
urlc.connect();
is = urlc.getInputStream();
int data;
while ((data = is.read()) != -1)
{
System.out.print((char)data);
}
System.out.println("\r\nSuccess!");
}
catch (IOException ex)
{
Logger.getGlobal().log(Level.SEVERE, ex.getMessage(), ex);
System.out.println("\r\nFailure!");
}
if (is != null)
is.close();
}
}
Here's the console output:
Starting...
Nov 18, 2013 3:01:48 PM org.bh.mi.Main main
SEVERE: Server returned HTTP response code: 403 for URL: http://prog.bhstudios.org/prog/bhmi/database/get/
java.io.IOException: Server returned HTTP response code: 403 for URL: http://prog.bhstudios.org/prog/bhmi/database/get/
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626)
at org.bh.mi.Main.main(Main.java:36)
Failure!
Note that 403 means the server is on and properly accepted the request, but refuses to do anything further.
Now here's the kicker: If I get, say, http://example.com, it works just fine!
How can I get my Java app to read this file from my webserver?
I tested against your server and if I submit the request - using TamperData - with User-Agent: Java/1.6.0_14 (I just picked a random java version), your webserver responds with 403 Forbidden.
My browser shows the following error message:
Error 1010
Access denied
What happened?
The owner of this website (prog.bhstudios.org) has banned your access based on your browser's signature (cf7ab9f58210755-ua21).
In other words, your server (or more likely: your proxy, as the headers both indicate use of cloadflare-nginx and ASP.net) filters based on user agent strings. This is probably done to prevent bots and screenscrapers from accessing your websites.
You either need to drop this filter (ask your proxy adminstrator), or set a different user agent for URLConnection, see Setting user agent of a java URLConnection and How to modify the header of a HttpUrlConnection
Your server for some reason is configured to forbid access when the request header
User-Agent: Java/...
is present. I was able to reproduce the problem and also got it to work by doing
URLConnection urlc = url.openConnection();
urlc.setRequestProperty("User-Agent", "");
urlc.connect();
Client Side
I have a java application that connects to a remote server using basic POST or GET methods:
URL url = new URL(urlStr);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setAllowUserInteraction(false);
conn.setRequestProperty("Content-type", "text/xml; charset=" + ENCODING);
conn.connect();
conn.getOutputStream().write(data.getBytes(ENCODING));
conn.getOutputStream().close();
(I cannot change this code, the only things I can change is the urlStr and the data sent to the server when calling the method).
[EDIT] : The client can be a java client or any other client (c++, objective-c, ..). The point here is that I can only access what's in the body of my post as well as the URL.
Server Side
On my server side, I would like to implement Spring Security (SecurityContext and session persistance).
I understand that spring security is based on the browser's cookies when it's a WebApp to hold the information about the session id. But in my case there's no Browser.
Do I need to simulate the storage of the JSESSIONID and send it back to the server? I'm not sure this is possible since I would need to call conn.addRequestProperty(key, value) which is not possible.
Is there any other way?
Thank you.
[EDIT]
as pointed out by #zagyi, I can use the URL to pass session token to Spring, but I still can't figure out how.
Passing the jsessionid in the url is just a matter of appending it at the end of the url like this:
http://localhost:8080/example/auth/login;jsessionid=A06F00609BBA8A4C2B005FB25F90C4C9
You can see this in working if you configure a browser not to accept any cookies, in which case the server automatically includes the session id in the url (assuming a default tomcat configuration). This topic is also discussed in this question.
There may be a client-side solution for that.
The action point where we can interact is here:
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
We will provide an own (wrapped) HttpURLConnection, which will handle the JSESSIONID. But unfortunately we have to start a bit further.
The trick is that we register a new protocol, e.g. "xhttp", that we use to wrap a real "http" protocol connection. So, your URL will look like:
xhttp://www.example.com/...
First, define a URLStreamHandlerFactory class
public class MyURLStreamHandlerFactory implements URLStreamHandlerFactory {
public URLStreamHandler createURLStreamHandler(String protocol) {
if ("xhttp".equals(protocol)) {
return new MyURLStreamHandler();
}
return null;
}
}
At Java (or application) init time we can set it. You can do it only once per JVM.
URLStreamHandlerFactory fac = new MyURLStreamHandlerFactory();
URL.setURLStreamHandlerFactory(fac);
So, let's go ahead with MyURLStreamHandler.
public class MyURLStreamHandler extends URLStreamHandler {
#Override
protected URLConnection openConnection(URL url) throws IOException {
return new MyHttpURLConnection(url);
}
}
This is quite simple, we create our own connection. Let's do the dirty stuff:
public final class MyHttpURLConnection extends HttpURLConnection {
private HttpURLConnection conn;
public MyHttpURLConnection(URL url) throws MalformedURLException, IOException {
super(url);
String newUrlString = url.toExternalForm().substring(1);
conn = (HttpURLConnection) new URL(newUrlString).openConnection();
}
#Override
public void disconnect() {
conn.disconnect();
}
#Override
public boolean usingProxy() {
return false;
}
#Override
public void connect() throws IOException {
conn.connect();
conn.setRequestProperty("JSESSIONID", "X");
}
}
And voilá, we managed to access our connection, and set the JSESSIONID header.
All you need is to compile your classes, add the class files to the client jar, and make the init code running some way in the same JVM where the above code runs.
If you cannot do it, there is another possibility: set the following system parameter to the client Java application:
-Djava.protocol.handler.pkgs=com.example.myprotocol
In this case create a com.example.myprotocol.xhttp (xhttp like your protocol name), and rename our MyURLStreamHandler class to com.example.myprotocol.xhttp.Handler. This is the fixed name where the protocol resolver will look for it. Note, that this java.protocol.handler.pkgs property is checked by the security manager.
I am trying out a simple program for reading the HTML content from a given URL. The URL I am trying in this case doesn't require any cookie/username/password, but still I am getting a io.IOException: Server returned HTTP response code: 403 error. Can anyone tell me what am I doing wrong here? (I know there are similar question in SO, but they didn't help):
import java.net.*;
import java.io.*;
import java.net.MalformedURLException;
import java.io.IOException;
public class urlcont {
public static void main(String[] args) {
try {
URL u = new URL("http://www.amnesty.org/");
URLConnection uc = u.openConnection();
uc.addRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");
uc.connect();
InputStream in = uc.getInputStream();
int b;
File f = new File("C:\\Users\\kausta\\Desktop\\urlcont.txt");
f.createNewFile();
OutputStream s = new FileOutputStream(f);
while ((b = in.read()) != -1) {
s.write(b);
}
}
catch (MalformedURLException e) {System.err.println(e);}
catch (IOException e) {System.err.println(e);}
}
}
If you can fetch the URL in a browser, but not via Java, that indicates, to me, that they are blocking programmatic access to the page via user-agent filtering. Try setting the user-agent on your connection so that your code appears, to the webserver, to be a web-browser.
See this thread for help on that: What is the proper way of setting headers in a URLConnection?
There is a permission problem:
A web server may return a 403 Forbidden HTTP status code in response to a request from a client for a web page or resource to indicate that the server refuses to allow the requested action
you are not doing anything "wrong", the server you are trying to access is blocking your request, as you are not allowed to access the file
Http-Error 403 means Forbidden --> the remote server blocks the request.
check if you need to give authentification to access the document you want and in that case provide it with the request ;)
I am currently pen-testing a web application and came across an interesting phenomenon. During my testing sessions, I gathered URLs using a proxy. Now I wanted to test my URL list for anonymous access, so i wrote this little tool
public static void main(String[] args) {
try {
TrustAllCerts.disableCertChecks();
FileReader fr = new FileReader(new File("urls.txt"));
BufferedReader br = new BufferedReader(fr);
String urlStr = br.readLine();
while (urlStr != null) {
if (urlStr.trim().length() > 0) {
URL url = new URL(urlStr);
HttpsURLConnection urlc = (HttpsURLConnection) url.openConnection();
urlc.connect();
if (urlc.getResponseCode() == HttpURLConnection.HTTP_OK) {
System.out.println(urlStr);
} else {
System.out.println("["+urlc.getResponseCode()+"] "+urlStr);
}
urlc.disconnect();
}
urlStr = br.readLine();
}
br.close();
} catch (Exception e) {
e.printStackTrace();
}
}
It does basically nothing, but opening an URL connection on a given URL and test the HTTP response code (actually I implemented some more tests, if I'm getting redirected to a login page). However, the problem is, that this specific application (some custom MS SQL Server Reporting Services) is configured to use NTLM WWW authentication. If I try to access some of the URLs using Firefox, i get an 401 Unauthorized + login dlg. Internet Exploder performs NTLM auth in the background and grants access. It seems that the Java URLConnection (or URL) class does the same, because I am getting no 401. Is there a way to disable implicit NTLM authentication in Java? This is a bad pitfall for me.
I think the Java Network Documentation is the best resource. Setting the http.auth.preference="basic" should get you what you want. Assuming you don't need digest or something else. I'm not sure if you can go beyond that to disable NTLM.
Another thing to consider is other Java HTTP client implementations, like Apache's or Google's.
I'm not sure that this will help, but I've been stumped by the opposite.
I wanted NTLM auth to take place, so on my local machine I use a free app called CNTLM. It's a local proxy server that will forward (and NT auth) incoming requests. Good for apps that can't use NTLM proxies.
I'm sorry, I know this isn't answering your question, but maybe it proves helpful to someone out there! :)