Develop Reference

Connecting to Azure storage account thru proxy server Microsoft Azure Storage SDK for Java

In our project we need to access the Blob Storage through a Proxy Server (squid).
We are planning to use the Microsoft Azure Storage SDK for Java version 2.2.0.
But it looks like setting the proxy is not provided by the API.
The only way I could make it go through the proxy is by setting the System properties
System.setProperty("http.proxyHost", "127.0.0.1");
System.setProperty("http.proxyPort", "3128");
But this affect all services that are running on my JVM which harms other services that not supposed to go via the Proxy.
Looking at the java code it looks like
com.microsoft.azure.storage.core.BaseRequest.createURLConnection(URI, RequestOptions, UriQueryBuilder, OperationContext). Is calling java.net.URL.openConnection() without proxy.
While using java.net.URL.openConnection(Proxy) could provide the required support?
It looks wired to me that this is not supported?
Do I miss something here?
UPDATE: I opened an issue on this in azure-storage-java git, I would be happy to get your input as I want to suggest a pull request for this.

So far there have been no Java SDK API support access directly Azure Storage through proxy server, because BaseRequest Class miss "url.openConnection(proxy)" in the function "public static HttpConnection createURLConnection(...)".
Per my experience, there are two ways to help you implement the access function.
The one is that you can use Azure Storage REST API through the java.net.Proxy Class to access storage service.
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(host, port));
URLConnection conn = url.openConnection(proxy);
And if you should be authorize proxy user & password, you can do it as the follows:
//Proxy-Authorization: Basic <Base64.encode(user:password)>
String headerKey = "Proxy-Authorization";
String headerValue = "Basic " + Base64.encode(user+":"+password);
conn.setRequestProperty(headerKey, headerValue);
The last one is that you can modify Azure SDK API and overwrite the method “createURLConnection” in Class “BaseRequest” to implement accessing. The Azure Storage SDK v2.2.0 project on GitHub is https://github.com/Azure/azure-storage-java/tree/v2.2.0/.
Note:
public static HttpURLConnection createURLConnection(final URI uri, final RequestOptions options, UriQueryBuilder builder, final OperationContext opContext, java.net.Proxy proxy)
and
final HttpURLConnection retConnection = (HttpURLConnection) resourceUrl.openConnection(proxy);
public static HttpURLConnection createURLConnection(final URI uri, final RequestOptions options, UriQueryBuilder builder, final OperationContext opContext, java.net.Proxy proxy) throws IOException, URISyntaxException, StorageException {
if (builder == null) {
builder = new UriQueryBuilder();
}
final URL resourceUrl = builder.addToURI(uri).toURL();
final HttpURLConnection retConnection = (HttpURLConnection) resourceUrl.openConnection(proxy);
if (options.getTimeoutIntervalInMs() != null && options.getTimeoutIntervalInMs() != 0) {
builder.add(TIMEOUT, String.valueOf(options.getTimeoutIntervalInMs() / 1000));
}
// Note: ReadTimeout must be explicitly set to avoid a bug in JDK 6.
// In certain cases, this bug causes an immediate read timeout exception to be thrown even if ReadTimeout is not set.
retConnection.setReadTimeout(Utility.getRemainingTimeout(options.getOperationExpiryTimeInMs(), options.getTimeoutIntervalInMs()));
// Note : accept behavior, java by default sends Accept behavior as text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
retConnection.setRequestProperty(Constants.HeaderConstants.ACCEPT, Constants.HeaderConstants.XML_TYPE);
retConnection.setRequestProperty(Constants.HeaderConstants.ACCEPT_CHARSET, Constants.UTF8_CHARSET);
// Note : Content-Type behavior, java by default sends Content-type behavior as application/x-www-form-urlencoded for posts.
retConnection.setRequestProperty(Constants.HeaderConstants.CONTENT_TYPE, Constants.EMPTY_STRING);
retConnection.setRequestProperty(Constants.HeaderConstants.STORAGE_VERSION_HEADER,
Constants.HeaderConstants.TARGET_STORAGE_VERSION);
retConnection.setRequestProperty(Constants.HeaderConstants.USER_AGENT, getUserAgent());
retConnection.setRequestProperty(Constants.HeaderConstants.CLIENT_REQUEST_ID_HEADER,
opContext.getClientRequestID());
return retConnection;
}
By the way, You need to call above method in every CloudXXXClient(CloudBlobClient, etc) Class.

Following the issue-48 opened by me based on this question and additional one opened by strazh issue-65, The proxy support was improved in version 4.2.0 see here.
Added support for setting a library-wide proxy. The default proxy can
be set on OperationContext.
See the JUnits for full example
https://github.com/Azure/azure-storage-java/blob/master/microsoft-azure-storage-test/src/com/microsoft/azure/storage/GenericTests.java
Look for testDefaultProxy and testProxy

Azure Storage team has released a new SDK (v10), where the Proxy is now supported through the HttpPipeline. You can share the pipeline across all operations by passing it to StorageURL or just use in a single Blob by passing it to the BlobURL object.
AnonymousCredentials creds = new AnonymousCredentials();
// Use PipelineOptions to define a retry strategy and a proxy - you can also pass your own HttpClient to this if you like
PipelineOptions po = new PipelineOptions();
// Proxy configuration shown here as a sample
HttpClientConfiguration configuration = new HttpClientConfiguration(
new Proxy(Proxy.Type.HTTP, new InetSocketAddress("localhost", 8888)), false); //pass true if the Proxy endpoint is HTTPS
po.client = HttpClient.createDefault(configuration);
// Create a URI with SAS token, and pass it to the PageBlobURL with an HttpPipeline created by createPipeline method
URL u = new URL("https://myaccount.blob.core.windows.net/mycontainer/myfile?sv=2017-04-17&sr=b&si=14169767-6354-41ed-a99b-c9db8dcc66bc&sig=8NUr%2BSpmRH%2BB2z%2FpQZDPDquTQ7rbgWfE9a6AePLlFT0%3D");
PageBlobURL blobURL = new PageBlobURL(u, PageBlobURL.createPipeline(creds, po));

Implementing Spring Security with Java Client

Client Side
I have a java application that connects to a remote server using basic POST or GET methods:
URL url = new URL(urlStr);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setAllowUserInteraction(false);
conn.setRequestProperty("Content-type", "text/xml; charset=" + ENCODING);
conn.connect();
conn.getOutputStream().write(data.getBytes(ENCODING));
conn.getOutputStream().close();
(I cannot change this code, the only things I can change is the urlStr and the data sent to the server when calling the method).
[EDIT] : The client can be a java client or any other client (c++, objective-c, ..). The point here is that I can only access what's in the body of my post as well as the URL.
Server Side
On my server side, I would like to implement Spring Security (SecurityContext and session persistance).
I understand that spring security is based on the browser's cookies when it's a WebApp to hold the information about the session id. But in my case there's no Browser.
Do I need to simulate the storage of the JSESSIONID and send it back to the server? I'm not sure this is possible since I would need to call conn.addRequestProperty(key, value) which is not possible.
Is there any other way?
Thank you.
[EDIT]
as pointed out by #zagyi, I can use the URL to pass session token to Spring, but I still can't figure out how.

Passing the jsessionid in the url is just a matter of appending it at the end of the url like this:
http://localhost:8080/example/auth/login;jsessionid=A06F00609BBA8A4C2B005FB25F90C4C9
You can see this in working if you configure a browser not to accept any cookies, in which case the server automatically includes the session id in the url (assuming a default tomcat configuration). This topic is also discussed in this question.

There may be a client-side solution for that.
The action point where we can interact is here:
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
We will provide an own (wrapped) HttpURLConnection, which will handle the JSESSIONID. But unfortunately we have to start a bit further.
The trick is that we register a new protocol, e.g. "xhttp", that we use to wrap a real "http" protocol connection. So, your URL will look like:
xhttp://www.example.com/...
First, define a URLStreamHandlerFactory class
public class MyURLStreamHandlerFactory implements URLStreamHandlerFactory {
public URLStreamHandler createURLStreamHandler(String protocol) {
if ("xhttp".equals(protocol)) {
return new MyURLStreamHandler();
}
return null;
}
}
At Java (or application) init time we can set it. You can do it only once per JVM.
URLStreamHandlerFactory fac = new MyURLStreamHandlerFactory();
URL.setURLStreamHandlerFactory(fac);
So, let's go ahead with MyURLStreamHandler.
public class MyURLStreamHandler extends URLStreamHandler {
#Override
protected URLConnection openConnection(URL url) throws IOException {
return new MyHttpURLConnection(url);
}
}
This is quite simple, we create our own connection. Let's do the dirty stuff:
public final class MyHttpURLConnection extends HttpURLConnection {
private HttpURLConnection conn;
public MyHttpURLConnection(URL url) throws MalformedURLException, IOException {
super(url);
String newUrlString = url.toExternalForm().substring(1);
conn = (HttpURLConnection) new URL(newUrlString).openConnection();
}
#Override
public void disconnect() {
conn.disconnect();
}
#Override
public boolean usingProxy() {
return false;
}
#Override
public void connect() throws IOException {
conn.connect();
conn.setRequestProperty("JSESSIONID", "X");
}
}
And voilá, we managed to access our connection, and set the JSESSIONID header.
All you need is to compile your classes, add the class files to the client jar, and make the init code running some way in the same JVM where the above code runs.
If you cannot do it, there is another possibility: set the following system parameter to the client Java application:
-Djava.protocol.handler.pkgs=com.example.myprotocol
In this case create a com.example.myprotocol.xhttp (xhttp like your protocol name), and rename our MyURLStreamHandler class to com.example.myprotocol.xhttp.Handler. This is the fixed name where the protocol resolver will look for it. Note, that this java.protocol.handler.pkgs property is checked by the security manager.

Suppress NTLM Auth in Java

I am currently pen-testing a web application and came across an interesting phenomenon. During my testing sessions, I gathered URLs using a proxy. Now I wanted to test my URL list for anonymous access, so i wrote this little tool
public static void main(String[] args) {
try {
TrustAllCerts.disableCertChecks();
FileReader fr = new FileReader(new File("urls.txt"));
BufferedReader br = new BufferedReader(fr);
String urlStr = br.readLine();
while (urlStr != null) {
if (urlStr.trim().length() > 0) {
URL url = new URL(urlStr);
HttpsURLConnection urlc = (HttpsURLConnection) url.openConnection();
urlc.connect();
if (urlc.getResponseCode() == HttpURLConnection.HTTP_OK) {
System.out.println(urlStr);
} else {
System.out.println("["+urlc.getResponseCode()+"] "+urlStr);
}
urlc.disconnect();
}
urlStr = br.readLine();
}
br.close();
} catch (Exception e) {
e.printStackTrace();
}
}
It does basically nothing, but opening an URL connection on a given URL and test the HTTP response code (actually I implemented some more tests, if I'm getting redirected to a login page). However, the problem is, that this specific application (some custom MS SQL Server Reporting Services) is configured to use NTLM WWW authentication. If I try to access some of the URLs using Firefox, i get an 401 Unauthorized + login dlg. Internet Exploder performs NTLM auth in the background and grants access. It seems that the Java URLConnection (or URL) class does the same, because I am getting no 401. Is there a way to disable implicit NTLM authentication in Java? This is a bad pitfall for me.

I think the Java Network Documentation is the best resource. Setting the http.auth.preference="basic" should get you what you want. Assuming you don't need digest or something else. I'm not sure if you can go beyond that to disable NTLM.
Another thing to consider is other Java HTTP client implementations, like Apache's or Google's.

I'm not sure that this will help, but I've been stumped by the opposite.
I wanted NTLM auth to take place, so on my local machine I use a free app called CNTLM. It's a local proxy server that will forward (and NT auth) incoming requests. Good for apps that can't use NTLM proxies.
I'm sorry, I know this isn't answering your question, but maybe it proves helpful to someone out there! :)

set proxy in java

I create a service that post something through the internet and everything's fine. But when I deploy it to our server I get connection status:403, forbidden. I think it is because our server won't allow direct access to the internet without login first. We have to login first in the browser with our username/password to access the internet.
I notice that if I have login and access the internet in the server, the service I deploy run alright. But I don't think it's practical because in that case my service won't run if someone or I don't login first.
I have tried setting the proxy in the java code but to no avail. Could someone help me with this problem? Here is I post my service snippet.
System.getProperties().put("http.proxySet", "true");
System.getProperties().put("http.proxyHost", myHost);
System.getProperties().put("http.proxyPort", "8080");
System.getProperties().put("http.proxyUser", myUser);
System.getProperties().put("http.proxyPassword", myPassword);
System.getProperties().put("http.nonProxyHosts", "localhost|127.0.0.1");
try {
URL url = new URL(urlAddress);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("POST");
con.setDoOutput(true);
con.setDoInput(true);
...
if (con.getResponseCode() == HttpURLConnection.HTTP_OK) {
System.out.println("connection OK");
istrm = con.getInputStream();
if (istrm == null) {
System.out.println("istrm == null");
}
...
} else {
System.out.println("Response: " + con.getResponseCode() + ", " + con.getResponseMessage());
}
}
And my process goes to else block and gets response message 403

Try using System.setProperty(String, String) instead.

This is how you set application-wide proxies..
System.setProperty("https.proxyHost", "myproxy.domain.com");
System.setProperty("https.proxyPort", "myport");
NOTE: use http.proxyHost and http.proxyPort if you don't require https.

I think that the problem is that you are setting the properties too late. The HTTP request router code is likely to read those properties once during its initialization. If your code sets the properties after the HTTP request router has initialized, the new property values will have no effect ... unless you can cause the request router to reinitialize.
One way to guarantee that your proxy property settings will take effect is to set them on the java command line that launches the web server; e.g. with Tomcat on Linux, you can do this by setting the relevant "-D..." options in the JAVA_OPTS environment variable.

If you are using a proxy on your system... (I mean on your operating system)
you can use this line of code:
System.setProperty("java.net.useSystemProxies", "true");

How to use an HTTP proxy in java

I am writing a code that connects to websites and checks some code, like a crawler. But I need to connect trough a proxy and change the IP address (so it doesn't show the client's IP in the server logs).
How can this be done through java?

You can use the java system properties to set up a proxy or pass it as command line options.
You can find some details and samples here.
Ex: Before opening the connection
System.setProperty("http.proxyHost", "myProxyServer.com");
System.setProperty("http.proxyPort", "80");
Or you can use the default network proxies configured in the sytem
System.setProperty("java.net.useSystemProxies", "true");
Since Java 1.5 you can create a instance of proxy and pass it to the openConnection() method.
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("123.0.0.1", 8080));
URL url = new URL("http://www.yahoo.com");
HttpURLConnection uc = (HttpURLConnection)url.openConnection(proxy);
uc.connect();
Or as lisak suggested, you can use some 3rd party libraries which supports your need better.

Or you can also use HttpClient which would suit your needs better. Check out the documentation, it's brief and very informative.
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpHost proxy = new HttpHost("someproxy", 8080);
httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);

Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("123.0.0.1", 8080));
URL url = new URL("http://www.yahoo.com");
HttpURLConnection uc = (HttpURLConnection)url.openConnection(proxy);
uc.connect();
This worked for me. I was able to use the proxy for the specific connection or transfer. Earlier we were using System.setProperty which used to set it at system level and all the requests internal/external started using the same proxy.
Also Proxy.Type.HTTP works for both http and https