Develop Reference

HTTP Post to API return 403 FORBIDDEN

I have to upload an XML file to an API. This is API is secured by a signed certificate, which I got from the issuer of the API.
Now, I got two use cases. First, I have to download some files from this API. This is working perfectly with the following code:
final Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ip, port));
final URL url = new URL(linkToFile);
final HttpsURLConnection conn = (HttpsURLConnection) url.openConnection(proxy);
conn.setSSLSocketFactory(this.http.createSSLContext().getSocketFactory());
try (
InputStream inputStream = zipUrlConn.getInputStream();
ZipInputStream stream = new ZipInputStream(inputStream);) {
// Do stuff with ZipInputStream here
}
The createSSLContext() method looks as follows:
public SSLContext createSSLContext() throws NoSuchAlgorithmException, CertificateException, IOException, KeyStoreException,
UnrecoverableKeyException, KeyManagementException {
final KeyStore clientStore = KeyStore.getInstance("PKCS12");
clientStore.load(new FileInputStream(this.certificateResource.getFile()), this.p12PW.toCharArray());
final KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
keyManagerFactory.init(clientStore, this.p12PW.toCharArray());
final KeyManager[] keyManagers = keyManagerFactory.getKeyManagers();
final KeyStore trustStore = KeyStore.getInstance("JKS");
trustStore.load(new FileInputStream(this.trustStoreResource.getFile()), this.trustStorePW.toCharArray());
final TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(trustStore);
final TrustManager[] trustManagers = trustManagerFactory.getTrustManagers();
final SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagers, trustManagers, new SecureRandom());
return sslContext;
}
I was following a guideline, which I got from the issuer, which showed how to do this with a cUrl command:
curl --cert Certificate-<id>.pem[:pem_password] https://api.url.com/
So I was basically trying to rebuild this command in java, which is working.
Now for the part that's not working, which is the file upload. Again, I was given a cUrl command which I have to rebuild:
curl --cert Certificate-<id>.pem[:pem_password] -F upload=#<Path_to_file>\DS<PartnerId>_<Timestamp>.XML https://api.url.com/in/upload.php
I tried several things to achieve that:
"Normal" Java
Firstly, I tried it with the standard HttpsURLConnection as follows:
final Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("192.168.1.25", 3128));
final HttpsURLConnection connection = (HttpsURLConnection) new URL(ARGE_UPLOAD_URL).openConnection(proxy);
connection.setSSLSocketFactory(this.http.createSSLContext().getSocketFactory());
connection.setDoOutput(true);
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8");
try (OutputStream outputStream = connection.getOutputStream()) {
outputStream.write(Files.readAllBytes(new File("src/main/resources/XML/example.xml").toPath()));
}
final InputStream result = connection.getInputStream();
But this always results in java.io.IOException: Server returned HTTP response code: 403 for URL: https://api.url.com/in/upload.php, even though I'm using the same configuration, with which I am able to download from the API.
Apache HttpClient
I found some resources claiming that the HttpClient is a lot easier to configure and use, so I gave it a try:
final CloseableHttpClient httpClient = HttpClientBuilder
.create()
.setSSLContext(this.http.createSSLContext())
.setProxy(new HttpHost(InetAddress.getByName(ip), port))
.build();
final HttpEntity requestEntity = MultipartEntityBuilder
.create()
.addBinaryBody("upload=#example.xml", new File("src/main/resources/XML/example.xml")) // Hardcoded for testing
.build();
final HttpPost post = new HttpPost(https://api.url.com/in/upload.php);
post.setEntity(requestEntity);
try (CloseableHttpResponse response = httpClient.execute(post)) {
this.logger.info(response.getStatusLine().toString());
EntityUtils.consume(response.getEntity());
}
Resulting in HTTP/1.1 403 FORBIDDEN
HttpClient (FileEntity instead of MultipartEntity)
As a last thing I tried to a FileEntity:
final HttpPost httpPost = new HttpPost(https://api.url.com/in/upload.php);
httpPost.addHeader("content-type", "application/x-www-form-urlencoded;charset=utf-8");
final FileEntity fileEntity = new FileEntity(new File("src/main/resources/XML/example.xml"));
httpPost.setEntity(fileEntity);
System.out.println("executing request " + httpPost.getRequestLine() + httpPost.getConfig());
try (CloseableHttpResponse response = httpClient.execute(httpPost)) {
final HttpEntity responseEntity = response.getEntity();
System.out.println("Status: " + response.getStatusLine());
if (responseEntity != null) {
System.out.println("Entity: " + EntityUtils.toString(responseEntity));
}
}
Resulting in Status: HTTP/1.1 403 FORBIDDEN
I just don't understand, how I can be able to download from the API, but not upload to it, despite using exactly the same configuration.
If you need any more information, I'll be happy to provide them.
EDIT
As suggested by oli, I used Fiddler to capture the HTTPS request. This is the result for method 1 (Normal Java):
POST https://hrbaxml.arbeitsagentur.de/in/upload.php HTTP/1.1
Accept-Charset: utf-8
Accept: */*
Content-Type: application/x-www-form-urlencoded;charset=utf-8
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
Host: hrbaxml.arbeitsagentur.de
Connection: keep-alive
Content-Length: 6738
And this is the result from the manual upload through Google Chrome:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 6948
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryxZxIJ5h19MEFbZQs
Cookie: cookie_consent=accepted
Host: hrbaxml.arbeitsagentur.de
Origin: https://hrbaxml.arbeitsagentur.de
Referer: https://hrbaxml.arbeitsagentur.de/in/
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
EDIT 2
Just to add, this is the result using method 2 (HttpClient with MultipartEntity):
POST https://hrbaxml.arbeitsagentur.de/in/upload.php HTTP/1.1
Content-Length: 7025
Content-Type: multipart/form-data; boundary=1sZvrqcGe-FuQ3r-_fFgt2SJtZ5_yo7Pfvq_
Host: hrbaxml.arbeitsagentur.de
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.5 (Java/1.8.0_161)
Accept-Encoding: gzip,deflate
--1sZvrqcGe-FuQ3r-_fFgt2SJtZ5_yo7Pfvq_
Content-Disposition: form-data; name="DSV000306700_2018-08-23_09-00-00.xml"; filename="DSV000306700_2018-08-23_09-00-00.xml"
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
EDIT 3
I tried copying all the HTTP Headers from the Chrome request, so that my request from Java looks like this:
POST https://hrbaxml.arbeitsagentur.de/in/upload.php HTTP/1.1
Accept: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: max-age=0
Content-Type: multipart/form-data; boundary=1535095530678
Cookie: cookie_consent=accepted
Referer: https://hrbaxml.arbeitsagentur.de/in/
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
Host: hrbaxml.arbeitsagentur.de
Connection: keep-alive
Content-Length: 6944
--1535095530678
Content-Disposition: form-data; name="uploadFile"; filename="DSV000306700_2018-08-23_09-00-00.xml"
Content-Type: application/xml
Content-Transfer-Encoding: binary
.. xml data ..
--1535095530678--
But still, without success. Any other possible solutions? Maybe it isn't a problem with the upload but something else?

I would capture the HTTP request (f.e. with Wireshark) that you send to the server with your Java-Application and compare it with the HTTP request that you send from the browser (yout can easy capture it with the build-in browser tools, try to press F12).
I'm 100% sure that you will see some differences, this is what allways works for me.
EDIT:
There is another possible problem. Please try to add
connection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0");
or the same one, that your browser sends in your first implementation. Also make sure that you have no problems with your SSL certificate and with encrypt algorithm (you use default one, which one is it in your case). In addition (if nothing helps) you can also check the key length of the negotiationed handshake key.

This information came from the server side, so you should ask the service provider for help, maybe upload needs auth

Okay, I finally figured it out. The error was in my POST request, as expected. First, here is the working code (with method 2):
final CloseableHttpClient httpClient = HttpClientBuilder
.create()
.setSSLContext(this.http.createSSLContext())
.setProxy(new HttpHost(InetAddress.getByName(ip), port))
.build();
final HttpEntity requestEntity = MultipartEntityBuilder
.create()
.addBinaryBody("upload", new File("src/main/resources/XML/example.xml")) // Hardcoded for testing
.build();
final HttpPost post = new HttpPost(https://api.url.com/in/upload.php);
post.setEntity(requestEntity);
try (CloseableHttpResponse response = httpClient.execute(post)) {
this.logger.info(response.getStatusLine().toString());
EntityUtils.consume(response.getEntity());
}
As you can see, the only line that changed is .addBinaryBody("upload", new File("src/main/resources/XML/example.xml")). The "upload" is crucial, since the cUrl call I was trying to rebuilt had the -F flag. From the cUrl manual:
(HTTP SMTP IMAP) For HTTP protocol family, this lets curl emulate a filled-in form in which a user has pressed the submit button. This causes curl to POST data using the Content-Type multipart/form-data according to RFC 2388.
Each part then consists of a name and the data of the file. The API I'm working with relies on the name being "upload", in order to process the request. Otherwise it doesn't know what to do and returns 403 FORBIDDEN.

How to cache okHTTP response from Web server?

I want to know how can okHTTP response from Web server (which returns json data) be cached?
I want my app to download all the data needed for RecycleView and cache it once the user runs the app first time - and avoid re-downloading and parsing all the same data from Web server, if data has not changed.
I tried to get response headers, but this is what I get:
Request URL: https://somedomain.com/wp-json/?categories=3&per_page=100&status=publish
Request Method: GET
Status Code: 200 OK
Remote Address: 176.28.12.139:443
Referrer Policy: no-referrer-when-downgrade
Access-Control-Allow-Headers: Authorization, Content-Type
Access-Control-Expose-Headers: X-WP-Total, X-WP-TotalPages
Allow: GET
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json; charset=UTF-8
Date: Fri, 23 Mar 2018 15:41:23 GMT
Link: <https://somedomain.com/wp-json/>; rel="https://api.w.org/"
Server: nginx
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Powered-By: PleskLin
X-Powered-By: PHP/7.1.15
X-Robots-Tag: noindex
X-WP-Total: 43
X-WP-TotalPages: 1
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: hr,en-US;q=0.9,en;q=0.8,de;q=0.7,nb;q=0.6
Connection: keep-alive
Host: somedomain.com
Referer: https://somedomain.com/somedir/
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
categories: 3
per_page: 100
status: publish
I have read that I could find ETAG or Last-Modified from the Server response to check for any changes, but as you can see there is no such thing.
Do you have any idea what to do if app only needs to download data at the first run - and after that only if the data has changed?

You need cache interceptor like this:
public class CacheInterceptor implements Interceptor {
#Override
public Response intercept(Chain chain) throws IOException {
Response response = chain.proceed(chain.request());
CacheControl cacheControl = new CacheControl.Builder()
.maxAge(15, TimeUnit.MINUTES) // 15 minutes cache
.build();
return response.newBuilder()
.removeHeader("Pragma")
.removeHeader("Cache-Control")
.header("Cache-Control", cacheControl.toString())
.build();
}
}
Add this interceptor with Cache to your OkHttpClient like this:
File httpCacheDirectory = new File(applicationContext.getCacheDir(), "http-cache");
int cacheSize = 10 * 1024 * 1024; // 10 MiB
Cache cache = new Cache(httpCacheDirectory, cacheSize);
OkHttpClient okHttpClient = new OkHttpClient.Builder()
.addNetworkInterceptor(new CacheInterceptor())
.cache(cache)
.build();

Just add a cache to your OkHttp client.
First, you'll need to create the cache:
int cacheSize = 10 * 1024 * 1024;
Cache cache = new Cache(cacheDirectory, cacheSize);
Note that with this code you're providing a cache with a size limit of 10MB. After this, you'll create your client with the following code:
OkHttpClient client = new OkHttpClient.Builder()
.cache(cache)
.build();
The client will honor cache related headers. You can relate to the official documentation for further details:
https://github.com/square/okhttp/wiki/Recipes

HttpOnly cookies not sent by request

I want to use HttpOnly cookies and I set it in Java as follows:
...
Cookie accessTokenCookie = new Cookie("token", userToken);
accessTokenCookie.setHttpOnly(true);
accessTokenCookie.setSecure(true);
accessTokenCookie.setPath("/");
response.addCookie(accessTokenCookie);
Cookie refreshTokenCookie = new Cookie("refreshToken", refreshToken);
refreshTokenCookie.setHttpOnly(true);
refreshTokenCookie.setSecure(true);
refreshTokenCookie.setPath("/");
response.addCookie(refreshTokenCookie);
...
I got the client side the response with the cookies, but when I send the next request I do not have the cookies on the request. Maybe I miss something, but as I understood, these HttpOnly cookies has to be sent by the browser back on every request (JavaScript does not have access to those cookies) coming to the defined path.
I have the following Request Headers:
Accept:application/json, text/plain, */*
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.8,hu;q=0.6,ro;q=0.4,fr;q=0.2,de;q=0.2
Authorization:Basic dXNlcm5hbWU6cGFzc3dvcmQ=
Connection:keep-alive
Content-Length:35
content-type:text/plain
Host:localhost:8080
Origin:http://localhost:4200
Referer:http://localhost:4200/
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36
X-Requested-With:XMLHttpRequest
and the following response headers:
Access-Control-Allow-Credentials:true
Access-Control-Allow-Origin:http://localhost:4200
Access-Control-Expose-Headers:Access-Control-Allow-Origin, Content-Type, Date, Link, Server, X-Application-Context, X-Total-Count
Cache-Control:no-cache, no-store, max-age=0, must-revalidate
Content-Length:482
Content-Type:application/json;charset=ISO-8859-1
Date:Fri, 03 Feb 2017 13:11:29 GMT
Expires:0
Pragma:no-cache
Set-Cookie:token=eyJhbGciO;Max-Age=10000;path=/;Secure;HttpOnly
Set-Cookie:refreshToken=eyJhb8w;Max-Age=10000;path=/;Secure;HttpOnly
Vary:Origin
Also in the client side I use withCredentials: true in Angular2 and X-Requested-With:XMLHttpRequest as request header.
And it is Cross Domain.

Yes you are correct having the cookie your browser should send the cookie automatically while it is not expired and the httpOnly flag means it cannot be accessed or manipulated via JavaScript.
However
You need to ensure that the cookie you are sending is not cross domain, if you require it cross domain you will need to handle it differently.

Urlencoding data for post request body. Am I using wrong charset?

I want to replicate a working POST request in Java. For testing purpose, lets take message like: 'äöõüäöõüäöõüäöõü'
Working POST request (with encoded message of 'äöõüäöõüäöõüäöõü'):
Header
POST http://www.mysite.com/newreply.php?do=postreply&t=477352 HTTP/1.1
Host: www.warriorforum.com
Connection: keep-alive
Content-Length: 403
Origin: http://www.mysite.com
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko)Chrome/14.0.835.202 Safari/535.1
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Accept: */*
Referer: http://www.mysite.com/test-forum/477352-test.html
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: bblastvisit=1319205053; bblastactivity=0; bbuserid=265374; bbpassword=1125e9ec1ab41f532ab8ec6f77ddaf94; bbsessionhash=91444317c100996990a04d6c5bbd8375;
Body
securitytoken=1319806096-618e5f9012901e2d818bf2c74c2121baa064be57&ajax=1&ajax_lastpost=1319806096&**message=%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC**&wysiwyg=0&styleid=1&signature=1&fromquickreply=1&s=&do=postreply&t=477352&p=who%20cares&specifiedpost=0&parseurl=1&loggedinuser=265374
As we can see in the request body 'äöõüäöõüäöõüäöõü is encoded as: %u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC
Now i want to replicate it.
Lets Url encode the text with charset utf-8 in Java:
String userText = "äöõüäöõüäöõüäöõü";
String encoded = URLEncoder.encode(userText, "utf-8");
Result: %C3%A4%C3%B6%C3%B5%C3%BC%C3%A4%C3%B6%C3%B5%C3%BC%C3%A4%C3%B6%C3%B5%C3%BC%C3%A4%C3%B6%C3%B5%C3%BC%0A%0A%0A%5BSIZE%3D%221%22%5D%5BI%5D << NOT THE SAME
Lets try ISO-8859-1:
String userText = "äöõüäöõüäöõüäöõü";
String encoded = URLEncoder.encode(userText, "ISO-8859-1");
Result: %E4%F6%F5%FC%E4%F6%F5%FC%E4%F6%F5%FC%E4%F6%F5%FC%0A%0A%0A%5BSIZE%3D%221%22%5D%5BI%5D << NOT THE SAME
Neither of them produce the same encoded string as in the working example, but all of them have the same input. What am I missing here?

%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC%u00E4%u00F6%u00F5%u00FC
I don't know what the above data is encoded as, but it isn't application/x-www-form-urlencoded; charset=UTF-8 as the request claims. This is not legal data for this MIME type.
It looks like some UTF-16BE-encoded form.
URLEncoder.encode(userText, "utf-8"); would be the correct way to encode the application/x-www-form-urlencoded; charset=UTF-8 values if this was actually what the server was expecting. (ref)

HTTP POST request with authorization on android

When I set "Authorization" header with setHeader from HttpPost then hostname disappears from request and there is always error 400 (bad request) returned. Same code is working fine on pure java (without android) and when I remove setting "Authorization" header also on android it works fine, but I need authorization.
This is a code (domain changed):
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost("http://myhost.com/test.php");
post.setHeader("Accept", "application/json");
post.setHeader("User-Agent", "Apache-HttpClient/4.1 (java 1.5)");
post.setHeader("Host", "myhost.com");
post.setHeader("Authorization",getB64Auth());
List <NameValuePair> nvps = new ArrayList <NameValuePair>();
nvps.add(new BasicNameValuePair("data[body]", "test"));
AbstractHttpEntity ent=new UrlEncodedFormEntity(nvps, HTTP.UTF_8);
ent.setContentType("application/x-www-form-urlencoded; charset=UTF-8");
ent.setContentEncoding("UTF-8");
post.setEntity(ent);
post.setURI(new URI("http://myhost.com/test.php"));
HttpResponse response =client.execute(post);
Method getB64Auth() returns "login:password" encoded using Base64 like: "YnxpcYRlc3RwMTulHGhlSGs=" but it's not important.
This is a piece of lighttpd's error.log when above code is invoked on pure java:
2011-02-23 15:37:36: (request.c.304) fd: 8 request-len: 308
POST /test.php HTTP/1.1
Accept: application/json
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: myhost.com
Authorization: Basic YnxpcYRlc3RwMTulHGhlSGs=
Content-Length: 21
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Encoding: UTF-8
Connection: Keep-Alive
HTTP/1.1 200 OK
Content-type: text/html
Transfer-Encoding: chunked
and record from access.log (IP changed):
1.1.1.1 myhost.com - [23/Feb/2011:15:37:36 +0100] "POST /test.php HTTP/1.1" 200 32 "-" "Apache-HttpClient/4.1 (java 1.5)"
When the same code is invoked on android, I get this in logs:
POST /test.php HTTP/1.1
Accept: application/json
User-Agent: Apache-HttpClient/4.1 (java 1.5)
Host: myhost.com
Authorization: Basic YnxpcYRlc3RwMTulHGhlSGs=
Content-Length: 21
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Encoding: UTF-8
Connection: Keep-Alive
Expect: 100-Continue
2011-02-23 15:45:10: (response.c.128) Response-Header:
HTTP/1.1 400 Bad Request
Content-Type: text/html
Content-Length: 349
Connection: close
access.log:
1.1.1.1 - - [23/Feb/2011:15:45:10 +0100] "POST /test.php HTTP/1.1" 400 349 "-" "Apache-HttpClient/4.1 (java 1.5)"
How to get Authorization with POST working on android?
When I use HttpURLConnection instead of HttpClient it is no difference.

Thanks to Samuh for a hint :)
There was an extra newline character inserted which has no means in GET requests, but matters in POST ones.
This is proper way to generate Authorization header in android (in getB64Auth in this case):
private String getB64Auth (String login, String pass) {
String source=login+":"+pass;
String ret="Basic "+Base64.encodeToString(source.getBytes(),Base64.URL_SAFE|Base64.NO_WRAP);
return ret;
}
The Base64.NO_WRAP flag was lacking.

use simply this :
String authorizationString = "Basic " + Base64.encodeToString(
("your_login" + ":" + "your_password").getBytes(),
Base64.NO_WRAP); //Base64.NO_WRAP flag
post.setHeader("Authorization", authorizationString);