Develop Reference

Extract multiple JSON strings from multipart HTTP response

I'm using Apache HttpClient to work with a web service that returns a multipart/form-data response which contains json.
I'm having a very hard time extracting each JSON string separately so I can read the json string.
I did read similar posts on Stackoverflow, and some suggested using Apache commons fileupload, but I am not sure how that can separate the JSON strings from the whole response that has a bunch of other text such as the boundary string, content type, etc
The response looks something like below.
--Boundary_16003419_2104021487_1483424496169
Content-Type: application/json
Content-RequestId: GetMailboxes
Status-Code: 200
X-Server-Response-Time: 4ms
X-Server-Chain: domain.com
Content-RequestDuration: 5
{JSON}
--Boundary_16003419_2104021487_1483424496169
Content-Type: application/json
Content-RequestId: GetFolders
Status-Code: 200
X-Server-Response-Time: 8ms
X-Server-Chain: domain.com
Content-RequestDuration: 10
{JSON}
--Boundary_16003419_2104021487_1483424496169
Content-Type: application/json
Content-RequestId: GetAlerts
Status-Code: 200
X-Server-Response-Time: 10ms
X-Server-Chain: domain.com
Content-RequestDuration: 12
{JSON}
--Boundary_16003419_2104021487_1483424496169
Content-Type: application/json
Content-RequestId: GetAccounts
Status-Code: 200
X-Server-Response-Time: 11ms
X-Server-Chain: domain.com
Content-RequestDuration: 12
{JSON}
--Boundary_16003419_2104021487_1483424496169
Content-Type: application/json
Content-RequestId: GetAllSavedSearches
Status-Code: 200
X-Server-Response-Time: 10ms
X-Server-Chain: domain.com
Content-RequestDuration: 12
{JSON}
--Boundary_16003419_2104021487_1483424496169
Content-Type: application/json
Content-RequestId: GetAthenaSegment
Status-Code: 200
X-Server-Response-Time: 14ms
Content-RequestDuration: 21
{JSON}
--Boundary_16003419_2104021487_1483424496169
Content-Type: application/json
Content-RequestId: ListFolderThreads
Status-Code: 200
X-Server-Response-Time: 110ms
Content-RequestDuration: 116
{JSON}
--Boundary_16003419_2104021487_1483424496169
Content-Type: application/json
Content-RequestId: GetUserInfo
Status-Code: 200
X-Server-Response-Time: 197ms
Content-RequestDuration: 204
{JSON}
--Boundary_16003419_2104021487_1483424496169
Content-RequestId: Status
Content-Type: application/json
{JSON}
--Boundary_16003419_2104021487_1483424496169--
Any way to do this reliably?

One option would be Apache Mime4j. You would likely want to use MimeTokenStream or MimeStreamParser as described here if you want your application to handle response content without building a complete DOM tree in memory.

Stream using libstreaming to VLC 2.2.4 exception

I setup libstreaming to stream video from Android camera to VLC 2.2.4 on my macOS.
SharedPreferences.Editor editor = PreferenceManager.getDefaultSharedPreferences(context).edit();
editor.putString(RtspServer.KEY_PORT, String.valueOf(1777));
editor.apply();
SessionBuilder.getInstance()
.setContext(context)
.setCallback(this)
.setAudioEncoder(SessionBuilder.AUDIO_NONE)
.setVideoEncoder(SessionBuilder.VIDEO_H264)
.setVideoQuality(new VideoQuality(640, 480, 15, 500000));
context.startService(new Intent(context, RtspServer.class));
Then I builded url bellow and opened it in VLC 2.2.4:
rtsp://172.xx.xx.67:1777?h264=500000-15-640-480
Then I received some logs on Android side:
OPTIONS
RTSP/1.0 200 OK
Server: MajorKernelPanic RTSP Server
Cseq: 2
Content-Length: 0
DESCRIBE
SPS: Z0KAHtoCgPaAbQoTUA==
PPS: aM4G4g==
RTSP/1.0 200 OK
Server: MajorKernelPanic RTSP Server
Cseq: 3
Content-Length: 272
Content-Base: 172.xx.xx.53:1777/
Content-Type: application/sdp
v=0
o=- 0 0 IN IP4 172.xx.xx.53
s=Unnamed
i=N/A
c=IN IP4 172.xx.xx.26
t=0 0
a=recvonly
m=video 5006 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=42801e;sprop-parameter-sets=Z0KAHtoCgPaAbQoTUA==,aM4G4g==;
a=control:trackID=1
SETUP
PPS: aM4G4g==
SPS: Z0KAHtoCgPaAbQoTUA==
RTSP/1.0 200
Server: MajorKernelPanic RTSP
Cseq: 4
Content-Length: 0
Transport: RTP/AVP/UDP;unicast;destination=172.xx.xx.26;client_port=5006-5007;server_port=42104-42497;ssrc=68db94da;mode=play
Session: 1185d20035702ca
Cache-Control: no-cache
PLAY
RTSP/1.0 200 OK
Server: MajorKernelPanic RTSP Server
Cseq: 5
Content-Length: 0
RTP-Info: url=rtsp://172.xx.xx.53:1777/trackID=1;seq=0
Session: 1185d20035702ca
TEARDOWN
RTSP/1.0 200 OK
Server: MajorKernelPanic RTSP Server
Cseq: 6
Content-Length: 0
java.lang.IllegalStateException: No successful match so far
at java.util.regex.Matcher.ensureMatch(Matcher.java:471)
at java.util.regex.Matcher.group(Matcher.java:579)
at net.majorkernelpanic.streaming.rtsp.RtspServer$Request.parseRequest(RtspServer.java:643)
at net.majorkernelpanic.streaming.rtsp.RtspServer$WorkerThread.run(RtspServer.java:393)
E/RtspServer:Error parsing CSeq: Attempt to read from field 'java.util.HashMap net.majorkernelpanic.streaming.rtsp.RtspServer$Request.headers' on a null object reference
RTSP/1.0 400 Bad Request
Server: MajorKernelPanic RTSP Server
Content-Length: 0
Or shortly:
OPTIONS 200 -> DESCRIBE 200 -> SETUP 200 -> PLAY 200 -> TEARDOWN 200
-> Error parsing CSeq (IllegalStateException: No successful match so far)
- no method in the request
After successful connect (all steps above) I get exception in RtspServer.
As result I can't play the stream.
Here is the log for official example spydroid-ipcamera (connecting from VLC 2.2.4):
!!! It works on VLC 1.1.5:

Azure Billing API returning 403

I have spent several days trying to figure this out, looking at all the info I could find on the Azure website, the Azure git hubs, and all the relevant stack overflow posts. I hope I am missing something simple.
I am using the example java code being posted around the web to obtain a token:
try {
exec = Executors.newFixedThreadPool(1);
context = new AuthenticationContext("https://login.microsoftonline.com/8e4f0713-5eea-4da0-99c0-xxxxxxxxxxxx",
true, exec);
ClientCredential cred = new ClientCredential(webClientID, clientSecret);
Future<AuthenticationResult> future = context.acquireToken("https://management.azure.com/", cred, null);
result = future.get();
} catch(Exception e) {
logger.warn("Exception " + e);
} finally {
exec.shutdown();
}
if (result == null) {
return null;
}
return result.getAccessToken();
This generates a token which I place into the request header:
Authorization: Bearer -token-
The GET https://management.azure.com/subscriptions/758ad253-cbf5-4b18-8863-xxxxxxxxxxxx/providers/Microsoft.Commerce/RateCard?api-version=2015-06-01-preview%26%24filter%3DOfferDurableId+eq+%27MS-AZR-0003p%27+and+Currency+eq+%27USD%27+and+Locale+eq+%27en-US%27+and+RegionInfo+eq+%27US%27
Returns 403 code:
Exception: java.io.IOException: Server returned HTTP response code: 403 for URL: https://management.azure.com/subscriptions/758ad253-cbf5-4b18-8863-xxxxxxxxxxxx/providers/Microsoft.Commerce/RateCard?api-version=2015-06-01-preview%26%24filter%3DOfferDurableId+eq+%27MS-AZR-0003p%27+and+Currency+eq+%27USD%27+and+Locale+eq+%27en-US%27+and+RegionInfo+eq+%27US%27
Headers:
[0] null: HTTP/1.1 403 Forbidden
[1] Cache-Control: no-cache
[2] Pragma: no-cache
[3] Content-Type: application/json; charset=utf-8
[4] Expires: -1
[5] x-ms-failure-cause: gateway
[6] x-ms-request-id: e4ad9253-e034-481d-aba0-f46902b7057f
[7] x-ms-correlation-request-id: e4ad9253-e034-481d-aba0-f46902b7057f
[8] x-ms-routing-request-id: EASTUS:20151103T205103Z:e4ad9253-e034-481d-aba0-f46902b7057f
[9] Strict-Transport-Security: max-age=31536000; includeSubDomains
[10] Date: Tue, 03 Nov 2015 20:51:02 GMT
[11] Connection: close
[12] Content-Length: 303
I did all the setup on the Azure mgmt console, to create the App in Azure AD, get the clientID & client secret, etc. The SSL/HTTPS code is:
azureURL = new java.net.URL(url);
con = (HttpsURLConnection)azureURL.openConnection();
con.disconnect();
con.setDoOutput(true);
con.setDoInput(true);
con.setUseCaches(false);
con.setSSLSocketFactory(MyUtils.getSSLSocketFactory());
con.setRequestMethod("GET");
con.setRequestProperty("x-ms-version", "2015-06-01-preview");
con.setRequestProperty("Content-Type", "application/json");
String token = getAccessTokenFromServicePrincipalCredentials();
if (token != null) {
con.setRequestProperty("Authorization", "Bearer " + token);
con.connect();
in = (InputStream)con.getContent();
InputStreamReader inr = new InputStreamReader(in);
} else {
logger.warn("unable to obtain prices");
}
Any suggestions on how to debug the problem?

I see that you're getting an access token for the Service Principal (i.e. the application itself). Please make sure that you grant this Service Principal user at least Reader role on your Azure Subscription. I believe you're getting this error because this user doesn't have any access to your Azure Subscription.
Please see this link: https://azure.microsoft.com/en-in/documentation/articles/role-based-access-control-configure/ (Manage access using the Azure Management Portal section) on how you can assign role.

As we can’t call Usage & Rate Card APIs via certificate with HTTPS requests. As it is mentioned:
All of the tasks that you do on resources using the Azure Resource Manager must be authenticated with Azure Active Directory
on Authenticating Azure Resource Manager requests.
So you got a 403 issue.
Please try to build common HTTP request for the REST APIs with the request header Content-Type and Authorization which is mentioned on Resource Usage (Preview).
Also you can test to get the info you want in HTTP request build tool, like:
And here is a similar thread How to use Management certificate based authentication for making REST API calls to Azure? for your reference.

For the file not found the reply is:
java.io.FileNotFoundException:
https://management.azure.com/subscriptions/758ad253-cbf5-4b18-8863-3eed082xxxxx/providers/Microsoft.Commerce/RateCard?api-version=2015-06-01-preview%26%24filter%3DOfferDurableId+eq+%27MS-AZR-0003p%27+and+Currency+eq+%27USD%27+and+Locale+eq+%27en-US%27+and+RegionInfo+eq+%27US%27
HTTP/1.1 404 Not Found
[1] Cache-Control: no-cache
[2] Pragma: no-cache
[3] Content-Type: application/json; charset=utf-8
[4] Expires: -1
[5] x-ms-failure-cause: gateway
[6] x-ms-request-id: 8bd5ea3a-5a5f-4eb5-86b5-bd6581f94e00
[7] x-ms-correlation-request-id: 8bd5ea3a-5a5f-4eb5-86b5-bd6581f94e00
[8] x-ms-routing-request-id: EASTUS:20151119T181954Z:8bd5ea3a-5a5f-4eb5-86b5-bd6581f94e00
[9] Strict-Transport-Security: max-age=31536000; includeSubDomains
[10] Date: Thu, 19 Nov 2015 18:19:53 GMT
[11] Content-Length: 348
Let me see what else I can get...

JSIP (JAIN SIP) exchanging request URI and Route header when sending. Why? How to prevent it?

I am using JSIP (JAIN-SIP) version 1.2 (implementation build 2384). I am trying to send the following SIP message (I removed actual IP addresses in the post):
INVITE sip:bob#miniims.net SIP/2.0
Via: SIP/2.0/UDP xxx.yyy.zzz.www:5065;branch=z9hG4bKbranch1
Route: <sip:xxx.yyy.zzz.www>,<sip:xxx.yyy.zzz.www:5062>
Max-Forwards: 70
From: "Alice" <sip:alice#miniims.net>;tag=rawclientv1.0
To: "Bob" <sip:bob#miniims.net>
Call-ID: f2927406a96b0b3d3ef8213aa338d1c4#xxx.yyy.zzz.www
CSeq: 1 INVITE
Contact: "alice" <sip:alice#xxx.yyy.zzz.www:5065>
P-Preferred-Identity: <sip:alice#miniims.net>
Content-Length: 141
v=0
o=alice 2890844526 2890844526 IN IP4 xxx.yyy.zzz.www
s=
c=IN IP4 xxx.yyy.zzz.www
t=0 0
m=audio 12346 RTP/AVP 0 8 96
a=rtpmap:96 L16/8000
When I print it after parsing, it is correct. However, when I send it, the request URI is put to the end of the list of Route headers, and the first Route header is moved into SIP URI. No RFC or 3GPP standard prescribes such behavior. WireShark already captures the wrong message:
INVITE sip:xxx.yyy.zzz.www SIP/2.0
Via: SIP/2.0/UDP xxx.yyy.zzz.www:5065;branch=z9hG4bKbranch1
Route: <sip:xxx.yyy.zzz.www:5062>,<sip:bob#miniims.net>
Max-Forwards: 70
From: "Alice" <sip:alice#miniims.net>;tag=rawclientv1.0
To: "Bob" <sip:bob#miniims.net>
Call-ID: f2927406a96b0b3d3ef8213aa338d1c4#xxx.yyy.zzz.www
CSeq: 1 INVITE
Contact: "alice" <sip:alice#xxx.yyy.zzz.www:5065>
P-Preferred-Identity: <sip:alice#miniims.net>
Content-Length: 141
v=0
o=alice 2890844526 2890844526 IN IP4 xxx.yyy.zzz.www
s=
c=IN IP4 xxx.yyy.zzz.www
t=0 0
m=audio 12346 RTP/AVP 0 8 96
a=rtpmap:96 L16/8000
Is it normal behavior in JSIP? Or a bug? Is there a way to prevent it, i.e. a workaround? Thanks in advance!

Use lr; parameter in the Route Header.

HttpClient 4.2.2 and proxy with username/password

I have a problem with HttpClient 4.2.2 and proxy with username/password.
HttpClient works in this way:
List item
Send request without proxy stuff (however I set proxy parameters for each request)
Get response from Squid with 407 error
Send request with proxy stuff
It's very strange behavior, Is it possible to add info about proxy to each request? I've tried to add hardcoded "Proxy-Authorization" header to each request and it works fine, why HttpClient can't do the same?
Java code
DefaultHttpClient httpClient = new DefaultHttpClient();
httpClient.getParams().setParameter(PROTOCOL_VERSION, HTTP_1_1);
...
String proxyServer = getProxyServer();
int proxyPort = getProxyPort();
List<String> authpref = new ArrayList<String>();
authpref.add(AuthPolicy.BASIC);
httpClient.getParams().setParameter(AuthPNames.PROXY_AUTH_PREF, authpref);
String proxyUser = getProxyUser();
String proxyPassword = getProxyPassword();
CredentialsProvider credsProvider = httpClient.getCredentialsProvider();
credsProvider.setCredentials(new AuthScope(proxyServer, proxyPort), new UsernamePasswordCredentials(proxyUser, proxyPassword));
httpClient.setCredentialsProvider(credsProvider);
HttpHost proxy = new HttpHost(proxyServer, proxyPort, (proxyServer.indexOf("https") != 0) ? "http" : "https");
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
...
HttpPut put = new HttpPut(url);
/*** hardcoded header ***/
//put.addHeader("Proxy-Authorization", "Basic eHRlbmR4LmRuZXByOnF3ZXJ0eQ==");
/*** hardcoded header ***/
put.setEntity(entity);
httpClient.execute(put);
httpClient.getConnectionManager().shutdown();
[DefaultClientConnection] Sending request: PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg HTTP/1.1
[headers] >> PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg HTTP/1.1
[headers] >> Authorization: Basic eGRldjo0YTFmNmMwOTgyYWRkMWQ0NDg1YjRhMGE4YWMxY2JjMWNiMTA0ODc1
[headers] >> Content-Length: 2
[headers] >> Host: 172.26.27.22:8080
[headers] >> Proxy-Connection: Keep-Alive
[headers] >> User-Agent: Apache-HttpClient/4.2.2 (java 1.5)
[DefaultClientConnection] Receiving response: HTTP/1.0 407 Proxy Authentication Required
[headers] << HTTP/1.0 407 Proxy Authentication Required
[headers] << Server: squid/2.7.STABLE8
[headers] << Date: Thu, 08 Nov 2012 10:09:49 GMT
[headers] << Content-Type: text/html
[headers] << Content-Length: 1431
[headers] << X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
[headers] << Proxy-Authenticate: Basic realm="Please, enter username and password"
[headers] << X-Cache: MISS from 172.26.27.94
[headers] << X-Cache-Lookup: NONE from 172.26.27.94:3128
[headers] << Via: 1.0 172.26.27.94:3128 (squid/2.7.STABLE8)
[headers] << Connection: close
[DefaultHttpClient] Authentication required
[DefaultHttpClient] 172.26.27.94:3128 requested authentication
[ProxyAuthenticationStrategy] Authentication schemes in the order of preference: [Basic]
[DefaultHttpClient] Selected authentication options: [BASIC]
[DefaultClientConnection] Connection 0.0.0.0:63344<->172.26.27.94:3128 closed
[DefaultClientConnectionOperator] Connecting to 172.26.27.94:3128
[RequestAddCookies] CookieSpec selected: best-match
[RequestAuthCache] Re-using cached 'basic' auth scheme for http://172.26.27.22:8080
[RequestAuthCache] No credentials for preemptive authentication
[RequestProxyAuthentication] Proxy auth state: CHALLENGED
[RequestProxyAuthentication] Generating response to an authentication challenge using basic scheme
[DefaultHttpClient] Attempt 2 to execute request
[DefaultClientConnection] Sending request: PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg HTTP/1.1
[headers] >> PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg HTTP/1.1
[headers] >> Authorization: Basic eGRldjo0YTFmNmMwOTgyYWRkMWQ0NDg1YjRhMGE4YWMxY2JjMWNiMTA0ODc1
[headers] >> Content-Length: 2
[headers] >> Host: 172.26.27.22:8080
[headers] >> Proxy-Connection: Keep-Alive
[headers] >> User-Agent: Apache-HttpClient/4.2.2 (java 1.5)
[headers] >> Proxy-Authorization: Basic eHRlbmR4LmRuZXByOnF3ZXJ0eQ==
[DefaultClientConnection] Receiving response: HTTP/1.0 201 Created
[headers] << HTTP/1.0 201 Created
[headers] << Content-Length: 0
[headers] << Date: Thu, 08 Nov 2012 10:09:49 GMT
[headers] << X-Cache: MISS from 172.26.27.94
[headers] << X-Cache-Lookup: MISS from 172.26.27.94:3128
[headers] << Via: 1.1 172.26.27.94:3128 (squid/2.7.STABLE8)
[headers] << Connection: keep-alive
[headers] << Proxy-Connection: keep-alive
Squid log
1352370666.778 0 172.26.27.94 TCP_DENIED/407 1870 PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg - NONE/- text/html
1352370671.429 8 172.26.27.94 TCP_MISS/201 282 PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg proxyuser DIRECT/172.26.27.22 -
1352370671.474 0 172.26.27.94 TCP_DENIED/407 1882 PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image2.jpg - NONE/- text/html
1352370671.486 7 172.26.27.94 TCP_MISS/201 282 PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image2.jpg proxyuser DIRECT/172.26.27.22 -
With hardcoded header
There is no 407 error in the java log, and squid log
Squid log
1352370542.016 8 172.26.27.94 TCP_MISS/201 282 PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image1.jpg proxyuser DIRECT/172.26.27.22 -
1352370542.033 7 172.26.27.94 TCP_MISS/201 282 PUT http://172.26.27.22:8080/myapp/rest/2/3/29/image2.jpg proxyuser DIRECT/172.26.27.22 -

The authentication process with an HTTP proxy is described in RFC2616 §14.33 and §14.34 and is exactly as you see with HttpClient. It includes:
the proxy sending a 407 (Proxy Authentication Required) response with a Proxy-Authenticate header containing the challenge applicable to the requested resource,
the client issuing a new request with a Proxy-Authorization header consisting of the credentials containing the authentication information.
Implementing the preemptive authentication with HttpClient is described in this tutorial (section §4.8), and requires prepopulating the authentication cache of the client object.
Unfortunately, their code does not work when preauthenticating to a proxy server. It may be a bit tricky to understand how to get it right, but actually it is as simple as passing a parameter to the BasicScheme constructor:
AuthCache authCache = new BasicAuthCache();
AuthScheme basicAuthScheme = null;
if (isProxy) {
basicAuthScheme = new BasicScheme(ChallengeState.PROXY);
} else {
basicAuthScheme = new BasicScheme(ChallengeState.TARGET);
}
authCache.put(host, basicAuthScheme);
httpContext.setAttribute(ClientContext.AUTH_CACHE, authCache);

It's your AuthScope:
// You set proxyServer here vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
credsProvider.setCredentials(new AuthScope(proxyServer, proxyPort), new UsernamePasswordCredentials(proxyUser, proxyPassword));
httpClient.setCredentialsProvider(credsProvider);
// But here you are looking for the indexof https to determine if it is an SSL proxy
// is the String returned from getProxyServer() above a URL or a host name?
HttpHost proxy = new HttpHost(proxyServer, proxyPort, (proxyServer.indexOf("https") != 0) ? "http" : "https");
httpClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
Buried in my comment above - is the string returned by getProxyServer() a URL or a hostname?
EDIT
I think I have figured out what is going on. Basic Auth has to be sent on every request for it to work. If you are making multiple executions you are going to need to have some mechanism to cache the authentication data. By executing the client in the fashion you are right now, there is no "context" that can store that as one is created on every request.
The other item to look at is the client tutorial sections 4.7 and 4.8. If you are looking to completely eliminate the 407 error followed by a request with the BASIC authentication, then make sure you read section 4.8.
Pay very close attention to the "localcontext" variable they have defined as this acts as a state container for your client.