The previous ConceptNet 5.4 API version returns plain text JSON format (http://conceptnet5.media.mit.edu/data/5.4/c/en/library).
Does someone familiar with the new ConceptNet 5.5 API? Why the query returns html source and not plain text like previous version (http://api.conceptnet.io/related/c/en/library)? I didn't find option to select the plain text JSON format.
Therefore, this Java code is not usable with the new version.
JsonReader jsonReader = Json.createReader(new URL("http://conceptnet5.media.mit.edu/data/5.4/c/en/library").openStream());
The format of the API response depends on the Accept: header that your client sends. (I found this behavior really convenient when Django REST Framework does it, so I implemented it in ConceptNet's API code.) The default response format is JSON.
If you run curl http://api.conceptnet.io/related/c/en/library at the command line, for example, you will see plain, un-indented JSON. If you go there in a Web browser, it's indented and wrapped in HTML so it can be syntax-highlighted and linked. The difference is that the Web browser sends the header Accept: text/html.
I think it's a bug in the JsonReader you're using that it's explicitly sending Accept: text/html and yet it's expecting a JSON response, not HTML. See if you can work around it by configuring the headers it sends.
(If getting the HTML is unavoidable, note that the plain JSON is also present in the HTML, within the <script type="application/ld+json"> tag.)
Related
I understand that to fix the cross-site scripting, I need to validate the user input and encode the output to avoid browser execute malicious data.
However my application is just a pure Rest API which return JSON string and XML string, fortify reported cross-site scripting persistent (stored) because the code will query data from db and return to the response
#Java Code
#PostMapping(path = "${api.abc.endpoint}")
public ResponseEntity processRequest(#RequestBody String requestStr,
HttpServletRequest servletRequest) {
ResponseEntity<String> response = null;
String responseStr = "";
responseStr = processRequest(requestString, servletRequest);
response = ResponseEntity.ok().body(responseStr);
return response; //response can be JSON or XML
}
#Original JSON Response
{
"type":"order",
"responseCode":"001",
"responseText":"Success",
"transDesc":"Value from DB"
}
#Original XML Response
<abc:res xmlns:abc="http://sample.com/abc/">
<type>order</type>
<responseCode>001</responseCode>
<responseText>Success</responseText>
<transDesc>Value from DB</transDesc>
</abc:res>
I try to encode the output string using the OWASP Java Encoder and I got the below encoded string which changed the response format.
#Encoded JSON Response
{\"type\":\"order\",\"responseCode\":\"001\",\"responseText\":\"Success\",\"transDesc\":\"Value from DB\"}
#Encoded XML Response
<data contentType="application/xml;charset=UTF-8" contentLength="241">
<![CDATA[<abc:res xmlns:abc="http://sample.com/abc/"><type>order</type><responseCode>001</responseCode><responseText>Success</responseText><transDesc>Value from DB</type></abc:res>]]></data>
How can I actually fix the cross-site scripting persistent in fortify for JSON string and XML string?
Thanks.
Fortify may be too eager to detect XSS as it assumes any data you produce could end up directly interpreted as HTML. Content sent back to the browser with XML or JSON content types aren't vulnerable to XSS by themselves though. Check that the content-type header being sent back isn't text/html.
The issue may be that a client would read part of the response and output it as is onto the page. The encoding here would be the client's responsibility though as what encoding to use depends on the output context.
Many client-side frameworks will HTML encode data as necessary by default. If you control the client, you should check whether it's doing its own encoding here.
Input validation can help in general too. Either here or in related requests that are writing to the database. Input can be validated depending on what its content should be.
How the above Fortify cross site scripting persistent issue is solved for the database call and sending output as responsentity.
Leaving my solution in case this helps peeps in the future.
My app security team needed fortify to completely resolve the issue.
What worked for me was grabbing all the keys+values in the json and running them through the html encoder function from import org.apache.commons.lang3.StringUtils library.
As the user above mentioned, fortify tries to make sure that the user input it html encoded.
I have a server that handles a POST request with JSON. It also looks and decodes query parameters from the URI. My Java Client currently uses HTTPPost to send across the Json with ContentType application/json.
I wonder whehther URLEncodeUtil method format would be able to accomplish this. Except the documentation mentions
suitable for use as an application/x-www-form-urlencoded list of
parameters in an HTTP PUT or HTTP POST.
So my question is
1. Would this work with ContenType set to application/json.
2. Is there another way to accomplish what the Server requires, ie: have JSON as well as Query parameters encoded in the URI.
There are two official methods of posting form data via the (HTML spec). The pertinent value is application/x-www-form-urlencoded which adds a ? along with the name/value pairs encoded in the URL. If the form method is POST then it will be the first line after the HTTP POST statement.
Everything we do with HTTP in REST web services is valid HTTP, but not for HTML. So the application/json can have a combination of the application/x-www-form-urlencoded style parameters and the JSON payload.
The HTTP request will look something like this:
POST /blog/posts?myparam=Something%20Good&token=donotdothis
Accept: application/json
Content-Type: application/json
Content-Length: 57
{"title":"Hello World!","body":"This is my first post!"}
Also spelled out here: http://www.jsonrpc.org/historical/json-rpc-over-http.html
It's the ? that marks the beginning of extra parameters. So while that is technically legal, it does beg the question why everything you need to post can't be part of your JSON. The downside of this approach is that the query parameters are all part of your HTTP logs and is very visible. You definitely should not use this approach with passwords or any other personally identifiable information. Depending on privacy laws in your country, you want to minimize unnecessary records to make compliance much easier.
I am trying to connect to a website. After While authentication is successful, I get the following response:
{"code": "OK",
"data": {"session_id": "apimanager#taurusseo.com:EdiPNoBS2iYxOsAF9e2ceMbk"},
"user": "apimanager#taurusseo.com"}
What is the quick and easy way to parse through the (pure text) response given above?
Also, if the response is a JSON object, then how do I parse through the response, preferably using GSON library? I just want to use a library that is supported by Google App Engine for Java and GSON is one of those.
Finally, if a response analogous to the above is an XML response, then do I have to model a struct that resembles the response above? Again, is there a quick and easy way to get the response and parse through it, preferably using XML RPC Client? Again, I just want to use a library that is supported by Google App Engine for Java. XML RPC client is just a suggestion, if you can suggest something better then kindly do so.
I am confused as to why you would get back different data types of responses for the same request. However, you should be able to inspect the Content-Type header of the response, to determine if you should parse it as JSON or XML. As you already mentioned, you could use a library such as Gson to handle the response as JSON, or one of the build-in XML APIs (e.g. JAXP) to handle the response as XML.
I get this error while parsing an xml file using JDOM.
What is happening is, I receive a stream of data which is an xml combined with a pdf as an attachment within it. So when I try to create a document of it, this error is thrown.
I tried to print this stream and on the console I get the following, It is with lot of junk chars(the pdf contents) but in Wordpad it looks like -
------=_Part_2_23286828.1296553488632
Content-Type: text/xml; charset=utf-8
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
....
....
....
<Attachment>
<URI>Filename.pdf</URI>
</Attachment>
</SOAP-ENV:Envelope>
------=_Part_2_23286828.1296553488632
Content-Type: application/pdf; name="Filename.pdf"
Content-Transfer-Encoding: binary
Content-ID: </Attachment[1]/URI[1]>
Content-Disposition: attachment; filename="Filename.pdf"
%PDF-1.4
%âãÏÓ
4 0 obj <</Type/XObject/ColorSpace/DeviceRGB/Subtype/Image/BitsPerComponent 8/Width 579/Length 52722/Height 480/Filter/DCTDecode>>stream
ÿØÿà
Please note that the xml between <SOAP-ENV:Envelope> and </SOAP-ENV:Envelope> is well-formed.
How could I go about and create a JDOM document out of it? I guess, by removing the content before and after the xml start/end tags but how in a clean way?
I read that BOMInputStream from Apache IO Commons is helpful but I believe it is in version 2.* and I am using version 1.3.1
I hope this explains my problem, if not pls let me know.
Thank you.
UPDATE
At first I didnt realize it would be this cumbersome.
Actually, I am making a call from one servlet to another(doPost) using HttpURLConnection. The return is in the form of this stream.
Now, I am also trying to explore if in any way I can extract the xml part using some of the methods provided by Http/URLConnection.
Appreciate if anyone could shed some more light on this.
This message conforms to the SOAP with Attachment specification (http://www.w3.org/TR/SOAP-attachments). In java the way to parse these messages is to use an implementation of the SAAJ (Soap with Attachments API for Java: http://download.oracle.com/javaee/5/tutorial/doc/bnbhf.html.) There are a couple of different implementations of SAAJ out there. My personal favorite is the Spring-WS implementation another option is Apache Axiom.
My suggestion to you would be use either Spring-WS or Apache Axis to process this message rather than trying to do it manually from an input stream. Are you trying to do this on the server side or on the client side?
I'm working on setting up a RESTful request for the application I'm working on and I wanted to use xml as the request in the uri instead of allowing the client to supply the parameters in the URI itself.
I'm looking to have the URI like so: someurl/service/request
instead of: someurl/service/request?id={id}&name={name}
I have been searching the web to see what the convention should be when creating the POST request. Can anyone kind of help point me in the right direction on how I should set up this POST request allowing the client to use xml?
Not sure if it is relevant but I'm setting up the server side code in JAVA using the SPRING 3.0 framework. Please let me know if I need to supply more details.
Thanks for your help!!
You can put parameters into the body of the request. They are the same format as appending them to the URL. Eg:
POST /path/script.cgi HTTP/1.0
User-Agent: HTTPTool/1.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 32
home=Cosby&favorite+flavor=flies
You can do that in prototype pretty easily with:
new Ajax.Request('someurl/service', {
method: 'post',
postBody: 'home=Cosby&favorite+flavor=flies',
encoding: 'UTF-8'});
To add your xml file, just append it to your postBody with some sort of delimiter so your cgi knows where parameters end and where xml begins.
I think that's what you were looking for, hope it helps.
You can pass whatever you want in your POST body. So if you want to use XML, you can use XML. Example:
POST /car
Content-Type: text/xml
<car>
<date>10-10-2007<date>
<type>Corvette</type>
</car>
HTTP/1.1 201 CREATED
I think all the REST API frameworks let you easily specify XML in the client request and server response. See Restlet's quick start for an example.