JSON Data Limit - java

We have grails 2.2.4 application running on Tomcat that works with user camera and keystrokes, collects some data on the client side with Javascript and sends using POST.
In the view that collects data we have:
<g:form name="testResultsForm" id="testResultsForm" controller="customer" action="thankYou" method="post">
<h3>Dummy data!</h3>
<input type="text" style="visibility: hidden" name="testResults" id="testResults"/>
<button type="submit" class="btn btn-default">Submit dummy data</button>
</g:form>
In the JS, we assign all camera data to this html element and submit the form:
TestUtils.setValue('testResults', sendData);
$("#testResultsForm").submit();
In the grails controller we have the following line to parse the JSON:
def data = JSON.parse(params.testResults)
Everything works as expected except for when the user takes longer than normal and puts in lots of keystrokes. The errors looks something like:
2014-06-14 01:22:14,323 [http-8443-16] ERROR (org.codehaus.groovy.grails.web.errors.GrailsExceptionResolver) - JSONException occurred when processing request: [POST] /qbcheck/customer/thankYou
Expected a ',' or ']' at character 524288 of {"patkey":"","test_version":"1.4","data_version":"1.3","patientid":"","test_date":"","test_duration":0,"gender":"","dob":"","fov":62,"fps":26,"scale_factor":0,"country":46,"camera_data":{"x":[353,353,353,353,3......
It always fails at character 524288. This led us to investigate that there might be a limit on the amount of data, we looked at Tomcat and found that it allows 2MB data by default using maxpostsize property. Still we updated it to a bigger number just to be sure. Similarly, we tried looking on Grails and JS side but were not able to find any limitation.
Looking for any pointers in this regard. We are able to provide more details as required.

We found that the html "input" has a hard limit of 512 KB. Ideally, we should have received some kind of error/warning when trying to assign more data to an input value through JS. However, that does not happen
So we changed the input which was previously defined as:
<input type="text" style="visibility: hidden" name="testResults" id="testResults"/>
To a text-area:
<textarea style="visibility: hidden" name="testResults" id="testResults"/>
And this allowed us to transfer data greater than 512 KB.

Related

React REST request to Java backend failing

I have a simple React form. I am trying to send the data from this form using Fetch API to my Java backend. Here is my React Form file:
import React, {Component} from 'react';
class Form extends Component {
constructor(props){
super(props);
this.state={value:""};
this.handleChange = this.handleChange.bind(this);
this.handleSubmit = this.handleSubmit.bind(this);
}
handleChange(event){
event.preventDefault();
this.setState({value:event.target.value});
}
handleSubmit(event){
event.preventDefault();
const data = new FormData(event.target);
fetch('http://localhost:8080/add/person', {
method: 'POST',
body: data
});
}
render(){
return(
<form onSubmit={this.handleSubmit}>
<label>Name:
<input type="text" value={this.state.value} onChange={this.handleChange} />
</label>
<input type="submit" value="Submit"/>
</form>
);
}
}
For some reason, the data variable always has an empty JSON when I am in debug mode. In my Java backend, when I receive the request, I am seeing blank form data.
Any ideas as to why I am not able to send data across to my Java backend?
EDIT: I would also like to point out that my frontend is hosted on localhost:3000, while my Java backend server is on localhost:8080
why not just submit your data using the value stored in state?
handleSubmit(event){
event.preventDefault();
const data = this.state.value; //change here
fetch('http://localhost:8080/add/person', {
method: 'POST',
body: data
});
}
UPDATE: in your package.json add "proxy": "http://localhost:8080" if that doesn't work you will have to open it up using something like this but for what ever framework you're using on your backend.
https://www.npmjs.com/package/cors#enabling-cors-pre-flight
http://www.baeldung.com/spring-cors
as #Tadas Antanavicius said your input is also missing a name value. here is a nice and short medium article on the react portion of what you are trying to do with your code. you can even remove your onChange from your input field.
https://medium.com/#everdimension/how-to-handle-forms-with-just-react-ac066c48bd4f
Your problem is unrelated to your backend - the fetch code looks correct.
FormData is not being constructed as you would expect. You can try this out by opening Chrome Devtools' Network tab and watch the request as it goes by: empty request payload.
The problem is that the FormData constructor's argument relies on each input in the form having a name attribute, which you're missing. If you add it, (name="name") your front end should behave as expected:
<input type="text" name="name" value={this.state.value} onChange={this.handleChange} />
EDIT: As per your above conversation, seems like you also have a server side CORS issue. My answer fixes your original question, but yes you'll need to resolve the CORS one as well, the easiest way probably being to refer to the docs of whatever Java framework you're using. It's a very common problem and should be in FAQ.

Filling out a HTML-form with complex name (dot-notation in input-tag) with Java and Jaunt API

-
hey folks,
i am building a Java-tool, trying to automatically fill out some form input elements in an HTML-Page using Java and Jaunt API.
the HTML-Code is like:
<fieldset class = "fieldsetlong">
<legend>searchprofile</legend>
<label for="reference">reference:</label>
<input maxlength="50" name="reference" id="reference" type="text" />
</fieldset>
<fieldset class = "fieldsetlong">
<legend>searchcriteria</legend>
<label for="surname">surname:</label>
<input name="searchprofile.surname" id="surname" type="text" />
</fieldset>
The Java-Code for filling in the "normal" Input-field reference (it works) looks like:
form.set("reference", "123Test");
Unfortunately, I am not able to fill out the fields that use the dot-notation searchprofile.surname in the name
Here's a sample of what i've tried (without success):
form.set("surname", "TestPerson");
form.set("searchprofile.surname", "TestPerson");
form.set("name=\"searchprofile.surname\"", pers.getSurname());
form.set("id=\"surname\"", pers.getSurname());
For each of these commands I get a NotFoundException and don't know whether I can do this with Jaunt.
It would appreciate any kind of help in this regard.
Thanks in advance
Edit - is there a way to reach the dot-notated input-field searchprofile.surname with JSoup?
HTML allows dots in the name-Attribute, but does Jaunt accept this abc.name?
Not sure about Jaunt, never used it before. However Jsoup seems to be a pretty decent library to be used here. I myself have been using Jsoup for a fairly long time and it has been very successful in scraping web pages, filling input form and submit, and of course, HTML parsing!
I've posted a step by step guide to fill in form input fields and submit to server in the following answer: How to login with Jsoup
Basically it works very similar to your code, a very brief example would be:
Connection.Response response = Jsoup.connect(url)
.data("Name", "Value")
.method(Method.POST).execute();
Today, at work the Jaunt solution with
form.set("searchprofile.surname", "TestPerson");
worked like a charm.
I don't know what the problem was earlier but I am glad that it worked.
The HTML allows to use dots and minus, etc. which I misinterpreted as some kind of nested forms or hierarchies but the dot-notation is just a valid name-attribute in HTML.

Jsoup posting modified Document

I'm trying to create a web scraper for my coming android app. Therefore I need to use a simple search form on a website, fill it out and send my results back to the server.
As mentioned in the Jsoup-Cookbook, I scraped the site I needed from the Server and changed the values.
Now I just need to post my modified document back to the server and scrape the resulting page.
As far as I've seen in the Jsoup-API there is no way to post something back, except with the .data-Attribute in Jsoup.connection, which is unfortunately not able to fill out text fields by their id.
Any ideas or workarounds, how to post the modified document, or its parts back to the website ?
You seem to misunderstand how HTTP works in general. It is not true that the entire HTML document with modified input values is been sent from the client to the server. It's more so that the name=value pairs of all input elements are been sent as request parameters. The server will return the desired HTML response then.
For example, if you want to simulate a submit of the following form in Jsoup (you can find the exact HTML form syntax by opening the page with the form in your browser and do a rightclick, View Source)
<form method="post" action="http://example.com/somescript">
<input type="text" name="text1" />
<input type="text" name="text2" />
<input type="hidden" name="hidden1" value="hidden1value" />
<input type="submit" name="button1" value="Submit" />
<input type="submit" name="button2" value="Other button" />
</form>
then you need to construct the request as follows:
Document document = Jsoup.connect("http://example.com/somescript")
.data("text1", "yourText1Value") // Fill the first input field.
.data("text2", "yourText2Value") // Fill the second input field.
.data("hidden1", "hidden1value") // You need to keep it unmodified!
.data("button1", "Submit") // This way the server knows which button was pressed.
.post();
// ...
In some cases you'd also need to send the session cookies back, but that's a subject apart (and a question which has already been asked several times here before; in general, it's easier to use a real HTTP client for this and pass its response through Jsoup#parse()).
See also:
HTTP tutorial
HTTP specification
That's not the way. You should create a POST request (use Apache HTTP Components), get the response and then scrape it with JSoup.

InputStream closed in Apache FileUpload API

This is a very specific question. I hope there's someone here with good knowledge in FileUpload API.
If you have been worked with this API you should know that when the max size of a file has been exceeded the next items cannot be read because the FileSizeLimitExceededException is thrown and if you attempt to call one more time to hasNext() method to iterate to the next item an IOException is thrown because an inputstream is closed and you attempt to read it again.
This scenario is very bad because the next parameters cannot be read. Supose this form:
<form action="..." method="post" enctype="multipart/form-data">
<input type="hidden" name="param1" value="foo"/>
<input type="file" name="myFile"/><br/>
<input type="hidden" name="param2" value="bar"/>
<input type="submit"/>
</form>
Now the file exceeds the maximum size. Result:
"param1" is read.
"myFile" generates FileSizeLimitExceededException.
"param2" is not read.
I'm trying to rewrite part of the API -with no luck- because I want to read the next items ("param2" in this case).
Anyone has had the same problem? How have you fixed it?
Thanks.
EDIT: Well, I've implemented the solution proposed by BalusC. For the future:
FileUpload wrapper
FileUpload wrapper usage
If you consider that this wrapper needs anything else just say it. I don't accept it as a real solution because the ideal is to modify and fix the API itself.
A way without hacking the API would be so set the upload file size limit to "unlimited" and examine the upload file size yourself afterwards by FileItem#getSize(). This may only take longer before the enduser get feedback about that, because the entire file needs to be read fully first (but you need to do it anyway in order to get all subsequent items).
As a completely different alternative, you may want to look at the new HTML5 File API which offers you a way to examine the file size using JavaScript before the request is actually been sent.
var size = document.getElementById("fileFieldId").files[0].size;
// ...

Using PrintWriter, I am getting Chinese junk characters in browser

I am using PrintWriter as follows to get the output in the browser:
PrintWriter pw = response.getwriter();
StringBuffer sb = getTextFromDatabase();
pw.print(sb);
However, this prints the following Chinese junk characters:
格㸳潃浭湥獴⼼㍨‾琼扡敬㰾牴戠捧汯牯✽䔣䔷䔷❆㰾摴倾獯整⁤湏›〱㈭ⴷ〲〱ㄠ㨴㌰㔺਱‬祂›教桳慷瑮丠祡歡⠊湹祡歡捀獩潣挮浯਩硅散汬湥㱴琯㹤⼼牴㰾牴戠捧汯牯✽䔣䔷䔷❆㰾摴㰾琯㹤⼼牴㰾牴戠捧汯牯✽䔣䔷䔷❆㰾摴倾獯整⁤湏›〱㈭ⴷ〲〱ㄠ㨴㐰ㄺ਱‬祂›教桳慷瑮丠祡歡⠊湹祡歡捀獩潣挮浯਩敶祲朠潯㱤琯㹤⼼牴㰾牴戠捧汯牯✽䔣䔷䔷❆㰾摴㰾琯㹤⼼牴㰾牴戠捧汯牯✽䔣䔷䔷❆㰾摴倾獯整⁤湏›〱㈭ⴷ〲〱ㄠ㨴㜱㌺ਸ਼‬祂›教桳慷瑮丠祡歡⠊湹祡歡捀獩潣挮浯਩桔獩椠⁳潴琠獥㱴琯㹤⼼牴㰾琯扡敬㰾牢⼠‾格㸳潐瑳夠畯⁲潃浭湥㱴栯㸳㰠潦浲愠瑣潩㵮䌢浯敭瑮即牥汶瑥•敭桴摯∽敧≴渠浡㵥挢浯敭瑮潆浲•湯畳浢瑩∽爠瑥牵慖楬慤整潆浲⤨∻‾琼扡敬†眠摩桴∽〳∰栠楥桧㵴㌢〰㸢ठ琼㹲琼㹤氼扡汥映牯∽慮敭㸢潃浭湥㩴猼慰汣獡㵳洢湡呤汃獡≳⨾⼼灳湡㰾氯扡汥㰾牢㸯琼硥慴敲⁡慮敭∽潣瑮湥≴椠㵤挢浯敭瑮硔䅴敲≡挠慬獳∽整瑸牡慥氠牡敧•潣獬∽㠲•潲獷∽∶㸠⼼整瑸牡慥㰾琯㹤⼼牴㰾牴㰾摴㰾慬敢潦㵲渢浡≥举浡㩥猼慰汣獡㵳洢湡呤汃獡≳⨾⼼灳湡㰾氯扡汥㰾牢㸯椼灮瑵椠㵤渢浡≥琠灹㵥琢硥≴渠浡㵥渢浡≥挠慬獳∽慮敭•慶畬㵥∢洠硡敬杮桴∽㔲∵†楳敺∽㘳⼢㰾琯㹤⼼牴㰾牴㰾摴㰾慬敢潦㵲攢慭汩㸢ⵅ慍汩㰺灳湡挠慬獳∽慭摮䍔慬獳㸢㰪猯慰㹮⼼慬敢㹬戼⽲㰾湩異⁴摩∽浥楡≬琠灹㵥琢硥≴渠浡㵥攢慭汩•汣獡㵳攢慭汩•慶畬㵥∢洠硡敬杮桴∽㔲∵†楳敺∽㘳⼢㰾琯㹤⼼牴㰾牴㰾摴㰾湩異⁴琠灹㵥猢扵業≴†慮敭∽潰瑳•慶畬㵥倢獯≴㸯⼼摴㰾琯㹲⼼慴汢㹥⼼潦浲
I tried to use String instead of StringBuffer, but that didn't help. I also tried to set the content type header as follows
response.setContentType("text/html;charset=UTF-8");
before getting the response writer, but that did also not help.
In the DB there are no issues with the data as I have used the same data for 2 different purposes. In one I get correct output, but in other I get the above junk. I have used the above code in JSP using scriptlets. I have also given content type for the JSP.
Getting Chinese characters as Mojibake indicates that you're incorrectly showing UTF-16LE data as UTF-8. UTF16-LE stores each character in 4 bytes. In UTF-8, the 4-byte panels contains usually CJK (Chinese/Japanese/Korean) characters.
To fix this, you need to either show the data as UTF-16LE or to have stored the data in the DB as UTF-8 from the beginning on. Since you're attempting to display them as UTF-8, I think that your DB has to be reconfigured/converted to use UTF-8 instead of UTF-16LE.
Unrelated to the concrete problem, storing HTML (that was what those characters originally represent) in a database is really a bad idea ;) This was the original content:
<h3>Comments</h3> <table><tr bgcolor='#E7E7EF'><td>Posted On: 10-27-2010 14:03:51
, By: Yeshwant Nayak
(ynayak#cisco.com)
Excellent</td></tr><tr bgcolor='#E7E7EF'><td></td></tr><tr bgcolor='#E7E7EF'><td>Posted On: 10-27-2010 14:04:11
, By: Yeshwant Nayak
(ynayak#cisco.com)
very good</td></tr><tr bgcolor='#E7E7EF'><td></td></tr><tr bgcolor='#E7E7EF'><td>Posted On: 10-27-2010 14:17:36
, By: Yeshwant Nayak
(ynayak#cisco.com)
This is to test</td></tr></table><br /> <h3>Post Your Comment</h3> <form action="CommentsServlet" method="get" name="commentForm" onsubmit=" return ValidateForm();"> <table width="300" height="300"> <tr><td><label for="name">Comment:<span class="mandTClass">*</span></label><br/><textarea name="content" id="commentTxtArea" class="textarea large" cols="28" rows="6" ></textarea></td></tr><tr><td><label for="name">Name:<span class="mandTClass">*</span></label><br/><input id="name" type="text" name="name" class="name" value="" maxlength="255" size="36"/></td></tr><tr><td><label for="email">E-Mail:<span class="mandTClass">*</span></label><br/><input id="email" type="text" name="email" class="email" value="" maxlength="255" size="36"/></td></tr><tr><td><input type="submit" name="post" value="Post"/></td></tr></table></form
Here's how you can turn this incorrectly encoded Chinese back to normal characters:
String incorrect = "格㸳潃浭湥獴⼼㍨‾琼扡敬㰾牴戠捧汯";
String original = new String(incorrect.getBytes("UTF-16LE"), "UTF-8");
Note that this should not be used as solution! It was just posted as an evidence of the root cause of the problem.
Clearly, you have some kind of encoding problem here, but my guess is it is on the server or database side, not in the browser.
In the DB there are no issues with the data as i have used the same data for 2 different options,but in one i get correct output n in other junk.
I don't find that argument convincing. In fact, I think you may be overlooking the real cause of the problem.
What I think you need to do is add some server-side logging to capture what is actually in that StringBuffer that you are sending to the PrintWriter
Also, look at what is different about the way that the server side handles the "2 different options". (What do you mean by that phrase?).
Finally, please provide some REAL code, not just 3 line snippets that won't compile.

Categories

Resources