Problem: Trying to upload bigger than ~ 50 mb files, $_FILES return empty array.
I try change limits on php (a lot of combination) but it useleass..
my class at java:
public void upload() throws Exception {
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(this.url);
try {
FileBody bin = new FileBody(this.file);
#SuppressWarnings("deprecation")
MultipartEntity reqEntity = new MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);
reqEntity.addPart("file", bin);
httppost.setEntity(reqEntity);
System.out.println("Requesting : " + httppost.getRequestLine());
ResponseHandler<String> responseHandler = new BasicResponseHandler();
String responseBody = httpclient.execute(httppost, responseHandler);
System.out.println("responseBody : " + responseBody);
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
httpclient.getConnectionManager().shutdown();
}
}
my php code to print array for respawn:
<?php
ini_set('file_uploads', 1);
ini_set('max_file_uploads', '9999999999M');
ini_set('post_max_size', '2000M');
ini_set('max_input_time', 3600);
ini_set('max_execution_time', 3600);
print_r($_FILES);
?>
You could try to add the mime type of the file when you create the FileBody. For example:
FileBody bin = new FileBody(this.file, "application/octect-stream");
Another solution is to use the HttpURLConnection client that is easier to use and less error prone. Also, Google recommends to use it for new applications.
One side or the other is breaking the http connection being used by the POST action.
Do you know which side ( client or server )
If you do not know , maybe you should try to rule out the server by using CURL to do the M-Multipart Post of one of your big ( +50 ) files.
Invoke CURL to upload and add the -v switch for verbose log so you can analyse the process.
Android POST action on files of 50 - 100 meg should work fine.
Since you are doing big uploads , you would want a client that you can send headers "expect 100 Continue" and where you can employ chunked transfer encoding along with a client side thread that can provide data to a buffer that is wrapped by something like a byteArrayEntity
apache or nginX on std config should handle this fine. If your php process is abending then maybe you need to check phpinfo.
Related
I am trying to implement the MOT history API https://dvsa.github.io/mot-history-api-documentation/ and they give an example using CURL which works with the supplied api key successfully when using an online CURL tool.
I am trying to implement this in Android and realise I have to use something like HttpPost rather than CURL, this is my code:
//Tried with full URL and by adding the registration as a header.
//HttpPost httpPost = new HttpPost("https://beta.check-mot.service.gov.uk/trade/vehicles/mot-tests?registration=" + reg_selected);
HttpPost httpPost = new HttpPost("https://beta.check-mot.service.gov.uk/trade/vehicles/mot-tests");
httpPost.addHeader("Content-Type", "application/json");
httpPost.addHeader("Accept", "application/json+v6");
httpPost.addHeader("x-api-key", "abcdefgh123456");
httpPost.addHeader("registration", reg_selected);
StringEntity entity = new StringEntity(jsonObj.toString(), HTTP.UTF_8);
httpPost.setEntity(entity);
HttpClient client = new DefaultHttpClient();
try {
HttpResponse response = client.execute(httpPost);
if (response.getStatusLine().getStatusCode() == 200) {
InputStream inputStream = response.getEntity().getContent();
bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
String readLine = bufferedReader.readLine();
String jsonStr = readLine;
JSONObject myJsonObj = new JSONObject(jsonStr);
}else if (response.getStatusLine().getStatusCode() == 400){
//Bad Request Invalid data in the request. Check your URL and parameters
error_text = "Bad Request";
}else if (response.getStatusLine().getStatusCode() == 403){
//Unauthorised – The x-api-key is missing or invalid in the header
error_text = "Authentication error"; //<<<< FAILS HERE 403
}
response.getStatusLine().getStatusCode() returns • "403 – Unauthorised – The x-api-key is missing or invalid in the header".
However the x-api-key that I use works correctly with the online CURL test so the actual key is correct but how I am adding it to my android code request must be invalid or similar.
Can anyone throw any light as to the correct way to convert the CURL into Android java so that the server does not return 403?
Thanks
It's easy to do with Jsoup:
// CREATE CONNECTION
Connection conn=Jsoup.connect("URL_GOES_HERE");
// ADD POST/FORM DATA
conn.data("KEY", "VALUE");
// ADD HEADERS HERE
conn.header("KEY", "VALUE");
// SET METHOD AS POST
conn.method(Connection.Method.POST);
// ACCEPT RESPONDING CONTENT TYPE
conn.ignoreContentType(true);
try
{
// GET RESPONSE
String response = conn.execute().body();
// USE RESPONSE HERE
// CREATE JSON OBJECT OR ANYTHING...
} catch(HttpStatusException e)
{
int status = e.getStatusCode();
// HANDLE HTTP ERROR HERE
} catch (IOException e)
{
// HANDLE IO ERRORS HERE
}
Ps: I guess you are confused with Header and Post Data. The key etc (Credentials) must be used as Post Data and Content Type etc as Header.
I have a very large json file (about 500MB) that I am trying to upload using REST and java. It works using curl like this -
curl -H "content-type: application/json" --data-binary #2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations
However, I get "Software caused connection abort: socket write error" when I do it using REST like this-
String filePath = "C:\\2018-02-28.json";
String filename = "2018-02-28.json";
File uploadedFile = new File("C:\\2018-02-28.json");
try {
// HttpClient httpclient = HttpClientBuilder.create().build();
String authHeader = authToken();
HttpEntity entity = MultipartEntityBuilder
.create()
.addTextBody("name", "fileDate")
.addTextBody("fileName", filename)
.addTextBody("Content-Type", "application/json")
.addBinaryBody("fileData", new File(filePath), ContentType.create("application/json"), filename)
.build();
HttpClient httpClient = HttpClients.custom()
.setConnectionTimeToLive(2700, TimeUnit.SECONDS)
.setMaxConnTotal(400).setMaxConnPerRoute(400)
.setDefaultRequestConfig(RequestConfig.custom()
.setSocketTimeout(30000).setConnectTimeout(5000).build())
.setRetryHandler(new DefaultHttpRequestRetryHandler(5, true))
.build();
HttpPost request = new HttpPost("http://md:8086/InputGateway/rest/input-gateway-service/ABC/invocations");
request.setEntity(entity);
request.setHeader(HttpHeaders.AUTHORIZATION, authHeader);
HttpResponse resp = httpClient.execute(request);
HttpEntity entity1 = resp.getEntity();
System.out.println(EntityUtils.toString(entity1, "utf-8"));
System.out.println("File has been Uploaded successfully: " + uploadedFile);
} catch (Exception ex) {
throw new Exception( ex.toString());
}
}
what am I doing wrong here
curl -H "content-type: application/json" --data-binary #2018-02-28.json http://md01:8086/Gateway/rest/gateway-service/ABC/invocations
One thing you may want to do is run this command in verbose mode, to see what is really happening. It is possible that the example that works because curl is using an Expect header that allows the server to prepare for the data dump.
A packet analyzer, to see if the remote server is actually sending you a RST, or if something else is going on in the network stack.
(Obvious, but just in case: try using the same code to send a smaller file. Make sure that the size it the limiting factor. Doing a binary search to find out how big a file is acceptable might provide an extra clue.)
I use the following code to send binary data to CGI C++ program from Android app (it is called in doInBackground within AsyncTask):
public static HttpResponse makeRequestPost(String uri, MyObject obj) {
try {
HttpPost httpPost = new HttpPost(uri);
ByteArrayOutputStream stream = new ByteArrayOutputStream();
DataOutputStream dataStream = new DataOutputStream(stream);
obj.writeRawObject(dataStream);
byte[] byteArray = stream.toByteArray();
ByteArrayEntity entity = new ByteArrayEntity(byteArray);
entity.setContentType("application/octet-stream");
entity.setChunked(true);
httpPost.setEntity(entity);
return new DefaultHttpClient().execute(httpPost);
} catch (UnsupportedEncodingException e) {
Log.e(TAG, "UnsupportedEncodingException. " + e.getMessage());
} catch (ClientProtocolException e) {
Log.e(TAG, "ClientProtocolException. " + e.getMessage());
} catch (IOException e) {
Log.e(TAG, "IOException. " + e.getMessage());
}
return null;
}
I tested the same server using CURL with data written to a file using:
obj.writeRawObject(dataStream);
Also, both byteArray and entity variables in the code above contain correct data before calling execute(httpPost).
For some reason, on the server side
char* lenstr = getenv("CONTENT_LENGTH");
returns NULL when I run the app. Data I am trying to send are from a few bytes to 30 kB. It doesn't get them on the server side regardless the size. App can communicate with the server but do not attach a binary data. Is it something I am missing?
Something like this works from CURL:
curl -i -H "Content-Type:application/octet-stream" -X POST --data-binary #file_with_my_obj.bin <url>
Appreciate any ideas.
Can you try using binary/octet-stream instead of application/octet-stream in setContentType ?
I want to save a document in CouchDB server for that I am writing my own PUT request. I can do so using curl command:
> curl -vX PUT http://127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af/artwork.jpg?rev=2-2739352689 --data-binary #artwork.jpg -H "Content-Type: image/jpg"
How can I write the same in PUT Request form in Java like:
HttpPut request = new HttpPut(url);
StringEntity stringEntity = null;
try {
stringEntity = new StringEntity(
(new JSONObject(map)).toString());
} catch (UnsupportedEncodingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
stringEntity.setContentType(new BasicHeader(HTTP.CONTENT_TYPE,
"application/json"));
request.setHeader("Content-type", "application/json");
request.setEntity(stringEntity);
Any idea, how to write such a put request as written in Curl command given above.
You need to deal with binary data, I was just doing this today and answered another post:
Couchdb Upload Image via Httpclient
I'm trying out the twitter streaming api. I could succesfully filter tweets by using curl, as stated here:
curl -d #tracking http://stream.twitter.com/1/statuses/filter.json -u <user>:<pass>
where tracking is a plain file with the content:
track=Berlin
Now I tried to do the same thing in JavaSE, using Apache's HTTPComponents:
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(<user>, <pass>);
DefaultHttpClient httpClient = new DefaultHttpClient();
httpClient.getCredentialsProvider().setCredentials(AuthScope.ANY, creds);
HttpPost httpPost = new HttpPost("http://stream.twitter.com/1/statuses/filter.json");
HttpParams params = new BasicHttpParams();
params = params.setParameter("track", "Berlin");
httpPost.setParams(params);
try {
HttpResponse httpResponse = httpClient.execute(httpPost);
HttpEntity entity = httpResponse.getEntity();
if (entity != null) {
InputStream instream = entity.getContent();
String t;
BufferedReader br = new BufferedReader(new InputStreamReader(instream));
while(true) {
t = br.readLine();
if(t != null) {
linkedQueue.offer(t);
}
}
}
} catch (IOException ioe) {
System.out.println(ioe.getMessage());
}
finally{
httpClient.getConnectionManager().shutdown();
}
When I run that, I get:
No filter parameters found. Expect at least one parameter: follow track
as a single entry in my linkedQueue. Seems the api wants the parameter in a different form, but cannot find any hint in the documentation. Can somebody share some experiences with the api or see any other problem with the code? Thanks!
EDIT
Putting the filter parameter into the params was a bad idea. As it's post data, it needs to be defined as an Entity before the request is being made:
StringEntity postEntity = new StringEntity("track=Berlin", "UTF-8");
postEntity.setContentType("application/x-www-form-urlencoded");
httpPost.setEntity(postEntity);
That's what I was doing wrong. Thanks Brian!
I suspect you need to post the data as the contents of your HTTP post. The man page for curl -d says:
(HTTP) Sends the specified data in a
POST request to the HTTP server, in
the same way that a browser does when
a user has filled in an HTML form and
presses the submit button. This will
cause curl to pass the data to the
server using the content-type
application/x-www-form-urlencoded.
so I believe you have to set that content type and put the contents of the tracking file in the body of your post.