I am upgrading S3Client in aws cloud service to S3AsyncClient.
I have this function to convert to async:
public PutObjectResponse uploadFileByUrl(String fileUrl, String builderId, PbModel category, String categoryId)
URL url = new URL(fileUrl);
String[] fileNameArray = url.getFile().split("\\.");
var uniqueFileName = prepareFileName(fileNameArray[fileNameArray.length -1]);
URLConnection connection = url.openConnection();
long contentSize = connection.getContentLengthLong();
InputStream inputStream = connection.getInputStream();
return s3Client.putObject(myObjectRequestBuild, RequestBody.fromInputStream(inputStream, contentSize));
}
I have this function to convert to async:
public CompletableFuture<PutObjectResponse> uploadFileByUrl(String fileUrl, String builderId, PbModel category, String categoryId)
URL url = new URL(fileUrl);
String[] fileNameArray = url.getFile().split("\\.");
var uniqueFileName = prepareFileName(fileNameArray[fileNameArray.length -1]);
URLConnection connection = url.openConnection();
long contentSize = connection.getContentLengthLong();
InputStream inputStream = connection.getInputStream();
return asyncClient.putObject(myObjectRequestBuild, AsyncRequestBody.fromPublisher(???));
}
As you can see in second method above when I convert the first function above to async I need to use AsyncRequestBody instead of RequestBody.
AsyncRequestBody doesn't have fromInputStream method but it have fromPublisher method that I want to use, the fromPublisher method get as parameter type of Publisher.
So my question is how to convert my inputStream into an Publisher?
AsyncRequestBody doesn't have fromInputStream method
Correct however, it has a plethora of other ways to create an AsyncRequestBody:
fromByteBuffer(ByteBuffer byteBuffer)
fromBytes(byte[] bytes)
fromFile(File file)
fromFile(Path path)
fromPublisher(org.reactivestreams.Publisher<ByteBuffer> publisher)
fromString(String string)
fromString(String string, Charset cs)
Considering the above, you have a few solutions:
Convert the InputStream to a byte array using IOUtils.toByteArray(inputStream) (or in Java 9+, inputStream.readAllBytes()) and then use fromBytes directly
As above but then convert the byte[] to a ByteBuffer using ByteBuffer.wrap(byteArray) and then use fromByteBuffer
Create a new File object specifying a filename, copy the contents of the InputStream to the file's FileOutputStream using IOUtils.copy(), and then use fromFile(File file)
As above but instead of providing the File object, provide its path to fromFile(Path path) after you've written to it's FileOutputStream
Convert the InputSteam to a Publisher<ByteArray> using DataBufferUtils.readByteChannel from the Spring Framework, Akka StreamConverters etc. and then use fromPublisher
Convert the InputStream to a UTF-8 encoded String then use fromString(String string) (no need to specify the Charset if it is UTF-8 encoded)
Convert the InputStream to a non-UTF-8 encoded String then use fromString(String string, Charset cs), specifying the CharSet
Of course, some of the above are plain redundant in your case e.g. fromFile(Path path) is for files that you've already stored & converting the InputSteam to a Publisher<ByteArray> is going to be a pain but I've included all possible solutions for completeness.
I would approach this using solution #1, resulting in the cleanest, simplest code out of the above.
Convert the InputStream to byte[] using inputStream.readAllBytes() and then use AsyncRequestBody.fromBytes(...).
This should work:
public CompletableFuture<PutObjectResponse> uploadFileByUrl(String fileUrl, String builderId, PbModel category, String categoryId)
URL url = new URL(fileUrl);
String[] fileNameArray = url.getFile().split("\\.");
var uniqueFileName = prepareFileName(fileNameArray[fileNameArray.length -1]);
URLConnection connection = url.openConnection();
long contentSize = connection.getContentLengthLong();
InputStream inputStream = connection.getInputStream();
byte[] fileByteArray = inputStream.readAllBytes();
return asyncClient.putObject(myObjectRequestBuild, AsyncRequestBody.fromBytes(fileByteArray));
}
Related
I have http response object "httpResponse" which is instance of class javax.ws.rs.core.Response. I have to convert it into a human readable string object for debugging. How can I achieve this?
final InputStream inputStream = (InputStream) httpResponse.getEntity();
final ByteArrayDataSource dataSource = new ByteArrayDataSource(inputStream, ContentType.MULTIPART_FORM_DATA.getMimeType());
final MimeMultipart multipartResponse = new MimeMultipart(dataSource);
I tried doing the below using the inputStream object. The output is all binary. Do I need to do base64 decoding? Thanks.
IOUtils.toString(inputStream, StandardCharsets.UTF_8))
As you said you need decoding.
// string to byte[]
byte[] bytes = "hello".getBytes(StandardCharsets.UTF_8);
// byte[] to string
String s = new String(bytes, StandardCharsets.UTF_8);
Source:
https://mkyong.com/java/how-do-convert-byte-array-to-string-in-java/
Here it is explained not only the conversion from bytes to text, but also the conversion from bytes to image or any kind of binary type.
I use this open-open source library for creating thumbnails of images. My task is to convert File to Base64 String and then send this String as request parameter to REST Service. I convert file to Base64 String in client side by using this function:
public String convertToBase64(File file) throws IOException {
byte[] bytes = new byte[(int)file.length()];
FileInputStream fileInputStream = new FileInputStream(file);
fileInputStream.read(bytes);
return new String(Base64.encodeBase64(bytes), "UTF-8");
}
Then i retrieve this encodedString and send this String as request parameter to REST Service.
In REST Service i decode this String by using this function:
public static void uploadBase64Image(String file, String filename) throws Exception {
filename = filename.replaceAll(" ", "_");
byte[] data = DatatypeConverter.parseBase64Binary
(file.replaceFirst("data:image/jpg;base64,", "")
.replaceFirst("data:image/jpeg;base64,", "")
.replaceFirst("data:image/png;base64,", ""));
File tempFile = File.createTempFile("base64", filename);
FileUtils.writeByteArrayToFile(tempFile, data);
FileEntity fe = FileEntity.processAndCompressFile(tempFile, "pic");
fe.save();
renderJSON(fe.getJson());
}
The DatatypeConverter is located under package javax.xml.bind;
The problem is occurs when i try to make a thumbnail of this tempFile. The part of code where i try to make a thumbnail:
private static String compressFileAndGetFilenameWithExt(File originalFile, int height, String filename, String fileExt) throws Exception {
StringBuilder newFileName = new StringBuilder(filename)
.append("_")
.append(height)
.append(fileExt);
File compressedFile = new File(filesLocation + newFileName.toString());
Thumbnails.Builder<File> builder = Thumbnails.of(originalFile);
builder = builder.height(height);
builder.toFile(compressedFile);
return newFileName.toString();
}
I get IIOException occurred : Error reading PNG metadata in this line of code: builder.toFile(compressedFile);
Can someone explain and help me to solve this problem?
I found the solution of bag. This is because i don't configure headers to take file
There is a way to convert an InputStream to a String, and encode it to base64, right?
In my function, I get InputStream parameter, and need to insert it into the BLOB field in my Oracle database table.
Is there a way to do that?
(My database object contains string field to save the image, but I don't find any way to convert the InputStream to string in base 64 format.)
There is nice way to do this is using IOUtils to convert the InputStream into a Byte Array...
something like
InputStream is;
byte[] bytes = IOUtils.toByteArray(is);
Here you can use Base64 to convert Byte Array to String.
Sample Code
String encoded = Base64.getEncoder().encodeToString(bytes);
Now you can use your String.
You can try something like this using the Base64 API.
InputStream finput = new FileInputStream(file);
byte[] imageBytes = new byte[(int)file.length()];
finput.read(imageBytes, 0, imageBytes.length);
finput.close();
String imageStr = Base64.encodeBase64String(imageBytes);
Use this:
http://commons.apache.org/proper/commons-codec/archives/1.9/apidocs/org/apache/commons/codec/binary/Base64.html
The simplest way would be to use IOUtils from apache-commons to do that:
String result= IOUtils.toString(inputStream, ENCODING);
From the documentation:
toString(byte[] input, String encoding)
Gets the contents of a byte[] as a String using the specified character encoding.
After that To Encode/Decode in Base64:
// Encode
String resultBase64Encoded = Base64.getEncoder().encodeToString(result.getBytes("utf-8"));
// Decode
byte[] asBytes = Base64.getDecoder().decode(resultBase64Encoded);
String resultAsStringAgain= String(asBytes, "utf-8")
Note: I'm assuming you use JDK 8 for the Encode/Decode part.
Apparently the OP wants just to persist an InputStream to the DB. You can do that directly using JDBC:
InputStream inputStream = ......;
String sql = "INSERT INTO TABLE_NAME(COLUMN_NAME) values (?)";
PreparedStatement statement = connection.prepareStatement(sql);
statement.setBlob(1, inputStream);
statement.executeUpdate();
As I mentioned, you shouldn't use String for binary data. The base64-encoded data can be stored as a String though. But since your database column is a blob, I would continue to work with a byte[].
Use IOUtils to get a byte[] from the InputStream:
byte[] bytes = IOUtils.toByteArray(yourInputStream);
byte[] encoded = java.util.Base64.getEncoder().encode(bytes);
Then write it to the database. Writing a blob using jdbc and a PreparedStatement looks like this:
yourPreparedStatement.setBytes(nIndex, encoded);
In case someone is looking for solution without external libraries. Java8 adds Base64 utility class which contains getEncoder() method. Java9 adds fancy method on InputStream called readAllBytes() Link to api
So now you without any external libraries your code may look like
import java.io.InputStream;
import java.util.Base64;
public String encode(InputStream stream) throws IOException {
final var bytes = stream.readAllBytes();
return Base64.getEncoder().encodeToString(bytes);
}
I have this issue with GZIP compression:
I need to send by POST method a huge JSON string, which is too big to be accept like URL (Ex: http://localhost/app/send/JSON STRING ENCODED BY BASE64), than it result in HTTP error 403
so, I need to compress my json and I found a way to do it with GZIP compression, which I can decompress with gzdecode() in PHP.
but it doesn't work...
my functions compress() and decompress() works fine inside my Java App, but when I send it to webservice, something goes wrong and gzdecode() doesn't work.
I have no idea what I missing, I need some help
functions used in java app (client)
public String Post(){
String retorno = "";
String u = compress(getInput());
u = URLEncoder.encode(URLEncoder.encode(u, "UTF-8"));
URL uri = new URL(url + u);
HttpURLConnection conn = (HttpURLConnection) uri.openConnection();
conn.setDoOutput(false);
conn.setRequestMethod(getMethod());
conn.setRequestProperty("Content-encoding", "gzip");
conn.setRequestProperty("Content-type", "application/octet-stream");
BufferedReader buffer = new BufferedReader(
new InputStreamReader((conn.getInputStream())));
String r = "";
while ((r = buffer.readLine()) != null) {
retorno = r + "\n";
}
return retorno;
}
GZIP compress function (client)
public static String compress(String str) throws IOException {
byte[] blockcopy = ByteBuffer
.allocate(4)
.order(java.nio.ByteOrder.LITTLE_ENDIAN)
.putInt(str.length())
.array();
ByteArrayOutputStream os = new ByteArrayOutputStream(str.length());
GZIPOutputStream gos = new GZIPOutputStream(os);
gos.write(str.getBytes());
gos.close();
os.close();
byte[] compressed = new byte[4 + os.toByteArray().length];
System.arraycopy(blockcopy, 0, compressed, 0, 4);
System.arraycopy(os.toByteArray(), 0, compressed, 4,
os.toByteArray().length);
return Base64.encode(compressed);
}
method php used to receive a URL (server, using Slim/PHP Framework)
init::$app->post('/enviar/:obj/', function( $obj ) {
$dec = base64_decode(urldecode( $obj ));//decode url and decode base64 tostring
$dec = gzdecode($dec);//here is my problem, gzdecode() doesn't work
}
post method
public Sender() throws JSONException {
//
url = "http://192.168.0.25/api/index.php/enviar/";
method = "POST";
output = true;
//
}
As noticed in some of the comments.
Bigger data should be send as a POST request instead of GET. URL params should be used only for single variables. As you noticed the URL length is limited to few kB and it's not very good idea to send larger data this way (even though GZIP compressed).
Your GZIP compression code seems to be wrong. Please try this:
public static String compress(String str) throws IOException {
ByteArrayOutputStream os = new ByteArrayOutputStream(str.length());
GZIPOutputStream gos = new GZIPOutputStream(os);
gos.write(str.getBytes());
os.close();
gos.close();
return Base64.encodeToString(os.toByteArray(),Base64.DEFAULT);
}
I'm trying to download a file from a given URL which may or may not be a direct link to the file. Does anyone know how I can detect the filename to write to if the URL is an indirect link (i.e http://www.example.com/download.php?getFile=1) ?
It is no problem if the URL is a direct link to extract the filename from the URL and start writing to the extracted filename but with a redirect link the only method I have found so far is to write to an arbitrary filename - foo.txt - and then try and work with that. Problem is I really need the filename (and extension) to be correct.
A sample of the code I am using is: (the section in the 'else' clause is neither finished nor working):
public static boolean dlFile(String URL, String dest){
try{
URL grab = new URL(URL);
ReadableByteChannel rbc = Channels.newChannel(grab.openStream());
String fnRE = ".*/([a-zA-Z0-9\\-\\._]+)$";
Pattern pattern = Pattern.compile(fnRE);
Matcher matcher = pattern.matcher(URL);
String fName = "";
if(matcher.find()) fName = matcher.group(1);
else { //filename cannot be extracted - do something here - below doesn't work raises MalformedURLExcpetion
URL foo = new URL(URL);
HttpURLConnection fooConnection = (HttpURLConnection) foo.openConnection();
URL secondFoo = new URL(fooConnection.getHeaderField("Location"));
System.out.println("Redirect URL: "+secondFoo);
fooConnection.setInstanceFollowRedirects(false);
URLConnection fooURL = secondFoo.openConnection();
}
System.out.println("Connection to "+URL+" established!");
if(dest.endsWith("/")){}
else dest+="/";
System.out.println("Writing "+fName+" to "+dest);
FileOutputStream fos = new FileOutputStream(dest+fName);
fos.getChannel().transferFrom(rbc, 0, 1 << 24);
I am sure there must be a simple way to get the filename from the headers or something like that but I cannot work out how to get it. Thanks in advance,
Assuming the response has a "Location" header field, I was able to obtain the direct link to a url containing multiple redirects like this:
String location = "http://www.example.com/download.php?getFile=1";
HttpURLConnection connection = null;
for (;;) {
URL url = new URL(location);
connection = (HttpURLConnection) url.openConnection();
connection.setInstanceFollowRedirects(false);
String redirectLocation = connection.getHeaderField("Location");
if (redirectLocation == null) break;
location = redirectLocation;
}
//and finally:
String fileName = location.substring(location.lastIndexOf('/') + 1, location.length());
I think its better to use Java Jsoup library, then use the below method:
public static void downloadFileJsoup(String URL, String PATH) throws IOException {
Response res = Jsoup.connect(URL)
.userAgent("Mozilla")
.timeout(30000)
.followRedirects(true)
.ignoreContentType(true)
.maxBodySize(20000000)//Increase value if download is more than 20MB
.execute();
String remoteFilename=res.header("Content-Disposition").replaceFirst("(?i)^.*filename=\"?([^\"]+)\"?.*$", "$1");
String filename = PATH + remoteFilename;
FileOutputStream out = (new FileOutputStream(new java.io.File(filename)));
out.write( res.bodyAsBytes());
out.close();
}
No, in general no way. The response does'nt contain that information normally, since you do not add any own protocol information to the data stream (in case you can control the server).
Anyway, you ask for the file name extension. Maybe with the correct content-type you are done.