I am trying to download binary file using OkHttp with progress.
The file get download properly when BUFFER_SIZE is 1.
However file get corrupted when I set BUFFER_SIZE to 1024.
With BUFFER_SIZE set to 1 file takes long time to download
Below is the code snippet:
import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import okhttp3.Call;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.Response;
public class DownloadTest {
public static String url = "https://cdn.pixabay.com/photo/2017/02/06/12/34/reptile-2042906_1280.jpg";
public static void main(String[] args) throws Exception {
OkHttpClient client = new OkHttpClient();
Call call = client.newCall(new Request.Builder().url(url).get().build());
Response response = call.execute();
System.out.println("" + response.headers().toString());
System.out.println("" + response.body().contentLength());
InputStream inputStream = response.body().byteStream();
float contentLength = (float) response.body().contentLength();
OutputStream fileOutputStream = new FileOutputStream(new File("myfile.jpg"));
System.out.println("writing file " + contentLength);
float downloaded = 0;
/**
* IF BUFFER_SIZE IS 1 file is downloaded properly
* if BUFFER_SIZE is 1024 file is corrupted
* open the downloaded image to test
*/
//byte[] BUFFER_SIZE = new byte[1]; //Proper Download
byte[] BUFFER_SIZE = new byte[1024]; //File Corrupt
while (true) {
int byteRead = inputStream.read(BUFFER_SIZE);
if (byteRead == -1) {
break;
}
downloaded += byteRead;
fileOutputStream.write(BUFFER_SIZE);
System.out.println(" " + downloaded + "/" + contentLength + " = " + ((downloaded / contentLength) * 100));
}
fileOutputStream.flush();
fileOutputStream.close();
System.out.println("file closed");
}
}
If your BUFFER_SIZE is not full at the last read then you will have wrong data written in file:
You have
fileOutputStream.write(BUFFER_SIZE);
you should have:
fileOutputStream.write(BUFFER_SIZE, 0, byteRead);
EDIT1: I would also sugest to replace this part of the code:
while (true) {
int byteRead = inputStream.read(BUFFER_SIZE);
if (byteRead == -1) {
break;
}
With a better approach:
int byteRead;
while ( (byteRead = inputStream.read(BUFFER_SIZE)) > 0 ) {
Your code is a bit confusing because BUFFER_SIZE looks like a numeric constant. Apart from that, I think your problem is with fileOutputStream.write(BUFFER_SIZE). On the last write, when your byte[] is not "full," you will still write the entire content of the array. Use the overload that specifies an offset (0) and the bytes to write (byteRead.)
Related
I am able to create a 7z file but want to create the file with a password, I tried with the set compression method but there is not an option to set the key, Please help me how I can create a password-protected 7Z file in Java.
public static void main(String args[]) throws FileNotFoundException, IOException {
SevenZOutputFile sevenZOutput = new SevenZOutputFile(new File("D:\\Test\\outFile.7z"));
File entryFile = new File("D:\\Test\\Test_20200210200232.dat");
SevenZArchiveEntry entry = sevenZOutput.createArchiveEntry(entryFile, entryFile.getName());
sevenZOutput.putArchiveEntry(entry);
FileInputStream in = new FileInputStream(entryFile);
int len;
byte buffer[] = new byte[8192];
int transferedMegaBytes2=0;
while ((len = in.read(buffer)) > 0) {
sevenZOutput.write(buffer, 0, len);
transferredBytes += len;
int transferedMegaBytes = (int) (transferredBytes / 1048576);
if(transferedMegaBytes>transferedMegaBytes2){
System.out.println("Transferred: " + transferedMegaBytes + " Megabytes.");
transferedMegaBytes2=transferedMegaBytes;
}
}
sevenZOutput.closeArchiveEntry();
sevenZOutput.setContentCompression(SevenZMethod.AES256SHA256);
sevenZOutput.close();
}
Apache Commons Compress does not support creating 7Z with passwords.
https://commons.apache.org/proper/commons-compress/limitations.html
I've tidied up your code, added try-with-resources & annotated a couple of problem-areas:
import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import org.apache.commons.compress.archivers.sevenz.SevenZArchiveEntry;
import org.apache.commons.compress.archivers.sevenz.SevenZMethod;
import org.apache.commons.compress.archivers.sevenz.SevenZOutputFile;
public class Q66451111 {
private static final int KB = 1024;
private static final int MB = KB * KB;
public static void main(final String[] args) throws IOException {
final File entryFile = new File("D:\\Test\\Test_20200210200232.dat");
final File new7Z = new File("D:\\Test\\outFile.7z");
try ( final InputStream fin = new FileInputStream(entryFile);
final InputStream in = new BufferedInputStream(fin);
final SevenZOutputFile szof = new SevenZOutputFile (new7Z) )
{
final SevenZArchiveEntry entry = szof.createArchiveEntry(entryFile, entryFile.getName());
szof.putArchiveEntry(entry);
final byte buffer[] = new byte[8192];
int transferredBytes = 0;
int transferredBytesSincePrint = 0;
int len;
while ((len = in.read (buffer)) != -1 /* TODO Note: do NOT use '> 0' */) {
szof.write(buffer, 0, len);
transferredBytes += len;
transferredBytesSincePrint += len;
if (transferredBytesSincePrint > MB) {
transferredBytesSincePrint = 0;
System.out.println("Transferring.: " + ((double) transferredBytes / MB) + " Megabytes.");
}
}
System .out.println("Transferred..: " + ((double) transferredBytes / MB) + " Megabytes.");
szof.closeArchiveEntry();
szof.setContentCompression(SevenZMethod.AES256SHA256 /* FIXME Unsupported 7z Method!! */);
}
}
}
This question already has answers here:
Java multiple file transfer over socket
(3 answers)
Closed 5 years ago.
I'm trying to send a file through Java sockets from one program to the other. My code is based off of this question. The issue is that on the Ubuntu machine hosting the server, the file sent over is 0 bytes in size. The code has worked on a local connection on my Windows laptop, so perhaps the issue has to do with the remote connection?
Server code:
int g = Integer.parseInt(in.readLine());
if(g > -1) {
InputStream fIn = client.getInputStream();
for(int i = 0; i < g; i++) {
String name = in.readLine();
byte[] bytes = new byte[16*1024];
File f = new File("plugins/" + name + ".jar");
if(!f.exists()) f.createNewFile();
FileOutputStream fOut = new FileOutputStream(f);
int count;
while ((count = fIn.read(bytes)) >= 0) {
fOut.write(bytes, 0, count);
}
fOut.flush();
fOut.close();
}
System.out.println("[" + client.getRemoteSocketAddress().toString() + "] added " + g + " new plugins.");
client.close();
}else{
client.close();
}
Client code:
JFileChooser fd = new JFileChooser("C:\\");
fd.setFileSelectionMode(JFileChooser.FILES_ONLY);
fd.setMultiSelectionEnabled(true);
fd.setFileFilter(new FileNameExtensionFilter(null,"jar"));
int r = fd.showOpenDialog(null);
if(r == JFileChooser.APPROVE_OPTION) {
File[] files = fd.getSelectedFiles();
OutputStream fOut = sock.getOutputStream();
out.println(files.length);
for(File f : files) {
out.println(f.getName().split("\\.")[0]);
byte[] bytes = new byte[16 * 1024];
FileInputStream fIn = new FileInputStream(f);
int count;
while ((count = fIn.read(bytes)) >= 0) {
fOut.write(bytes, 0, count);
}
fIn.close();
}
}else{
out.println(-1);
}
sock.close();
As said in the comments there are different issues in you code.
To recognise the end of single file you can simply add the file size after the filename separated by a Semicolon;
out.println(f.getName().split("\\.")[0] + "; size" + f.length());
On the Server side you accumulate the file size read / written stop read ing the content of the file close it and read the next line (filename / file size).
As said in the comments there are different issues.
You use fIn and fOut which is not clear how they are defined.
On the client side you use fOut while the Socket OutputStream is named 'out'.
Similar thing on the server side. You read the number of files from in but afterwards you get the InputStream (again?) from the Socket and use that as fIn. Depending on your in that may or may not work.
Also reading a line from via readLine from an InputStream and also binary content is not that trivial. In my example code I'm using the DataInputStream for that even the readLine method is deprecated.
The Example also uses the "try with resources" Syntax to ensure that all resources (Streams, Sockets) get closed in the case of an Exception.
The Example does not show how to deal with the file size on the server side That's up to you.
The example is completely runnable on a single machine and shows how to copy a single file via socket based on your code.
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.io.PrintStream;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.ServerSocket;
import java.net.Socket;
public class ClientServerSocketExample {
private static class Server {
public void run() throws IOException {
try (ServerSocket serverSocket = new ServerSocket(41000, 10, InetAddress.getLocalHost())) {
try (Socket client = serverSocket.accept()) {
try (DataInputStream in = new DataInputStream(client.getInputStream())) {
int g = Integer.parseInt(in.readLine());
if(g > -1) {
for(int i = 0; i < g; i++) {
String[] filenameAndSize = in.readLine().split(";");
String name = filenameAndSize[0];
byte[] bytes = new byte[16*1024];
File f = new File("/tmp/" + name + ".jar");
if(!f.exists()) f.createNewFile();
try (FileOutputStream fOut = new FileOutputStream(f)) {
int count;
while ((count = in.read(bytes)) >= 0) {
fOut.write(bytes, 0, count);
}
fOut.flush();
}
}
System.out.println("[" + client.getRemoteSocketAddress().toString() + "] added " + g + " new plugins.");
}
}
}
}
}
}
private static class Client {
public void run() throws IOException {
try (Socket socket = new Socket()) {
socket.connect(new InetSocketAddress(InetAddress.getLocalHost(), 41000) );
sendFiles(socket);
}
}
private static void sendFiles(Socket sock) throws IOException {
File[] files = new File[]{new File("some.jar")};
OutputStream fOut = sock.getOutputStream();
PrintStream out = new PrintStream(fOut);
out.println(files.length);
for(File f : files) {
out.println(f.getName().split("\\.")[0] + "; size" + f.length());
byte[] bytes = new byte[16 * 1024];
try (FileInputStream fIn = new FileInputStream(f)) {
int count;
while ((count = fIn.read(bytes)) >= 0) {
out.write(bytes, 0, count);
}
out.flush();
}
}
}
}
public static void main(String ... args) throws IOException, InterruptedException {
new Thread(new Runnable() {
#Override
public void run() {
try {
new Server().run();
} catch (IOException ioe) {
System.out.println(ioe);
}
}
}).start();
System.out.println("Waiting for the Server to come up");
Thread.sleep(500);
new Client().run();
}
}
There may be an issue in this solution when the client sends a \r as new line separator and the file starts with \n. The server may count the \n as part of the line separator and the file will be corrupt (one byte will be missing).
I'm trying to find a way to use
copyInputStreamToFile(InputStream source, File destination)
to make a small progress bar in the console by file size. Is there a way to do this?
The short answer you can't, look at the source code of this method, I tried to track its execution path and it goes to this method at IOUtils class:
public static long copyLarge(final InputStream input, final OutputStream output, final byte[] buffer)
throws IOException {
long count = 0;
int n = 0;
while (EOF != (n = input.read(buffer))) {
output.write(buffer, 0, n);
count += n;
}
return count;
}
So, this functionality is encapsulated by an API.
The long answer you can implement downloading method by yourself, by using relative parts of IOUtils and FileUtils libraries and add functionality to print percentage of downloaded file in a console:
This is a working kick-off example:
package apache.utils.custom;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
public class Downloader {
private static final int EOF = -1;
private static final int DEFAULT_BUFFER_SIZE = 1024 * 4;
public static void copyInputStreamToFileNew(final InputStream source, final File destination, int fileSize) throws IOException {
try {
final FileOutputStream output = FileUtils.openOutputStream(destination);
try {
final byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
long count = 0;
int n = 0;
while (EOF != (n = source.read(buffer))) {
output.write(buffer, 0, n);
count += n;
System.out.println("Completed " + count * 100/fileSize + "%");
}
output.close(); // don't swallow close Exception if copy completes normally
} finally {
IOUtils.closeQuietly(output);
}
} finally {
IOUtils.closeQuietly(source);
}
}
You should provide expected file size to this method which you can calculate by using this code:
URL url = new URL(urlString);
URLConnection urlConnection = url.openConnection();
urlConnection.connect();
int file_size = urlConnection.getContentLength();
Of course the better idea is to encapsulate the whole functionality in a single method.
Hope it will help you.
I'm trying to make a simplified HDFS (Hadoop Distributed File System) for a final project in a Distributed System course.
So, the first thing that I'm trying is to write a program which split an arbitrary file into blocks (chunks) of an arbitrary dimension.
I found this useful example, which code is:
package javabeat.net.io;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
/**
* Split File Example
*
* #author Krishna
*
*/
public class SplitFileExample {
private static String FILE_NAME = "TextFile.txt";
private static byte PART_SIZE = 5;
public static void main(String[] args) {
File inputFile = new File(FILE_NAME);
FileInputStream inputStream;
String newFileName;
FileOutputStream filePart;
int fileSize = (int) inputFile.length();
int nChunks = 0, read = 0, readLength = PART_SIZE;
byte[] byteChunkPart;
try {
inputStream = new FileInputStream(inputFile);
while (fileSize > 0) {
if (fileSize <= 5) {
readLength = fileSize;
}
byteChunkPart = new byte[readLength];
read = inputStream.read(byteChunkPart, 0, readLength);
fileSize -= read;
assert (read == byteChunkPart.length);
nChunks++;
newFileName = FILE_NAME + ".part"
+ Integer.toString(nChunks - 1);
filePart = new FileOutputStream(new File(newFileName));
filePart.write(byteChunkPart);
filePart.flush();
filePart.close();
byteChunkPart = null;
filePart = null;
}
inputStream.close();
} catch (IOException exception) {
exception.printStackTrace();
}
}
}
But I think that there is a big issue: the value of PART_SIZE cannot be greater than 127, otherwise an error: possible loss of precision will occur.
How can I solve without totally changing the code?
The problem is that PART_SIZE is a byte; its maximum value is therefore indeed 127.
The code you have at the moment however is full of problems; for one, incorrect resource handling etc.
Here is a version using java.nio.file:
private static final String FILENAME = "TextFile.txt";
private static final int PART_SIZE = xxx; // HERE
public static void main(final String... args)
throws IOException
{
final Path file = Paths.get(FILENAME).toRealPath();
final String filenameBase = file.getFileName().toString();
final byte[] buf = new byte[PART_SIZE];
int partNumber = 0;
Path part;
int bytesRead;
byte[] toWrite;
try (
final InputStream in = Files.newInputStream(file);
) {
while ((bytesRead = in.read(buf)) != -1) {
part = file.resolveSibling(filenameBase + ".part" + partNumber);
toWrite = bytesRead == PART_SIZE ? buf : Arrays.copyOf(buf, bytesRead);
Files.write(part, toWrite, StandardOpenOption.CREATE_NEW);
partNumber++;
}
}
}
List<PDDocument> Pages=new ArrayList<PDDocument>();
Document.load(filePath);
try {
Splitter splitter = new Splitter();
splitter.setSplitAtPage(NoOfPagesDocumentWillContain);
Pages = splitter.split(document);
}catch(Exception e)
{
l
e.getCause().printStackTrace();
}
I am trying to save .doc, .pdf, .txt, and image files into my database using hibernate, jsf, and mysql.
I have created a column to save the file of type BLOB. If I am saving .txt type then files are saved correctly.
If i am trying to save file of any other format then i am getting an exception.
In my bean I have created a field name: byte[] file;
How i can save it correctly without any exceptions? Do I need to change datatype for mysql column or use a different field for java class?
(in response to BalusC)
This is the code which I am using for file writing. I am using fileInputStream and then saving the file using hibernate framework.
Iterator iter = items.iterator();
while (iter.hasNext()) {
FileItem item = (FileItem) iter.next();
if (item.isFormField()) {
String name = item.getFieldName();
String value = item.getString();
} else {
String fieldName = item.getFieldName();
String fileName = item.getName();
String contentType = item.getContentType();
boolean isInMemory = item.isInMemory();
long sizeInBytes = item.getSize();
byte[] fileInBytes=item.get();
try {
File uploadedFile = new File("/home/db/webApp", fileName);
uploadedFile.createNewFile();
FileInputStream fileInputStream = new FileInputStream(uploadedFile);
//convert file into array of bytes
fileInputStream.read(fileInBytes);
fileInputStream.close();
} catch (Exception e) {
e.printStackTrace();
}
UploadedFile object= new UploadedFile();
object.setFile(fileInBytes);
uploadedObject.setFileName(fileName);
session.save(object);
UploadedFile is jsf managed bean:
public class UploadedFile{
private String fileName;
private byte[] file;
/**
* #return the fileName
*/
public String getFileName() {
return fileName;
}
/**
* #param fileName the fileName to set
*/
public void setFileName(String fileName) {
this.fileName = fileName;
}
/**
* #return the file
*/
public byte[] getFile() {
return file;
}
/**
* #param file the file to set
*/
public void setFile(byte[] file) {
this.file = file;
}
}
and my database table has following structure:
Create UploadFile(FILE_NAME` VARCHAR(1000) NOT NULL,
`FILE` BLOB NOT NULL);
Your problem looks like it's a data type issue. A BLOB in MySQL is not very big. Try setting your table's column data type to a LONGBLOB instead.
Its better to save the file in a location
and save the location in the database
Store the file meta info such as location in db, but synchronize db and file system will be an issue.
package com.server;
import java.io.*;
import java.sql.*;
import java.util.*;
import java.text.*;
import java.util.regex.*;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.fileupload.disk.DiskFileItemFactory;
import org.apache.commons.fileupload.*;
import org.mortbay.jetty.Response;
import javax.servlet.*;
import javax.servlet.http.*;
import java.io.*;
import java.sql.*;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.ServletInputStream.*;
import java.io.PrintWriter;
public class XmlServlet extends HttpServlet {
public void doPost(HttpServletRequest req,HttpServletResponse res)
{
File uploadedFile;
System.out.print("on server");
try{
Class.forName("com.mysql.jdbc.Driver");
Connection conn = DriverManager.getConnection("jdbc:mysql://localhost:3308/image","root","root1");
PrintWriter out=res.getWriter();
//out.println("<br>Content type is :: " +contentType);
//to get the content type information from JSP Request Header
String contentType = req.getContentType();
int flag=0;
FileInputStream fis=null;
FileOutputStream fileOut=null;
//here we are checking the content type is not equal to Null and as well as the passed data from mulitpart/form-data is greater than or equal to 0
if ((contentType != null) && (contentType.indexOf("multipart/form-data") >= 0))
{
DataInputStream in = new DataInputStream(req.getInputStream());
//we are taking the length of Content type data
int formDataLength = req.getContentLength();
byte dataBytes[] = new byte[formDataLength];
int byteRead = 0;
int totalBytesRead = 0;
//this loop converting the uploaded file into byte code
while (totalBytesRead < formDataLength) {
byteRead = in.read(dataBytes, totalBytesRead,formDataLength);
totalBytesRead += byteRead;
}
String file = new String(dataBytes);
//for saving the file name
String saveFile = file.substring(file.indexOf("filename=\"") + 10);
saveFile = saveFile.substring(0, saveFile.indexOf("\n"));
out.println("savefiledddd"+saveFile);
int extension_save=saveFile.lastIndexOf("\"");
String extension_saveName=saveFile.substring(extension_save);
//Here we are invoking the absolute path out of the encrypted data
saveFile = saveFile.substring(saveFile.lastIndexOf("\\")+ 1,saveFile.indexOf("\""));
int lastIndex = contentType.lastIndexOf("=");
String boundary = contentType.substring(lastIndex + 1,contentType.length());
int pos;
//extracting the index of file
pos = file.indexOf("filename=\"");
pos = file.indexOf("\n", pos) + 1;
pos = file.indexOf("\n", pos) + 1;
pos = file.indexOf("\n", pos) + 1;
int boundaryLocation = file.indexOf(boundary, pos) - 4;
int startPos = ((file.substring(0, pos)).getBytes()).length;
int endPos = ((file.substring(0, boundaryLocation)).getBytes()).length;
out.println("savefile"+saveFile);
int file_No=22;
uploadedFile=new File("./war/img");
uploadedFile.mkdir();
String kk=uploadedFile.getAbsolutePath();
String pathname_dir=kk+"/"+saveFile;
//String pathname_dir="C:\\Program Files\\Apache Software Foundation\\Tomcat 6.0\\jk\\"+saveFile;
File filepath=new File(pathname_dir);
out.println("filepath_ "+filepath);
fileOut = new FileOutputStream(filepath);
fileOut.write(dataBytes, startPos, (endPos - startPos));
fileOut.flush();
out.println("<h1> your files are saved</h1></body></html>");
out.close();
File database_filename=new File(pathname_dir);
fis=new FileInputStream(database_filename);
int len=(int)database_filename.length();
PreparedStatement ps = conn.prepareStatement("insert into new (image) values (?)");
ps.setBinaryStream(1,fis,len);
ps.executeUpdate();
ps.close();
flag=1;
}
if(flag==1)
{
fileOut.close();
fis.close();
}
}catch(Exception e)
{
System.out.println("Exception Due to"+e);
e.printStackTrace();
}
}
}
It's a server code. By that code you can upload any file which is stored in database and store that file on server side at (img) folder. By using the reference of that you can access the file.