Extracting files in .iso using jSevenZip - java

I'm trying to extract the files in an .iso file I have. There are no errors and the console prints out all the files inside. However, none of the files get extracted. Any help is greatly appreciated!
I was able to find this link but I can't seem to integrate it in to my project.
Link to code example
class DownloadWorker extends SwingWorker<String, Object> {
private String flocation;
private JLabel out;
public DownloadWorker(String location, JLabel fout)
{
this.flocation = location;
this.out = fout;
}
#Override
public String doInBackground() {
//download here
out.setText("Beginning download. This may take a few minutes.");
try {
//ProgressBar/Install
System.out.println("FILELOCATION:\n----------");
System.out.println(flocation);
String URL_LOCATION = "http://www.futureretrogaming.tk/gamefiles/ProfessorPhys.iso";
String LOCAL_FILE = (flocation + "\\ProfessorPhys\\");
File localfile = new File(LOCAL_FILE);
if (localfile.exists()) {
System.out.println("Directory exists!");
}
else {
System.out.println("Directory doesn't exist! Creating...");
localfile.mkdir();
if (localfile.exists())
System.out.println("Directory created!");
}
System.out.println("LOCALFILE:\n-------");
System.out.println(LOCAL_FILE);
URL website = new URL(URL_LOCATION);
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream(LOCAL_FILE+"\\ProfessorPhys.iso\\");
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
System.out.println("--------\nDone Downloading\n---------");
RandomAccessFile randomAccessFile = null;
ISevenZipInArchive inArchive = null;
try {
randomAccessFile = new RandomAccessFile(localfile +"\\ProfessorPhys.iso\\", "r");
inArchive = SevenZip.openInArchive(null, // autodetect archive type
new RandomAccessFileInStream(randomAccessFile));
// Getting simple interface of the archive inArchive
ISimpleInArchive simpleInArchive = inArchive.getSimpleInterface();
System.out.println(" Hash | Size | Filename");
System.out.println("----------+------------+---------");
for (ISimpleInArchiveItem item : simpleInArchive.getArchiveItems()) {
final int[] hash = new int[] { 0 };
if (!item.isFolder()) {
ExtractOperationResult result;
final long[] sizeArray = new long[1];
result = item.extractSlow(new ISequentialOutStream() {
public int write(byte[] data) throws SevenZipException {
hash[0] ^= Arrays.hashCode(data); // Consume data
sizeArray[0] += data.length;
return data.length; // Return amount of consumed data
}
});
if (result == ExtractOperationResult.OK) {
System.out.println(String.format("%9X | %10s | %s", //
hash[0], sizeArray[0], item.getPath()));
} else {
System.err.println("Error extracting item: " + result);
}
}
}
} catch (Exception e) {
System.err.println("Error occurs: " + e);
System.exit(1);
} finally {
if (inArchive != null) {
try {
inArchive.close();
} catch (SevenZipException e) {
System.err.println("Error closing archive: " + e);
}
}
if (randomAccessFile != null) {
try {
randomAccessFile.close();
} catch (IOException e) {
System.err.println("Error closing file: " + e);
}
}
}
} catch (Exception e) {
System.out.println(e);
}
return "done";
}
#Override
protected void done() {
//done
out.setText(out.getText() + "\n Operation Done");
}
}
Code I'm trying to implement:
public class ExtractorUtil {
private static Logger logger = Logger.getLogger(ExtractorUtil.class.getCanonicalName());
public static void extract(File file, String extractPath) throws Exception {
ISevenZipInArchive inArchive = null;
RandomAccessFile randomAccessFile = null;
randomAccessFile = new RandomAccessFile(file, "r");
inArchive = SevenZip.openInArchive(null, new RandomAccessFileInStream(randomAccessFile));
inArchive.extract(null, false, new MyExtractCallback(inArchive, extractPath));
if (inArchive != null) {
inArchive.close();
}
if (randomAccessFile != null) {
randomAccessFile.close();
}
}
public static class MyExtractCallback implements IArchiveExtractCallback {
private final ISevenZipInArchive inArchive;
private final String extractPath;
public MyExtractCallback(ISevenZipInArchive inArchive, String extractPath) {
this.inArchive = inArchive;
this.extractPath = extractPath;
}
#Override
public ISequentialOutStream getStream(final int index, ExtractAskMode extractAskMode) throws SevenZipException {
return new ISequentialOutStream() {
#Override
public int write(byte[] data) throws SevenZipException {
String filePath = inArchive.getStringProperty(index, PropID.PATH);
FileOutputStream fos = null;
try {
File dir = new File(extractPath);
File path = new File(extractPath + filePath);
if (!dir.exists()) {
dir.mkdirs();
}
if (!path.exists()) {
path.createNewFile();
}
fos = new FileOutputStream(path, true);
fos.write(data);
} catch (IOException e) {
logger.severe(e.getLocalizedMessage());
} finally {
try {
if (fos != null) {
fos.flush();
fos.close();
}
} catch (IOException e) {
logger.severe(e.getLocalizedMessage());
}
}
return data.length;
}
};
}

Related

Android - Zip Path Traversal in play store

I am uploading my App on play store but get me bellow error:
Zip Path Traversal Your app contains an unsafe unzipping pattern that
may lead to a Path Traversal vulnerability. Please see this Google
Help Center article to learn how to fix the issue.
org.apache.cordova.Zip.unzipSync
I edited my source code like this LINK, but get me error.
Here is my source code changed:
public class Zip extends CordovaPlugin {
private static final String LOG_TAG = "Zip";
// Can't use DataInputStream because it has the wrong endian-ness.
private static int readInt(InputStream is) throws IOException {
int a = is.read();
int b = is.read();
int c = is.read();
int d = is.read();
return a | b << 8 | c << 16 | d << 24;
}
#Override
public boolean execute(String action, CordovaArgs args, final CallbackContext callbackContext) throws JSONException {
if ("unzip".equals(action)) {
unzip(args, callbackContext);
return true;
}
return false;
}
private void unzip(final CordovaArgs args, final CallbackContext callbackContext) {
this.cordova.getThreadPool().execute(new Runnable() {
public void run() {
unzipSync(args, callbackContext);
}
});
}
private void unzipSync(CordovaArgs args, CallbackContext callbackContext) {
InputStream inputStream = null;
try {
String zipFileName = args.getString(0);
String outputDirectory = args.getString(1);
// Since Cordova 3.3.0 and release of File plugins, files are accessed via cdvfile://
// Accept a path or a URI for the source zip.
Uri zipUri = getUriForArg(zipFileName);
Uri outputUri = getUriForArg(outputDirectory);
CordovaResourceApi resourceApi = webView.getResourceApi();
File tempFile = resourceApi.mapUriToFile(zipUri);
if (tempFile == null || !tempFile.exists()) {
String errorMessage = "Zip file does not exist";
callbackContext.error(errorMessage);
Log.e(LOG_TAG, errorMessage);
return;
}
File outputDir = resourceApi.mapUriToFile(outputUri);
outputDirectory = outputDir.getAbsolutePath();
outputDirectory += outputDirectory.endsWith(File.separator) ? "" : File.separator;
if (outputDir == null || (!outputDir.exists() && !outputDir.mkdirs())) {
String errorMessage = "Could not create output directory";
callbackContext.error(errorMessage);
Log.e(LOG_TAG, errorMessage);
return;
}
OpenForReadResult zipFile = resourceApi.openForRead(zipUri);
ProgressEvent progress = new ProgressEvent();
progress.setTotal(zipFile.length);
inputStream = new BufferedInputStream(zipFile.inputStream);
inputStream.mark(10);
int magic = readInt(inputStream);
if (magic != 875721283) { // CRX identifier
inputStream.reset();
} else {
// CRX files contain a header. This header consists of:
// * 4 bytes of magic number
// * 4 bytes of CRX format version,
// * 4 bytes of public key length
// * 4 bytes of signature length
// * the public key
// * the signature
// and then the ordinary zip data follows. We skip over the header before creating the ZipInputStream.
readInt(inputStream); // version == 2.
int pubkeyLength = readInt(inputStream);
int signatureLength = readInt(inputStream);
inputStream.skip(pubkeyLength + signatureLength);
progress.setLoaded(16 + pubkeyLength + signatureLength);
}
// The inputstream is now pointing at the start of the actual zip file content.
ZipInputStream zis = new ZipInputStream(inputStream);
inputStream = zis;
ZipEntry ze;
byte[] buffer = new byte[32 * 1024];
boolean anyEntries = false;
while ((ze = zis.getNextEntry()) != null) {
try {
anyEntries = true;
String compressedName = ze.getName();
if (ze.isDirectory()) {
try {
File dir = new File(outputDirectory + compressedName);
File f = new File(dir, ze.getName());
String canonicalPath = f.getCanonicalPath();
if (!canonicalPath.startsWith(dir.toString())){
dir.mkdirs();
}else {
if (inputStream != null) {
try {
inputStream.close();
} catch (IOException e) {
}
}
}
} catch (Exception e) {
String errorMessage = "An error occurred while unzipping.";
callbackContext.error(errorMessage);
Log.e(LOG_TAG, errorMessage, e);
}
} else {
File file = new File(outputDirectory + compressedName);
File f = new File(file, ze.getName());
String canonicalPath = f.getCanonicalPath();
if (!canonicalPath.startsWith(file.toString())) {
file.getParentFile().mkdirs();
if (file.exists() || file.createNewFile()) {
try {
Log.w("Zip", "extracting: " + file.getPath());
FileOutputStream fout = new FileOutputStream(file);
int count;
while ((count = zis.read(buffer)) != -1) {
fout.write(buffer, 0, count);
}
fout.close();
} catch (Exception e) {
String errorMessage = "An error occurred while unzipping.";
callbackContext.error(errorMessage);
Log.e(LOG_TAG, errorMessage, e);
}
}
}else {
if (inputStream != null) {
try {
inputStream.close();
} catch (IOException e) {
}
}
}
}
progress.addLoaded(ze.getCompressedSize());
updateProgress(callbackContext, progress);
zis.closeEntry();
} catch (Exception e) {
String errorMessage = "An error occurred while unzipping.";
callbackContext.error(errorMessage);
Log.e(LOG_TAG, errorMessage, e);
}
}
// final progress = 100%
progress.setLoaded(progress.getTotal());
updateProgress(callbackContext, progress);
if (anyEntries)
callbackContext.success();
else
callbackContext.error("Bad zip file");
} catch (Exception e) {
String errorMessage = "An error occurred while unzipping.";
callbackContext.error(errorMessage);
Log.e(LOG_TAG, errorMessage, e);
} finally {
if (inputStream != null) {
try {
inputStream.close();
} catch (IOException e) {
}
}
}
}
private void updateProgress(CallbackContext callbackContext, ProgressEvent progress) throws JSONException {
PluginResult pluginResult = new PluginResult(PluginResult.Status.OK, progress.toJSONObject());
pluginResult.setKeepCallback(true);
callbackContext.sendPluginResult(pluginResult);
}
private Uri getUriForArg(String arg) {
CordovaResourceApi resourceApi = webView.getResourceApi();
Uri tmpTarget = Uri.parse(arg);
return resourceApi.remapUri(
tmpTarget.getScheme() != null ? tmpTarget : Uri.fromFile(new File(arg)));
}
private static class ProgressEvent {
private long loaded;
private long total;
public long getLoaded() {
return loaded;
}
public void setLoaded(long loaded) {
this.loaded = loaded;
}
public void addLoaded(long add) {
this.loaded += add;
}
public long getTotal() {
return total;
}
public void setTotal(long total) {
this.total = total;
}
public JSONObject toJSONObject() throws JSONException {
return new JSONObject(
"{loaded:" + loaded +
",total:" + total + "}");
}
}
}

Flatbuffer writing in binary file in android give only single response

I am new at flatbuffer. I had created a schema file from outside the project and adding the user(Monster as in Flatbuffer document) inside the binary via java code in android. everything works fine. but the time of reading binary file data it only gives me the user length 1. I had added 3 people but it's give me the length of monsters is 1 at the time of reading. can anyone help to figure this out. Here is the code->
this is the full code ->
enter code here
builder = new FlatBufferBuilder(1024);
public void addNewData(String emailOffset, String nameOffset,String
contactNoOffset, String DOJOffset, String departmentOffset,
String empIdOffset, float[] embeddingOffset)
{
int storeembedd = SingleJson.createEmbeddingVector(builder,
embeddingOffset);
int hereEmailOffset = builder.createString(emailOffset);
int hereNameOffset = builder.createString(nameOffset);
int hereContactOffset = builder.createString(contactNoOffset);
int hereDOJOffset = builder.createString(DOJOffset);
int hereDepartmentOffset=builder.createString(departmentOffset);
int hereImpIdOffset = builder.createString(empIdOffset);
int test = SingleJson.createSingleJson(builder, hereEmailOffset,
hereNameOffset, hereContactOffset
, hereDOJOffset, hereDepartmentOffset, hereImpIdOffset,
storeembedd);
int [] offsetOfMonster = new int[1];
offsetOfMonster[0] = test;
int temp = Monsters.createMonstersVector(builder,
offsetOfMonster);
Monsters.startMonsters(builder);
Monsters.addMonsters(builder, temp);
int orc = Monsters.endMonsters(builder);
builder.finish(orc);
byte[] buf1 = builder.sizedByteArray();
openAndAppenedInBinary(buf1);
}
private void openAndAppenedInBinary(byte[] buf1) {
FileOutputStream output = null;
try {
InputStream inputStream = new FileInputStream(newFile());
byte[] buffer = new byte[inputStream.available()];
if (inputStream.available() == 0) {
output = new FileOutputStream(newFile(), true);
output.write(buf1);
} else {
while (inputStream.read(buffer) != -1) {
output = new FileOutputStream(newFile(), true);
output.write(buffer);
output.write(buf1);
}
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
output.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
private String newFile() {
File file = new File(Environment.getExternalStorageDirectory()
+ "/Android/data/"
+ context.getPackageName()
+ "/binFile");
if (!file.exists()) {
if (!file.mkdir()) {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
try {
Path dir = Paths.get(file.getAbsolutePath());
Files.createDirectory(dir);
} catch (IOException e) {
Path parentDir = Paths.get(file.getParent());
if (!Files.exists(parentDir)) {
try {
Files.createDirectories(parentDir);
} catch (IOException e1) {
e1.printStackTrace();
}
}
}
}
}
}
File binFile = new File(file, "renamed.bin");
try {
FileWriter writer = new FileWriter(binFile);
writer.close();
} catch (IOException e) {
e.printStackTrace();
}
return binFile.getAbsolutePath();
}
here is my reading code of flatbiffer->
new FlatParsingForPerticularId().execute(readRawResource(R.raw.renamed));//renamed is my bin file
private class FlatParsingForPerticularId extends AsyncTask {
#Override
protected String doInBackground(Object... params) {
byte[] buffer = (byte[]) params[0];
long startTime = System.currentTimeMillis();
ByteBuffer bb = ByteBuffer.wrap(buffer);
Monsters monsterList = Monsters.getRootAsMonsters(bb);
int length = monsterList.monstersLength();
SingleJson monster = null;
for (int i = 0; i < length; i++) {//here I m getting length 1 intead of 3
monster = monsterList.monsters(i);
if (i == outputval[0]) {
outputval[0] = (int) monster.EmpNo();
break;
}
}
long endTime = System.currentTimeMillis() - startTime;
String textToShow = "Elements: " + monsterList.monstersLength() + ": load time: " + endTime + "ms";
String[] monsterArr = monster.Name().split(" ");
return monsterArr[0];
}

Thrown IllegalArgumentException when using java.nio

Trying to download file by url. For that using java.nio library. The code above work perfect for sdk 24 and above but getting IllegalArgumentException for android sdk 23 and below. I'm getting the error when trying transfer the data. Dear Friends can you please clarify which is the problem.
private void downloadFile(final String itemUrl) {
new Thread(new Runnable() {
#Override
public void run() {
ReadableByteChannel readableByteChannel = null;
FileOutputStream fOutStream = null;
String root =
getApplicationContext().getApplicationInfo().dataDir;
File myDir = new File(root + "/downloadedSongs");
if (!myDir.exists()) {
myDir.mkdirs();
}
File file = new File(myDir, itemTitle);
if (file.exists()) {
file.delete();
}
try {
URL url = new URL(itemUrl);
readableByteChannel =
Channels.newChannel(url.openStream());
fOutStream = new FileOutputStream(file);
fOutStream.getChannel().transferFrom(readableByteChannel, 0, Long.MAX_VALUE);
} catch (IOException e) {
orderAction = Enums.OrderAction.Delete;
} catch (IllegalArgumentException e) {
orderAction = Enums.OrderAction.Delete;
Crashlytics.logException(e);
} finally {
try {
if (fOutStream != null) {
fOutStream.close();
}
if (readableByteChannel != null) {
readableByteChannel.close();
}
} catch (IOException ioExObj) {
orderAction = Enums.OrderAction.Delete;
}
closeNotification(orderAction);
}
}
}).start();
}
Fatal Exception: java.lang.IllegalArgumentException: position=0 count=9223372036854775807 at java.nio.FileChannelImpl.transferFrom(FileChannelImpl.java:370) at am.itsoft.youtomp3.services.DnlService$1.run(DnlService.java:169) at java.lang.Thread.run(Thread.java:818)
The reason is in this argument: Long.MAX_VALUE
Right code:
FileChannel destChannel = fOutStream.getChannel();
long blockSize;
if (Build.VERSION.SDK_INT > 23) {
long blockSize = Long.MAX_VALUE;
} else {
long blockSize = 8*1024;
}
long position = 0;
long loaded;
while ((loaded = destChannel.transferFrom(readableByteChannel, position, blockSize)) > 0) {
position += loaded;
}

How to build a zip file with a size of 400 GB in java

I need to download all the documents from an alfresco site that contains 400GB of documents.
The code below is ok for create a small zip file (about 1GB) otherwise it takes too much memory.
I would not like to keep ZipOutputStream in memory, i would like to use memory only for every document copied to the Zip file or use a temporary file that is overwritten for each document.
What is the best practice for this kind of problem?
This piece of code is called from my main:
FolderImpl sitoFolder = (FolderImpl) cmisObject;
List<Tree<FileableCmisObject>> sitoFolderDescendants = sitoFolder.getDescendants(-1);
byte[] zipFile = createZipFILE(sitoFolderDescendants);
String rootPath = cartella_download_file;
File dir = new File(rootPath + File.separator);
if (!dir.exists()) {
dir.mkdirs();
}
Date date = new Date();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
String stringDate = sdf.format(date);
String nameZipFile = sitoFolder.getName().replaceAll("\\s","");
File serverFile = new File(dir.getAbsolutePath() + File.separator + stringDate+"_"+nameZipFile+".zip");
BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(new FileOutputStream(serverFile));
IOUtils.write(zipFile, bufferedOutputStream);
bufferedOutputStream.close();
//Returns the zip file
private byte[] createZipFILE(List<Tree<FileableCmisObject>> list) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ByteTransform byteTransform = new ByteTransform();
try {
ReportDocument reportDocument = new ReportDocument();
ZipOutputStream zos = new ZipOutputStream(baos);
for (Tree<FileableCmisObject> aList : list) {
traverseTree(aList, zos, reportDocument);
}
zos.close();
return baos.toByteArray();
} catch (IOException exc) {
reportLog.error(exc.getMessage());
} finally {
baos.close();
}
return new byte[0];
}
private void traverseTree(Tree<FileableCmisObject> tree, ZipOutputStream zos, ReportDocument reportDocument) {
for (int i=0; i<tree.getChildren().size(); i++) {
Tree<FileableCmisObject> child = tree.getChildren().get(i);
if (CmisUtil.isDocument(child.getItem())) {
Document document = (Document) child.getItem();
try {
addToZipFile(document, zos);
} catch (IOException ioExc) {
appLog.error(ioExc.getMessage());
}
} else if(CmisUtil.isFolder(child.getItem())) {
Folder folder = (Folder) child.getItem();
if (folder.getChildren().getTotalNumItems() == 0) {
try {
addToZipFolder(folder, zos);
} catch (IOException ioExc) {
appLog.error(ioExc.getMessage());
}
}
}
traverseTree(child, zos, reportDocument);
}
}
//Service method to add documents to the zip file
private void addToZipFile(Document document, ZipOutputStream zos) throws IOException {
InputStream inputStream = document.getContentStream().getStream();
String path = document.getPaths().get(0).replace(sito_export_path, "");
ZipEntry zipEntry = new ZipEntry(path);
zos.putNextEntry(zipEntry);
IOUtils.copy(inputStream, zos, 1024);
inputStream.close();
zos.closeEntry();
}
//Service method to add empty folder to the zip file
private void addToZipFolder(Folder folder, ZipOutputStream zos) throws IOException {
String path = folder.getPaths().get(0).replace(sito_export_path, "");
ZipEntry zipEntry = new ZipEntry(path.concat("/"));
zos.putNextEntry(zipEntry);
}
I solved it. I first created a directory on the server and then created the zip file on this directory directly.
The error was to save all the files first on: ByteArrayOutputStream and then on the zip file.
File serverFile = new File(dir.getAbsolutePath() + File.separator + stringDate+"_"+nameZipFile+".zip");
FileOutputStream fileOutputStream = new FileOutputStream(serverFile);
ZipArchiveOutputStream zos = new ZipArchiveOutputStream(fileOutputStream);
for (Tree<FileableCmisObject> aList : sitoFolderDescendants) {
traverseTree(aList, zos, reportDocument);
}
zos.close();
In the finally block I close the FileOutputStream.
Than I changed the services method using: ZipArchiveOutputStream and ZipArchiveEntry.
private void addToZipFolder(Folder folder, ZipArchiveOutputStream zos) throws IOException {
String path = folder.getPaths().get(0).replace(sito_export_path, "");
ZipArchiveEntry zipEntry = new ZipArchiveEntry(path.concat("/"));
appLog.info("aggiungo cartella vuota "+folder.getName()+" al file zip");
zos.putArchiveEntry(zipEntry);
zos.closeArchiveEntry();
}
private void addToZipFile(Document document, ZipArchiveOutputStream zos) throws IOException {
InputStream inputStream = document.getContentStream().getStream();
String path = document.getPaths().get(0).replace(sito_export_path, "");
ZipArchiveEntry entry = new ZipArchiveEntry(path);
entry.setSize(document.getContentStreamLength());
zos.putArchiveEntry(entry);
byte buffer[] = new byte[1024];
while (true) {
int nRead = inputStream.read(buffer, 0, buffer.length);
if (nRead <= 0) {
break;
}
zos.write(buffer, 0, nRead);
}
inputStream.close();
zos.closeArchiveEntry();
}
Actually i have create downlod as zip functionality for alfresco 3.4.d version and used following code.i have not checked it for GB's file because i don't have that much data.it may be help to you.
This is Java Backed WebScript.
/*
* this class create a zip file base on given(parameter) node
* */
public class ZipContents extends AbstractWebScript {
private static Log logger = LogFactory.getLog(ZipContents.class);
private static final int BUFFER_SIZE = 1024;
private static final String MIMETYPE_ZIP = "application/zip";
private static final String TEMP_FILE_PREFIX = "alf";
private static final String ZIP_EXTENSION = ".zip";
private ContentService contentService;
private NodeService nodeService;
private NamespaceService namespaceService;
private DictionaryService dictionaryService;
private StoreRef storeRef;
private String encoding;
public void setNodeService(NodeService nodeService) {
this.nodeService = nodeService;
}
public void setContentService(ContentService contentService) {
this.contentService = contentService;
}
public void setNamespaceService(NamespaceService namespaceService) {
this.namespaceService = namespaceService;
}
public void setDictionaryService(DictionaryService dictionaryService) {
this.dictionaryService = dictionaryService;
}
public void setStoreUrl(String url) {
this.storeRef = new StoreRef(url);
}
public void setEncoding(String encoding) {
this.encoding = encoding;
}
public void execute(WebScriptRequest req, WebScriptResponse res) throws IOException {
String nodes = req.getParameter("nodes");
if (nodes == null || nodes.length() == 0) {
throw new WebScriptException(HttpServletResponse.SC_BAD_REQUEST, "nodes");
}
List<String> nodeIds = new ArrayList<String>();
StringTokenizer tokenizer = new StringTokenizer(nodes, ",");
if (tokenizer.hasMoreTokens()) {
while (tokenizer.hasMoreTokens()) {
nodeIds.add(tokenizer.nextToken());
}
}
String filename = req.getParameter("filename");
if (filename == null || filename.length() == 0) {
throw new WebScriptException(HttpServletResponse.SC_BAD_REQUEST, "filename");
}
String noaccentStr = req.getParameter("noaccent");
if (noaccentStr == null || noaccentStr.length() == 0) {
throw new WebScriptException(HttpServletResponse.SC_BAD_REQUEST, "noaccent");
}
try {
res.setContentType(MIMETYPE_ZIP);
res.setHeader("Content-Transfer-Encoding", "binary");
res.addHeader("Content-Disposition", "attachment;filename=\"" + unAccent(filename) + ZIP_EXTENSION + "\"");
res.setHeader("Cache-Control", "must-revalidate, post-check=0, pre-check=0");
res.setHeader("Pragma", "public");
res.setHeader("Expires", "0");
createZipFile(nodeIds, res.getOutputStream(), new Boolean(noaccentStr));
} catch (RuntimeException e) {
throw new WebScriptException(HttpServletResponse.SC_BAD_REQUEST, e.getMessage());
}
}
public void createZipFile(List<String> nodeIds, OutputStream os, boolean noaccent) throws IOException {
File zip = null;
try {
if (nodeIds != null && !nodeIds.isEmpty()) {
zip = TempFileProvider.createTempFile(TEMP_FILE_PREFIX, ZIP_EXTENSION);
FileOutputStream stream = new FileOutputStream(zip);
CheckedOutputStream checksum = new CheckedOutputStream(stream, new Adler32());
BufferedOutputStream buff = new BufferedOutputStream(checksum);
ZipArchiveOutputStream out = new ZipArchiveOutputStream(buff);
out.setEncoding(encoding);
out.setMethod(ZipArchiveOutputStream.DEFLATED);
out.setLevel(Deflater.BEST_COMPRESSION);
if (logger.isDebugEnabled()) {
logger.debug("Using encoding '" + encoding + "' for zip file.");
}
try {
for (String nodeId : nodeIds) {
NodeRef node = new NodeRef(storeRef, nodeId);
addToZip(node, out, noaccent, "");
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
throw new WebScriptException(HttpServletResponse.SC_BAD_REQUEST, e.getMessage());
} finally {
out.close();
buff.close();
checksum.close();
stream.close();
if (nodeIds.size() > 0) {
InputStream in = new FileInputStream(zip);
try {
byte[] buffer = new byte[BUFFER_SIZE];
int len;
while ((len = in.read(buffer)) > 0) {
os.write(buffer, 0, len);
}
} finally {
IOUtils.closeQuietly(in);
}
}
}
}
} catch (Exception e) {
logger.error(e.getMessage(), e);
throw new WebScriptException(HttpServletResponse.SC_BAD_REQUEST, e.getMessage());
} finally {
// try and delete the temporary file
if (zip != null) {
zip.delete();
}
}
}
public void addToZip(NodeRef node, ZipArchiveOutputStream out, boolean noaccent, String path) throws IOException {
QName nodeQnameType = this.nodeService.getType(node);
// Special case : links
if (this.dictionaryService.isSubClass(nodeQnameType, ApplicationModel.TYPE_FILELINK)) {
NodeRef linkDestinationNode = (NodeRef) nodeService.getProperty(node, ContentModel.PROP_LINK_DESTINATION);
if (linkDestinationNode == null) {
return;
}
// Duplicate entry: check if link is not in the same space of the
// link destination
if (nodeService.getPrimaryParent(node).getParentRef().equals(nodeService.getPrimaryParent(linkDestinationNode).getParentRef())) {
return;
}
nodeQnameType = this.nodeService.getType(linkDestinationNode);
node = linkDestinationNode;
}
String nodeName = (String) nodeService.getProperty(node, ContentModel.PROP_NAME);
nodeName = noaccent ? unAccent(nodeName) : nodeName;
if (this.dictionaryService.isSubClass(nodeQnameType, ContentModel.TYPE_CONTENT)) {
ContentReader reader = contentService.getReader(node, ContentModel.PROP_CONTENT);
if (reader != null) {
InputStream is = reader.getContentInputStream();
String filename = path.isEmpty() ? nodeName : path + '/' + nodeName;
ZipArchiveEntry entry = new ZipArchiveEntry(filename);
entry.setTime(((Date) nodeService.getProperty(node, ContentModel.PROP_MODIFIED)).getTime());
entry.setSize(reader.getSize());
out.putArchiveEntry(entry);
byte buffer[] = new byte[BUFFER_SIZE];
while (true) {
int nRead = is.read(buffer, 0, buffer.length);
if (nRead <= 0) {
break;
}
out.write(buffer, 0, nRead);
}
is.close();
out.closeArchiveEntry();
} else {
logger.warn("Could not read : " + nodeName + "content");
}
} else if (this.dictionaryService.isSubClass(nodeQnameType, ContentModel.TYPE_FOLDER)
&& !this.dictionaryService.isSubClass(nodeQnameType, ContentModel.TYPE_SYSTEM_FOLDER)) {
List<ChildAssociationRef> children = nodeService.getChildAssocs(node);
if (children.isEmpty()) {
String folderPath = path.isEmpty() ? nodeName + '/' : path + '/' + nodeName + '/';
ZipArchiveEntry entry = new ZipArchiveEntry(folderPath);
entry.setSize(0);
entry.setTime(((Date) nodeService.getProperty(node, ContentModel.PROP_MODIFIED)).getTime());
out.putArchiveEntry(entry);
out.closeArchiveEntry();
} else {
for (ChildAssociationRef childAssoc : children) {
NodeRef childNodeRef = childAssoc.getChildRef();
addToZip(childNodeRef, out, noaccent, path.isEmpty() ? nodeName : path + '/' + nodeName);
}
}
} else {
logger.info("Unmanaged type: " + nodeQnameType.getPrefixedQName(this.namespaceService) + ", filename: " + nodeName);
}
}
/**
* ZipEntry() does not convert filenames from Unicode to platform (waiting
* Java 7) http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4244499
*
* #param s
* #return
*/
public static String unAccent(String s) {
String temp = Normalizer.normalize(s, Normalizer.NFD, 0);
return temp.replaceAll("[^\\p{ASCII}]", "");
}
}

Program for file transfer in java using sockets

I have written a java code to transfer files from one server to another using the concept of socket programming. I got the codes from another java forum that meet my requirements. The program is said to transfer large sized files (like .mkv , .mprg movies) from one machine to another and can be used to transfer files of all formats. But after running the codes I found that the program is not able to transfer large sized files such as movies and even pdf of sizes 80mb or 111mb. The program has used bytebuffer but still error occurs. The codes are as follows (I got them from this site http://www.coderpanda.com/java-socket-programming-transferring-large-sized-files-through-socket/)
ClientMain.java
import java.io.IOException;
import java.net.Socket;
public class ClientMain
{
private DirectoryTxr transmitter = null;
Socket clientSocket = null;
private boolean connectedStatus = false;
private String ipAddress;
String srcPath = null;
String dstPath = "";
public ClientMain()
{
}
public void setIpAddress(String ip)
{
this.ipAddress = ip;
}
public void setSrcPath(String path)
{
this.srcPath = path;
}
public void setDstPath(String path)
{
this.dstPath = path;
}
private void createConnection()
{
Runnable connectRunnable = new Runnable()
{
public void run()
{
while (!connectedStatus)
{
try
{
clientSocket = new Socket(ipAddress, 22);
connectedStatus = true;
transmitter = new DirectoryTxr(clientSocket, srcPath, dstPath);
}
catch (IOException io)
{
io.printStackTrace();
}
}
}
};
Thread connectionThread = new Thread(connectRunnable);
connectionThread.start();
}
public static void main(String[] args)
{
ClientMain main = new ClientMain();
main.setIpAddress("localHost");
main.setSrcPath("C:/Transfer/");
main.setDstPath("C:/Receive");
main.createConnection();
}
}
DirectoryTxr.java
import java.io.*;
import java.net.Socket;
import java.net.SocketException;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
public class DirectoryTxr
{
Socket clientSocket = null;
String srcDir = null;
String dstDir = null;
byte[] readBuffer = new byte[1024];
private InputStream inStream = null;
private OutputStream outStream = null;
int state = 0;
final int permissionReqState = 1;
final int initialState = 0;
final int dirHeaderSendState = 2;
final int fileHeaderSendState = 3;
final int fileSendState = 4;
final int fileFinishedState = 5;
private boolean isLive = false;
private int numFiles = 0;
private int filePointer = 0;
String request = "May I send?";
String respServer = "Yes,You can";
String dirResponse = "Directory created...Please send files";
String fileHeaderRecvd = "File header received ...Send File";
String fileReceived = "File Received";
String dirFailedResponse = "Failed";
File[] opFileList = null;
public DirectoryTxr(Socket clientSocket, String srcDir, String dstDir)
{
try
{
this.clientSocket = clientSocket;
inStream = clientSocket.getInputStream();
outStream = clientSocket.getOutputStream();
isLive = true;
this.srcDir = srcDir;
this.dstDir = dstDir;
state = initialState;
readResponse();
sendMessage(request);
state = permissionReqState;
}
catch (IOException io)
{
io.printStackTrace();
}
}
private void sendMessage(String message)
{
try
{
sendBytes(request.getBytes("UTF-8"));
}
catch (UnsupportedEncodingException e)
{
e.printStackTrace();
}
}
private void readResponse()
{
Runnable readRunnable = new Runnable()
{
public void run()
{
while (isLive)
{
try
{
int num = inStream.read(readBuffer);
if (num > 0)
{
byte[] tempArray = new byte[num];
System.arraycopy(readBuffer, 0, tempArray, 0, num);
processBytes(tempArray);
}
}
catch (SocketException se)
{
System.exit(0);
}
catch (IOException io)
{
io.printStackTrace();
isLive = false;
}
}
}
};
Thread readThread = new Thread(readRunnable);
readThread.start();
}
private void sendDirectoryHeader()
{
File file = new File(srcDir);
if (file.isDirectory())
{
try
{
String[] childFiles = file.list();
numFiles = childFiles.length;
String dirHeader = "$" + dstDir + "#" + numFiles + "&";
sendBytes(dirHeader.getBytes("UTF-8"));
}
catch (UnsupportedEncodingException en)
{
en.printStackTrace();
}
}
else
{
System.out.println(srcDir + " is not a valid directory");
}
}
private void sendFile(String dirName)
{
File file = new File(dirName);
if (!file.isDirectory())
{
try
{
int len = (int) file.length();
int buffSize = len / 8;
RandomAccessFile raf = new RandomAccessFile(file, "rw");
FileChannel channel = raf.getChannel();
int numRead = 0;
while (numRead >= 0)
{
ByteBuffer buf = ByteBuffer.allocate(1024 * 100000);
numRead = channel.read(buf);
if (numRead > 0)
{
byte[] array = new byte[numRead];
System.arraycopy(buf.array(), 0, array, 0, numRead);
sendBytes(array);
}
}
System.out.println("Finished");
}
catch (IOException io)
{
io.printStackTrace();
}
}
}
private void sendHeader(String fileName)
{
try
{
File file = new File(fileName);
if (file.isDirectory())
return;
String header = "&" + fileName + "#" + file.length() + "*";
sendHeader(header);
sendBytes(header.getBytes("UTF-8"));
}
catch (UnsupportedEncodingException e)
{
e.printStackTrace();
}
}
private void sendBytes(byte[] dataBytes)
{
synchronized (clientSocket)
{
if (outStream != null)
{
try
{
outStream.write(dataBytes);
outStream.flush();
}
catch (IOException io)
{
io.printStackTrace();
}
}
}
}
private void processBytes(byte[] data)
{
try
{
String parsedMessage = new String(data, "UTF-8");
System.out.println(parsedMessage);
setResponse(parsedMessage);
}
catch (UnsupportedEncodingException u)
{
u.printStackTrace();
}
}
private void setResponse(String message)
{
if (message.trim().equalsIgnoreCase(respServer) && state == permissionReqState)
{
state = dirHeaderSendState;
sendDirectoryHeader();
}
else if (message.trim().equalsIgnoreCase(dirResponse) && state == dirHeaderSendState)
{
state = fileHeaderSendState;
if (LocateDirectory())
{
createAndSendHeader();
}
else
{
System.out.println("Vacant or invalid directory");
}
}
else if (message.trim().equalsIgnoreCase(fileHeaderRecvd) && state == fileHeaderSendState)
{
state = fileSendState;
sendFile(opFileList[filePointer].toString());
state = fileFinishedState;
filePointer++;
}
else if (message.trim().equalsIgnoreCase(fileReceived) && state == fileFinishedState)
{
if (filePointer < numFiles)
{
createAndSendHeader();
}
System.out.println("Successfully sent");
}
else if (message.trim().equalsIgnoreCase(dirFailedResponse))
{
System.out.println("Going to exit....Error ");
}
else if (message.trim().equalsIgnoreCase("Thanks"))
{
System.out.println("All files were copied");
}
}
private void closeSocket()
{
try
{
clientSocket.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
private boolean LocateDirectory()
{
boolean status = false;
File file = new File(srcDir);
if (file.isDirectory())
{
opFileList = file.listFiles();
numFiles = opFileList.length;
if (numFiles <= 0)
{
System.out.println("No files found");
}
else
{
status = true;
}
}
return status;
}
private void createAndSendHeader()
{
File opFile = opFileList[filePointer];
String header = "&" + opFile.getName() + "#" + opFile.length() + "*";
try
{
state = fileHeaderSendState;
sendBytes(header.getBytes("UTF-8"));
}
catch (UnsupportedEncodingException e)
{
}
}
private void sendListFiles()
{
createAndSendHeader();
}
}
ServerMain.java
public class ServerMain {
public ServerMain() {
}
public static void main(String[] args) {
DirectoryRcr dirRcr = new DirectoryRcr();
}
}
DirectoryRcr.java
import java.io.*;
import java.net.ServerSocket;
import java.net.Socket;
import java.net.SocketException;
public class DirectoryRcr
{
String request = "May I send?";
String respServer = "Yes,You can";
String dirResponse = "Directory created...Please send files";
String dirFailedResponse = "Failed";
String fileHeaderRecvd = "File header received ...Send File";
String fileReceived = "File Received";
Socket socket = null;
OutputStream ioStream = null;
InputStream inStream = null;
boolean isLive = false;
int state = 0;
final int initialState = 0;
final int dirHeaderWait = 1;
final int dirWaitState = 2;
final int fileHeaderWaitState = 3;
final int fileContentWaitState = 4;
final int fileReceiveState = 5;
final int fileReceivedState = 6;
final int finalState = 7;
byte[] readBuffer = new byte[1024 * 100000];
long fileSize = 0;
String dir = "";
FileOutputStream foStream = null;
int fileCount = 0;
File dstFile = null;
public DirectoryRcr()
{
acceptConnection();
}
private void acceptConnection()
{
try
{
ServerSocket server = new ServerSocket(22);
socket = server.accept();
isLive = true;
ioStream = socket.getOutputStream();
inStream = socket.getInputStream();
state = initialState;
startReadThread();
}
catch (IOException io)
{
io.printStackTrace();
}
}
private void startReadThread()
{
Thread readRunnable = new Thread()
{
public void run()
{
while (isLive)
{
try
{
int num = inStream.read(readBuffer);
if (num > 0)
{
byte[] tempArray = new byte[num];
System.arraycopy(readBuffer, 0, tempArray, 0, num);
processBytes(tempArray);
}
sleep(100);
} catch (SocketException s)
{
}
catch (IOException e)
{
e.printStackTrace();
}
catch (InterruptedException i)
{
i.printStackTrace();
}
}
}
};
Thread readThread = new Thread(readRunnable);
readThread.start();
}
private void processBytes(byte[] buff) throws InterruptedException
{
if (state == fileReceiveState || state == fileContentWaitState)
{
if (state == fileContentWaitState)
state = fileReceiveState;
fileSize = fileSize - buff.length;
writeToFile(buff);
if (fileSize == 0)
{
state = fileReceivedState;
try
{
foStream.close();
}
catch (IOException io)
{
io.printStackTrace();
}
System.out.println("Received " + dstFile.getName());
sendResponse(fileReceived);
fileCount--;
if (fileCount != 0)
{
state = fileHeaderWaitState;
}
else
{
System.out.println("Finished");
state = finalState;
sendResponse("Thanks");
Thread.sleep(2000);
System.exit(0);
}
System.out.println("Received");
}
}
else
{
parseToUTF(buff);
}
}
private void parseToUTF(byte[] data)
{
try
{
String parsedMessage = new String(data, "UTF-8");
System.out.println(parsedMessage);
setResponse(parsedMessage);
}
catch (UnsupportedEncodingException u) {
u.printStackTrace();
}
}
private void setResponse(String message)
{
if (message.trim().equalsIgnoreCase(request) && state == initialState)
{
sendResponse(respServer);
state = dirHeaderWait;
}
else if (state == dirHeaderWait)
{
if (createDirectory(message))
{
sendResponse(dirResponse);
state = fileHeaderWaitState;
}
else
{
sendResponse(dirFailedResponse);
System.out.println("Error occurred...Going to exit");
System.exit(0);
}
} else if (state == fileHeaderWaitState)
{
createFile(message);
state = fileContentWaitState;
sendResponse(fileHeaderRecvd);
}
else if (message.trim().equalsIgnoreCase(dirFailedResponse))
{
System.out.println("Error occurred ....");
System.exit(0);
}
}
private void sendResponse(String resp)
{
try
{
sendBytes(resp.getBytes("UTF-8"));
}
catch (UnsupportedEncodingException e)
{
e.printStackTrace();
}
}
private boolean createDirectory(String dirName)
{
boolean status = false;
dir = dirName.substring(dirName.indexOf("$") + 1, dirName.indexOf("#"));
fileCount = Integer.parseInt(dirName.substring(dirName.indexOf("#") + 1, dirName.indexOf("&")));
if (new File(dir).mkdir())
{
status = true;
System.out.println("Successfully created directory " + dirName);
}
else if (new File(dir).mkdirs())
{
status = true;
System.out.println("Directories were created " + dirName);
}
else if (new File(dir).exists())
{
status = true;
System.out.println("Directory exists" + dirName);
}
else
{
System.out.println("Could not create directory " + dirName);
status = false;
}
return status;
}
private void createFile(String fileName)
{
String file = fileName.substring(fileName.indexOf("&") + 1, fileName.indexOf("#"));
String lengthFile = fileName.substring(fileName.indexOf("#") + 1, fileName.indexOf("*"));
fileSize = Integer.parseInt(lengthFile);
dstFile = new File(dir + "/" + file);
try
{
foStream = new FileOutputStream(dstFile);
System.out.println("Starting to receive " + dstFile.getName());
}
catch (FileNotFoundException fn)
{
fn.printStackTrace();
}
}
private void writeToFile(byte[] buff)
{
try
{
foStream.write(buff);
}
catch (IOException io)
{
io.printStackTrace();
}
}
private void sendBytes(byte[] dataBytes)
{
synchronized (socket)
{
if (ioStream != null)
{
try
{
ioStream.write(dataBytes);
}
catch (IOException io)
{
io.printStackTrace();
}
}
}
}
}
Note that:-
ClientMain.java and DirectoryTxr.java are the two classes under client application.
ServerMain.java and DirectoryRcr.java are the two classes under Server application.
run the ClientMain.java and ServerMain.java simultaneously
Also specify the source directory, destination directory and host address of the machine in which server is running in the ClientMain.java(as per your computer). Here we are not specifying source file ,instead a source directory or folder is specifying.So the entire files of source directory will be transferred.
I would really appreciate if someone can help me with the problem.
You can check this code. This code can send files till 2GB of file size. Checked and working. Basically what I do here is first I send the file size and file name in an object then I start sending the file. First of all let me show you the class whose object will carry the File's details and send it to Receiver.
FileTransfer Class:
public class FileDetails implements Serializable {
String name;
long size;
public void setDetails(String name, long size) {
this.name = name;
this.size = size;
}
public String getName() {
return name;
}
public long getSize() {
return size;
}
}
Sender:
ServerSocket sendServer = null;
Socket sendSocket;
FileDetails details;
byte data[];
try {
File file = new File("File Url");
// Getting file name and size
if (file.length() > Integer.MAX_VALUE) {
System.out.println("File size exceeds 2 GB");
} else {
sendServer = new ServerSocket(5050);
System.out.println("Waiting for Client...");
sendSocket = sendServer.accept();
// File Object for accesing file Details
System.out.println("Connected to Client...");
data = new byte[2048]; // Here you can increase the size also which will send it faster
details = new FileDetails();
details.setDetails(file.getName(), file.length());
// Sending file details to the client
System.out.println("Sending file details...");
ObjectOutputStream sendDetails = new ObjectOutputStream(sendSocket.getOutputStream());
sendDetails.writeObject(details);
sendDetails.flush();
// Sending File Data
System.out.println("Sending file data...");
FileInputStream fileStream = new FileInputStream(file);
BufferedInputStream fileBuffer = new BufferedInputStream(fileStream);
OutputStream out = sendSocket.getOutputStream();
int count;
while ((count = fileBuffer.read(data)) > 0) {
System.out.println("Data Sent : " + count);
out.write(data, 0, count);
out.flush();
}
out.close();
fileBuffer.close();
fileStream.close();
}
} catch (Exception e) {
System.out.println("Error : " + e.toString());
}
Receiver's code :
int port = 5050;
try {
System.out.println("Connecting to Server...");
Socket receiveSocket = new Socket("IP of Server", port);
System.out.println("Connected to Server...");
// Getting file details
System.out.println("Getting details from Server...");
ObjectInputStream getDetails = new ObjectInputStream(receiveSocket.getInputStream());
FileDetails details = (FileDetails) getDetails.readObject();
System.out.println("Now receiving file...");
// Storing file name and sizes
String fileName = details.getName();
System.out.println("File Name : " + fileName);
byte data[] = new byte[2048]; // Here you can increase the size also which will receive it faster
FileOutputStream fileOut = new FileOutputStream("D:\\" + fileName);
InputStream fileIn = receiveSocket.getInputStream();
BufferedOutputStream fileBuffer = new BufferedOutputStream(fileOut);
int count;
int sum = 0;
while ((count = fileIn.read(data)) > 0) {
sum += count;
fileBuffer.write(data, 0, count);
System.out.println("Data received : " + sum);
fileBuffer.flush();
}
System.out.println("File Received...");
fileBuffer.close();
fileIn.close();
} catch (Exception e) {
System.out.println("Error : " + e.toString());
}
Hope this helps you.
You don't need all this. The canonical way to copy a stream in Java is as follows. It works for any buffer size greater than zero. I generally use 8192. There is certainly no necessity to read entire files into memory. It just wastes time and space.
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
If you know the size in advance and need to keep the socket open for another transfer:
while (total < length && (count = in.read(buffer, 0, length-total > buffer.length ? buffer.length : (int)(length-total))) > 0)
{
out.write(buffer, 0, count);
total += count;
}
where total is a long initialized to zero before this loop, and length is the length you know in advance.

Categories

Resources