i have an application that want to write a file using fileoutputstream
here's the code, method patch
public static Response patch() {
try {
System.out.println("PATCH");
System.out.println(request.contentType);
String file = params.get("filename");
System.out.println("patch file: " + file);
Map<String, Header> MapOffset = request.headers;
for (Entry<String, Header> entry : MapOffset.entrySet()) {
System.out.println("Header['" + entry.getKey() + "]: "
+ entry.getValue().value());
}
Header offsetParam = MapOffset.get("offset");
Long offset = 0L;
if (offsetParam != null) {
offset = Long.parseLong(offsetParam.value());
}
InputStream input = request.body;
File f = new File(UPLOAD_DIR + System.getProperty("file.separator")
+ file);
System.out.println("address: " + f.getAbsolutePath());
System.out.println("offset: " + offset);
System.out.println("length: " + f.length());
fileBasicUpload(f, offset, input);
Response respon = new Response();
respon.status = OK;
return respon;
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
and this is where i write a file
private static void fileBasicUpload(File f, Long offset, InputStream input)
throws IOException {
FileOutputStream output = null;
try {
int c = -1;
byte[] b = new byte[1024];
try {
output = new FileOutputStream(f, true);
while ((c = input.read(b)) != -1) {
output.write(b, 0, c);
}
} catch (Exception e) {
System.out.println("Error: " + e.getMessage());
}
} finally {
output.close();
}
}
but when my application called, then stream closed error is show up at while ((c = input.read(b)) != -1) that line.
i don't know how that error is called. sorry for my poor english and thanks
i found the answer. in my application i found like this
public static Response upload(File file){
System.out.println("Appliaction.upload");
response = ResumableUpload.post();
return response;
// render(response);
}
the parameter file, it must be delete, then it work!
Related
I'm uploading bulk files (say 50000) to Box through Box Java API's. To do that we call the uploadFile method 50000 times. The method also creates and close the file input stream.
Is there a way to keep the stream open until the bulk load is done? Even if I close the stream in finally block, it will close it every time I call the method.
private static String uploadFile(String pathFileName, BoxAPIConnection api, BoxFolder folder) {
boolean fileExists = false;
String fileId = null;
FileInputStream stream = null;
log.debug("Invoked uploadFileAsBoxAppUser-uploadFile :" + pathFileName + ":" + api + ":" + folder);
try {
String fileName = pathFileName.substring(pathFileName.lastIndexOf("\\") + 1, pathFileName.length());
log.debug("fileName :" + fileName);
for (BoxItem.Info itemInfo : folder) {
if (itemInfo instanceof BoxFile.Info) {
BoxFile.Info fileInfo = (BoxFile.Info) itemInfo;
if (fileName.equals(fileInfo.getName())) {
fileExists = true;
fileId = fileInfo.getID();
log.debug("fileExists in Destination box Folder fileID " + fileId);
}
}
}
if (!fileExists) {
log.debug("uploading new file: " + fileName);
stream = new FileInputStream(pathFileName);
BoxFile.Info boxInfo = folder.uploadFile(stream, pathFileName);
fileId = boxInfo.getID();
} else {
log.debug("uploading new version of file: " + fileName);
BoxFile file = new BoxFile(api, fileId);
stream = new FileInputStream(pathFileName);
file.uploadVersion(stream);
}
} catch (IOException e) {
log.debug("Exception in uploadFileAsBoxAppUser :" + e);
}
finally {
if (stream != null)
try {
stream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return fileId;
}
The method I wrote to download files always produce corrupted files.
public static String okDownloadToFileSync(final String link, final String fileName, final boolean temp, DownloadStatusManager statusManager, ErrorDisplayerInterface errorDisplayerInterface) {
Request request = new Request.Builder()
.url(link)
.build();
OkHttpClient client = Api.getInstance().getOkHttpClient();
OutputStream output = null;
InputStream input = null;
try {
Response response = client.newCall(request).execute();
//Add the file length to the statusManager
final int contentLength = Integer.parseInt(response.header("Content-Length"));
if (statusManager != null) {
statusManager.add(Hash.md5(link), contentLength);
}
//Get content type to know extension
final String contentType = response.header("Content-Type");
final String ext = contentTypeMap.get(contentType);
Log.i(TAG, link + "\n --> contentType = " + contentType + "\n --> ext = " + ext);
if (ext == null) {
Log.e(TAG, "-----------\next is null, seems like there is a problem with that url : \n " + link + "\n----------");
return null;
} else if (ext.equals("json")) {
Log.e(TAG, "-----------\ndownloadable file seems to be a json, seems like there is a problem with that url : \n " + link + "\n----------");
return null;
}
//Check if file already exists
if (!temp && fileName != null) {
File test = new File(M360Application.getContext().getFilesDir(), fileName + "." + ext);
if (test.exists()) {
Log.i(TAG, "File exists ! : " + test.getPath());
test.delete();
//return test.getAbsolutePath();
}
}
// expect HTTP 200 OK, so we don't mistakenly save error report instead of the file
if (!response.isSuccessful()) {
errorDisplayerInterface.popWarn(null, "Error while downloading " + link, "connection.getResponseCode() != HttpURLConnection.HTTP_OK");
return null;
}
input = response.body().byteStream();
File file;
if (temp) {
file = File.createTempFile(UUID.randomUUID().toString(), ext, M360Application.getContext().getCacheDir());
} else {
file = new File(M360Application.getContext().getFilesDir(), fileName + "." + ext);
}
output = new FileOutputStream(file);
output.write(response.body().bytes());
// byte data[] = new byte[4096];
// long total = 0;
// int count;
// while ((count = input.read(data)) != -1) {
// output.write(data, 0, count);
// total++;
//
// if (statusManager != null) {
// statusManager.update(Hash.md5(link), contentLength - total);
// }
// }
return file.getAbsolutePath();
} catch (IOException e) {
e.printStackTrace();
errorDisplayerInterface.popError(null, e);
} finally {
if (statusManager != null) {
statusManager.finish(Hash.md5(link));
}
try {
if (output != null)
output.close();
if (input != null)
input.close();
} catch (IOException ignored) {
ignored.printStackTrace();
}
}
return null;
}
I access these file via adb, transfer them to my sccard, and there I see that they seem to have the proper size, but has no type according to for instance Linux file command.
Would you know what is missing and how to fix it?
Thank you.
Edit
Simpler version of the code ( but the bug is the same )
public static String okioDownloadToFileSync(final String link, final String fileName) throws IOException {
Request request = new Request.Builder()
.url(link)
.build();
OkHttpClient client = Api.getInstance().getOkHttpClient();
Response response = client.newCall(request).execute();
final int contentLength = Integer.parseInt(response.header("Content-Length"));
//Get content type to know extension
final String contentType = response.header("Content-Type");
final String ext = contentTypeMap.get(contentType);
// expect HTTP 200 OK, so we don't mistakenly save error report instead of the file
if (!response.isSuccessful()) {
return null;
}
File file = new File(M360Application.getContext().getFilesDir(), fileName + "." + ext);
BufferedSink sink = Okio.buffer(Okio.sink(file));
sink.writeAll(response.body().source());
sink.close();
Log.i(TAG, "file.length : " + file.length() + " | contentLength : " + contentLength);
return file.getAbsolutePath();
}
The log : file.length : 2485394 | contentLength : 1399242
Solution
The problem was that I was getting the OkHttpClient from my API singleton, which was used by retrofit and had multiples interceptors. Those interceptors were polluting the response.
So I OkHttpClient client = Api.getInstance().getOkHttpClient(); became OkHttpClient client = new OkHttpClient.Builder().build(); and everything is now ok !
Thanks a lot. I'm dividing the method into smaller pieces right now.
Instead of
output.write(response.body().bytes());
try something like this
byte[] buff = new byte[1024 * 4];
while (true) {
int byteCount = input.read(buff);
if (byteCount == -1) {
break;
}
output.write(buff, 0, byteCount);
}
I'm writing a code that needs to continually updates a text file in a loop: remove the actual content and write another one. The problem is the update in not done correctly. It uses the previous version of the file.
For example, in loop 4, the called file is the file updated in the loop 3.
My question is how to update my file correctly ?
public static void main(String[] args) throws IOException {
for(int level:levels){
RecipeTree.CloneTree(root, partialroot);
for(int i =0; i<100; i++){
RecipeTree.removalcondition = RecipeTree.levelOfConditions(depth, length, recipe, level);
System.out.println(removalcondition);
RecipeTree.PartialTree(partialroot, RecipeTree.removalcondition);
// Here the update of the file
InitSTRIPSPlanner(partialroot);
for(RecipeTree leaf: partialroot.getLeaves()){
...........
}
}
}
public static void InitSTRIPSPlanner(RecipeTree root) throws IOException{
String adresseBut = System.getProperty("user.dir") + "/prolog/test-2p/Domain_knowledge.pl";
String adresseSource = System.getProperty("user.dir") + "/prolog/test-2p/STRIPS_planner.pl";
try {
copyFileUsingStream(new File(adresseSource), new File(adresseBut));
} catch (IOException e) {
e.printStackTrace();
}
File fw = new File (adresseBut);
OutputStream output = new FileOutputStream(fw,true);
output.write("\n".getBytes());
output.flush();
FromTreeToProlog(root,output);
output.close();
}
public static void FromTreeToProlog(RecipeTree root, OutputStream output) throws IOException{
for(RecipeTree leaf: root.getLeaves()){
if (leaf.getHead().getPostconditions() != null) {
output.write("\n".getBytes());
output.flush();
output.write("\n".getBytes());
output.flush();
if (leaf.getHead().getPreconditions() != null) {
output.write(("strips_preconditions("
+ leaf.getHead().getName().toLowerCase() + ",["
+ leaf.getHead().getPreconditions().toLowerCase()+ "]).").getBytes());
output.write("\n".getBytes());
output.flush();
}
else {
output.write(("strips_preconditions("
+ leaf.getHead().getName().toLowerCase() + ",[_]).").getBytes());
output.write("\n".getBytes());
output.flush();
}
output.write(("strips_achieves("
+ leaf.getHead().getName().toLowerCase() + ","
+ leaf.getHead().getPostconditions().toLowerCase()
+ ").").getBytes());
}
}
for (String i : conditions) {
output.write(("strips_primitive(" + i.toLowerCase() + ").").getBytes());
output.write("\n".getBytes());
output.flush();
}
for (String recipe : RecipeTree.RecipeCondition) {
output.write(("strips_preconditions(" + recipe.toLowerCase() + ",[_]).").getBytes());
output.write("\n".getBytes());
output.flush();
output.write(("strips_achieves(" + recipe.toLowerCase() + ",c"
+ recipe.toLowerCase() + ").").getBytes());
output.write("\n".getBytes());
output.flush();
output.write(("strips_primitive(c" + recipe.toLowerCase() + ").").getBytes());
output.write("\n".getBytes());
output.flush();
}
}
private static void copyFileUsingStream(File source, File dest) throws IOException {
InputStream is = null;
OutputStream os = null;
try {
is = new FileInputStream(source);
os = new FileOutputStream(dest);
byte[] buffer = new byte[1024];
int length;
while ((length = is.read(buffer)) > 0) {
os.write(buffer, 0, length);
}
} finally {
is.close();
os.close();
}
}
}
new FileOutputStream(fw, true);
is appending and should be either one of
new FileOutputStream(fw, false);
new FileOutputStream(fw);
if I understand you correctly.
BTW.
copyFileUsingStream(new File(adresseSource), new File(adresseBut));
could be (since Java 7):
Files.copy(Paths.get(adresseSource), Paths.get(adresseBut),
StandardCopyOption.REPLACE_EXISTING);
I'm retrieving files list from FTPClient using ftpClient.listFiles(); and it returns an array with correct file names and appropriate quantity.
I'm trying to read content from files, but only ftpClient.retrieveFileStream(fileName) is available for that purpose. And it somehow breaks after reading the first file and returns null.
Is there a way to convert FTPFile directly into String?
ftp.connect();
ftp.changeWorkingDirectoryToFrom();
List<String> idsList = new ArrayList<String>();
List<String> names = ftp.listFileNames();
for (String fileName : names) {
String content = fromFTPFileToString(fileName);
Matcher matcher = FILES_PATTERN.matcher(content);
String id = extractId(content);
if (matcher.find()) {
boolean duplicate = idsList.contains(id);
LOG.info("MATCHED: " + fileName);
if (!duplicate) {
ftp.moveFileFromTo(fileName);
idsList.add(id);
} else {
LOG.info("DUPLICATE: " + fileName);
duplicated++;
ftp.deleteFileOnFromFtp(fileName);
}
}
processed++;
}
ftp.disconnect();
private String fromFTPFileToString(String fileName) {
String content = "";
try {
InputStream is = ftp.readContentFromFTPFile(fileName);
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, ENCODING);
content = writer.toString();
IOUtils.closeQuietly(is);
IOUtils.closeQuietly(writer);
} catch (Exception ex) {
LOG.error(ex);
}
return content;
}
void deleteFileOnFromFtp(String fileName) {
changeWorkingDirectory(properties.getProperty(PropertiesType.FOLDER_FROM.toString()));
deleteFile(fileName);
}
InputStream readContentFromFTPFile(String fileName) {
changeWorkingDirectory(properties.getProperty(PropertiesType.FOLDER_FROM.toString()));
InputStream inputStream = null;
try {
inputStream = ftpClient.retrieveFileStream(fileName);
} catch (IOException ex) {
LOG.error("Unable to extract content from file:" + O_Q + fileName + C_Q);
}
return inputStream;
}
void moveFileFromTo(String fileName) {
String from = FORWARD_SLASH + properties.getProperty(PropertiesType.FOLDER_FROM.toString()) + FORWARD_SLASH + fileName;
String to = FORWARD_SLASH + properties.getProperty(PropertiesType.FOLDER_TO.toString()) + FORWARD_SLASH + fileName;
try {
ftpClient.rename(from, to);
} catch (IOException ex) {
LOG.error("Unable to move file file from:" + O_Q + from + C_Q + " to: " + O_Q + to + C_Q);
throw new RuntimeException(ex.getCause());
}
}
3 Hints
1) You can use retrieveFile
And use a ByteArrayOutputStream as the 2nd parameter.
To get a String from it simply "new String(baos.toByteArray(),[The Charset you used]);"
2) Check for ftpClient.completePendingCommand();
If some commands are still pending
3) I had once a similar issue setting ftp.enterLocalPassiveMode() helped
I am trying to use asynchronousFileChannel to write the date into a text file. I made 3 jar file of the program with the AsynchronousFileChannel and compiled all 3 jars simultaneously through command prompt to read 3 different text files and output to one common temporary file
I have 2000 records in my test files(3) to be read,but the output in the common temporary file is missing some of the records,the output should have 6000 records but it shows only 5366 or 5666 or sometimes less than that.
I am not able to figure out why some data is lost as it is the functionality of a asynchronousFileChannel.
Here is the code for the java program using asynchronousfilechannel.
class Writer(){
public void writeOut(ReadableData fileData)
throws InterruptedException {
Path file = null;
AsynchronousFileChannel asynchFileChannel = null;
String filePath = tempFileName;
try {
file = Paths.get(filePath);
asynchFileChannel = AsynchronousFileChannel.open(file,
StandardOpenOption.WRITE, StandardOpenOption.CREATE);
CompletionHandler<Integer, Object> handler = new CompletionHandler<Integer, Object>() {
#Override
public void completed(Integer result, Object attachment) {
if (result == Integer.MAX_VALUE) {
log.debug("Attachment: " + attachment + " " + result
+ " bytes written");
log.debug("CompletionHandler Thread ID: "
+ Thread.currentThread().getId());
}
result++;
}
#Override
public void failed(Throwable e, Object attachment) {
try {
throw e;
} catch (Throwable e1) {
e1.printStackTrace();
}
log.debug("File Write Failed Exception:");
e.printStackTrace();
}
};
String printData = fileData.getId() + "|"
+ fileData.getName() + "|" + fileData.getEmpId()
+ "|" + fileData.getServieId() + "|" + "\n";
asynchFileChannel.write(ByteBuffer.wrap(printData.getBytes()),
asynchFileChannel.size(), "file write", handler);
log.debug(printData);
}
}
catch (IOException e) {
e.printStackTrace();
log.error(e.getMessage());
} finally {
}
}
}
}
and this is my class to read data from 3 files:
public class FileReader1 {
static Logger log = Logger.getLogger(FileHandlerNorthBoundMain.class
.getName());
Writer wrO=new Writer();
public static void main(String[] args) throws IOException,
IllegalFileFormatException, InterruptedException {
String filePath = "C:\\Users\\Public\\testdata1.csv"; //"C:\\Users\\Public\\testdata2.csv"; "C:\\Users\\Public\\testdata3.csv";
File file = new File(filePath);
log.info("Fetching data.... from: " + filePath);
ArrayList<ReadableData> list = new ArrayList<ReadableData>();
FileInputStream fs = null;
BufferedReader reader = null;
String Name;
int Id, EmpId, ServiceId;
ReadableData readableData = null;
int count = 0;
fs = new FileInputStream(file);
reader = new BufferedReader(new InputStreamReader(fs));
String line = reader.readLine();
while (line != null) {
StringTokenizer st = new StringTokenizer(line, "\\|");
while (st.hasMoreTokens()) {
try {
Id = Integer.parseInt(st.nextToken());
Name = st.nextToken();
EmpId = Integer.parseInt(st.nextToken());
ServiceId = Integer.parseInt(st.nextToken());
readableData = new ReadableData(Id,
, Name, EmpId,ServiceId);
wrO.writeOut(readableData);
list.add(count, readableData);
count = count++;
} catch (Exception ex) {
log.error("Illegal File Format");
throw new IllegalFileFormatException("Illegal File Format");
}
}
line = reader.readLine();
}
reader.close();
}
Modify your Writer class with the following code part with asynchronousFileChannel lock()
byte[] test = printData.getBytes();
Future<FileLock> featureLock = asynchFileChannel.lock();
log.info("Waiting for the file to be locked ...");
FileLock lock = featureLock.get();
if (lock.isValid()) {
log.debug(printData);
Future<Integer> featureWrite = asynchFileChannel.write(
ByteBuffer.wrap(test), asynchFileChannel.size());
log.info("Waiting for the bytes to be written ...");
int written = featureWrite.get();
log.info("I’ve written " + written + " bytes into "
+ file.getFileName() + " locked file!");
lock.release();
}
This might be because asynchronousFileChannel is thread safe but Bytebuffer is not,care should be taken to ensure that the buffer is not accessed until after the operation has completed.
check the documentation http://openjdk.java.net/projects/nio/javadoc/java/nio/channels/AsynchronousFileChannel.html