I've a java program that writes files to a directory on Linux VM. After writing 4.9 million files, it is failing with the following error:
java.io.FileNotFoundException: /home/user/test/6BA30639CA0A2772AA0217312B3E847E2399E9A25F50F9960D6A670F4F2533EF.blob.lock (No space left on device)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:124)
at com.xyz.azure.AsyncADLSHashFileUploader.SaveToLocalStore(AsyncADLSHashFileUploader.java:231)
My logic is:
public String SaveToLocalStore(byte[] bytes, String subfolder) throws Exception {
MessageDigest md = MessageDigest.getInstance("SHA-256");
md.update(bytes);
byte[] sha256 = md.digest();
String sha256str = bytesToHex(sha256);
Path tmpDir = Paths.get(LocalRootDirectory.toString(), subfolder);
Path tmpFile = Paths.get(tmpDir.toString(), sha256str + ".tmp");
Path localFile = Paths.get(LocalRootDirectory.toString(), subfolder, sha256str + ".blob");
String dstFile = getDestFileFromSrcFile(localFile.toString());
//noinspection ResultOfMethodCallIgnored
tmpDir.toFile().mkdirs();
//We can make a safe assumption that if the .blob file is present, that means that it has been fully written to
if (!Files.exists(localFile)) {
try (
RandomAccessFile randomAccessFile = new RandomAccessFile(localFile + ".lock", "rw");
FileChannel fc = randomAccessFile.getChannel();
FileLock fileLock = fc.tryLock()) {
//if other thread/process is already handling this file... no point of us doing anything
if (fileLock != null) {
//local file is already there, no need to write it again
try (FileOutputStream fos = new FileOutputStream(tmpFile.toString())) {
fos.write(bytes);
fos.flush();
fos.close();
Files.move(tmpFile, localFile, REPLACE_EXISTING);
if(statisticsEnabled) {
synchronized (StatsLock) {
StatsSavedHashes.add(localFile.toString());
}
}
} catch (Exception e) {
LOGGER.error("Failed to create local temp file: " + tmpFile + ": " + e, e);
//cleanup temp if it exists
Files.deleteIfExists(tmpFile);
throw e;
}
}
} catch (OverlappingFileLockException e) {
//other thread is handling it already, so just carry one as if(fileLock == null) was at play
return dstFile;
}
catch (Exception e) {
LOGGER.error("Error while saving local file: " + localFile + ": " + e, e);
throw e;
}
}
return dstFile;
}
I can see that there is more than 80% disk space available.
I've also checked for available inodes on the file system.
I don't know where might be the problem
Can someone please help me with this regard?
Related
I have a piece of code to copy a specific file, I've used this functions for ages and it works properly.
The problem is that right now I'm writing a program with java awt/swing and my copyFile functions only works in debug mode...I can't understand why..
This is the error it throws:
can't copy directory: QQQ.wav
source file is unreadable: QQQ.wav
Error occoured QQQ.wav (The system cannot find the file specified)
But when I run the program in debug mode it works..!!
could anyone help me please???
Function copyFile:
public static void copyFile(File varFromFile, File varToFile) throws IOException {
// First make sure the source file exists, is a file, and is readable.
if (!varFromFile.exists())
System.err.println("no such source file: " + varFromFile);
if (!varFromFile.isFile())
System.err.println("can't copy directory: " + varFromFile);
if (!varFromFile.canRead())
System.err.println("source file is unreadable: " + varFromFile);
// If the destination is a directory, use the source file name
// as the destination file name
if (varToFile.isDirectory())
varToFile = new File(varToFile, varFromFile.getName());
// If the destination exists, make sure it is a writable file
// and ask before overwriting it. If the destination doesn't
// exist, make sure the directory exists and is writable.
if (varToFile.exists()) {
if (!varToFile.canWrite())
System.out.println("destination file is unwriteable: "
+ varToFile);
} else {
// If file doesn't exist, check if directory exists and is
// writable. If getParent() returns null, then the directory is
// the current directory. so look up the user. Directory system
// property to
// find out what that is.
// The destination directory
String varParent = varToFile.getParent();
// If none, use the current directory
if (varParent == null)
varParent = System.getProperty("user.dir");
// Convert it to a file.
File vardir = new File(varParent);
if (!vardir.exists())
System.out.print("destination directory doesn't exist: "
+ varParent);
if (vardir.isFile())
System.out
.print("destination is not a directory: " + varParent);
if (!vardir.canWrite())
System.out.print("destination directory is unwriteable: "
+ varParent);
}
// If we've gotten this far, then everything is okay.
// So we copy the file, a buffer of bytes at a time.
// Stream to read from source
FileInputStream varFromSource = null;
// Stream to write to destination
FileOutputStream VarToDestination = null;
try {
// Create input stream
varFromSource = new FileInputStream(varFromFile);
// Create output stream
VarToDestination = new FileOutputStream(varToFile);
// To hold file contents
byte[] buffer = new byte[4096];
// How many bytes in buffer
int bytes_read;
// Read until EOF
while ((bytes_read = varFromSource.read(buffer)) != -1)
VarToDestination.write(buffer, 0, bytes_read);
//System.out.println("File copied !!!");
} catch (Exception e) {
System.err.println("Error occoured " + e.getMessage());
} finally {
if (varFromSource != null) {
try {
varFromSource.close();
} catch (IOException e) {
System.err.println("Error is " + e.getMessage());
}
}
if (VarToDestination != null) {
try {
VarToDestination.close();
} catch (IOException e) {
System.err.println("Error is " + e.getMessage());
}
}
}
Hello i m trying to upload file using java file.. but i don't get it.. i get file size=0 i'm providing here my java code. tell me why i cant upload on particular folder. i want to store my file in particular folder. i am trying to get file size, file name but i got the null value where am i wrong please tell me.
public void updateTesti(ActionRequest actionRequest,ActionResponse actionResponse) throws IOException, PortletException
{
//image upload logic
String folder_for_upload =(getPortletContext().getRealPath("/"));
//String folder=actionRequest.getParameter("uploadfolder");
realPath=getPortletContext().getRealPath("/");
logger.info("RealPath is" + realPath);
logger.info("Folder is :" + folder_for_upload);
try
{
logger.info("Admin is try to upload");
UploadPortletRequest uploadRequest = PortalUtil.getUploadPortletRequest(actionRequest);
if (uploadRequest.getSize("fileName") == 0) {
SessionErrors.add(actionRequest, "error");
}
String sourceFileName = uploadRequest.getFileName("fileName");
File uploadedFile = uploadRequest.getFile("fileName");
System.out.println("Size of uploaded file: " + uploadRequest.getSize("fileName"));
logger.info("Uploded file name is: " + uploadRequest.getFileName("fileName"));
String destiFolder=("/home/ubuntu/liferay/liferay-portal-6.1.1-ce-ga2/tomcat-7.0.27/webapps/imageUpload-portlet/image");
String newsourcefilename = (uploadRequest.getFileName("fileName"));
File newFile = new File(destiFolder +"/"+ newsourcefilename);
logger.info("New file name: " + newFile.getName());
logger.info("New file path: " + newFile.getPath());
InputStream in = new BufferedInputStream(uploadRequest.getFileAsStream("fileName"));
FileInputStream fis = new FileInputStream(uploadedFile);
FileOutputStream fos = new FileOutputStream(newFile);
byte[] bytes_ = FileUtil.getBytes(in);
int i = fis.read(bytes_);
while (i != -1) {
fos.write(bytes_, 0, i);
i = fis.read(bytes_);
}
fis.close();
fos.close();
Float size = (float) newFile.length();
System.out.println("file size bytes:" + size);
System.out.println("file size Mb:" + size / 1048576);
logger.info("File created: " + newFile.getName());
SessionMessages.add(actionRequest, "success");
}
catch (FileNotFoundException e)
{
System.out.println("File Not Found.");
e.printStackTrace();
SessionMessages.add(actionRequest, "error");
}
catch (NullPointerException e)
{
System.out.println("File Not Found");
e.printStackTrace();
SessionMessages.add(actionRequest, "error");
}
catch (IOException e1)
{
System.out.println("Error Reading The File.");
SessionMessages.add(actionRequest, "error");
e1.printStackTrace();
}
}
You need to do this to upload small files < 1kb
File f2 = uploadRequest.getFile("fileupload", true);
They are stored in memory only. I have it in my catch statement incase I get a null pointer - or incase my original file (f1.length) == 0
I have executed your code.It is working as per expectation.There might be something wrong in your jsp page.I am not sure but might be your name attribute is not same as the one which you are using in processAction(assuming that you are using portlet).Parameter is case sensitive,so check it again.
You will find more on below link.It has good explanation in file upload.
http://www.codeyouneed.com/liferay-portlet-file-upload-tutorial/
I went through a file upload code, and when i implement that in my local system what i got is, portlet is saving the file i upload in tomcat/webbapp/abc_portlet_project location, what i dont understand is from where portlet found
String folder = getInitParameter("uploadFolder");
String realPath = getPortletContext().getRealPath("/");
System.out.println("RealPath" + realPath +"\\" + folder); try {
UploadPortletRequest uploadRequest =
PortalUtil.getUploadPortletRequest(actionRequest);
System.out.println("Size: "+uploadRequest.getSize("fileName"));
if (uploadRequest.getSize("fileName")==0)
{SessionErrors.add(actionRequest, "error");}
String sourceFileName = uploadRequest.getFileName("fileName"); File
file = uploadRequest.getFile("fileName");
System.out.println("Nome file:" +
uploadRequest.getFileName("fileName")); File newFolder = null;
newFolder = new File(realPath +"\" + folder);
if(!newFolder.exists()){ newFolder.mkdir(); }
File newfile = null;
newfile = new File(newFolder.getAbsoluteFile()+"\"+sourceFileName);
System.out.println("New file name: " + newfile.getName());
System.out.println("New file path: " + newfile.getPath());
InputStream in = new
BufferedInputStream(uploadRequest.getFileAsStream("fileName"));
FileInputStream fis = new FileInputStream(file); FileOutputStream fos
= new FileOutputStream(newfile);
trying to get length of file which from local storage. File exists 100% (because I even tried to create it straight before getting the length (and checked it exists). Code is as simple as:
try {
InputStream is = new FileInputStream("errorlog2.txt");
// Get the size of the file
long length = file.length();
// Close the input stream and return length
is.close();
return length;
}
catch (IOException e) {
Log.w(BaseHelper.TAG_MAIN_ACTIVITY, "bad stuff: ", e);
return 0;
in 100% cases it throws an exception. What might be the problem? What`s correct way to get length of local storage files?
thanks a lot!
UPDATE (full code) - file exists and readable, but no length :(
//creating file
String someFileName = "errlogtest.log";
try {
FileOutputStream fOut = openFileOutput(someFileName, Context.MODE_APPEND);
OutputStreamWriter osw = new OutputStreamWriter(fOut);
osw.write("dsgaadfg0df9g0sdf90sg9058349 sdf");
osw.flush();
osw.close();
} catch (IOException ioe) {
ioe.printStackTrace();
}
//reading file
try {
FileInputStream fileToOpen = openFileInput(someFileName);
byte[] readerByte = new byte[fileToOpen.available()];
while (fileToOpen.read(readerByte) != -1) {
}
String fileContents = new String(readerByte);
fileContents.toString();
// next line works fine
Toast.makeText(getApplicationContext(), "FILE CONTENTS: " + fileContents, Toast.LENGTH_LONG).show();
} catch (FileNotFoundException e) {
Log.w("Reading file", "Unable to open file");
} catch (IOException e) {
Log.w("Reading file", "Error reading file");
}
// trying to get length
// simply not working:
File file = new File(someFileName);
long length2 = file.length();
Toast.makeText(getApplicationContext(), "FILE LENGTH:" + length2, Toast.LENGTH_LONG).show();
// shows zero and filenotfoundexception: /errlogtest.log
try {
RandomAccessFile raffile = new RandomAccessFile(new File(someFileName), "r");
long length = raffile.length();
Toast.makeText(getApplicationContext(), "FILE LENGTH:" + length, Toast.LENGTH_LONG).show();
catch (IOException e) {
Log.w("Reading file", "Error reading file", e);
}
help please
Depending on where you are creating the file then you'll need to provide a full path and not just a file name.
For example, if the file is being created using Context.openFileOutput(...) then you would use something like...
long length = new File(getFilesDir().getAbsolutePath() + "/errorlog2.txt").length();
Without opening the file you can get its length:
Method 1:
File file = new File("someFile.txt");
long length = file.length();
Method 2:
RandomAccessFile raf = new RandomeAccessFile(new File("someFile.txt"), "r");
long length = raf.length();
I had a problem with reading /proc/%d/stat files using my Java method copyFiles() (source code below).
I have found workaround using similar readProc() method.
Now I am wondering what was the problem. Output files were created, but each file had 0 bytes (in /proc/ all files are 0 bytes because it is not standard filesystem). FileUtils is from the Apache Commons IO library.
I've tried to do the same using java.nio - again, IOException is being thrown that attributes are wrong for each file.
I removed some part of the code regarding parsing exceptions etc.
Why does this work with FileInputStream, but not with FileUtils.copyFile()?
public void copyFiles() {
final File dir = new File("/proc");
final String[] filedirArray = dir.list();
long counter = 0;
for(String filedir : filedirArray) {
final File checkFile = new File(dir, filedir);
if (checkFile.isDirectory()) {
try {
Integer.parseInt(filedir);
File srcFile = new File(checkFile, "stat");
File dstFile = new File("/home/waldekm/files/stat" + "." + Long.toString(counter++));
try {
FileUtils.copyFile(srcFile, dstFile);
} catch (IOException e1) {}
} catch (NumberFormatException e) {
// not a number, do nothing
}
}
}
}
public static void readProc(final String src, final String dst) {
FileInputStream in = null;
FileOutputStream out = null;
File srcFile = new File(src);
File dstFile = new File(dst);
try {
in = new FileInputStream(srcFile);
out = new FileOutputStream(dstFile);
int c;
while((c = in.read()) != -1) {
out.write(c);
}
} catch (IOException e1) {
} finally {
try {
if (in != null) {
in.close();
}
} catch (IOException e1) {}
try {
if (out != null) {
out.close();
}
} catch (IOException e1) {}
}
The reason is most likely that the operating system is reporting the file size as zero.
On my machine, man 2 stat says this:
"For most files under the /proc directory, stat() does not return the file size in the st_size field; instead the field is returned with the value 0."
(The stat system call will be what the JVM uses to find out what a file's size is.)
Here is a code snipped that would read specific fields from a proc file, using methods that are available (but not documented directly) in the Process class of Android. Modify the FORMAT buffer and the output buffer size to read more/different values from the proc file,
int PROC_SPACE_TERM = (int)' ';
int PROC_OUT_LONG = 0x2000
public static final int[] PROCESS_STATS_FORMAT = new int[] {
PROC_SPACE_TERM,
PROC_SPACE_TERM,
PROC_SPACE_TERM,
PROC_SPACE_TERM,
PROC_SPACE_TERM,
PROC_SPACE_TERM,
PROC_SPACE_TERM,
PROC_SPACE_TERM,
PROC_SPACE_TERM,
PROC_SPACE_TERM,
PROC_SPACE_TERM,
PROC_SPACE_TERM,
PROC_SPACE_TERM,
PROC_SPACE_TERM|PROC_OUT_LONG, // 13: utime
PROC_SPACE_TERM|PROC_OUT_LONG // 14: stime
};
long buf[] = new long[2];
try {
int pid = 1000; // Assume 1000 is a valid pid for a process.
Method mReadProcFile =
Process.class.getMethod("readProcFile", String.class,
int[].class, String[].class,
long[].class, float[].class);
mReadProcFile.invoke(null, "/proc/" + pid + "/stat",
PROCESS_STATS_FORMAT, null, buf, null);
return buf;
} catch(NoSuchMethodException e) {
Log.e(TAG, "Error! Could not get access to JNI method - readProcFile");
} catch (InvocationTargetException e) {
Log.e(TAG, "Error! Could not invoke JNI method - readProcFile");
} catch (IllegalAccessException e) {
Log.e(TAG, "Error! Illegal access while invoking JNI method - readProcFile");
}
return null;
I see you are creating a FileInputStream to read a /proc file. Instead I suggest you create a FileReader object. FileInputStream gets tripped up by the lack of file length for /proc files but FileReader does not.
Could you please suggest how to deal with these situations ? I understand that in the second example, it is very rare that it would happen on unix, is it ? If access rights are alright. Also the file wouldn't be even created. I don't understand why the IOException is there, either it is created or not, why do we have to bother with IOException ?
But in the first example, there will be a corrupted zombie file. Now if you tell the user to upload it again, the same thing may happen. If you can't do that, and the inputstream has no marker. You loose your data ? I really don't like how this is done in Java, I hope the new IO in Java 7 is better
Is it usual to delete it
public void inputStreamToFile(InputStream in, File file) throws SystemException {
OutputStream out;
try {
out = new FileOutputStream(file);
} catch (FileNotFoundException e) {
throw new SystemException("Temporary file created : " + file.getAbsolutePath() + " but not found to be populated", e);
}
boolean fileCorrupted = false;
int read = 0;
byte[] bytes = new byte[1024];
try {
while ((read = in.read(bytes)) != -1) {
out.write(bytes, 0, read);
}
} catch (IOException e) {
fileCorrupted = true;
logger.fatal("IO went wrong for file : " + file.getAbsolutePath(), e);
} finally {
IOUtils.closeQuietly(in);
IOUtils.closeQuietly(out);
if(fileCorrupted) {
???
}
}
}
public File createTempFile(String fileId, String ext, String root) throws SystemException {
String fileName = fileId + "." + ext;
File dir = new File(root);
if (!dir.exists()) {
if (!dir.mkdirs())
throw new SystemException("Directory " + dir.getAbsolutePath() + " already exists most probably");
}
File file = new File(dir, fileName);
boolean fileCreated = false;
boolean fileCorrupted = false;
try {
fileCreated = file.createNewFile();
} catch (IOException e) {
fileCorrupted = true;
logger.error("Temp file " + file.getAbsolutePath() + " creation fail", e);
} finally {
if (fileCreated)
return file;
else if (!fileCreated && !fileCorrupted)
throw new SystemException("File " + file.getAbsolutePath() + " already exists most probably");
else if (!fileCreated && fileCorrupted) {
}
}
}
I really don't like how this is done in Java, I hope the new IO in Java 7 is better
I'm not sure how Java is different than any other programming language/environment in the way you are using it:
a client sends some data to your over the wire
as you read it, you write it to a local file
Regardless of the language/tools/environment, it's possible for the connection to be interrupted or lost, for the client to go away, for the disk to die, or for any other error to occur. I/O errors can occur in any and all environments.
What you can do in this situation is highly dependent on the situation and the error that occured. For example, is the data structured in some way where you could ask the user to resume uploading from record 1000, for example? However, there is no single solution that fits all here.