Develop Reference

Decrease the number of calls

I have arraylist of filenames (java) and I want to delete these files using rm
for(String check: checks){
here I am removing each file using rm
}
but it is time consuming can I do batching using xargs or something else which can help to delete files faster.

Don’t use rm. Use Java.
As others have pointed out, spawning a process is much slower than doing it in your program. Also, it’s just bad design to use a system command for something that can be done in code.
And finally, by using Files.delete, you gain platform independence. Your code is now write once, run anywhere!
Here’s what your loop would look like:
for (String check : checks) {
Files.delete(Path.of(check));
}
This is essentially the same as what rm would do anyway.

How to run multiple CMDs with quotes in one terminal in java

I need to execute in java 2 commands in one terminal,
each command contains a path with spaces.
For example:
CMD 1: "C:\user\with space\bat1.bat"
CMD 2: "C:\user\with space\bat2.bat"
The purpose is running CMD 2 only if CMD 1 ran successfully, in the same terminal.
I've tried to use && :
Runtime.getRuntime().exec("\"C:\\user\\with space\\bat1.bat\" && \"C:\\user\\with space\\bat2.bat\"");
But I got the following error:
'\"C:\user\with space\bat1.bat\"' is not recognized as an internal or external command,
operable program or batch file.
How can I run them both and make it recognize each command separately?

You've confused a terminal with execute. These are not the same thing. Your OS can run applications. The terminal isn't "the OS". It's an application. When you type stuff in there, you are not sending it 'to the system', you're just typing in that application. No different from me typing in this very text box in a web browser. I type a letter and it appears. I hit enter and stuff happens.
The shell does a whole boatload of 'parsing' on what you type and then uses the equivalent of Runtime.getRuntime().exec in the language it is written in.
There are a lot of things that /bin/bash, /bin/zsh, /bin/fish, windows's cmd.exe, and all those other shells do. For some of these, it really IS the OS that does it... depending on which OS you're using. It's best to rely on absolutely none of these:
Scour the PATH when you type a non-absolute-path executable to know which one to run.2
Append .exe, .bat, .com, or similar.
A whole bunch of well known commands like cd, ls, mkdir, etcetera.
&&, || and other ways to chain things.
filename substitutions, such as whatever.exe *.txt (the *.txt is 'substitution').1, same goes for ?.
variable substitution, such as /bin/foo $fn.
Redirect pipes such as 2>/dev/null or <myfile.txt.
splitting the command up by separating on spaces and considering the first thing the path of a command and all other things, each individually, a single argument... but then looking at quotes, using those to figure out certain spaces are not separators between parameters, and removing the actual quote characters of course.
Replace ~ with the user's home dir.
And much, much more.
You need to do all of it yourself. That's why you should never use Runtime.getRuntime().exec and always use ProcessBuilder instead. You can set up redirects with that, do the argument splitting on your own. Always specify an absolute path. Don't use * or ? for anything, etcetera.
You can't write, from within java, a magical way to make 2 batch files not open up 2 black boxes. Instead, create a new batch file that runs both batch files (I think you have to do CALL bat1.bat to avoid 3 black boxes showing up), and then run that, and you can't just run the batch file, you run cmd.exe and tell IT to run the batch file. If it's very very simple, you may be able to tell CMD to do it in one go.
Path workingDir = Paths.get(".");
List<String> batOfBatsContent = List.of(
"#ECHO OFF",
"CALL bat1.bat || GOTO :fail",
"CALL bat2.bat",
":fail");
Path megaBat = workingDir.resolve("doEverything.bat");
Files.write(megaBat, batOfBats.stream().collect(Collectors.joining("\r\n"));
ProcessBuilder pb = new ProcessBuilder();
pb.command(System.getenv("COMSPEC"), "/c", batOfBats.getAbsolutePath());
pb.start();
Or possibly:
Path workingDir = Paths.get(".");
ProcessBuilder pb = new ProcessBuilder();
pb.command(System.getenv("COMSPEC"), "/c",
"\"" +
workingDir.resolve("bat1.bat").getAbsolutePath() +
"\"&&\"" +
workingDir.resolve("bat2.bat").getAbsolutePath()
+ "\"");
pb.start();
[1] Actually, windows does do this, probably. Posix OSes (everything that isn't windows, at this point in time) definitely don't.
[2] Java tries to do this, sort of, but it's not particularly reliably. If you don't have to rely on it, don't. It's security-wise dubious in any case to do so.

Is it possible to prepend data to an file without rewriting?

I deal with very large binary files ( several GB to multiple TB per file ). These files exist in a legacy format and upgrading requires writing a header to the FRONT of the file. I can create a new file and rewrite the data but sometimes this can take a long time. I'm wondering if there is any faster way to accomplish this upgrade. The platform is limited to Linux and I'm willing to use low-level functions (ASM, C, C++) / file system tricks to make this happen. The primimary library is Java and JNI is completely acceptable.

There's no general way to do this natively.
Maybe some file-systems provide some functions to do this (cannot give any hint about this), but your code will then be file-system dependent.
A solution could be that of simulating a file-system: you could store your data on a set of several files, and then provide some functions to open, read and write data as if it was a single file.

Sounds crazy, but you can store the file data in reverse order, if it is possible to change function that reads data from file. In that case you can append data (in reverse order) at the end of the file. It is just a general idea, so I can't recommend anything particular.
The code for reversing of current file can looks like this:
std::string records;
ofstream out;
std::copy( records.rbegin(), records.rend(), std::ostream_iterator<string>(out));

It depends on what you mean by "filesystem tricks". If you're willing to get down-and-dirty with the filesystem's on-disk format, and the size of the header you want to add is a multiple of the filesystem block size, then you could write a program to directly manipulate the filesystem's on-disk structures (with the filesystem unmounted).
This enterprise is about as hairy as it sounds though - it'd likely only be worth it if you had hundreds of these giant files to process.

I would just use the standard Linux tools to do it.
Writting another application to do it seems like it would be sub-optimal.
cat headerFile oldFile > tmpFile && mv tmpFile oldFile

I know this is an old question, but I hope this helps someone in the future. Similar to simulating a filesystem, you could simply use a named pipe:
mkfifo /path/to/file_to_be_read
{ echo "HEADER"; cat /path/to/source_file; } > /path/to/file_to_be_read
Then, you run your legacy program against /path/to/file_to_be_read, and the input would be:
HEADER
contents of /path/to/source_file
...
This will work as long as the program reads the file sequentially and doesn't do mmap() or rewind() past the buffer.

MATLAB - Delete elements of binary files without loading entire file

This may be a stupid question, but Google and MATLAB documentation have failed me. I have a rather large binary file (>10 GB) that I need to open and delete the last forty million bytes or so. Is there a way to do this without reading the entire file to memory in chunks and printing it out to a new file? It took 6 hours to generate the file, so I'm cringing at the thought of re-reading the whole thing.
EDIT:
The file is 14,440,000,000 bytes in size. I need to chop it to 14,400,000,000.

There is no ftruncate() in Matlab, but you've got access to the full Java standard library in the JVM embedded in Matlab, and can use java.io.RandomAccessFile or the Java NIO classes to truncate a file.
Here's a Matlab function that calls to Java to lop the last n bytes off a file. Should have minimal I/O cost.
function remove_last_n_bytes_from_file(file, n)
jFile = java.io.RandomAccessFile(file, 'rw');
currentLength = jFile.length();
wantLength = currentLength - n;
fprintf('Truncating file %s: Resizing to %d to remove %d bytes\n', file, wantLength, n);
jFile.setLength(wantLength);
jFile.close();
You could also do it as a one-liner.
java.io.RandomAccessFile('/path/to/my/file.bin', 'rw').setLength(n);

I found Perl is much quicker to do this than MATLAB.
Here are two examples from Perl Cookbook:
truncate(HANDLE, $length)
or die "Couldn't truncate: $!\n";
truncate("/tmp/$$.pid", $length)
or die "Couldn't truncate: $!\n";
You can run Perl script from MATLAB with PERL function.

Since you don't want to read the file into MATLAB (understandably), you are dealing with system level commands. MATLAB has a facility to call system commands using the "system" command
system
So now your problem is reduced to finding the shell command in your OS that will do it for you. Or you can write a program using truncate() (unix -- KennyTM) or SetEndOfFile (windows)

I don't know if MATLAB supports this, but see ftruncate() and truncate().

Java exec on Unix

I have the Java code below running on Unix (both AIX and Linux), but it doesn't work. If I run this code the file q1.01 is not compressed, and I don't get any exceptions thrown (The file q1.01 exists, and I expect to find the file q1.01.Z after the command runs.) At the command prompt if I type "which compress" it reports back with "/usr/bin/compress". If I type the command "/usr/bin/compress q1.01" at the Unix prompt it works fine. Any ideas on what might be wrong?
String cmd = "/usr/bin/compress q1.01";
Runtime.getRuntime().exec(cmd);

[Later edit: the problem was in the initial description; the OP was passing a wildcard and not q.01. So my answer below is wrong, except for the part in bold. I'm leaving it so the comments after it will make sense.]
It's trying to run /usr/bin/compress as the program name without arguments.
There are many forms of the Runtime.exec() method. You're using the .exec(String) version, which just takes the executable. Instead, you need to use the .exec(String[]) array version, which takes the executable in String[0] and the parameters in String[1..].
.exec() wants a String array for passing arguments.
Try
String[] cmd = new String[] { "/usr/bin/compress", "q1.01" };
Runtime.getRuntime().exec(cmd);
Note that .exec does not call the local command shell. That means we have to do, among other things, wildcard expansion and even some argument parsing before calling .exec(). This is why you can't just pass it your full command line.

There were a couple of problems. One was that I had tried using wildcards, and since the shell isn't invoked they weren't being expanded. The other problem was that I had created very small test files like this: "echo 'abc' >q1.01". This file was so small that compress couldn't compress it any further and so left it alone. (Stupidly, I think when I typed in the command at the shell I used a different filename, which did compress.)
Thanks everyone for the answers. It did help!

You probably need to use an absolute path to the file. Capture the output though, to see what the problem is - see this page for info on how to do that.

This site may be able to provide some clues.

If the process input stream is null, I suspect that Java wasn't even able to spawn the subprocess. What does Process#exitValue() return?
I'd recommend using strace to see what actually happens on the system-call level. The actual exec() arguments and return code would be especially interesting to see.