Develop Reference

Java - Download file from URL with matching file name pattern

I want to download few files from a URL. I know the starting of the file name. But the next part would be different. Mostly a date. But it could be different for different files. From Java code, is there any way to download file with matching pattern?
If I hit the below URL in chrome, all the files are listed and I have to download the required files manually.
http://<ip_address>:<port>/MR/build/report/scan/daily/2021-12-13_120/data/
File names can b like below. It will have known file name and date. The date can be different. Either the same as in URL or some older one.
scan_report_2021_12_13_120.txt
build_report_2021_12_10_110.txt
my_reportdata_2021_11_30_110.txt
As of now, my Java code is like below. I have to pass the complete URL with exact file name to download the files. Most of the cases it would be same as the date and number in URL. So in the program I take the date part from URL and add it to my file name nd pass as the URL. But for some files it might change and for those I have to manually download.
private static void downloadFile(String remoteURLPath, String localPath) {
System.out.println("DownloadFileTest.downloadFile() Downloading from " + remoteURLPath + " to = " + localPath);
FileOutputStream fos = null;
try {
URL website = new URL(remoteURLPath);
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
fos = new FileOutputStream(localPath);
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (fos != null) {
try {
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
The argument remoteURLPath is passed like http://<ip_address>:<port>/MR/build/report/scan/daily/2021-12-13_120/data/scan_report_2021_12_13_120.txt
And localPath is passed like C:\\MyDir\\MyData\\scan_report_2021_12_13_120.txt
Similarly other files also with date as 2021_12_13_120. Other files wont get downloaded. But will create empty file in the same directory which I will delete later since size is 0.
Is there any way we can pass pattern here?
Like http://<ip_address>:<port>/MR/build/report/scan/daily/2021-12-13_120/data/scan_report_*.txt
And instead of passing complete local path, is there any way to pass only directory where the file should get downloaded with exact same name as in the remote system?
In Linux I can use wget with pattern matching. But was looking for Java way to download in all platforms.
wget -r -np -nH --cut-dirs=10 -A "scan_report*.txt" "http://<ip_address>:<port>/MR/build/report/scan/daily/2021-12-13_120/data/"

Thanks to comment from #FedericoklezCulloca. I modified my code using this answer
The solution I did is read all html page and get all href values as it had only the file names with extension. From there I had another list which I used to get the matching files and those I downloaded then using my code in the Question.
Method to get all href list from URL. may be optimisation can be done. Also I did not use any extra library.
private static List<String> getAllHREFListFromURL(String downloadURL) {
URL url;
InputStream is = null;
List<String> hrefListFromURL = new ArrayList<>();
try {
url = new URL(downloadURL);
is = url.openStream();
byte[] buffer = new byte[1024];
int bytesRead = -1;
StringBuilder page = new StringBuilder(1024);
while ((bytesRead = is.read(buffer)) != -1) {
String str = new String(buffer, 0, bytesRead);
page.append(str);
}
StringBuilder htmlPage = new StringBuilder(page);
String search_start = "href=\"";
String search_end = "\"";
while (!htmlPage.isEmpty()) {
int indexOf = htmlPage.indexOf(search_start);
if (indexOf != -1) {
String substring = htmlPage.substring(indexOf + search_start.length());
String linkName = substring.substring(0, substring.indexOf(search_end));
hrefListFromURL.add(linkName);
htmlPage = new StringBuilder(substring);
} else {
htmlPage = new StringBuilder();
}
}
} catch (MalformedURLException e1) {
e1.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
} finally {
try {
is.close();
} catch (Exception e) {
}
}
return hrefListFromURL;
}
Method to get list of files that I needed.
private static List<String> getDownloadList(List<String> allHREFListFromURL) {
List<String> filesList = getMyFilesList();
List<String> downloadList = new ArrayList<>();
for (String fileName : filesList) {
Predicate<String> fileFilter = Pattern.compile(fileName + "*").asPredicate();
List<String> collect = allHREFListFromURL.stream().filter(fileFilter).collect(Collectors.toList());
downloadList.addAll(collect);
}
return downloadList;
}
private static List<String> getMyFilesList() {
List<String> filesList = new ArrayList<>();
filesList.add("scan_report");
filesList.add("build_report");
filesList.add("my_reportdata");
return filesList;
}
The downloadList I iterate and uses my original download method to download.

Java too many open files exception

I have a problem on my code; basically I have an array containing some key:
String[] ComputerScience = { "A", "B", "C", "D" };
And so on, containing 40 entries.
My code reads 900 pdf from 40 folder corresponding to each element of ComputerScience, manipulates the extracted text and stores the output in a file named A.txt , B.txt, ecc ...
Each folder "A", "B", ecc contains 900 pdf.
After a lot of documents, an exception "Too many open files" is thrown.
I'm supposing that I am correctly closing files handler.
static boolean writeOccurencesFile(String WORDLIST,String categoria, TreeMap<String,Integer> map) {
File dizionario = new File(WORDLIST);
FileReader fileReader = null;
FileWriter fileWriter = null;
try {
File cat_out = new File("files/" + categoria + ".txt");
fileWriter = new FileWriter(cat_out, true);
} catch (IOException e) {
e.printStackTrace();
}
try {
fileReader = new FileReader(dizionario);
} catch (FileNotFoundException e) { }
try {
BufferedReader bufferedReader = new BufferedReader(fileReader);
if (dizionario.exists()) {
StringBuffer stringBuffer = new StringBuffer();
String parola;
StringBuffer line = new StringBuffer();
int contatore_index_parola = 1;
while ((parola = bufferedReader.readLine()) != null) {
if (map.containsKey(parola) && !parola.isEmpty()) {
line.append(contatore_index_parola + ":" + map.get(parola).intValue() + " ");
map.remove(parola);
}
contatore_index_parola++;
}
if (! line.toString().isEmpty()) {
fileWriter.append(getCategoryID(categoria) + " " + line + "\n"); // print riga completa documento N x1:y x2:a ...
}
} else { System.err.println("Dictionary file not found."); }
bufferedReader.close();
fileReader.close();
fileWriter.close();
} catch (IOException e) { return false;}
catch (NullPointerException ex ) { return false;}
finally {
try {
fileReader.close();
fileWriter.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return true;
}
But the error still comes. ( it is thrown at:)
try {
File cat_out = new File("files/" + categoria + ".txt");
fileWriter = new FileWriter(cat_out, true);
} catch (IOException e) {
e.printStackTrace();
}
Thank you.
EDIT: SOLVED
I found the solution, there was, in the main function in which writeOccurencesFile is called, another function that create a RandomAccessFile and doesn't close it.
The debugger sais that Exception has thrown in writeOccurencesFile but using Java Leak Detector i found out that the pdf were already opened and not close after parsing to pure text.
Thank you!

Try using this utility specifically designed for the purpose.
This Java agent is a utility that keeps track of where/when/who opened files in your JVM. You can have the agent trace these operations to find out about the access pattern or handle leaks, and dump the list of currently open files and where/when/who opened them.
When the exception occurs, this agent will dump the list, allowing you to find out where a large number of file descriptors are in use.

i have tried using try-with resources; but the problem remains.
Also running in system macos built-in console print out a FileNotFound exception at the line of FileWriter fileWriter = ...
static boolean writeOccurencesFile(String WORDLIST,String categoria, TreeMap<String,Integer> map) {
File dizionario = new File(WORDLIST);
try (FileWriter fileWriter = new FileWriter( "files/" + categoria + ".txt" , true)) {
try (FileReader fileReader = new FileReader(dizionario)) {
try (BufferedReader bufferedReader = new BufferedReader(fileReader)) {
if (dizionario.exists()) {
StringBuffer stringBuffer = new StringBuffer();
String parola;
StringBuffer line = new StringBuffer();
int contatore_index_parola = 1;
while ((parola = bufferedReader.readLine()) != null) {
if (map.containsKey(parola) && !parola.isEmpty()) {
line.append(contatore_index_parola + ":" + map.get(parola).intValue() + " ");
map.remove(parola);
}
contatore_index_parola++;
}
if (!line.toString().isEmpty()) {
fileWriter.append(getCategoryID(categoria) + " " + line + "\n"); // print riga completa documento N x1:y x2:a ...
}
} else {
System.err.println("Dictionary file not found.");
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
return true;
}
This is the code that i am using now, although the bad managing of Exception, why the files seem to be not closed?
Now i am making a test with File Leak Detector

Maybe your code raises another exception that you are not handling. Try add catch (Exception e) before finally block
You also can move BufferedReader declaration out the try and close it in finally

Importing CSV into MySQL through JAVA

So I'm trying to import a CSV file into my MySQL database through my Java program. The program imports everything that's in the file, like it's suppose to, but the first row, it send to the end of the table, and the program see it's there, but if I search for that nr, it says it doesn't exists. And if I go directly to the database table and edit the nr(if the nr is 137, and I edit and write 137 again) the program recognize that nr, and if I search for it, it will find, and the database table organizes itself and sends that entry where is suppose to be.
I just don't see any logic in this. I someone could help me out, I'd appreciated.
LOAD DATA INFILE 'C:\\Users\\carla.DESKTOP-9364K9K\\Desktop\\Alunos_1.csv'
INTO TABLE utentes character set utf8
FIELDS TERMINATED BY ','
(NrProcesso, Nome, #Nome_Resumido, Ano, Turma, #Subsidio, #Nome_EE, #NIF, #email, #Obs)
SET
Subsidio = IF(#Subsidio='','Nenhum',#Subsidio),
Nome_Resumido = IF(#Nome_Resumido='',NULL,#Nome_Resumido),
Nome_EE = IF(#Nome_EE='',NULL,#Nome_EE),
NIF = IF(#NIF = '', NULL,#NIF),
email = IF(#email='',NULL,#email),
Obs = IF(#Obs='',NULL,#Obs);
Thanks in advance.

You have do do something to check cell/column value and form a sql to inject in MySQL.
public List<Object> getRecordingsListFromCsv(String csvFileLocation, String mp3FileLocation, String mp3FileLocation2, String saveFileLocation, ChannelSftp sftp) {
Map<String, File> recordingsFilesMap = null;
BufferedReader br = null;
List<String> errorFilesList = new ArrayList<>();
List<Object> tempList = new LinkedList<>();
try {
csvRows = 0;
recordingsFilesMap = new LinkedHashMap<>();
br = new BufferedReader(new FileReader(csvFileLocation));
String line = br.readLine();
scriptLog.info("\n" + csvFileLocation + " loaded. Parsing File...");
while ((line = br.readLine()) != null) {
String[] csvArray = parseCsvLineToArray(line);
// System.out.println(Arrays.asList(csvArray) + "\n\n");
if (csvArray[0].trim().isEmpty()) {
continue;
}
/* Do your stuff here */
csvRows++;
}
} catch (FileNotFoundException e) {
scriptLog.error("\n---ERROR---\n FILE NOT FOUND: " + csvFileLocation);
String errorStr = "Type=" + e.toString();
errorStr += "StackTrace=" + Arrays.toString(e.getStackTrace());
scriptLog.error(errorStr);
} catch (IOException e) {
String errorStr = "Type=" + e.toString();
errorStr += "StackTrace=" + Arrays.toString(e.getStackTrace());
scriptLog.error(errorStr);
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
System.out.println(e.toString());
}
}
}
Hope it will help you at some extent!!

JPEG encoder - set quality from command line

I'm building a JPEG image encoder. As it stands, in order to encode an image the user enters the name of the file they wish to encode and name of the file to be created as a result.
I'd like the user to be able to set the quality of the encoding in the command line. I tried renaming the second argument (100) in new JpegEncoder(image, 100, new FileOutputStream(args[1]));
encoder.Compress(); to args[2] but that didn't work.
public class JPGencoder {
public static void main ( String[] args ) {
String[] names = ImageIO.getWriterFormatNames();
BufferedImage image = null;
JpegEncoder encoder = null;
try {
image = ImageIO.read( new File( args[0] ) );
System.err.println("Process image " + args[0]);
System.err.println(image.toString());
} catch (Exception e) {
System.err.println("Problems with image " + args[0]);
}
try {
encoder = new JpegEncoder(image, 100, new FileOutputStream(args[1]));
encoder.Compress();
} catch (Exception e) {
System.out.println("well that didn't work");
}
}
}

Based on this definition of JpegEncoder the second argument to the JpegEncode constructor is an int.
The type of args[2] is a String so presumably by "did not work" you mean "did not compile". To convert args[2] to an int:
Integer.parseInt(args[2]);
This will throw a NumberFormatException if args[2] is not a valid int.

It is not difficult to set the JPG compression/quality using ImageIO. Here are some snippets that might get you started.
private ImageWriteParam imageWriterParams;
private ImageWriter imageWriter;
File out = new File("some.jpg");
// ...
Iterator it = ImageIO.getImageWritersBySuffix("jpg");
// presume every iterator has precisely 1 writer
imageWriter = (ImageWriter)it.next();
imageWriterParams = imageWriter.getDefaultWriteParam();
if ( imageWriterParams.canWriteCompressed() ) {
try {
imageWriterParams.setCompressionMode( ImageWriteParam.MODE_EXPLICIT );
} catch(Exception e) {
e.printStackTrace();
}
} else {
logger.log(Level.WARNING, "ImageWriter cannot compress!");
}
imageWriterParams.setCompressionQuality(qualF);
FileImageOutputStream fios = new FileImageOutputStream(out);
imageWriter.setOutput(fios);
imageWriter.write(
null,
new IIOImage(image,null,null),
imageWriterParams );
fios.flush();
fios.close();

How to clear Mediastore before setting ringtone

When I set a ringtone from my app works once, but when running the code again, it tries to create a duplicate entry in the media store, which creates problems. Without creating seperate unique file names for every sound file, I want to fix this problem.
I found this solution posted in an answer here: setting audio file as Ringtone
and am trying to use that to fix mine.
When I try it in my code below, I get two errors. One is an SQLiteException and the other is a RuntimeException which is caused by the squlite error, which is after the Java code.
String TAG = "CFFS";
File dir = new File(Environment.getExternalStorageDirectory()+ "/ringtones"); // Set base DIR where new ringtone will live
dir.mkdirs(); // create if directors don't exist
File outputFile = new File(dir, "College Fight Song.mp3"); // Define out new output file
Uri inURI = null;
try {
inURI = Uri.parse(getIntent().getStringExtra("com.carboni.fightsongs.FILE_RES_ID"));
} catch (Exception e) {
e.printStackTrace();
Log.e(TAG, "Could not get URI " + e);
}
// If we didn't parse a good URI then don't execute the code below
if (inURI != null) {
InputStream in = null;
// Get the input stream
try { in = new BufferedInputStream(this.getContentResolver().openInputStream(inURI)); }
catch (Exception e) { Log.e(TAG, "Exception getting input stream " + e); }
// Get the output stream
OutputStream out = null;
try { out = new FileOutputStream(outputFile); }
catch (Exception e) { Log.e(TAG, "Exception getting output stream " + e); }
// Again, if we don't have 2 good handles then don't try to read/write them
if ((in != null) && (out != null)) {
byte[] buf = new byte[1024]; // Define our buffer size
int bytesRead = 0;
while (bytesRead >= 0) {
try {
bytesRead = in.read(buf, 0, buf.length); // Read max of 1024 bytes
if (bytesRead > 0)
out.write(buf); // Write buffer to new file if we got a good read
} catch (Exception e) {
Log.e(TAG,"Exception reading " + e);
e.printStackTrace();
}
}
}
// Close out handles and proceed
try {
in.close();
out.close();
}
catch (Exception e) { Log.e(TAG, "Exception closing streams " + e); }
ContentValues v = new ContentValues();
v.put(MediaStore.MediaColumns.DATA, outputFile.getAbsolutePath());
v.put(MediaStore.MediaColumns.TITLE, "College Football Fight Song");
v.put(MediaStore.MediaColumns.SIZE, outputFile.length());
v.put(MediaStore.MediaColumns.MIME_TYPE, "audio/mp3");
v.put(MediaStore.Audio.Media.IS_RINGTONE, true);
Uri pURI = MediaStore.Audio.Media.getContentUriForPath(outputFile.getAbsolutePath());
// remove entry every time so we don't get duplicate entries and have a problem setting a 2nd time
getContentResolver().delete(pURI, MediaStore.MediaColumns.DATA + "\"" + outputFile.getAbsolutePath() + "\"", null);
Uri nURI = this.getContentResolver().insert(pURI, v);
Log.i(TAG, "Setting ringtone URI to " + nURI);
// Set ringtone
RingtoneManager.setActualDefaultRingtoneUri(this, RingtoneManager.TYPE_RINGTONE, nURI);
Toast.makeText(this, "Ringtone set", Toast.LENGTH_LONG).show();
ERROR:
09-03 14:16:08.343: ERROR/DatabaseUtils(11968): android.database.sqlite.SQLiteException: near ""/mnt/sdcard/ringtones/College Fight Song.mp3"": syntax error: , while compiling: DELETE FROM audio_meta WHERE _data"/mnt/sdcard/ringtones/College Fight Song.mp3"

It looks like the error is this line:
getContentResolver().delete(pURI, MediaStore.MediaColumns.DATA + "\"" + outputFile.getAbsolutePath() + "=\"", null);