Develop Reference

Java - Download file from URL with matching file name pattern

I want to download few files from a URL. I know the starting of the file name. But the next part would be different. Mostly a date. But it could be different for different files. From Java code, is there any way to download file with matching pattern?
If I hit the below URL in chrome, all the files are listed and I have to download the required files manually.
http://<ip_address>:<port>/MR/build/report/scan/daily/2021-12-13_120/data/
File names can b like below. It will have known file name and date. The date can be different. Either the same as in URL or some older one.
scan_report_2021_12_13_120.txt
build_report_2021_12_10_110.txt
my_reportdata_2021_11_30_110.txt
As of now, my Java code is like below. I have to pass the complete URL with exact file name to download the files. Most of the cases it would be same as the date and number in URL. So in the program I take the date part from URL and add it to my file name nd pass as the URL. But for some files it might change and for those I have to manually download.
private static void downloadFile(String remoteURLPath, String localPath) {
System.out.println("DownloadFileTest.downloadFile() Downloading from " + remoteURLPath + " to = " + localPath);
FileOutputStream fos = null;
try {
URL website = new URL(remoteURLPath);
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
fos = new FileOutputStream(localPath);
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (fos != null) {
try {
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
The argument remoteURLPath is passed like http://<ip_address>:<port>/MR/build/report/scan/daily/2021-12-13_120/data/scan_report_2021_12_13_120.txt
And localPath is passed like C:\\MyDir\\MyData\\scan_report_2021_12_13_120.txt
Similarly other files also with date as 2021_12_13_120. Other files wont get downloaded. But will create empty file in the same directory which I will delete later since size is 0.
Is there any way we can pass pattern here?
Like http://<ip_address>:<port>/MR/build/report/scan/daily/2021-12-13_120/data/scan_report_*.txt
And instead of passing complete local path, is there any way to pass only directory where the file should get downloaded with exact same name as in the remote system?
In Linux I can use wget with pattern matching. But was looking for Java way to download in all platforms.
wget -r -np -nH --cut-dirs=10 -A "scan_report*.txt" "http://<ip_address>:<port>/MR/build/report/scan/daily/2021-12-13_120/data/"

Thanks to comment from #FedericoklezCulloca. I modified my code using this answer
The solution I did is read all html page and get all href values as it had only the file names with extension. From there I had another list which I used to get the matching files and those I downloaded then using my code in the Question.
Method to get all href list from URL. may be optimisation can be done. Also I did not use any extra library.
private static List<String> getAllHREFListFromURL(String downloadURL) {
URL url;
InputStream is = null;
List<String> hrefListFromURL = new ArrayList<>();
try {
url = new URL(downloadURL);
is = url.openStream();
byte[] buffer = new byte[1024];
int bytesRead = -1;
StringBuilder page = new StringBuilder(1024);
while ((bytesRead = is.read(buffer)) != -1) {
String str = new String(buffer, 0, bytesRead);
page.append(str);
}
StringBuilder htmlPage = new StringBuilder(page);
String search_start = "href=\"";
String search_end = "\"";
while (!htmlPage.isEmpty()) {
int indexOf = htmlPage.indexOf(search_start);
if (indexOf != -1) {
String substring = htmlPage.substring(indexOf + search_start.length());
String linkName = substring.substring(0, substring.indexOf(search_end));
hrefListFromURL.add(linkName);
htmlPage = new StringBuilder(substring);
} else {
htmlPage = new StringBuilder();
}
}
} catch (MalformedURLException e1) {
e1.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
} finally {
try {
is.close();
} catch (Exception e) {
}
}
return hrefListFromURL;
}
Method to get list of files that I needed.
private static List<String> getDownloadList(List<String> allHREFListFromURL) {
List<String> filesList = getMyFilesList();
List<String> downloadList = new ArrayList<>();
for (String fileName : filesList) {
Predicate<String> fileFilter = Pattern.compile(fileName + "*").asPredicate();
List<String> collect = allHREFListFromURL.stream().filter(fileFilter).collect(Collectors.toList());
downloadList.addAll(collect);
}
return downloadList;
}
private static List<String> getMyFilesList() {
List<String> filesList = new ArrayList<>();
filesList.add("scan_report");
filesList.add("build_report");
filesList.add("my_reportdata");
return filesList;
}
The downloadList I iterate and uses my original download method to download.

Getting current jar file's name

I have this code which on the dev-environment return the information.
But when I run from the jar the code doesn't follow how it should.
The name of the jar is hardcoded and would like to get it's name, because versions vary.
private static String getManifestUrlForClass(Class<?> cl) throws URISyntaxException, IOException {
URL url = cl.getResource(cl.getSimpleName() + ".class");
String s = url.toString();
System.out.println("URL Path: " + url.getPath());
System.out.println("URL File: " + url.getFile());
String path = MYCLASS.class.getProtectionDomain().getCodeSource().getLocation().getPath();
String revisionNumber = "";
String decodedPath = "";
JarFile jarfile = null;
try {
decodedPath = URLDecoder.decode(path, "UTF-8").replace("classes", "");
try {
jarfile = new JarFile(decodedPath + "MYJAR-ver.si.on.jar");
} catch (IOException e1) {
System.out.println("or Path to file cannot decode...");
e1.printStackTrace();
}
Manifest manifestFromJar = jarfile.getManifest(); //
System.out.println("Manifest from " + jarfile.getName().toString() + " = "
+ manifestFromJar.getMainAttributes().getValue("Revision-Number").toString());
revisionNumber = manifestFromJar.getMainAttributes().getValue("Revision-Number").toString();
} catch (IOException e) {
System.out.println(url.getFile().toString() + "is not jar");// TODO Auto-generated catch block
System.out.println("or Path to file cannot decode...");
e.printStackTrace();
}
return revisionNumber;
}
MYJAR will always be the same but the |ver.si.on| will most likely vary and hardcoding the name isn't a best practice.
What I want to do?
1. Get the MYJAR-ver.si.on.jar's location no matter where it is located
2. Use the location to access it's Manifest
3. Use the Manifest to extract revision number
4. Show the revision number in the ui
I'm new yet to java and don't understand it pretty well. I've read something about using "rsrc:" to get to the jar, or something similar to this https://stackoverflow.com/a/40680501/6756124 .

How to extract data from Facebook pages using FQL in Java

I want to extract data such as name of page, likes,talking_about_count and page information from any page using FQL in Java. I have written code like below:
public static void main(String[] args) {
// TODO Auto-generated method stub
String STR_FACEBOOK_FQL_URL = "https://api.facebook.com/method/fql.query?query=";
String STR_FACEBOOK_RETURN_FORMAT = "&format=json";
String STR_FACEBOOK_ACCESS_TOKEN = "&access_token=";
String FQL_GET_QUERY= "SELECT name,likes,about FROM page WHERE id={page-id}";
String encodedQuery;
try
{
encodedQuery = URLEncoder.encode(FQL_GET_QUERY, "UTF-8");
String faceBookURL = STR_FACEBOOK_FQL_URL + encodedQuery +
STR_FACEBOOK_RETURN_FORMAT
+ STR_FACEBOOK_ACCESS_TOKEN ;
URL url = new URL(faceBookURL);
InputStream is = url.openStream();
JSONTokener tok = new JSONTokener(is);
JSONObject result = new JSONObject(tok);
System.out.println(result.toString());
is.close();
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
But this is shownig error like:
{"request_args":[{"value":"fql.query","key":"method"},{"value":"SELECT likes,name,about FROM page WHERE uid=me()","key":"query"},{"value":"json","key":"format"},{"value":"id|secret","key":"access_token"}],"error_code":602,"error_msg":"likes is not a member of the page table."}
Can anybody tell me, what is the wrong in this code?

First of all, please don't use the deprecated endpoint:
String STR_FACEBOOK_FQL_URL = "https://graph.facebook.com/fql?q=";
is the way to go now. You're using non-existing fields for the page table (see https://developers.facebook.com/docs/reference/fql/page/). likes needs to be replaced by fan_count:
String FQL_GET_QUERY= "SELECT name,fan_count,about FROM page WHERE id={page-id}";

Input formatted date and time in Swing to get nodes using GET REST

I am required to input start time and end time and in swing java that I will further send to a URL to get some selected nodes created in this time using GET REST call.
URL is:
http://wisekar.iitd.ernet.in/active/api_resources.php/method/mynode
?key=YOUR_API_KEY_HERE
&startTime=START_TIME[Optional]
&endTime=END_TIME[Optional]en
The website will take take the input (time stamps) as they are given in the image.
Screenshot Of my window
Now my code is here:
class Algorithm extends JFrame implements ActionListener {
private static String ENDPOINT =
"http://wisekar.iitd.ernet.in/active/api_"
+ "resources.php/method/mynode.json?key=api_key";
Algorithm() {
// label1 = new JLabel();.....
panel = new JPanel(new GridLayout(5, 2));
//adding in panel label1,text1 ...
add(panel, BorderLayout.CENTER);
SUBMIT.addActionListener(this);
setTitle("Optimal Travel Route");
}
public void actionPerformed(ActionEvent ae) {
try {
//String value1 = text1.getText();..
URL url = new URL(ENDPOINT + "&datasetId=" + value3
+ "&startTime=" + value1 + "&endTime=" + value2);
System.out.println(url);
HttpURLConnection httpCon;
httpCon = (HttpURLConnection) url.openConnection();
httpCon.setDoInput(true);
System.out.println(httpCon.getResponseCode());
System.out.println(httpCon.getResponseMessage());
BufferedReader in = new BufferedReader(new InputStreamReader(
httpCon.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
System.out.println(inputLine);
}
in.close();
} catch (IOException ex) {
Logger.getLogger(Algorithm.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
class AlgorithmDemo {
public static void main(String arg[]) {
try {
Algorithm frame = new Algorithm();
frame.setSize(450, 200);
frame.setVisible(true);
} catch (Exception e) {
JOptionPane.showMessageDialog(null, e.getMessage());
}
}
}
I have tried everything; I have commented it. Using this code, I am getting the nodes when I am typing the generated URL in the browser, but it is not giving the result on the console when I am printing. What is wrong in my code? Can someone tell me how should date and time should be passed to the GET. Please help. When I only input the data set ID, it gives me all the nodes of that data set ID. According to me, there is some problem in passing the time stamps.

Unfortunately when I ran a quick test I got an unauthorized response.
But it looks like the issue is the string is not url safe.
Yo need to ensure your spaces are converted to %20.
This will also be why it works using and browser as the browser address bar will do this for you behind the scenes.
If you use:
String urlSafeValue1 = URLEncoder.encode(value1, "UTF-8");
String urlSafeValue2 = URLEncoder.encode(value2, "UTF-8");
String urlSafeValue3 = URLEncoder.encode(value3, "UTF-8");
The arguments will be made url safe.

try this my friend ...
String httpURL = ENDPOINT + URLEncoder.encode("&datasetId=" + value3
+ "&startTime=" + value1 + "&endTime=" + value2, "UTF-8");
URLConnection urlConnection = new URL(httpURL).openConnection();
urlConnection.connect();

If your time stamps are on the format in the screenshot, they contain spaces which must be escaped in URLs. Try
value1 = value1.replaceAll(" ", "+");
and similarly for value2 before constructing the URL.

How to deal with the URISyntaxException

I got this error message :
java.net.URISyntaxException: Illegal character in query at index 31: http://finance.yahoo.com/q/h?s=^IXIC
My_Url = http://finance.yahoo.com/q/h?s=^IXIC
When I copied it into a browser address field, it showed the correct page, it's a valid URL, but I can't parse it with this: new URI(My_Url)
I tried : My_Url=My_Url.replace("^","\\^"), but
It won't be the url I need
It doesn't work either
How to handle this ?
Frank

You need to encode the URI to replace illegal characters with legal encoded characters. If you first make a URL (so you don't have to do the parsing yourself) and then make a URI using the five-argument constructor, then the constructor will do the encoding for you.
import java.net.*;
public class Test {
public static void main(String[] args) {
String myURL = "http://finance.yahoo.com/q/h?s=^IXIC";
try {
URL url = new URL(myURL);
String nullFragment = null;
URI uri = new URI(url.getProtocol(), url.getHost(), url.getPath(), url.getQuery(), nullFragment);
System.out.println("URI " + uri.toString() + " is OK");
} catch (MalformedURLException e) {
System.out.println("URL " + myURL + " is a malformed URL");
} catch (URISyntaxException e) {
System.out.println("URI " + myURL + " is a malformed URL");
}
}
}

Use % encoding for the ^ character, viz. http://finance.yahoo.com/q/h?s=%5EIXIC

You have to encode your parameters.
Something like this will do:
import java.net.*;
import java.io.*;
public class EncodeParameter {
public static void main( String [] args ) throws URISyntaxException ,
UnsupportedEncodingException {
String myQuery = "^IXIC";
URI uri = new URI( String.format(
"http://finance.yahoo.com/q/h?s=%s",
URLEncoder.encode( myQuery , "UTF8" ) ) );
System.out.println( uri );
}
}
http://java.sun.com/javase/6/docs/api/java/net/URLEncoder.html

Rather than encoding the URL beforehand you can do the following
String link = "http://example.com";
URL url = null;
URI uri = null;
try {
url = new URL(link);
} catch(MalformedURLException e) {
e.printStackTrace();
}
try{
uri = new URI(url.toString())
} catch(URISyntaxException e {
try {
uri = new URI(url.getProtocol(), url.getUserInfo(), url.getHost(),
url.getPort(), url.getPath(), url.getQuery(),
url.getRef());
} catch(URISyntaxException e1 {
e1.printStackTrace();
}
}
try {
url = uri.toURL()
} catch(MalfomedURLException e) {
e.printStackTrace();
}
String encodedLink = url.toString();

A general solution requires parsing the URL into a RFC 2396 compliant URI (note that this is an old version of the URI standard, which java.net.URI uses).
I have written a Java URL parsing library that makes this possible: galimatias. With this library, you can achieve your desired behaviour with this code:
String urlString = //...
URLParsingSettings settings = URLParsingSettings.create()
.withStandard(URLParsingSettings.Standard.RFC_2396);
URL url = URL.parse(settings, urlString);
Note that galimatias is in a very early stage and some features are experimental, but it is already quite solid for this use case.

A space is encoded to %20 in URLs, and to + in forms submitted data (content type application/x-www-form-urlencoded). You need the former.
Using Guava:
dependencies {
compile 'com.google.guava:guava:28.1-jre'
}
You can use UrlEscapers:
String encodedString = UrlEscapers.urlFragmentEscaper().escape(inputString);
Don't use String.replace, this would only encode the space. Use a library instead.

Coudn't imagine nothing better for
http://server.ru:8080/template/get?type=mail&format=html&key=ecm_task_assignment&label=Согласовать с контрагентом&descr=Описание&objectid=2231
that:
public static boolean checkForExternal(String str) {
int length = str.length();
for (int i = 0; i < length; i++) {
if (str.charAt(i) > 0x7F) {
return true;
}
}
return false;
}
private static final Pattern COLON = Pattern.compile("%3A", Pattern.LITERAL);
private static final Pattern SLASH = Pattern.compile("%2F", Pattern.LITERAL);
private static final Pattern QUEST_MARK = Pattern.compile("%3F", Pattern.LITERAL);
private static final Pattern EQUAL = Pattern.compile("%3D", Pattern.LITERAL);
private static final Pattern AMP = Pattern.compile("%26", Pattern.LITERAL);
public static String encodeUrl(String url) {
if (checkForExternal(url)) {
try {
String value = URLEncoder.encode(url, "UTF-8");
value = COLON.matcher(value).replaceAll(":");
value = SLASH.matcher(value).replaceAll("/");
value = QUEST_MARK.matcher(value).replaceAll("?");
value = EQUAL.matcher(value).replaceAll("=");
return AMP.matcher(value).replaceAll("&");
} catch (UnsupportedEncodingException e) {
throw LOGGER.getIllegalStateException(e);
}
} else {
return url;
}
}

I had this exception in the case of a test for checking some actual accessed URLs by users.
And the URLs are sometime contains an illegal-character and hang by this error.
So I make a function to encode only the characters in the URL string like this.
String encodeIllegalChar(String uriStr,String enc)
throws URISyntaxException,UnsupportedEncodingException {
String _uriStr = uriStr;
int retryCount = 17;
while(true){
try{
new URI(_uriStr);
break;
}catch(URISyntaxException e){
String reason = e.getReason();
if(reason == null ||
!(
reason.contains("in path") ||
reason.contains("in query") ||
reason.contains("in fragment")
)
){
throw e;
}
if(0 > retryCount--){
throw e;
}
String input = e.getInput();
int idx = e.getIndex();
String illChar = String.valueOf(input.charAt(idx));
_uriStr = input.replace(illChar,URLEncoder.encode(illChar,enc));
}
}
return _uriStr;
}
test:
String q = "\\'|&`^\"<>)(}{][";
String url = "http://test.com/?q=" + q + "#" + q;
String eic = encodeIllegalChar(url,'UTF-8');
System.out.println(String.format(" original:%s",url));
System.out.println(String.format(" encoded:%s",eic));
System.out.println(String.format(" uri-obj:%s",new URI(eic)));
System.out.println(String.format("re-decoded:%s",URLDecoder.decode(eic)));

If you're using RestangularV2 to post to a spring controller in java you can get this exception if you use RestangularV2.one() instead of RestangularV2.all()

Replace spaces in URL with + like If url contains dimension1=Incontinence Liners then replace it with dimension1=Incontinence+Liners.