In Wikipedia, it says:
Windows
Here are some examples which may be accepted by some applications on Windows systems, referring to the same, local file c:\WINDOWS\clock.avi
file://localhost/c:/WINDOWS/clock.avi
file:///c:/WINDOWS/clock.avi
Here is the URI as understood by the Windows Shell API:[2]
file:///c:/WINDOWS/clock.avi
Suppose, I need conversion of filenames type: C:\WINDOWS\clock.avi to URIs for use in a JavaFX WebView as href for tags, css, links etc. etc.
The app intercepts clicks on the links, analyzes and performs actions, depending on the link (runs pdf, internet, music etc.)
But, I don't manage to generate the proper URI from filenames to be used in those webview pages.
These are the results that Java provides:
Code
private static void print(String s) {
System.out.println(s);
}
public static void main(String[] args) throws Exception {
String fileName = "c:\\WINDOWS\\clock.avi";
print("0 - " + new File(fileName).toString());
print("1 - " + new File(fileName).toURI().toString());
print("2 - " + new File(fileName).toURI().toURL().toString());
print("3 - " + new URL(new File(fileName).toURI().toString()).toString());
}
Results:
0 - c:\WINDOWS\clock.avi
1 - file:/c:/WINDOWS/clock.avi
2 - file:/c:/WINDOWS/clock.avi
3 - file:/c:/WINDOWS/clock.avi
So, No 1, 2 and 3 are all same, and none of them are correct - as I infer from URI rules. Is it a Java bug?
Is there a proper way to get exactly the file:///c:/WINDOWS/clock.avi uri from a file path c:\WINDOWS\clock.avi instead of the resulting file:/c:/WINDOWS/clock.avi, which is reported as wrong in various places?
The StackOverflow most popular answers is use No. 1 or No. 2, but the above shows that the results are not correct URIs.
Calling toUri() on a Path yields the URI you want. I don't see anything in the documentation that states this is guaranteed behavior, however; or if it is in the documentation, I'm just not recognizing it. So, in case this behavior is subject to change, note that I only tried this using JDK 10.0.1.
In my opinion, if you can switch your code to use the file API introduced with NIO2 (Java 7), rather than java.io that would be ideal. If you can't or don't want to for whatever reason, then you can convert a File into a Path using File.toPath() (and vice versa using Path.toFile()).
Using this code:
File file = new File("C:\\Users\\Username\\Desktop");
System.out.printf("File.toURI() = %s%nFile.toPath().toUri() = %s%n",
file.toURI(), file.toPath().toUri());
Gave me this output:
File.toURI() = file:/C:/Users/Username/Desktop/
File.toPath().toUri() = file:///C:/Users/Username/Desktop/
Related
I am working on fixing Veracode issues in my application. Veracode has highlighted the flaw "External Control of File Name or Path (CWE ID 73) " in below code.
Thread.currentThread().getContextClassLoader().getResourceAsStream(lookupName)
How do I validate the parameter? If I need to use below ESAPI validation, then what is the exact parameter I should be passing in getValidFileName() method. Currently I am passing the parameters as below.
ESAPI.validator().getValidFileName(lookupName, lookupName,
ESAPI.securityConfiguration().getAllowedFileExtensions(), false);
Correct me whether I am following the right approach for fixing this issue.
There are several suggestions at: https://community.veracode.com/s/article/how-do-i-fix-cwe-73-external-control-of-file-name-or-path-in-java
You can use hardcoded values, if these files are stored in the server side.
(i.e.: in a HashMap).
Another solution is to use a custom validator (from veracode page) :
// GOOD Code
String extension = request.getParameter("extension");
File f = new File(buildValidAvatarPath(extension))
#FilePathCleanser
public String buildValidAvatarPath(extension) {
String[] allowedExtensions = new String[]{"jpg","gif","png"};
String extension = "png"; // Default extension
for (String allowedExtension: allowedExtensions) {
if (allowedExtension.equals(request.getParameter("extension"))) {
extension = request.getParameter("extension");
}
}
// See "Note on authorization"
User user = getCurrentUser();
if (!userMayAccessFile(user, path)) {
throw new AuthorizationException("User may not access this file", user);
}
File(configPath + "avatar." + extension)
return path;
}
Okay, so the problem is that you are allowing user-control of that file path. Imagine its on a UNIX box and they enter:
../../../../../../../etc/shadow
Whatever user privileges are granted to the user running that java Thread is possible to expose to the user in question. I don't know what processing is going on in your application, but the danger is that you need to prevent user control of that lookup variable.
The call you're making is consistent with the single test in ValidatorTest.java, which is definitely a deficiency in code coverage on our behalf.
Now, there's an excellent chance that even if you use this call that Veracode might still flag it: the default file list in ESAPI.properties will need to be either truncated for your use case, or you'll have to create your own Validator rule for legal file extensions for your specific use case.
Which brings up the next bit: There's a lot of mischief that can happen in regards to file uploads.
In short, to be actually secure about file uploads will require more than what ESAPI currently offers, which is unfortunately, only an extension check. In your particular case, make sure you try some directory traversal attacks. And use that OWASP link to help analyze your application.
Given that the OP wants to clear the issue in Veracode, you would want to chain a couple calls:
ESAPI.validator().getValidDirectoryPath() and ESAPI.Validator.getValidFileName()
But be sure you've properly truncated the extension list in HttpUtilities.ApprovedUploadExtensions in validator.properties as the default list is too permissive, at least until we release 2.1.0.2.
I have to stress however that even with this particular combination there is absolutely nothing ESAPI does to prevent a user from renaming "netcat.exe" to "puppies.xlsx" and bypassing your validation check, that's why the rant on the first part of this answer.
ESAPI's file validation is NOT secure, it's quite simply better than nothing at all.
Doing this correctly requires more work than just using 1-2 calls to ESAPI.
DISCLAIMER: as of this writing I am the project co-lead for ESAPI.
You can change file name by sanitizing it as below code snippet:
private static String sanitizeFileName(String name) {
return name
.chars()
.mapToObj(i -> (char) i)
.map(c -> Character.isWhitespace(c) ? '_' : c)
.filter(c -> Character.isLetterOrDigit(c) || c == '-' || c == '_' || c == ':')
.map(String::valueOf)
.collect(Collectors.joining());
}
I am currently trying to implement file exports in background so that the user can do some actions while the file is downloading.
I used the apache isis CommandExexuteIn:Background action attribute. However, I got an error
"Not an (encodable) value", this is an error thrown by the ScalarValueRenderer class.
This is how my method looks like:
#Action(semantics = SemanticsOf.SAFE,
command = CommandReification.ENABLED)
commandExecuteIn = CommandExecuteIn.BACKGROUND)
public Blob exportViewAsPdf() {
final Contact contact = this;
final String filename = this.businessName + " Contact Details";
final Map<String, Object> parameters = new HashMap<>();
parameters.put("contact", contact);
final String template = templateLoader.buildFromTemplate(Contact.class, "ContactViewTemplate", parameters);
return pdfExporter.exportAsPdf(filename, template);
}
I think the error has something to do with the command not actually invoking the action but returns the persisted background command.
This implementation actually worked on the method where there is no return type. Did I miss something? Or is there a way to implement background command and get the expected results?
interesting use case, but it's not one I anticipated when that part of the framework was implemented, so I'm not surprised it doesn't work. Obviously the error message you are getting here is pretty obscure, so I've raised a
JIRA ticket to see if we could at least improve that.
I'm interested to know in what user experience you think the framework should provide here?
In the Estatio application that we work on (that has driven out many of the features added to the framework over the last few years) we have a somewhat similar requirement to obtain PDFs from a reporting server (which takes 5 to 10 seconds) and then download them. This is for all the tenants in a shopping centre, so there could be 5 to 50 of these to generate in a single go. The design we went with was to move the rendering into a background command (similar to the templateLoader.buildFromTemplate(...) and pdfExporter.exportAsPdf(...) method calls in your code fragment, and to capture the output as a Document, via the document module. We then use the pdfbox addon to stitch all the document PDFs together as a single downloadable PDF for printing.
Hopefully that gives you some ideas of a different way to support your use case
Thx
Dan
I have a problem with java tensorflow API. I have run the training using the python tensorflow API, generating the files output_graph.pb and output_labels.txt. Now for some reason I want to use those files as input to the LabelImage module in java tensorflow API. I thought everything would have worked fine since that module wants exactly one .pb and one .txt. Nevertheless, when I run the module, I get this error:
2017-04-26 10:12:56.711402: W tensorflow/core/framework/op_def_util.cc:332] Op BatchNormWithGlobalNormalization is deprecated. It will cease to work in GraphDef version 9. Use tf.nn.batch_normalization().
Exception in thread "main" java.lang.IllegalArgumentException: No Operation named [input] in the Graph
at org.tensorflow.Session$Runner.operationByName(Session.java:343)
at org.tensorflow.Session$Runner.feed(Session.java:137)
at org.tensorflow.Session$Runner.feed(Session.java:126)
at it.zero11.LabelImage.executeInceptionGraph(LabelImage.java:115)
at it.zero11.LabelImage.main(LabelImage.java:68)
I would be very grateful if you help me finding where the problem is. Furthermore I want to ask you if there is a way to run the training from java tensorflow API, because that would make things easier.
To be more precise:
As a matter of fact, I do not use self-written code, at least for the relevant steps. All I have done is doing the training with this module, https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/image_retraining/retrain.py, feeding it with the directory that contains the images divided among subdirectories according to their description. In particular, I think these are the lines that generate the outputs:
output_graph_def = graph_util.convert_variables_to_constants(
sess, graph.as_graph_def(), [FLAGS.final_tensor_name])
with gfile.FastGFile(FLAGS.output_graph, 'wb') as f:
f.write(output_graph_def.SerializeToString())
with gfile.FastGFile(FLAGS.output_labels, 'w') as f:
f.write('\n'.join(image_lists.keys()) + '\n')
Then, I give the outputs (one some_graph.pb and one some_labels.txt) as input to this java module: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/java/src/main/java/org/tensorflow/examples/LabelImage.java, replacing the default inputs. The error I get is the one reported above.
The model used by default in LabelImage.java is different that the model that is being retrained, so the names of inputs and output nodes do not align. Note that TensorFlow models are graphs and the arguments to feed() and fetch() are names of nodes in the graph. So you need to know the names appropriate for your model.
Looking at retrain.py, it seems that it has a node that takes the raw contents of a JPEG file as input (the node DecodeJpeg/contents) and produces the set of labels in the node final_result.
If that's the case, then you'd do something like the following in Java (and you don't need the bit that constructs a graph to normalize the image since that seems to be a part of the retrained model, so replace LabelImage.java:64 with something like:
try (Tensor image = Tensor.create(imageBytes);
Graph g = new Graph()) {
g.importGraphDef(graphDef);
try (Session s = new Session(g);
// Note the change to the name of the node and the fact
// that it is being provided the raw imageBytes as input
Tensor result = s.runner().feed("DecodeJpeg/contents", image).fetch("final_result").run().get(0)) {
final long[] rshape = result.shape();
if (result.numDimensions() != 2 || rshape[0] != 1) {
throw new RuntimeException(
String.format(
"Expected model to produce a [1 N] shaped tensor where N is the number of labels, instead it produced one with shape %s",
Arrays.toString(rshape)));
}
int nlabels = (int) rshape[1];
float[] probabilities = result.copyTo(new float[1][nlabels])[0];
// At this point nlabels = number of classes in your retrained model
DoSomethingWith(probabilities);
}
}
Hope that helps.
Regarding the "No operation" error, I was able to resolve that by using input and output layer names "Mul" and "final_result", respectively. See:
https://github.com/tensorflow/tensorflow/issues/2883
In a DirectoryWalker class I want to find out if a File instance is actually a symbolic link to a directory (assuming, the walker walks on UNIX systems). Given, I already know the instance is a directory, would the following be a reliable condition to determine the symbolic link?
File file;
// ...
if (file.getAbsolutePath().equals(file.getCanonicalPath())) {
// real directory ---> do normal stuff
}
else {
// possible symbolic link ---> do link stuff
}
The technique used in Apache Commons uses the canonical path to the parent directory, not the file itself. I don't think that you can guarantee that a mismatch is due to a symbolic link, but it's a good indication that the file needs special treatment.
This is Apache code (subject to their license), modified for compactness.
public static boolean isSymlink(File file) throws IOException {
if (file == null)
throw new NullPointerException("File must not be null");
File canon;
if (file.getParent() == null) {
canon = file;
} else {
File canonDir = file.getParentFile().getCanonicalFile();
canon = new File(canonDir, file.getName());
}
return !canon.getCanonicalFile().equals(canon.getAbsoluteFile());
}
Java 1.6 does not provide such low level access to the file system. Looks like NIO 2, which should be included in Java 1.7, will have support for symbolic links. A draft of the new API is available. Symbolic links are mentioned there, creating and following them is possible. I'm not exactly sure that which method should be used to find out whether a file is a symbolic link. There's a mailing list for discussing NIO 2 - maybe they will know.
Also, watch out for file.isFile() and file.isDirectory() both returning results based on the resolved file and therefore both returning false when file refers to a symlink where the target doesn't exist.
(I know this isn't a useful answer in itself but it tripped me up a couple of times so thought I should share)
It looks like getCanonicalPath() can do other things that might make it different from the absolute path.
This method first converts this pathname to absolute form if necessary, as if by invoking the getAbsolutePath() method, and then maps it to its unique form in a system-dependent way. This typically involves removing redundant names such as "." and ".." from the pathname, resolving symbolic links (on UNIX platforms), and converting drive letters to a standard case (on Microsoft Windows platforms).
But it might work for the vast majority of your use cases; your mileage may vary.
If you are already coding something specifically for *nix, then you could do a shell command from Java like this:
Process p = Runtime.getRuntime().exec(new String[]{"test", "-h", yourFileName});
p.waitFor();
if (p.exitValue() == 0)
System.out.println("This file is a symbolic link");
else
System.out.println("This file is not a symbolic link");
That's very specific to *nix, but it does at least work.
Sorry to reply to such an old post, but I was looking for a solution for Windows systems some time back, and some of the previous answers didn't work out for me. If you're not concerned with cross platform compatibility and only need a solution for Windows, the following technique worked well for my purposes.
File f = new File("whatever file or folder");
if (f instanceof ShellFolder) {
ShellFolder sf = (ShellFolder)f;
if (sf.isLink()) {
// Your code when it's a link
}
}
I thought I would share some good fortune I had in dealing with this issue. I am using JDK 1.6.0_23 and so I cannot benefit from NIO2. I am building and running on Windows 7 /x64 ONLY so mileage may vary in other environments. Unfortunately, other solutions here did not work for me in avoiding NullPointerExceptions caused when attempting to traverse a junction (probably because junction != symlink....). While I am not constrained by JDK version, I decided to keep at the problem for a bit longer.
I had this code which would cause a NullPointerException if used on a symbolic link or when encountering the 'System Volume Information' directory. (Note, traverseItem.f() returns an object of type java.io.File)
if (traverseItem.f().isDirectory) {
for (File item : traverseItem.f().listFiles()) {
So, it is supposedly a directory but calling listFiles() on it causes an NPE. What to do? I spied the list() method and wondered if it would exhibit the same behavior. What I discovered was the following:
Calling list() on a File describing an empty folder returns a String[] array of length zero. However, calling list() on a File describing a junction which would otherwise crash from listFiles() returns null
I was able to avoid the NullPointerExceptions by adding the following test before calling listFiles()
String[] contents = traverseItem.f().list();
if (contents != null) { //Non-traversible if null, possibly junction or ???
It remains to exhaustively test all cases of junction, symbolic link, hard link, and dare I mention it, shortcut, but this may help some.
I have been developing a Java application which executes a long series of queries and calculations, and presents its results as a series of HTML pages. For visualizing graphs I have been playing around with JUNG library for a while, and it appears as the real strength of the library is the support for user interaction, which is of course unavailable when the graph is saved as a static image (PNG in my case).
I was wondering if it would be:
a) possible
b) feasible
c) sensible
... to create an applet, during execution of the main application, which then can be insert to the HTML reports and can be used interactively after the application has finished execution and the user goes through the report pages.
If this is not possible due to technical reasons; do you have any alternative recommendations/ suggestions as to how I can achieve something like this?
Thanks,
EDIT: Just to clarify the concept, the "main" application is a link in a chain of events, and thus has so separate GUI. The idea with the applet is NOT to mimic or transport all the stuff from the main app to a HTML page, but to make it possible to use interactive tools that come with JUNG library, when the user is reviewing the graphical results AFTER the execution of the main software has finished.
Let me know if the concept is still rather unclear and I'll give a second attempt to explain things in further detail.
UPDATE: Following the advices I got, thnx to #boffinBrain & #AndrewThompson, I wrote my applet, and placed in a package in my project along with other visualization related classes. The hierarchy is as follows:
my.domain.project
my.domain.project.tests
my.domain.project.visualization
Now the HTML reports are created at an arbitrary location on the local drive, this is a feature as the user gives an output folder prior to running the "main" application. In my ReportGenerator class (which generates these HTML files) I have the following bit of code:
File bin = new File(getClass().getProtectionDomain().getCodeSource().getLocation().toString());
String codebase = bin.getParent();
System.out.println(codebase);
String archive = "lib/collections-generic-4.01/collections-generic-4.01.jar";
String applet_name = "bin/my.domain.project.visualization.HierarchyGraphApplet.class";
codebase printout shows: file:/home/username/workspace/project which is correct what I'd expected. Under the project folder there's bin/ and lib/, and inside bin there is the right hierarchy of folders all the way down to my applet class which also exists.
Now why did I write all this down; well because when I try to run my applet on the reports I get:
java.lang.NoClassDefFoundError: bin/my/domain/project/visualization/HierarchyGraphApplet (wrong name: my/domain/project/visualization/HierarchyGraphApplet)
I have read similar questions like: this or this but it seems like the problem is somewhere else. I double checked the spelling etc...
Is there something simple I am missing, or is there a more complicated problem at hand?
Maybe this example will give you some ideas to pursue. It creates data files used as 'reports' for consumption by the applet(s).
Because the applet gains the data via an input file whose title is specified in an applet param. The content of the data file is only limited by the requirements of the report, your skill to create it & parse it, ..and available disk space. ;)
Compile & run the main(String[]) to (hopefully) see 2 web pages open in tabs of your browser.
import java.awt.Desktop;
import javax.swing.*;
import java.net.*;
import java.io.*;
/** Simplistic example, not intended to show good I/O practices
or Exception handling for the sake of brevity. */
public class Reporter extends JApplet {
public void init() {
String input = getParameter("input");
JEditorPane report = new JEditorPane();
report.setText("Problem loading input file");
add(report);
URL url;
try {
url = new URL(getDocumentBase(), input);
report.setPage(url);
} catch(Exception e) {
e.printStackTrace();
}
}
/** The main represents our report generator. It is part
of the applet class only in order to create an SSCCE. Good
design would imply that it might be in a class ReportGenerator,
while the applet is in class ReportViewer. */
public static void main(String[] args) throws Exception {
File f;
String title = "1";
String data = "apples";
createInput(title, data);
f = createHTML(title);
Desktop.getDesktop().browse(f.toURI());
title = "2";
data = "oranges";
createInput(title, data);
f = createHTML(title);
Desktop.getDesktop().browse(f.toURI());
System.out.println( "End of report generation.." );
}
public static void createInput(String title, String data) throws Exception {
File f = new File("data" + title + ".txt");
PrintWriter pw = new PrintWriter(f);
pw.println(data);
pw.flush();
pw.close();
}
public static File createHTML(String title) throws Exception {
File f = new File("Data" + title + ".html");
PrintWriter pw = new PrintWriter(f);
pw.println("<html>");
pw.println("<title>");
pw.println("Data " + title);
pw.println("<title>");
pw.println("<body>");
pw.println("<h1>");
pw.println("Data " + title);
pw.println("</h1>");
pw.println("<applet ");
pw.println("code='Reporter'");
pw.println("width='400'");
pw.println("height='400'");
pw.println(">");
pw.println("<param name='input' value='data" + title + ".txt'>");
pw.println("</applet>");
pw.println("</body>");
pw.println("</html>");
pw.flush();
pw.close();
return f;
}
}
In relation to further questions:
..does the given code assume that the html reports and the applet are located in the same folder?
Not necessarily. The input parameter might specify ../other/data3.txt for the other directory at the same level as the one contained by the HTML, or /reports/data3.txt for a reports directory at the root of the site.
..As you have also noted, in a real-life example the code for the applet would most likely be in its own class, would that pose any complications as to how it would be incorporated into the html files (which are generated in a separate class, named ReportGenerator)?
It would require only slight changes to point to the applet.jar as opposed to the application.jar. Use a codebase to separate the HTML from the directory of the applet.jar (though archives can also be accessed via relative or absolute URLs).
It's definitely feasible to create an applet to display the data, but you don't want to dynamically generate a new one each time. You want to create a separate, stand-alone applet which can generate your graphs/reports from a set of input data in text format, and then when you create the HTML page, supply the report data using an applet parameter (using the PARAM tag).