We have a java project where we have some default configurations under src/main/resources and there are overwrites under project_root/configDeploy
For our mapred jars we want to copy both configs but allow config/deploy files to overwrite defaults in resources. So we can have myconf.xml in resource and myconf.xml in deploy, but the mapred fat jar generated only has myconf.xml
I have tried two different methods, try to have deploy copy and overwrite the resources conf:
from 'src/main/resources'
from 'conf/deploy'
but this will add two files inside the jar, so it didn't work
Then I tried to add only files from src/main/resources that are not in conf/deploy, something like:
into('conf'){
from{
'src/main/resources'
}
exclude{file('deploy/conf/')}
}
into('conf'){
from{
'deploy/conf'
}
}
but this didn't work, as a result none of the confs from resources were copies.
So question is if I have a two folders with files which some of them have same name how can I include their files in jar so I get files from both folders but for files that are in both I get only the version in second folder.
Thanks for your help!
To avoid the duplicate files in the JAR you can set the duplicates strategy to EXCLUDE on the task.
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
This will cause subsequent attempts to add the file to be ignored. Therefore, if you want files in 'deploy/conf' to take precedence you should define that copy spec first.
Related
I need to get a resource from inside the root of the application when its packed into jar. My project is like this:
ProjectRoot
resource.txt //want to access from here
src
main
java
package1
package2
package3
Main.java
target
app.jar
classes
resource.txt //works when here
package1
package2
package3
Main.class
I use this code:
Path path = Paths.get("resource.txt");
When run before packaging into a jar, it finds the file just fine (inside ProjectRoot). When running the jar, it can't find it, and transforms this path to target/resource.txt.
This code:
BufferedReader br = new BufferedReader(new InputStreamReader(new Main().getClass().getClassLoader().getResourceAsStream(
"resource.txt")));
when run before packaging looks for the resource inside target/classes. After packaging it claims to taking the resource from .../target/app.jar!/resource.txt.
This code:
BufferedReader br = new BufferedReader(new InputStreamReader(new Main().getClass().getClassLoader().getResourceAsStream(
"/resource.txt")));
I can't understand where's looking for the resource, but it doesn't seem to be ProjectRoot.
All I want to do is to place the resource inside ProjectRoot and be able to access it from both outside jar (when running the class files from IDE) and inside (after having packaged the files into a jar file using Maven).
EDIT: I NEED THE CODE TO WORK BOTH FOR PRE- AND POST- packaging. MEANING: If I run a Main.java FROM INSIDE IDE IT WOULD GET THE RESOURCE; IF I PACKAGE EVERYTHING INTO JAR AND RUN JAR IT WOULD GET THE RESOURCE - ALL WITH THE SAME CODE.
Use: Main.class.getResource("/resource.txt").
Note that your attempt using any call to getClassLoader is strictly worse (it's more text, and will fail more often, because that class loader can in exotic cases be null (specifically, when you're part of the bootstrap loader), whereas calling getResource directly on the class always works.
The reason your snippet does not work is because when invoking getResource on the classloader, you must NOT start the resource with a slash. When invoking on a class directly, you can (if you don't, it'll be relative to the package of the class you're calling it on, if you do, it'll be relative to the root).
TL;DR: Of the forms SomeClass.class.getClassLoader().getResource, getClass().getResource and MyClass.class.getResource, only the last one is correct, the rest are strictly inferior and therefore should not be used at all.
Maven uses something called the Standard Directory Layout. If you don't follow this layout then the plugins can't do their job correctly. Technically, you can configure Maven to use different directories but 99.999% of the time this is not necessary.
One of the features of this layout is that production files go in:
<project-dir>/src/main/java
All *.java files
<project-dir>/src/main/resources
All non-*.java files (that are meant to be resources)
When you build your project the Java source files are compiled and the *.class files are put into the target/classes directory; this is done by the maven-compiler-plugin. Meanwhile, the resource files are copied from src/main/resources into target/classes as well; the maven-resources-plugin is responsible for this.
Note: See Introduction to the Build Lifecycle for more information about phases and which plugins are executed by which phase. This Stack Overflow question may also be useful.
When you launch your application from the IDE (possibly via the exec-maven-plugin) the target/classes directory is put on the classpath. This means all the compiled classes from src/main/java and all the copied resources from src/main/resources are available to use via the classpath.
Then, when you package your application in a JAR file, all the files in target/classes are added to the resulting JAR file (handled by the maven-jar-plugin). This includes the resources copied from src/main/resources. When you launch the application using this JAR file the resources are still available to use via the classpath, because they're embedded in the JAR file.
To make resource.txt available on the classpath, just move:
<project-dir>/resource.txt
To:
<project-dir>/src/main/resources/resource.txt.
Then you can use Class#getResource with /resource.txt as the path and everything should work out for you. The URL returned by getResource will be different depending on if you're executing against target/classes or against the JAR file.
When executing against target/classes you'll get something like:
file:///.../<project-dir>/target/classes/resource.txt
When executing against the JAR file you'll get something like:
jar:file:///.../<project-dir>/target/projectname-version.jar!/resource.txt
Note: This all assumes resource.txt is actually supposed to be a resource and not an external file. Resources are typically read-only once deployed in a JAR file; if you need a writable file then it's up to you to use a designated location for the file (e.g. a folder in the user's home directory). One typically accesses external files via either java.io.File or java.nio.file.*. Remember, resources are not the same thing as normal files.
Now, if you were to put resource.txt directly under <project-dir> that would mean nothing to Maven. It would not be copied to target/classes or end up in the JAR file which means the resource is never available on the classpath. So just to reiterate, all resources go under src/main/resources.
Check out the Javadoc of java.lang.Class#getResource(String) for more information about the path, such as when to use a leading / and when not to. The link points to the Javadoc for Java 12 which includes information about resources and modules (JPMS/Jigsaw modules, not Maven modules); if you aren't using modules you can ignore that part of the documentation.
So, I've recently (partially) completed a Java project with Gradle. Importantly, the project uses absolute pathing to access files in my resources folder, since those files will change after the JAR is made. When I use Eclipse's "export as runnable JAR" functionality, I get something that works perfectly - putting the .jar file in my main directory lets it find everything. However, using Gradle's build function doesn't, because Gradle adds extra layers between the .jar and the resources. To demonstrate, here's my "normal" directory:
./program root directory
|_program.jar
|_resources
|_[actual resources]
And here's the directory Gradle makes:
./build folder
|_libs
| |_program.jar
|_resources
|_main
|_[actual resources]
What I want from Gradle is:
./build folder
|_program.jar
|_resources
|_[actual resources]
Yes, I could manually move the resources and program.jar around in the directory to achieve this, but that feels wrong - this is exactly what Gradle is supposed to do for me, right? I know there has to be SOME way to do it. I just don't know how. So that's why I'm asking for help - how do I do this?
To change the output of resources:
sourceSets.main.output.resourcesDir = "$buildDir/resources"
To change where the JAR file is put:
jar {
// use destinationDir for Gradle < 5.1
destinationDirectory = buildDir
}
If all your resources are meant to be external you may want to exclude them from the JAR file:
jar {
include '**/*.class'
destinationDirectory = buildDir
}
That will only include .class files from the jar task's input. You can customize this using the include and exclude options.
I currently have two projects:
api-test
...
/config/config.json
...
and
ui-test
...
/config/config.json
...
In eclipse, I am adding api-test in the build path of ui-test, so that api-test is the dependency of ui-test.
However the build failed, because api-test is looking for the config.json located in api-test/config/config.json by calling:
System.getProperty("user.dir") + "/config/config.json"
which does not exist in ui-test project.
the two config.json include different contents - what would be the best solution to let each project refer to their own config.json while ui-test is referring to api-test project?
Put the files into the projects' src/main/resources directories as suggested by Maven's Standard Directory Layout. You can use relative paths to access these resources then.
See How to get file resource from Maven src/test/resources/ folder in JUnit test? For instance:
Test file existence
#Test
public void testStreamToString() {
assertNotNull("Test file missing", getClass().getResource("/sample.txt"));
...
}
I'm trying to add a bunch of dependencies stored on hdfs to distributed cache. I've been following the advice from this article: http://www.datasalt.com/2011/05/handling-dependencies-and-configuration-in-java-hadoop-projects-efficiently/. My question is: is it possible to add a folder containing the dependencies to the classpath?
DistributedCache.addFileToClassPath(new Path("/tmp/lib/"), job.getConfiguration());
Or would I need to add each dependency individually?
for (Path dependency : dependencies) {
DistributedCache.addFileToClassPath(dependency, job.getConfiguration());
}
And how would I check that the dependencies were actually added to the classpath on all the slave nodes?
Thanks.
You'll need to iterate the jars and add them one at a time as you suggested. Or you can bundle the jars into a single zip file and then use the DistributedCache.addArchiveToClassPath(Path, Configuration) method.
To check they were added to the classpath, try examining the System property java.class.path in the setup method of a mapper / reducer.
I have a small Java/Gradle project. I'm using the Application plugin to create a zip distribution (using the distZip task). Using the standard configuration I get the following directories in my zip file:
/bin - The scripts to start the application go in here
/lib - Contains my project code in a JAR file and all dependency JAR files.
The trouble is that I would like a third directory: /conf where I can put my configuration files (instead of having them packaged inside my application JAR file.
I imagine that this is a pretty common requirement because things like log4j.xml and hibernate.properties would be better placed outside the JAR file. I just can't figure out how I can customise the behavior of the Application plugin to do this however.
I revisited this problem several months later and I finally have an elegant solution. The following code should be added to the gradle file:
distZip {
into(project.name) {
from '.'
include 'conf/*'
}
}
This adds an additional include to the distZip task. This copies the "conf" directory (including contents) into the Zip distribution.
The generated zip file contains a single directory which is the same as the project name. This is why the "into" part is required.
Actually, create a dist dir under the src dir in your project. Anything in this dir is copied by the application plugin (under applicationDistribution) when installApp or distZip is run.
Or edit applicationDistribution to do other things, if a simple copy is not enough.
For me, a simple
applicationDistribution.from("src/main/config/") {
into "config"
}
did the job. Of course you need to have your properties loaded correctly from within code. Especially if you move them from src/main/resources where they have been usable via classpath, into the new location. I circumvented this by adding a command line parameter which points to the configuration file.
I am not sure whether you can customize the application plugin, I have never used it. There is however other ways to achieve what you want to achieve.
You may create a /conf directory like this:
confDir = new File("$buildDir/conf")
You can then copy the files you need into this directory like this:
task copyConfFiles(type: Copy) {
from _wherever your files reside_
into confDir
include('**/*.properties') // your configuration files
}
You may then hook this copy task into the process like this:
distZip.dependsOn copyConfFiles
And last if you do not want your configurations in the final zip, you can do this:
distZip {
exclude('**/*.properties') // your configuration files
}
Again, there might be a better way. This is a way.
OP's self-answer may be good for his use case, but there are a few things I'd like to improve on:
His answer suggests that he has a directory conf parallel to the build.gradle. There is no such thing in the Maven Standard Directory Layout. The general consensus is to have a src/main/conf as had been hinted to in the docs:
If there are other contributing sources to the artifact build, they
would be under other subdirectories: for example src/main/antlr would
contain Antlr grammar definition files.
The target directory name is NOT project.name as had been pointed out in a comment.
If resource filtering is required, and it often is, then having a separate task is desirable. During local development, this task can be run to generate the filtered files. The distribution would merely use the output of this task (and unlike OP's answer, this also makes conf available to the tar distribution).
def props = new Properties()
file("src/main/filters/application.properties")
.withInputStream { props.load(it) }
import org.apache.tools.ant.filters.ReplaceTokens
task copyConf(type: Copy) {
from("src/main/conf/")
into("$buildDir/conf")
filesMatching("**/*.y*ml") {
filter(tokens: props, ReplaceTokens)
}
}
distributions {
main {
contents {
from(copyConf) {
into("conf")
}
}
}
}