Due to the way my build system is designed (RTC Build Engine), I would like to provide maven with property values via a properties file, instead of specifying -Dkey=value for every property.
I found a couple of questions on S.O. (How to set build properties from a file in Maven POM? and How to read an external properties file in Maven) that relate precisely to this question, but they are relatively old, and both require custom plugins to work (in alpha state).
I realize that passing parameters to Maven like this is probably not the best solution, but the other option is specifying everything on the command line via -D settings which is not ideal either.
Furthermore, given that this properties file is only really used by the build engine (and not by the individual developer), I don't truly believe it belongs in the pom. But I cannot find any other mechanism that would allow me to specify a plugin to use - settings.xml does not permit specifying plugins.
Is my only choice in this case to use a plugin and specify it in the project pom?
in the pom you can place...
<properties>
<core-version>1234</core-version>
<lib-version>1234</lib-version>
<build-version>9999</lib-version>
<build-date>20150101</build-date>
</properties>
with all the properties you require.
Or you can use...
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>properties-maven-plugin</artifactId>
<version>1.0-alpha-2</version>
<executions>
<execution>
<phase>initialize</phase>
<goals>
<goal>read-project-properties</goal>
</goals>
<configuration>
<files>
<file>dev.properties</file>
</files>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
and the file dev.properties will contain the properties
core-version=1234
lib-version=1234
build-version=9999
build-date=20150101
...
Or... you can inject the properties using a settings.xml file as shown here
You may also find the Maven build number plugin useful... here
The best in such cases is to upgrade to at least Maven 3.2.1 which supports defining such properties on the command line like the following:
mvn -Drevision=1234 -Dchangelist=WhatEver -Dsha1=XXXX clean package
But you can only use the above names.
Excerpt from release notes:
A simple change to prevent Maven from emitting warnings about versions
with property expressions. Allowed property expressions in versions
include ${revision}, ${changelist}, and ${sha1}. These properties can
be set externally, but eventually a mechanism will be created in Maven
where these properties can be injected in a standard way. For example
you may want to glean the current Git revision and inject that value
into ${sha1}. This is by no means a complete solution for continuous
delivery but is a step in the right direction.
Related
I have a project compose by more than one module, and an integration test (in the test folder) where I want to run this script using the #sql annotation. By default the class path resource is used.
The test is inside this folder:
mainFolder/module1/src/test/java/com/.../.../controllers/TestClass.java
while the script is present in this folder:
mainFolder/scripts/postgres/script.sql
Basically I'm not sure which string (relative path) I should put in the value parameter of the #Sql annotation.
I am afraid that if your scripts are not copied to the class path there are not a lot of options.
Please, try something like:
#Sql("file:/path-to-mainFolder/mainFolder/scripts/postgres/script.sql")
As you can see in the documentation you can use any valid resource type.
Having said that, I think the best option is to make these resources available in the classpath. If you are using maven, you can use for instance the copy-resources goal of the maven resources plugin to copy your resources when running your tests:
<project>
...
<build>
<plugins>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<id>copy-resources</id>
<!-- here the phase you need: validate, test-compile... -->
<phase>validate</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${basedir}/target/scripts</outputDirectory>
<resources>
<resource>
<!-- Depending on your project, try defining the scripts src location as you consider more appropriate -->
<directory>mainFolder/scripts/postgres/script.sql</directory>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
...
</build>
...
</project>
With this setup, now you can define your script location as a classpath resource:
#Sql("/scripts/postgres/script.sql")
Perhaps, I am not sure about that, in a similar fashion you can use the plugin testResources goal as well.
If you want to use the scripts on your local environment machine you can also add the directory mainFolder/scripts/postgres as a dependency. In Intellij, you con go to Project Structure -> Modules -> your module -> dependencies -> + -> Jars or directories -> your script directory -> Classes.
Then you will be able to use
#Sql("/yourScript.sql")
In DevOps pipeline you would need to add the dedicated directory to your application server classpath.
Keep in mind that you will have to keep unique names for your sql files, because if you intend to use a structure with multiple directory levels, I think the jvm will load only the first entry. It is not a recommended approach, but it is fast if you only need to generate a report or test something on your local environment.
Another approach would be to specify at runtime -Xbootclasspath/a:. Then you would be able to use #Sql with relative paths as you initially wanted.
E.g.
-Xbootclasspath/a:path-to-mainFolder
then you would be able to use
#Sql("/scripts/postgres/script.sql")
import software.amazon.awscdk.services.lambda.Function;
Function helloLambda = new Function(helloStack, "hellocdkworld123", FunctionProps.builder()
.functionName("HelloLambda")
.code(Code.fromAsset("target/cdkhello-0.1.jar")) // <- x ?
.runtime(Runtime.JAVA_8)
.handler("com.myorg.functions.HelloLambda::sayHello") <- y?
.build());
There is also a possibility to reference it by S3 bucket. But when I run cdk bootstrap I get a generated bucket with generated name of the jar file. How should I be able to reference that before hand from code? Of course now I could write the exact bucket + file but then purpose of defining it from code is lost right?
First of all, assuming that the method that you want to execute when the Lambda is invoked is sayHello, from the com.myorg.functions.HelloLambda class, then that part of your solution is correct. The more difficult part is actually accessing the JAR with your Lambda code in it.
NOTE: I've updated my original answer with what I think is a better way to accomplish this. In order to avoid confusion and making this answer too wordy, I've removed the original answer, though much of it is common with this one. I credit this answer for helping to improve this answer.
Pass the path to the dependent resource's JAR to CDK
TL;DR
Create a new property for the full path to your Lambda JAR.
Associate dependency and execution related goals into the package phase of the build.
Update cdk.json to point to the the package phase.
Pass the full path via a system property to your CDK code.
Use the System property to pass to Code.asset(...).
Preparation
I've separated out the Lambda and the CDK infrastructure code into separate Maven modules. The intention being that once the Lambda code is compiled, packaged up into an uber JAR (its code plus all of its dependencies' code), the infrastructure module can refer to it as a dependency, passing the full path to the Lambda JAR to the App/Stack class to that it can use it as an asset.
Create a new property for the full path to your Lambda JAR.
In the properties section of your pom.xml, create a new property to refer to your Lambda JAR. Something like this:
<properties>
...
<lambda.jar>${GROUP_ID:ARTIFACT_ID:jar}</lambda.jar>
...
</properties>
Populate a property with the full path to your Lambda dependency's JAR, using the dependency plugin.
<build>
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>3.1.1</version>
<executions>
<execution>
<goals>
<goal>properties</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
</plugin>
...
<plugins>
</build>
This associates the properties goal with the process-resources phase. Whenever that phase of the build occurs, the property you've created previously will be populated with the full path to the JAR in your local repository.
Associate dependency and execution related goals into a single phase of the build.
When you create a new CDK Java project, it outputs a file called cdk.json, which points by default to the Maven exec:java goal. In order for your new lambda.jar property to be populated correctly, you need to associate the exec:java goal with the same phase as above.
<build>
<plugins>
...
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.6.0</version>
<executions>
<execution>
<goals>
<goal>java</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
<configuration>
<mainClass>com.myorg.TestingApp</mainClass>
</configuration>
</plugin>
...
</plugins>
</build>
In order for your code to get access to the JAR file that you've generated, you need to create a System property (I couldn't get environment variables to work) to your App class. Your pom.xml started with something like this:
Pass the full path via a system property to your CDK code.
In the configuration section (after mainClass), add a system property for your assets directory, something like this:
<systemProperties>
<systemProperty>
<key>lambda.jar</key>
<value>${lambda.jar}</value>
</systemProperty>
</systemProperties>
Update cdk.json to point to the the common phase you've used.
Your cdk.json of your CDK project should be changed to point to the process-resources phase. Once done it will look like this:
{
"app": "mvn package"
}
It will cause both the goals to be run in succession, and upon execution the path to your Lambda's JAR will be passed as a system property.
Access the property from your App/Stack code.
Finally, now that the system property is created, you can access it from your code by calling System.getProperty("lambda.jar"). Something like this:
final Code code = Code.fromAsset(System.getProperty("lambda.jar"));
You can then use the code reference wherever needed when defining your Lambda functions.
We use Jenkins which use md5 fingerprinting to identify artifacts and whether the artifact has changed since the last build. Unfortunately Maven builds always generate binary different artifacts.
Therefore I am looking into making Maven generate the same jar artifact for the same set of input files regardless of where and when they were built, which amongst other things mean that the entries in the jar file must be sorted - not only in the index, but in the order they are written to the jar file.
After examining maven-jar-plugin which use maven-assembly-plugin, my conclusions are that they do not collect all files to be written in memory before writing them all at once, but write one at a time. This mean that it may be better to postprocess the generated jar instead of changing the current behavior so I at that time can sort the entries, zero the timestamps, etc.
I am unfamiliar with writing Maven plugins, so my question is, how should I write a plugin which Maven knows how to tell where the artifact-jar-in-progress is located and how I hook it up in my pom.xml?
(At first I need this to work for jar files, but war files would be nice too).
As mentioned, this can be done based on something similar to maven-shade-plugin. I went ahead and wrote a simple plugin to add this capability -- see https://github.com/manouti/jar-timestamp-normalize-maven-plugin (available on the Central repo).
The behavior is based on the shade plugin. It consists of a single goal called normalize which can be bound to the package lifecycle phase and configured in the project's POM:
<plugins>
<plugin>
<groupId>com.github.manouti</groupId>
<artifactId>jar-timestamp-normalize-maven-plugin</artifactId>
<version>1.0-SNAPSHOT</version>
<executions>
<execution>
<id>jar-normalize</id>
<goals>
<goal>normalize</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
</plugin>
</plugins>
A few notes about the plugin:
The artifact under build is accessed via project#getArtifact() where project is a org.apache.maven.project.MavenProject.
Normalization consists of mainly three steps:
Setting the last modified time of all Jar entries to a specific timestamp (default value is 1970-01-01 00:00:00AM but can be changed via -Dtimestamp system property).
Reordering (alphabetically) of attributes in the manifest except for Manifest-Version which always comes first.
Removing comments from the pom.properties file which contain a timestamp that causes the Jar to differ from one build to another.
Once invoked, the goal will generate the output file next to the original artifact (named artifactId-version-normalized.jar), i.e. in the project.build.directory directory.
To create maven plugin project
mvn archetype:generate \
-DgroupId=sample.plugin \
-DartifactId=hello-maven-plugin \
-DarchetypeGroupId=org.apache.maven.archetypes \
-DarchetypeArtifactId=maven-archetype-plugin
invoke this command it will generate a skeleton project with a class called MyMojo.java
write your stuff inside execute() method, and install that plugin to your repository by mvn clean install
then attach its execution with your project, in your project pom.xml
<build>
<plugins>
<plugin>
<groupId>sample.plugin</groupId>
<artifactId>hello-maven-plugin</artifactId>
<version>1.0-SNAPSHOT</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>sayhi</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
to access project properties inside your Mojo
/**
* The Maven project.
*
* #parameter expression="${project}"
* #required
* #readonly
*/
private MavenProject project;
and then
project.getProperties("build.directory")
and read other properties to determine your jar file packed
See
maven: guide-java-plugin-development
I agree on creating a custom maven plugin seems like a better option. I dont know about an existing plugin provides solution for what you asked.
md5 checksum (or sha-1 in my repository) is generated with install plugin, so seems like you need to extend this or write a new plugin which works after install phase.
I have 2 suggestions about this plugin:
1) When thinking simple, this plugin should:
Read generated jar:
Extract all entries.
Exclude some entries (e.g. MANIFEST.MF).
Sort remaining entries .
Extract md5s for each in memory.
Generate a single md5 from all of those extracted.
However when considering about where & when independency: Accordig to .class file structure Java_class_file there is minor, major versions entries are held in compiled class files. So if compiler changes, .class files will be changed. In this case we need a check on source code level from this point :( So this solution become useless if there is no guarantee on copiler version.
2) As very dirty but easy solution, this plugin may only extract your module's pom.xml file's md5 code. But you must guarantee each change in your jar reflects to a minor version (or built number) manually.
Instead of writing your own plugin you can write a Groovy script that is executed by groovy-maven-plugin:
<plugin>
<groupId>org.codehaus.gmaven</groupId>
<artifactId>groovy-maven-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>execute</goal>
</goals>
<configuration>
<source>
import java.util.jar.*
String fileName = '${project.build.directory}/${project.build.finalName}.jar'
println "Editing file ${fileName}"
JarFile file = new JarFile(fileName);
// do your edit
</source>
</configuration>
</execution>
</executions>
</plugin>
I'm on creating a maven based java project, which contains the PMD maven plugin. I use my own rule set XML and it works like a charm, except two rule sets: the emptycode and the unnecessary: when I run the build, maven says: "can't find resource". The role definitions look like:
<role ref="rulesets/emptycode" />
and
<role ref="rulesets/unnecessary" />
In every other cases, this kind of definition works. What I found out is that: there is a rule set with the name "unnecessary" under ecmasrcipt category, so maybe this definition needs some suggestion to use java version. I tried multiple thinks, like set language attribute to the ruleset xml node ("JAVA", based on PMD JavaDoc), and some pre-postfix in ref, but it doesn't work and I found no working solution over the web. Does someone has an idea, what I forgot to set, or what I fail? Thanks for any help!
PMD seems to be a fiddly beastie to use from Maven. I've just figured this out with version 3.0 of the plugin - there are two solutions:
The quick-and-dirty solution: put rulesets in your project:
download the PMD jar (http://sourceforge.net/projects/pmd/files/latest/download)
extract lib/pmd-x.x.x.jar
extract from that PMD jar file the rulesets/<type>/<ruleset>.xml files you want to use
place them in a folder under your project - something like ${basedir}/pmd/...
reference them as follows:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-pmd-plugin</artifactId>
<configuration>
<rulesets>
<ruleset>${basedir}/pmd/<ruleset>.xml</ruleset>
</rulesets>
</configuration>
</plugin>
The advantage is this is easy, the disadvantage is if you update the PMD version in future you'll need to remember to update these files.
The nice solution: reference rulesets in pmd-x.x.x.jar.
create a custom ruleset such as: ${basedir}/pmd/custom.xml (see http://pmd.sourceforge.net/pmd-5.0.2/howtomakearuleset.html)
reference the PMD rulesets in the following way: <rule ref="rulesets/java/imports.xml"/>
NB: the path is the path inside pmd-x.x.x.jar (see quick-and-dirty above) with no leading slash
reference your custom ruleset as follows:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-pmd-plugin</artifactId>
<configuration>
<rulesets>
<ruleset>${basedir}/pmd/custom.xml</ruleset>
</rulesets>
</configuration>
</plugin>
The advantage is this will always reference the current PMD rulesets from the PMD jar, the disadvantage is it's a bit fiddly to get right.
To experiment with this until it was working (maven-pmd-plugin version 3.0) I kept running mvn pmd:pmd (<linkXref>false</linkXref> in pom.xml) and tweaked the paths until I stopped getting errors.
My Hudson job calls a single XML database collection and a single log file. E.g.
/db/project
${user.home}/logs/logging.log
I'd like to inject Hudson's env.EXECUTOR_NUMBER in both paths to avoid concurrent execution clashes. E.g.
/db/project {$EXECUTOR_NUMBER}
{$user.home}/logs {$EXECUTOR_NUMBER}/logging.log
I've found out the following:
Hudson site states that I need to pass Hudson's EXECUTOR_NUMBER to Maven as part of the build goal.
Log4J can only use system variables not environment variables (as these are a platform specific concept)
Now that I've got Hudson's EXECUTOR_NUMBER variable specified in the build goal, how can I use this as a system property for use by Log4J and Java's System.getProperties() class?
You can pass System Properties to any Java process using the -D syntax.
mvn clean install -DEXECUTOR_NUMBER={$EXECUTOR_NUMBER}
For a test class in a forked run, you will additionally have to configure the surefire plugin to pass the system property to the forked vm:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.9</version>
<configuration>
<systemPropertyVariables>
<EXECUTOR_NUMBER>${EXECUTOR_NUMBER}</EXECUTOR_NUMBER>
</systemPropertyVariables>
</configuration>
</plugin>