The project I am working on needs to use Google Protobuf for serialization, therefore a number of stub has to be generated before my code is built.
The command line arguments I use is:
protoc -I=src/proto --java_out=src/main/java src/proto/*.proto
This works fine in the console.
I now wants to use Maven exec plugin so that this manual process becomes part of the Maven build. The pom section I used is:
<build>
<plugins>
<plugin>
<artifactId>exec-maven-plugin</artifactId>
<groupId>org.codehaus.mojo</groupId>
<version>1.3.2</version>
<executions>
<execution>
<id>Google Protobuf Stub Generation</id>
<phase>generate-sources</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>protoc</executable>
<commandlineArgs>-I=src/proto --java_out=src/main/java src/proto/*.proto</commandlineArgs>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
An error is given to complain there is no file called: src/proto/*.proto
However, it works fine if I remove the wildcard and specify a specific file, e.g.
<commandlineArgs>-I=src/proto --java_out=src/main/java src/proto/model.proto</commandlineArgs>
I think it is the wildcard * that cause the problem as Maven might have a different way to handle it.
My question is that how can I specify "All files with .proto extension in that folder" in Maven?
If you have bash, you can cheat and use 'bash' as your executable, '-c' as the first argument, and your full command as the second argument.
Essentially, the second argument will be ran in the shell, so wildcard(*) characters work and you generally get the same functionality as using the command line directly.
If you are calling the protoc command on a shell, the wildcard-operator is replaced by the shell with all matching files. protoc command itself is not able to handle the wildcard.
For example:
By calling "ls *.txt" on your shell, the ls command is not called with the argument "*.txt". The shell translates the command-call to "ls file1.txt file2.txt ..."
Solution: Create a command out of find xargs and protoc.
Related
import software.amazon.awscdk.services.lambda.Function;
Function helloLambda = new Function(helloStack, "hellocdkworld123", FunctionProps.builder()
.functionName("HelloLambda")
.code(Code.fromAsset("target/cdkhello-0.1.jar")) // <- x ?
.runtime(Runtime.JAVA_8)
.handler("com.myorg.functions.HelloLambda::sayHello") <- y?
.build());
There is also a possibility to reference it by S3 bucket. But when I run cdk bootstrap I get a generated bucket with generated name of the jar file. How should I be able to reference that before hand from code? Of course now I could write the exact bucket + file but then purpose of defining it from code is lost right?
First of all, assuming that the method that you want to execute when the Lambda is invoked is sayHello, from the com.myorg.functions.HelloLambda class, then that part of your solution is correct. The more difficult part is actually accessing the JAR with your Lambda code in it.
NOTE: I've updated my original answer with what I think is a better way to accomplish this. In order to avoid confusion and making this answer too wordy, I've removed the original answer, though much of it is common with this one. I credit this answer for helping to improve this answer.
Pass the path to the dependent resource's JAR to CDK
TL;DR
Create a new property for the full path to your Lambda JAR.
Associate dependency and execution related goals into the package phase of the build.
Update cdk.json to point to the the package phase.
Pass the full path via a system property to your CDK code.
Use the System property to pass to Code.asset(...).
Preparation
I've separated out the Lambda and the CDK infrastructure code into separate Maven modules. The intention being that once the Lambda code is compiled, packaged up into an uber JAR (its code plus all of its dependencies' code), the infrastructure module can refer to it as a dependency, passing the full path to the Lambda JAR to the App/Stack class to that it can use it as an asset.
Create a new property for the full path to your Lambda JAR.
In the properties section of your pom.xml, create a new property to refer to your Lambda JAR. Something like this:
<properties>
...
<lambda.jar>${GROUP_ID:ARTIFACT_ID:jar}</lambda.jar>
...
</properties>
Populate a property with the full path to your Lambda dependency's JAR, using the dependency plugin.
<build>
<plugins>
...
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>3.1.1</version>
<executions>
<execution>
<goals>
<goal>properties</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
</plugin>
...
<plugins>
</build>
This associates the properties goal with the process-resources phase. Whenever that phase of the build occurs, the property you've created previously will be populated with the full path to the JAR in your local repository.
Associate dependency and execution related goals into a single phase of the build.
When you create a new CDK Java project, it outputs a file called cdk.json, which points by default to the Maven exec:java goal. In order for your new lambda.jar property to be populated correctly, you need to associate the exec:java goal with the same phase as above.
<build>
<plugins>
...
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.6.0</version>
<executions>
<execution>
<goals>
<goal>java</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
<configuration>
<mainClass>com.myorg.TestingApp</mainClass>
</configuration>
</plugin>
...
</plugins>
</build>
In order for your code to get access to the JAR file that you've generated, you need to create a System property (I couldn't get environment variables to work) to your App class. Your pom.xml started with something like this:
Pass the full path via a system property to your CDK code.
In the configuration section (after mainClass), add a system property for your assets directory, something like this:
<systemProperties>
<systemProperty>
<key>lambda.jar</key>
<value>${lambda.jar}</value>
</systemProperty>
</systemProperties>
Update cdk.json to point to the the common phase you've used.
Your cdk.json of your CDK project should be changed to point to the process-resources phase. Once done it will look like this:
{
"app": "mvn package"
}
It will cause both the goals to be run in succession, and upon execution the path to your Lambda's JAR will be passed as a system property.
Access the property from your App/Stack code.
Finally, now that the system property is created, you can access it from your code by calling System.getProperty("lambda.jar"). Something like this:
final Code code = Code.fromAsset(System.getProperty("lambda.jar"));
You can then use the code reference wherever needed when defining your Lambda functions.
We use Jenkins which use md5 fingerprinting to identify artifacts and whether the artifact has changed since the last build. Unfortunately Maven builds always generate binary different artifacts.
Therefore I am looking into making Maven generate the same jar artifact for the same set of input files regardless of where and when they were built, which amongst other things mean that the entries in the jar file must be sorted - not only in the index, but in the order they are written to the jar file.
After examining maven-jar-plugin which use maven-assembly-plugin, my conclusions are that they do not collect all files to be written in memory before writing them all at once, but write one at a time. This mean that it may be better to postprocess the generated jar instead of changing the current behavior so I at that time can sort the entries, zero the timestamps, etc.
I am unfamiliar with writing Maven plugins, so my question is, how should I write a plugin which Maven knows how to tell where the artifact-jar-in-progress is located and how I hook it up in my pom.xml?
(At first I need this to work for jar files, but war files would be nice too).
As mentioned, this can be done based on something similar to maven-shade-plugin. I went ahead and wrote a simple plugin to add this capability -- see https://github.com/manouti/jar-timestamp-normalize-maven-plugin (available on the Central repo).
The behavior is based on the shade plugin. It consists of a single goal called normalize which can be bound to the package lifecycle phase and configured in the project's POM:
<plugins>
<plugin>
<groupId>com.github.manouti</groupId>
<artifactId>jar-timestamp-normalize-maven-plugin</artifactId>
<version>1.0-SNAPSHOT</version>
<executions>
<execution>
<id>jar-normalize</id>
<goals>
<goal>normalize</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
</plugin>
</plugins>
A few notes about the plugin:
The artifact under build is accessed via project#getArtifact() where project is a org.apache.maven.project.MavenProject.
Normalization consists of mainly three steps:
Setting the last modified time of all Jar entries to a specific timestamp (default value is 1970-01-01 00:00:00AM but can be changed via -Dtimestamp system property).
Reordering (alphabetically) of attributes in the manifest except for Manifest-Version which always comes first.
Removing comments from the pom.properties file which contain a timestamp that causes the Jar to differ from one build to another.
Once invoked, the goal will generate the output file next to the original artifact (named artifactId-version-normalized.jar), i.e. in the project.build.directory directory.
To create maven plugin project
mvn archetype:generate \
-DgroupId=sample.plugin \
-DartifactId=hello-maven-plugin \
-DarchetypeGroupId=org.apache.maven.archetypes \
-DarchetypeArtifactId=maven-archetype-plugin
invoke this command it will generate a skeleton project with a class called MyMojo.java
write your stuff inside execute() method, and install that plugin to your repository by mvn clean install
then attach its execution with your project, in your project pom.xml
<build>
<plugins>
<plugin>
<groupId>sample.plugin</groupId>
<artifactId>hello-maven-plugin</artifactId>
<version>1.0-SNAPSHOT</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>sayhi</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
to access project properties inside your Mojo
/**
* The Maven project.
*
* #parameter expression="${project}"
* #required
* #readonly
*/
private MavenProject project;
and then
project.getProperties("build.directory")
and read other properties to determine your jar file packed
See
maven: guide-java-plugin-development
I agree on creating a custom maven plugin seems like a better option. I dont know about an existing plugin provides solution for what you asked.
md5 checksum (or sha-1 in my repository) is generated with install plugin, so seems like you need to extend this or write a new plugin which works after install phase.
I have 2 suggestions about this plugin:
1) When thinking simple, this plugin should:
Read generated jar:
Extract all entries.
Exclude some entries (e.g. MANIFEST.MF).
Sort remaining entries .
Extract md5s for each in memory.
Generate a single md5 from all of those extracted.
However when considering about where & when independency: Accordig to .class file structure Java_class_file there is minor, major versions entries are held in compiled class files. So if compiler changes, .class files will be changed. In this case we need a check on source code level from this point :( So this solution become useless if there is no guarantee on copiler version.
2) As very dirty but easy solution, this plugin may only extract your module's pom.xml file's md5 code. But you must guarantee each change in your jar reflects to a minor version (or built number) manually.
Instead of writing your own plugin you can write a Groovy script that is executed by groovy-maven-plugin:
<plugin>
<groupId>org.codehaus.gmaven</groupId>
<artifactId>groovy-maven-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>execute</goal>
</goals>
<configuration>
<source>
import java.util.jar.*
String fileName = '${project.build.directory}/${project.build.finalName}.jar'
println "Editing file ${fileName}"
JarFile file = new JarFile(fileName);
// do your edit
</source>
</configuration>
</execution>
</executions>
</plugin>
I'm using a plugin in my pom that looks like this:
<plugin>
<groupId>org.codehaus.enunciate</groupId>
<artifactId>maven-enunciate-plugin</artifactId>
<!-- check for the latest version -->
<version>1.27</version>
<executions>
<execution>
<goals>
<goal>docs</goal>
</goals>
<configuration>
<docsDir>${project.build.directory}/docs</docsDir>
</configuration>
</execution>
</executions>
</plugin>
And I'm wondering, if I run package will this plugin run or is it during site or something else? Is there an easy way to tell by looking at this, or do I have to either
Read the plugin documentation
Experiment through trial and error
I'm hoping there's an easier way. I'm using intellij-idea, if that provides a means I'd be happy with that. Assuming I can't tell without one of these two methods, is it a best practice to always define the phase in the pom so that I can save myself and others time in the future?
You can let maven print out informations regarding the plugin using mavens help plugin - for enunciate simply use the following command:
mvn help:describe -Dplugin=org.codehaus.enunciate:maven-enunciate-plugin -Ddetail
It has actually 6 goals bound to different phases - goal docs will be bound to process-sources-phase
To only extract the goal you are interested in you can furthermore use the following command:
mvn help:describe -Dmojo=docs -DgroupId=org.codehaus.enunciate -DartifactId=maven-enunciate-plugin -Ddetail
You could also omit the -Ddetail part, but it won't give you then any information on the phase it is running.
I've been banging my head against a wall for about an hour on this: I'm trying to pass a simple property (java.library.path) to exec-maven-plugin. The goal is to have it integrate with Netbeans Right Click file > Run File procedure.
So I set my POM like this:
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.1.1</version>
<configuration>
<environmentVariables>
<java.library.path>native/win32-x86</java.library.path>
</environmentVariables>
</configuration>
</plugin>
</plugins>
</build>
(I use an old version so I can see the execution args, but its fully reproducible with 1.2)
Then I right click my file and click "Run File". Netbeans starts this process:
W:\programming\apache-maven-2.2.1\bin\mvn.bat -Dexec.classpathScope=runtime -Dexec.args=-classpath %classpath org.quackedcube.camera.CameraDemo -Dexec.executable=C:\Program Files\Java\jdk1.6.0_21\bin\java.exe -Dnetbeans.execution=true -Dmaven.repo.local=W:\programming\maven-repo process-classes exec:exec
(The original full classpath execution was changed to exec:exec so hopefully my configuration applied)
But my environment variable is apparently ignored, as the resulting executed program is:
Result of cmd.exe /X /C ""C:\Program Files\Java\jdk1.6.0_21\bin\java.exe" -classpath *snip* org.quackedcube.camera.CameraDemo" execution is: '1'.
I've tried
Using separate Key and Value tags inside an enviornmentVariable tag
Use a key and value tag directly inside an enviornmentVariables tag (worth a try)
binding to a phase
passing as a maven arg and using exec:java instead
Passing -Djava.library.path=native/win32-x86 as a Run argument and VM option in Project Configuration page
and all have failed. I'm really at a loss here.
I guess this is the disadvantage of using JNI in maven: You have to pass as an argument to your tests, your runtime, your module run POM, and your parent POM.
So my question: How can I pass a java.library.path property to an executed file? It would be nice if it integrated with Netbeans Run File functionality (therefor I don't have to change the class name in a POM, build, then run)
Didn't know this, but apparently when doing this you need to put this property first. I didn't think it was necessary since the classpath isn't immediately executed, but apparently it does make a difference.
To fix it, I simply changed this in Project Properties > Actions > Run File via Main
exec.classpathScope=${classPathScope}
exec.args=-Djava.library.path="native/win32-x86" -classpath %classpath ${packageClassName}
exec.executable=java
The reason you can't specifcy it in the POM is that NB passes the classpath and what its execution via command line exec.args, which overrides whats in your POM.
While this might be ugly and platform dependant, its what happens when you mix JNI and Maven. There isn't really another way that I can see.
Not sure if you tried this but as long as you need to set property on a level of JVM it should be done with -Djava.library.path=/some/path
So in order to specify it for exec-maven-plugin you could write something like this:
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.1</version>
<configuration>
<executable>java</executable>
<arguments>
<argument>-Djava.library.path=${java.library.path}</argument>
</arguments>
</configuration>
</plugin>
You need, of course, to update the executable and maybe add another attributes.
I'm working on a project which involves maven, java and clojure. The problem I'm facing is this, I have some UTF-8 chars in my clojure source files because of which my source code is not interpreted correctly by the java compiler, I kinda got it working by setting the environment variable JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8, but what I want is to pass this property through MAVEN.
I have already tried setting MAVEN_OPTS=-Dfile.encoding but this doesn't seem to work.
I have also tried setting configuration for the compiler plugin of maven... something like this:
<configuration>
<compilerArgument>-Dfile.encoding=UTF8</compilerArgument>
</configuration>
This doesn't work either.
I'm I doing something wrong, or is there another way.
thanks,
RD
Ok, Here's some more detail.
This is my parent pom,
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.5</source>
<target>1.5</target>
<encoding>UTF-8</encoding> <! also tried <encoding>UTF8</encoding>
</configuration>
</plugin>
Nothing interesting in the child's pom except...
<resources>
<resource>
<directory>src/main/clojure</directory>
</resource>
</resources>
;; clojure code snippet which causes problems
(let [char "대"]
(not (empty? (filter #(s/contains? % char) <some-list>)))
;; The list is always empty because I never find a match if I do not set the env. variable
Did you try passing compiler options? [-encoding UTF-8]
Update: Based on your comments, this is a runtime, not a compile issue. As a workaround, you could try escaping the character as unicode.
i.e. change the character to '\uXXXX' in the clojure file, where XXXX is the Unicode point in hexadecimal.
If your problem is happening in your unit tests. You can configure the surefire plugin by setting the argLine property. This allows you to set arbitrary JVM options on the command line.
Did you set the parameter through Compiler Plugin like this?
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<compilerArgument>-Dfile.encoding=UTF8</compilerArgument>
</configuration>
</plugin>
</plugins>
Given the fact that Clojure actually hardcodes the expected encoding of input files to UTF8 (see src/jvm/clojure/lang/Compiler.java, loadFile-method), I'm surprised that using file.encoding does have any effect at all.
Try adding this property to your pom
UTF-8
For me, this code works without problem in cider REPL in Emacs.
;; returns sequence ("대")
(filter #(= % "대") ["대" "한" "민" "국"])
Can you provide code which emits error?