Develop Reference

Jenkinsfile syntax highlighting in Java project using IntelliJ IDEA

We already tried the approaches as listed below:
https://github.com/oliverlockwood/jenkinsfile-idea-plugin
https://st-g.de/2016/08/jenkins-pipeline-autocompletion-in-intellij
After having searched the web for many hours on multiple days, we still haven't found a helpful resource on this matter. Thus, it appears to make sense to ask a new question here.
We are developing our Java projects in IntelliJ idea and want to integrate our builds with Jenkins. When we create a Jenkinsfile in Idea, we do not get syntax highlighting or auto completion. Since we are new to Jenkins, those features would be really useful to us. How can we make Idea be more supportive with Jenkinsfiles?
If there is no way to get syntax highlighting and auto completion for a Jenkinsfile in IntelliJ IDEA, what other editors would be helpful?
Please note:
we are working with Java projects, not Groovy projects.
We've already tried the plugin https://github.com/oliverlockwood/jenkinsfile-idea-plugin. When the plugin is activated, the Jenkinsfile is recognized as such, but instead of syntax highlighting we get an error message, please see below.
pipeline {
agent { docker 'maven:3.3.3' }
stages {
stage('build') {
steps {
sh 'echo Hello, World!'
}
}
}
}
IntelliJ IDEA highlights the p of pipeline as error. The error message reads:
JenkinsTokenType.COMMENT, JenkinsTokenType.CRLF or
JenkinsTokenType.STEP_KEY expected, got 'p'
Thanks for any help!

If you want IDEA to recognize a Jenkinsfile as a Groovy file, then you can add the String "Jenkinsfile" as a valid file name pattern (normally contains file endings) for Groovy files. This is supported "out of the box" without requiring any additional Plugin (except the "Groovy" Plugin, but that is already part of IDEA).
To do that go to the settings menu, open the "Editor" item and then "File Types". Now select "Groovy" in the upper list and add "Jenkinsfile". You can also use a regex like "Jenkinsfile*" if you want to be more flexible regarding an optional file ending for the Jenkinsfile.
The setting should now look like this:
Your example now looks like this in IDEA (with the Dracula theme):
So IDEA now provides syntax highlighting and auto completion as far as I can tell. It suggests existing function/method names while writing, but I'm not a Groovy developer, thus I can't tell if some suggestions are missing.

At long last we found a solution that works for us and provides syntax highlighting and code completion for the Jenkinsfile present in an otherwise normal Java project in Idea.
The solution is taken from here, here (and from additional personal experiments / research)
Download the Groovy SDK (if you did not do so already) from the Groovy Page and configure it on your Java project. For help on this see here
Download the pipeline GDSL file from your Jenkins instance which should be available under a link like https://yourJenkinsInstance.tld/pipeline-syntax/gdsl, and add it to the classpath of your Java project. E.g. by creating a new folder src/main/jenkins, putting the pipeline gdsl file there and marking the folder as source root in IntelliJ Idea
Add "Jenkinsfile" as a valid file name pattern for groovy files as described here
To avoid the error message 'node' cannot be applied to '(groovy.lang.Closure<java.lang.Object>), you can add this line at the top of your Jenkinsfile:
// noinspection GroovyAssignabilityCheck

If you add
#!groovy​
header to your jenkinsfile then you should get groovy syntax highlighting in IDE.

Another option is to use a shabang on top of the Jenkinsfile like this #!/usr/bin/env groovy. Also you can try out gdsl: https://st-g.de/2016/08/jenkins-pipeline-autocompletion-in-intellij
but so far it doesn't support declarative pipelines: https://issues.jenkins-ci.org/browse/JENKINS-40127

Looking at the source code, it looks like COMMENTS are not defined (they are commented out in the code)
The STEP_KEY is defined as: STEP_NAME="sh" | "parallel"
I'm not sure the plugin does much more and it hasn't been updated in 2 years.

go to the settings menu, open the "Editor"--> "File Types". Now select "Groovy" in the upper list and add ".Jenkinsfile". You can also use a regex like ".Jenkinsfile" if you want to be more flexible regarding an optional file ending for the Jenkinsfile.

Jenkinsfile is a groovy like script, but normally IntelliJ map it to TextMate editor.
To do that go to the settings menu, open the "Editor" item and then "File Types". Now select "TextMate" in the upper list and add "Jenkinsfile". You can also use a regex like "Jenkinsfile*" if you want to be more flexible regarding an optional file ending for the Jenkinsfile.

Use sh like this and the error should go away (worked for me)...
steps {
sh """
echo 'Hello, World!'
"""
}

Maven and Intellij builds differently

I have a parent pom.
When I am on the parent and write mvn clean install my dropwizard application builds fine.
However, in Intellij it fails because it cannot find my config.yml file.
The problem is that I need to include my module directory in Intellij, but not in Maven. Why is that?
Running mvn install this works
public static void main(String[] args) throws Exception {
new CivilizationApplication().run(new String[]{"server", "src/main/resources/config.yml"});
}
In Intellij I must change to the following
public static void main(String[] args) throws Exception {
new CivilizationApplication().run(new String[]{"server", "civilization-rest/src/main/resources/config.yml"});
}
How can I configure Intellij, so it will work without specifying the module directory?

You could try setting your working directory in your run configuration...
I would also look into your pom.xml configuration to see if you can change the working source sets to use this directory upon importing the maven project.

Set Working directory to $MODULE_DIR$. Maven always uses $MODULE_DIR$, but you probably do not have $MODULE_DIR$ by default for IntelliJ Application executions.

You can just tell Intellij to do exactly the same thing Maven does when you start it.
You can do so by clicking on the drop down menu next to the "Launch App" button. There you can define a new Run configuration that uses Maven by clicking on the + to the left and then choosing Maven.
Then you can just enter clean install exec:java where it says "Command line" and it should work exactly like it does when you execute it manually from the command line. Debugging should work like this as well without any problem.

The problem here is that you are refering to a resource file by using a relative path instead of using the classpath.
So, instead of
new CivilizationApplication().run(new String[]{"server", "src/main/resources/config.yml"});
You better have this
new CivilizationApplication().run("server", getClass().getResourceAsStream("config.yml"));
(obviously, make sure to change the signature of the #run() method as well)

Just a thought but maybe it's to do with the Resource not being copied.
I'm on OSX so the exact terms may differ if you're on Windows but for me... in Preferences, select "Build, Execution, Deployment", then "Compiler", and try adding ";?*.yml" to the Resource patterns.

How to execute a specific plugin/Mojo from a pom.xml programmatically?

I'm the author of one of the Maven plugins (not Apache/Codehaus, completely indie). Sometimes I get support requests or test cases where I'd really need to debug the execution of my plugin with an existing pom.xml. Basically the test cases I get are sample/test project (pom.xml with src/main/resoures, src/main/java and so on).
What I need is a way to:
Load an existing pom.xml.
Find a specific execution of my plugin there (usually it's the only one).
Get an instance of MyMojo - fully initialized/condigured, with all the components and parameters corectly injected.
Execute MyMojo.
What's important is that test projects are separate projects, I don't want to copy them into the Maven module of my plugin.
I'd like to be able to do this without remote debugging.
By debugging I mean to be able to set and halt on breakpoints (also conditional), step in/out/over on the source code.
Ideally I'd like to be able to executeMyMojoFrom(new File("pom.xml")) - for instance in a JUnit test or a main method of some class. (I can supply groupId, artifactId etc. All other definitions should just be loaded from that pom.xml.)
How can I achieve this?
What I've tried so far:
Debug As... on pom.xml in Eclipse - does not work well enough (source code not found, breakpoint don't work as its not a Java project context)
Maven Embedder/Invoker solutions - spawn things in separate processes via CLI. Forget breakpoints, no debugging.
Remote debugging with mvnDebug and then remote debugging from Eclipse as suggested by Pascal Thivent here. This is so far the best option. However, remote debugging means starting mvnDebug separately, and there's also not guarantee that the JARs I have in Eclipse are exactly the same that mvnDebug is using. So there's a certain distance here.
maven-plugin-testing-harness - I actually thought this this will do the task. But first I was jumping through hoops for a few hours just to make it start. All of the important dependencies are "provided" so I first had to figure out the right combination of versions of these artifacts. And then - only to discover that AbstractMojoTestCase only works within the plugin module you want to test. Probably I was mistaken when I thought that maven-plugin-testing-harness was a testing harness for Maven plugins. It seems that it's a testing harness for the plugin from that plugin's module. Which is not illogical but does not help my case. I'd like to test my plugin in other modules.
So right now I've got the best results with the remote debugging solution. But what I'm looking for is really something like maven-plugin-testing-harness but not hardwired to the plugin module. Does anyone happen to have a hint, if such a method exists somewhere in Maven artifacts?
To be even more specific, I'd like to write something like:
public void testSomething()
throws Exception
{
File pom = getTestFile( "pom.xml" );
assertNotNull( pom );
assertTrue( pom.exists() );
MyMojo myMojo = (MyMojo) lookupMojo( "myGroupId", "myArtifactid", ...,
"myGoal", pom );
assertNotNull( myMojo );
myMojo.execute();
...
}
Compare it to the MyMojoTest here - it's almost there. Should just not be hardwired into the mymojo Maven module (as it is in maven-plugin-testing-harness).
Update
Few answers to the questions in comments:
You mean you don't want such a test class, i.e MyMojoTest to reside inside the same project as the MyMojo, i.e your plugin project? Why is that?
Exactly. I want to debug the plugin execution in an existing Maven project, I don't want to move that project to my plugin project first to be able to run a test. I want to be able to test/debug an existing project. Ideally, I'd just need to add my-maven-plugin-testing dependency and subclass MyMojoTest in the project's src/test/jaca. This would be a good instrument to debug executions. Dragging the target project into my Mojo project ist just too much overhead - and mostly these aren't really the test cases I want to keep long-term. I hope, this answers, why.
Anyway, it's merely a convention to keep the project-to-test/pom.xml inside the src/test/resources of your plugin module, not a rule...
My problem is not the location of the pom.xml of the project-to-test, that is easily configurable. My difficulty is that maven-plugin-testing-harness is is somehow hardcoded to be in the Mojo's project. It uses the pom.xml of the Mojo, looks for other special files/descriptors in the containing project. So I somehow can't use it in a non-Mojo project, or can I? This is my question.
And I'm not sure why Debug as... didn't help you...
Not sure either, but (1) breakpoints did not work and (2) the source code was not "attached" for some reason.

If the Debug as didn't work for you as well as it should, you can try to use the mojo-executor with a bit of work.
https://github.com/TimMoore/mojo-executor
This is how you would execute the copy-dependencies goal of the Maven Dependency Plugin programmatically:
executeMojo(
plugin(
groupId("org.apache.maven.plugins"),
artifactId("maven-dependency-plugin"),
version("2.0")
),
goal("copy-dependencies"),
configuration(
element(name("outputDirectory"), "${project.build.directory}/foo")
),
executionEnvironment(
mavenProject,
mavenSession,
pluginManager
)
);
The project, session, and pluginManager variables should be injected via the normal Mojo injection. Yes, that means this should be executed from the context of another maven plugin. Now that I think about it, whether this would help you in any way is still a question because this still relies on injection of such components by the underlying plexus container.
My original idea was though to have you build a maven plugin that would invoke your jaxb2 plugin thru the mojo-executor like above, then serialize the mavenProject, mavenSession, pluginManager, i.e, all the plexus injected components and then use those objects to invoke your jaxb2 plugin in future from a standalone class without the plugin that you built.

Maven - parsing command line arguments?

Can anyone think of a way to do this? At the request of our build engineers, I have parameterized a Maven build of mine like this:
mvn clean install -Dmyprop1=kimi -Dmyprop2=seb -Dmyprop3=alonso
These line up with some properties I've defined in my pom, like this:
<properties>
<myprop1>default value for prop1</myprop1>
<myprop2>default value for prop2</myprop3>
<myprop3>default value for prop3</myprop4>
</properties>
In my pom, I can then of course refer to these properties anywhere I'd like - for exampe:
${myprop2}
AFAIK, this is all pretty standard Maven stuff for handling custom command line args, but here is where my question comes in - what our build engineers would actually like to see is something more along the lines of:
mvn clean install -Dmyprops=prop1>kimi,prop2>seb,prop3>alonso
In other words, a single command line argument that they could stuff all of the properties into (using some sort of delimiting scheme, like ">" and "," for instance as I've shown above) that I would then be expected to parse within my pom accordingly.
I've done a fair amount of research and I'm really not seeing anything that would allow me to do this & I'm inclined to push back and tell our build engineers they need to just update their machinery to handle multiple command line args (1 per property, as would be standard).

Anyone heard about a standard property deployment.name?

I've just fought for a whole day with a strange maven problem:
I had a custom property called "deployment.name" that was never to resolved to what I configured for it, but rather the maven filtering mechanism always replaced it by the project's name.
I tried the goal "help:expressions" to find out whether this is a preconfigured property, but that goal only throws exceptions in m2eclipse. Google does not seem to know a pre-configured
property by that name.
The strangest bit: deployment.somethingelse works perfectly fine, so I ended up replacing ".name" with ".depname", then it works ;.)

The Maven Super POM defines the common configuration for all Maven projects. The values in that are accessible as properties (and , so that is where most of the properties you generally use come from (e.g. ${project.build.directory}), these are the pretty much the same as the output of help:expressions.
There is no deployment section in the super POM. The only thing I can think of is that the property is being set somewhere else, e.g. in a profile, or overridden by a plugin (though that seems unlikely). You could try running mvn help:effective-pom to see if the property is being set by a profile.
Are you able to post your POM? that might help diagnose it.

... I just ran help:effective-pom, and there is no trace of "deployment.name" in the output.
I can see all the other properties that I defined though (e.g. "deployment.depname").
Maybe "name" is a reserved attribute of some sort? Maybe debugging into m2eclipse will shed light on this riddle.