I have a azure pipeline that runs in the cloud and in there, I basically want to run a Blackduck scan. To make it easy, the blackduck task runs a maven command to build the dependency tree. In order to do so, it scans all the dependencies in my pom.xml.
I currently have 2 feeds in Azure DevOps. 1 feed that store external libraries and 1 to store internal libraries developed in house. All the dependencies that are external, the maven command is able to retrieve but not the internal one. I keep getting a 401 Unauthorized error.
I have updated my pom.xml to include the credentials to connect to azure artifact as shown here. I have also updated the settings.xml file to include the connection as well. Before the maven command is run, I have added the Maven authenticate task
Yml pipeline:
- task: PowerShell#2
displayName: "Modify Settings.xml"
inputs:
targetType: 'inline'
script: |
$xmlPom = [xml]"<server>
<id>Pack-All</id>
<username>Bob</username>
<password>$(ado.pat)</password>
</server>"
$file = "settings.xml"
$origin = "$(M2_HOME)\conf"
$xdoc = new-object System.Xml.XmlDocument
$fileXml = resolve-path(“$origin\$file”)
$xdoc.load($fileXml)
$xdoc.settings.servers.AppendChild($xdoc.ImportNode($xmlPom.server, $true))
$xdoc = [xml] $xdoc.OuterXml.Replace(" xmlns=`"`"", "")
$xdoc.Save(“$origin\$file”)
- task: MavenAuthenticate#0
displayName: 'Maven Authenticate'
inputs:
artifactsFeeds: 'Pack-All'
- task: SynopsysDetectTask#2
displayName: "Run Black Duck analysis"
condition: and(succeeded(), eq('${{ parameters.blackduck }}', 'true'))
continueOnError: true
inputs:
Products: 'BD'
BlackDuckService: 'Black Duck'
DetectVersion: 'latest'
DetectArguments: '--detect.project.name=$(Build.Repository.Name)Test --detect.binary.scan.file.path=$(Build.SourcesDirectory)\app.war --detect.maven.build.command=-DmavenFeedAuthenticate=true'
Maven command that the Blackduck task executes:
C:\ProgramData\chocolatey\lib\maven\apache-maven-3.6.3\bin\mvn.cmd -DmavenFeedAuthenticate=true dependency:tree -T1
Error Log:
[ERROR] Failed to execute goal on project dimload-ms-app-agg: Could not resolve dependencies for project ca.test-ms-app-agg:war:0.0.1-SNAPSHOT: Failed to collect dependencies at ca.cn.boot:helpers:jar:0.4.28950: Failed to read artifact descriptor for ca.test.boot:helpers:jar:0.4.28950: Could not transfer artifact ca.test.boot:helpers:pom:0.4.28950 from/to Test-All (https://pkgs.dev.azure.com/Test-Int/_packaging/Pack-All/maven/v1): Authentication failed for https://pkgs.dev.azure.com/Test-Int/_packaging/Pack-All/maven/v1/ca/test/boot/helpers/0.4.28950/helpers-0.4.28950.pom 401 Unauthorized -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
From your Yaml Sample, you have added the MavenAuthenticate task. This task will automatically generate a settings.xml for the target feed.
This file exists in the path xxx\.m2\settings.xml
So you don't need to add this file(settings.xml)manually. This can also help you simplify your code.
Authentication failed for https://pkgs.dev.azure.com/.... 401 Unauthorized
The possible cause of this issue is that the build service account does not have sufficient permissions for the feed.
You could try the following steps:
Navigate to Artifacts ->Target Feed ->Feed Settings -> Permission.
Set the Project Build Service(ProjectName Build Service(OrganizationName)) As Contributor role within the target feed.
Or you could enable the option Allow Project-Scoped Builds.
On the other hand, from the feed URL, it seems to be an Organization-Scope feed.
You could check the Limit job authorization scope to current project for non-release pipelines option is Enabled in Project Settings -> Pipelines.
You could try to disable the option.
Note: To disable this option, you need to disable the option in Organization Settings-> Settings first. Then you could disable the option in Project level.
I run the command below with GitLab CI / CD:
mvn -s /etc/gitlab-runner/maven/settings-maven.xml --batch-mode test sonar:sonar
-Dsonar.java.source=1.8
-Dsonar.host.url=${SONARHOST}
-Dsonar.projectKey=${SONARKEY}
-Dsonar.projectName=${SONARNAME}
-Dsonar.projectVersion=${SONARVERSION}
-Dsonar.sources=myproject_core/src/main/java,myproject_service/src/main/java,myproject_persistance/src/main/java
-Dsonar.language=java
-Dsonar.java.binaries=myproject_core/target/classes,myproject_service/target/classes,myproject_persistance/target/classes
-Dsonar.java.libraries=**/*.jar
-Dsonar.java.coveragePlugin=jacoco
-Dsonar.surefire.reportPaths=myproject_core/target/surefire-reports,myproject_service/target/surefire-reports,myproject_persistance/target/surefire-reports
-Dsonar.junit.reportPaths=myproject_core/target/surefire-reports,myproject_persistance/target/surefire-reports,myproject_service/target/surefire-reports
-Dsonar.verbose=true
-Dsonar.binaries=myproject_core/target/classes,myproject_service/target/classes,myproject_persistance/target/classes
-Dsonar.jacoco.reportPaths=myproject_core/target/jacoco.exec,myproject_service/target/jacoco.exec,myproject_persistance/target/jacoco.exec
-Dsonar.exclusions=**/*Test*/**
-Dsonar.tests=myproject_core/src/test,myproject_service/src/test,myproject_persistance/src/test
So, I want to analyse 3 project core, service and persistance (I have 7 modules in my project)
myproject-parent
myproject-core
myproject-persistance
myproject-service
myproject- ...
But, I have this error :
[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.7.0.1746:sonar (default-cli) on project myproject-parent: The directory '/builds/git-myproject/myproject-parent/**myproject_persistance/myproject_core**/src/main/java' does not exist for Maven module fr.xxx.xxx.myproject:myproject_persistance:jar:1.1. Please check the property sonar.sources -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.7.0.1746:sonar (default-cli) on project myproject-parent: The directory '/builds/git-myproject/myproject-parent/myproject_persistance/myproject_core/src/main/java' does not exist for Maven module fr.xxx.xxx.myproject:myproject_persistance:jar:1.1. Please check the property sonar.sources
On this sonarqube error, i can see that in the path, module persistance is include on module core, but this is not the case
I'm looking for why but I can't find. Have you ever had this problem ?
We built an app with 0.18.0 quarkus version and we have following application.properties:
quarkus.datasource.url=${POSTGRES_DATABASE_URL}
quarkus.datasource.driver=org.postgresql.Driver
quarkus.hibernate-orm.database.generation=update
quarkus.hibernate-orm.dialect=org.hibernate.dialect.PostgreSQL10Dialect
quarkus.http.port=${PORT:8080}
%dev.quarkus.datasource.url=jdbc:h2:mem:db
%dev.quarkus.datasource.driver=org.h2.Driver
%dev.quarkus.hibernate-orm.dialect=org.hibernate.dialect.H2Dialect
When I tried upgrading to 0.20.0 (or any version higher, including 1.0.0.Final) I get following error when building with maven:
[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:0.20.0:build (default) on project thats-my-spot: Failed to build a runnable JAR: Failed to build a runner jar: Failed to augment application classes: For input string: "${PORT:8080}" -> [Help 1]
Why?
${PORT:8080} just means that it should take either the PORT environment variable if available or use 8080 if not.
I looked at potential issues, changes in semantics but found only one bug that might touch defaults in properties: https://github.com/quarkusio/quarkus/issues/3030 But the description and solution is different, so I don't think that it broke my app.
Can you try with 1.2.1.Final? We had some issues with handling defaults of environment properties at some point but it has been solved for quite a while.
If you can reproduce it with the latest, please open an issue in our tracker with a reproducer and we will have a look.
I was doing some testing using Maven and realized that I can execute the findbugs goal of the Findbugs plugin without adding the plugin to the POM file. On the other hand, when I needed to run the run goal of the Jetty plugin, I was forced to add the plugin to the POM file or the build failed.
Why Jetty needed configuration in the POM while Findbugs didn't?
How does Maven know which Findbugs to execute (suppose we have to plugins with the same name but different group id)?
When I run the first command the build is successful without any changes in POM file:
mvn findbugs:findbugs
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building module-mytest 1.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- findbugs-maven-plugin:3.0.4:findbugs (default-cli) # module-mytest ---
[INFO] Fork Value is true
[java] Warnings generated: 6
[INFO] Done FindBugs Analysis....
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 24.165s
[INFO] Finished at: Sun Oct 23 18:40:26 WEST 2016
[INFO] Final Memory: 21M/111M
[INFO] -----------------------------------------------------------------------
But when I run the second I get this:
mvn jetty:run
[INFO] Scanning for projects...
Downloading: http://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml (13 KB at 30.0 KB/sec)
Downloaded: http://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml (20 KB at 41.0 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.129s
[INFO] Finished at: Sun Oct 23 18:43:27 WEST 2016
[INFO] Final Memory: 12M/104M
[INFO] ------------------------------------------------------------------------
[ERROR] No plugin found for prefix 'jetty' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/home/hp-pc/.m2/repository), central (http://repo.maven.apache.org/maven2)] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoPluginFoundForPrefixException
So in order to pass the build I needed to add the following to the pom file:
<plugin>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-maven-plugin</artifactId>
<version>9.2.11.v20150529</version>
</plugin>
What is a prefix and why do we need it?
You just encountered the Plugin Prefix Resolution of Maven. This is a feature which enables the user to invoke goals of a specific Maven plugin, by using its prefix. When you invoke directly a goal on the command-line, you could use the fully-featured form of:
mvn my.plugin.groupId:foo-maven-plugin:1.0.0:bar
This would invoke the goal bar of the Foo Maven plugin having the coordinates my.plugin.groupId:foo-maven-plugin:1.0.0 (in the form of groupId:artifactId:version). It works well, but it is a bit verbose. It would be nice to invoke this goal in a simpler manner, without specifying all those coordinates. Maven makes this possible by assigning prefixes to plugins, so that you can refer to this prefix instead of the whole coordinates, with:
mvn foo:bar
^^^ ^^^
| |
prefix |
|
goal
How is this prefix determined?
You can define a prefix for each Maven plugin. This corresponds to a simple name used to identify it:
The conventional artifact ID formats to use are:
maven-${prefix}-plugin - for official plugins maintained by the Apache Maven team itself (you must not use this naming pattern for your plugin, see this note for more informations)
${prefix}-maven-plugin - for plugins from other sources
If your plugin's artifactId fits this pattern, Maven will automatically map your plugin to the correct prefix in the metadata stored within your plugin's groupId path on the repository.
Put another way, if your plugin's artifact id is named foo-maven-plugin, Maven will automatically assign it a prefix of foo. If you don't want this default assignment, you can still configure your own with the help of the maven-plugin-plugin and its goalPrefix parameter.
How does Maven map prefixes to plugins?
In the command
mvn foo:bar
Maven must have a way to deduce that foo actually means my.plugin.groupId:foo-maven-plugin. In the settings.xml file, you can add plugin groups, in the form of:
<pluginGroups>
<pluginGroup>org.mortbay.jetty</pluginGroup>
</pluginGroups>
What this does, is telling Maven which group id it is supposed to consider when you're using a prefix in a command. By default, and in addition to the groups specified in the settings, Maven also searches the group ids org.apache.maven.plugins and org.codehaus.mojo. It searches those default ones after the ones you configured in the settings. Therefore, with the configuration above, and a command of mvn foo:bar, Maven will look for a plugin having a prefix of foo inside the group id org.mortbay.jetty, org.apache.maven.plugins and org.codehaus.mojo.
The second step is how that search is actually performed. Maven will download metadata files (or look them into your local repository if they are already downloaded), called maven-metadata.xml, from each remote repositories at those group ids. If we take the example where the only remote repository we have is Maven Central, Maven will first download http://repo1.maven.org/maven2/org/mortbay/jetty/maven-metadata.xml, and look inside this file if we have something mapping foo. Notice how the group id was transformed into a directory structure in the remote repository. The structure of this metadata file is:
<metadata>
<plugins>
<plugin>
<name>Some Awesome Maven Plugin</name>
<prefix>somePrefix</prefix>
<artifactId>some-maven-plugin</artifactId>
</plugin>
</plugins>
</metadata>
If none of the <plugin> section contain a <prefix> that is equal to the one we specified (foo), Maven will continue with the next group id, hitting http://repo1.maven.org/maven2/org/codehaus/mojo/maven-metadata.xml. Again, if none are found, Maven will finally hit http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-metadata.xml (notice the Downloading: logs in your mvn jetty:run command, exactly fetching those last two files). If none are still found, there is nothing Maven can do for you anymore, and it will error:
[ERROR] No plugin found for prefix 'foo' in the current project and in the plugin groups [org.mortbay.jetty, org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (.../.m2/repository), central (http://repo.maven.apache.org/maven2)] -> [Help 1]
This is the error you have here. However, if one match was made during this search, then Maven can deduce the <artifactId> to use.
It now means it has the group id, and the artifact id. The final piece of the puzzle is the version
Which version is going to be used?
Maven will take the latest one available, unless explicitly configured in the POM (see next section). All possible versions are retrieved by fetching another metadata file, still called maven-metadata.xml, but this time living alongside the artifact id folder in the repository (contrary to the ones above, where it was alongside the group id). Taking the example of the Maven Clean plugin (whose group id and artifact id would be found with the above mechanism and a command of mvn clean:clean), the maven-metadata.xml looks like:
<metadata>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-clean-plugin</artifactId>
<versioning>
<latest>3.0.0</latest>
<release>3.0.0</release>
<versions>
<version>2.0-beta-1</version>
<version>2.0-rc1</version>
<version>2.0</version>
<version>2.1</version>
<!-- more versions -->
<version>3.0.0</version>
</versions>
<lastUpdated>20151022205339</lastUpdated>
</versioning>
</metadata>
Maven will select as version the <release> version, which represents the latest release version of the plugin. If that tag isn't there, it will select <latest> which represent the latest version of the plugin, release or snapshot. It can happen that both tags are not there, in which case, Maven will select the first release, or the first snapshot for lack of a release, of the list of <version> elements.
If that still fails, there is nothing Maven can do for you anymore, the version couldn't be deduced and it errors. This isn't very likely to happen though. We now have gathered the group id, the artifact id, and the version; time to finally invoke the bar goal of our plugin.
What's the issue with my configuration?
As said above, Maven looks in certain pre-defined group ids inside the active remote repositories to look for matches with a given prefix. With the command
mvn findbugs:findbugs
Maven starts the search with the findbugs prefix. Since our configuration does not have any <pluginGroup> in our settings, Maven looks into org.codehaus.mojo and org.apache.maven.plugins group id for a prefix match.
And it does find one: Findbugs Maven Plugin is published under the org.codehaus.mojo group id; indeed, you can find it in the maven-metadata.xml:
<plugin>
<name>FindBugs Maven Plugin</name>
<prefix>findbugs</prefix>
<artifactId>findbugs-maven-plugin</artifactId>
</plugin>
And you can also find the version that is going to be used by peeking into the maven-metadata.xml file under the findbugs-maven-plugin just deduced (3.0.4 at the time of this writing; and notice how it exactly matches the version in the mvn findbugs:findbugs logs of your question). So the resolution succeeded, and then Maven can continue to invoke the findbugs goal of this plugin.
The second example is the command
mvn jetty:run
As before, the same resolution steps happen, but, in this case, you'll find out that the prefix <jetty> does not appear in any of the maven-metadata.xml for the group ids org.codehaus.mojo and org.apache.maven.plugins. So the resolution fails, and Maven returns the error that you have.
But we've seen how to make it work! We can add a <pluginGroup> inside our settings, so that this group id can also be searched during resolution. The Jetty Maven Plugin is published under the group id org.eclipse.jetty, and if we peek into the corresponding maven-metadata.xml in Maven Central, you'll see that <prefix>jetty</prefix> is there. So the fix is simple: just define this new group id to search inside the settings:
<pluginGroups>
<pluginGroup>org.eclipse.jetty</pluginGroup>
</pluginGroups>
Now, Maven will also look into this group id, and match the jetty prefix to the org.eclipse.jetty:jetty-maven-plugin successfully.
How can I use a specific version? Or, I don't want to modify my settings!
Of course, all of this resolution can be side-tracked if you define the plugin explicitly in your POM, which is the other solution you found:
<plugin>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-maven-plugin</artifactId>
<version>9.2.11.v20150529</version>
</plugin>
and use
mvn jetty:run
If you configure the plugin directly in the POM, the prefix resolution still happens, but it is a bit masked: Maven will download the plugin from the configured remote repositories, and will download and install all metadata files along the way, including the maven-metadata.xml containing the mapping for the prefix jetty. So since it downloads it automatically, the search always succeeds.
Note also that since the plugin was defined in the POM, you wouldn't need any <pluginGroup> in the settings: the group id was written in the POM. Furthermore, it makes sure that the version 9.2.11.v20150529 is going to be used, instead of the latest.
mvn -version
mvn archetype:create \
> -DgroupId=com.tobilko \
> -DartifactId=site \
> -DarchetypeArtifactId=maven-archetype-webapp
I'm going to create a project with Maven using a maven-archetype-webapp archetype. Why can't I do it? I've just started learning Maven so the error doesn't tells me anything:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-
plugin:2.3:create (default-cli) on project standalone-pom: Unable to
parse configuration of mojo org.apache.maven.plugins:maven-archetype
plugin:2.3:create for parameter #: Abstract class or interface
'org.apache.maven.artifact.repository.ArtifactRepository' cannot be
instantiated: org.apache.maven.artifact.repository.ArtifactRepository.
<init>() -> [Help 1]
I don't want to have any other archetypes or a default archetype.
Dont use create use generate . Something like this
mvn archetype:generate
-DgroupId=com.tobilko
-DartifactId=site
-DarchetypeArtifactId=maven-archetype-webapp
archetype:create is deprecated
instead use
archetype:generate
See this link for the reference since you have started learning maven