I am in the process of setting up Jenkins pipeline builds and am starting to use the same methods across multiple jobs, so it's time to put these common methods into a shared library.
The first function I have created is to update GitHub with the result of some unit tests. I am having an issue where I can run this function from the command line fine but when it comes to using it within my Jenkins build it does not work and I cannot seem to get any debug output in the Jenkins console
This is the directory structure of my shared library
my-project
src
vars
- getCommitId.groovy
- gitUpdateStatus.groovy
So the first function getCommitId works fine
#!/usr/bin/env groovy
def call() {
commit_id = sh script: 'git rev-parse HEAD', returnStdout: true
commit_id = commit_id.replaceAll("\\s","") // Remove Whitespace
return commit_id
}
This returns the correct value
This is gitUpdateStatus
#!/usr/bin/env groovy
#Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')
import static groovyx.net.http.ContentType.JSON
import static groovyx.net.http.Method.POST
import groovyx.net.http.HTTPBuilder
String targetUrl = 'https://api.github.com/repos/myRepo/'
def http = new HTTPBuilder(targetUrl)
http.request(POST) {
uri.path = "repo/statuses/12345678"
requestContentType = JSON
body = [state: 'success', description: 'Jenkins Unit Tests', target_url: 'http://test.co.uk', context: 'unit tests']
headers.'Authorization' = "token myOauthTokenHere"
headers.'User-Agent' = 'Jenkins Status Update'
headers.Accept = 'application/json'
response.success = { resp, json ->
println "GitHub updated successfully! ${resp.status}"
}
response.failure = { resp, json ->
println "GitHub update Failure! ${resp.status} " + json.message
}
}
I can run this fine via the command line, but I get no output when run as a Jenkins build.
My Jenkinsfile
#Library('echo-jenkins-shared')_
node {
GIT_COMMIT_ID = getGitCommitId()
echo "GIT COMMIT ID: ${GIT_COMMIT_ID}"
gitUpdateStatus(GIT_COMMIT_ID)
}
Why would this not work or could this be converted to just use native Groovy methods?
First off, I would advise you to use as service like https://requestb.in to check if your code actually perform HTTP calls.
Second, I would recommend NOT to use #Grab-based dependencies like HTTPBuilder in Jenkins pipelines, but the http_request plugin instead, downloadable & installable as a .hpi:
https://jenkins.io/doc/pipeline/steps/http_request/
Finally, you can find an example of utility class to perform HTTP requests here:
https://github.com/voyages-sncf-technologies/hesperides-jenkins-lib/blob/master/src/com/vsct/dt/hesperides/jenkins/pipelines/http/HTTPBuilderRequester.groovy
With the rationale behind it explained there: https://github.com/voyages-sncf-technologies/hesperides-jenkins-lib#httprequester
Related
I am trying to understand if what I want to get is faceable.
We want to write a shared library step that allows us to produce a Kafka message.
To do this we use
#Grab(group = 'org.apache.kafka', module = 'kafka-clients', version = '3.2.0')
...
...
def producer = new KafkaProducer([
"bootstrap.servers": bootstrapServers,
// serializers
"value.serializer" : "org.apache.kafka.common.serialization.StringSerializer",
"key.serializer" : "org.apache.kafka.common.serialization.StringSerializer",
// acknowledgement control
"acks" : "all",
// TLS config
"ssl.truststore.type": "JKS",
"security.protocol": "SSL",
"ssl.enabled.protocols": "TLSv1.2",
"ssl.protocol": "TLSv1.2",
"ssl.truststore.location" : "<cacets_location>",
"ssl.truststore.password" : "changeit"
])
The method gets all params from outside except ssl.truststore.location
that is provided through the node volume.
The problem I realized is that JAVA commands are executed in a different workspace.
Let's say we have ssl.truststore.location in "/etc/pki/java/cacerts" location,
while using the readFile command I'm able to read it
, however, when I try to read using the pure JAVA command, I'm getting NoSuchFileFoundExecption
I found that when I execute a Java command like
File folder = new File("").getAbsoluteFile();
My working dir is empty "\", as if it's executed in an absolutely empty sandbox that is not related to the Jenkins workspace.
My question is if what I'm trying to obtain is doable in the Jenkins pipeline scope
and how to get it working.
I have written code that leverages Azure SDK for Blobs in order to interact with the blob storage.
As a clever and dutiful developer, I have not tested my code by navigating the live application, but rather created a Spring Boot JUnit test and spent a few hours fixing all my mistakes. I didn't use anyh kind of mocking, in fact, as my problem was using the library the correct way. I ran the code against a live instance of a blob storage and checked that all my Java methods worked as expected.
I am writing here because
To call it a day, I hardcoded the credentials in my source files. The repository is a company-private repository, not that harm. Credentials can be rotated, developers can all access from Azure portal and get the credentials. But still I don't like the idea of pushing credentials into code
Having these junit tests work on Azure DevOps pipelines could be some of a good idea
I know from the very beginning that hardcoding credentials into code is a worst practice, but since this morning I wanted to focus on my task. Now I want to adopt the best practices. I am asking about redesigning the test structure
Testing code is this.
The code creates an ephemeral container and tries to store/retrieve/delete blobs. It uses a GUID to create a unique private workspace, to clear after test is finished.
#SpringBootTest(classes = FileRepositoryServiceAzureBlobImplTest.class)
#SpringBootConfiguration
#TestConfiguration
#TestPropertySource(properties = {
"azure-storage-container-name:amlcbackendjunit",
"azure-storage-connection-string:[not going to post it on Stackoverflow before rotating it]"
})
class FileRepositoryServiceAzureBlobImplTest {
private static final Resource LOREM_IPSUM = new ClassPathResource("loremipsum.txt", FileRepositoryServiceAzureBlobImplTest.class);
private FileRepositoryServiceAzureBlobImpl uut;
private BlobContainerClient blobContainerClient;
private String loremChecksum;
#Value("${azure-storage-connection-string}")
private String azureConnectionString;
#Value("${azure-storage-container-name}")
private String azureContainerName;
#BeforeEach
void beforeEach() throws IOException {
String containerName = azureContainerName + "-" + UUID.randomUUID();
blobContainerClient = new BlobContainerClientBuilder()
.httpLogOptions(new HttpLogOptions().setApplicationId("az-sp-sb-aml"))
.clientOptions(new ClientOptions().setApplicationId("az-sp-sb-aml"))
.connectionString(azureConnectionString)
.containerName(containerName)
.buildClient()
;
blobContainerClient.create();
uut = spy(new FileRepositoryServiceAzureBlobImpl(blobContainerClient));
try (InputStream loremIpsumInputStream = LOREM_IPSUM.getInputStream();) {
loremChecksum = DigestUtils.sha256Hex(loremIpsumInputStream);
}
blobContainerClient
.getBlobClient("fox.txt")
.upload(BinaryData.fromString("The quick brown fox jumps over the lazy dog"));
}
#AfterEach
void afterEach() throws IOException {
blobContainerClient
.delete();
}
#Test
void store_ok() {
String desiredFileName = "loremIpsum.txt";
FileItemDescriptor output = assertDoesNotThrow(() -> uut.store(LOREM_IPSUM, desiredFileName));
assertAll(
() -> assertThat(output, is(notNullValue())),
() -> assertThat(output, hasProperty("uri", hasToString(Matchers.startsWith("azure-blob://")))),
() -> assertThat(output, hasProperty("size", equalTo(LOREM_IPSUM.contentLength()))),
() -> assertThat(output, hasProperty("checksum", equalTo(loremChecksum))),
() -> {
String localPart = substringAfter(output.getUri().toString(), "azure-blob://");
assertAll(
() -> assertTrue(blobContainerClient.getBlobClient(localPart).exists())
);
}
);
}
}
In production (but also in SIT/UAT), the real Spring Boot application will get the configuration from the Container environment, including the storage connection string. Yes, for this kind of test I could also avoid using Spring and #TestPropertySource, because I'm not leveraging any bean from the context.
Question
I want to ask how can I amend this test in order to
Decouple the connection string from code
Softly-ignore the test if for some reason the connection string is not present (e.g. developer downloaded the project the first time and wants to kick-start) (note 1)
Integrate this test (with a working connection string) from Azure DevOps pipelines, where I can configure virtually any environment variable and such
Here is the build job comprised of tests
- task: Gradle#2
displayName: Build with Gradle
inputs:
gradleWrapperFile: gradlew
gradleOptions: -Xmx3072m $(gradleJavaProperties)
options: -Pci=true -PbuildId=$(Build.BuildId) -PreleaseType=${{parameters.releaseType}}
jdkVersionOption: 1.11
jdkArchitectureOption: x64
publishJUnitResults: true
sqAnalysisEnabled: true
sqGradlePluginVersionChoice: specify
sqGradlePluginVersion: 3.2.0
testResultsFiles: '$(System.DefaultWorkingDirectory)/build/test-results/**/TEST-*.xml'
tasks: clean build
Note 1: the live application can be kick-started without the storage connection string. It falls back to a local temporary directory.
The answer is a bit complex to explain, so I did my best
TL;DR
Note that the original variable names are redacted and YMMV if you try to recreate the example with the exact keys I used
Create a secret pipeline variable containing the connection string, and bury* it into the pipeline
Example name testStorageAccountConnectionString
Change the Gradle task
- task: Gradle#3
displayName: Build with Gradle
inputs:
gradleWrapperFile: gradlew
gradleOptions: -Xmx10240m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 -DAZURE_STORAGE_CONNECTION_STRING=$(AZURE_STORAGE_CONNECTION_STRING)
options: --build-cache -Pci=true -PgitCommitId=$(Build.SourceVersion) -PbuildId=$(Build.BuildId) -Preckon.stage=${{parameters.versionStage}} -Preckon.scope=${{parameters.versionScope}}
jdkVersionOption: 1.11
jdkArchitectureOption: x64
publishJUnitResults: true
sqAnalysisEnabled: true
sqGradlePluginVersionChoice: specify
sqGradlePluginVersion: 3.2.0
testResultsFiles: '$(System.DefaultWorkingDirectory)/build/test-results/**/TEST-*.xml'
tasks: clean build
env:
AZURE_STORAGE_CONNECTION_STRING: $(testStorageAccountConnectionString)
Explanation
Spring Boot accepts placeholder ${azure.storageConnectionString} from an environment variable AZURE_STORAGE_CONNECTION_STRING. Please read the docs and try it locally first. This means we need to run the test with an environment variable propely set in order to resolve the placeholder
Gradle can run with -D to add an environment variable. -DAZURE_STORAGE_CONNECTION_STRING=$(AZURE_STORAGE_CONNECTION_STRING) adds an environment variable AZURE_STORAGE_CONNECTION_STRING to the test run equal to the pipeline environment variable AZURE_STORAGE_CONNECTION_STRING (not that fantasy)
Azure DevOps pipelines protect secret variables from unwanted access. We created the pipeline variable as secret, so there is another trick to do first
Gradle's env attributes set environment variable for the pipeline container. In this case, we make sure that Gradle runs with AZURE_STORAGE_CONNECTION_STRING set to testStorageAccountConnectionString. Env is the only place where Azure pipelines agent will resolve and set free the content of the secret variable
Secrets cannot be retrieved any more from web interface. Azure Pipelines are designed for this
I have a Jenkins pipeline;
#Library('sharedLib#master')
import org.foo.point
pipeline {
agent { label 'slaveone' }
// agent { label 'master' }
stages {
stage('Data Build'){
steps{
script{
def Point = new point()
Point.hello("mememe")
}
}
}
}
}
which runs a small bit of code in a library called 'jenkins-shared-library/src/sharedLib';
package org.foo
import java.io.File
class point{
def hello(name){
File saveFile = new File("c:/temp/jenkins_log.txt")
saveFile.write "hello"
}
}
It runs fine on both 'master' and 'slaveone', but in both cases the 'jenkins_log.txt' file appears on the master. The log file contains this;
Running on slaveone in d:\Jenkins_WorkDir\workspace\mypipeline
How is this code running on slaveone and writing files to master?
Edit: I should also mention that this is my third attempt at doing this. The first one was with Groovy code direct in the pipeline, and the second was using a 'def' type call in the vars directory. Both produced the same behaviour, seemingly oblivious to the agent it was being run on.
I think everything inside the script runs on master, but here I found a workaround: Jenkins Declarative Pipeline, run groovy script on slave agent
Jenkins stores all logs on master only, that's why you cannot find any log on nodes.
Suppose I've got a minimal Scala WORKSPACE file like this:
workspace(name = "scala_example")
git_repository(
name = "io_bazel_rules_scala",
commit = "e9e65ada59823c263352d10c30411f4739d5df25",
remote = "https://github.com/bazelbuild/rules_scala",
)
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_repositories")
scala_repositories()
load("#io_bazel_rules_scala//scala:toolchains.bzl", "scala_register_toolchains")
scala_register_toolchains()
And then a BUILD:
load("#io_bazel_rules_scala//scala:scala.bzl", "scala_binary")
scala_binary(
name = "example-bin",
srcs = glob(["*.scala"]),
main_class = "Example",
)
And an Example.scala:
object Example { def main(args: Array[String]): Unit = println("running") }
I can run bazel run example-bin and everything works just fine. My problem is that this recent rules_scala PR changed the way the Java binary path is set to use the following:
ctx.attr._java_runtime[java_common.JavaRuntimeInfo].java_executable_exec_path
…instead of the previous ctx.executable._java.short_path.
After this change the Java binary path includes an external directory in the path, which seems to be a legacy thing (?). This means that after this change, if I run the following:
bazel run --nolegacy_external_runfiles example-bin
It no longer works:
INFO: Running command line: bazel-bin/example-bin
.../.cache/bazel/_bazel_travis/03e97e9dbbfe483081a6eca2764532e8/execroot/scala_example/bazel-out/k8-fastbuild/bin/example-bin.runfiles/scala_example/example-bin_wrapper.sh: line 4: .../.cache/bazel/_bazel_travis/03e97e9dbbfe483081a6eca2764532e8/execroot/scala_example/bazel-out/k8-fastbuild/bin/example-bin.runfiles/scala_example/external/local_jdk/bin/java: No such file or directory
ERROR: Non-zero return code '127' from command: Process exited with status 127
It also breaks some scripts I have that expect non-external paths.
Why is java_executable_exec_path giving me this external path? Is there some option I can give bazel to convince it not to do this?
Sorry for the slow reply -- it appears that this is because the Scala rules erroneously used java_executable_exec_path whereas they should have used java_executable_runfiles_path.
I sent a pull request to fix it, then I realized that you already did in https://github.com/bazelbuild/rules_scala/commit/4235ef58782ce2ec82981ea70b808397b64fe7df
Since the latter is now available at HEAD with Bazel, I'll remove the ugly if at least.
I'm using jenkins as CI tool. I used restful api to build a job remotely but I don't know how to get test result remotely as well.
I can't be more thankful if anybody know a solution
Use the XML or Json API. At most pages on Jenkins you can add /api/ to the url and get data in xml, json and similar formats. So for a job you can go to <Jenkins URL>/job/<Job Name>/api/xml and get informaiton about the job, builds, etc. For a build you can go to <Jenkins URL>/job/<Job Name>/<build number>/api/xml and you will get a summary for the build. Note that you can use the latestXXXBuild in order to get the latest successful, stable, failing, complete build, like this; <Jenkins URL>/job/<Job Name>/lastCompletedBuild/api/xml.
Additionally if youre using any plugin which publishes test results to the build, then for a given job you can go to <Jenkins URL>/job/<Job Name>/lastCompletedBuild/testReport/api/xml and you will get an xml report with results.
There is a lot more to it, you can control what is exported with the tree parameter and depth parameter. For a summary go to <Jenkins URL>/api/
Well, if you are using a jenkins shared library or decided to permit the security exceptions (a less good approach) then you can access them via a job and send them out to whatever you like - push vs pull
def getCurrentBuildFailedTests() {
def failedTests = []
def build = currentBuild.build()
def action = build.getActions(hudson.tasks.junit.TestResultAction.class)
if (action) {
def failures = build.getAction(hudson.tasks.junit.TestResultAction.class).getFailedTests()
println "${failures.size()} Test Results Found"
for (def failure in failures) {
failedTests.add(['name': failure.name, 'url': failure.url, 'details': failure.errorDetails])
}
}
return failedTests
}