I have written code that leverages Azure SDK for Blobs in order to interact with the blob storage.
As a clever and dutiful developer, I have not tested my code by navigating the live application, but rather created a Spring Boot JUnit test and spent a few hours fixing all my mistakes. I didn't use anyh kind of mocking, in fact, as my problem was using the library the correct way. I ran the code against a live instance of a blob storage and checked that all my Java methods worked as expected.
I am writing here because
To call it a day, I hardcoded the credentials in my source files. The repository is a company-private repository, not that harm. Credentials can be rotated, developers can all access from Azure portal and get the credentials. But still I don't like the idea of pushing credentials into code
Having these junit tests work on Azure DevOps pipelines could be some of a good idea
I know from the very beginning that hardcoding credentials into code is a worst practice, but since this morning I wanted to focus on my task. Now I want to adopt the best practices. I am asking about redesigning the test structure
Testing code is this.
The code creates an ephemeral container and tries to store/retrieve/delete blobs. It uses a GUID to create a unique private workspace, to clear after test is finished.
#SpringBootTest(classes = FileRepositoryServiceAzureBlobImplTest.class)
#SpringBootConfiguration
#TestConfiguration
#TestPropertySource(properties = {
"azure-storage-container-name:amlcbackendjunit",
"azure-storage-connection-string:[not going to post it on Stackoverflow before rotating it]"
})
class FileRepositoryServiceAzureBlobImplTest {
private static final Resource LOREM_IPSUM = new ClassPathResource("loremipsum.txt", FileRepositoryServiceAzureBlobImplTest.class);
private FileRepositoryServiceAzureBlobImpl uut;
private BlobContainerClient blobContainerClient;
private String loremChecksum;
#Value("${azure-storage-connection-string}")
private String azureConnectionString;
#Value("${azure-storage-container-name}")
private String azureContainerName;
#BeforeEach
void beforeEach() throws IOException {
String containerName = azureContainerName + "-" + UUID.randomUUID();
blobContainerClient = new BlobContainerClientBuilder()
.httpLogOptions(new HttpLogOptions().setApplicationId("az-sp-sb-aml"))
.clientOptions(new ClientOptions().setApplicationId("az-sp-sb-aml"))
.connectionString(azureConnectionString)
.containerName(containerName)
.buildClient()
;
blobContainerClient.create();
uut = spy(new FileRepositoryServiceAzureBlobImpl(blobContainerClient));
try (InputStream loremIpsumInputStream = LOREM_IPSUM.getInputStream();) {
loremChecksum = DigestUtils.sha256Hex(loremIpsumInputStream);
}
blobContainerClient
.getBlobClient("fox.txt")
.upload(BinaryData.fromString("The quick brown fox jumps over the lazy dog"));
}
#AfterEach
void afterEach() throws IOException {
blobContainerClient
.delete();
}
#Test
void store_ok() {
String desiredFileName = "loremIpsum.txt";
FileItemDescriptor output = assertDoesNotThrow(() -> uut.store(LOREM_IPSUM, desiredFileName));
assertAll(
() -> assertThat(output, is(notNullValue())),
() -> assertThat(output, hasProperty("uri", hasToString(Matchers.startsWith("azure-blob://")))),
() -> assertThat(output, hasProperty("size", equalTo(LOREM_IPSUM.contentLength()))),
() -> assertThat(output, hasProperty("checksum", equalTo(loremChecksum))),
() -> {
String localPart = substringAfter(output.getUri().toString(), "azure-blob://");
assertAll(
() -> assertTrue(blobContainerClient.getBlobClient(localPart).exists())
);
}
);
}
}
In production (but also in SIT/UAT), the real Spring Boot application will get the configuration from the Container environment, including the storage connection string. Yes, for this kind of test I could also avoid using Spring and #TestPropertySource, because I'm not leveraging any bean from the context.
Question
I want to ask how can I amend this test in order to
Decouple the connection string from code
Softly-ignore the test if for some reason the connection string is not present (e.g. developer downloaded the project the first time and wants to kick-start) (note 1)
Integrate this test (with a working connection string) from Azure DevOps pipelines, where I can configure virtually any environment variable and such
Here is the build job comprised of tests
- task: Gradle#2
displayName: Build with Gradle
inputs:
gradleWrapperFile: gradlew
gradleOptions: -Xmx3072m $(gradleJavaProperties)
options: -Pci=true -PbuildId=$(Build.BuildId) -PreleaseType=${{parameters.releaseType}}
jdkVersionOption: 1.11
jdkArchitectureOption: x64
publishJUnitResults: true
sqAnalysisEnabled: true
sqGradlePluginVersionChoice: specify
sqGradlePluginVersion: 3.2.0
testResultsFiles: '$(System.DefaultWorkingDirectory)/build/test-results/**/TEST-*.xml'
tasks: clean build
Note 1: the live application can be kick-started without the storage connection string. It falls back to a local temporary directory.
The answer is a bit complex to explain, so I did my best
TL;DR
Note that the original variable names are redacted and YMMV if you try to recreate the example with the exact keys I used
Create a secret pipeline variable containing the connection string, and bury* it into the pipeline
Example name testStorageAccountConnectionString
Change the Gradle task
- task: Gradle#3
displayName: Build with Gradle
inputs:
gradleWrapperFile: gradlew
gradleOptions: -Xmx10240m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 -DAZURE_STORAGE_CONNECTION_STRING=$(AZURE_STORAGE_CONNECTION_STRING)
options: --build-cache -Pci=true -PgitCommitId=$(Build.SourceVersion) -PbuildId=$(Build.BuildId) -Preckon.stage=${{parameters.versionStage}} -Preckon.scope=${{parameters.versionScope}}
jdkVersionOption: 1.11
jdkArchitectureOption: x64
publishJUnitResults: true
sqAnalysisEnabled: true
sqGradlePluginVersionChoice: specify
sqGradlePluginVersion: 3.2.0
testResultsFiles: '$(System.DefaultWorkingDirectory)/build/test-results/**/TEST-*.xml'
tasks: clean build
env:
AZURE_STORAGE_CONNECTION_STRING: $(testStorageAccountConnectionString)
Explanation
Spring Boot accepts placeholder ${azure.storageConnectionString} from an environment variable AZURE_STORAGE_CONNECTION_STRING. Please read the docs and try it locally first. This means we need to run the test with an environment variable propely set in order to resolve the placeholder
Gradle can run with -D to add an environment variable. -DAZURE_STORAGE_CONNECTION_STRING=$(AZURE_STORAGE_CONNECTION_STRING) adds an environment variable AZURE_STORAGE_CONNECTION_STRING to the test run equal to the pipeline environment variable AZURE_STORAGE_CONNECTION_STRING (not that fantasy)
Azure DevOps pipelines protect secret variables from unwanted access. We created the pipeline variable as secret, so there is another trick to do first
Gradle's env attributes set environment variable for the pipeline container. In this case, we make sure that Gradle runs with AZURE_STORAGE_CONNECTION_STRING set to testStorageAccountConnectionString. Env is the only place where Azure pipelines agent will resolve and set free the content of the secret variable
Secrets cannot be retrieved any more from web interface. Azure Pipelines are designed for this
Related
I am trying to understand if what I want to get is faceable.
We want to write a shared library step that allows us to produce a Kafka message.
To do this we use
#Grab(group = 'org.apache.kafka', module = 'kafka-clients', version = '3.2.0')
...
...
def producer = new KafkaProducer([
"bootstrap.servers": bootstrapServers,
// serializers
"value.serializer" : "org.apache.kafka.common.serialization.StringSerializer",
"key.serializer" : "org.apache.kafka.common.serialization.StringSerializer",
// acknowledgement control
"acks" : "all",
// TLS config
"ssl.truststore.type": "JKS",
"security.protocol": "SSL",
"ssl.enabled.protocols": "TLSv1.2",
"ssl.protocol": "TLSv1.2",
"ssl.truststore.location" : "<cacets_location>",
"ssl.truststore.password" : "changeit"
])
The method gets all params from outside except ssl.truststore.location
that is provided through the node volume.
The problem I realized is that JAVA commands are executed in a different workspace.
Let's say we have ssl.truststore.location in "/etc/pki/java/cacerts" location,
while using the readFile command I'm able to read it
, however, when I try to read using the pure JAVA command, I'm getting NoSuchFileFoundExecption
I found that when I execute a Java command like
File folder = new File("").getAbsoluteFile();
My working dir is empty "\", as if it's executed in an absolutely empty sandbox that is not related to the Jenkins workspace.
My question is if what I'm trying to obtain is doable in the Jenkins pipeline scope
and how to get it working.
I have just received a new project, I have a fresh repo clone of a java spring project.
When I build it with Gradle, all the dependencies are downloaded but when one of the Gradle tasks execute, the unit tests, the build fails.
I think the problem resides in the argThat() method of Mockito that is not getting well integrated with JUnit. This is one of the places where the issue occurs:
Any time a unit tests have this kind of logic, it fails with:
The console output is not for the above test but it is a similar method with more complex logic.
The above tests still fail with the same issue.
This only happens in my machine and not on others that are on a Unix distribution, fedora.
I think the problem is due to the dependencies version, but I have tested with different ones to no avail.
I can give you more information if needed.
Thank you.
EDIT: Code - not a screenshot
#Test
void shouldAbortEventExecutionWhenJobFails() throws JobParametersInvalidException, JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException {
when(jobLauncher.run(eq(job1), argThat(jobParametersForPath(TEST_PATH_1)))).thenReturn(jobExecutionFailed);
when(job1.getName()).thenReturn("job1");
ExecutionState result = executor.execute(asList(event1, event2));
assertThat(result).isEqualTo(ExecutionState.FAILED);
verify(jobLauncher).run(eq(job1), argThat(jobParametersForPath(TEST_PATH_1)));
verify(jobLauncher, never()).run(eq(job2), argThat(jobParametersForPath(TEST_PATH_1)));
verify(jobLauncher).run(eq(job1), argThat(jobParametersForPath(TEST_PATH_2)));
verify(jobLauncher).run(eq(job2), argThat(jobParametersForPath(TEST_PATH_2)));
verifyNoMoreInteractions(jobLauncher);
}
private ArgumentMatcher<JobParameters> jobParametersForPath(String inputPath) {
return jobParameters ->
jobParameters.getParameters().get("inputFilePath").toString().equals(inputPath) &&
jobParameters.getParameters().get("outputFilePath").toString().equals(TEST_OUTPUT_PATH + "/" + inputPath) &&
jobParameters.getParameters().containsKey("timestamp");
}
I can't tell you the exact problem without inspecting your code or without reproducing your issue. But I guess the problem should be related to file paths;
I can see that there is a variable called outputFilePath inside your assertation object. in Linux environments, we use slash / for file paths, but in windows environments it's back-slashes \.
[1] https://www.howtogeek.com/181774/why-windows-uses-backslashes-and-everything-else-uses-forward-slashes/
[2] https://stackoverflow.com/a/1589959/3728639
You need to debug your Junit test and compare actual assertation object with the expected one
I am in the process of setting up Jenkins pipeline builds and am starting to use the same methods across multiple jobs, so it's time to put these common methods into a shared library.
The first function I have created is to update GitHub with the result of some unit tests. I am having an issue where I can run this function from the command line fine but when it comes to using it within my Jenkins build it does not work and I cannot seem to get any debug output in the Jenkins console
This is the directory structure of my shared library
my-project
src
vars
- getCommitId.groovy
- gitUpdateStatus.groovy
So the first function getCommitId works fine
#!/usr/bin/env groovy
def call() {
commit_id = sh script: 'git rev-parse HEAD', returnStdout: true
commit_id = commit_id.replaceAll("\\s","") // Remove Whitespace
return commit_id
}
This returns the correct value
This is gitUpdateStatus
#!/usr/bin/env groovy
#Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7')
import static groovyx.net.http.ContentType.JSON
import static groovyx.net.http.Method.POST
import groovyx.net.http.HTTPBuilder
String targetUrl = 'https://api.github.com/repos/myRepo/'
def http = new HTTPBuilder(targetUrl)
http.request(POST) {
uri.path = "repo/statuses/12345678"
requestContentType = JSON
body = [state: 'success', description: 'Jenkins Unit Tests', target_url: 'http://test.co.uk', context: 'unit tests']
headers.'Authorization' = "token myOauthTokenHere"
headers.'User-Agent' = 'Jenkins Status Update'
headers.Accept = 'application/json'
response.success = { resp, json ->
println "GitHub updated successfully! ${resp.status}"
}
response.failure = { resp, json ->
println "GitHub update Failure! ${resp.status} " + json.message
}
}
I can run this fine via the command line, but I get no output when run as a Jenkins build.
My Jenkinsfile
#Library('echo-jenkins-shared')_
node {
GIT_COMMIT_ID = getGitCommitId()
echo "GIT COMMIT ID: ${GIT_COMMIT_ID}"
gitUpdateStatus(GIT_COMMIT_ID)
}
Why would this not work or could this be converted to just use native Groovy methods?
First off, I would advise you to use as service like https://requestb.in to check if your code actually perform HTTP calls.
Second, I would recommend NOT to use #Grab-based dependencies like HTTPBuilder in Jenkins pipelines, but the http_request plugin instead, downloadable & installable as a .hpi:
https://jenkins.io/doc/pipeline/steps/http_request/
Finally, you can find an example of utility class to perform HTTP requests here:
https://github.com/voyages-sncf-technologies/hesperides-jenkins-lib/blob/master/src/com/vsct/dt/hesperides/jenkins/pipelines/http/HTTPBuilderRequester.groovy
With the rationale behind it explained there: https://github.com/voyages-sncf-technologies/hesperides-jenkins-lib#httprequester
Recently got the code to write bdd tests with cucumber on Java. There is already maven project with couple of tests and test framework. I need to continue writing bdd tests using this framework.
I am writing API tests and try to run them and i get the error. I found where it fails to run further but I want to figure out what's the idea of doing so in the code. Let me share some code:
So the test framework is collecting info about the API host name this way:
public class AnyClass {
private static final String API_HOSTNAME = "hostname";
private static String getAPIHostName() {
String apiHostName = System.getProperty(API_HOSTNAME);
...
}
When i leave it as is, and run the test, i get the error that host name is empty.
Can you advise on what might be expected to have under System property key "hostname"?
p.s. I tried to use http://localhost and http://127.0.0.1, where my api is located instead of assigning system property but it cannot find such host name.
Can you advise on what might be expected to have under System property key "hostname"?
Yes, I needed to run tests in command line with the syntax like:
mvn clean verify -Dhostname=http://127.0.0.1:8080
I'm using jenkins as CI tool. I used restful api to build a job remotely but I don't know how to get test result remotely as well.
I can't be more thankful if anybody know a solution
Use the XML or Json API. At most pages on Jenkins you can add /api/ to the url and get data in xml, json and similar formats. So for a job you can go to <Jenkins URL>/job/<Job Name>/api/xml and get informaiton about the job, builds, etc. For a build you can go to <Jenkins URL>/job/<Job Name>/<build number>/api/xml and you will get a summary for the build. Note that you can use the latestXXXBuild in order to get the latest successful, stable, failing, complete build, like this; <Jenkins URL>/job/<Job Name>/lastCompletedBuild/api/xml.
Additionally if youre using any plugin which publishes test results to the build, then for a given job you can go to <Jenkins URL>/job/<Job Name>/lastCompletedBuild/testReport/api/xml and you will get an xml report with results.
There is a lot more to it, you can control what is exported with the tree parameter and depth parameter. For a summary go to <Jenkins URL>/api/
Well, if you are using a jenkins shared library or decided to permit the security exceptions (a less good approach) then you can access them via a job and send them out to whatever you like - push vs pull
def getCurrentBuildFailedTests() {
def failedTests = []
def build = currentBuild.build()
def action = build.getActions(hudson.tasks.junit.TestResultAction.class)
if (action) {
def failures = build.getAction(hudson.tasks.junit.TestResultAction.class).getFailedTests()
println "${failures.size()} Test Results Found"
for (def failure in failures) {
failedTests.add(['name': failure.name, 'url': failure.url, 'details': failure.errorDetails])
}
}
return failedTests
}