I have an app that does CRUD basically. I am able to run my unit tests locally but on the CI(GitHub Action) it's failing. I am getting the error because of PostgreSQL. Here you can see the error. I couldn't be able to fix that. You can access the whole repository on this LINK. You can see my ci.yaml file below;
name: CI
on:
pull_request:
push:
branches: [develop, main]
concurrency:
group: ci-${{ github.ref }}-group
cancel-in-progress: true
jobs:
default:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Set up JDK
uses: actions/setup-java#v3
with:
java-version: '17'
distribution: 'temurin'
- name: Build with Maven
run: mvn -B package --file pom.xml
- name: Update dependency graph
uses: advanced-security/maven-dependency-submission-action#571e99aab1055c2e71a1e2309b9691de18d6b7d6
- name: Build Jar file
run: ./project-dev build-jar
- name: Save Jar file
uses: actions/upload-artifact#v3
with:
name: demo-0.0.1-SNAPSHOT
path: target/demo-0.0.1-SNAPSHOT.jar
retention-days: 1
Can someone help me to run my unit tests on the CI, please?
You need to make sure that the database runs.
Your program expects a Posgres DB named school_management to be available under localhost:5432.
However, such a database isn't available in your script.
For setting up the database, you could use the an existing action like this one :
steps:
- uses: harmon758/postgresql-action#v1
with:
postgresql version: '11'
postgresql db: school_management
postgresql user: learning
postgresql password: sa123456
Alternatively, you could use PosgreSQL service containers as described here:
# Service containers to run with `container-job`
services:
# Label used to access the service container
postgres:
# Docker Hub image
image: postgres
# Provide the password for postgres
env:
POSTGRES_PASSWORD: sa123456
POSTGRES_USER: learning
POSTGRES_DB: school_management
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
However this makes it run using a different hostname so you have to change your spring.datasource.url to jdbc:postgresql://localhost:5432/school_management or similar.
Integrated in your workflow, it could look like the following:
name: CI
on:
pull_request:
push:
branches: [develop, main]
concurrency:
group: ci-${{ github.ref }}-group
cancel-in-progress: true
jobs:
default:
runs-on: ubuntu-latest
# Service containers to run with `container-job`
services:
# Label used to access the service container
postgres:
# Docker Hub image
image: postgres
# Provide the password for postgres
env:
POSTGRES_PASSWORD: sa123456
POSTGRES_USER: learning
POSTGRES_DB: school_management
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout#v3
- name: Set up JDK
uses: actions/setup-java#v3
with:
java-version: '17'
distribution: 'temurin'
# override spring.datasource.url
- name: Setup config
run: |
mkdir config
echo 'spring.datasource.url=jdbc:postgresql://postgres:5432/school_management' > config/application.properties
- name: Build with Maven
run: mvn -B package --file pom.xml
- name: Update dependency graph
uses: advanced-security/maven-dependency-submission-action#571e99aab1055c2e71a1e2309b9691de18d6b7d6
- name: Build Jar file
run: ./project-dev build-jar
- name: Save Jar file
uses: actions/upload-artifact#v3
with:
name: demo-0.0.1-SNAPSHOT
path: target/demo-0.0.1-SNAPSHOT.jar
retention-days: 1
Another possibility is to use an embedded database like H2 for tests.
With this, you don't have to setup any database.
Looking at your logs line 1351
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Your tests are trying to connect to a local Postgres instance that is not available. Also looking at your tests you have both unit and integration tests. Whereas an integration test needs to load the application context meaning that your running application inside of the pipeline will not be able to connect to Postgres. Hence, all of your integration tests will fail that utilize Postgres.
However, your other tests are passing, line 2085:
2023-02-14 12:13:39.378 INFO 1740 --- [ main] o.s.j.d.e.EmbeddedDatabaseFactory : Starting embedded database: url='jdbc:h2:mem:d00124ab-b172-4fd1-bf29-b4836ae2f938;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false', username='sa'
these are working since your application is connecting correctly to the h2 database that you have.
the StudentRepositoryTest are working since you have the following annotation in your class #DataJpaTest which will boot up this integration test and connect to the in-memory database.
I think the test that is failing is the following DemoApplicationTests:
#SpringBootTest
class DemoApplicationTests {
#Test
void contextLoads() {
}
}
Since this test load the application context (the whole application) and will automatically try to connect with postgres.
So to fix the issue just delete the file. or a better solution which I would recommend (which is a bit more advanced) is to use something called testcontainers and actually run a postgres database inside of a container.
The reason why am suggesting the latter solution is normally once you want to run an integration test you try to have the exact solution that your application runs on production. Hence, an h2 database might have edge cases that does not match postgres database
Any pointer why my codeql analysis is failing ? I am not getting any indication in log. Once Analysis ends, it also deletes all files from _work/_temp thus I am unable to see what's happening.
Here is my yml to start the code ql analysis
name: "CodeQL CI Workflow"
env:
CATALINA_HOME: "/apps/tomcat/apache-tomcat-9.0.44"
JAVA_HOME: "/apps/jdk1.8.0_231"
on:
push:
branches:
- master
- support/*
pull_request:
branches:
- master
- support/*
jobs:
code_ql_scan:
runs-on: [ rhelcicd ]
environment: dev
strategy:
fail-fast: false
matrix:
language: [ 'java' ]
steps:
- run: env
- uses: actions/checkout#v2
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init#v1
with:
debug: true
languages: ${{ matrix.language }}
config-file: ./.github/codeql/codeql-config.yml
- name: Building application using ant
run: ant -f build.xml -Dfile.encoding=ISO-8859-1 -Ddir.javadevlib=./javadevlib -Ddir.deploy=./built_artifact -Dapp.name=my-webapp
# Perform the CodeQL Analysis on compiled code by Ant
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze#v1
with:
debug: true
- name: Cleanup
if: ${{ always() }}
run: docker ps -q | xargs -n 1 -P 8 -I {} docker stop {}
- run : pwd
- run : ls
Here is my query config file ( ./.github/codeql/codeql-config.yml )
name: "CodeQL config"
queries:
- uses: security-and-quality
- uses: security-extended
paths-ignore:
- .idea
- .m2
- conf
- etc
- public_html
- ssccloud
- xmlfiles
- lib
- javadevlib
Here is the log
...................................... lots of other CWE processing logs ...........................................................
2022-02-04T20:51:05.1870388Z [27/178] Found in cache: /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-java/Security/CWE/CWE-502/UnsafeDeserialization.ql.
2022-02-04T20:51:05.1871797Z Compiling query plan for /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-java/Security/CWE/CWE-611/XXE.ql.
2022-02-04T20:51:05.1873100Z Resolving imports for /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-java/Security/CWE/CWE-611/XXE.ql.
2022-02-04T20:51:05.1874485Z Compiling query plan for /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-java/Security/CWE/CWE-614/InsecureCookie.ql.
2022-02-04T20:51:05.1876013Z Resolving imports for /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-java/Security/CWE/CWE-614/InsecureCookie.ql.
2022-02-04T20:51:05.1877647Z Compilation cache hit for /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-java/Security/CWE/CWE-614/InsecureCookie.ql.
2022-02-04T20:51:05.1879200Z [28/178] Found in cache: /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-java/Security/CWE/CWE-614/InsecureCookie.ql.
2022-02-04T20:51:05.1880653Z Compiling query plan for /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-java/Security/CWE/CWE-643/XPathInjection.ql.
2022-02-04T20:51:05.1882101Z Resolving imports for /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-java/Security/CWE/CWE-643/XPathInjection.ql.
2022-02-04T20:51:05.1883909Z Compilation cache hit for /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-java/Security/CWE/CWE-611/XXE.ql.
2022-02-04T20:51:05.1885185Z [29/178] Found in cache: /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-java/Security/CWE/CWE-611/XXE.ql.
2022-02-04T20:51:05.1886658Z Compiling query plan for /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-java/Security/CWE/CWE-681/NumericCastTainted.ql.
2022-02-04T20:51:05.1887950Z Resolving imports for /home/actions/actions-runner/_work/_tool/CodeQL/0.0.0-20211005/x64/codeql/qlpacks/codeql-j
2022-02-04T20:51:05.1889117Z at runQueries (/home/actions/actions-runner/_work/_actions/github/codeql-action/v1/lib/analyze.js:193:19)
2022-02-04T20:51:05.1890035Z at processTicksAndRejections (internal/process/task_queues.js:93:5) {
2022-02-04T20:51:05.1890832Z name: 'CodeQLAnalysisError',
2022-02-04T20:51:05.1891564Z queriesStatusReport: { analyze_failure_language: 'java' }
2022-02-04T20:51:05.1892054Z }
2022-02-04T20:51:05.3776434Z Post job cleanup.
2022-02-04T20:51:05.8506780Z Cleaning up orphan processes
I'm currently on my final phase of my bachelor thesis, implementing a web application. Now I need to host it on Azure. I've an application.war with Angular Frontend in it and a services.war with the Java REST Calls. Both are packed through Maven.
When I deploy it from Github with GitHub Actions to Azure, I get the following Error:
"Deployment Failed with Error: Error: More than one package matched with specified pattern: *.war. Please restrain the search pattern."
Do you have a tutorial or can help me how to setup the yml-file to deploy the application.war to ROOT and services.war to /services/...
on:
push:
branches:
- master
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Set up Java version
uses: actions/setup-java#v1
with:
java-version: '11'
- name: Build with Maven
run: mvn clean install
- name: Upload artifact for deployment job
uses: actions/upload-artifact#v2
with:
name: java-app
path: '${{ github.workspace }}/**/target/*.war'
deploy:
runs-on: ubuntu-latest
needs: build
environment:
name: 'Production'
url: ${{ steps.deploy-to-webapp.outputs.webapp-url }}
steps:
- name: Download artifact from build job
uses: actions/download-artifact#v2
with:
name: java-app
- name: Deploy to Azure Web App
id: deploy-to-webapp
uses: azure/webapps-deploy#v2
with:
app-name: 'app-name'
slot-name: 'Production'
publish-profile: ${{ secrets.AZUREAPPSERVICE_PUBLISHPROFILE }}
package: '*.war'
Thank you very much!
Janick
If you check this Deploy Angular app to Azure App Service running Linux from GitHub, you will understand how to deploy an Angular single page application, from GitHub to Azure App Service.
In order to deploy to App Service using GitHub Actions we need a workflow file. A workflow is defined by a YAML (.yml) file in the /.github/workflows/ path in your repository. This definition contains the various steps and parameters that make up the workflow.
Build and deploy a Java app to Azure using an Azure publish profile. The publish-profile input references the AZURE_WEBAPP_PUBLISH_PROFILE secret that you created earlier.
name: Java CI with Maven
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: Build with Maven
run: mvn -B package --file pom.xml
working-directory: my-app-path
- name: Azure WebApp
uses: Azure/webapps-deploy#v2
with:
app-name: my-app-name
publish-profile: ${{ secrets.AZURE_WEBAPP_PUBLISH_PROFILE }}
package: my/target/*.war
For more information check this Deploy to App Service using GitHub Actions document from Microsoft. Also read this Create Azure Web App and deploy using GitHub Actions document from GitHub and check the example templates.
I have a Github action pipeline that can successfully create an S3 and then upload my war file into there, but when deploying to the beanstalk, always got s3 access denied error. below is my build.yml file:
# This workflow will build a package using Maven and then publish it to GitHub packages when a release is created
# For more information see: https://github.com/actions/setup-java/blob/main/docs/advanced-usage.md#apache-maven-with-a-settings-path
name: Maven Package
on:
pull_request:
branches:
- main
push:
branches:
- develop
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout#v2
name: Set up JDK 8
- uses: actions/setup-java#v2
with:
java-version: '8'
distribution: 'adopt'
server-id: github # Value of the distributionManagement/repository/id field of the pom.xml
settings-path: ${{ github.workspace }} # location for the settings.xml file
- name: Build with Maven
run: mvn -B package --file pom.xml
- name: make a new dir and upload war in there
run: mkdir staging && cp -r target/* staging
- uses: actions/upload-artifact#v2
with:
name: Package
path: staging
- name: list all files
run: ls && cd target && ls
- name: Publish to GitHub Packages Apache Maven
run: mvn deploy -s $GITHUB_WORKSPACE/settings.xml
env:
GITHUB_TOKEN: ${{ github.token }}
- name: Deploy to EB
uses: einaregilsson/beanstalk-deploy#v18
with:
aws_access_key: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws_secret_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
application_name: springbootwebapi
environment_name: Springbootwebapi-env
version_label: v1.0.10
region: us-east-2
deployment_package: target/login-0.0.2-SNAPSHOT.war
below is the some log snippet from GitHub action:
No existing bucket name given, creating/requesting storage location
Uploading file to bucket elasticbeanstalk-us-east-2-148565102071 New
build successfully uploaded to S3,
bucket=elasticbeanstalk-us-east-2-148565102071,
key=/springbootwebapi/v1-0-10.zip Created new application version
v1.0.10 in Beanstalk. Starting deployment of version v1.0.10 to
environment Springbootwebapi-env Deployment started,
"wait_for_deployment" was true...
18:17:02 INFO: Environment update is starting. 18:17:06 ERROR:
Service:Amazon S3, Message:Access Denied 18:17:06 ERROR: Failed to
deploy application. 18:17:07 ERROR: Service:Amazon S3, Message:Access
Denied: S3Bucket=elasticbeanstalk-us-east-2-148565102071,
S3Key=resources/environments/e-fp5bx3gtdn/_runtime/_versions/springbootwebapi/v1.0.10
18:17:13 ERROR: Deployment failed! Current State: Version: Sample
Application, Health: Red, Health Status: Degraded Error: Deployment
failed: Error: Deployment failed! Current State: Version: Sample
Application, Health: Red, Health Status: Degraded
I don't know why got accessed denied even right after the uploading successfully.
UPDATE 1:
I already have the below permissions added see the below, but not working:
As per docs, you need to attach the below policies for the AWS user to be able to deploy your project when using the GitHub action you have specified:
AWSElasticBeanstalkWebTier
AWSElasticBeanstalkManagedUpdatesCustomerRolePolicy
Adding the above will fix the problem, while also ensuring that you have no future issues when using this GitHub action.
After removing AWSCompromisedKeyQuarantineV2 from the permission list, it works successfully. The reason is this permission actually denies several related operations to the user, see the below JSON for AWSCompromisedKeyQuarantineV2 details:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": [
"ec2:RequestSpotInstances",
"ec2:RunInstances",
"ec2:StartInstances",
"iam:AddUserToGroup",
"iam:AttachGroupPolicy",
"iam:AttachRolePolicy",
"iam:AttachUserPolicy",
"iam:ChangePassword",
"iam:CreateAccessKey",
"iam:CreateInstanceProfile",
"iam:CreateLoginProfile",
"iam:CreatePolicyVersion",
"iam:CreateRole",
"iam:CreateUser",
"iam:DetachUserPolicy",
"iam:PassRole",
"iam:PutGroupPolicy",
"iam:PutRolePolicy",
"iam:PutUserPermissionsBoundary",
"iam:PutUserPolicy",
"iam:SetDefaultPolicyVersion",
"iam:UpdateAccessKey",
"iam:UpdateAccountPasswordPolicy",
"iam:UpdateAssumeRolePolicy",
"iam:UpdateLoginProfile",
"iam:UpdateUser",
"lambda:AddLayerVersionPermission",
"lambda:AddPermission",
"lambda:CreateFunction",
"lambda:GetPolicy",
"lambda:ListTags",
"lambda:PutProvisionedConcurrencyConfig",
"lambda:TagResource",
"lambda:UntagResource",
"lambda:UpdateFunctionCode",
"lightsail:Create*",
"lightsail:Delete*",
"lightsail:DownloadDefaultKeyPair",
"lightsail:GetInstanceAccessDetails",
"lightsail:Start*",
"lightsail:Update*",
"organizations:CreateAccount",
"organizations:CreateOrganization",
"organizations:InviteAccountToOrganization",
"s3:DeleteBucket",
"s3:DeleteObject",
"s3:DeleteObjectVersion",
"s3:PutLifecycleConfiguration",
"s3:PutBucketAcl",
"s3:DeleteBucketOwnershipControls",
"s3:DeleteBucketPolicy",
"s3:ObjectOwnerOverrideToBucketOwner",
"s3:PutAccountPublicAccessBlock",
"s3:PutBucketPolicy",
"s3:ListAllMyBuckets"
],
"Resource": [
"*"
]
}
]
}
I created a job Maven Project in Jenkins for the purpose to Compile and execute all my automatic tests, but when i build the job i got this error :
i got the same message error for all the scenarios
should i create a pipeline in place of project maven ?
i recupare my project with the link ssh GitLab and im working behind a proxy
Thnaks(y)
Here is a docker-compose file that will open 7 instances of FF and 1 instance on chrome. I use it with azure pipeline but you can integrate it with jenkins. You'll will have to add a jenkins task that runs docker-compose
To try on command line , just install docker desktop ( i use it with mac) and run below command
docker-compose -f /path/of/file up
version: "3"
services:
selenium-hub:
image: selenium/hub:3.141.59-20210607
container_name: selenium-hub
ports:
- "65299:4444"
chrome:
image: selenium/node-chrome:3.141.59-20210607
depends_on:
- selenium-hub
environment:
- HUB_HOST=selenium-hub
- HUB_PORT=4444
firefox:
image: selenium/node-firefox:3.141.59-20210607
depends_on:
- selenium-hub
environment:
- HUB_HOST=selenium-hub
- HUB_PORT=4444
deploy:
mode: replicated
replicas: 7
For using docker-compose with azure pipeline , I am using the below.
Make sure you have dockerRegistryEndpoint setup (in below ex: Dockerhub) . I use this to run my cucumber tests and integrate third party cucumber report (PublishCucumberReport#1)in pipeline
trigger:
- master
resources:
- repo: self
variables:
tag: '$(Build.BuildId)'
stages:
- stage: Build
displayName: Build and Push image
jobs:
- job: Build
displayName: Build and Push
pool:
vmImage: 'ubuntu-latest'
steps:
- task: DockerCompose#0
displayName: open browser instances
inputs:
containerregistrytype: 'Container Registry'
dockerRegistryEndpoint: Dockerhub
dockerComposeFile: '**/docker-compose.yml'
action: 'Run a Docker Compose command'
dockerComposeCommand: 'up -d'
detached: true
- task: Maven#3
inputs:
mavenPomFile: 'pom.xml'
mavenOptions: '-Xmx3072m'
jdkArchitectureOption: 'x64'
publishJUnitResults: true
testResultsFiles: '**/target/cucumber.html'
goals: 'clean verify -P acceptanceTests -e -X'
- task: PublishPipelineArtifact#1
displayName: Publish cucumber report
inputs:
pathToPublish: $(System.DefaultWorkingDirectory)/s/target/cucumber-report/
artifactName: 'cucumber.html'
- task: PublishCucumberReport#1
inputs:
jsonDir: ./target/cucumber-report/
outputPath: ./target/
For documentation , refer - https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/build/docker-compose?view=azure-devops