I have an app that does CRUD basically. I am able to run my unit tests locally but on the CI(GitHub Action) it's failing. I am getting the error because of PostgreSQL. Here you can see the error. I couldn't be able to fix that. You can access the whole repository on this LINK. You can see my ci.yaml file below;
name: CI
on:
pull_request:
push:
branches: [develop, main]
concurrency:
group: ci-${{ github.ref }}-group
cancel-in-progress: true
jobs:
default:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Set up JDK
uses: actions/setup-java#v3
with:
java-version: '17'
distribution: 'temurin'
- name: Build with Maven
run: mvn -B package --file pom.xml
- name: Update dependency graph
uses: advanced-security/maven-dependency-submission-action#571e99aab1055c2e71a1e2309b9691de18d6b7d6
- name: Build Jar file
run: ./project-dev build-jar
- name: Save Jar file
uses: actions/upload-artifact#v3
with:
name: demo-0.0.1-SNAPSHOT
path: target/demo-0.0.1-SNAPSHOT.jar
retention-days: 1
Can someone help me to run my unit tests on the CI, please?
You need to make sure that the database runs.
Your program expects a Posgres DB named school_management to be available under localhost:5432.
However, such a database isn't available in your script.
For setting up the database, you could use the an existing action like this one :
steps:
- uses: harmon758/postgresql-action#v1
with:
postgresql version: '11'
postgresql db: school_management
postgresql user: learning
postgresql password: sa123456
Alternatively, you could use PosgreSQL service containers as described here:
# Service containers to run with `container-job`
services:
# Label used to access the service container
postgres:
# Docker Hub image
image: postgres
# Provide the password for postgres
env:
POSTGRES_PASSWORD: sa123456
POSTGRES_USER: learning
POSTGRES_DB: school_management
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
However this makes it run using a different hostname so you have to change your spring.datasource.url to jdbc:postgresql://localhost:5432/school_management or similar.
Integrated in your workflow, it could look like the following:
name: CI
on:
pull_request:
push:
branches: [develop, main]
concurrency:
group: ci-${{ github.ref }}-group
cancel-in-progress: true
jobs:
default:
runs-on: ubuntu-latest
# Service containers to run with `container-job`
services:
# Label used to access the service container
postgres:
# Docker Hub image
image: postgres
# Provide the password for postgres
env:
POSTGRES_PASSWORD: sa123456
POSTGRES_USER: learning
POSTGRES_DB: school_management
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout#v3
- name: Set up JDK
uses: actions/setup-java#v3
with:
java-version: '17'
distribution: 'temurin'
# override spring.datasource.url
- name: Setup config
run: |
mkdir config
echo 'spring.datasource.url=jdbc:postgresql://postgres:5432/school_management' > config/application.properties
- name: Build with Maven
run: mvn -B package --file pom.xml
- name: Update dependency graph
uses: advanced-security/maven-dependency-submission-action#571e99aab1055c2e71a1e2309b9691de18d6b7d6
- name: Build Jar file
run: ./project-dev build-jar
- name: Save Jar file
uses: actions/upload-artifact#v3
with:
name: demo-0.0.1-SNAPSHOT
path: target/demo-0.0.1-SNAPSHOT.jar
retention-days: 1
Another possibility is to use an embedded database like H2 for tests.
With this, you don't have to setup any database.
Looking at your logs line 1351
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Your tests are trying to connect to a local Postgres instance that is not available. Also looking at your tests you have both unit and integration tests. Whereas an integration test needs to load the application context meaning that your running application inside of the pipeline will not be able to connect to Postgres. Hence, all of your integration tests will fail that utilize Postgres.
However, your other tests are passing, line 2085:
2023-02-14 12:13:39.378 INFO 1740 --- [ main] o.s.j.d.e.EmbeddedDatabaseFactory : Starting embedded database: url='jdbc:h2:mem:d00124ab-b172-4fd1-bf29-b4836ae2f938;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false', username='sa'
these are working since your application is connecting correctly to the h2 database that you have.
the StudentRepositoryTest are working since you have the following annotation in your class #DataJpaTest which will boot up this integration test and connect to the in-memory database.
I think the test that is failing is the following DemoApplicationTests:
#SpringBootTest
class DemoApplicationTests {
#Test
void contextLoads() {
}
}
Since this test load the application context (the whole application) and will automatically try to connect with postgres.
So to fix the issue just delete the file. or a better solution which I would recommend (which is a bit more advanced) is to use something called testcontainers and actually run a postgres database inside of a container.
The reason why am suggesting the latter solution is normally once you want to run an integration test you try to have the exact solution that your application runs on production. Hence, an h2 database might have edge cases that does not match postgres database
I run tests on GitHub Actions, like this:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Runs Elasticsearch
uses: elastic/elastic-github-actions/elasticsearch#master
with:
stack-version: 7.16.1
- name: Set up JDK 17
uses: actions/setup-java#v1
with:
java-version: 17
- name: Build with Maven
run: mvn -B package --file pom.xml
The version of ElasticSearch is currently fixed; but in my project I include the ElasticSearch client 7.16.3. I would like to use that version in the stack-version as well. Is there a preferred way to extract a property from the pom.xml and use that in the Action?
You can use get-xml-info to read a property from the pom.xml
- name: Get elasticsearch version from pom.xml
id: get-elasticsearch-version
uses: mavrosxristoforos/get-xml-info#1.0
with:
xml-file: 'pom.xml'
xpath: '//*[local-name()="elasticsearch.version.property.tagname"]'
- name: Runs Elasticsearch
uses: elastic/elastic-github-actions/elasticsearch#master
with:
stack-version: ${{ steps.get-elasticsearch-version.outputs.info }}
I have a Github action pipeline that can successfully create an S3 and then upload my war file into there, but when deploying to the beanstalk, always got s3 access denied error. below is my build.yml file:
# This workflow will build a package using Maven and then publish it to GitHub packages when a release is created
# For more information see: https://github.com/actions/setup-java/blob/main/docs/advanced-usage.md#apache-maven-with-a-settings-path
name: Maven Package
on:
pull_request:
branches:
- main
push:
branches:
- develop
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout#v2
name: Set up JDK 8
- uses: actions/setup-java#v2
with:
java-version: '8'
distribution: 'adopt'
server-id: github # Value of the distributionManagement/repository/id field of the pom.xml
settings-path: ${{ github.workspace }} # location for the settings.xml file
- name: Build with Maven
run: mvn -B package --file pom.xml
- name: make a new dir and upload war in there
run: mkdir staging && cp -r target/* staging
- uses: actions/upload-artifact#v2
with:
name: Package
path: staging
- name: list all files
run: ls && cd target && ls
- name: Publish to GitHub Packages Apache Maven
run: mvn deploy -s $GITHUB_WORKSPACE/settings.xml
env:
GITHUB_TOKEN: ${{ github.token }}
- name: Deploy to EB
uses: einaregilsson/beanstalk-deploy#v18
with:
aws_access_key: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws_secret_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
application_name: springbootwebapi
environment_name: Springbootwebapi-env
version_label: v1.0.10
region: us-east-2
deployment_package: target/login-0.0.2-SNAPSHOT.war
below is the some log snippet from GitHub action:
No existing bucket name given, creating/requesting storage location
Uploading file to bucket elasticbeanstalk-us-east-2-148565102071 New
build successfully uploaded to S3,
bucket=elasticbeanstalk-us-east-2-148565102071,
key=/springbootwebapi/v1-0-10.zip Created new application version
v1.0.10 in Beanstalk. Starting deployment of version v1.0.10 to
environment Springbootwebapi-env Deployment started,
"wait_for_deployment" was true...
18:17:02 INFO: Environment update is starting. 18:17:06 ERROR:
Service:Amazon S3, Message:Access Denied 18:17:06 ERROR: Failed to
deploy application. 18:17:07 ERROR: Service:Amazon S3, Message:Access
Denied: S3Bucket=elasticbeanstalk-us-east-2-148565102071,
S3Key=resources/environments/e-fp5bx3gtdn/_runtime/_versions/springbootwebapi/v1.0.10
18:17:13 ERROR: Deployment failed! Current State: Version: Sample
Application, Health: Red, Health Status: Degraded Error: Deployment
failed: Error: Deployment failed! Current State: Version: Sample
Application, Health: Red, Health Status: Degraded
I don't know why got accessed denied even right after the uploading successfully.
UPDATE 1:
I already have the below permissions added see the below, but not working:
As per docs, you need to attach the below policies for the AWS user to be able to deploy your project when using the GitHub action you have specified:
AWSElasticBeanstalkWebTier
AWSElasticBeanstalkManagedUpdatesCustomerRolePolicy
Adding the above will fix the problem, while also ensuring that you have no future issues when using this GitHub action.
After removing AWSCompromisedKeyQuarantineV2 from the permission list, it works successfully. The reason is this permission actually denies several related operations to the user, see the below JSON for AWSCompromisedKeyQuarantineV2 details:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": [
"ec2:RequestSpotInstances",
"ec2:RunInstances",
"ec2:StartInstances",
"iam:AddUserToGroup",
"iam:AttachGroupPolicy",
"iam:AttachRolePolicy",
"iam:AttachUserPolicy",
"iam:ChangePassword",
"iam:CreateAccessKey",
"iam:CreateInstanceProfile",
"iam:CreateLoginProfile",
"iam:CreatePolicyVersion",
"iam:CreateRole",
"iam:CreateUser",
"iam:DetachUserPolicy",
"iam:PassRole",
"iam:PutGroupPolicy",
"iam:PutRolePolicy",
"iam:PutUserPermissionsBoundary",
"iam:PutUserPolicy",
"iam:SetDefaultPolicyVersion",
"iam:UpdateAccessKey",
"iam:UpdateAccountPasswordPolicy",
"iam:UpdateAssumeRolePolicy",
"iam:UpdateLoginProfile",
"iam:UpdateUser",
"lambda:AddLayerVersionPermission",
"lambda:AddPermission",
"lambda:CreateFunction",
"lambda:GetPolicy",
"lambda:ListTags",
"lambda:PutProvisionedConcurrencyConfig",
"lambda:TagResource",
"lambda:UntagResource",
"lambda:UpdateFunctionCode",
"lightsail:Create*",
"lightsail:Delete*",
"lightsail:DownloadDefaultKeyPair",
"lightsail:GetInstanceAccessDetails",
"lightsail:Start*",
"lightsail:Update*",
"organizations:CreateAccount",
"organizations:CreateOrganization",
"organizations:InviteAccountToOrganization",
"s3:DeleteBucket",
"s3:DeleteObject",
"s3:DeleteObjectVersion",
"s3:PutLifecycleConfiguration",
"s3:PutBucketAcl",
"s3:DeleteBucketOwnershipControls",
"s3:DeleteBucketPolicy",
"s3:ObjectOwnerOverrideToBucketOwner",
"s3:PutAccountPublicAccessBlock",
"s3:PutBucketPolicy",
"s3:ListAllMyBuckets"
],
"Resource": [
"*"
]
}
]
}
As the docs state in order to cache the Maven dependencies with GitHub Actions all we have to use is the actions/cache action like this:
steps:
- uses: actions/checkout#v2
- name: Set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: Cache Maven packages
uses: actions/cache#v2
with:
path: ~/.m2
key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
restore-keys: ${{ runner.os }}-m2
- name: Build with Maven
run: mvn --batch-mode --update-snapshots verify
However using the windows-2016 GitHub Actions environment, this doesn't provides us with a working cache - as the logs states:
Post job cleanup.
"C:\Program Files\Git\usr\bin\tar.exe" --posix --use-compress-program "zstd -T0" -cf cache.tzst -P -C D:/a/spring-boot-admin/spring-boot-admin --files-from manifest.txt --force-local
/usr/bin/tar: C\:\\Users\runneradmin\\.m2\repository: Cannot stat: No such file or directory
/usr/bin/tar: Exiting with failure status due to previous errors
Warning: Tar failed with error: The process 'C:\Program Files\Git\usr\bin\tar.exe' failed with exit code 2
How to fix this?
It seems that the path to the Maven repository isn't correctly initialized. As this issue describes the paths are written with \\ instead of / which GNU tar expects. The fix was already provided in Dec 2020, so it made it to the version v2.1.4. The last version v2.1.3 was released in November. But sadly there is a bug in pointing the v2 to the latest v2.1.4 (as normally expected by GitHub Actions users). Therefore to solve this issue, we need to explicitely specifiy the full actions/cache version v2.1.4 like this:
steps:
- uses: actions/checkout#v2
- name: Set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: Cache Maven packages
uses: actions/cache#v2.1.4
with:
path: ~/.m2
key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
restore-keys: ${{ runner.os }}-m2
- name: Build with Maven
run: mvn --batch-mode --update-snapshots verify
Now it should work like a charm (see logs here).
I'm working on a project where we use Firebase to store data and we're doing some unit tests. I have set a GitHub Action that executes mvn package and mvn test on every push or pull request, and the problem is that I receive the following error when it executes tests:
java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
I have created the secret with the .json and this is how I wrote the GitHub Action:
name: Maven CI/CD
on:
push:
branches: [ develop ]
pull_request:
branches: [ develop ]
jobs:
build_and_test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Set up JDK 15
uses: actions/setup-java#v1
with:
java-version: 15
- name: Cache the Maven packages to speed up build
uses: actions/cache#v1
with:
path: ~/.m2
key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
restore-keys: ${{ runner.os }}-m2
- name: Build project with Maven
run: mvn -B package --file pom.xml
env:
GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
- name: Run (J)Unit tests
run: mvn clean test
env:
GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
I have found that it somehow wasn't using the updated version of the action. Anyway, I solved using google-github-actions/setup-gcloud#master set as follow:
- uses: google-github-actions/setup-gcloud#master
with:
project_id: ${{ secrets.GCP_PROJECT_ID }}
service_account_key: ${{ secrets.GCP_SA_KEY }}
export_default_credentials: true
- name: Set GCP credentials
run: gcloud info
GCP_PROJECT_ID contains the project id and GCP_SA_KEY contains the service account .json file obtained from Firebase.