Develop Reference

How to set up a selenium test of download a file in Gitlab CI

I am writing a selenium test to verify a file being downloaded. It works fine in locally. And I can easily access the file both through the 'target' folder and inside the container /home/seluser/Downloads.
The test script is:
#BeforeMethod
public void setUp() throws MalformedURLException {
folder = new File("target");
for(File file : folder.listFiles()) {
file.delete();
}
System.setProperty("webdriver.chrome.driver", "chromedriver.exe");
ChromeOptions options = new ChromeOptions();
Map<String, Object> prefs = new HashMap<String, Object>();
prefs.put("profile.default_content_settings.popups", 0);
prefs.put("download.default_directory", folder.getAbsolutePath());
options.setExperimentalOption("prefs", prefs);
DesiredCapabilities cap = new DesiredCapabilities();
cap.setBrowserName("chrome");
cap.setCapability(ChromeOptions.CAPABILITY, options);
//driver = new ChromeDriver(cap);
//driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), cap);
driver = new RemoteWebDriver(new URL("http://docker:4444/wd/hub"), cap);
}
#AfterMethod
public void tearDown() {
driver.quit();
}
#Test
public void downloadFileTest() throws InterruptedException {
driver.get("http://the-internet.herokuapp.com/download");
driver.findElement(By.linkText("some-file.txt")).click();
Thread.sleep(2000);
File listOffFiles[] = folder.listFiles();
Assert.assertTrue(listOffFiles.length > 0);
for(File file : listOffFiles) {
Assert.assertTrue(file.length() > 0);
}
}
Let me explain a little. First I create a folder named "target" in the project root repository. Then I set the download path in the chrome docker comtainer via container volumes in docker-compose file.
version: "3"
services:
chrome:
image: selenium/node-chrome:4.0.0-20211013
container_name: chrome
shm_size: 2gb
depends_on:
- selenium-hub
volumes:
- ./target:/home/seluser/Downloads
environment:
- SE_EVENT_BUS_HOST=selenium-hub
- SE_EVENT_BUS_PUBLISH_PORT=4442
- SE_EVENT_BUS_SUBSCRIBE_PORT=4443
- SE_NODE_GRID_URL=http://localhost:4444
ports:
- "6900:5900"
selenium-hub:
image: selenium/hub:4.0.0-20211013
container_name: selenium-hub
ports:
- "4444:4444"
This setting works fine locally. when I run it in the gitlab CI, i cannot push a empty fold to gitlab, so i have to create a file and store it in the folder and push it to gitlab. But it the test script, I delete this file in the setup stage in case it disturb the assertion. But the pipeline fails. The result does not give me more details , just said the assertionException. Here is the gitlab-ci.yml:
image: adoptopenjdk/openjdk11
stages:
- gradle-build
- docker-test
.gradle_template: &gradle_definition
variables:
GRADLE_OPTS: "-Dorg.gradle.daemon=false"
before_script:
- export GRADLE_USER_HOME=${CI_PROJECT_DIR}/.gradle
gradle-build:
<<: *gradle_definition
stage: gradle-build
script:
- chmod +x ./gradlew
- ./gradlew --build-cache assemble
cache:
key: "$CI_COMMIT_REF_NAME"
paths:
- build
- .gradle
artifacts:
paths:
- build/libs/*.jar
expire_in: 1 day
only:
- feature/multi-browsers
chrome-test:
stage: docker-test
image:
name: docker/compose:1.29.2
entrypoint: [ "/bin/sh", "-c" ]
services:
- docker:19.03.12-dind
variables:
DOCKER_TLS_CERTDIR: ""
DOCKER_DRIVER: overlay2
DOCKER_HOST: tcp://docker:2375/
cache:
key: "$CI_COMMIT_REF_NAME"
policy: pull
paths:
- build
- .gradle
dependencies:
- gradle-build
before_script:
- docker info
- docker-compose --version
script:
- apk add --no-cache docker-compose
- apk add openjdk11
- docker-compose down
- docker-compose up --build --force-recreate --no-deps -d
- echo "Hello, chrome-test"
- chmod +x ./gradlew
- ./gradlew :test --tests "LogInTest_chrome"
artifacts:
when: always
reports:
junit: build/test-results/test/**/TEST-*.xml
paths:
- build/reports/*
expire_in: 1 day
only:
- feature/multi-browsers
I wonder if someone has experience with this download test in gitlab CI. I think the download path I set maybe not work in gitlab CI. I even have no ideas how to check if a file is downloaded or not it gitlab CI.

I dont exactly know what you expect to run on http://docker:4444, but that seems not correct at: driver = new RemoteWebDriver(new URL("http://docker:4444/wd/hub"), cap);.
I geuss you want to connect with the selenium-hub instead. Personally I always prefer the use of GitLab services over running docker-compose in your pipeline. Maybe this answer about running E2E tests with Docker in GitLab helps.

Statement : 'Let me explain a little. First I create a folder named "target" in the project root repository.' I assume that you are creating this by executing folder = new File("target"); This actually will not create the directory, what you need to apply something like the following:
String directory = "/target";
directory.mkdir()
However, when this part of your code runs, this will not actually create the directory due to your docker-compose.yml where the volume mapping configuration from your chrome-node is mapped to the host /target directory.
volumes:
./target:/home/seluser/Downloads
This will already exist, as docker will create it, so the java code will return a boolean value of 'false' (as it does not need to create the directory) and will proceed to the next step.
docker-compose.yml: there is a mapping for your 'selenium/node-chrome' image where you have mapped /target to /home/seluser/Downloads. As this directory does not initially exist when 'docker-compose up' is run, docker will create the directory on the host, so it can be mapped to your desired volume. Here lies 2 problems, first of all docker runs as root and will create the new '/target' directory on the host, but only gives it linux permissions 'drwxr-xr-x' (755). This means that only root user can write to that directory. So even though the configuration has mapped the volume, when you download to the directory as 'seluser', it will not be able to write and the browser will be returned with a 'Permission denied' response.
The other issue is the java code declares to the RemoteWebDriver that the download will need to be saved to prefs.put("download.default_directory", folder.getAbsolutePath());, which has been declared as "target", this is an issue as the chrome-node where the browser resides does not have a directory called '/target', so will fail to
download anyway. This is the cause of your 'assertion exception'
Propose that the following should be done for efficiency and stability:
Update the gitlab-ci.yml prior to the 'docker-compose down & docker-compose up --build --force-recreate --no-deps -d' commands
mkdir /target
chmod -R 777 /target
This will ensure that the directory can be written to by 'seluser' on the node-chrome container
Update the docker-compose.yml
For Windows Host => GitLab CI Docker(WSL2) => Chrome-Node Docker
volumes:
./target:/home/seluser/Downloads
For Linux Host => GitLab CI Docker(Linux) => Chrome-Node Docker
volumes:
/target:/home/seluser/Downloads
This will ensure that the Chrome browser can find a directory named '/target' and that the java test can see a file written to the host '/target' directory
Also your selenium-hub configuration needs to be updated, ports for publish events and subscribe events have not been mapped, update to the following:
selenium-hub:
image: selenium/hub:4.0.0-20211013
container_name: selenium-hub
ports:
- "4442:4442"
- "4443:4443"
- "4444:4444"

How do I get Java testcontainers to work in Docker Multistage builds?

I have a problem similar to Run (Docker) Test Container in gitlab with Maven. The difference is that rather than my script running mvn directly it runs a docker multistage build that runs the test inside of the docker image. Unfortunately this doesn't appear to work for the PostgreSQL Test Container.
Dockerfile
#############
### build ###
#############
# base image
FROM maven:3-jdk-11 as build
# set working directory
WORKDIR /app
# add app
COPY . .
RUN export MAVEN_OPTS="-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true" && export MAVEN_CLI_OPTS="-B -U --batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"
RUN mvn $MAVEN_CLI_OPTS clean install
############
### prod ###
############
# Yea this isn't right, but it crashes before it gets to this point. This is for example purposes only.
FROM openjdk:15-jdk-alpine
COPY --from=build /app/reproducer-testcontainer/target/reproducer-testcontainer.jar /reproducer-testcontainer.jar
CMD java -jar reproducer-testcontainer.jar
When I run mvn clean install it works properly and runs my test using the PostgreSQL Test Container. However, when I run docker build . it fails at the mvn clean install step with the below stack trace.
Stack trace:
13:05:01.250 [main] ERROR org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - ping failed with configuration Environment variables, system properties and defaults. Resolved:
dockerHost=unix:///var/run/docker.sock
apiVersion='{UNKNOWN_VERSION}'
registryUrl='https://index.docker.io/v1/'
registryUsername='root'
registryPassword='null'
registryEmail='null'
dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'
due to org.rnorth.ducttape.TimeoutException: Timeout waiting for result with exception
org.rnorth.ducttape.TimeoutException: Timeout waiting for result with exception
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:51)
<snip>
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.io.IOException: com.sun.jna.LastErrorException: [2] No such file or directory
at org.testcontainers.shaded.org.scalasbt.ipcsocket.UnixDomainSocket.<init>(UnixDomainSocket.java:62)
<snip>
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.sun.jna.LastErrorException: [2] No such file or directory
at org.testcontainers.shaded.org.scalasbt.ipcsocket.UnixDomainSocketLibrary.connect(Native Method)
at org.testcontainers.shaded.org.scalasbt.ipcsocket.UnixDomainSocket.<init>(UnixDomainSocket.java:57)
... 35 common frames omitted
In my CI pipeline I'd like to only run docker build . and not worry about having another stage that does the mvn clean install.
How do I fix the configuration to get the java PostgreSQL Testcontainers to work inside of a Docker build so that I can use it in a multi-stage build?
Full Code example: https://gitlab.com/raymondcg/reproducer-testcontainer

Not really Testcontainers related.
Testcontainers requires a valid Docker daemon. When you build images, there is no daemon mounted into the image build context.
You can easily verify that by doing:
RUN curl --unix-socket /var/run/docker.sock http:/_/_ping
Make this command return "OK" (no need to run the Testcontainers code), and your tests will pass as well.

You can overwrite testcontainers default docker host by adding:
ENV DOCKER_HOST=tcp://host.docker.internal:2375
to your build stage.

Concourse CI + MVN

I am trying to create a pipeline using Concourse CI using Maven. The pipeline should:
Get the code from git.
Build and run test using maven.
the project generates html report in target folder
above steps are executing properly. The question how to access the output i.e target folder generated by the maven project.
I am not able to access the generated folder and copy to the required folder for later usage.
File pipeline.yml:
resources:
- name: branch-master
type: git
source:
uri: {{git-url}}
branch: master
jobs:
- name: MavenJob
serial: true
plan:
- get: branch-master
trigger: true
- task: mvn-test
privileged: true
file: branch-dev/AppDemo/test.yml
File test.yml:
platform: linux
image_resource:
type: docker-image
source:
repository: maven
tag: latest
inputs:
- name: branch-master
outputs:
- name: mvn-output
run:
path: "mvn"
args: ["-f", "branch-master/AppDemo/pom.xml", "test"]
Please some body help me.
Thanks in advance.

For sake of simplicity and unambiguity I renamed your git repo to project. Let's assume it contains concourse's yamls. And there is a directory AppDemo with java-maven app
project/pipeline.yml:
resources:
- name: project
type: git
source:
uri: {{git-url}}
branch: master
jobs:
- name: MavenJob
serial: true
plan:
- get: project
trigger: true
- task: mvn-test
privileged: true
file: project/test.yml
the job above should locate and trigger the test.yml task
project/test.yml:
platform: linux
image_resource:
type: docker-image
source:
repository: maven # let's hope bin/bash is available there. if no, use sh
tag: latest
inputs:
- name: project # project is your git-repo. all paths are relative to it's location
outputs:
- name: mvn-output
run:
path: /bin/bash
args:
- project/test-script.sh:
project/test-script.sh:
_ROOT=$(pwd)
echo "starting test-script from directory: $ROOT"
cd _ROOT/AppDemo
mvn test
if you want to pass outputs of maven job somewhere further, then you should just copy all those files into mvn-output directory later in the script.
I would also recommend using this bash-script as wrapper rather than raw "maven call" - it's more convenient for debugging the whole process as you can, e.g. echo'ing paths.
in case of any error, if container still alive, try hijacking into it to see what has actually happened there and where files are located:
fly -t <target> hijack -u <url-of-failed-job-from-your-browser>
hope this helps

is it possible to deploy a simple java web application to a local server using gitlab-ci without docker?

I'm simply trying to copy the war to the webapps folder of my tomcat and restart the server, something like
copy /y .\WebAppFolder\target\WebApp*.war C:\server\webapps\ROOT.war
He can't find this specified path when try to copy, but locally this command works.
My runner is on windows.

The problem is that I was trying to use the "target" created by the previous stage, but the runner deletes everything that is not tracked at the end of each stage.
I add this code in the stage was generate the war so the runner does not destroy the directory
artifacts:
untracked: true
paths:
- WebApp/target
Getting something like that
buildWebApp:maven:
stage: buildWebApp
tags:
- tag
script: mvn clean install -f WebApp/pom.xml
artifacts:
untracked: true
paths:
- WebApp/target
only:
- master
test:
stage: test
tags:
- tag
script:
- copy /y my copy
- mvn test -f testes/pom.xml
only:
- master
Sorry but i lost the link of gitlab doc where i find this solution

Using Gitlab pages to publish jar file

I am using gitlab.io. I have a Java project which builds using Maven.
I have setup the following yml file to do the builds:
image: maven:3-jdk-7
build:
script:
- "mvn install -B"
- mkdir .public
- cp -r ./gl_pages/* .public
- mv ./target/MyProg_installer-0.0.1-SNAPSHOT.jar .public/Prog_Latest.jar
- mv .public public
artifacts:
paths:
- public
only:
- master
If I go to the following URL's I can see index.html without a problem.
https://<<MYUSER>>.gitlab.io/<<MY_PROJECT>/
https://<<MYUSER>>.gitlab.io/<<MY_PROJECT>/index.html
I expected
https://<<MYUSER>>.gitlab.io/<<MY_PROJECT>/Prog_Latest.jar
to enable me to download the jar file, but this url simply returns a 404.
I have downloaded the zip of the artifacts from the build and this works without an issue.
Am I doing something wrong?