Develop Reference

Can't start localstack on Gitlab runner with LocalstackTestRunner

I have an issue of running Java integration tests with LocalstackTestRunner on Gitlab agent.
I've taken example from official localstack site:
import cloud.localstack.LocalstackTestRunner;
import cloud.localstack.TestUtils;
import cloud.localstack.docker.annotation.LocalstackDockerProperties;
#RunWith(LocalstackTestRunner.class)
#LocalstackDockerProperties(services = { "s3", "sqs", "kinesis:77077" })
public class MyCloudAppTest {
#Test
public void testLocalS3API() {
AmazonS3 s3 = TestUtils.getClientS3();
List<Bucket> buckets = s3.listBuckets();
}
}
and run it with help of gradle as gradle clean test.
If I run it locally on my Mac Book - all is ok but if it's run on Gitlab agent - there is an issue:
com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to localhost.localstack.cloud:4566 [localhost.localstack.cloud/127.0.0.1] failed: Connection refused (Connection refused)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
My gitlab ci job looks likes as follows:
Localstack_test:
stage: test
services:
- docker:dind
when: always
script:
- ./gradlew clean test --stacktrace
It happens that S3 client can't connect to localhost.localstack.cloud:4566 because docker container created by LocalstackTestRunner is started within parent docker:dind container and AmazonS3 client can't access it. I've tried with other AWS services - result is the same - AWS client can't access localstack endpoint.
I've found some workaround as follows:
add localstack as service in gitlab-ci
add alias to it
expose env variable HOSTNAME_EXTERNAL=alias
Make an implementation of IHostNameResolver to return my alias as HOSTNAME_EXTERNAL specified in gitlab-ci.
Something like that:
Gitlab-ci:
Localstack_test:
stage: test
services:
- docker:dind
- name: localstack/localstack
alias: localstack-it
variables:
HOSTNAME_EXTERNAL: "localstack-it"
when: always
script:
- ./gradlew clean test --stacktrace |& tee -a ./gradle.log
Java IT test:
#RunWith(LocalstackTestRunner.class)
#LocalstackDockerProperties(
services = { "s3", "sqs", "kinesis:77077" },
hostNameResolver = SystemEnvHostNameResolver.class
)
public class MyCloudAppTest {
#Test
public void testLocalS3API() {
AmazonS3 s3 = TestUtils.getClientS3();
List<Bucket> buckets = s3.listBuckets();
}
}
public class SystemEnvHostNameResolver implements IHostNameResolver {
private static final String HOSTNAME_EXTERNAL = "HOSTNAME_EXTERNAL";
#Override
public String getHostName() {
String external = System.getenv(HOSTNAME_EXTERNAL);
return !Strings.isNullOrEmpty(external) ?
external :
new LocalHostNameResolver().getHostName();
}
}
It works but as a result 2 localstack Docker containers are run and internal docker container is still not available. Maybe does somebody know better solution?
STR:
gradle-6.7
cloud.localstack:localstack-utils:0.2.5

Spring Boot - How to start my Spring Boot Application from sister module in a multi-module project?

I have a multi-module project with two projects: backend and client. The backend is a normal Spring Boot Rest API, nothing special. The client module is just a Java Library using the Rest API.
The backend has packaging of "war" as the backend as it uses JSPs, too and needs to be deployed to a servlet container. The backend is still easily testable with #SpringBootTest.
Now I want to have some integration tests inside the client module using the backend module as a sandbox server.
To use all the backend classes in the client module I added
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<configuration>
<attachClasses>true</attachClasses>
</configuration>
</plugin>
and configured the backend as a test dependency in client with classes
In my client/src/test/java I have a helper class which starts up the backend module
#Configuration
public class SandboxServer {
#Bean
public ConfigurableApplicationContext backend() {
return
new SpringApplicationBuilder(BackendApplication.class)
.sources(SandboxServerConfig.class)
.run("spring.profiles.active=sandbox")
}
}
The profile "sandbox" is used to setup a test database etc. But I had more problems. First problem was regarding the document root, so I configured it:
public class SandboxServerConfig
implements WebServerFactoryCustomizer<TomcatServletWebServerFactory> {
#Override
public void customize(TomcatServletWebServerFactory factory) {
factory.setDocumentRoot(new File("../backend/src/main/webapp"));
}
}
But it still does not work as Spring is not picking up backend/src/main/resources/application.properties
That might be correct as it is not in the root classpath of the client module.
So it does not really work. I guess it is not possible to just start up the sibling module in an Integration test.
How can I achieve to start up the sibling spring boot module for integration testing? What is the best practice for szenarios like this?

You can override the application.properties location using TestPropertySource like this:
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = BlaApplication.class)
#TestPropertySource(locations="/path/to/backend/src/main/resources/application.properties")
public class ExampleApplicationTests {
}

I found a much more solid solution. In my sibling Project "frontend" I have a Component which is starting up the backend server in integration mode if and only if it is not already running.
Benefits:
The real WAR is tested
You can start the WAR before in your IDE and let the tests run fast
If you run it with maven it is started up before all tests only once
No build configuration needed (like pre-integration in maven)
process is seperated from Junit runtime so no hassle with complex setups.
Drawbacks:
You need to build the package before you can run any integration test in the frontend. But hey, you should build your package before you test it. That's what integration test is all about.
And here is my SandboxServerProcess.class.
import org.springframework.stereotype.Component;
import javax.annotation.*;
import javax.annotation.*;
import java.io.*;
import java.net.*;
import java.util.*;
#Component
#Profile("integration")
public class SandboxServerProcess {
private static final String WAR = "../backend/target/backend.war";
private final static int PORT = 8081;
private boolean startedByMe;
#PostConstruct
public void start() throws Exception {
if (isStarted()) {
return;
}
testWarExists();
packagedWar("start");
if (waitForStartup()) {
startedByMe = true;
return;
}
throw new RuntimeException("Sandbox Server not started");
}
private void testWarExists() {
File file = new File(WAR);
if (!file.exists()) {
throw new RuntimeException("WAR does not exist:" + file.getAbsolutePath());
}
}
#PreDestroy
public void stop() throws IOException {
if (startedByMe) {
packagedWar("stop");
}
}
private void packagedWar(String command) throws IOException {
ProcessBuilder builder = new ProcessBuilder();
builder.environment().put("MODE", "service");
builder.environment().put("SPRING_PROFILES_ACTIVE", "integration");
builder.environment().put("APP_NAME", "backend");
builder.environment().put("PID_FOLDER", "./");
builder.environment().put("LOG_FOLDER", "./");
List<String> commands = new ArrayList<>();
commands.add(WAR);
commands.add(command);
builder.command(commands);
builder.inheritIO();
builder.redirectErrorStream(true);
builder.start();
}
private boolean isStarted() {
try {
Socket socket = new Socket();
InetSocketAddress sa = new InetSocketAddress("localhost", PORT);
socket.connect(sa, 500);
logger.warn("SandboxServer is started");
return true;
} catch (IOException e) {
return false;
}
}
private boolean waitForStartup() throws InterruptedException {
for (int i = 1; i < 30; i++) {
if (isStarted()) {
return true;
}
logger.warn("SandboxServer not yet ready, tries: " + i);
Thread.sleep(1000);
}
return false;
}
}

Java.lang.ExceptionInInitializerError while listing docker images

I am trying to list docker images using java client api through https://github.com/docker-java/docker-java
This my code snippet
import java.util.List;
import com.github.dockerjava.api.DockerClient;
import com.github.dockerjava.api.model.Image;
import com.github.dockerjava.core.DockerClientBuilder;
public class StartContainers{
DockerClient dockerClient ;
public StartContainers() {
startContainers();
}
public void startContainers() {
dockerClient = DockerClientBuilder.getInstance("tcp://localhost:2375").build();
List<Image> images = dockerClient.listImagesCmd().exec();
for(int i=0; i < images.size(); i++){
System.out.println(images.get(i));
}
}
public static void main(String args[])
{
StartContainers startContainers=new StartContainers();
}
}
and I got this Exception
Exception in thread "main" java.lang.ExceptionInInitializerError at
org.glassfish.jersey.client.JerseyWebTarget.(JerseyWebTarget.java:71)
at
org.glassfish.jersey.client.JerseyClient.target(JerseyClient.java:290)
at
org.glassfish.jersey.client.JerseyClient.target(JerseyClient.java:76)
at
com.github.dockerjava.jaxrs.JerseyDockerCmdExecFactory.init(JerseyDockerCmdExecFactory.java:234)
at
com.github.dockerjava.core.DockerClientImpl.withDockerCmdExecFactory(DockerClientImpl.java:161)
at
com.github.dockerjava.core.DockerClientBuilder.build(DockerClientBuilder.java:47)
at
com.cit.security.controllers.StartContainers.startContainers(StartContainers.java:18)
at
com.cit.security.controllers.StartContainers.(StartContainers.java:13)
at
com.cit.security.controllers.StartContainers.main(StartContainers.java:29)
Caused by: java.lang.RuntimeException: Unable to create jax-rs
RuntimeDelegate at
javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:149)
at javax.ws.rs.core.UriBuilder.(UriBuilder.java:56)

Can you describe the environment where you are running that code?
I managed to run that code (as a maven project) with success on a Debian host, running the docker daemon as root (sudo nohup docker daemon -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock &) and started a dummy container.

java.lang.ClassNotFoundException: org.zeromq.ZContext when trying to start windows service

I have a basic Maven java app that I created and it depends on JeroMQ which is a full Java implemenetation of ZeroMQ. Since I also need to wrap this java app as a windows service, I chose to use Apache Commons Daemon and specifically, followed this excellent example: http://web.archive.org/web/20090228071059/http://blog.platinumsolutions.com/node/234 Here's what the Java code looks like:
package com.org.SubscriberACD;
import java.nio.charset.Charset;
import org.zeromq.ZContext;
import org.zeromq.ZMQ;
import org.zeromq.ZMQ.Socket;
/**
* JeroMQ Subscriber for Apache Commons Daemon
*
*/
public class Subscriber
{
/**
* Single static instance of the service class
*/
private static Subscriber subscriber_service = new Subscriber();
/**
* Static method called by prunsrv to start/stop
* the service. Pass the argument "start"
* to start the service, and pass "stop" to
* stop the service.
*/
public static void windowsService(String args[]) {
String cmd = "start";
if(args.length > 0) {
cmd = args[0];
}
if("start".equals(cmd)) {
subscriber_service.start();
}
else {
subscriber_service.stop();
}
}
/**
* Flag to know if this service
* instance has been stopped.
*/
private boolean stopped = false;
/**
* Start this service instance
*/
public void start() {
stopped = false;
System.out.println("My Service Started "
+ new java.util.Date());
ZContext context = new ZContext();
Socket subscriber = context.createSocket(ZMQ.SUB);
subscriber.connect("tcp://localhost:5556");
String subscription = "MySub";
subscriber.subscribe(subscription.getBytes(Charset.forName("UTF-8")));
while(!stopped) {
System.out.println("My Service Executing "
+ new java.util.Date());
String topic = subscriber.recvStr();
if (topic == null)
break;
String data = subscriber.recvStr();
assert(topic.equals(subscription));
System.out.println(data);
synchronized(this) {
try {
this.wait(60000); // wait 1 minute
}
catch(InterruptedException ie){}
}
}
subscriber.close();
context.close();
context.destroy();
System.out.println("My Service Finished "
+ new java.util.Date());
}
/**
* Stop this service instance
*/
public void stop() {
stopped = true;
synchronized(this) {
this.notify();
}
}
}
Then I created the following folder structure just like the tutorial suggested:
E:\SubscriberACD
\bin
\subscriberACD.exe
\subscriberACDw.exe
\classes
\com\org\SubscriberACD\Subscriber.class
\logs
I then navigated to the bin directory and issued the following command to install the service:
subscriberACD.exe //IS//SubscriberACD --Install=E:\SubscriberACD\bin\subscriberACD.exe --Descriptio
n="Subscriber using Apache Commons Daemon" --Jvm=c:\glassfish4\jdk7\jre
\bin\server\jvm.dll --Classpath=E:\SubscriberACD\classes --StartMode=jvm
--StartClass=com.org.SubscriberACD.Subscriber --StartMethod=windowsSer
vice --StartParams=start --StopMode=jvm --StopClass=com.org.SubscriberA
CD.Subscriber --StopMethod=windowsService --StopParams=stop --LogPath=E:\SubscriberACD\logs --StdOutput=auto --StdError=auto
The install works fine since I can see it in Windows Services. However, when I try to start it from there, I get an error saying "Windows cannot start the SubscriberACD on Local Computer".
I checked the error logs and see the following entry:
2016-04-14 14:38:40 Commons Daemon procrun stderr initialized
Exception in thread "main" ror: org/zeromq/ZContext
at com.org.SubscriberACD.Subscriber.start(Subscriber.java:57)
at com.org.SubscriberACD.Subscriber.windowsService(Subscriber.java:33)
Caused by: java.lang.ClassNotFoundException: org.zeromq.ZContext
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
... 2 more
It's worth noting that JeroMQ is currently a jar under my Maven Dependencies. I configured it from my POM.xml file.
I think the problem might be that my service doesn't have access to the JeroMQ jar that is under my Maven Dependencies. My assumption is that the class file doesn't contain the dependencies. So what I tried was exporting my entire project as a jar and stuck that baby under E:\SubscriberACD\classes\
So my structure now looks like this:
E:\SubscriberACD
\bin
\subscriberACD.exe
\subscriberACDw.exe
\classes
\com\org\SubscriberACD\
\Subscriber.class
\Subscriber.jar
\logs
However, that didn't fix the issue. Can anyone shed some light on this?

Change your --Classpath argument to :
--Classpath=E:\SubscriberACD\classes\your-jar-filename.jar
You almost certainly have other jarfiles you'll need, so just append them to the end of the --Classpath using ; (semi-colon) delimiters...
--Classpath=E:\SubscriberACD\classes\your-jar-filename.jar;e:\other-dir\classes\some-other.jar;etc...

jgit need 2 arguments error message

I am trying a simple java class to test the functionality of jGit (see below).
import org.eclipse.jgit.lib.Repository;
import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
import java.io.File;
import java.io.IOException;
public class CreateRepository {
public static void main( String[] args ){
Repository myrepo = createRepository("/mypath");
}
public static Repository createRepository(String repoPath) {
FileRepositoryBuilder builder = new FileRepositoryBuilder();
Repository repo = null;
try {
repo = builder.setGitDir(new File(repoPath))
.readEnvironment() // scan environment GIT_* variables
.findGitDir() // scan up the file system tree
.build();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return repo;
}
}
When i run this on Eclipse Indigo with latest jgit jar in my build path i get the error message "Need 2 arguments" - nothing else, no exceptions! :S
Thanks for any help in advance.

The only part of JGit which displays that error message is in the main() function of MyersDiff.
/**
* #param args two filenames specifying the contents to be diffed
*/
public static void main(String[] args) {
if (args.length != 2) {
System.err.println(JGitText.get().need2Arguments);
System.exit(1);
}
// ...
}
So check your classpath and make sure your project (and your main()) are before the jgit.jar, and that you don't somehow calls the wrong main().

First look into package org.eclipse.jgit.api.
The easiest start is from class Git:
// clone a repository
Git git = Git.cloneRepository().setURI("git://yourserver/repo.git").call();
// init a fresh new repository in the current directory
Git git = Git.init().call();
// open a repository on your disk
Git git = Git.open(new File("/path/of/repo");
Then explore the commands available on the git object you get
from these starting points.