Develop Reference

Test containers: ignore parent `EXPOSE` instruction from Dockerfile

I'm trying to run Couchbase v.5.1.1 docker container for test purposes via Test container with fixed exposed ports, like:
trait CouchbaseTestEnvironment extends ForAllTestContainer {
this: Suite =>
def couchbaseContainer: FixedHostPortGenericContainer = {
val consumer = new Slf4jLogConsumer(LoggerFactory.getLogger(getClass))
/*
* Couchbase should know which ports are exposed for client, because this is how it exposes services.
* E.g. client ask only for on port - say 8091. And query service port is 8093. So client, won't ask for every port,
* instead CB will tell client on which port query service exposed, that's why CB should be aware about port mapping.
* That's why we need to give CB port mappings
*
* See for more details:
* https://stackoverflow.com/questions/59277436/couchbase-in-docker-for-integration-tests-make-the-ports-8092-8093-8094-and-8
*/
def randomPort: Int = {
val (from, to) = (32768, 35000) //linux private port range
from + Random.nextInt(to - from)
}
val random8091Port = randomPort
val random8092Port = randomPort
val random8093Port = randomPort
val random8094Port = randomPort
val random11210Port = randomPort
val container = FixedHostPortGenericContainer(
imageName = "couchbase:community-5.0.1",
exposedHostPort = random8091Port,
exposedContainerPort = random8091Port,
env = Map(
"COUCHBASE_RANDOM_PORT_8091" -> random8091Port.toString,
"COUCHBASE_RANDOM_PORT_8092" -> random8092Port.toString,
"COUCHBASE_RANDOM_PORT_8093" -> random8093Port.toString,
"COUCHBASE_RANDOM_PORT_8094" -> random8094Port.toString,
"COUCHBASE_RANDOM_PORT_11210" -> random11210Port.toString
)
)
container.container.withFixedExposedPort(random8092Port, random8092Port)
container.container.withFixedExposedPort(random8093Port, random8093Port)
container.container.withFixedExposedPort(random8094Port, random8094Port)
container.container.withFixedExposedPort(random11210Port, random11210Port)
container.container.withLogConsumer(consumer)
container
}
}
So as you can see 5 FIXED ports should be exposed.
But, when I'm running tests I actually can see, that instead other ports exposed with random ports:
docker ps
f4fc1ce06544 couchbase:community-5.0.1 "/entrypoint.sh /opt…" 59 seconds ago Up 1 second 0.0.0.0:55264->8091/tcp, 0.0.0.0:55263->8092/tcp, 0.0.0.0:55262->8093/tcp, 0.0.0.0:55261->8094/tcp, 0.0.0.0:55260->11207/tcp, 0.0.0.0:55259->11210/tcp, 0.0.0.0:55258->11211/tcp, 0.0.0.0:55257->18091/tcp, 0.0.0.0:55256->18092/tcp, 0.0.0.0:55255->18093/tcp, 0.0.0.0:55254->18094/tcp unruffled_mendel
03b491ac2ea8 testcontainersofficial/ryuk:0.3.0
So as you can see another ports was exposed, and mapped to random ports instead fixed.
As far as I understand, test containers, ignores ports I gave, and instead exposes ports from Couchbase Dockerfile: https://github.com/couchbase/docker/blob/master/community/couchbase-server/5.1.1/Dockerfile#L74
EXPOSE 8091 8092 8093 8094 8095 8096 11207 11210 11211 18091 18092 18093 18094 18095 18096
Can I somehow force Test containers to ignore EXPOSE instruction?
Partially helped question: Couchbase in docker for integration tests: Make the ports 8092, 8093, 8094 and 8095 configurable to be able to use docker’s random ports

Can I somehow force Test containers to ignore EXPOSE instruction?
I don't know if there is a simple configuration option for this, but a workaround solution I found is to use an advanced feature of the docker-java create container command customization. I'm providing an example in Java, translate it to Scala yourself, please. Apply it as the last command before returning a container object from your function:
container.withCreateContainerCmdModifier(
cmd -> cmd.getHostConfig().withPublishAllPorts(false)
);
The main point here is the usage of .withPublishAllPorts(false). From my understanding, this is the same as --publish-all (or -P) arguments of the docker run command. Testcontainers library sets this value to true by default. This modification overrides it to false.
With this configuration no ports are published at all for your example, not the 5 fixed as expected:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2ee4fb91b97c couchbase:community-5.0.1 "/entrypoint.sh couc…" 33 seconds ago Up 32 seconds 8091-8094/tcp, 11207/tcp, 11210-11211/tcp, 18091-18094/tcp trusting_keldysh
This is because in the answer you provided, the author created a special custom docker image of couchbase, which "understands" environment variables like COUCHBASE_RANDOM_PORT_8091. Your code uses the standard couchbase image couchbase:community-5.0.1, which basically just ignores these environment variables. So in order to run counchbase on not standard internal ports, you need to build a custom image with the "magic" configure-node.sh script, which tunes couchbase config using values provided in environment variables.
I hope it helps anyhow :)

Creating a mysql docker container setting env variables

I am using the spotify-docker-client to create and start a mysql container for testing. It works perfect, but I am having a hard time trying to find how to set certain values to connect to the database like MYSQL_ROOT_PASSWORD, MYSQL_DATABASE, MYSQL_USER, and MYSQL_PASSWORD. This is my code:
final ContainerConfig containerConfig = ContainerConfig.builder()
.hostConfig(hostConfig)
.image(image)
.env("MYSQL_ROOT_PASSWORD","testrootpwd","MYSQL_DATABASE", "test", "MYSQL_USER", "test", "MYSQL_PASSWORD", "test")
.build();
LOG.debug("Creating container for image: {}", image);
final ContainerCreation creation = this.docker.createContainer(containerConfig);
I am assuming that .env call is to set environment variables. And according to the mysql container documentation, setting those env variables is the way to do it:
https://hub.docker.com/_/mysql
But still, I can't connect to the container, I connected to bash and I see that those env variables are not set.
Does anyone know how to do it?
I could create a dockerfile and create my own image, but I don't want to do that, I want to do it with the spotify client.

This client uses docker API, so if the client is lacking documentation you can always check the original API.
Check the CREATE A CONTAINER section in Docker Engine API.
You can see that there is a JSON request example with env field:
"Env": [
"FOO=bar",
"BAZ=quux"
],
So my guess is that you can do just that in your Java code:
final ContainerConfig containerConfig = ContainerConfig.builder()
.hostConfig(hostConfig)
.image(image)
.env("MYSQL_ROOT_PASSWORD=testrootpwd", "MYSQL_DATABASE=test", ...)
.build();
P.S. Also please note what the documentation says regarding this param:
A list of environment variables to set inside the container in the
form ["VAR=value", ...]. A variable without = is removed from the
environment, rather than to have an empty value.
Might help you avoiding bugs later.

How to run downloaded App Router via Service Marketplace

I downloaded XS_JSCRIPT14_10-70001363 package from Service Marketplace.
Please suggest me how to run this App Router Login form with localhost
I am trying with npm startcommand, but getting UAA service exception. How to handle from localhost.

When you download the approuter, either via npm or service marketplace you have to provide two additional files for a basic setup inside the AppRouter directory (besides package.json, xs-app.json, etc.).
The default-services.json holds the variables that tell the approuter where to find the correct authentication server (e.g., XSUAA). You have to provide at least the clientid, clientsecret, and URL of the authorization server as part of this file like this:
{
"uaa": {
"url" : "http://my.uaa.server/",
"clientid" : "client-id",
"clientsecret" : "client-secret",
"xsappname" : "my-business-application"
}
}
You can get this parameters, for example, after binding on SAP Cloud Platform, CloudFoundry your application to an (empty) instance of XSUAA where you can retrieve the values via cf env <appname> from the `VCAP_SERVICES/xsuaa' properties (they have exactly the same property names).
In addition, you require the default-env.json file which holds at least the destination variable to which backend microservice you want to send the received Json Web Token to. It may look like this:
{
"destinations": [ {
"name": "my-destination", "url": "http://localhost:1234", "forwardAuthToken": true
}]
}
Afterwards, inside the approuter directory you can simply run npm start which runs the approuter per default under http://localhost:5000. It also writes nice console output you can use to debug the parameters above.

EDIT: Turns out I was incorrect, it is apparently possible to run the approuter locally.
First of all, here is the documentation for the approuter: https://help.sap.com/viewer/65de2977205c403bbc107264b8eccf4b/Cloud/en-US/01c5f9ba7d6847aaaf069d153b981b51.html
As far as I understood, you need to provide to files to the approuter for it to run locally, default-services.json and default-env.json (put them in the same directory as your package.json.
The default-services.json has a format like this:
{
"uaa": {
"url" : "http://my.uaa.server/",
"clientid" : "client-id",
"clientsecret" : "client-secret",
"xsappname" : "my-business-application"
}
}
The default-env.json is simply a json file holding the environment variables that the approuter needs to access, like so:
{
"VCAP_SERVICES": <env>,
...
}
Unfortunately, the documentation does not state which variables are required, therefore I cannot provide you with a working example.
Hope this helps you! Should you manage to get this running, I'm sure others would appreciate if you share your knowledge here.

Google Cloud public hostname

Is there any solution to get a public hostname in google cloud like in other cloud platforms?
Currently the machine name is:
computername.c.googleprojectid.internal
but I want something like in Amazon or in Azure:
computername.cloudapp.net

You can use the Google Cloud DNS service to update the DNS record for your host on startup. (You could also use a service like dyn-dns, but I'm assuming that you want to us the Google tools where possible.) It looks like you'd want to use the "create change" API, using a service account associated with your VM. This would look something like:
POST https://www.googleapis.com/dns/v1beta1/projects/*myProject*/managedZones/*myZone.com*/changes
{
"additions": [
{
"name": "computername.myZone.com.",
"type": "A",
"ttl": 600,
"rrdatas": [
"200.201.202.203"
]
}
],
"deletions": [
],
}
Note that 200.201.202.203 needs to be the external IP address of your VM.

Launching Instance: VPC security groups may not be used for a non-VPC launch

I'm attempting to create an instance in another region, but I get this error:
AWS Error Code: InvalidParameterCombination, AWS Error Message: VPC security groups may not be used for a non-VPC launch
Here is the code I'm executing.
RunInstancesRequest instancereq = new RunInstancesRequest();
instancereq.setInstanceType("m3.medium");
instancereq.setImageId("ami-37b1b45e");
instancereq.setMinCount(1);
instancereq.setMaxCount(1);
ArrayList<String> secgroup = new ArrayList<String>();
instancereq.setKeyName("testkey");
secgroup.add("testdefault");
instancereq.setSecurityGroups(secgroup);
instancereq.setPlacement(getAzPlacement());
RunInstancesResult instanceresult = ec2.runInstances(instancereq);
I've also tried, instead of using the name "testdefault", using the actual groupid (sg-########), but I'll get an error saying that security group doesn't exist (which is wrong, it does). Which, based on the API doc, if using a non-default VPC, you should pass the actual groupid but I'll get an error like this:
InvalidGroup.NotFound, AWS Error Message: The security group 'sg-########' does not exist
If I use "default" as the setSecurityGroups it will use the default VPC. It just doesn't seem like like the groupid I'm passing, despite it being accurate.
Also, if I comment out the setSecurityGroups code, and use setSubnetId instead and pass the subnet id, it will create the instance just fine, but it goes into the "default" security group, not "testdefault" like I want.
All I'm trying to accomplish is creating an instance and having it use the already existing VPC group.

My Answer will focus on below statement:
All I'm trying to accomplish is creating an instance and having it use the already existing VPC group.
So, as I understand, you want to launch an instance in a non-default VPC and assign it an existing VPC security group to it.
I am not a java guy, but I could do what you wanted in ruby as below.
require 'aws-sdk-core'
Aws.config = {
:access_key_id => "my_access_key",
:secret_access_key => "my_secret_key",
:region => 'us-west-2'
}
ec2 = Aws::EC2.new
ec2.run_instances(
min_count: 1,
max_count: 1,
image_id: 'ami-8635a9b6',
instance_type: 't1.micro',
placement: {
availability_zone: 'us-west-2a'
},
network_interfaces: [
{
subnet_id: 'subnet-e881bd63',
groups: ['sg-fd53bf5e'],
device_index: 0,
associate_public_ip_address: true
}
],
key_name: 'my-key'
).each do |resp|
resp.instances.each do |x|
puts x.instance_id
end
end
Although this is a Ruby code, it is pretty straight forward and should give you some clear hints on how to go about doing it in Java as all these AWS SDKs are polling the same web service APIs.
I guess, the things that you should be concentrating in above code is:
:region => 'us-west-2'
and
placement: {
availability_zone: 'us-west-2a'
},
network_interfaces: [
{
subnet_id: 'subnet-e881bd63',
groups: ['sg-fd53bf5e'],
device_index: 0,
associate_public_ip_address: true
}
],
Make sure you explicitly specify the region.
Check how I have defined the subnet ID and security group ID. This code will launch my EC2 instance in subnet-e881bd63 of my VPC and will apply VPC security group ID sg-fd53bf5e to its 0th network interface. Besides, it will also assign a public IP address to my instance. (by default, it will not assign a public IP address when you launch instances in VPC).
FYI. When you launch instances in VPC, you must provide Security group ID instead of security group name.

This same error occurs using the command line program so I'm adding a separate answer helped by QuickNull. Simply make sure you specify the security group and subnet. For example:
aws ec2 run-instances --image-id ami-XXXXXXXX --count 1 --instance-type t1.micro --key-name XXXXXXXX --security-group-ids sg-XXXXXXXX --subnet-id subnet-XXXXXXXX

You can't specify security group names for VPC launch (setSecurityGroups). For a non-default VPC, you must use security group IDs instead. See EC2 run-instances page (withSecurityGroupIds , or --security-group-ids from CLI).
When you specify a security group for a nondefault VPC to the CLI or the API actions, you must use the security group ID and not the security group name to identify the security group.
See: Security Groups for EC2-VPC
Related:
Terraform throws "groupName cannot be used with the parameter subnet" or "VPC security groups may not be used for a non-VPC launch"

Thanks to #slayedbylucifer for his ruby code, here's the java solution for reference:
// Creates an instance in the specified subnet of a non-default VPC and using the
// security group with id sg-1234567
ec2.runInstances(new RuntInstancesRequest()
...
.withSubnetId("subnet-1234abcd")
.withSecurityGroupIds("sg-1234567"));