Develop Reference

Keycloak with custom user federation docker deployment

I'm trying to deploy Keycloak with custom user federation provider on docker. I want to use external database as an additional source of user authentication. I've tested configuration on my host (whole project extracted from .tar.gz) and it's working - I can search users from external db in admin panel or log into keycloak.
The problem is, when I run my container I get following error:
12:36:36,127 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "custom-user-storage-jpa.jar")]) - failure description: {
"WFLYCTL0412: Required services that are not installed:" => ["jboss.naming.context.java.jboss.datasources.ExternalPostgresDS"],
"WFLYCTL0180: Services with missing/unavailable dependencies" => [
"jboss.persistenceunit.\"custom-user-storage-jpa.jar#custom-user-storage-jpa\" is missing [jboss.naming.context.java.jboss.datasources.ExternalPostgresDS]",
"jboss.persistenceunit.\"custom-user-storage-jpa.jar#custom-user-storage-jpa\".__FIRST_PHASE__ is missing [jboss.naming.context.java.jboss.datasources.ExternalPostgresDS]"
]
so I assume my jar do not see datasource configured in standalone.xml with (with name ExternalPostgresDS)
There is persistance.xml in my custom-user-storage provider, later built in jar by maven clean install
<?xml version="1.0" encoding="UTF-8" ?>
<persistence version="2.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
http://java.sun.com/xml/ns/persistence
http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">
<persistence-unit name="custom-user-storage-jpa">
<jta-data-source>java:jboss/datasources/ExternalPostgresDS</jta-data-source>
<properties>
<property name="hibernate.hbm2ddl.auto" value="none" />
<property name="hibernate.show_sql" value="false" />
<property name="hibernate.dialect" value="org.hibernate.dialect.PostgreSQL95Dialect" />
</properties>
</persistence-unit>
</persistence>
There is also section in standalone.xml defining that datasource and driver
<xa-datasource jndi-name="java:jboss/datasources/ExternalPostgresDS" pool-name="ExternalPostgresDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
<xa-datasource-property name="ServerName">
address...
</xa-datasource-property>
<xa-datasource-property name="PortNumber">
5432
</xa-datasource-property>
<xa-datasource-property name="DatabaseName">
dbname...
</xa-datasource-property>
<xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
<driver>postgresql</driver>
<security>
<user-name>username...</user-name>
<password>password...</password>
</security>
<validation>
<valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLValidConnectionChecker"/>
<exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLExceptionSorter"/>
</validation>
</xa-datasource>
<drivers>
<driver name="postgresql" module="org.postgresql">
<xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
</driver>
<driver name="h2" module="com.h2database.h2">
<xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
</driver>
</drivers>
And finally there is a Dockerfile which should customize official keycloak image to my needs:
FROM jboss/keycloak:11.0.2
ENV DB_VENDOR postgres
ENV DB_ADDR addr..
ENV DB_DATABASE dbname...
ENV DB_USER user...
ENV DB_PASSWORD password...
ENV PROXY_ADDRESS_FORWARDING true
ENV KEYCLOAK_USER admin
ENV KEYCLOAK_PASSWORD password
COPY ./_resources/standalone.xml /opt/jboss/keycloak/standalone/configuration/standalone.xml
COPY ./_resources/custom-user-storage-jpa.jar /opt/jboss/keycloak/standalone/deployments/custom-user-storage-jpa.jar
COPY ./_resources/postgresql/main/module.xml /opt/jboss/keycloak/modules/system/layers/keycloak/org/postgresql/main/module.xml
COPY ./_resources/postgresql/main/postgresql-42.2.18.jar /opt/jboss/keycloak/modules/system/layers/keycloak/org/postgresql/main/postgresql-42.2.18.jar
ENV JAVA_OPTS -server -Xms2048m -Xmx6144m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m
EXPOSE 8080
Of course I checked that copied files are correct and on their place inside running container, after startup everything works fine but without my custom-user-storage deployment.
What am I missing?

It seems like your standalone.xml file was not read and jboss tries to use a default configuration instead.
There is a mention of updating the default filename to standalone-ha.xml here
https://lists.jboss.org/pipermail/keycloak-dev/2018-October/011304.html.
updating the image building command to
COPY ./_resources/standalone.xml /opt/jboss/keycloak/standalone/configuration/standalone-ha.xml
should help

Where to put -ds.xml?

I am having trouble adding a datasource to my wildfly server using a -ds.xml file as recommended by the jboss doc. I have a maven project with a persistence.xml:
<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.0"
xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">
<persistence-unit name="primary">
<jta-data-source>java:jboss/datasources/DvdRental4</jta-data-source>
<properties>
<property name="hibernate.hbm2ddl.auto" value="create-drop" />
<property name="hibernate.show_sql" value="true" />
</properties>
</persistence-unit>
</persistence>
and a postgres-ds.xml in the root of the project:
<?xml version="1.0" encoding="UTF-8"?>
<datasources>
<local-tx-datasource>
<jndi-name>DvdRental4</jndi-name>
<connection-url>jdbc:postgresql://localhost:5432/dvdrental4</connection-url>
<driver>postgresql-9.4.1208</driver>
<user-name>postgres</user-name>
<password>1234</password>
<metadata>
<type-mapping>PostgreSQL</type-mapping>
</metadata>
</local-tx-datasource>
</datasources>
When running mvn package and deploying the .war, I get an error that DvdRental4 is missing. So I suppose I need to place the postgres-ds.xml somewhere specific, but where? Or maybe the xml files are wrong? If needed, I can also provide the pom.xml. I already created the database dvdrental4, so this shouldn't be an issue.
I also read somewhere that you have to place the -ds.xml file in the deploy folder, I only have a wildfly/standalone/deployments folder, and putting it in there didn't work.
EDIT:
Wildfly gives the following error:
ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2) WFLYCTL0013: Operation ("full-replace-deployment") failed - address: ([]) - failure description: {
"WFLYCTL0412: Required services that are not installed:" => ["jboss.naming.context.java.jboss.datasources.DvdRental4"],
"WFLYCTL0180: Services with missing/unavailable dependencies" => [
"jboss.persistenceunit.\"dvdrental-gruppe-4.war#primary\".__FIRST_PHASE__ is missing [jboss.naming.context.java.jboss.datasources.DvdRental4]",
"jboss.persistenceunit.\"dvdrental-gruppe-4.war#primary\" is missing [jboss.naming.context.java.jboss.datasources.DvdRental4]"
]
}

I think I solved it by just putting the *-ds.xml file in the project/src/main/webapp/WEB-INF of my maven project, at least there is no error saying the datasource couldn't be found. I also changed the postgres-ds.xml a bit:
<?xml version="1.0" encoding="UTF-8"?>
<datasources>
<datasource jndi-name="java:jboss/datasources/DvdRental4" pool-name="DvdRental4"
enabled="true" use-java-context="true">
<connection-url>jdbc:postgresql://localhost:5432/dvdrental4</connection-url>
<driver>postgresql-9.4.1208.jar</driver>
<security>
<user-name>postgres</user-name>
<password>1234</password>
</security>
</datasource>
</datasources>
Note: I deployed the driver as a .jar, so that is why I can specify the driver directly as a .jar, I think.

The JNDI name you specified for your datasource is "DvdRental4". But the persistence context specifies "java:jboss/datasources/DvdRental4". I am not sure whether that is correct. Either try "java:jboss/datasources/DvdRental4" in both places or (even better) use the Boss JNDI view to see where the datasource is actually deployed.

The ability to deploy datasources was not a feature that made it between JBoss 5 and the AS7 rewrite, only the deployable jms destinations. You will need to configure the server to define the datasource, before you deploy your application

How to apply DRY for Liquibase data source config and persistence.xml?

I am using liquibase and Java EE with JBoss. I configure the persistence.xml file in the project to specify the Data Source I want to use.
In JBoss configs:
<datasources>
<datasource jta="true" jndi-name="java:jboss/datasources/WebStoriesDS" pool-name="java:jboss/datasources/WebStoriesDS" enabled="true" use-java-context="true" use-ccm="true">
<connection-url>jdbc:postgresql://127.0.0.1:5432/mydatabase</connection-url>
<driver>org.postgresql</driver>
<security>
<user-name>postgres</user-name>
<password>postgres</password>
</security>
</datasource>
<drivers>
<driver name="org.postgresql" module="org.postgresql">
<xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
</driver>
</drivers>
</datasources>
In the project's persistence.xml:
<persistence version="2.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">
<persistence-unit name="PersistenceUnit">
<jta-data-source>java:jboss/datasources/WebStoriesDS</jta-data-source>
</persistence-unit>
</persistence>
And the project's web.xml:
<context-param>
<param-name>liquibase.datasource</param-name>
<param-value>java:jboss/datasources/WebStoriesDS</param-value>
</context-param>
Well, as you can see I have 1 data source reference in JBoss and 2 references in my project. How would I apply the DRY principle here and use only 1 reference in JBoss configs and 1 reference in my project? Use a properties file to hold the data source name?
I need just to configure it in a way that, when I need to change the name, I change only in 2 places: In my project and in JBoss configs.

Both standard Liquibase and JPA require the datasource to be explicitly listed. It is a logical reference, so you can change where the datasource points without affecting either configuration unless you want to change the datasource name (as you listed).
If you are wanting to remove the explicit reference from either Liquibase or JPA, the easiest is probably to subclass liquibase.integration.servlet.LiquibaseServletListener and override the getDataSource() method to look up reference from your PersitanceUnit and return it. That will allow you to only specify the datasource in persistance.xml and no longer need the liquibase.datasource configuration in web.xml.

Configuring data source - tomee.xml, persistence.xml

I'm doing some EJB with JPA project that maps/persists some entities to mysql database.
I have defined persistence unit in persistence.xml like this:
<persistence-unit name="MyAppPU" transaction-type="JTA">
<provider>org.apache.openjpa.persistence.PersistenceProviderImpl</provider>
<jta-data-source>MyAppDS</jta-data-source>
<properties>
<property name="openjpa.jdbc.SynchronizeMappings" value="buildSchema(ForeignKeys=true)" />
<property name="openjpa.jdbc.DBDictionary" value="mysql" />
<property name="openjpa.Log" value="DefaultLevel=WARN, Tool=INFO" />
</properties>
</persistence-unit>
Then, in tomee/conf/tomee.xml file i have defined data source like this:
<Resource id="MyAppDS" type="DataSource">
JdbcDriver com.mysql.jdbc.Driver
JdbcUrl jdbc:mysql://127.0.0.1:3306/MyAppDB
UserName root
Password 123
JtaManaged true
DefaultAutoCommit false
</Resource>
All this works fine, i create MyApp.jar, deploy it to TomEE server, test it and i get mysql tables in database.
My question is "Is there any other place where I could define data source resource?"
Or it has to be in tomee/conf/tomee.xml file?
Can it be defined somewhere inside application structure, in some xml file, and deployed inside apps jar file to server?

That's the whole point of a JNDI data source, to externalize it outside of your application, so you can modify it without recompiling or repackaging. So it is better to leave it this way.
For testing purpose, some EE server such as JBoss (Wildfly) let you define this in your project.

It might be a bit late to answer this, You can in tomee place the resource definition in WEB-INF/resources.xml.

You only have to set the tomee.xml. If you are using Eclipse, you must copy the tomee.xml into the servers configuration/Tomee to get recognize it into the web project, otherwise you will get troubles.

Services with missing/unavailable dependencies

Any idea why I'm getting this error:
JBAS014775: New missing/unsatisfied dependencies:
service jboss.jdbc-driver.mysql (missing) dependents: [service jboss.data-source.jboss/datasources/UserDS]
ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) `{"JBAS014653: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-2" => {"JBAS014771: Services with missing/unavailable dependencies" => ["jboss.data-source.jboss/datasources/UserDSjboss.jdbc-driver.com_mysql_jdbcMissing[jboss.data-source.jboss/datasources/UserDSjboss.jdbc-driver.com_mysql_jdbc]"]}}}`
persistence.xml
<?xml version="1.0" encoding="UTF-8"?>
<persistence version="2.0"
xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://java.sun.com/xml/ns/persistence
http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">
<persistence-unit name="primary">
<jta-data-source>java:jboss/datasources/UserDS</jta-data-source>
<properties>
<!-- Properties for Hibernate -->
<property name="hibernate.hbm2ddl.auto" value="create-drop" />
<property name="hibernate.show_sql" value="true" />
</properties>
</persistence-unit>
</persistence>
mydatasource-ds.xml
<?xml version="1.0" encoding="UTF-8"?>
<datasources xmlns="http://www.jboss.org/ironjacamar/schema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.jboss.org/ironjacamar/schema http://docs.jboss.org/ironjacamar/schema/datasources_1_0.xsd">
<datasource jndi-name="java:jboss/datasources/UserDS" pool-name="kitchensink-quickstart"
enabled="true" use-java-context="true">
<!-- jdbc:h2:mem:kitchensink-quickstart;DB_CLOSE_ON_EXIT=FALSE;DB_CLOSE_DELAY=-1 -->
<connection-url>
jdbc:mysql://localhost:3306/test
</connection-url>
<driver>mysql</driver>
<security>
<user-name>root</user-name>
<password>root</password>
</security>
</datasource>
</datasources>
module.xml
<module xmlns="urn:jboss:module:1.0" name="com.mysql">
<resources>
<resource-root path="mysql-connector-java-5.1.22.jar"/>
</resources>
<dependencies>
<module name="javax.api"/>
</dependencies>
</module>

If you are specifying the data source as a resource reference in web.xml, then match the name exactly with that in standalone.xml (or domain.xml):
web.xml:
<resource-ref>
<res-ref-name>java:jboss/datasources/OracleDS</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
</resource-ref>
standalone.xml:
<datasource jndi-name="java:jboss/datasources/OracleDS" pool-name="OracleDS" enabled="true" use-java-context="false">

the reason for the error is you are missing the dependence java:jboss/datasources/UserDS.
With Jboss 7.x+ these datasource can be added directly to the app servers configuration as you discovered.
the difference between Standalone and Domain configuration is the standalone configuration is designed for only one app server w/ said configuration. If you look closely at the domain.xml you will see several app server configurations (aka profiles). These will be much like standalone, standalone-full, standalone-ha, standalone-full-ha config files found under the standalone/conf* directory. Operating in domain mode allows you to control many different server instances running on that domain from a central location (ie the domain controller). ( this includes nodes of a cluster if you have ha configured)
This is closely related to your original question in that the domain controller has the ability to gracefully share this datasource configuration to all of its nodes.

Wildfly Version 10.0.1
I am running a non clustered Wildlfy set up.
I had 2 Wildlfy instances already running and was trying to deploy the 3rd one when I encountered the error.
I had to stop the other two instances and then try again and the wildfly deployment went through successfully.