Develop Reference

ServletContainerInitializer ignored by Tomee because of jar name and web-fragment "ordering"

The library I'm maintaining contains a implementation of ServletContainerInitializer which is declared in META-INF/services/javax.servlet.ServletContainerInitializer file.
This library is used by web applications running on Tomee 8 (Apache Tomcat (TomEE)/9.0.63 (8.0.12)).
Recently I added the log4j-web dependency to the project. Before that my initializer was called as expected but since it is ignored.
After creating a minimal example I identified that my jar is ignored because :
Its name starts with spring- (it is not a library of the Spring Framework, only a poor naming choice) and,
Another jar in the classpath (log4j-web) contains a web-fragment.xml with "ordering" before or after "others" :
<web-fragment xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
http://java.sun.com/xml/ns/javaee/web-fragment_3_0.xsd"
version="3.0" metadata-complete="true">
<name>my_web_fragment_xxx</name>
<distributable />
<ordering>
<before>
<others />
</before>
</ordering>
</web-fragment>
I activated Tomcat traces logs and noticed that "spring" is listed in some exclusion prefixes as well as many other names.
TRACE [Thread-2] org.apache.openejb.config.NewLoaderLogic.readDefaultExclusions Loaded default.exclusions
TRACE [Thread-2] org.apache.openejb.config.NewLoaderLogic.logExclusions Exclusion prefixes: [
...
TRACE [Thread-2] org.apache.openejb.config.NewLoaderLogic.logExclusions -snappy-java-
TRACE [Thread-2] org.apache.openejb.config.NewLoaderLogic.logExclusions -spring-
TRACE [Thread-2] org.apache.openejb.config.NewLoaderLogic.logExclusions -sshd-
...
I noticed that "google" is listed too and when I prefix my jar with "google-" it reproduces the same issue.
The default exclusion list comes from openejb-core library :
https://github.com/apache/tomee/blob/c0928e2aed1713d16d28bffaed11ad5024bc3728/container/openejb-core/src/main/resources/default.exclusions
Workarounds
Rename the library (but it will have an impact on applications depending on it),
Add the name of the ignored jar to the tomcat.util.scan.StandardJarScanFilter.jarsToScan jars list (but it has to be done to every environments from dev to production for customers using our products).
Question
What I would like to understand is, why this issue is only triggered when a jar in the classpath contains a web-fragment with an <ordering> element ?
Minimal reproducible sample
https://github.com/KWSimon/tomcat_jarscanning

Getting "http://javax.xml.XMLConstants/property/accessExternalDTD property not supported" in JBoss and using springboot

After upgrading our application to spring boot 2.6.6 in combination with JBoss 7.3 we get on every xml validation the following warnings:
2022-04-13 14:18:39,433 WARN [org.springframework.xml.validation.Jaxp15ValidatorFactory] (default task-2) http://javax.xml.XMLConstants/property/accessExternalDTD property not supported by org.apache.xerces.jaxp.validation.ValidatorImpl
2022-04-13 14:18:39,433 WARN [org.springframework.xml.validation.Jaxp15ValidatorFactory] (default task-1) http://javax.xml.XMLConstants/property/accessExternalSchema property not supported by org.apache.xerces.jaxp.validation.ValidatorImpl
Due to the intensive number of validations, the JBoss disk was filled with a log warnings resulting in a full disk.
The most obvious solution is not to use JBoss, but unfortunately that is not possible in our production environment.

The reason for the warning is that JBoss 7.3 is using an old Xerces library (2.2.12.0.SP03) which does not support the properties: accessExternalDTD and accessExternalSchema and spring-boot (2.6.6) expect support for this and log a warning when it is not supported.
Normally you should update to a newer version of the Xerces library but in the case of JBoss this is not possible because the Xerces library is packaged with JBoss.
The solution for this is to change the logging settings in JBoss to the error level for this particular logging. This can be done with the following commands:
${JBOSS_HOME}/bin/jboss-cli.sh --connect controller=localhost:9990 --user=<admin user> --password=<password> --command="/subsystem=logging/logger=org.springframework.xml.validation.Jaxp15ValidatorFactory:add"
${JBOSS_HOME}/bin/jboss-cli.sh --connect controller=localhost:9990 --user=<admin user> --password=<password> --command="/subsystem=logging/logger=org.springframework.xml.validation.Jaxp15ValidatorFactory:write-attribute(name=level, value=ERROR)"

The provided implementation of the ValidatorFactory doesn't support the mentioned property.
As it's mentioned in the answer of #Gerard, JBoss org.apache.xerces dependency provides its implementation.
You can configure your program to use the implementation that supports or ignores the mentioned property.
I elaborated on the details in another similar answer.

CWNEN0030E error on DataSource JNDI lookup in Websphere Liberty

I am try to convert a WAS traditional 8.5 websphere configuration to a websphere liberty 20.x configuration and getting an error. I believe the connection settings are right, the names are correct, the jar file for the driver exists but getting the error below.
Here is essentially the configuration.
<dataSource jndiName="jdbc/db2a" type="javax.sql.DataSource">
<jdbcDriver javax.sql.DataSource="com.ibm.db2.jcc.DB2Driver" libraryRef="DB2JCCLib"/>
<properties.db2.jcc driverType="4" databaseName="DB1" serverName="host.name" portNumber="446"/>
</dataSource>
<web-bnd
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://websphere.ibm.com/xml/ns/javaee"
xsi:schemaLocation="http://websphere.ibm.com/xml/ns/javaee http://websphere.ibm.com/xml/ns/javaee/ibm-web-bnd_1_0.xsd" version="1.0">
<virtual-host name="default_host"/>
<resource-ref name="jdbc/db2a" binding-name="jdbc/db2a"/>
</web-bnd>
...
Configuration at top of server.xml
<!-- Enable features -->
<featureManager>
<feature>appSecurity-2.0</feature>
<feature>jaxrs-2.0</feature>
<feature>jsp-2.3</feature>
<feature>localConnector-1.0</feature>
<feature>jaxws-2.2</feature>
<feature>ldapRegistry-3.0</feature>
</featureManager>
The jar file is : db2jcc4-4.22.29.jar
And has as reference to that jar / class file.
The java code for the connection is standard jdbc connection:
DriverManager.getConnection (connStr);
And this:
InitialContext initialContext = new InitialContext();
Context context = (Context)initialContext.lookup("java:comp/env");
DataSource dataSource = (DataSource)context.lookup(string5);
logger.info((Object)"CVDBBackendHandler.getConnection() out");
return dataSource.getConnection();
The error is below.
Root exception is com.ibm.wsspi.injectionengine.InjectionException: CWNEN0030E: The server was unable to obtain an object instance for the java:comp/env/jdbc/db2a reference. The exception message was: CWNEN1003E: The server was unable to find the jdbc/db2a binding with the java.lang.Object type for the java:comp/env/jdbc/db2a reference.

You need to add one of the JDBC features to your feature list, e.g.
<featureManager>
<feature>jdbc-4.3</feature>
</featureManager>
(or one of the earlier JDBC features, e.g.jdbc-4.2, etc.).
Though your messages.log should show a feature set like this (including features pulled in from the features you explicitly enabled):
CWWKF0012I: The server installed the following features: [appSecurity-2.0, distributedMap-1.0, el-3.0, federatedRegistry-1.0, jaxb-2.2, jaxrs-2.0, jaxrsClient-2.0, jaxws-2.2, jndi-1.0, json-1.0, jsp-2.3, ldapRegistry-3.0, localConnector-1.0, servlet-3.1, ssl-1.0].
this list doesn't include any activating the JDBC feature, so you must explicitly enable it. (On the other hand, note the jndi-1.0 feature required to do JNDI lookups is included even though it wasn't one you added directly to server.xml, since one of the other features included it).
Some helpful links for reference:
https://openliberty.io/docs/latest/reference/feature/jdbc-4.3.html
https://openliberty.io/docs/latest/relational-database-connections-JDBC.html

There is an error in the configuration. com.ibm.db2.jcc.DB2Driver is not an implementation of javax.sql.DataSource.
You can remove the configuration attribute javax.sql.DataSource="com.ibm.db2.jcc.DB2Driver" and let the built-in knowledge in Liberty infer it from the JDBC driver jar, or you can specify it as,
javax.sql.DataSource="com.ibm.db2.jcc.DB2DataSource"
The above data source class name can be found in DB2 documentation here

Log4j 2.8 cannot load custom converters in OSGi environment

I'm currently switching to log4j2 and I'm working in OSGi environment. There's been couple of issues with OSGi in log4j 2.7 branch which encouraged me to grab 2.8 as the final library.
The thing is I have couple of custom PatternConverters implemented in my project and I can't manage to have them loaded and recognized
DEBUG Initializing configuration XmlConfiguration[location=C:\Program Files\...\etc\log4j2.xml]
.
.
DEBUG Took 0,001043 seconds to load 0 plugins from package common.audit.sink.impl.log4j2.converter
although there's at least 10 of them. This later results in errors like:
DEBUG Building Plugin[name=layout, class=org.apache.logging.log4j.core.layout.PatternLayout].
DEBUG PatternLayout$Builder(pattern="%ED;...
ERROR Unrecognized format specifier [ED]
ERROR Unrecognized conversion specifier [ED] starting at position 3 in conversion pattern.
I have pretty much everything set up as necessary (it worked with 2.7 anyways).
The plugin implementation annotated and both newInstance, format methods implemented according to this guide
The packages declaration in log4j2.xml:
<Configuration status="trace" name="Systemtest_Server" packages="common.audit.sink.impl.log4j2.converter">
The plugins metadata is generated and present in *.jar bundle containing plugins
META-INF\org\apache\logging\log4j\core\config\plugins\Log4j2Plugins.dat
The log4j-core-2.8.jar bundle is activated together with log4j-api-2.8.jar at the startup of Equinox container. The configuration file is pointed by system property and works (obviously from the debug message I receive).
However, I found this issue reported on log4j Jira, which is kind of my problem as I was facing the same exception until I modified log4j-api-2.8.jar to make it import org.apache.logging.log4j.core.util package so my server would start normally. There are some comments suggesting that this might lead to non-extensibility of log4j2 in OSGi.
So is there some way how to overcome this issue (quick fix is enough)? Alternatively, is there something that I might be forgetting to configure here?
Thanks a lot for any help.

Ok, so I'm gonna answer this for myself and maybe it will help somebody too. I did a lot of playing around and the fault was in OSGi with which I'm not so familiar, to be honest.
I found out in the end that you have to ensure these things:
The bundle has to be activated in OSGi container. Inactive bundles are not scanned for plugins. The fact that you mark bundle for activation in your configuration doesn't necessarily mean it will actually happen. E.g., some fault in my bundle's manifest prevent this from happening. So to be sure I build bundle with
Export-Package: common.audit.sink.impl.log4j2.*
Import-Package: *
At least one of following has to hold to get your plugins loaded:
Your bundle with custom plugins is a fragment of log4j-core by stating this in its manifest:
Fragment-Host: org.apache.logging.log4j.core
Probably the right approach.
Your bundle is activated with the same priority as log4j bundles (not only API but also Core needs to be active to enable plugin-search)
Of course the configuration mentioned in my question is needed as well. After this I got no errors and saw a trace like this:
DEBUG Oracle Nashorn Version: 1.8.0_102, Language: ECMAScript, Threading: Not Thread Safe, Compile: true, Names: {nashorn, Nashorn, js, JS, JavaScript, javascript, ECMAScript, ecmascript}
DEBUG Took 0,008133 seconds to load 16 plugins from package common.audit.sink.impl.log4j2.converter
DEBUG PluginManager 'Core' found 112 plugins
DEBUG PluginManager 'Level' found 0 plugins
...
DEBUG Building Plugin[name=layout, class=org.apache.logging.log4j.core.layout.PatternLayout].
DEBUG PatternLayout$Builder(pattern="%ED;%EU;%EL;...
DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.FileAppender].

NoSuchMethodError - Calling Class/Method from Class in Same Package

We are integrating an internal framework into our weblogic application and we are running into deployment problems.
Technologies Used
Weblogic 10.3.6 application
Spring 3.0
Maven 2
Eclipse J2EE
The Problem
On startup of the weblogic application, we receive the following NoSuchMethodError while initializing one of the beans. This error is occuring when calling classes in the org.joda.time (2.0) jar.
Caused By: java.lang.NoSuchMethodError: org.joda.time.DateTimeZone.convertLocalToUTC(JZ)J
at org.joda.time.LocalDate.toDateTimeAtStartOfDay(LocalDate.java:715)
at org.joda.time.LocalDate.toDateTimeAtStartOfDay(LocalDate.java:690)
. . . excluded . . .
Things We Have Tried
After Googling "NoSuchMethodError spring", many of the problems seem to be incompatible Spring versions. After printing the dependency tree, the only Spring version in use is 3.0.
Googling "NoSuchMethodError" usually gave JAR hell solutions.
Multiple versions of the same dependency. After doing some maven dependency management, the only joda-time jar in use is 2.0. Additionally, the local repository was purged of any unnecessary jars.
.war / runtime may not have the correct jars included in the lib directory. After looking into the WEB_INF/lib directory, the only joda-time jar is version 2.0, which contains all of the appropriate class files
One mysterious thing is that the DateTimeZone.convertLocalToUTC(JZ)J has been a part of the org.joda.time project since 1.0, so even if we have incompatible versions, the method should still be found, especially if the class and package are able to be found.
Finally there are no other DateTimeZone classes in the project (ctrl+shift+T search in eclipse) so I'm confused as to which class is being loaded if the org.joda.DateTimeZone class is not being loaded.
Questions:
Can anyone explain why the method could not be found?
Are there more places to check for existing or conflicting jars?
Is there a way to check the DateTimeZone class that the LocalDate class is using during runtime via Eclipse debug?

Here's some interesting reading:
prefer-web-inf-classes Element
The weblogic.xml Web application deployment descriptor contains a
element (a sub-element of the
element). By default, this element is set to
False. Setting this element to True subverts the classloader
delegation model so that class definitions from the Web application
are loaded in preference to class definitions in higher-level
classloaders. This allows a Web application to use its own version of
a third-party class, which might also be part of WebLogic Server. See
“weblogic.xml Deployment Descriptor Elements.”
taken from: http://docs.oracle.com/cd/E15051_01/wls/docs103/programming/classloading.html
Other troubleshooting tips:
You can try: -verbose:class and check your managed server's logs to check if the class is being loaded properly.
An efficient way to confirm which intrusive jar might be getting loaded is by running a whereis.jsp within the same webcontext (i.e., JVM instance) of this app.
--whereis.jsp --
<%# page import="java.security.*" %>
<%# page import="java.net.URL" %>
<%
Class cls = org.joda.time.DateTimeZone.class;
ProtectionDomain pDomain = cls.getProtectionDomain();
CodeSource cSource = pDomain.getCodeSource();
URL loc = cSource.getLocation();
out.println(loc);
// it should print something like "c:/jars/MyJar.jar"
%>
You can also try jarscan on your $WEBLOGIC_HOME folder to see if you can find the jar that contains this class: https://java.net/projects/jarscan/pages/Tutorial

A NoSuchMethodError is almost always due to conflicting library versions. In this case I'm guessing there are multiple versions of joda libraries in the two projects.
Weblogic is pulling the org.joda jar.
Tryu adding this in your weblogic.xml to exclude the jar that weblogic is pulling, and instead use your appllication jar.
The below is from my application, you can have a look what all we have to removed for our application.
<wls:container-descriptor>
<wls:prefer-application-packages>
<wls:package-name>antlr.*</wls:package-name>
<wls:package-name>org.slf4j.*</wls:package-name>
<wls:package-name>org.slf4j.helpers.*</wls:package-name>
<wls:package-name>org.slf4j.impl.*</wls:package-name>
<wls:package-name>org.slf4j.spi.*</wls:package-name>
<wls:package-name>org.hibernate.*</wls:package-name>
<wls:package-name>org.springframework.*</wls:package-name>
<wls:package-name>javax.persistence.*</wls:package-name>
<wls:package-name>org.apache.commons.*</wls:package-name>
<wls:package-name>org.apache.xmlbeans.*</wls:package-name>
<wls:package-name>javassist.*</wls:package-name>
<wls:package-name>org.joda.*</wls:package-name>
<wls:package-name>javax.xml.bind.*</wls:package-name>
<wls:package-name>com.sun.xml.bind.*</wls:package-name>
<wls:package-name>org.eclipse.persistence.*</wls:package-name>
</wls:prefer-application-packages>
<wls:show-archived-real-path-enabled>true</wls:show-archived-real-path-enabled>
</wls:container-descriptor>