(I already know the answer, but because I often find myself re-looking for the answer, so I post it here as documentation for myself and others. This is encouraged on Stackoverflow.)
Background intro
Many Servlet developers have read the book "Head First Serlet & JSP" to get their "Certified Web Component Developer Exam" or just to learn Servlets. But the book hasn't been updated since 2009, and only covers Servlet 2.4. A lot has changed since then. The current latest version is 4.0.
One of the things that changed is the boot process of servlet web applications, which can make it unclear what happens during startup and how the web application is initialized.
Question
In Servlet version 2.4 and lower, web.xml is used to fully configure web applications. But higher version seem to have other ways to configure web applications, without touching web.xml and without annotations. For example, web frameworks provided as .jar files are somehow able to hook into the Servlet container and add url mappings.
How does this mechanism work?
Introduction
In Servlet 2.4 (November 2003) a servlet container (such as Tomcat and Jetty) simply booted the web application by looking for the file WEB-INF/web.xml (the deployment descriptor). The file web.xml contains references to servlets, filters and listeners, with their associated url patterns and parameters. Using web.xml the servlet container knows exactly where to find everything and how to configure them.
Since Servlet 3.0 (December 2009), web.xml is optional, you can also use annotations or a programmatic configuration instead.
Annotations are simpler in use. They are located in the javax.servlet.annotation package, and allow you to annotate a servlet with #WebServlet, a filter with #WebFilter, and a listener with #WebListener. The servler container will then automatically find and detect these classes. Annotations however do provide less configuration features than web.xml and a programmatic configuration.
This text further focuses at how programmatic configuration can be configured and how it starts Spring MVC. It's a bit more complex than annotations, but does give you and framework designers more control over the booting process.
If you wanted to use a web framework before Servlet version 3.0, you have to add a servlet or filter to your web.xml and configure the framework from there. After this initialization, you could start writing classes (usually non-Servlet classes) known by the web framework to create a web application.
Since Servlet 3.0, the system is modular. This allows framework and library designers to initialize the servlet container without you having to configure the framework via web.xml. You can immediately start writing the web framework specific classes to create a web application without touching Servlet classes.
(It's also possible in Servlet 3.0 to create your own web.xml and still let the framework do the initialization of the framework, without it being defined in web.xml.)
How can a framework or library hook into the Servlet container automatically?
When booting up, the Servlet container first looks for the deployment descriptor located at WEB-INF/web.xml. If this file has a metadata-complete attribute set to false, or if it isn't defined at all, the container will also search for annotated classes, such as #WebServlet.
Apart from looking for web.xml and annotated classes, since Servlet 3.0, the container will also look for META-INF/web-fragment.xml files within the .jar files located in the WEB-INF/lib directory.
The file web-fragment.xml is a web fragment, which is
(quote from Java Servlet Specification)
a logical partitioning of the web application in such a way that the
frameworks being used within the web application can define all the
artifacts without asking developers to edit or add information in the
web.xml. It can include almost all the same elements that the web.xml
descriptor uses. However the top level element for the descriptor MUST
be web-fragment and the corresponding descriptor file MUST be called
web-fragment.xml. The ordering related elements also differ between
the web-fragment.xml and web.xml
The content of web-fragment.xml is like web.xml, but with a web-fragment root element instead of web-app element:
<web-fragment xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
https://java.sun.com/xml/ns/javaee/web-fragment_3_0.xsd"
version="3.0">
<filter>
<filter-name>FrameworkFilter</filter-name>
<filter-class>framework.FrameworkFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>FrameworkFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
</web-fragment>
The exact ordering of loading of a single web.xml and multiple web-fragment.xml files, can be configured using the <absolute-ordering> and <ordering>.
Apart from web fragments, there is also a programmatic method to partition the web application: by writing an implementation of the interface javax.servlet.ServletContainerInitializer. ServletContainerInitializer gives access to the ServletContext, which contains methods to programmatically add servlets, filters and listeners.
To use ServletContainerInitializer, it has to be specified in a file located in META-INF/services/javax.servlet.ServletContainerInitializer. The content of this file has to be the fully qualified path to the implementation class.
How does this work for Spring MVC?
Although Spring MVC does contain a web-fragment.xml, it doesn't define any servlets, filters or listeners. Spring uses the META-INF/services/javax.servlet.ServletContainerInitializer file to refer to its own ServletContainerInitializer implementation class, which is the class SpringServletContainerInitializer.
SpringServletContainerInitializer is a ServletContainerInitializer, so it receives a ServletContext on startup. The goal of SpringServletContainerInitializer is to pass the servletContext to a more developer friendly WebApplicationInitializer, so that you can add servlets, such as Springs DispatcherServlet, which is a front-controller that directs incoming requests to other controllers. (See spring-framework-reference about how to configure a DispatcherServlet to Spring.)
Spring MVC doesn't provide a concrete implementation of WebApplicationInitializer, only a number of abstract classes, so that you have control over the booting process. In case of Spring Boot, a concrete implementation is provided: SpringApplicationWebApplicationInitializer to reduce the amount of boilerplate code.
Further info
A detailed description about how the Servlet container booting process exactly works, can be found in the official Java Servlet Specification. Further info about Spring MVC can be found in the Spring Framework reference.
I recently upgraded a project to Java 8, targeting Tomcat 9. It relies heavily on cookies that use commas and possibly spaces in the value so I need to use LegacyCookieProcessor because the cookies aren't RFC 6265 compliant. The only two ways I can find on how to do this require Spring Boot or editing context.xml in Tomcat. This project isn't using Spring Boot so the first one isn't an option, and I'd really rather not break portability by requiring context.xml to be changed wherever it is deployed.
Is it possible to configure this in Java config or web.xml (or any other way)?
Tomcat supports per application config in /META-INF/context.xml bundled in the application package just like the web.xml file.
That file supports the cookie processor config
<CookieProcessor className="org.apache.tomcat.util.http.LegacyCookieProcessor" />
I want to create a simple web application that runs on tomcat with pure java configuration.
I know how to do it with Spring frameworks WebApplicationInitializer. But i don't know without Spring.
I have heard about #WebServlet
Please guide me.
This is true that from Java EE 6, web.xml is no longer required for a Webapp to deploy. Instead of using web.xml, you can use annotations to replace the xml file (#WebServlet, etc.)
More information is given in this thread : How to use annotations instead of web.xml in the servlet to specify url
We are trying to introduce Spring Security 4.0.2 into a Spring MVC application which uses Spring 4.2.3 but also is running on servlet 2.5. I am having trouble configuring Security:
Caused by: java.lang.ClassNotFoundException: javax.servlet.FilterRegistration$Dynamic
I believe this class is only in Servlet 3.0. Is it possible to use Spring Security 4.0 with Servlet 2.5? Or should we regress to Security 3.2?
My suspicion is you are using Jetty and this explains your situation.
The javax.servlet.FilterRegistration class was introduced in Servlet
3.0. This exception suggests that you have still libraries of an older Servlet API version in your runtime classpath which got precedence in
classloading.
Apparently, Java configuration demands this class. I used the old XML configuration and things worked fine.
I would like to create a Spring Boot application to be deployed on Google AppEngine infrastructure. GAE currently only supports servlet 2.5 web applications.
Is it possible to use Spring Boot - using auto-configuration - in combination with a old-fashioned web.xml?
Can I for example use a contextClass/contextConfigLocation pointing to a #Configration class including #EnableAutoConfiguration?
All Spring Boot examples seem to use a simple Application class with main method to run the application. So I would like to know if Spring Boot supports using a web.xml as start point to boot the application?
More than one question there:
There's nothing stopping you from using web.xml (it's still part of the Servlet spec). Most people prefer the Java initializers these days.
#EnableAutoConfiguration works for any application context (it just uses features of Spring).
But Spring Boot out of the box doesn't have a ContextLoaderListener that knows about SpringApplication, so you miss some of the benefits of Boot if you do as you describe. There's one you might find useful here.
Spring Boot Actuator relies on a few Servlet 3.0 features here and there so you need workarounds for a 2.5 environment (see this spring-boot-legacy prototype for details).
There's a sample app here that runs on GAE, currently deployed here: http://dsyerboot.appspot.com/.