Develop Reference

How do frameworks (like Spring) configure the Servlet container without web.xml?

(I already know the answer, but because I often find myself re-looking for the answer, so I post it here as documentation for myself and others. This is encouraged on Stackoverflow.)
Background intro
Many Servlet developers have read the book "Head First Serlet & JSP" to get their "Certified Web Component Developer Exam" or just to learn Servlets. But the book hasn't been updated since 2009, and only covers Servlet 2.4. A lot has changed since then. The current latest version is 4.0.
One of the things that changed is the boot process of servlet web applications, which can make it unclear what happens during startup and how the web application is initialized.
Question
In Servlet version 2.4 and lower, web.xml is used to fully configure web applications. But higher version seem to have other ways to configure web applications, without touching web.xml and without annotations. For example, web frameworks provided as .jar files are somehow able to hook into the Servlet container and add url mappings.
How does this mechanism work?

Introduction
In Servlet 2.4 (November 2003) a servlet container (such as Tomcat and Jetty) simply booted the web application by looking for the file WEB-INF/web.xml (the deployment descriptor). The file web.xml contains references to servlets, filters and listeners, with their associated url patterns and parameters. Using web.xml the servlet container knows exactly where to find everything and how to configure them.
Since Servlet 3.0 (December 2009), web.xml is optional, you can also use annotations or a programmatic configuration instead.
Annotations are simpler in use. They are located in the javax.servlet.annotation package, and allow you to annotate a servlet with #WebServlet, a filter with #WebFilter, and a listener with #WebListener. The servler container will then automatically find and detect these classes. Annotations however do provide less configuration features than web.xml and a programmatic configuration.
This text further focuses at how programmatic configuration can be configured and how it starts Spring MVC. It's a bit more complex than annotations, but does give you and framework designers more control over the booting process.
If you wanted to use a web framework before Servlet version 3.0, you have to add a servlet or filter to your web.xml and configure the framework from there. After this initialization, you could start writing classes (usually non-Servlet classes) known by the web framework to create a web application.
Since Servlet 3.0, the system is modular. This allows framework and library designers to initialize the servlet container without you having to configure the framework via web.xml. You can immediately start writing the web framework specific classes to create a web application without touching Servlet classes.
(It's also possible in Servlet 3.0 to create your own web.xml and still let the framework do the initialization of the framework, without it being defined in web.xml.)
How can a framework or library hook into the Servlet container automatically?
When booting up, the Servlet container first looks for the deployment descriptor located at WEB-INF/web.xml. If this file has a metadata-complete attribute set to false, or if it isn't defined at all, the container will also search for annotated classes, such as #WebServlet.
Apart from looking for web.xml and annotated classes, since Servlet 3.0, the container will also look for META-INF/web-fragment.xml files within the .jar files located in the WEB-INF/lib directory.
The file web-fragment.xml is a web fragment, which is
(quote from Java Servlet Specification)
a logical partitioning of the web application in such a way that the
frameworks being used within the web application can define all the
artifacts without asking developers to edit or add information in the
web.xml. It can include almost all the same elements that the web.xml
descriptor uses. However the top level element for the descriptor MUST
be web-fragment and the corresponding descriptor file MUST be called
web-fragment.xml. The ordering related elements also differ between
the web-fragment.xml and web.xml
The content of web-fragment.xml is like web.xml, but with a web-fragment root element instead of web-app element:
<web-fragment xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
https://java.sun.com/xml/ns/javaee/web-fragment_3_0.xsd"
version="3.0">
<filter>
<filter-name>FrameworkFilter</filter-name>
<filter-class>framework.FrameworkFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>FrameworkFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
</web-fragment>
The exact ordering of loading of a single web.xml and multiple web-fragment.xml files, can be configured using the <absolute-ordering> and <ordering>.
Apart from web fragments, there is also a programmatic method to partition the web application: by writing an implementation of the interface javax.servlet.ServletContainerInitializer. ServletContainerInitializer gives access to the ServletContext, which contains methods to programmatically add servlets, filters and listeners.
To use ServletContainerInitializer, it has to be specified in a file located in META-INF/services/javax.servlet.ServletContainerInitializer. The content of this file has to be the fully qualified path to the implementation class.
How does this work for Spring MVC?
Although Spring MVC does contain a web-fragment.xml, it doesn't define any servlets, filters or listeners. Spring uses the META-INF/services/javax.servlet.ServletContainerInitializer file to refer to its own ServletContainerInitializer implementation class, which is the class SpringServletContainerInitializer.
SpringServletContainerInitializer is a ServletContainerInitializer, so it receives a ServletContext on startup. The goal of SpringServletContainerInitializer is to pass the servletContext to a more developer friendly WebApplicationInitializer, so that you can add servlets, such as Springs DispatcherServlet, which is a front-controller that directs incoming requests to other controllers. (See spring-framework-reference about how to configure a DispatcherServlet to Spring.)
Spring MVC doesn't provide a concrete implementation of WebApplicationInitializer, only a number of abstract classes, so that you have control over the booting process. In case of Spring Boot, a concrete implementation is provided: SpringApplicationWebApplicationInitializer to reduce the amount of boilerplate code.
Further info
A detailed description about how the Servlet container booting process exactly works, can be found in the official Java Servlet Specification. Further info about Spring MVC can be found in the Spring Framework reference.

Have two servlet contexts in one Spring Boot application context under Jetty

Imagine I have one Spring Boot instance, pretty boring, one ApplicationContext, starter-jetty. By default, it does one ServletContext on /, puts a DispatcherServlet on / too. Every servlet and filter then placed under this context.
Now, I want to have one servlet context with contextPath=/api with a few servlets and another servlet context under /internal with a different set of servlets. They have to share ApplicationContext and most beans. And yes, it has to be plain old servlets & filters, not just two WebMvc instances.
How do I configure that? The most crucial question is how to make two ServletContext's for Jetty to consume. Just class names are enough, links to examples are encouraged. Hints on how to painlessly configure the distribution of Servlet's and Filter's between servlet contexts are appreciated too.

which servlets are not part of Web application

Servlet specification SRV.3.2 says
Servlets in a container that were not deployed as part of aWeb
application are implicitly part of a “default” Web application and
have a default ServletContext.
so how can we create serlvets those are not part of web application and what is the use of them?

Take a look at section SRV.9.1:
SRV.9.1 Web Applications Within Web Servers
A web application is rooted at a specific path within a web server. For
example, a catalog application could be located at
http://www.example.com/catalog. All requests that start with this prefix will be routed to the
ServletContext which represents the catalog application.
Servlets that are not part of a web application are therefore servlets that are not rooted at a specific path. In other words, they're rooted at the / path, and their use is to serve content from there.
Note that this is pretty awkward to begin with (as the verbose text already shows), so most web application servers let you designate an application to run at the 'root' context path in some other way.

when container call deployment descriptor (web.xml)

We know that whenever a request for a servlet comes, servlet container will first check web.xml file for url and corresponding servlet classes. This is fine, But the confusion comes after that. Suppose I am using load-on-startup property. SO, the servlet should be ready before first call comes in. In that case container need servlet Config object to make servlet in work. But again load-on-start up and init-parameter for servlet is defined in web.xml file. So when exactly container uses web.xml?( when load-on-startup property is used then container can not obviously wait for first call, again container has to read web.xml file to know whether that property is used with any servlet.)
It will be better if some one can clear my confusion. Please provide some dependable link also if possible.

Loading and Instantiation:
The servlet container is responsible for loading and instantiating servlets. The loading and instantiation can occur when the container is started, or delayed until the container determines the servlet is needed to service a request.
When the servlet engine is started, needed servlet classes must be located by the servlet container. The servlet container loads the servlet class using normal Java class loading facilities. The loading may be from a local file system, a remote file system, or other network services. And as the servlets are declared in the web.xml, this file is loaded and read by the container during container startup.
Initialization:
After the servlet object is instantiated, the container must initialize the servlet before
it can handle requests from clients.The container initializes the servlet instance by calling the init method of the Servlet interface with a unique (per servlet declaration) object implementing the ServletConfig interface.
Ref: JSR-000315 JavaTM Servlet 3.0

web.xml is read as soon as you deploy your application on a web server. For the sake of understanding, you can assume container is nothing but your web server. Although web server has more than just a web container.
Web server reads the web.xml, and loads the context config, load on startup servelts,etc. web.xml is the file through which you tell your container/server about your application. Your web application sits inside the web server, and server intercepts all the incoming requests, decides to which application the request should be forwarded depending on the context.

How to retrieve Spring context loaded in Tomcat from outside application

I'm trying to do the following:
Imagine you have a Java EE application running on Tomcat using Spring as the IoC engine.
I have another jar in the application that has full access to all the resources. i.e. I can instantiate the same application context that is running in tomcat but it takes around 30 seconds to instantiate all the dependencies.
Anyone knows if it is feasible to retrieve current tomcats ApplicationContext from the outside?
There is a way to ask for the WebApplicationContext inside a servlet but I'm not on it, I only have a jar with a main method.

No. You have to be running within tomcat (a webapp) to be able to access the servlet context (and from there - the application context). You are not even in the same runtime with the main method.
If you want to get some information from the context, you should expose it as a service. For example:
a restful service, via Spring-MVC
via JMX
via JNDI