I'm trying to integrate Spring in a pretty large application with thousands of classes, and i'm experiencing huge delays starting my container because of component-scanning.
I have already narrowed the number of directories specified in the "base-package", to the minimum in order to reduce the time wasted in scanning irrelevant directories, but the class-path scanning part of initialization still takes about 1-2 mins.
So, is there a way to optimize the scanning process ? I've thought of storing the candidate classes path in a file and make the container then get them from the file instead of scanning the class-path with every startup, but i don't really know where to start or if that is even possible.
Any advice is much appreciated. Thanks in advance.
Edit1: Loading bean definitions form an autogenerated xml file, reduced the Spring bootstrap time to 9~10 secs which confirms that the reflection api used by Spring for the components class-path scanning is the major source of startup delays.
As for generating the xml file here is the code, since it might be helpful for someone with the same issues.
import java.io.File;
import java.io.FileNotFoundException;
import java.io.PrintWriter;
import java.util.ArrayList;
public class ConfigurationWriter {
public ArrayList<String> beanDefinitions = new ArrayList<String>();
public ConfigurationWriter() {
// the context loaded with old fashioned way (classpath scanning)
ApplicationContext context = SpringContainerServiceImpl.getInstance().getContext();
String[] tab = context.getBeanDefinitionNames();
for (int i = 0; i < tab.length - 6; i++) {
Class clazz = context.getType(tab[i]);
String scope = context.isPrototype(tab[i]) ? "prototype" : "singleton";
String s = "<bean id=\"" + tab[i] + "\" class=\"" + clazz.getName() + "\" scope=\"" + scope + "\"/>";
beanDefinitions.add(s);
}
// Collections.addAll(beanDefinitions, tab);
}
#SuppressWarnings("restriction")
public void generateConfiguration() throws FileNotFoundException {
File xmlConfig = new File("D:\\dev\\svn\\...\\...\\src\\test\\resources\\springBoost.xml");
PrintWriter printer = new PrintWriter(xmlConfig);
generateHeader(printer);
generateCorpse(printer);
generateTail(printer);
printer.checkError();
}
#SuppressWarnings("restriction")
private void generateCorpse(PrintWriter printer) {
for (String beanPath : beanDefinitions) {
printer.println(beanPath);
}
}
#SuppressWarnings("restriction")
private void generateHeader(PrintWriter printer) {
printer.println("<?xml version=\"1.0\" encoding=\"UTF-8\"?>");
printer.println("<beans xmlns=\"http://www.springframework.org/schema/beans\"");
printer.println("xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"");
printer.println("xmlns:context=\"http://www.springframework.org/schema/context\"");
printer.println("xsi:schemaLocation=\"");
printer.println("http://www.springframework.org/schema/mvc");
printer.println("http://www.springframework.org/schema/mvc/spring-mvc-3.0.xsd");
printer.println("http://www.springframework.org/schema/beans");
printer.println("http://www.springframework.org/schema/beans/spring-beans-3.0.xsd");
printer.println("http://www.springframework.org/schema/context");
printer.println("http://www.springframework.org/schema/context/spring-context-3.0.xsd\"");
printer.println("default-lazy-init=\"true\">");
}
#SuppressWarnings("restriction")
private void generateTail(PrintWriter printer) {
// printer.println("<bean class=\"com.xxx.frmwrk.spring.processors.xxxBeanFactoryPostProcessor\"/>");
printer.println("<bean class=\"com.xxx.frmwrk.spring.processors.xxxPostProcessor\"/>");
printer.println("</beans>");
}
}
Edit 2: With Spring 5 including an important set of optimizations for speeding up the context initialization, It also comes with an interesting and handy feature that enables generating an index of candidate components at compile time : Spring Context Indexer
Question: How many (in %) of the classes in the directories are Spring Beans?
Answer: I'm not really sure (it's a really big project) , but from what i saw i believe it's arround 90 to 100%, since xml and properties files are isolated in separate locations)
If the problem is really the component scan and not the bean initializing process itself (and I highly doubt that), then the only solution I can imagine is to use Spring XML configuration instead of component scan. - (May you can create the XML file automatically).
But if you have many classes and 90% - 100% of them are Beans, then, the reduction of scanned files will have a maximal improvement of 10%-0%.
You should try other ways to speed up your initialization, may using lazy loading or any lazy loading related techniques, or (and that is not a joke) use faster hardware (if it is not a stand alone application).
A easy way to generate the Spring XML is to write a simple spring application that uses the class path scanning like your original application. After all Beans are initialize, it iterates through the Beans in the Spring Context, check if the bean belongs to the important package and write the XML Config for this bean in a file.
Auto discovery of annotated classes currently requires to scan all classes in the specified package(s) and can take a long time, a known problem of the current class loading mechanism.
Java 9 is going to help here with Jigsaw.
From the Java Platform Module System requirements by Mark Reinold, http://openjdk.java.net/projects/jigsaw/spec/reqs/ :
Efficient annotation detection —
It must be possible to identify all of the class files in a module artifact in which a particular annotation is present without actually reading all of the class files. At run time it must be possible to identify all of the classes in a loaded module in which a particular annotation is present without enumerating all of the classes in the module, so long as the annotation was retained for run time. For efficiency it may be necessary to specify that only certain annotations need to be detectable in this manner.
One potential approach is to augment a module’s definition with an index of the annotations that are present in the module, together with an indication of the elements to which each annotation applies. To limit the size of the index, only annotations which themselves are annotated with a new meta-annotation, say #Indexed, would be included.
Not much you can do about the performance there, I guess you aren't concerned about the startup in production environment, but the startup time of your tests*.
Two tips:
Review that your test-appcontext only uses the minimally required components of your app
instead of having a list of component-scan directives, use one, with a comma-separated value like this: base-package="com.package.one,com.package.two..."
I know it is an old question, and as you will see the situation was different at that time, but hopefully it can help others researching this issue as I did.
According to this answer to a different question, The #ComponentScan annotation now supports a lazyInit flag, which should help in reducing start-up time.
https://stackoverflow.com/a/29832836/4266381
Note: Your edit made it sound like switching to XML by itself was the magic. Yet, looking closer at the code, you had default-lazy-init="true". I wonder if that was the true reason.
The only thing that comes in my mind, beside reducing the directories to be scanned, is the use of lazy bean initialization.
May this could help if you have a lot of beans
You could use Spring's Java-based container configuration instead of component scan.
In comparison to XML-based configuration the Java-based container configuration is type-safe.
But first of all you should check whether your component scan paths are specific enough so that they do not include classes of third party libraries.
Related
I am working on a project where every model has this line:
#Model(adaptables = { SlingHttpServletRequest.class,Resource.class },
defaultInjectionStrategy = DefaultInjectionStrategy.OPTIONAL)
In my understanding:
If Resource or SlingHTTPRequest is not to be used, this dependency injection must be removed from the model
SlingHTTPRequest can help obtain resource with the use of .getResource method anyway, so using SlingHTTPServeltRequest class alone, with required dependencyInjectionStrategy should be sufficient, and Resource class as an adaptable is never needed?
Please share your thoughts. Thanks in advance!
Question 1)
A SlingModel MUST be either created/adapted from SlingHttpServletRequest or a Resource. It cannot be created from nothing.
The the adaptables-property specifies, from which object it can be created.
If the SlingModel can be created from both, the scripting-environment (e.g. HTL scripts) will use the Resource. But SlingModels can be used elsewhere too, so the source will be random.
Hint 1: Do not use both adaptables. So either decide for SlingHttpServletRequest or Resource. Because both will work, but the injecting will be different - and can cause weird bugs (at least it is thin ice, and hard to test). The example with #Self is simple, but some other injectors are even more complicated, as the implicitly the #Via changes.
#Model(adaptables = { SlingHttpServletRequest.class, Resource.class },
defaultInjectionStrategy = DefaultInjectionStrategy.OPTIONAL)
public class MySlingModel {
#Self
// will be null, if adapted from Resource!!!
private SlingHttpServletRequest request;
#Self
// will be null, if adapted from SlingHttpServletRequest!!!
private Resource resource;
Question 2
Components (and so SlingModels) should be context-free, and be represented by a Resource (= JCR node and evtl. some sub-nodes).
So a SlingModel should normally adapted from a Resource. It is also easier to use elsewhere (in other services or sling-models).
Only, if you need something from the request, then switch to the SlingHttpServletRequest. Unfortunately this is often needed for URL mapping. But limit yourself to access RequestAttributes. Even something like WcmMode should not be used in a SlingModel. It is better to see SlingModels as a Wrapper around a Resource, a small Java-Layer to access data.
Hint 2: Not everything is a SlingModel! You can create Services, Servlets, AdapterFactories, Filters, Rewriter, ...
yes its not mandatory. but if your data is in a nested structure (think of slides inside a carousel), and you need to adapt a nested resource (slide) to model, you need resource as adaptable. request.getResource will get you the component resource,
additionally, you might have to work with resources without a request object, say in a workflowprocessstep or a sling job processor. capability to just do resource.adaptTo saves you a bit of time.
I have a (web-)application that needs special configurations and/or extensions based on the customer using the application. I call these additions "plugins" and they are auto discovered by classpath scanning when the application starts. For extensions that is incredibly easy. Let's say I want to have a plugin which adds an API that prints "hello world" when the URL /myplugin/greet is called: I just create a #Controller annotated class with the according #RequestMapping, put this in a myplugin.jar, copy that on the classpath and that's it.
Problems come up when I want to change some defaults and especially if I want to do this multiple times. Let's say my core application has a config like this:
#Configuration
public class CoreConfiguration {
#Bean
public Set<String> availableModules() {
return Collections.singleton("core");
}
}
Now I have two plugins that don't know about each other (but they do know the CoreConfig), but they both want to add themselves to the list of available modules. How would I do that? If I only had a single plugin that wants to override the module list I could override the existing bean from CoreConfiguration, but with two plugins that becomes a problem. What I imagine is something like this:
#Configuration
public class FirstPluginConfiguration {
#Bean
public Set<String> availableModules(Set<String> availableModules) {
Set<String> extendedSet = new HashSet<>(availableModules);
extendedSet.add("FirstPlugin");
return extendedSet;
}
}
Of course a SecondPluginConfiguration would look nearly exactly like this, except that the Set is not extended by "FirstPlugin", but by "SecondPlugin". I tested it to check what would happen and spring will just never call the First/SecondPluginConfiguration "availableModules" methods but it does not show an error either.
Now of course in this case this could easily be solved by using a mutable Set in the CoreConfiguration and then autowiring and extending the set in the other configurations, but for example I also want to be able to add method interceptors to some beans. So for example I might have an interface CrashLogger which has a logCrash(Throwable t) method and in CoreConfiguration a ToFileCrashLogger is created that writes stack traces to files as the name suggests. Now a plugin could say that he also wants to get notified about crashes, for example the plugin wants to ADDITIONALLY send the stacktrace to someone by email. For that matter that plugin could wrap the CrashLogger configured by the CoreConfiguration and fire BOTH. A second plugin could wrap the wrapper again and do something totally different with the stacktrace and still call both of the other CrashLoggers.
The later does sound somewhat like AOP and if I'd just let ALL my beans be proxied (I did not test that) I could autowire them into my plugin configurations, cast them to org.springframework.aop.framework.Advised and then add advices that manipulate behaviour. However it does seem like a huge overkill to generate proxies for each and everyone of my beans just so that that plugin can potentially add one or two advices one one or two beans.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I want to know about how spring servlet component scan works? and it scan #Service,#Component etc scan.But I am not clear how it works? What flow works behind this?
The best thing to do when you're curious about how something is implemented in an open source project is to look at the source code. Here's how I would go about it. These steps are specific to my particular configuration (XML based Spring contexts written with Eclipse) but the general theory is applicable to any environment....
1) I know from poking around the source code that the classes that parse particular XML config elements usually have a name that ends with "BeanDefinitionParser". So to see how:
<context:component-scan base-package="com.yourproject" />
is implemented, look for classes named "Component*BeanDefinitionProcessor".
2) Doing a CTRL + SHIFT + T in Eclipse and searching for "Component*BeanDefinitionProcessor" I came up with ComponentScanBeanDefinitionParser. That sounds like it's probably what I'm looking for...
3) Within that class (lines 83 - 85), there's this chunk of code:
ClassPathBeanDefinitionScanner scanner = configureScanner(parserContext, element);
Set<BeanDefinitionHolder> beanDefinitions = scanner.doScan(basePackages);
registerComponents(parserContext.getReaderContext(), beanDefinitions, element);
So the class that processes XML tags delegates to ClassPathBeanDefinitionScanner. This makes sense since Spring supports both XML configuration as well as Java based configuration. It wouldn't make sense to duplicate the actual scanning logic in the annotations config handler and the XML config handler.
4) Looking at the code for ClassPathBeanDefinitionScanner, I don't see anything that looks like it's looking for things on the classpath. However, it does extend a class named ClassPathScanningCandidateComponentProvider.
public class ClassPathBeanDefinitionScanner extends ClassPathScanningCandidateComponentProvider {
5) Looking in ClassPathScanningCandidateComponentProvider, I can see the for loop that's looping over everything in a package
public Set<BeanDefinition> findCandidateComponents(String basePackage) {
Set<BeanDefinition> candidates = new LinkedHashSet<BeanDefinition>();
try {
String packageSearchPath = ResourcePatternResolver.CLASSPATH_ALL_URL_PREFIX +
resolveBasePackage(basePackage) + "/" + this.resourcePattern;
Resource[] resources = this.resourcePatternResolver.getResources(packageSearchPath);
boolean traceEnabled = logger.isTraceEnabled();
boolean debugEnabled = logger.isDebugEnabled();
for (int i = 0; i < resources.length; i++) {
6) It's using ResourcePatternResolver to find stuff on the classpath:
private ResourcePatternResolver resourcePatternResolver = new PathMatchingResourcePatternResolver();
7) In PathMatchingResourcePatternResolver you can see the code that actually searches the classpath.
How component scan in spring?
It boils down to doing some gymnastics with class loaders and resource path matching and using the ClassLoader.getResources(String name) method which returns an Enumeration containing URLs representing classpath resources. Those resources (Java classes) are then checked to see if they contain the #Component annotation (or a specialization of it) and if so, are considered a "candidate" component. Other filtering can take place but by default those components are then defined programmatically as Spring beans. When annotation configuration is enabled, autowiring of components also takes place, so I can have Spring scan the classpath for all my web controllers, and then automatically inject dependencies such as services or data access objects.
The actual classes that perform this magic are the ClassPathScanningCandidateComponentProvider which, by default, uses a PathMatchingResourcePatternResolver to find matching classpath resources using ClassLoader.getResources().
As per Spring documentation Spring beans are singletons by default.
As with Spring-managed components in general, the default and by far
most common scope is 'singleton'. However, there are times when other
scopes are needed. Therefore Spring 2.5 introduces a new #Scope
annotation as well. Simply provide the name of the scope within the
annotation
#Scope(StandardScopes.PROTOTYPE)
#Repository
public class MovieFinderImpl implements MovieFinder {
// ...
}
Different Bean scopes are avialable here.
I'm working on a JSR-303 validation framework for GWT. Some of you may have heard of it even though it is a small project. Here is gwt-validation.
In the old days (v1.0) it used a marker interface for each class and each class had metadata generated separately. This was bad because it was not part of the JSR-303 standard and we moved on to the next idea.
In version 2.0 it scans the classpath at runtime using Reflections. This is great. The downside is that it doesn't seem to be able to work inside of containerized environments or those with special restrictions.
This is probably my fault, look at the following code:
//this little snippet goes through the classpath urls and ommits jars that are on the forbidden list.
//this is intended to remove jars from the classpath that we know are not ones that will contain patterns
Set<URL> classPathUrls = ClasspathHelper.forJavaClassPath();
Set<URL> useableUrls = new HashSet<URL>();
for(URL url : classPathUrls) {
boolean use = true;
for(String jar : this.doNotScanJarsInThisList) {
if(url.toString().contains(jar)) {
use = false;
break;
}
}
if(use) {
useableUrls.add(url);
}
use = false;
}
ConfigurationBuilder builder = new ConfigurationBuilder()
.setUrls(useableUrls)
.setScanners( new TypeAnnotationsScanner(),
new FieldAnnotationsScanner(),
new MethodAnnotationsScanner(),
new SubTypesScanner()
)
.useParallelExecutor()
;
this.reflections = new Reflections(builder);
I'm using the filter to remove jars that I know can't have annotations in them that I'm interested in. As I mention this gives a huge speed boost (especially on large classpaths) but the ClasspathHelper.forJavaClassPath() that I'm basing this on probably isn't the best way to go in container environments. (e.g. Tomcat, JBoss)
Is there a better way or at least a way that will work with a container environment and still let my users filter out classes they don't want?
I've looked, some, into how the Hibernate Validation project (the reference implementation for JSR-303) and they appear to at least be using (at least in part) the Annotations Processing in Java 6. This can't be all of the story because that didn't show up until JDK6 and Hibernate Validator is JDK5 compatible. (See: hibernate documentation)
So, as always, there's more to the story.
I've read these threads, for reference:
About Scannotation which has been pretty much replaced by Reflections.
This one but it uses File and I'm not sure what the implications are of that in things like GAE (Google App Engine) or Tomcat.
Another that goes over a lot of the things I've talked about already.
These threads have only helped so much.
I've also read about the annotation processing framework and I must be missing something. It appears to do what I want but then again it appears to only work at compile time which I know isn't what is done by Hibernate Validator. (Can anyone explain how it does scanning? It works on GAE which means it can't use any of the IO packages.)
Further, would this code work better than what I have above?
Set<URL> classPathUrls = ClasspathHelper.forClassLoader(Thread.currentThread().getContextClassLoader());
Could that correctly get the classloader inside of a Tomcat or JBoss container? It seems scan a smaller set of classes and still finish okay.
So, in any case, can anyone help me get pointed in the right direction? Or am I just stuck with what I've got?
You could take a look at Spring's annotation support.
Spring can scan annotations in files (using asm IIRC), and works in and out of a container.
It may not be easy because it goes through Spring's Resource abstraction, but it should be doable to reuse (or extract) the relevant code.
I am trying to speed up the Integration tests in our environment. All our classes are autowired. In our applicationContext.xml file we have defined the following:
<context:annotation-config/>
<context:component-scan base-package="com.mycompany.framework"/>
<context:component-scan base-package="com.mycompany.service"/>
...additional directories
I have noticed that Spring is scanning all directories indicated above and then iterates over each bean and caches the properties of each one. (I went over the DEBUG messages from spring)
As a result, the following test takes about 14 seconds to run:
public class MyTest extends BaseSpringTest {
#Test
def void myTest(){
println "test"
}
}
Is there any way to lazy load the configuration? I tried adding default-lazy-init="true" but that didn't work.
Ideally, only the beans required for the test are instantiated.
thanks in advance.
Update: I should have stated this before, I do not want to have a context file for each test. I also do not think one context file for just the tests would work. (This test context file would end up including everything)
If you really want to speed up your application context, disable your <component-scan and performs the following routine before running any test
Resource resource = new ClassPathResource(<PUT_XML_PATH_RIGHT_HERE>); // source.xml, for instance
InputStream in = resource.getInputStream();
Document document = new SAXReader().read(in);
Element root = document.getRootElement();
/**
* remove component-scanning
*/
for ( Iterator i = root.elementIterator(); i.hasNext(); ) {
Element element = (Element) i.next();
if(element.getNamespacePrefix().equals("context") && element.getName().equals("component-scan"))
root.remove(element);
}
in.close();
ClassPathScanningCandidateComponentProvider scanner = new ClassPathScanningCandidateComponentProvider(true);
for (String source: new String[] {"com.mycompany.framework", "com.mycompany.service"}) {
for (BeanDefinition bd: scanner.findCandidateComponents(source)) {
root
.addElement("bean")
.addAttribute("class", bd.getBeanClassName());
}
}
//add attribute default-lazy-init = true
root.addAttribute("default-lazy-init","true");
/**
* creates a new xml file which will be used for testing
*/
XMLWriter output = new XMLWriter(new FileWriter(<SET_UP_DESTINATION_RIGHT_HERE>));
output.write(document);
output.close();
Besides that, enable <context:annotation-config/>
As you need to perform the routine above before running any test, you can create an abstract class where you can run the following
Set up a Java system property for testing environment as follows
-Doptimized-application-context=false
And
public abstract class Initializer {
#BeforeClass
public static void setUpOptimizedApplicationContextFile() {
if(System.getProperty("optimized-application-context").equals("false")) {
// do as shown above
// and
System.setProperty("optimized-application-context", "true");
}
}
}
Now, for each test class, just extends Initializer
One approach is to skip the auto detection completely and either load up a separate context (with the components required for the test) or redefine your beans at runtime (prior to the test running).
This thread discusses redefinition of beans and a custom test class for doing this:
Spring beans redefinition in unit test environment
This is the price you pay for auto-detection of components - it's slower. Even though your test only requires certain beans, your <context:component-scan> is much broader, and Spring will instantiate and initialise every bean it finds.
I suggest that you use a different beans file for your tests, one which only defines the beans necessary for the test itself, i.e. not using <context:component-scan>.
Probably what you need is to refactor your config to use less autowiring. My approach is almost always wire the beans by name, trying to be explicit with the design but, at the same time, not being too verbose either, using autowiring when is clear that you are using it in order to hide minor details.
Addendum:
If that is not enough and you are using junit, you may want to use a utility from the JUnit Addons project. The class DirectorySuiteBuilder dynamically builds up a test suite from a directory structure. So you can make something like
DirectorySuiteBuilder builder = new DirectorySuiteBuilder();
Test suite = builder.suite("project/tests");
Initializing the Spring context before this code, you can run all tests at once. However, if each test assume a "clean" Spring context, then you are probably lost.
In this kind of situation, you will need to find a balance.
On one hand, you would rightly want to run the tests in a shortest possible time to get the results quick. This is especially important when working in a team environment with continuous integration working.
On the other hand, you would also rightly want to keep the configuration of tests as simple as possible so the maintenance of test suite would not become too cumbersome to be useful.
But at the end of the day, you will need to find your own balance and make a decision.
I would recommend creating a few context configuration files for testing to group some tests so such a simple test would not take long time simply being configured by Spring, while keeping the number of configuration files to minimum you can manage.
Convention bean factory is designed to solve this problem and speeds up the whole process significantly, 3x or more.
Since none of the answers here solved this problem for me, I add my own experience.
My problem was that Spring, Hibernate and EhCache grouped up in the attempt of drowning my console with verbose DEBUG messages, resulting unreadable log and - far worse - unbearable low performance.
Configuring their log levels fixed all up:
Logger.getLogger("org.hibernate").setLevel(Level.INFO);
Logger.getLogger("net.sf.ehcache").setLevel(Level.INFO);
Logger.getLogger("org.springframework").setLevel(Level.INFO);