I want to generate the IR for some java bytecode. The project I am using is using Wala as the static analysis tool for other reasons. One of the steps that I need to do is to create class hierarchy for a given scope of interest. Now it seems that the creation of the class hierarchy is an expensive process, thus minimizing it is desirable specially if I need to call it repeatedly.
I tried to narrow the scope of the class hierarchy by using AnalysisScopeReader.readJavaScope(...) where I specify a scope file that contains only a single entry, mainly the class that I want to analyse, which is loaded using Application loader. However that does not seem to work because I get an error that com.ibm.wala.ipa.cha.ClassHierarchyException: failed to load root <Primordial,Ljava/lang/Object> of class hierarchy. It seems that Wala is expecting to have the jre.jar in the scope file too, so it can find Ljava/lang/Object. However that is problematic, because if I added with line to the scope file, the creation of class hierarchy will include so many classes that I am not going to use, and which i'd rather not spend time or memory on.
It seems that people have been struggling with this issue here too but was never resolved.
To be concrete, here is what I do
scope2 = AnalysisScopeReader.readJavaScope(somePath, null, urlcl);
cha = ClassHierarchyFactory.makeWithRoot(scope2);
I really do not want/need to create class hierarchy in my analysis, I am not using any call graph analysis. I need only to create the IR for a single class that I am analyzing without having to pay for creating hierarchy of classes which I am not going to use. Or at least, I want the class hierarchy is contains a single file that I am currently analyzing.
It seems that ClassHierarchyFactor.makeWithRoot() almost works except that it still requires Object class to use for missing superclasses, which means that I need to add the java lang class hierarchy into the analysis.
The solution here (How to get rid of the unrelated classes when using WALA to analyze Java bytecode?) seems to be a hack rather than a fix.
I am not sure if there is a way to do that, can anyone help? Is there anyway in Wala that I can get ssa IR for a single class without having to construct a class hierarchy that is so big java lang class hierarchy?
Many thanks!
The following should get you minimal class hierarchy for a single targetClass
private ClassHierarchy getClassHierarchy(Class<?> targetClass) {
AnalysisScope scope = getAnalysisScope(targetClass);
return ClassHierarchyFactory.makeWithRoot(scope);
}
where
private AnalysisScope getAnalysisScope(Class<?> targetClass) {
String targetFile = targetClass.getResource(targetClass.getSimpleName() + ".class").getFile();
FileSystem fs = FileSystems.getFileSystem(URI.create("jrt:/"));
byte[] bytes = Files.readAllBytes(fs.getPath("modules", "java.base", "java/lang/Object.class"));
Path path = Path.of(System.getProperty("user.dir"), "target", "test-classes", "Object.class");
Files.write(path, bytes);
AnalysisScope scope = AnalysisScope.createJavaAnalysisScope();
scope.addClassFileToScope(ClassLoaderReference.Primordial, path.toFile());
scope.addClassFileToScope(ClassLoaderReference.Application, new File(targetFile));
return scope;
}
Related
I am working on a project where every model has this line:
#Model(adaptables = { SlingHttpServletRequest.class,Resource.class },
defaultInjectionStrategy = DefaultInjectionStrategy.OPTIONAL)
In my understanding:
If Resource or SlingHTTPRequest is not to be used, this dependency injection must be removed from the model
SlingHTTPRequest can help obtain resource with the use of .getResource method anyway, so using SlingHTTPServeltRequest class alone, with required dependencyInjectionStrategy should be sufficient, and Resource class as an adaptable is never needed?
Please share your thoughts. Thanks in advance!
Question 1)
A SlingModel MUST be either created/adapted from SlingHttpServletRequest or a Resource. It cannot be created from nothing.
The the adaptables-property specifies, from which object it can be created.
If the SlingModel can be created from both, the scripting-environment (e.g. HTL scripts) will use the Resource. But SlingModels can be used elsewhere too, so the source will be random.
Hint 1: Do not use both adaptables. So either decide for SlingHttpServletRequest or Resource. Because both will work, but the injecting will be different - and can cause weird bugs (at least it is thin ice, and hard to test). The example with #Self is simple, but some other injectors are even more complicated, as the implicitly the #Via changes.
#Model(adaptables = { SlingHttpServletRequest.class, Resource.class },
defaultInjectionStrategy = DefaultInjectionStrategy.OPTIONAL)
public class MySlingModel {
#Self
// will be null, if adapted from Resource!!!
private SlingHttpServletRequest request;
#Self
// will be null, if adapted from SlingHttpServletRequest!!!
private Resource resource;
Question 2
Components (and so SlingModels) should be context-free, and be represented by a Resource (= JCR node and evtl. some sub-nodes).
So a SlingModel should normally adapted from a Resource. It is also easier to use elsewhere (in other services or sling-models).
Only, if you need something from the request, then switch to the SlingHttpServletRequest. Unfortunately this is often needed for URL mapping. But limit yourself to access RequestAttributes. Even something like WcmMode should not be used in a SlingModel. It is better to see SlingModels as a Wrapper around a Resource, a small Java-Layer to access data.
Hint 2: Not everything is a SlingModel! You can create Services, Servlets, AdapterFactories, Filters, Rewriter, ...
yes its not mandatory. but if your data is in a nested structure (think of slides inside a carousel), and you need to adapt a nested resource (slide) to model, you need resource as adaptable. request.getResource will get you the component resource,
additionally, you might have to work with resources without a request object, say in a workflowprocessstep or a sling job processor. capability to just do resource.adaptTo saves you a bit of time.
I am building a dynamic Web Project (in Eclipse with Tomcat as server) using servlets and JSPs. The general purpose of the project is to let the user solve small code problems. In order to do that I take a method written by the user and use it to build a java file that I can run via Reflection. The problem I can't seem to figure out is that Tomcat (or Eclipse?) does not update the file at runtime. So when I create the file using the code of the current user and try to compile it, my program always executes the file as it was when I started the server using the code of the previous user. How can I tell it to update the file before running it?
Edit:
That's how I create the file:
public boolean writeFile() {
try {
PrintWriter writer = new PrintWriter(relativePath + "src\\testfiles\\TestFile.java");
writer.print(content);
writer.close();
return true; }...
Here I call the writer and try running the file:
FileWriter writer = new FileWriter(content);
if(writer.writeFile()){
Class<?> TestFile;
Method m;
try {
TestFile = cl.loadClass("testfiles.TestFile");
m = TestFile.getDeclaredMethod("testSolution");
m.invoke(null);
Thanks in advance!
Ok, it's now clear what the issue is. Your issue is not with Tomcat not reloading the file, but with the classloader not reloading the class.
Normal classloaders will only load a class once, and keep it cached forever. The only way for a class to get unloaded is by its classloader being garbage collected. To reload a class you either have to use a different classloader each time (with the previous one getting garbage collected or you'll run out of memory), or to have a custom loader thar doesn't cache.
See this article for an implementation of a custom classloader that does what you want.
You could theoretically just have a new class each time (by changing its name on each save) but you'd eventually run out of memory.
Maybe the easiest way is to instantiate a new classloader in the method itself, load a class, run the method on it, and don't keep any other references to the loader or the class. That way, you'll get a freshly loaded class each time, and the old instances of both classes and loaders will get garbage collected.
UPDATE: I was operating under the assumption that you already know how to compile a class at runtime but, based on the comments, that's not the case.
A classloader can, of course, only load a compiled class, so a source directly is not very useful.
Java internally provides a a compiler interface under javax.tools.JavaCompiler, but it's not easy to use and requires a different handling of Java versions before and after Java 9. So it is much easier to use a library like jOOR that hides the gory parts:
Class clazz = Reflect.compile("com.example.Test",
"package com.example;" +
"public class Test {\n" +
" public String hello() {\n" +
" return \"hello\";\n" +
" }\n" +
" }")
.type();
Instead of type() to simply get the class, you can actually keep using jOOR's fluent reflection API to trigger the methods on the generated class or whatever it is you'd normally do via regular reflection.
For direct JavaCompiler usage, see e.g. this or, even better, jOOR's source code.
I recently discovered the library kryonet, which is super awesome and fits my needs excellently.
However, the one problem that I am having is developing a good strategy for registering all of the classes that can be transferred.
I know that I can write a static method in each object that will return a list of all of the classes that it uses, but I would really rather not have to do that (for my own time purposes, as well as those who will be extending these objects).
I was playing around with trying to see if there was a way to get all of the classes that an object references (in it's fields as well as it's inheritance) from the getClass() method, but I was unable to have any success.
Finally, I know that kryo has kryo.setRegistrationOptional(true) but I am having a very difficult time trying to figure out how to use it. When this option is turned on, kryo still seems to throw exceptions if I haven't registered classes. Also, this method supposed is much slower than being able to register all of the classes. I'm fine if the first time you need to send an object using this method is slow, but I don't know if I'm okay with serious performance degradation every time that I want to send an object.
Ideally, I'll have a package of objects that I will want to send using kryonet. If there was just some was to scan that package and determine all of the classes that I need to register, that would be excellent. Now not all of my clients would need to register every object, but that's something of a separate issue, and I don't know if there is a solution to that.
If anyone could point me in the right direction that would be excellent.
Classes may come from different places such as disk, network, memory (dynamically generated). Therefore, obtaining information about classes to be registered with Kryo has to be handled separately for each specific case.
If you can read classes from a jar file then the following snippet should get you started.
private static List<Class<?>> getFromJarFile(final String jar, final String packageName) throws ClassNotFoundException, IOException {
final List<Class<?>> classes = new ArrayList<Class<?>>();
final JarInputStream jarFile = new JarInputStream(new FileInputStream(jar));
JarEntry jarEntry = null;
do {
jarEntry = jarFile.getNextJarEntry();
if (jarEntry != null) {
String className = jarEntry.getName();
if (className.endsWith(".class")) {
className = className.substring(0, className.lastIndexOf('.')); // strip filename extension
if (className.startsWith(packageName + "/")) { // match classes in the specified package and its subpackages
classes.add(Class.forName(className.replace('/', '.')));
}
}
}
} while (jarEntry != null);
return classes;
}
I'm trying to integrate Spring in a pretty large application with thousands of classes, and i'm experiencing huge delays starting my container because of component-scanning.
I have already narrowed the number of directories specified in the "base-package", to the minimum in order to reduce the time wasted in scanning irrelevant directories, but the class-path scanning part of initialization still takes about 1-2 mins.
So, is there a way to optimize the scanning process ? I've thought of storing the candidate classes path in a file and make the container then get them from the file instead of scanning the class-path with every startup, but i don't really know where to start or if that is even possible.
Any advice is much appreciated. Thanks in advance.
Edit1: Loading bean definitions form an autogenerated xml file, reduced the Spring bootstrap time to 9~10 secs which confirms that the reflection api used by Spring for the components class-path scanning is the major source of startup delays.
As for generating the xml file here is the code, since it might be helpful for someone with the same issues.
import java.io.File;
import java.io.FileNotFoundException;
import java.io.PrintWriter;
import java.util.ArrayList;
public class ConfigurationWriter {
public ArrayList<String> beanDefinitions = new ArrayList<String>();
public ConfigurationWriter() {
// the context loaded with old fashioned way (classpath scanning)
ApplicationContext context = SpringContainerServiceImpl.getInstance().getContext();
String[] tab = context.getBeanDefinitionNames();
for (int i = 0; i < tab.length - 6; i++) {
Class clazz = context.getType(tab[i]);
String scope = context.isPrototype(tab[i]) ? "prototype" : "singleton";
String s = "<bean id=\"" + tab[i] + "\" class=\"" + clazz.getName() + "\" scope=\"" + scope + "\"/>";
beanDefinitions.add(s);
}
// Collections.addAll(beanDefinitions, tab);
}
#SuppressWarnings("restriction")
public void generateConfiguration() throws FileNotFoundException {
File xmlConfig = new File("D:\\dev\\svn\\...\\...\\src\\test\\resources\\springBoost.xml");
PrintWriter printer = new PrintWriter(xmlConfig);
generateHeader(printer);
generateCorpse(printer);
generateTail(printer);
printer.checkError();
}
#SuppressWarnings("restriction")
private void generateCorpse(PrintWriter printer) {
for (String beanPath : beanDefinitions) {
printer.println(beanPath);
}
}
#SuppressWarnings("restriction")
private void generateHeader(PrintWriter printer) {
printer.println("<?xml version=\"1.0\" encoding=\"UTF-8\"?>");
printer.println("<beans xmlns=\"http://www.springframework.org/schema/beans\"");
printer.println("xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"");
printer.println("xmlns:context=\"http://www.springframework.org/schema/context\"");
printer.println("xsi:schemaLocation=\"");
printer.println("http://www.springframework.org/schema/mvc");
printer.println("http://www.springframework.org/schema/mvc/spring-mvc-3.0.xsd");
printer.println("http://www.springframework.org/schema/beans");
printer.println("http://www.springframework.org/schema/beans/spring-beans-3.0.xsd");
printer.println("http://www.springframework.org/schema/context");
printer.println("http://www.springframework.org/schema/context/spring-context-3.0.xsd\"");
printer.println("default-lazy-init=\"true\">");
}
#SuppressWarnings("restriction")
private void generateTail(PrintWriter printer) {
// printer.println("<bean class=\"com.xxx.frmwrk.spring.processors.xxxBeanFactoryPostProcessor\"/>");
printer.println("<bean class=\"com.xxx.frmwrk.spring.processors.xxxPostProcessor\"/>");
printer.println("</beans>");
}
}
Edit 2: With Spring 5 including an important set of optimizations for speeding up the context initialization, It also comes with an interesting and handy feature that enables generating an index of candidate components at compile time : Spring Context Indexer
Question: How many (in %) of the classes in the directories are Spring Beans?
Answer: I'm not really sure (it's a really big project) , but from what i saw i believe it's arround 90 to 100%, since xml and properties files are isolated in separate locations)
If the problem is really the component scan and not the bean initializing process itself (and I highly doubt that), then the only solution I can imagine is to use Spring XML configuration instead of component scan. - (May you can create the XML file automatically).
But if you have many classes and 90% - 100% of them are Beans, then, the reduction of scanned files will have a maximal improvement of 10%-0%.
You should try other ways to speed up your initialization, may using lazy loading or any lazy loading related techniques, or (and that is not a joke) use faster hardware (if it is not a stand alone application).
A easy way to generate the Spring XML is to write a simple spring application that uses the class path scanning like your original application. After all Beans are initialize, it iterates through the Beans in the Spring Context, check if the bean belongs to the important package and write the XML Config for this bean in a file.
Auto discovery of annotated classes currently requires to scan all classes in the specified package(s) and can take a long time, a known problem of the current class loading mechanism.
Java 9 is going to help here with Jigsaw.
From the Java Platform Module System requirements by Mark Reinold, http://openjdk.java.net/projects/jigsaw/spec/reqs/ :
Efficient annotation detection —
It must be possible to identify all of the class files in a module artifact in which a particular annotation is present without actually reading all of the class files. At run time it must be possible to identify all of the classes in a loaded module in which a particular annotation is present without enumerating all of the classes in the module, so long as the annotation was retained for run time. For efficiency it may be necessary to specify that only certain annotations need to be detectable in this manner.
One potential approach is to augment a module’s definition with an index of the annotations that are present in the module, together with an indication of the elements to which each annotation applies. To limit the size of the index, only annotations which themselves are annotated with a new meta-annotation, say #Indexed, would be included.
Not much you can do about the performance there, I guess you aren't concerned about the startup in production environment, but the startup time of your tests*.
Two tips:
Review that your test-appcontext only uses the minimally required components of your app
instead of having a list of component-scan directives, use one, with a comma-separated value like this: base-package="com.package.one,com.package.two..."
I know it is an old question, and as you will see the situation was different at that time, but hopefully it can help others researching this issue as I did.
According to this answer to a different question, The #ComponentScan annotation now supports a lazyInit flag, which should help in reducing start-up time.
https://stackoverflow.com/a/29832836/4266381
Note: Your edit made it sound like switching to XML by itself was the magic. Yet, looking closer at the code, you had default-lazy-init="true". I wonder if that was the true reason.
The only thing that comes in my mind, beside reducing the directories to be scanned, is the use of lazy bean initialization.
May this could help if you have a lot of beans
You could use Spring's Java-based container configuration instead of component scan.
In comparison to XML-based configuration the Java-based container configuration is type-safe.
But first of all you should check whether your component scan paths are specific enough so that they do not include classes of third party libraries.
I am writing a static analysis tool for an assignment, it analyses Java bytecode using the ASM library. One of the parts of ASM that we use requires (or at least, appears to require) that the class be loaded from the ClassLoader.
We were hoping the tool would be able to analyse .class files without requiring them on the classpath. We already load the .classes from a specified directory at run time and read them in using an InputStream. This is acceptable for ASM in most cases. There are some classes, such as SimpleVerifier, which attempt to load the classes though.
Is it possible, under this scenario, to register the .class files to be loaded so that calls to Class.forName() will load them? Or is there an easy way to extend the ClassLoader to allow this?
Edit: the information on URLClassLoader was useful. Unfortunately, using Thread.currentThread().setContextClassLoader() to an instance of that didn't work in this scenario. The library code I'm calling into uses a loader it retrieves on instance initialisation using getClass().getClassLoader().
By the time I set the URLClassLoader the class hasn't been initialised so I guess the contextClassLoader does not load that class.
Have I understand the responses correctly? Would using the URLClassLoader to load the 3rd party class be a possibility?
Almost.
If you have classes compiled somewhere, you can load them with a URLClassLoader. You can then set this ClassLoader to be the ClassLoader for the current Thread: Thread.setContextClassLoader(ClassLoader)
Users can that get the current threads context class loader and use that to access the class definition.
First of all, ASM can be used in a such way that it won't use ClassLoader to obtain information about classes.
There are several places in ASM framework where it loads classes by default but all those places can be overridden in your own subclasses. Out of the top of my head:
ClassWriter.getCommonSuperClass() method is called only when ClassWriter.COMPUTE_FRAMES flag is used and can be overwriten to not use ClassLoader to get inforamtion about classes. You can find an example of that in ClassWriterComputeFramesTest that introduces a ClassInfo abstraction
Similarly SimpleVerifier.getClass() method is used by SimpleVerifier.isAssignableFrom() and you can overwrite the latter and use the ClassInfo abstraction to find the common super type. If I am not mistaken, AspectWerkz project had implemented similar thing in its type pattern matching code. Also note that there is SimpleVerifier.setClassLoader() method, which you can use if you still want to load your own classes.
On a side note, on a Sun's JVMs, loaded classes gets to PermGen area and can't be unloaded, so it is not a good idea to load classes only for static code analysis purposes if you can avoid that, especially if tool would be integrated into a long-live process, such as IDE.
You can't, as far as I know, extend the System class loader at runtime, but you can dynamically load classes from an arbitrary location (jar or directory) using URLClassLoader.
You could try to setup a "launcher" in the startup of your application that creates an URLClassLoader passing it the locations on the classpath and your own .class locations and start the application from that classloader.
When the SimpleVerifier is loaded by the URLClassLoader it will also be able to load the classes from the extra locations.
Yes, you can use URLClassLoader
I have a test where I do load the class at runtime. This class is not in the classpath (nor even exist when the test is run for that matter ), later is it loaded and works great.
Here's the code.
void testHello() throws MalformedURLException, ClassNotFoundException {
URL[] url = {
new URL("file:/home/oreyes/testwork/")
};
try {
new URLClassLoader(url).loadClass("Hello");
throw new AssertionError("Should've thrown ClassNotFoundException");
} catch ( ClassNotFoundException cnfe ){}
c.process();// create the .class file
new URLClassLoader(url).loadClass("Hello");
// it works!!
}
Taken from this question.
I created my own ClassLoader its quite simple.
/**
* Used to hold the bytecode for the class to be loaded.
*/
private final static ThreadLocal<byte[]> BYTE_CODE = new ThreadLocal<byte[]>();
#Override
protected Class<?> findClass(final String name) throws ClassNotFoundException {
final byte[] bytes = BYTE_CODE.get();
if (null == bytes) {
throw new ClassNotFoundException(name);
}
return this.defineClass(null, bytes, 0, bytes.length);
}