Develop Reference

Integrating Cloudinary with Adobe AEM

I am trying to integrate Adobe AEM 6.3 (running on Java 1.8) with Cloudinary SDK. I have done the following but, keep hitting an exception that I am not able to resolve. Has anyone integrated Cloudinary with AEM and run into similar issues?
Add the dependency in the pom.xml for compiling the code.
<dependency>
<groupId>com.cloudinary</groupId>
<artifactId>cloudinary-core</artifactId>
<version>1.24.0</version>
</dependency>
<dependency>
<groupId>com.cloudinary</groupId>
<artifactId>cloudinary-http44</artifactId>
<version>1.24.0</version>
</dependency>
Build an OSGI plugin to ensure AEM gets the right jar files. For this purpose, I followed the steps to create a third party RESTful service example. To build the bundle, I had to explicitly download the following jar files: cloudinary-1.0.14.jar, cloudinary-core-1.21.0.jar, cloudinary-http44-1.21.0.jar, commons-codec-1.10.jar, commons-collections-3.2.2.jar, commons-lang3-3.1.jar, commons-logging-1.2.jar, httpclient-4.4.jar, httpmime-4.4.jar, jsp-api-2.0.jar
Despite creating a bundle that has httpclient, I get the following exception when trying to upload an image to Cloudinary. Here's code and the exception.
Code snippet
import com.cloudinary.*;
..
Cloudinary cloudinary = new Cloudinary("<<credentials>>");
...
File toUpload = new File("/Users/akshayranganath/Downloads/background-2633962_1280.jpg");
try {
Map uploadResult = cloudinary.uploader().upload(toUpload, ObjectUtils.emptyMap());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Exception
Caused by: java.lang.NoClassDefFoundError: javax/net/ssl/HostnameVerifier
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.defineClass(BundleWiringImpl.java:2370)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.findClass(BundleWiringImpl.java:2154)
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1542)
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:79)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:2018)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.http.impl.conn.SchemeRegistryFactory.createDefault(SchemeRegistryFactory.java:52)
at org.apache.http.impl.client.AbstractHttpClient.createClientConnectionManager(AbstractHttpClient.java:321)
at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:484)
at org.apache.http.impl.client.AbstractHttpClient.createHttpContext(AbstractHttpClient.java:301)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:818)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at com.cloudinary.Uploader.callApi(Uploader.java:317)
at com.cloudinary.Uploader.upload(Uploader.java:57)
at com.aem.community.core.models.HelloWorldModel.init(HelloWorldModel.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.sling.models.impl.ModelAdapterFactory.invokePostConstruct(ModelAdapterFactory.java:792)
at org.apache.sling.models.impl.ModelAdapterFactory.createObject(ModelAdapterFactory.java:607)
... 211 common frames omitted
Caused by: java.lang.ClassNotFoundException: javax.net.ssl.HostnameVerifier not found by MyBundle [550]
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1574)
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:79)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:2018)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 236 common frames omitted
This is the first time I am working with AEM and I may not be following the right steps. Please let me know if anyone has been able to get past this issue.
Update
Based on Alexander's suggestion and a pointer from another source, I added the following code to the parent pom.xml file.
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<version>3.5.0</version>
<configuration>
<instructions>
<Embed-Dependency>*;scope=compile|runtime</Embed-Dependency>
<Embed-Directory>OSGI-INF/lib</Embed-Directory>
<Embed-Transitive>true</Embed-Transitive>
</instructions>
</configuration>
</plugin>
After making this change, the cloudinary libraries were being added to the bundle. Here's the output from AEM: http://localhost:4502/system/console/bundles
Embedded-Artifacts: OSGI-INF/lib/cloudinary-http44-1.21.0.jar; g="com.cloudinary"; a="cloudinary-http44"; v="1.21.0", OSGI-INF/lib/commons-lang3-3.1.jar; g="org.apache.commons"; a="commons-lang3"; v="3.1", OSGI-INF/lib/httpclient-4.4.jar; g="org.apache.httpcomponents"; a="httpclient"; v="4.4", OSGI-INF/lib/httpcore-4.4.jar; g="org.apache.httpcomponents"; a="httpcore"; v="4.4", OSGI-INF/lib/commons-logging-1.2.jar; g="commons-logging"; a="commons-logging"; v="1.2", OSGI-INF/lib/commons-codec-1.9.jar; g="commons-codec"; a="commons-codec"; v="1.9", OSGI-INF/lib/httpmime-4.4.jar; g="org.apache.httpcomponents"; a="httpmime"; v="4.4", OSGI-INF/lib/cloudinary-core-1.21.0.jar; g="com.cloudinary"; a="cloudinary-core"; v="1.21.0"
However, I now get an error with this message:
org.apache.avalon.framework.logger -- Cannot be resolved
org.apache.log -- Cannot be resolved
I am able to resolve the org.apache.avalon.framework.logger error by adding a dependency Avalon framework. But, I am not able to get over the org.apache.log issue. It looks like there is a version conflict that is causing the problem.
This new error starts when I include the Cloudinary http44 library. This library doesn't appear to directly reference logging (see here for dependencies). Due to this error, the application still fails to go from Installed to Active state.

Cloudinary-libs are available as Maven artifacts. Such JAR-files can be put in your bundle as private libraries with the maven-bundle-plugin.
The following sample works for me (even with Cloudinary test account)
...
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<executions>
<execution>
<!-- Create the bundle late in the compile-phase instead of the package-phase.
So the generated OSGi meta-data is available during JUnit tests. -->
<id>run-before-tests</id>
<phase>process-classes</phase>
<goals>
<goal>bundle</goal>
</goals>
</execution>
</executions>
<configuration>
<instructions>
<Bundle-Name>Test Bundle</Bundle-Name>
<Embed-Dependency>*;groupId=com.cloudinary;scope=compile|runtime</Embed-Dependency>
<Embed-Directory>OSGI-INF/lib</Embed-Directory> <!-- not needed, but nice -->
<Embed-Transitive>true</Embed-Transitive>
</instructions>
</configuration>
</plugin>
...
<dependencies>
<dependency>
<groupId>com.cloudinary</groupId>
<artifactId>cloudinary-core</artifactId>
<version>1.24.0</version>
</dependency>
<dependency>
<groupId>com.cloudinary</groupId>
<artifactId>cloudinary-http44</artifactId>
<version>1.24.0</version>
</dependency>
...
In general embedding an external library can be from simple, cumbersome to impossible. It depends on the dependencies of the imported artifacts.
Check the dependency tree manually! (e.g. https://mvnrepository.com/)
You have to fiddle with 3 instructions:
Embed-Dependency
This are the libraries, that are put in your bundle. Be careful with the asterisk operator, otherwise you may include way too many dependencies (in case of AEM easily half of the internet). But do not include too less! Extract the built bundle.jar, to see what is actually included (in case of cloudinary it was easy).
Import-Package
Often the libs have way too many dependencies, especially if libs come an other ecosystem (like Spring or JEE containers), or have a lot of semi-optional dependencies. With this setting you can tell OSGi, that a bundle can be activated, even if certain dependencies are not available.
This is a real world example :
<Import-Package>
!com.sun.msv.*,
!org.apache.log4j.jmx.*,
!sun.misc.*,
!org.jboss.logging.*,
!org.apache.zookeeper.*,
*
</Import-Package>
Export-Package
Normally the library should be bundle-private. But sometimes you have to import differently, or the lib does something automatically. So you should always check in the system console, what your bundle is exporting. In case it is not right, you have to manually fiddle with this setting:
Here is an example:
<Export-Package>
!*.internal,
!*.internal.*,
!*.impl,
!*.impl.*,
com.mycompany.myproject.mybundle.*
</Export-Package>
By default all packages * are exported, except they are named impl or internal. Also their child packages are private (the !*.impl.* rule). If the default doesn't work, then export with this instruction only what you need.
Whatever you export goes to the global OSGi space. As also the AEM- and Sling-Bundles are not perfect nor 100%-bugfree, please make sure
the startup/shutdown order of out-of-the-box AEM bundles should not be changed
a deployment, re-deployment or un-deployment of your code should not start/stop any out-of-the-box AEM bundles.
If you don't ensure this, you might experience strange deployment issues - that are very difficult to find/solve.
So the best is, NOT to export anything that is imported by any AEM out-of-the-box bundle. Everything else is for Experts-only. And even they overestimate themselves, and underestimate the long-term costs of patching AEM manually.
PS: the _removeheaders instruction could remove all osgi-instructions that are not needed for runtime. But only do this, if you want to provide a bundle to the public and make it totally shiny. I would leave it in, as it is some kind of documentation.

AEM OSGI 3rd party dependency, bundle not Active

I'm working off an AEM project, I'm new to AEM btw. I'm trying to use a couple of 3rd party non-OSGI libraries, like Dozer and Spring. I'm really struggling with this, it is a pain in the ass, maybe it is because OSGI is kind of old school. My bundles get installed, but not Active in the OSGI console, as they complain about missing packages.
I want to do step by step. First I want to make work my bundles with just Dozer and nothing else. This is my pom dependencies:
<dependency>
<groupId>net.sf.dozer</groupId>
<artifactId>dozer-osgi</artifactId>
<version>5.5.1</version>
</dependency>
Here is my maven-bundle-plugin configuration:
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<version>4.1.0</version>
<configuration>
<instructions>
<Export-Package>somepackages*</Export-Package>
<Private-Package/>
<Import-Package>*</Import-Package>
<Sling-Bundle-Resources>/META-INF</Sling-Bundle-Resources>
</instructions>
</configuration>
</plugin>
As you can see, I'm using dozer-osgi already. I'm not sure why it doesn't pick it up. It complains about this:
org.dozer,version=[5.5,6) -- Cannot be resolved
org.dozer.loader.api,version=[5.5,6) -- Cannot be resolved
First, I don't understand why it says [5.5,6), because I'm telling it to use 5.5.1. Second, I'm using dozer-osgi already, I believe it should pick it up automatically.
I tried also using:
<Embed-Dependency>dozer-osgi</Embed-Dependency>
Whit dozer-osgi added, things start to get better, and more complex at the same time. It seems like now it loads dozer, but starts complaining about transitive dependencies:
android.dalvik -- Cannot be resolved
dalvik.system -- Cannot be resolved
javassist.util.proxy -- Cannot be resolved
org.apache.commons.beanutils,version=[1.9,2) -- Cannot be resolved
org.apache.commons.beanutils.converters,version=[1.9,2) -- Cannot be resolved
The beanutils utils is easy to fix. I just need to add the regarding dependencies like this:
<dependency>
<groupId>commons-beanutils</groupId>
<artifactId>commons-beanutils</artifactId>
<version>1.9.2</version>
</dependency>
and add "commons-beanutils" here:
<Embed-Dependency>dozer-osgi,commons-beanutils</Embed-Dependency>
Now, it complains about:
android.dalvik -- Cannot be resolved
dalvik.system -- Cannot be resolved
javassist.util.proxy -- Cannot be resolved
org.hibernate.proxy -- Cannot be resolved
The hibernate dependency error is new. I thought beanutils depended on hibernate, but not, is Dozer the one depending on it.
I tried adding hibernate, like this:
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>3.6.9.Final</version>
</dependency>
<Embed-Dependency>dozer-osgi,commons-beanutils,hibernate-core</Embed-Dependency>
And so on, it is like a look, it gets worse and worse everytime I try to add a transitive dependency. I also tried this, only keeping dozer-osgi in my dependencies:
<Embed-Dependency>dozer-osgi</Embed-Dependency>
<Embed-Transitive>true</Embed-Transitive>
Result:
android.dalvik -- Cannot be resolved
dalvik.system -- Cannot be resolved
javassist.util.proxy -- Cannot be resolved
org.hibernate.proxy -- Cannot be resolved
org.apache.commons.beanutils -- Cannot be resolved
org.apache.commons.beanutils.converters -- Cannot be resolved
It is like if Embed-Transitive doesn't work at all.
What is the best way to work with 3rd party libraries when using Maven and AEM? In my case, Maven is using to install the bundles in my AEM instance.

In OSGi you have to distinguish between the build and the runtime. At build time your initial approach was totally fine.
You use dozer and the maven-bundle-plugin creates suitable Import-Package statements. You can check these by looking into the Manifest of the jar.
The rules for package imports result in a range of version that should work with your code. So [5.5,6) is exactly the expected import range.
Now to runtime. Here you have to supply all dependencies of your code (including the transitive ones) as bundles in AEM (or more generally in the OSGi runtime).
So you also need to install dozer OSGi as bundle. If this complains again then you also need to install its dependencies.
This is the default approach and normally totally fine.
Now if you want to make your bundle standalone in respect to having no additional runtime depenencies then you can try to embed all dependencies. Then you only need to install your bundle.
Be aware though that this is not easy. If your code uses some of the embedded classes in its own API then you get into lots of problems with embedding. So if your are not very experienced then better go the way of installing all dependencies as bundles.
To simplify the install process you can create a content package that contains all needed bundles.
You also have to check that all dependencies actually are bundles. In some cases the normal maven dependencies are not suitable. In this case have a look at apache servicemix bundles. It is a project that creates OSGi bundles for popular dependencies and deploys these to maven central.

Which lifecycle for a Java annotation that generates documentation?

I would like to be able to document the dependencies of my services directly in my code, using Java annotations. Those annotations could bear the information about the target system, whether the connection is incoming/outgoing/2-ways, and the type of connection (REST, RabbitMQ...).
It could look like this:
#Dependency(target = "Twitter API", type = "Outgoing", medium = "REST")
The idea would be to generate a DOT file from all the annotations inside the project.
I have a fair idea on how to create my own annotation, with the required attributes. However, i am not sure at which part of the lifecycle of the compilation/processing i should handle those annotations.
I understand that the annotation processors generate source files, but in my case the generated files are not at all required by the compiler nor the application itself.
Ideally i would like to be able to trigger the annotation processing, and DOT file generation, by a dedicated Maven or Gradle task.
Is it something that is easily doable when writing my own annotations?

If you want to create documentation via maven than you need add the bellow two dependencies as plugins and then execute site maven goal.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
<version>3.7.1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-project-info-reports-plugin</artifactId>
<version>3.0.0</version>
</plugin>
If you just want to document REST endpoints than you can use swagger.
If your project uses spring than the integration of swagger is pretty easy. You can use this tutorial.
If you want to save in a file the dependency graph of your project you can simply execute the following maven command.
mvn dependency:tree -Doutput=/path/to/file

Annotations are processed by the Java compiler directly.
For Maven, one can use the maven-compiler-plugin. It is also possible to perform the compilation in 2 steps, as explained here.
For Gradle, one can add the processors in the dependencies block under the annotationProcessor configuration, as explained here.
Annotation retention can be specified as SOURCE, so they won't be kept after compilation.

JasperException - File Not Found even when tag is present

Alright StackOverflow, I'm coming to you in a time of need.
I've inherited a project with a custom tag library. The project is in AEM, but the problem is more of an issue with the straight Java side of things and is nothing specific to AEM that I am aware of. AEM is built on OSGI, so that could be related, but again it's sort of unlikely to be part of the answer.
Essentially, I'm getting an exception on only one of two servers. It says:
org.apache.sling.api.scripting.ScriptEvaluationException: org.apache.sling.scripting.jsp.jasper.JasperException: File "/META-INF/tags/helloWorld.tagx" not found
Now - this helloWorld.tagx is sort of garbage left in from a template project, but for some reason it's essential. There's a few issues here.
The tag is in the right location (META-INF/tags/helloWorld.tagx) and I'm using the jsptld-maven-plugin to generate the tld file which looks correct to me.
The configuration for the plugin (as well as the maven-bundle-plugin):
<plugin>
<groupId>com.squeakysand.jsp</groupId>
<artifactId>jsptld-maven-plugin</artifactId>
<configuration>
<shortName>myproject</shortName>
<processTagFiles>true</processTagFiles>
</configuration>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<instructions>
<Bundle-Activator>com.rebny.taglib.osgi.Activator</Bundle-Activator>
<Include-Resource>
META-INF/${project.artifactId}-${project.version}.tld=${project.build.outputDirectory}/META-INF/${project.artifactId}-${project.version}.tld,
{maven-resources}
</Include-Resource>
<Sling-Bundle-Resources>
/META-INF/tags
</Sling-Bundle-Resources>
</instructions>
</configuration>
</plugin>
And the segment from the resulting tld file:
<tag-file>
<name>helloWorld</name>
<path>/META-INF/tags/helloWorld.tagx</path>
</tag-file>
So it looks fine to me, and I've done a lot of searching and found people with syntax errors, etc. but I don't believe that to be the case here, especially since it works on one server.
The weird part is the tag isn't actually used anywhere, but removing it doesn't solve the issue. The jsp throwing this error uses other tags, but not this one. There is literally no references to this tag in the entire project - I'm thinking it might be hiding another issue. The error says that it's on line 6 of the JSP, but neither the source JSP or the compiled JSP have anything interesting on line 6, or in the file at all. Again, even compiled JSPs have no reference to this tag!
I'm happy to provide other information. My knowledge of this is pretty poor, so I'm not exactly sure what information is relevant. Any help or troubleshooting tips is greatly appreciated!
Full stack trace here

Your best bet is to go through Adobe Daycare. They are pretty good about getting you patches. Especially if your client has an existing relationship with them which they should if they are using AEM.

The issue relates to Scripted tag files are not found. Reference.
It's a bug that states, If a taglib in a bundle, references a tag implemented as a script (located in /META-INF/tags) this script is not found as it is not searched within the bundle containing the tld. It's rather used as a resource on the classpath.
The bug has been Fixed in JSP 2.3.2 version. So make use of this JSP version in your project. Moreover, this bug would have been fixed in AEM 6.3.1.1 Or higher,
Here, Is a helpful thread related to this issue.

Maven-dependency-plugin and annotations with SOURCE RetentionPolicy

In a mvn project where I am utilizing maven-dependency-plugin to detect unused dependencies, there is seemingly no dependency scope I can specify for Google's AutoValue (com.google.auto.value:auto-value) to that will convince the plugin that the dependency is being used in spite of the fact that annotations from the package are being used (e.g. #AutoValue) and the project won't build if auto-value is excluded.
Now one solution is simply adding a configuration entry to my plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<configuration>
<usedDependencies>
<usedDependency>com.google.auto.value:auto-value</usedDependency>
</usedDependencies>
</configuration>
</plugin>
But I would be curious to know whether it's possible to configure either the maven-dependency-plugin or the dependency entry for auto-value in a way that would detect usage of the dependency per its annotations?
My suspicion is that this isn't possible because the RetentionPolicy of the annotations I'm using from auto-value are of RetentionPolicy.SOURCE and are discarded by the compiler. Is this correct?

Unfortunately, your suspicion is correct. The maven-dependency-plugin documentation specifically lists this as a concern here for source level annotations: http://maven.apache.org/shared/maven-dependency-analyzer/
Warning: Analysis is not done at source but bytecode level, then some cases are not detected (constants, annotations with source-only retention, links in javadoc) which can lead to wrong result if they are the only use of a dependency.
You can force AutoValue as used with usedDependencies as you have in your example or use the ignoredUnusedDeclaredDependencies configuration instead (which is what I did recently).
I don't believe it is possible to configure the dependency section to avoid this because maven doesn't provide a scope level that is compile only. I mark AutoValue with the provided scope to keep it out of any shaded jars I might make.
Lastly, you could write (or find if it exists) a custom dependency analyzer that takes source level annotations into account. See the documentation here http://maven.apache.org/plugins/maven-dependency-plugin/analyze-mojo.html#analyzer. Probably not worth the effort.