Develop Reference

Kotlin Library Exported As Jar Usable In Java Code, But Not In Kotlin Code

I've written a utility library for a project I am working on in Kotlin. The project has a number of dependencies (such as AWS libraries). I want to package the library as a 'fat' .jar so that I can use the library in other projects without issue.
Currently, I am using the gradle shadow plugin. I am able to successfully use the .jar classes in Java code/projects without issue. However, when using the .jar classes in Kotlin projects/code (or attempting to) I am facing visibility issues (code does not compile b/c of unresolved references). I have no idea what I am doing wrong and am not sure if this is a common issue.
I've already tried:
Gradle Shadow plugin here
Modifying the gradle jar task to include all dependencies myself
The fat jar is successfully created in both cases, and usable in Java code, but Kotlin is not able to resolve the references. However, when I create a non fat jar, the classes are visible, but obviously broken (bc of missing dependencies). So, essentially, only when I create a fat .jar, only my Kotlin code cannot use the library.
Does anyone have any insight?

Most likely, your re-packaged JAR is missing the META-INF/*.kotlin_module files from the original JARs. These are the Kotlin package metadata files which the Kotlin compiler needs to be able to read top-level declarations from the classes.
If these files are lost, you will face visibility issues with top-level declarations and extensions.
You need to configure your fat JAR tools to also copy these files into the resulting JAR.

You would also need to add both the compile time and run time dependencies when you compile your library jar file.
One way to add them would be for example:
tasks {
jar{
from({
configurations.runtimeClasspath.get().filter { it.name.endsWith("jar") }.map { zipTree(it) }
configurations.compileClasspath.get().filter { it.name.endsWith("jar") }.map { zipTree(it) }
})
exclude ("META-INF/*.RSA", "META-INF/*.SF", "META-INF/*.DSA")
}
}

Gradle: What is the difference between classpath and compile dependencies?

When adding dependencies to my project I am never sure what prefix I should give them, e.g. "classpath" or "compile".
For example, should my dependencies below be compile time or classpath?
Also, should this be in my applications build.gradle or in the module specific build.gradle?
Current build.gradle (at application level):
apply plugin: 'java'
repositories {
mavenCentral()
}
dependencies {
compile 'org.hibernate:hibernate-core:5.0.5.Final'
compile 'mysql:mysql-connector-java:5.1.38'
}

If buildscript itself needs something to run, use classpath.
If your project needs something to run, use compile.
The buildscript{} block is for the build.gradle itself.
For multi-project building, the top-level build file is for the root project, the specific build file is for sub-project (module).
Top-level build file where you can add configuration options common to all sub-projects/modules.
Do not place your application dependencies in top-level build file, they belong in the individual module build.gradle files

I'm going to guess that you're referencing compile and classpath within the dependencies {} block. If that is so, those are dependency Configurations.
A configuration is simply a named set of dependencies.
The compile configuration is created by the Java plugin. The classpath configuration is commonly seen in the buildScript {} block where one needs to declare dependencies for the build.gradle, itself (for plugins, perhaps).

If I understand correctly, you're confusing Project.dependencies script block with the Project.buildscript.dependencies script block (just like I did when I reached this question).
I'll try to answer this with what I found.
I think you should be already familiar with the Project.dependencies script block. In this block, we declare dependencies that are required by our source code. There are several ways to declare a dependency that we need for the project. See Gradle Tutorial: Dependency Types. I'll only mention the part that is the most relevant to this problem:
compile 'org.hibernate:hibernate-core:5.0.5.Final' is a module dependency declaration. The compile configuration (which is now deprecated by the implementation configuration.) is merely a keyword for Implementation only dependencies. It is not a keyword describing which type of dependency it is (by type here I'm following the three types defined in the tutorial, i.e. module, file, and project.)
In Gradle Tutorial: Organizing Build Logic it says:
If your build script needs to use external libraries, you can add them
to the script’s classpath in the build script itself. You do this
using the buildscript() method, passing in a closure which declares
the build script classpath.
This is the same way you declare, for example, the Java compilation
classpath. You can use any of the dependency types described in
Dependency Types, except project dependencies.
Having declared the build script classpath, you can use the classes in
your build script as you would any other classes on the classpath.
I hope things are getting clear to you now.
With classpath "com.android.tools.build:gradle:${Versions.android_gradle_plugin}" we're setting classpath method with com.android.tools.build:gradle:${Versions.android_gradle_plugin} which is a module dependency that is used by the build script itself rather than the source in your project.
On the other hand, with compile 'org.hibernate:hibernate-core:5.0.5.Final' we're declaring a module dependency required for your project with the compile configuration.
tl;dr: The classpath, compile, and implementation are all keywords that can be used against dependencies under different circumstances. The former is used when you want to pass in a dependency to the build script, and the latter is one of the configuration you may want to declare.

Android:
classpath in project build.gradle —— the implementation after classpath is only used by gradle it self, used in build script. So if i add the implementation (such as retrofit) in the project build.gradle classpath 'retrofit...', i can't get retrofit in my code!! Because —— my code can't see it, only the buildscript can see it.
implementation in app build.gradle —— add the implementation your code can use!!

Can I force the order of dependencies in my classpath with Gradle?

A project runs on Google App Engine. The project has dependency that uses a class that can't be invoked on App Engine due to security constraints (it's not on the whitelist). My (very hacky) solution was to just copy a modified version of that class into my project (matching the original Class's name and package) that doesn't need the restricted class. This works on both dev and live, I assume because my source appears in the classpath before my external dependencies.
To make it a bit cleaner, I decided to put my modified version of that class into it's own project that can be packaged up in a jar and published for anyone else to use should they face this problem.
Here's my build.gradle:
// my jar that has 'fixed' version of Class.
compile files('path/to/my-hack-0.0.1.jar')
// dependency that includes class that won't run on appengine
compile 'org.elasticsearch:elasticsearch:1.4.4'
On my local dev server, this works fine, the code finds my hacked version of the class first at runtime. On live, for some unknown reason, the version in the elasticsearch dependency is loaded first.
I know having two versions of the same class in the classpath isn't ideal but I was hoping I could reliably force my version to be at the start of the classpath. Any ideas? Alternatively, is there a better way to solve this problem?

Not really sure if this is what people visiting this question were looking for, but this was what my problem and a solution that I reached at.
Jar A: contains class XYZ
Jar B: also contains class XYZ
My Project needs Jar B on the classpath before Jar A to be able to get compiled.
Problem is Gradle sorts the dependencies based on alphabetical order post resolving them which meant Jar B will be coming after Jar A in the generated classpath leading to error while compiling.
Solution:
Declare a custom configuration and patch the compileClasspath. This is how the relevant portion of build.gradle might look like.
configurations {
priority
sourceSets.main.compileClasspath = configurations.priority + sourceSets.main.compileClasspath
}
dependencies {
priority 'org.blah:JarB:2.3'
compile 'org.blah:JarA:2.4'
...
}

It's the app engine classloader I should have been investigating, not gradle...
App Engine allows you to customise the class loader JAR ordering with a little bit of xml in your appengine-web.xml. In my case:
<class-loader-config>
<priority-specifier filename="my-hack-0.0.1.jar"/>
</class-loader-config>
This places my-hack-0.0.1.jar as the first JAR file to be searched for classes, barring those in the directory war/WEB-INF/classes/.
...Thanks to a nudge in the right direction from #Danilo Tommasina :)
UPDATE 2020:
I just hit the same problem again and came across my own question... This time, live appengine was loading a different version of org.json than was being loaded in dev. Very frustrating and no amount of fiddling the build script would fix it. For future searchers, if you're getting this:
java.lang.NoSuchMethodError: org.json.JSONObject.keySet()Ljava/util/Set;
It's because it's loading an old org.json dependency from god-knows-where. I fixed it by adding this to my appengine-web.xml:
<class-loader-config>
<priority-specifier filename="json-20180130.jar"/>
</class-loader-config>
You'll also need a matching dependency in build.gradle if you don't already have one:
compile 'org.json:json:20180130'

According to gradle dependencies documentation, the order of dependencies defines the order in the classpath. So, we can simply put the libraries in the correct order in "dependencies".
But beware! here are two rules with higher priorities:
For a dynamic version, a 'higher' static version is preferred over a 'lower' version.
Modules declared by a module descriptor file (Ivy or POM file) are preferred over modules that have an artifact file only.

Using Gradle with native dependencies

I am trying to use Sigar in a Gradle project. Sigar distribution is by default provided with 2 types of files:
a JAR that contains classes
some native files (.so, dylib, .dll)
My purpose is to repackage these files so that I can use them as dependencies deployed and downloaded on-demand from a personal Maven repository.
My first try was to define dependencies as files in order to check that my application is working as expected before to repackage. Below is the Gradle code I used for my first test that works:
dependencies {
compile files("${rootDir}/lib/sigar/sigar.jar")
runtime fileTree(dir: "${rootDir}/lib/sigar/", exclude: "*.jar")
}
Then, I have repackaged Sigar native files into a JAR and renamed the other one to match rules for maven artifacts since I want to deploy them in a Maven repository. Below is what I get:
sigar-1.6.4.jar (contains .class files)
sigar-1.6.4-native.jar (contains .dylib, .so, and .dll files at the root)
The next step was to deploy these files in my custom repository. Then, I have updated my build.gradle as follows:
dependencies {
compile 'sigar:sigar:1.6.4'
runtime 'sigar:sigar:1.6.4:native'
}
Unfortunately, when I do a gradle clean build, new dependencies are fetched but native libraries can no longer be found at runtime since now I get the following exception:
Error thrown in postRegister method: rethrowing <java.lang.UnsatisfiedLinkError: org.hyperic.sigar.Sigar.getCpuInfoList()[Lorg/hyperic/sigar/CpuInfo;>
Consequently, I am looking for a solution to fetch and to link native files to my Java app like for other dependencies. Any advice, comment, suggestion, help, solution, etc. are welcome ;)

A solution is to define a new gradle configuration that unzips JAR files at the desired location:
project.ext.set('nativeLibsDir', "$buildDir/libs/natives")
configurations {
nativeBundle
}
dependencies {
nativeBundle 'sigar:sigar:1.6.4:native'
}
task extractNativeBundle(type: Sync) {
from {
configurations.nativeBundle.collect { zipTree(it) }
}
into file(project.nativeLibsDir)
}
dist.dependsOn extractNativeBundle
Then, this location must be put in java.library.path for tasks that depend on native libraries:
systemProperty "java.library.path", project.nativeLibsDir

android gradle -- compile two pure java projects to dex

I have two pure java projects (:java:libs:proj1, :java:libs:proj2) that produce two classes.jars when they are built. The two jars "combined" include the exact classes that I would like to convert into dex jar i.e. a JAR file with classes.dex in it.
Can someone help me with converting the two pure java projects into a single JAR with classes.dex in it?
Here is what I tried doing.
1) Create a separate Android Library project (apply plugin: 'android-library') and include the two java projects as dependencies.
apply plugin: 'android-library'
dependencies {
compile project(':java:libs:proj1')
compile project(':java:libs:proj2')
}
However, with this approach, it doesn't generate a dex file. Instead, it produces an ".aar" file which contains only the .class files of the two projects. If I can convert this .aar file to its dex, then this would be perfect but apparently the "android-library" plugin does not have a "dex" task.
2) Create an Android project (apply plugin: 'android') and include the two projects as depenendencies. (Similar to above but using 'android' instead of 'android-library').
apply plugin: 'android'
dependencies {
compile project(':java:libs:proj1')
compile project(':java:libs:proj2')
}
This approach does create the dex files that I need, however it also includes ALL the dependencies for my two pure java projects. So for example, my java projects depend on JUnit, BouncyCastle and some other external JAva projects. Inside of the classes.dex, I see that it has included everything. Is there a way to tell the "dex" task to not include dependencies "transitively"?
I only want the classes directly associated with my two java projects into dex. I wish there was an easy way :(
Thanks,
J

will modify compile in dependencies to provided help? please also check this
25.4.7. Excluding transitive dependencies
You can exclude a transitive dependency either by configuration or by dependency:
Example 25.14. Excluding transitive dependencies