Develop Reference

Duplicate handling strategy error with gradle while using protobuf for java

I am using the below configuration build.gradle
plugins {
id "com.google.protobuf" version "0.8.17"
id "java"
}
group "de.prerna.aws.tests"
version "1.0-SNAPSHOT"
repositories {
mavenCentral()
}
ext {
protobufVersion = "3.18.1"
}
dependencies {
implementation "com.google.protobuf:protobuf-java:$protobufVersion"
sourceSets {
main {
proto {
srcDir 'src/main/proto'
}
java {
// include self written and generated code
srcDirs 'src/main/java'
}
}
}
protobuf {
protoc {
artifact = 'com.google.protobuf:protoc:4.0.0-rc-2'
}
plugins {
grpc {
artifact = "io.grpc:protoc-gen-grpc-java:1.39.0"
}
}
generateProtoTasks.generatedFilesBaseDir = 'generated-sources'
generateProtoTasks {
all().each { task ->
task.plugins { grpc{} }
}
ofSourceSet('main')
}
}
Error
* What went wrong:
Execution failed for task ':processResources'.
> Entry Person.proto is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/7.2/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

A variant of BParolini for build.gradle (Groovy DSL)
tasks.withType(Copy) {
filesMatching("**/*.proto") {
duplicatesStrategy = DuplicatesStrategy.INCLUDE
}
}

I could fix this problem by adding the following code to my build.gradle.kts:
tasks {
withType<Copy> {
filesMatching("**/*.proto") {
duplicatesStrategy = DuplicatesStrategy.INCLUDE
}
}
}
Extra info: I'm using Gradle 7.3-rc-3 and Java 17.

Unfortunately nobody explains reasons for this problem, so here is some of my explorations and guesses. Please correct me if you know more.
If found that following build script code causes this error:
proto { srcDir 'src/main/proto' }
If look inside "build/extracted-include-protos" directory, there are original .proto files copied into "build/extracted-include-protos/test" (but not into main).
My guess is that those auto-copied .proto files are originally uses as the only sources, but when adding "src/main/proto" source set we give some compiler tool second set of same files.
Removing this srcDir is not a good idea, because it required for IDEA to correctly open included .proto on Ctrl+click (otherwise it is opened extracted copies which is useless).

How to combine multiple Javadoc into one using Gradle

This question was answered before but the chosen answer doesn't explain a lot for me on how this is doable on Gradle.
That and the fact that I can't comment on the solution to ask for more info forced me to make this question.
I have a Gradle project that has several modules available and I now want to set up the Javadoc task to combine the Javadoc comments of all the modules into a single location where I could browse it.
How would I now be able to do this using Gradle? I run Gradle 5.5 if the version is of any importance and I have the following things set in the build.gradle file:
allprojects {
ext {
// Convenience method to configure Javadoc
configureJavadoc = { Object jDocConfig ->
jDocConfig.options {
it.author()
it.encoding = 'UTF-8'
it.memberLevel = JavadocMemberLevel.PROTECTED
if (it instanceof StandardJavadocDocletOptions) {
def opt = it as StandardJavadocDocletOptions
opt.links(
"https://docs.example.com/java/"
)
if (JavaVersion.current().isJava9Compatible()) {
opt.addBooleanOption("html5", true)
opt.addStringOption("-release", "8")
}
if (JavaVersion.current().isJava11Compatible()) {
opt.addBooleanOption("-no-module-directories", true)
}
}
}
}
}
}
subprojects {
javadoc {
destinationDir = file("$rootDir/docs/")
configureJavadoc(it)
}
}

I was able to do it with:
def exportedProjects = [
":",
":module-a",
":module-b",
":module-c"
]
task allJavadoc(type: Javadoc) {
source exportedProjects.collect { project(it).sourceSets.main.allJava }
classpath = files(exportedProjects.collect { project(it).sourceSets.main.compileClasspath })
destinationDir = file("${buildDir}/docs/javadoc-all")
}

Adding support for Kotlin sources to existing Java multi-project Gradle build

I have a Spring Boot Java project that builds using Gradle (v6.2.2).
build.gradle.kts
plugins {
base
java
id("org.springframework.boot") apply false
}
val gradleVersionProperty: String by project
val javaVersion: String by project
val springBootVersion: String by project
val springCloudStarterParentBomVersion: String by project
if (JavaVersion.current() != JavaVersion.VERSION_11) {
throw GradleException("This build must be run with JDK 11")
} else {
println("Building with JDK " + JavaVersion.current())
}
tasks.withType<Wrapper> {
gradleVersion = gradleVersionProperty
distributionType = Wrapper.DistributionType.ALL
}
allprojects {
group = "com.meanwhile.in.hell"
version = "$version"
// Repos used in dependencyManagement section of settings.gradle
repositories {
mavenLocal()
mavenCentral()
maven("https://repo.spring.io/snapshot")
maven("https://repo.spring.io/milestone")
}
}
subprojects {
if (!project.name.startsWith("platform")) {
apply {
plugin("java-library")
}
java.sourceCompatibility = JavaVersion.VERSION_11
java.targetCompatibility = JavaVersion.VERSION_11
// Change the default test logging settings
tasks.withType<Test>() {
useJUnitPlatform()
testLogging {
events = setOf(
org.gradle.api.tasks.testing.logging.TestLogEvent.FAILED,
org.gradle.api.tasks.testing.logging.TestLogEvent.PASSED,
org.gradle.api.tasks.testing.logging.TestLogEvent.SKIPPED
)
exceptionFormat = org.gradle.api.tasks.testing.logging.TestExceptionFormat.FULL
}
enableAssertions = false
ignoreFailures = gradle.startParameter.isContinueOnFailure
maxParallelForks =
(Runtime.getRuntime().availableProcessors() / 2).takeIf { it > 0 } ?: 1
}
dependencies {
"api"(enforcedPlatform("org.springframework.boot:spring-boot-dependencies:$springBootVersion"))
"api"(enforcedPlatform("org.springframework.cloud:spring-cloud-dependencies:$springCloudStarterParentBomVersion"))
"implementation"(enforcedPlatform(project(":platform-dependencies")))
"testCompile"("org.springframework.boot:spring-boot-starter-test")
}
}
}
However, I would like to add support for Spring Boot Kotlin sub-projects within it. I have used a very simple sample project from a Kotlin-only project I have that builds fine within it. Without any changes to my root build.gradle.kts file, my current build error is:
* What went wrong:
Execution failed for task ':kotlin-sample-project:bootJar'.
> Main class name has not been configured and it could not be resolved
I have not configured the main class for any of the Java sub-projects and neither have I in my Kotlin-only other project.
My build.gradle.kts in kotlin-sample-project is very simple:
plugins {
id("org.springframework.boot")
}
dependencies {
implementation("org.springframework.cloud:spring-cloud-starter-gateway")
}
And my main class looks like:
src/main/kotlin/sample/KotlinSampleApplication.kts
package com.meanwhile.in.hell.kotlinsample
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.boot.runApplication
#SpringBootApplication
open class KotlinSampleApplication
fun main(args: Array<String>) {
runApplication<KotlinSampleApplication>(*args)
}
I have tried to add the kotlin plugin, but the build fails instantly not knowing what it is.
plugins {
base
java
kotlin
}
Error:
Line 9: kotlin
^ Unresolved reference. None of the following candidates is applicable because of receiver type mismatch:
public val <T : Any> Class<TypeVariable(T)>.kotlin: KClass<TypeVariable(T)> defined in kotlin.jvm

I have tried to add the kotlin plugin, but the build fails instantly not knowing what it is.
It should be:
build.gradle.kts:
plugins {
kotlin("jvm").version("1.3.72")
}
dependencies {
implementation(kotlin("stdlib-jdk8"))
}
Bot Kotlin JVM plugin should be applied and Kotlin stdlib be present on compile classpath.

Obfuscating Java in Gradle using yGuard ~ How to exclude methods?

I want to obfuscate Java code in Gradle using yGuard. So far, I have managed to get the obfuscation working using:
dependencies {
compile 'com.yworks:yguard:2.9.2'
}
task yGuardObfuscate {
group 'yGuard'
description 'Obfuscates existing archives.'
dependsOn "installDist"
doLast {
ant.taskdef(
name: 'yguard',
classname: 'com.yworks.yguard.YGuardTask',
classpath: sourceSets.main.runtimeClasspath.asPath
)
ant.yguard {
inoutpairs {
fileset(dir: "./build/install/module") {
include(name: "module*.jar")
exclude(name: "*_obf.jar")
}
}
rename(logFile: "./build/install/module/rename.log") {
property(name: "naming-scheme", value: "mix")
}
}
}
}
However, that has the problem that the main function gets obscured, and I can't run it anymore using java -jar module_obf.jar (which works with the unobfuscated jar).
I have tried several ways of excluding the main function from obfuscation.
First, I tried using the #Obfuscate annotation as described in the documentation (https://yworks.github.io/yGuard/task_documentation/#controlling-obfuscation-exclusion-with-annotations), like this:
#com.yworks.util.annotation.Obfuscation( exclude = true, applyToMembers = false)
public static void main(String[] args) {
try {
new Start();
} catch (SQLException e) {
LOGGER.error(e, e);
}
}
In my IDE (Eclipse 2019-14), this line was shown as valid. However, when trying to compile it with the gradle task I created, I got the following error:
...\Start.java:22: error: package com.yworks.util.annotation does not exist
#com.yworks.util.annotation.Obfuscation( exclude = true, applyToMembers = false)
^
I tried a number of things to get this to work, but could not arrive at a working solution.
So next, I tried to use a keep > method element in the yGuardObfuscate task to explicitly keep the main function of the Start class, like this:
task yGuardObfuscate {
group 'yGuard'
description 'Obfuscates existing archives.'
dependsOn "installDist"
doLast {
ant.taskdef(
name: 'yguard',
classname: 'com.yworks.yguard.YGuardTask',
classpath: sourceSets.main.runtimeClasspath.asPath
)
ant.yguard {
inoutpairs {
fileset(dir: "./build/install/module") {
include(name: "module*.jar")
exclude(name: "*_obf.jar")
}
}
rename(logFile: "./build/install/module/rename.log") {
property(name: "naming-scheme", value: "mix")
keep {
method("class": "com.company.project.module.Start", name: "public static void main(String[] args)")
}
}
}
}
}
This caused the Gradle build to fail with the following exception:
: java.lang.IllegalArgumentException: '(' expected but found void
[...]
Caused by: java.lang.IllegalArgumentException: '(' expected but found void
at com.yworks.yguard.ObfuscatorTask.toNativeMethod(ObfuscatorTask.java:188)
at com.yworks.yguard.ant.MethodSection.addEntries(MethodSection.java:35)
at com.yworks.yguard.ant.ExposeSection.createEntries(ExposeSection.java:170)
at com.yworks.yguard.ObfuscatorTask.execute(ObfuscatorTask.java:745)
at com.yworks.yguard.YGuardTask.execute(YGuardTask.java:116)
[...]
Root cause: java.lang.IllegalArgumentException: '(' expected but found void
at com.yworks.yguard.ObfuscatorTask.toNativeMethod(ObfuscatorTask.java:188)
at com.yworks.yguard.ant.MethodSection.addEntries(MethodSection.java:35)
at com.yworks.yguard.ant.ExposeSection.createEntries(ExposeSection.java:170)
at com.yworks.yguard.ObfuscatorTask.execute(ObfuscatorTask.java:745)
at com.yworks.yguard.YGuardTask.execute(YGuardTask.java:116)
Again, I tried several things here, such as writing "class" without the "", adding extra {}, but nothing helped.
So the question here is: What am I doing wrong? And how can I prevent yGuard from obfuscating the main function?

I now figured out a way to make it work, even though it sorta feels more like a workaround than a proper solution, so if anyone knows what I did wrong with the "official" solutions, please do tell me.
For this, I used the Annotation approach, and instead of using the default annotation, I created a custom annotation that is pretty much an exact copy in my project.
The annotation class looks like this:
package com.company.project.module.annotations;
public #interface Obfuscation {
boolean exclude() default true;
boolean applyToMembers() default true;
}
I use it in my Start class like this:
import com.company.project.module.annotations.*;
[...]
#Obfuscation( exclude = true, applyToMembers = false)
public static void main(String[] args) {
[...]
}
And finally, I added this custom annotation class to the rename element of my task like this:
rename(logFile: "./build/install/module/rename.log", annotationClass: "com.company.project.module.annotations.Obfuscation") {
property(name: "naming-scheme", value: "mix")
}
So the entire gradle task now looks like this:
task yGuardObfuscate {
group 'yGuard'
description 'Obfuscates existing archives.'
dependsOn "installDist"
doLast {
ant.taskdef(
name: 'yguard',
classname: 'com.yworks.yguard.YGuardTask',
classpath: sourceSets.main.runtimeClasspath.asPath
)
ant.yguard {
inoutpairs {
fileset(dir: "./build/install/module") {
include(name: "module*.jar")
exclude(name: "*_obf.jar")
}
}
rename(logFile: "./build/install/module/rename.log", annotationClass: "com.company.project.module.annotations.Obfuscation") {
property(name: "naming-scheme", value: "mix")
}
}
}
}
With that, the build now works, and I can successfully run my program using java -jar module_obf.jar.

Shading dependencies of scala (jar) library

I would like to distribute a jar of a library I created with all my dependencies bundled inside. However I would like to avoid version conflicts of dependencies with the adopting project.
I think maven shade can do this but I could not find a way to do this with Scala / SBT. I found OneJar however from my experiments with it seems to work only for executables.
How could I achieve this?
Thanks!

You can do this with your own classloader.
The classLoader:
Write a class loader which loads class files from diferent classloader using a rewrite.
For example you could add library as a prefix to the classpath when fetching the resource.
I have created a classloader using this teqnuiqe.
https://github.com/espenbrekke/dependent/blob/master/src/main/java/no/dependent/hacks/PathRewritingClassLoader.java
It replaces the method findClass in URLClassLoader with one adding a prefix.
protected Class<?> findClass(final String name) throws ClassNotFoundException {
Class result;
try {
result = (Class)AccessController.doPrivileged(new PrivilegedExceptionAction() {
public Class<?> run() throws ClassNotFoundException {
// This is where the prefix is added:
String path = PathRewritingClassLoader.this.prefix + name.replace('.', '/').concat(".class");
Resource res = PathRewritingClassLoader.this._ucp.getResource(path, false);
if(res != null) {
try {
return PathRewritingClassLoader.this._defineClass(name, res);
} catch (IOException var4) {
throw new ClassNotFoundException(name, var4);
}
} else {
return null;
}
}
}, this._acc);
} catch (PrivilegedActionException var4) {
throw (ClassNotFoundException)var4.getException();
}
if(result == null) {
throw new ClassNotFoundException(name);
} else {
return result;
}
}
We also have to rewrite resource loading
#Override
public URL getResource(String name){
return super.getResource(prefix+name);
}
Here is how it is used:
_dependentClassLoader = new PathRewritingClassLoader("private", (URLClassLoader)DependentFactory.class.getClassLoader());
Class myImplementationClass=_dependentClassLoader.loadClass("my.hidden.Implementation");
Building your jar:
In your build you place all the library and private classes under your selected prefix. In my gradle build I have a simple loop collecting all the dependencies.
task packageImplementation {
dependsOn cleanImplementationClasses
doLast {
def paths = project.configurations.runtime.asPath
paths.split(':').each { dependencyJar ->
println "unpacking" + dependencyJar
ant.unzip(src: dependencyJar,
dest: "build/classes/main/private/",
overwrite: "true")
}
}
}

Proguard can rename packages inside jar and obfuscate code. It is a bit complicated but you can achieve you goal with it. sbt-proguard plugin is actively maintained
Also you can check answers from similar thread:
maven-shade like plugin for SBT
UPDATE:
from version 0.14.0 sbt-assembly plugin seemed to have shading ability

Have you tried sbt-assembly plugin? It has set of merging strategies in case of conflicts and has pretty much nice start guide.