Develop Reference

Using external classes with dataflow

I am trying to make a custom transform class that takes in a side input and I am using PipelineTester and PAssert to try and test it, but I keep getting a no such method exception on methods I am trying to bring into the transform from other classes.
Caused by: java.lang.NoSuchMethodError:
com.org.utils.MyUtils.createMap(Ljava/lang/Iterable;)Ljava/util/Map;
at com.org.beam.MyTransform.ProcessElement(MyTransform.java:51)
I have tried using the #Autowired annotation to bring in the class like
#Autowired
private MyUtils myutils;
as well as just creating a static instance in the code like
private static MyUtils myUtils = new MyUtils();
and then calling
this.myUtils.createMap(mapThisToThat(inputCollection, this.myMap));
I have also tried making the methods static and calling them like
MyUtils.createMap(mapThisToThat(inputCollection, this.myMap));
the signature to mapThisToThat is
private Iterable<MyObject> mapThisToThat(Iterable<MyObject> objectIterator, Map<String, Integer> myMap) {
which is being passed into the createMap method which has this signature -
public Map<String, MyObject> createMap(Iterable<MyObject> inputCollection){
so it is passing in an Iterable of MyObjects correctly, but it says the method doesn't exist for some reason. does this mean beam transforms can't have external methods or am I doing something wrong?

For me in python, there are a variety of things I need to do for that to work:
https://cloud.google.com/dataflow/faq#how-do-i-handle-nameerrors
https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/
For you in java, they don't have reciprocal documentation for some reason, but over here https://beam.apache.org/documentation/runners/dataflow/ they say things like this:
In some cases, such as starting a pipeline using a scheduler such as Apache AirFlow, you must have a self-contained application. You can pack a self-executing JAR by explicitly adding the following dependency on the Project section of your pom.xml, in addition to the adding existing dependency shown in the previous section.
In their examples readme https://github.com/mbrukman/apache-beam/tree/master/examples/java they say this
Alternatively, you may choose to bundle all dependencies into a single JAR and execute it outside of the Maven environment. For example, you can execute the following commands to create the bundled JAR of the examples and execute it both locally and in Cloud Platform
If you continue to browse that examples repo, there is a common folder with utils. Hopefully you can copy how they did it.

Using jar from scala project from python via Py4j

I have built a jar from my Scala project.
I have the following structure for what I want to use from this jar
package aaa.bbb.ccc
case class FooResult(...)
trait Foo(...) {
def bar(): FooResult
}
object Foo {
private class FooImpl(...) extends Foo {
...
}
def apply(...): Foo
}
First question: Maybe I have misunderstood something in what Py4J offers,
but do I have to write a java/scala class to start the Py4J gateway if I want to use my own classes? Or is it enough to add it to the gateway's jvm's classpath?
Second question (which I guess doesn't apply depending on the answer to above): How do I add my jar when starting the java gateway in order to make it available? To solve this temporarily, I just started the jvm manually with my jar along with the Py4J jar with this command
java -classpath "path/to/py4j.jar:path/to/my.jar" py4j.GatewayServer 0
and then connected to it manually from the Python code. Then I tried to import my classes via
java_import(gateway.jvm, "aaa.bbb.ccc.*")
which didn't throw any error but I'm not sure it worked because it doesn't throw any error if I input some fake classpath.
Third question (which applies if the answer to the first is that I have to write the entry point to access my classes): How does this work when using scala?
object Main extends App {
val gw = new GatewayServer(// TODO: how to expose my classes here?
}

how to access internal properties from java test code

I've got some class with property marked as internal.
Then I try to set that property from test code which is in java.
How can I access those properties? test code and class code are in the same package.
example:
class MainActivity : AppCompatActivity() {
interal var someProperty = "test"
}
test code:
#Test
public void firstStartTest() {
val activity = MainActivity()
activity.setSomeProperty("something") //does not compile
}
Android Studio is suggesting activity.setSomeProperty$production_sources_for_module_app();
but this also does not compile.

Both classes (MainActivity and test class) must be in one module. This is a module definition:
More specifically, a module is a set of Kotlin files compiled together:
an IntelliJ IDEA module;
a Maven or Gradle project;
a set of files
compiled with one invocation of the Ant task.
https://kotlinlang.org/docs/reference/visibility-modifiers.html
It means, check your project structure.

add #JvmField annotation.
It treats variable as java protected

There are two ways of doing this:
Make the property protected. Note on how Java & Kotlin treat protected differently. In Java it's possible that other classes in the same package access protected members. Thus your test class (in Java) can access it.
Access the property via its ugly name. It should be sort of like activity.setSomeProperty$production_.... Make use the autocomplete. From the documentation:
Members of internal classes go through name mangling, to make it
harder to accidentally use them from Java and to allow overloading for
members with the same signature that don't see each other according to
Kotlin rules;

Loading a class and checking for method presence without resolving the class

I am trying to load a class from a folder to check for the implementation of a specific method. The class has some imports that are not present in the folder or its subfolders. Loading the class with Class clazz = Class.forName(className, false, classLoader); works fine, but when I call clazz.getDeclaredMethod("methodName") then I get a NoClassDefFoundError because some imports cannot be resolved.
Is there a possibility to examine a class at runtime (I do not intend to call methods or instantiate it) without loading dependencies?
If not, how else can I check a class for a specific method implementation when I have a classes-folder as a starting point?

Interesting question.
I don't think you want to roll your own byte code parser, so try Apache BCEL or Spring ASM. Both allow you to read/write class files without having to load them.

You might be able to do that with a library such as Apache Commons BCEL.
The Byte Code Engineering Library (Apache Commons BCEL™) is intended to give users a convenient way to analyze, create, and manipulate (binary) Java class files (those ending with .class).

you could try executing `javap ' from your program and parsing the output. For example:
javap Driver
Compiled from "Driver.java"
public class Driver {
public Driver();
public static void main(java.lang.String[]);
}

How to keep a jar file external but still use its classes in my Android project?

I need to have a jar file located in a main/assets directory within an Android project. It is important the jar file is located there.
With my main Android project is there a way to reference this jar file in my code and to use its classes?
To be clear I don't want to add the jar to the main project once compiled.
EDIT: I have tried the link below and it seems to load the Class file I've stated. But I'm strugging how to define constructor arguments for the dynamically loaded Class.
android-custom-class-loading-sample
EDIT2
Nearly there. I've confirmed the class is loaded from my classes.jar. I'm stuck instantiating it though.
On the licenseValidatorClazz.getConstructor line I get the error below. I'm guessing I'm missing something from my Interface file?
java.lang.NoSuchMethodException: [interface com.google.android.vending.licensing.Policy, interface com.google.android.vending.licensing.DeviceLimiter, interface com.google.android.vending.licensing.LicenseCheckerCallback, int, class java.lang.String, class java.lang.String]
public Class licenseValidatorClazz = null;
public LicenseValidator validator;
...
// Initialize the class loader with the secondary dex file.
DexClassLoader cl = new DexClassLoader(dexInternalStoragePath.getAbsolutePath(),
optimizedDexOutputPath.getAbsolutePath(),
null,
mContext.getClassLoader());
try {
// Load the library class from the class loader.
licenseValidatorClazz = cl.loadClass("com.google.android.vending.licensing.LicenseValidator");
validator = (LicenseValidator) licenseValidatorClazz.getConstructor(Policy.class,DeviceLimiter.class,LicenseCheckerCallback.class,int.class,String.class,String.class).newInstance(ddd, new NullDeviceLimiter(),
callback, generateNonce(), mPackageName, mVersionCode);
} catch (Exception exception) {
// Handle exception gracefully here.
exception.printStackTrace();
}
I have an Interface which contains the functions to pass to the loaded class.
public interface LicenseValidator
{
public LicenseCheckerCallback getCallback();
public int getNonce();
public String getPackageName();
public void verify(PublicKey publicKey, int responseCode, String signedData, String signature);
public void handleResponse(int response, ResponseData rawData);
public void handleApplicationError(int code);
public void handleInvalidResponse();
}

TO use an external jar to be associated with your application and use it during runtime, it needs to be in dalvik format since normal jars cannot work under dalvikVM.
Convert your files using the dx tool
using aapt cmd , add those classes.dex to your jar file.
Now this jar which contains files in dalvik format can be loaded into our project.
Here is a post which explains the procedure to accomplish it.

There are steps to accomplish this.
You have to make a copy of your JAR file into the private internal storage of your aplication.
Using the dx tool inside the android folder, you have to generate a classes.dex file associated with the JAR file. The dx tool will be at the location /android-sdks/build-tools/19.0.1 (this file is needed by the Dalvik VM, simply jar can not be read by the dalvik VM))
Using the aapt tool command which is also inside the same location, you have to add the classes.dex to the JAR file.
This JAR file could be loaded dynamically using DexClassLoader.
If you are making a JAR from any one your own library, you have to do this steps (1-4) every time when there is a change in your library source code. So you can automate this steps by creating a shell script(in Mac/Linux/Ubuntu) or batch scripts(in Windows). You can refere this link to understand how to write shell scripts.
Note : One situation for implementing this method is, when it is impossible to add the JAR files directly to the build path of core project and need to be loaded dynamically at run time. In normal cases the JAR files could be added to the build path.
please check this link for the detailed code and implementation.
How to load a jar file at runtime
Android: How to dynamically load classes from a JAR file?
Hope this helps!!

You should try out the Services API - java.util.ServiceLoader
You define a service interface and its implementations in your jar.
package com.my.project;
public interface MyService { ... }
public class MyServiceBarImpl implements MyService { ... }
public class MyServiceFooImpl implements MyService { ... }
Then you define the services contained within the jar file in the META-INF/services/ directory. For instance, in the file 'META-INF/services/com.my.project.MyService', you list the provider classes.
# Known MyService providers.
com.my.project.MyServiceBarImpl # The original implementation for handling "bar"s.
com.my.project.MyServiceFooImpl # A later implementation for "foo"s.
Then, in your main codebase, you can instantiate a MyService instance with the ServiceLoader:
for (MyService service : ServiceLoader.load(MyService.class)) {
//Perform some test to determine which is the right MyServiceImpl
//and then do something with the MyService instance
}
These examples are taken more-or-less straight from the API, although I've changed the package names to make them slightly less annoying to read.