Using external classes with dataflow - java

I am trying to make a custom transform class that takes in a side input and I am using PipelineTester and PAssert to try and test it, but I keep getting a no such method exception on methods I am trying to bring into the transform from other classes.
Caused by: java.lang.NoSuchMethodError:
com.org.utils.MyUtils.createMap(Ljava/lang/Iterable;)Ljava/util/Map;
at com.org.beam.MyTransform.ProcessElement(MyTransform.java:51)
I have tried using the #Autowired annotation to bring in the class like
#Autowired
private MyUtils myutils;
as well as just creating a static instance in the code like
private static MyUtils myUtils = new MyUtils();
and then calling
this.myUtils.createMap(mapThisToThat(inputCollection, this.myMap));
I have also tried making the methods static and calling them like
MyUtils.createMap(mapThisToThat(inputCollection, this.myMap));
the signature to mapThisToThat is
private Iterable<MyObject> mapThisToThat(Iterable<MyObject> objectIterator, Map<String, Integer> myMap) {
which is being passed into the createMap method which has this signature -
public Map<String, MyObject> createMap(Iterable<MyObject> inputCollection){
so it is passing in an Iterable of MyObjects correctly, but it says the method doesn't exist for some reason. does this mean beam transforms can't have external methods or am I doing something wrong?

For me in python, there are a variety of things I need to do for that to work:
https://cloud.google.com/dataflow/faq#how-do-i-handle-nameerrors
https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/
For you in java, they don't have reciprocal documentation for some reason, but over here https://beam.apache.org/documentation/runners/dataflow/ they say things like this:
In some cases, such as starting a pipeline using a scheduler such as Apache AirFlow, you must have a self-contained application. You can pack a self-executing JAR by explicitly adding the following dependency on the Project section of your pom.xml, in addition to the adding existing dependency shown in the previous section.
In their examples readme https://github.com/mbrukman/apache-beam/tree/master/examples/java they say this
Alternatively, you may choose to bundle all dependencies into a single JAR and execute it outside of the Maven environment. For example, you can execute the following commands to create the bundled JAR of the examples and execute it both locally and in Cloud Platform
If you continue to browse that examples repo, there is a common folder with utils. Hopefully you can copy how they did it.

Related

Add an annotation to a class from a place other than class definition

I need to add an annotation to a class which is in a sperate dependency (in a separate jar) of the project that I am working on. I am wondering that can I do this in Java?
As an example, I need to add an annotation HisClass without touching the source code of HisClass.
public class HisClass{
// ...
}
So, that the above class should be look like below at the build time,
#MyAnnot ( me = MyClass.class )
public class HisClass{
// ...
}
There are many ways:
Write a compiler plugin for javac, and make it add the annotations. This will be quite difficult, as the plugin API has nearly no documentation.
2.(Maybe not possible) Inject the annotation after compiling. Add an extra buildstep after compiling and use a library like ASM to add this annotation to the classfile.

How to import/load classes into a jenkinsfile from another file

I need to import custom classes defined outside of my jenkinsfile. These classes have constructors which require parameters. I've already tried using load() for each of the class files, but it seems this only works for script files that just define static methods. When I tried using this to load my class files, it threw an error that it could not find an <init> method.
I currently have a way of importing these classes that works, but it requires re-cloning the repository as a library (code below). I'd like to move away from this since it's inefficient and seems like it should be unnecessary.
current working (but not great) implementation:
Jenkinsfile:
lib = library(identifier: "<libraryName>#${env.currentBranch}",
retriever: modernSCM([$class: 'GitSCMSource',
credentialsId: <credentialsId>,
id: '<id>',
remote: env.projectRemote,
traits: [[$class: 'jenkins.plugins.git.traits.BranchDiscoveryTrait']]])).com.company.jenkins.sdk
def git = lib.Git.new(this, currentStage, currentStep)
...
src/com/company/jenkins/sdk/Git.groovy:
package com.company.jenkins.sdk.Git
class Git implements Serializable {
def script
def stage
def step
Git(script, stage, step){
...
}
}
I'd like to get rid of that library command and replace it with a simpler way of importing these classes.

how to access internal properties from java test code

I've got some class with property marked as internal.
Then I try to set that property from test code which is in java.
How can I access those properties? test code and class code are in the same package.
example:
class MainActivity : AppCompatActivity() {
interal var someProperty = "test"
}
test code:
#Test
public void firstStartTest() {
val activity = MainActivity()
activity.setSomeProperty("something") //does not compile
}
Android Studio is suggesting activity.setSomeProperty$production_sources_for_module_app();
but this also does not compile.
Both classes (MainActivity and test class) must be in one module. This is a module definition:
More specifically, a module is a set of Kotlin files compiled together:
an IntelliJ IDEA module;
a Maven or Gradle project;
a set of files
compiled with one invocation of the Ant task.
https://kotlinlang.org/docs/reference/visibility-modifiers.html
It means, check your project structure.
add #JvmField annotation.
It treats variable as java protected
There are two ways of doing this:
Make the property protected. Note on how Java & Kotlin treat protected differently. In Java it's possible that other classes in the same package access protected members. Thus your test class (in Java) can access it.
Access the property via its ugly name. It should be sort of like activity.setSomeProperty$production_.... Make use the autocomplete. From the documentation:
Members of internal classes go through name mangling, to make it
harder to accidentally use them from Java and to allow overloading for
members with the same signature that don't see each other according to
Kotlin rules;

What is a good practice or design to swap algorithms at runtime?

I have several data processing algorithms that can be assembled into a pipeline to transform data. The code is split into two components: A pre-processing component that does data loading-related tasks, and a processing pipeline component.
I currently have the two parts compiled and packaged into two separate jars. The idea is that the same pre-processing jar can be shipped to all customers, but the pipeline jar can be exchanged depending on customer requirements. I would like to keep the code simple and minimize configuration, so that rules out the use of OSGi or CDI frameworks.
I've gotten some hints by looking at SLF4J's implementation. That project is split into two parts: A core API, and a bunch of implementations that wrap different logging APIs. The core API makes calls to dummy classes (which exist in the core project simply to allow compilation) that are meant to be overridden by the same classes found in the logging projects. At build time, the compiled dummy classes are deleted from the core API before packaging into jar. At run time, the core jar and a logging jar are required to be included in the class path, and the missing class files in the core jar will be filled in by the files from the logging jar. This works fine, but it feels a little hacky to me. I'm wondering if there is a better design, or if this is the best that be done without using CDI frameworks.
Sounds like the strategy software design pattern.
https://en.wikipedia.org/wiki/Strategy_pattern
Take a look at the ServiceLoader.
Example Suppose we have a service type com.example.CodecSet which is
intended to represent sets of encoder/decoder pairs for some protocol.
In this case it is an abstract class with two abstract methods:
public abstract Encoder getEncoder(String encodingName);
public abstract Decoder getDecoder(String encodingName);
Each method returns an appropriate object or null if the provider does
not support the given encoding. Typical providers support more than one
encoding. If com.example.impl.StandardCodecs is an implementation of
the CodecSet service then its jar file also contains a file named
META-INF/services/com.example.CodecSet
This file contains the single line:
com.example.impl.StandardCodecs # Standard codecs
The CodecSet class creates and saves a single service instance at
initialization:
private static ServiceLoader<CodecSet> codecSetLoader
= ServiceLoader.load(CodecSet.class);
To locate an encoder for a given encoding name it defines a static factory method which iterates
through the known and available providers, returning only when it has
located a suitable encoder or has run out of providers.
public static Encoder getEncoder(String encodingName) {
for (CodecSet cp : codecSetLoader) {
Encoder enc = cp.getEncoder(encodingName);
if (enc != null)
return enc;
}
return null;
}
A getDecoder method is defined similarly.
You already understand the gist of how to use it:
Split your project into parts (core, implementation 1, implementation 2, ...)
Ship the core API with the pre-processor
Have each implementation add the correct META-INF file to its .jar file.
The only configuration files that are necessary are the ones you package into your .jar files.
You can even have them automatically generated for you with an annotation:
package foo.bar;
import javax.annotation.processing.Processor;
#AutoService(Processor.class)
final class MyProcessor extends Processor {
// …
}
AutoService will generate the file
META-INF/services/javax.annotation.processing.Processor
in the output classes folder. The file will contain:
foo.bar.MyProcessor

Testing ServiceLoader in Eclipse Plugins

I have four Eclipse plugin projects (create a new Java Project, right-click, configure, Convert to Plugin Project) in my workspace. The first (my.runtime) contains an interface (MyFactoryInterface) and a class (MyClient) that defines a method (List<String> getAllFactoryNames()) that loads all implementations of that interface via java.util.ServiceLoader and calls a method (String getName()) on them, collecting the results and returning them in a list.
To test that class, I have a JUnit test in the second project (my.runtime.test, set up with my.runtime as Fragment-Host), checking if the name returned by a dummy implementation(MyDummy, returning "Dummy") I have in the my.runtime.test project is in the list returned by MyClient.getAllFactoryNames(). So far, it works fine.
In the third project (my.extension, with my.runtime as dependency) I have a class (MyHello) that uses the names returned by MyClient.getAllFactoryNames() to return a list of greetings ("Hello "+name).
Again, to test this, I have a project (my.extension.test, with my.extension as Fragment-Host) containing another Implementation (MyWorld, returning "World" as name) and a JUnit test case checking if "Hello World" is in the greetings returned by MyHello.getGreetings(). This test fails, as MyClient still only finds the MyDummy implementation, and not the MyWorld implementation. Both implementations are accompanied by matching entries in META-INF/services/my.runtime.MyFactoryInterface files.
I currently use the following code to load the implementations:
ServiceLoader<MyFactoryInterface> myFactoryLoader = ServiceLoader.load(MyFactoryInterface.class);
for (MyFactoryInterface myFactory : myFactoryLoader) {
I know that I can supply a ClassLoader as a second argument to ServiceLoader.load, but I have no idea how to get one that knows all plugin projects... any suggestions? Or is ServiceLoader not the right tool for this problem?
If anyone stumbles over the same problem: the combination of ServiceLoader.load(MyFactoryInterface.class, Thread.currentThread().getContextClassLoader()) in MyClient and Thread.currentThread().setContextClassLoader(MyWorld.class.getClassLoader()); in the second test did the job.

Categories

Resources