Develop Reference

Checkpoint with spark file streaming in java

I want to implement checkpoint with spark file streaming application to process all unprocessed files from hadoop if in any case my spark streaming application stop/terminates. I am following this : streaming programming guide, but not found JavaStreamingContextFactory. Please help me what should I do.
My Code is
public class StartAppWithCheckPoint {
public static void main(String[] args) {
try {
String filePath = "hdfs://Master:9000/mmi_traffic/listenerTransaction/2020/*/*/*/";
String checkpointDirectory = "hdfs://Mongo1:9000/probeAnalysis/checkpoint";
SparkSession sparkSession = JavaSparkSessionSingleton.getInstance();
JavaStreamingContextFactory contextFactory = new JavaStreamingContextFactory() {
#Override public JavaStreamingContext create() {
SparkConf sparkConf = new SparkConf().setAppName("ProbeAnalysis");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
JavaStreamingContext jssc = new JavaStreamingContext(sc, Durations.seconds(300));
JavaDStream<String> lines = jssc.textFileStream(filePath).cache();
jssc.checkpoint(checkpointDirectory);
return jssc;
}
};
JavaStreamingContext context = JavaStreamingContext.getOrCreate(checkpointDirectory, contextFactory);
context.start();
context.awaitTermination();
context.close();
sparkSession.close();
} catch(Exception e) {
e.printStackTrace();
}
}
}

You must use Checkpointing
For checkpointing use stateful transformations either updateStateByKey or reduceByKeyAndWindow. There are a plenty of examples in spark-examples provided along with prebuild spark and spark source in git-hub. For your specific, see JavaStatefulNetworkWordCount.java;

How to enable journal in embedded MongoDB from Spring Boot test

I am trying to write a Spring Boot test that uses embedded MondoDB 4.0.2; the code to test requires Mongo ChangeStreams which requires MongoDB start as a replica set. MongoDB as a replica set at MongoDB v4 requires journaling enabled. I was not able to find a way to start with journaling enabled so posted this here looking for answers. I subsequently found out how to do it - below.
I have spring-boot 2.1.3.RELEASE. Spring-data-mongodb 2.1.5.RELEASE
This is what I'd been trying:
#RunWith(SpringRunner.class)
#DataMongoTest(properties= {
"spring.mongodb.embedded.version= 4.0.2",
"spring.mongodb.embedded.storage.repl-set-name = r_0",
"spring.mongodb.embedded.storage.journal.enabled=true"
})
public class MyStreamWatcherTest {
#SpringBootApplication
#ComponentScan(basePackages = {"my.package.with.dao.classes"})
#EnableMongoRepositories( { "my.package.with.dao.repository" })
static public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}
#Before
public void startup() {
MongoDatabase adminDb = mongoClient.getDatabase("admin");
Document config = new Document("_id", "rs0");
BasicDBList members = new BasicDBList();
members.add(new Document("_id", 0).append("host",
mongoClient.getConnectPoint()));
config.put("members", members);
adminDb.runCommand(new Document("replSetInitiate", config));
However, when the test starts the options used to start mongo did not include enabling journal.
The fix was to add this class:
#Configuration
public class MyEmbeddedMongoConfiguration {
private int localPort = 0;
public int getLocalPort() {
return localPort;
}
#Bean
public IMongodConfig mongodConfig(EmbeddedMongoProperties embeddedProperties) throws IOException {
MongodConfigBuilder builder = new MongodConfigBuilder()
.version(Version.V4_0_2)
.cmdOptions(new MongoCmdOptionsBuilder().useNoJournal(false).build());
// Save the local port so the replica set initializer can come get it.
this.localPort = Network.getFreeServerPort();
builder.net(new Net("127.0.0.1", this.getLocalPort(), Network.localhostIsIPv6()));
EmbeddedMongoProperties.Storage storage = embeddedProperties.getStorage();
if (storage != null) {
String databaseDir = storage.getDatabaseDir();
String replSetName = "rs0"; // Should be able to: storage.getReplSetName();
int oplogSize = (storage.getOplogSize() != null)
? (int) storage.getOplogSize().toMegabytes() : 0;
builder.replication(new Storage(databaseDir, replSetName, oplogSize));
}
return builder.build();
}
This got journal enabled and the mongod started with replica set enabled. Then I added another class to initialize the replica set:
#Configuration
public class EmbeddedMongoReplicaSetInitializer {
#Autowired
MyEmbeddedMongoConfiguration myEmbeddedMongoConfiguration;
MongoClient mongoClient;
// We don't use this MongoClient as it will try to wait for the replica set to stabilize
// before address-fetching methods will return. It is specified here to order this class's
// creation after MongoClient, so we can be sure mongod is running.
EmbeddedMongoReplicaSetInitializer(MongoClient mongoClient) {
this.mongoClient = mongoClient;
}
#PostConstruct
public void initReplicaSet() {
//List<ServerAddress> serverAddresses = mongoClient.getServerAddressList();
MongoClient mongo = new MongoClient(new ServerAddress("127.0.0.1", myEmbeddedMongoConfiguration.getLocalPort()));
MongoDatabase adminDb = mongo.getDatabase("admin");
Document config = new Document("_id", "rs0");
BasicDBList members = new BasicDBList();
members.add(new Document("_id", 0).append("host", String.format("127.0.0.1:%d", myEmbeddedMongoConfiguration.getLocalPort())));
config.put("members", members);
adminDb.runCommand(new Document("replSetInitiate", config));
mongo.close();
}
}
That's getting the job done. If anyone has tips to make this easier, please post here.

Executing a single command and exiting from Spring Shell 2

I stumbled upon this question from a while back, explaining how to get a Spring Shell application to exit after calling it from the command line with a single command. However, testing this in 2.0.0 with Spring Boot, it does not seem to be the case any more that invoking the JAR with command arguments will execute that command and then exit. The shell just starts as normal without executing the supplied command. Is it still possible to do this? If not, would it be possible to pass the arguments from the JAR execution to Spring Shell and then trigger an exit after execution?
For example, let's say I have a command, import that has a couple options. It could be run in the shell like this:
$ java -jar my-app.jar
> import -f /path/to/file.txt --overwrite
Successfully imported 'file.txt'
> exit
But it would be nice to be able to simply execute and exit, for the sake of building a script that can utilize this function:
$ java -jar my-app.jar import -f /path/to/file.txt --overwrite
Successfully imported 'file.txt'

Run it with #my-script, like so:
java -jar my-app.jar #my-script
Where my-script is a file with your commands:
import -f /path/to/file.txt --overwrite

Just to add, I found another way of doing this, doesn't give you the option to run in interactive mode but using your profiles above you can of course swap the configuration. Please note I'm using lombok and jool (just in case anyone copy pastes and gets funny issues!)
Entry
#SpringBootApplication
public class Righter {
public static void main(String[] args) {
SpringApplication.run(Righter.class, args);
}
#Bean
public ApplicationRunner shellRunner(Shell shell) {
return new NonInteractiveShellRunner(shell);
}
Application runner:
#Order(0)
public class NonInteractiveShellRunner implements ApplicationRunner{
private final Shell shell;
public NonInteractiveShellRunner(Shell shell) {
this.shell = shell;
}
#Override
public void run(ApplicationArguments args) throws Exception {
shell.run(new CommandInputProvider(args.getSourceArgs()));
}
public static class PredefinedInputProvider implements InputProvider{
private final Input input;
private boolean commandExecuted = false;
public PredefinedInputProvider(String[] args) {
this.input = new PredefinedInput(args);
}
#Override
public Input readInput() {
if (!commandExecuted){
commandExecuted=true;
return input;
}
return new PredefinedInput(new String[]{"exit"});
}
#AllArgsConstructor
private static class PredefinedInput implements Input{
private final String[] args;
#Override
public String rawText() {
return Seq.of(args).toString(" ");
}
#Override
public List<String> words(){
return Arrays.asList(args);
}
}
}
}

A way to add a single command run mode without excluding interactive mode and script mode (Tested on spring-shell-starter::2.0.0.RELEASE).
Create a runner by analogy with ScriptShellApplicationRunner.
// Runs before ScriptShellApplicationRunner and InteractiveShellApplicationRunner
#Order(InteractiveShellApplicationRunner.PRECEDENCE - 200)
public class SingleCommandApplicationRunner implements ApplicationRunner {
private final Parser parser;
private final Shell shell;
private final ConfigurableEnvironment environment;
private final Set<String> allCommandNames;
public SingleCommandApplicationRunner(
Parser parser,
Shell shell,
ConfigurableEnvironment environment,
Set<CustomCommand> customCommands
) {
this.parser = parser;
this.shell = shell;
this.environment = environment;
this.allCommandNames = buildAllCommandNames(customCommands);
}
private Set<String> buildAllCommandNames(Collection<CustomCommand> customCommands) {
final Set<String> result = new HashSet<>();
customCommands.stream().map(CustomCommand::keys).flatMap(Collection::stream).forEach(result::add);
// default spring shell commands
result.addAll(asList("clear", "exit", "quit", "help", "script", "stacktrace"));
return result;
}
#Override
public void run(ApplicationArguments args) throws Exception {
final boolean singleCommand = haveCommand(args.getSourceArgs());
if (singleCommand) {
InteractiveShellApplicationRunner.disable(environment);
final String fullArgs = join(" ", args.getSourceArgs());
try (Reader reader = new StringReader(fullArgs);
FileInputProvider inputProvider = new FileInputProvider(reader, parser)) {
shell.run(inputProvider);
}
}
}
private boolean haveCommand(String... args) {
for (String arg : args) {
if (allCommandNames.contains(arg)) {
return true;
}
}
return false;
}
}
Register the runner as bean.
#Configuration
class ContextConfiguration {
#Autowired
private Shell shell;
#Bean
SingleCommandApplicationRunner singleCommandApplicationRunner(
Parser parser,
ConfigurableEnvironment environment,
Set<CustomCommand> customCommands
) {
return new SingleCommandApplicationRunner(parser, shell, environment, customCommands);
}
}
So that the runner starts only when the command is sent, we create an interface.
public interface CustomCommand {
Collection<String> keys();
}
Implement CustomCommand interface in each commands.
#ShellComponent
#RequiredArgsConstructor
class MyCommand implements CustomCommand {
private static final String KEY = "my-command";
#Override
public Collection<String> keys() {
return singletonList(KEY);
}
#ShellMethod(key = KEY, value = "My custom command.")
public AttributedString version() {
return "Hello, single command mode!";
}
}
Done!
Run in interactive mode:
java -jar myApp.jar
// 2021-01-14 19:28:16.911 INFO 67313 --- [main] com.nao4j.example.Application: Starting Application v1.0.0 using Java 1.8.0_275 on Apple-MacBook-Pro-15.local with PID 67313 (/Users/nao4j/example/target/myApp.jar started by nao4j in /Users/nao4j/example/target)
// 2021-01-14 19:28:16.916 INFO 67313 --- [main] com.nao4j.example.Application: No active profile set, falling back to default profiles: default
// 2021-01-14 19:28:18.227 INFO 67313 --- [main] com.nao4j.example.Application: Started Application in 2.179 seconds (JVM running for 2.796)
// shell:>my-command
// Hello, single command mode!
Run script from file script.txt (contains text "my-command"):
java -jar myApp.jar #script.txt
// 2021-01-14 19:28:16.911 INFO 67313 --- [main] com.nao4j.example.Application: Starting Application v1.0.0 using Java 1.8.0_275 on Apple-MacBook-Pro-15.local with PID 67313 (/Users/nao4j/example/target/myApp.jar started by nao4j in /Users/nao4j/example/target)
// 2021-01-14 19:28:16.916 INFO 67313 --- [main] com.nao4j.example.Application: No active profile set, falling back to default profiles: default
// 2021-01-14 19:28:18.227 INFO 67313 --- [main] com.nao4j.example.Application: Started Application in 2.179 seconds (JVM running for 2.796)
// Hello, single command mode!
Run in single command mode:
java -jar myApp.jar my-command
// 2021-01-14 19:28:16.911 INFO 67313 --- [main] com.nao4j.example.Application: Starting Application v1.0.0 using Java 1.8.0_275 on Apple-MacBook-Pro-15.local with PID 67313 (/Users/nao4j/example/target/myApp.jar started by nao4j in /Users/nao4j/example/target)
// 2021-01-14 19:28:16.916 INFO 67313 --- [main] com.nao4j.example.Application: No active profile set, falling back to default profiles: default
// 2021-01-14 19:28:18.227 INFO 67313 --- [main] com.nao4j.example.Application: Started Application in 2.179 seconds (JVM running for 2.796)
// Hello, single command mode!

In addition to Alex answers, here is the simpler version of NonInteractiveApplicationRunner I made.
#Component
#Order(InteractiveShellApplicationRunner.PRECEDENCE - 100)
class NonInteractiveApplicationRunner implements ApplicationRunner {
private final Shell shell;
private final ConfigurableEnvironment environment;
public NonInteractiveApplicationRunner(Shell shell, ConfigurableEnvironment environment) {
this.shell = shell;
this.environment = environment;
}
#Override
public void run(ApplicationArguments args) {
if (args.getSourceArgs().length > 0) {
InteractiveShellApplicationRunner.disable(environment);
var input = String.join(" ", args.getSourceArgs());
shell.evaluate(() -> input);
shell.evaluate(() -> "exit");
}
}
}
Using #Component, we don't need to add bean method. In addition, using shell.evaluate() method looks much simpler compare to shell.run(...).

I found a nice little work-around. Rather than creating an ApplicationRunner that mimics the v1 behavior (which is tricky, since JLineInputProvider is a private class), I created one that is optionally loaded, based on active Spring profile. I used JCommander to define the CLI parameters, allowing me to have identical commands for the interactive shell and the one-off executions. Running the Spring Boot JAR with no args triggers the interactive shell. Running it with arguments triggers the one-and-done execution.
#Parameters
public class ImportParameters {
#Parameter(names = { "-f", "--file" }, required = true, description = "Data file")
private File file;
#Parameter(names = { "-t", "--type" }, required = true, description = "Data type")
private DataType dataType;
#Parameter(names = { "-o", "--overwrite" }, description = "Flag to overwrite file if it exists")
private Boolean overwrite = false;
/* getters and setters */
}
public class ImportCommandExecutor {
public void run(ImportParameters params) throws Exception {
// import logic goes here
}
}
/* Handles interactive shell command execution */
#ShellComponent
public class JLineInputExecutor {
// All command executors are injected here
#Autowired private ImportCommandExecutor importExecutor;
...
#ShellMethod(key = "import", value = "Imports the a file of a specified type.")
public String importCommand(#ShellOption(optOut = true) ImportParameters params) throws Exception {
importCommandExecutor.run(params);
}
...
}
/* Handles one-off command execution */
public class JCommanderInputExecutor implements ApplicationRunner {
// All command executors are injected here
#Autowired private ImportCommandExecutor importExecutor;
...
#Override
public void run(ApplicationArguments args) throws Exception {
// Create all of the JCommander argument handler objects
BaseParameters baseParameters = new BaseParameters();
ImportParameters importParameters = new ImportParameters();
...
JCommander jc = newBuilder().
.acceptUnknownOptions(true)
.addObject(baseParameters)
.addCommand("import", importParameters)
...
.build();
jc.parse(args);
String mainCommand = jc.getParsedCommand();
if ("import".equals(mainCommand)){
importExecutor.run(importParameters);
} else if (...) {
...
}
}
}
#Configuration
#Profile({"CLI"})
public class CommandLineInterfaceConfiguration {
// All of my command executors are defined as beans here, as well as other required configurations for both modes of execution
#Bean
public ImportCommandExecutor importExecutor (){
return new ImportCommandExecutor();
}
...
}
#Configuration
#Profile({"SINGLE_COMMAND"})
public class SingleCommandConfiguration {
#Bean
public JCommanderInputExecutor commandLineInputExecutor(){
return new JCommanderInputExecutor();
}
}
#SpringBootApplication
public class Application {
public static void main(String[] args) throws IOException {
String[] profiles = getActiveProfiles(args);
SpringApplicationBuilder builder = new SpringApplicationBuilder(Application.class);
builder.bannerMode((Mode.LOG));
builder.web(false);
builder.profiles(profiles);
System.out.println(String.format("Command line arguments: %s Profiles: %s",
Arrays.asList(args), Arrays.asList(profiles)));
builder.run(args);
}
private static String[] getActiveProfiles(String[] args){
return Arrays.asList(args).contains("-X") ? new String[]{"CLI", "SINGLE_COMMAND"} : new String[]{"CLI"};
}
}
So now I can trigger the interactive client by simply running my executable JAR:
java -jar app.jar
> import -f /path/to/file.txt -t GENE -o
> quit()
Or, if I pass the '-X' argument on the command line, the application will execute and then exit:
java -jar app.jar -X import -f /path/to/file.txt -t GENE -o

In linux works this way too:
echo "import -f /path/to/file.txt --overwrite" | java -jar my-app.jar
despite the fact that it ends with a failure; it happens only after the command is successfully executed.

Passing varargs to Java from Python using Py4j

I am trying to pass varargs to Java code from python.
Java code :
LogDebugCmd.java
public class LogDebugCmd implements Command {
private Class clazz;
private String format;
private Object[] args;
public LogDebugCmd() {}
public void setLog(String format, Object... args) {
this.format = format;
this.args = args.clone();
clazz = getClass();
}
#Override
public void execute() {
VLog.getLog(clazz).debug(format, args);
}
CommandEntryPoint.java
public class CommandEntryPoint {
private LogDebugCmd logDebugCmd;
public CommandEntryPoint() {
logDebugCmd = new LogDebugCmd();
public LogDebugCmd getLogDebugCmd(String str, Object... args) {
setLog(str, args);
return logDebugCmd;
}
public static void main(String args[]) {
GatewayServer gatewayServer = new GatewayServer(new CommandEntryPoint());
gatewayServer.start();
System.out.println("Gateway Server Started");
}
Python code
test.py:
from py4j.java_gateway import JavaGateway
gateway = JavaGateway()
logDebugCmd = gateway.entry_point.getLogDebugCmd("Step 01 - Test initiated.")
logDebugCmd..execute()
Error :
Traceback (most recent call last):
File "C:/Python27/Lib/site-packages/py4j-0.10.3-py2.7.egg/py4j/sampleTLStest.py", line 4, in <module>
logDebugCmd = gateway.entry_point.getLogDebugCmd("Step 01 - Test initiated.")
File "C:\Python27\lib\site-packages\py4j-0.10.3-py2.7.egg\py4j\java_gateway.py", line 1133, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "C:\Python27\lib\site-packages\py4j-0.10.3-py2.7.egg\py4j\protocol.py", line 323, in get_return_value
format(target_id, ".", name, value))
***Py4JError: An error occurred while calling t.getLogDebugCmd.
Trace:
py4j.Py4JException: Method getLogDebugCmd([class java.lang.String]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
at py4j.Gateway.invoke(Gateway.java:272)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)***
Is there a way to pass varargs to Java ? I am trying to pass String as the second argument to getLogDebugCmd() which also errors out.

Varargs are actually represented as an array so you can do this:
# Pass a None if you have no varargs
logDebugCmd = gateway.entry_point.getLogDebugCmd("Step 01 - Test initiated.", None)
# Create an array of objects
object_class = gateway.jvm.Object
object_array = gateway.new_array(object_class, 2)
object_array[0] = "foo"
object_array[1] = "bar"
logDebugCmd = gateway.entry_point.getLogDebugCmd("Step 01 - Test initiated.", object_array)
There is a pull request to improve varargs support in Py4J, but it has some compilation issues so it has not been merged yet.

Flyway: slow performance migrating a large number of scripts

We are using flywayDB for some years to manage a quite large database application running on oracle 12c. This works fine and very reliable.
But recently we are running into performance problems during database migration. The number of database skripts we manage in our versions-table has risen beyond 10,000. The time flyway takes to migrate a single script has increased from some originally milliseconds to currently about a second. It seems that flyway per single migration step selects the full content of the versions table to calculate at client side its installation rank. This doesnt scale well.
Is there any possiblity to speed up flyway, may be by caching the content of the versions table?
We are using the Java-API of flyway version 3.2.1.
For completeness I have written a testcase to demonstrate this behaviour.
#RunWith(Parameterized.class)
public class PerformanceTestcase {
private static Logger LOG = Logger.getLogger( PerformanceTestcase.class.getName() );
#Parameter
public int noOfScripts;
#Before
public void generateLotsOfInstallerSkripts() throws IOException {
LOG.log(Level.INFO, "generating {0} skripts", noOfScripts);
Path baseVersion = getBaseVersionPath();
generateSkripts( noOfScripts, baseVersion, BASE_SKRIPT_NAME );
}
#Test
public void testPerformance() throws IOException, SQLException {
// this one does not scale well with increasing noOfScripts
migrate();
}
private static final String SCHEMA_TABLE_NAME = "test_versions";
private static final String SKRIPT_NAME_FORMAT = "%s.%05d__test.sql";
private static final String SKRIPT_CONTENT = "select %05d from dual;";
private static final String FILESYSTEM = "filesystem:";
private static final String BASE_SKRIPT_NAME = "V00.00.00";
private static final String BASE_DIR = "/tmp/performanceTest";
private void migrate() throws SQLException {
Flyway flyway = new Flyway();
flyway.setDataSource( getDataSource() );
flyway.setLocations( FILESYSTEM + BASE_DIR );
flyway.setTable( SCHEMA_TABLE_NAME );
flyway.setBaselineVersionAsString(BASE_SKRIPT_NAME.substring(1) );
flyway.setBaselineOnMigrate(true);
flyway.setValidateOnMigrate(false);
flyway.migrate();
}
#Parameters(name="noOfScripts={0}")
public static Iterable<? extends Object> data() {
List<Integer> retval = new LinkedList<Integer>();
for ( int i=0; i<16000; i+=1000 ) {
if ( i>0 ) retval.add( Integer.valueOf(i) );
retval.add( Integer.valueOf(i+100) );
}
return retval;
}
private Path getBaseDirPath() throws IOException {
Path base = Paths.get(BASE_DIR);
if ( !Files.exists(base) ) {
Files.createDirectory(base);
}
return base;
}
private Path getBaseVersionPath() throws IOException {
Path base = getBaseDirPath();
Path baseVersion = base.resolve(BASE_SKRIPT_NAME);
if ( !Files.exists(baseVersion) ) {
Files.createDirectories(baseVersion);
}
return baseVersion;
}
private void generateSkripts( int numberOfSkripts, Path baseDir, String baseName ) throws IOException {
for (int i = 0; i < numberOfSkripts; i++) {
Path file = baseDir.resolve( String.format(SKRIPT_NAME_FORMAT, baseName, i) );
Files.write( file
, Arrays.asList( new String[] { String.format( SKRIPT_CONTENT, i ) } )
, StandardOpenOption.CREATE
, StandardOpenOption.TRUNCATE_EXISTING
);
}
}
private DataSource getDataSource() throws SQLException {
OracleDataSource ds = new OracleDataSource();
ds.setURL(CONNECTION_URL);
return ds;
}
}
Update
I just run the testcase with the current version 4.0.3 of flywayDB. It ran about half the time compared to 3.2.1, but the scaling problem still persists. Flyway is selecting the complete versions table per single migration step, which significantly slows down a migration when the version table is quite populated.
Update again
I've looked into the source-code of flywayDB version 4.0.3: In org.flywaydb.core.internal.command.DbMigrate#migrate the MigrationInfoServiceImpl is created and refreshed. This selects the full schema_versions-Table. But after that step only a single migration script is executed. I would expect migrating all pending scripts instead.
I've opened an issue on this one at github.

The issue is fixed. Today I tested flyway version 4.1.2 and the performance problem is gone. Thanks for the great work at boxfuse!