Develop Reference

Load neo4j dump in test container during integration test run

I am trying to write integration test for neo4j using spring boot. I am using test container. Can anyone help me how to load database dump file to testcontainer?

here's one way to do this (using current Neo4j 5.3). I have a dump file called neo4j.dump created from a local instance and I use withCopyFileToContainer to copy it to the container before it starts.
As I use the community edition in this example, there is no online backup/restore, but only dump/load. So therefor I have to change the startup command. The load needs to happen before Neo4j starts. I can't stop Neo4j in the container because it would stop the container.
Therefor I create a small shell script that executes the desired other command and than delegates to the original entry point.
This script is transferred with file mode 0100555, corresponding to r-xr-xr-x so that it is executable.
Finally, the container is started with the above script as command.
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestInstance;
import org.neo4j.driver.AuthTokens;
import org.neo4j.driver.Driver;
import org.neo4j.driver.GraphDatabase;
import org.testcontainers.containers.Neo4jContainer;
import org.testcontainers.images.builder.Transferable;
import org.testcontainers.utility.MountableFile;
#TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class LoadDumpTest {
Neo4jContainer<?> neo4j;
Driver driver;
#BeforeAll
void initNeo4j() {
neo4j = new Neo4jContainer<>("neo4j:5.3.0")
.withCopyFileToContainer(MountableFile.forClasspathResource("neo4j.dump"),
"/var/lib/neo4j/data/dumps/neo4j.dump")
.withCopyToContainer(Transferable.of("""
#!/bin/bash -eu
/var/lib/neo4j/bin/neo4j-admin database load neo4j
/startup/docker-entrypoint.sh neo4j
""", 0100555), "/startup/load-dump-and-start.sh")
.withCommand("/startup/load-dump-and-start.sh")
.withLogConsumer(f -> System.out.print(f.getUtf8String()));
neo4j.start();
driver = GraphDatabase.driver(neo4j.getBoltUrl(), AuthTokens.basic("neo4j", neo4j.getAdminPassword()));
}
#Test
void dataShouldHaveBeenLoaded() {
try (var session = driver.session()) {
var numNodes = session.run("MATCH (n) RETURN count(n)").single().get(0).asLong();
Assertions.assertTrue(numNodes > 0);
}
}
#AfterAll
void stopNeo4j() {
neo4j.stop();
}
}
Edit:
If you are on a recent enterprise edition, Christophe suggested the following solution on Twitter, which I do personally think it's superior, as it is less hacky.
https://github.com/ikwattro/neo4j-5-testcontainers-restore-backup/blob/main/src/test/java/dev/ikwattro/Neo4j5RestoreBackupExampleTest.java
It makes use of seed URIs for databases. Copying or binding the resource in his example works the same as in mine.

If you're using Neo4j 5, I suggest you make backups instead of dumps. Using backups you can take advantage of the seedUri option when creating a database, meaning you can create a database from an URI pointing to a backup on disk ( or in the neo4j container ). https://neo4j.com/docs/operations-manual/current/clustering/databases/#cluster-seed-uri
Here is an example using Testcontainers
package dev.ikwattro;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestInstance;
import org.neo4j.driver.AuthTokens;
import org.neo4j.driver.Driver;
import org.neo4j.driver.GraphDatabase;
import org.neo4j.driver.Session;
import org.neo4j.driver.SessionConfig;
import org.testcontainers.containers.BindMode;
import org.testcontainers.containers.Neo4jContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import static org.assertj.core.api.Assertions.assertThat;
#TestInstance(TestInstance.Lifecycle.PER_CLASS)
#Testcontainers(disabledWithoutDocker = true)
public class Neo4j5RestoreBackupExampleTest {
#Container
private Neo4jContainer<?> neo4j = new Neo4jContainer<>("neo4j:5.3.0-enterprise")
.withAdminPassword("password")
.withEnv("NEO4J_dbms_memory_heap_max__size", "256M")
.withEnv("NEO4J_dbms_databases_seed__from__uri__providers", "URLConnectionSeedProvider")
.withClasspathResourceMapping("backups", "/backups", BindMode.READ_ONLY)
.withEnv("NEO4J_ACCEPT_LICENSE_AGREEMENT", "yes");
#BeforeAll
void beforeAll() {
neo4j.start();
createDbFromBackup();
}
#Test
void testCreatingDbFromBackup() {
try (Driver driver = GraphDatabase.driver(neo4j.getBoltUrl(), AuthTokens.basic("neo4j", "password"))) {
try (Session session = driver.session(SessionConfig.forDatabase("worldcup22"))) {
var result = session.run("MATCH (n) RETURN count(n) AS c").single().get("c").asLong();
assertThat(result).isPositive();
}
}
}
private void createDbFromBackup() {
try (Driver driver = GraphDatabase.driver(neo4j.getBoltUrl(), AuthTokens.basic("neo4j", "password"))) {
try (Session session = driver.session(SessionConfig.forDatabase("system"))) {
session.run("""
CREATE DATABASE worldcup22 OPTIONS { existingData: "use", seedUri: "file:///backups/world-cup-2022-neo4j.backup"}
""");
}
}
}
}
You can find a working maven project here https://github.com/ikwattro/neo4j-5-testcontainers-restore-backup

How to retrieve the query string in the HiveMetastoreListener?

I am trying to create a listener on the HiveMetastore where i need to retrieve the query submitted to the metastore. Is there any way to retrieve the queryString?
In the MetatstoreListener we get events such as onCreate, onDelete etc.
It is possible to have postHook on the hive but need to have the listener on the metastore so that all the DDL commands are getting catched executed from anywhere
Is there any way to capture the events in the Metastore and apply the same events to another parallel Metastore setup?
context:- I am trying to upgrade the hive from 1.x version to 3.x.x
where the idea is to have the stateless setup of Metastore-service in Kubernetes.
but not sure how much the query syntax is compatible between both versions. So wanted to set hot-hot setup parallelly and monitor the results of queries. So if there is any way on the MetastoreListener to transfer the DDL events from one Metastore to another and execute simultaneously?
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hive.metastore.events.AlterTableEvent;
import org.apache.hadoop.hive.metastore.MetaStoreEventListener;
import org.apache.hadoop.hive.metastore.events.CreateTableEvent;
import org.apache.hadoop.hive.metastore.api.MetaException;
import org.codehaus.jackson.map.ObjectMapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.time.LocalDateTime;
public class HiveMetastoreListener extends MetaStoreEventListener {
private static final Logger LOGGER = LoggerFactory.getLogger(HiveMetastoreListener.class);
private static final ObjectMapper objMapper = new ObjectMapper();
private final DataProducer dataProducer = DataProducer.getInstance();
public HiveMetastoreListener(Configuration config) {
super(config);
}
/**
* Handler for a CreateTable Event
*/
#Override
public void onCreateTable(CreateTableEvent tableEvent) throws MetaException{
super.onCreateTable(tableEvent);
try {
String data = null;
dataProducer.produceToKafka("metastore_topic", LocalDateTime.now().toString(), data);
}catch (Exception e) {
System.out.println("Error:- " + e);
}
}

Extracting Rally Defect Discussion using the Java Rally Rest API

I am attempting to create a simple Java script which will connect to Rally, fetch all of the defects and return the defect details including the discussion as a Java object. The problem here is that the Discussion is returned as what I believe is a collection because only a URL is given. I am stuck on how to return the discussion for the defect as an object within the JSON rather than only another query which would have to be run separately (thousands of times I presume since we have thousands of defects).
Here is my code:
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import com.rallydev.rest.RallyRestApi;
import com.rallydev.rest.request.GetRequest;
import com.rallydev.rest.request.QueryRequest;
import com.rallydev.rest.request.UpdateRequest;
import com.rallydev.rest.response.QueryResponse;
import com.rallydev.rest.util.Fetch;
import com.rallydev.rest.util.QueryFilter;
import com.rallydev.rest.util.Ref;
import org.json.simple.JSONArray;
public class ExtractData{
public static void main(String[] args) throws URISyntaxException, IOException, NumberFormatException
{
RallyRestApi restApi = new RallyRestApi(new URI("https://rally1.rallydev.com"), "apiKeyHere");
restApi.setProxy(URI.create("http://usernameHere:passwordHere0#proxyHere:8080"));
restApi.setApplicationName("QueryExample");
//Will store all of the parsed defect data
JSONArray defectData = new JSONArray();
try{
QueryRequest defects = new QueryRequest("defect");
defects.setFetch(new Fetch("FormattedID","Discussion","Resolution"));
defects.setQueryFilter(new QueryFilter("Resolution","=","Configuration Change"));
defects.setPageSize(5000);
defects.setLimit(5000);
QueryResponse queryResponse = restApi.query(defects);
if(queryResponse.wasSuccessful()){
System.out.println(String.format("\nTotal results: %d",queryResponse.getTotalResultCount()));
for(JsonElement result: queryResponse.getResults()){
JsonObject defect = result.getAsJsonObject();
System.out.println(defect);
}
}else{
System.err.print("The following errors occured: ");
for(String err: queryResponse.getErrors()){
System.err.println("\t+err");
}
}
}finally{
restApi.close();
}
}
}
Here is an example of what I am getting when I attempt this:
{"_rallyAPIMajor":"2","_rallyAPIMinor":"0","_ref":"https://rally1.rallydev.com/slm/webservice/v2.0/defect/30023232168","_refObjectUUID":"cea42323c2f-d276-4078-92cc-6fc32323ae","_objectVersion":"6","_refObjectName":"Example defect name","Discussion":{"_rallyAPIMajor":"2","_rallyAPIMinor":"0","_ref":"https://rally1.rallydev.com/slm/webservice/v2.0/Defect/32323912168/Discussion","_type":"ConversationPost","Count":0},"FormattedID":"DE332322","Resolution":"Configuration Change","Summary":{"Discussion":{"Count":0}},"_type":"Defect"}
As you can see the discussion is being returned as a URL rather than fetching the actual discussion. As this query will be used at runtime I'd prefer the entire object.

Unfortunately there is no way to get all of that data in one request- you'll have to load the Discussion collection for each defect you read. Also of note, the max page size is 2000.
This isn't exactly the same as what you're trying to do, but this example shows loading child stories much like you'd load discussions...
https://github.com/RallyCommunity/rally-java-rest-apps/blob/master/GetChildStories.java#L37

Error Remote Rest API Client jBPM 6.3

I've created remote REST API client but I've got an error whereas it ran properly on jBPM console but REST having problem like this
[main] ERROR org.kie.services.client.api.command.AbstractRemoteCommandObject - Response with status 200 returned.
Exception in thread "main" org.kie.remote.client.api.exception.RemoteApiException: WorkflowRuntimeException thrown with message '[Transport.Performance:207 - :5] -- Exception when trying to evaluate constraint in split ':
org.kie.remote.services.rest.exception.KieRemoteRestOperationException: [Transport.Performance:207 - :5] -- Exception when trying to evaluate constraint in split
this is my code
package org.transportclient;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.jbpm.workflow.instance.WorkflowRuntimeException;
import org.kie.api.runtime.KieSession;
import org.kie.api.runtime.manager.RuntimeEngine;
import org.kie.api.runtime.process.ProcessInstance;
import org.kie.api.task.TaskService;
import org.kie.api.task.model.TaskSummary;
import org.kie.remote.client.api.RemoteRuntimeEngineFactory;
import org.kie.services.client.api.command.RemoteRuntimeException;
import org.kie.services.client.api.command.exception.RemoteApiException;
public class TransportRest {
public static void main(String[] args) throws MalformedURLException {
RuntimeEngine engine = RemoteRuntimeEngineFactory.newRestBuilder()
.addUrl(new URL("http://localhost:8080/jbpm-console"))
.addUserName("krisv").addPassword("krisv")
.addDeploymentId("RnD:transport:1.5.1")
.build();
KieSession ksession = engine.getKieSession();
//TaskService taskService = engine.getTaskService();
// start a new process instance
Map<String, Object> params = new HashMap<String, Object>();
params.put("entry_date", "04-22-2016 00:00:00");
params.put("ne_id", "NY");
params.put("ping_time","900");
System.out.println(params);
ProcessInstance processInstance = ksession.startProcess("Transport.Performance", params);
System.out.println(processInstance.getParentProcessInstanceId());
System.out.println("Start Performance process " + processInstance.getId());
}
}
Anyone can help ?
Thank you

I have never seen that error, but I had problems with jBPM REST API a few weeks ago when I upgraded jBPM version from 6.2 to 6.3. I solved it adding "rest-all" role to all users who need to use REST API (like "krisv" in your case"). Could you try this?
Regards.

IBM Case Manager Connection

I'm stuck at work for like a week now,
can some1 with CM JavaAPI exprience guide me what am I doing wrong?
I try to connect to the server where the Case Manger is installed and start a
session, maybe I'm doing it all wrong but IBM Knowledge Center did not help.

GOT IT!
package *packacge*;
// jars from the the CM different installation folders on the server
import java.util.List;
import java.util.Locale;
import javax.security.auth.Subject;
import com.filenet.api.core.Connection;
import com.filenet.api.core.ObjectStore;
import com.filenet.api.util.UserContext;
import com.ibm.casemgmt.api.CaseType;
import com.ibm.casemgmt.api.DeployedSolution;
import com.ibm.casemgmt.api.context.CaseMgmtContext;
import com.ibm.casemgmt.api.context.P8ConnectionCache;
import com.ibm.casemgmt.api.context.SimpleP8ConnectionCache;
import com.ibm.casemgmt.api.context.SimpleVWSessionCache;
import com.ibm.casemgmt.api.objectref.ObjectStoreReference;
public class CaseMgmtSession {
public static void main(String[] args) {
P8ConnectionCache connCache = new SimpleP8ConnectionCache();
Connection conn = connCache.getP8Connection("http://*ip of the server CM is installed on*/wsi/FNCEWS40MTOM/");
Subject subject = UserContext.createSubject(conn, *user of CM builder admin*, *pass of CM builder admin*, "FileNetP8WSI");
UserContext uc = UserContext.get();
uc.pushSubject(subject);
Locale origLocale = uc.getLocale();
uc.setLocale(Locale.ENGLISH);
CaseMgmtContext origCmctx = CaseMgmtContext.set(new CaseMgmtContext(new SimpleVWSessionCache(), connCache));
try {
// Code that calls the Case Java API or
// directly calls the CE Java API
// checking the connection is working
ObjectStore os = P8Connector.getObjectStore(*some object store name*);
ObjectStoreReference osRef = new ObjectStoreReference(os);
DeployedSolution someSolution = DeployedSolution.fetchInstance(osRef, *some deployed solution name*);
System.out.println(someSolution.getSolutionName());
List<CaseType> caseTypes = someSolution.getCaseTypes();
for(CaseType ct : caseTypes) {
System.out.println(ct.getName());
}
}
finally {
CaseMgmtContext.set(origCmctx);
uc.setLocale(origLocale);
uc.popSubject();
}
}
}
where P8Connector is a class i wrote that returns an objectstore

I dont know which version of Case Manager you are talking about. However, for 5.2.1.x, you will find ample references on IBM's site. For example - here and here.