I'm developing a neo4j procedure in java. I can test it with the custom data below.
#Test
public void commonTargetTest2() {
// This is in a try-block, to make sure we close the driver after the test
try (Driver driver = GraphDatabase.driver(embeddedDatabaseServer.boltURI(), driverConfig);
Session session = driver.session()) {
// And given I have a node in the database
session.run(
"CREATE (n1:Person {name:'n1'}) CREATE (n2:Person {name:'n2'}) CREATE (n3:Person {name:'n3'}) CREATE (n4:Person {name:'n4'}) CREATE (n5:Person {name:'n5'})"
+ "CREATE (n6:Person {name:'n6'}) CREATE (n7:Person {name:'n7'}) CREATE (n8:Person {name:'n8'}) CREATE (n9:Person {name:'n9'}) CREATE (n10:Person {name:'n10'})"
+ "CREATE (n11:Person {name:'n11'}) CREATE (n12:Person {name:'n12'}) CREATE (n13:Person {name:'n13'})"
+ "CREATE (n14:Person {name:'n14'}) CREATE "
+ "(n1)-[:KNOWS]->(n6),(n2)-[:KNOWS]->(n7),(n3)-[:KNOWS]->(n8),(n4)-[:KNOWS]->(n9),(n5)-[:KNOWS]->(n10),"
+ "(n7)-[:KNOWS]->(n11),(n8)-[:KNOWS]->(n12),(n9)-[:KNOWS]->(n13),"
+ "(n11)-[:KNOWS]->(n14),(n12)-[:KNOWS]->(n14),(n13)-[:KNOWS]->(n14);");
// name of the procedure I defined is "p1", below I'm calling it in cypher
StatementResult result = session
.run("CALL p1([1,3], [], 3, 0) YIELD nodes, edges return nodes, edges");
InternalNode n = (InternalNode) result.single().get("nodes").asList().get(0);
assertThat(n.id()).isEqualTo(13);
}
}
This works fine but the data is newly generated with CREATE statements and it is very small. I want to test my procedure with an existing neo4j database server. So that I can see the performance/results of my procedure with real/big data.
I can also achieve that with the below code. I can connect to an up and running neo4j database.
#Test
public void commonTargetTestOnImdb() {
// This is in a try-block, to make sure we close the driver after the test
try (Driver drv = GraphDatabase.driver("bolt://localhost:7687", AuthTokens.basic("neo4j", "123"));
Session session = drv.session()) {
// find 1 common downstream of 3 nodes
StatementResult result = session.run(
"CALL commonStream([1047255, 1049683, 1043696], [], 3, 2) YIELD nodes, edges return nodes, edges");
InternalNode n = (InternalNode) result.single().get("nodes").asList().get(0);
assertThat(n.id()).isEqualTo(5);
}
}
NOW, my problem is that I can't debug the codes of my procedure if I connect to an existing database. I package a JAR file and put it inside plugin folder of my neo4j database so that neo4j can call my procedure. I think I should debug the JAR file. I'm using vscode and java extensions to debug and run tests. How can I debug JAR file with vscode?
For the record, I find a way to debug my neo4j stored procedure. I'm using Java 8. I used IntelliJ idea. I added the config dbms.jvm.additional=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 to the neo4j.conf file
Inside IntelliJ IDEA, I added a new configuration for remote debugging.
Here note that syntax in Java 9+ is different. At the end, to give port parameter it uses address=*:5005. There is post about it here. https://stackoverflow.com/a/62754503/3209523
Related
I have been trying to call a java jar file from DB2 stored procedure as external jar.
Below is my external jar file code.
package connection;
public class Connect {
static Connection con = null;
static Statement stmt = null;
public static void main(String[] args) throws SQLException {
javastp("v1");
}
public static void javastp( String name) throws SQLException{
try {
con = DriverManager.getConnection( "jdbc:default:connection" );
stmt = con.createStatement();
stmt.executeUpdate("INSERT INTO SCHEMA_NAME.TEST(NAME, FLAG) VALUES ('"+name+"',true) ");
}catch (Exception e) {
// ...
}finally {
//Close open resources
if (stmt != null) stmt.close();
if (con != null) con.close();
}
}
}
i install it using below command in db2 server
db2 call sqlj.install_jar('file:E:/jarpath../jarname.jar','jarname',0);
Below is the stored procedure for DB2 external jar.
CREATE OR REPLACE PROCEDURE SCHEMA_NAME.PROC6()
LANGUAGE java
PARAMETER STYLE java
FENCED
EXTERNAL NAME 'jarname:connection.Connect.main'
The stored procedure executes successfully without errors.
But when i try to call the stored procedure using below,
CALL SCHEMA_NAME.PROC6()
i get the below error
SQL Error [38503]: A stored procedure process has been terminated abnormally. Routine name: "SCHEMA_NAME.PROC6". Specific name: "SQL201016144125554".. SQLCODE=-1131, SQLSTATE=38503, DRIVER=4.8.86
The java jar works fine when executed directly in the command prompt and the records are being inserted into the table, but i cant seem to figure out why the external stored procedure gives me error when i try to call it that way.
Can somebody here please help me out as am stuck on this since a very long time and i have no background of DB2.
the db2level command gives the below output:
DB21085I This instance or install (instance name, where applicable: "DB2")
uses "64" bits and DB2 code release "SQL10058" with level identifier
"0609010E".
Informational tokens are "DB2 v10.5.800.381", "s160901", "IP24000", and Fix
Pack "8".
Product is installed at "C:\PROGRA~1\IBM\SQLLIB" with DB2 Copy Name "DB2COPY1".
Linux Suse db2 version 11.1.2
Database server = DB2/LINUXX8664 11.1.2
The log is as below:
2020-10-20-09.24.56.746633+330 I4952989679E1230 LEVEL: Error
PID : 31693 TID : 140179352315648 PROC : db2sysc 0
INSTANCE: NODE : 000 DB :
APPHDL : APPID:
AUTHID : HOSTNAME:
EDUID : 2768 EDUNAME: db2agent () 0
FUNCTION: DB2 UDB, routine_infrastructure, sqlerWaitForFencedInvocation, probe:12115
MESSAGE : ZRC=0xFFFFFB38=-1224
SQL1224N The database manager is not able to accept new requests,
has terminated all requests in progress, or has terminated the
specified request because of an error or a forced interrupt.
Error SQL1131N has the following description
The DB2 architecture is designed so that applications run in a different address space than the database server. Running applications in a different address space prevents application programming errors from overwriting database manager internal buffers or files, and prevents application errors from crashing the database manager. The fenced mode process (db2fmp) is responsible for executing fenced stored procedures and user-defined functions in a different address space than the database server.
This message is returned when the db2fmp process terminates abnormally while running the named routine. The db2fmp process could have terminated abnormally for many reasons, including the following reasons:
There was a coding error, such as division by zero or an out-of-bounds pointer reference, in the implementation of the stored procedure or user-defined function that the db2fmp process was executing.
Another process terminated the db2fmp process using a signal, such as the termination signal SIGTERM.
SQLJ.INSTALL_JAR failed while installing a Java routine because the fenced user does not have permission to create or write to the necessary directories on the server.
Db2 is unforgiving when you make mistakes with external non-SQL code, in this case, it just throws an exception and expects you to sort it out.
Only use java for Db2 stored procedures when there is a compelling reason to NOT use SQL PL procedures.
You have these mistakes
You cannot use a main() for a Db2 java stored procedure entry point
The stored procedure entry point should be in your case the javastp() method
The EXTERNAL NAME clause in the DDL must not reference main() but should use javastp instead
The parameters in the create procedure line must match those of the method
in number and type and sequence. (In your case that is (IN name varchar(255) or similar).
In pig, you can pass a configuration from your pig script to pig UDF via UDFContext. For example,
// in pig script
SET my.conf dummy-conf
// in UDF java code
Configuration conf = UDFContext.getUDFContext().getJobConf();
String myConf = conf.get("my.conf");
So, is there a similar way to pass configuration from a hive script to a hive UDF? For example, if I have set MY_CONF='foobar' in a hive script, how can I retrieve that in a java UDF, which needs to consume the value of MY_CONF?
Instead of extending UDF class, you can try subclassing GenericUDF. This class has the following method you can override:
/**
* Additionally setup GenericUDF with MapredContext before initializing.
* This is only called in runtime of MapRedTask.
*
* #param context context
*/
public void configure(MapredContext context) {
}
MapredContext has a method just like UDFContext from Pig to retrieve the Job configuration. So you could just do the following:
#Override
public void configure(MapredContext context) {
Configuration conf = context.getJobConf();
}
As of hive 1.2 there are 2 approaches.
1. Overriding configure method from GenericUDF
#Override
public void configure(MapredContext context) {
super.configure(context);
someProp = context.getJobConf().get(HIVE_PROPERTY_NAME);
}
Above(1) won't work in all the cases. Works only in MapredContext.
Every query has to be force map/reduce jobs, to do that set
set hive.fetch.task.conversion=minimal/none;
set hive.optimize.constant.propagation=false;
.
with above properties set, you will hit major performance problems, especially for smaller queries.
2. Using SessionState
SessionState ss = SessionState.get();
if (ss != null) {
this.hiveConf = ss.getConf();
someProp = this.hiveConf.get(HIVE_PROPERTY_NAME);
LOG.info("Got someProp: " + someProp);
}
Go to hive command line
hive> set MY_CONF='foobar';
your variable should be listed when hitting the command
hive> set;
Now, consider you have below
Jar: MyUDF.jar
UDF calss: MySampleUDF.java which accepts a String value.
Table: employee
hive> ADD JAR /MyUDF.jar
hive> CREATE TEMPORARY FUNCTION testUDF AS 'youpackage.MySampleUDF';
hive> SELECT testUDF(${MY_CONF}) from employee;
there are lots of example, shared , so you can find all required details over google :).
A Small Example which was describe in shared link:
hive> ADD JAR assembled.jar;
hive> create temporary function hello as 'com.test.example.UDFExample';
hive> select hello(firstname) from people limit 10;
Please check link for reference which I normally Used to:
Link1
Link2
How to handle DB inside of your script using TestNG framework?
How to delete DB before each run of test script?
How to load sql file into clean DB before running test script?
Goal: Each test case must be independent
Framework: TestNG
Language: Java
Each test case must be independent against other test cases. Goal is to run Test cases randomly, no order is required.
Previously I have used PHPUnit framework where each test case was independent.
Before running each test script, I would:
dropdatabase
create a new database
load sql file into database with initial data
I was using this inside of shell script, and I would call shell script via command line:
mysql -u$DB_USER -p$DB_PWD -h$HOST -e "DROP DATABASE $DB_NAME"
mysql -u$DB_USER -p$DB_PWD -h$HOST -e "CREATE DATABASE $DB_NAME"
mysql -u$DB_USER -p$DB_PWD -h$HOST $DB_NAME < sql/dbinit.sql
Google-ing was not helpfull, therefore I am posting question here. I would need something like this for TestNG but I have not found anything similiar.
Could someone give advice to fellow QA.
How do you handle with oracle database, how do you delete data from the DB and load them inside of your test script?
Any advice, book, tutorial would be very helpfull.
You should look into DbUnit. I've recently started using it myself (I'm a TestNG user) and, as of yet, I haven't come across scenarios where you absolutely need JUnit itself. You can restore your database between tests, populate it, export it, etc ...
http://dbunit.sourceforge.net/
basic example:
private DatabaseHelper dbh;
private EntityManager em;
private IDatabaseConnection connection;
private IDataSet dataset;
#BeforeClass
private void setupDatabaseResource() throws Exception {
// using JPA and a custom helper class
em = dbh.getEntityManager();
connection = new DatabaseConnection(((SessionImpl) (em.getDelegate())).connection());
connection.getConfig().setProperty(DatabaseConfig.PROPERTY_DATATYPE_FACTORY, new HsqldbDataTypeFactory());
// full database export
IDataSet fullDataSet = connection.createDataSet();
FlatXmlDataSet.write(fullDataSet, new FileOutputStream("target/generated-sources/test-dataset.xml"));
FlatXmlDataSetBuilder flatXmlDataSetBuilder = new FlatXmlDataSetBuilder();
flatXmlDataSetBuilder.setColumnSensing(true);
// keep the dataset in memory, for later restore points
dataset = flatXmlDataSetBuilder.build(new FileInputStream("target/generated-sources/test-dataset.xml"));
}
edit: an example where #BeforeMethod restores your database between tests
#BeforeMethod
public void cleanDB() throws Exception {
DatabaseOperation.CLEAN_INSERT.execute(connection, dataset);
}
I want to create an embedded H2 database in my simple java project in Eclipse. How do I do this programatically and package the db inside my code ? I tried a SO post for this
and got an error in my code.
Code -
public static void main(String[]args){
JdbcDataSource ds = new JdbcDataSource();
ds.setURL("jdbc:h2:˜/test");
ds.setUser("sa");
ds.setPassword("sa");
try {
Connection conn = ds.getConnection();
} catch (SQLException e) {
e.printStackTrace();
}
}
Error -
org.h2.jdbc.JdbcSQLException: A file path that is implicitly relative to the
current working directory is not allowed in the database URL "jdbc:h2:˜/test".
Use an absolute path, ~/name, ./name, or the baseDir setting instead. [90011-181]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.engine.ConnectionInfo.getName(ConnectionInfo.java:398)
at org.h2.engine.Engine.openSession(Engine.java:45)
at org.h2.engine.Engine.openSession(Engine.java:167)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:145)
at org.h2.engine.Engine.createSession(Engine.java:128)
at org.h2.engine.Engine.createSession(Engine.java:26)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:347)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:108)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)
at org.h2.Driver.connect(Driver.java:72)
at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:190)
at org.h2.jdbcx.JdbcDataSource.getConnection(JdbcDataSource.java:161)
at MyCode.main(MyCode.java:8)
I saw this link - https://groups.google.com/forum/#!msg/h2-database/SlSwte0DLSU/eWj0UaejdkEJ and Where are my H2 database files?. Its not clear how I can get the exact path to test database on my windows pc.
How do I first access the test database and then create another database inside my java project ?
Thank you.
You have used the wrong character. You need to use ~ (tilde) and you have use ˜ (I don't know what it is, but it's not a tilde).
The location of the H2 files is very nicely documented. To view the contents, execute the h2.jar. It is not only the driver, but also an executable that will start a web-based applications for DB management.
I need to get a list of chunks after sharding inside my Java code. My code is simple and looks like this:
Mongo m = new Mongo( "localhost" , 27017 );
DB db = m.getDB( "admin" );
Object cr = db.eval("db.printShardingStatus()", 1);
A call of eval() returns an error:
Exception in thread "main" com.mongodb.CommandResult$CommandFailure: command failed [$eval]: { "serverUsed" : "localhost/127.0.0.1:27017" , "errno" : -3.0 , "errmsg" : "invoke failed: JS Error: ReferenceError: printShardingStatus is not defined src/mongo/shell/db.js:891" , "ok" : 0.0}
at com.mongodb.CommandResult.getException(CommandResult.java:88)
at com.mongodb.CommandResult.throwOnError(CommandResult.java:134)
at com.mongodb.DB.eval(DB.java:340)
at org.sm.mongodb.MongoTest.main(MongoTest.java:35)
And, really, if we look into the code of db.js, in line 891 there is a call to a method printShardingStatus() that is not defined inside a file. Inside of sh.status() method in utils_sh.js file, there is even a comment:
// TODO: move the actual commadn here
Important to mention, when I run these commands in mongo command line, everything works properly!
My questions are:
Is there any other possibility of getting a full sharding status within Java code? (eg. with DB.command() method)
If not, any other suggestions how to avoid my problem?
Many of the shell's helper functions are not available for server-side code execution. In the case of printShardingStatus(), it makes sense because there isn't a console to use for printing output and you'd rather have a string returned. Thankfully, you should be able to pull up the source of the shell function and reimplement it in your application (e.g. concatenating a returned string instead of printing directly).
$ mongo
MongoDB shell version: 2.2.0
connecting to: test
> db.printShardingStatus
function (verbose) {
printShardingStatus(this.getSiblingDB("config"), verbose);
}
So, let's look at the printShardingStatus() function...
> printShardingStatus
function (configDB, verbose) {
if (configDB === undefined) {
configDB = db.getSisterDB("config");
}
var version = configDB.getCollection("version").findOne();
// ...
}
Before turning all of the output statements into string concatenation, you'd want to make sure the other DB methods are all available to you. Performance-wise, I think the best option is to port the innards of this function to Java and avoid server-side JS evaluation altogether. If you dive deeper into the printShardingStatus() function, you'll see it's just issuing find() on the config database along with some group() queries.
If you do want to stick with evaluating JS and would rather not keep this code within your Java application, you can also look into storing JS functions server-side.
Have you deployed a shard cluster properly?
If so, you could connect to a mongo database that has sharding enabled.
Try calling the method db.printShardingStatus() with a that database within the mongo shell and see what happens.
Apparently the Javascript function 'printShardingStatus' is only available for the mongo shell and not for execution with server commands, to see the code start mongo.exe and type only 'printShardingStatus' and press enter.
In this case writing an extension method would be the best for solving this...
Javascript way of printing output of MongoDB query to a file
1] create a javascript file
test.js
cursor = db.printShardingStatus();
while(cursor.hasNext()){
printjson(cursor.next());
}
2] run
mongo admin --quiet test.js > output.txt