Trying to use a similar example from the sample code found here
My sample function is:
void query()
{
String nodeResult = "";
String rows = "";
String resultString;
String columnsString;
System.out.println("In query");
// START SNIPPET: execute
ExecutionEngine engine = new ExecutionEngine( graphDb );
ExecutionResult result;
try ( Transaction ignored = graphDb.beginTx() )
{
result = engine.execute( "start n=node(*) where n.Name =~ '.*79.*' return n, n.Name" );
// END SNIPPET: execute
// START SNIPPET: items
Iterator<Node> n_column = result.columnAs( "n" );
for ( Node node : IteratorUtil.asIterable( n_column ) )
{
// note: we're grabbing the name property from the node,
// not from the n.name in this case.
nodeResult = node + ": " + node.getProperty( "Name" );
System.out.println("In for loop");
System.out.println(nodeResult);
}
// END SNIPPET: items
// START SNIPPET: columns
List<String> columns = result.columns();
// END SNIPPET: columns
// the result is now empty, get a new one
result = engine.execute( "start n=node(*) where n.Name =~ '.*79.*' return n, n.Name" );
// START SNIPPET: rows
for ( Map<String, Object> row : result )
{
for ( Entry<String, Object> column : row.entrySet() )
{
rows += column.getKey() + ": " + column.getValue() + "; ";
System.out.println("nested");
}
rows += "\n";
}
// END SNIPPET: rows
resultString = engine.execute( "start n=node(*) where n.Name =~ '.*79.*' return n.Name" ).dumpToString();
columnsString = columns.toString();
System.out.println(rows);
System.out.println(resultString);
System.out.println(columnsString);
System.out.println("leaving");
}
}
When I run this in the web console I get many results (as there are multiple nodes that have an attribute of Name that contains the pattern 79. Yet running this code returns no results. The debug print statements 'in loop' and 'nested' never print either. Thus this must mean there are not results found in the Iterator, yet that doesn't make sense.
And yes, I already checked and made sure that the graphDb variable is the same as the path for the web console. I have other code earlier that uses the same variable to write to the database.
EDIT - More info
If I place the contents of query in the same function that creates my data, I get the correct results. If I run the query by itself it returns nothing. It's almost as the query works only in the instance where I add the data and not if I come back to the database cold in a separate instance.
EDIT2 -
Here is a snippet of code that shows the bigger context of how it is being called and sharing the same DBHandle
package ContextEngine;
import ContextEngine.NeoHandle;
import java.util.LinkedList;
/*
* Class to handle streaming data from any coded source
*/
public class Streamer {
private NeoHandle myHandle;
private String contextType;
Streamer()
{
}
public void openStream(String contextType)
{
myHandle = new NeoHandle();
myHandle.createDb();
}
public void streamInput(String dataLine)
{
Context context = new Context();
/*
* get database instance
* write to database
* check for errors
* report errors & success
*/
System.out.println(dataLine);
//apply rules to data (make ContextRules do this, send type and string of data)
ContextRules contextRules = new ContextRules();
context = contextRules.processContextRules("Calls", dataLine);
//write data (using linked list from contextRules)
NeoProcessor processor = new NeoProcessor(myHandle);
processor.processContextData(context);
}
public void runQuery()
{
NeoProcessor processor = new NeoProcessor(myHandle);
processor.query();
}
public void closeStream()
{
/*
* close database instance
*/
myHandle.shutDown();
}
}
Now, if I call streamInput AND query in in the same instance (parent calls) the query returns results. If I only call query and do not enter ANY data in that instance (yet web console shows data for same query) I get nothing. Why would I have to create the Nodes and enter them into the database at runtime just to return a valid query. Shouldn't I ALWAYS get the same results with such a query?
You mention that you are using the Neo4j Browser, which comes with Neo4j. However, the example you posted is for Neo4j Embedded, which is the in-process version of Neo4j. Are you sure you are talking to the same database when you try your query in the Browser?
In order to talk to Neo4j Server from Java, I'd recommend looking at the Neo4j JDBC driver, which has good support for connecting to the Neo4j server from Java.
http://www.neo4j.org/develop/tools/jdbc
You can set up a simple connection by adding the Neo4j JDBC jar to your classpath, available here: https://github.com/neo4j-contrib/neo4j-jdbc/releases Then just use Neo4j as any JDBC driver:
Connection conn = DriverManager.getConnection("jdbc:neo4j://localhost:7474/");
ResultSet rs = conn.executeQuery("start n=node({id}) return id(n) as id", map("id", id));
while(rs.next()) {
System.out.println(rs.getLong("id"));
}
Refer to the JDBC documentation for more advanced usage.
To answer your question on why the data is not durably stored, it may be one of many reasons. I would attempt to incrementally scale back the complexity of the code to try and locate the culprit. For instance, until you've found your problem, do these one at a time:
Instead of looping through the result, print it using System.out.println(result.dumpToString());
Instead of the regex query, try just MATCH (n) RETURN n, to return all data in the database
Make sure the data you are seeing in the browser is not "old" data inserted earlier on, but really is an insert from your latest run of the Java program. You can verify this by deleting the data via the browser before running the Java program using MATCH (n) OPTIONAL MATCH (n)-[r]->() DELETE n,r;
Make sure you are actually working against the same database directories. You can verify this by leaving the server running. If you can still start your java program, unless your Java program is using the Neo4j REST Bindings, you are not using the same directory. Two Neo4j databases cannot run against the same database directory simultaneously.
Related
I have a stored procedure in Cosmos DB Emulator. All this procedure does is: delete ALL documents from mycollection. When I run it in browser (https://localhost:8081/_explorer/index.html), it works great. Then I try to call it from Java:
RequestOptions requestOptions = new RequestOptions();
requestOptions.setPartitionKey(new PartitionKey(null));
System.out.println("START DELETE PROCEDURE");
StoredProcedureResponse spr = client.executeStoredProcedure(sprocLink, requestOptions, null);
System.out.println(spr.getResponseAsString());
And get the following result: {"deleted": 0,"continuation": false}
This is CRAZY. I'm running this stored procedure from the browser and getting this result: {"deleted": 10,"continuation": false}. Then (of course adding back those 10 documents) running this result from Java and getting this result: {"deleted": 0,"continuation": false}
So when the stored procedure is ran by Java, it is called but didn't do the job. Deleted nothing.... Why would this happen?
Below is the stored procedure
function testStoredProcedure( ) {
var collection = getContext().getCollection();
var collectionLink = collection.getSelfLink();
var response = getContext().getResponse();
var responseBody = {
deleted: 0,
continuation: true
};
var query = 'SELECT * FROM mycollection ';
// Validate input.
if (!query) throw new Error("The query is undefined or null.");
tryQueryAndDelete();
// Recursively runs the query w/ support for continuation tokens.
// Calls tryDelete(documents) as soon as the query returns documents.
function tryQueryAndDelete(continuation)) {
var requestOptions = {continuation: continuation};
var isAccepted = collection.queryDocuments(collectionLink, query, requestOptions, function (err, retrievedDocs, responseOptions) {
if (err) throw err;
if (retrievedDocs.length > 0) {
// Begin deleting documents as soon as documents are returned form the query results.
// tryDelete() resumes querying after deleting; no need to page through continuation tokens.
// - this is to prioritize writes over reads given timeout constraints.
tryDelete(retrievedDocs);
} else if (responseOptions.continuation) {
// Else if the query came back empty, but with a continuation token; repeat the query w/ the token.
tryQueryAndDelete(responseOptions.continuation);
} else {
// Else if there are no more documents and no continuation token - we are finished deleting documents.
responseBody.continuation = false;
response.setBody(responseBody);
}
});
// If we hit execution bounds - return continuation: true.
if (!isAccepted) {
response.setBody(responseBody);
}
}
// Recursively deletes documents passed in as an array argument.
// Attempts to query for more on empty array.
function tryDelete(documents) {
if (documents.length > 0) {
// Delete the first document in the array.
var isAccepted = collection.deleteDocument(documents[0]._self, {}, function (err, responseOptions) {
if (err) throw err;
responseBody.deleted++;
documents.shift();
// Delete the next document in the array.
tryDelete(documents);
});
// If we hit execution bounds - return continuation: true.
if (!isAccepted) {
response.setBody(responseBody);
}
} else {
// If the document array is empty, query for more documents.
tryQueryAndDelete();
}
}
}
For partitioned containers, when executing a stored procedure, a partition key value must be provided in the request options. Stored procedures are always scoped to a partition key. Items that have a different partition key value will not be visible to the stored procedure. This also applied to triggers as well.
You are setting partition key to "null" in requestOptions. "null" is a valid partition key value. Looks like "null" is not a partition key value for your 10 documents.
Humbly reposting #Jay Gong answer How to specify NONE partition key for deleting a document in Document DB java SDK?
Maybe it will help someone. Put:
PartitionKey partitionKey = new PartitionKey(Undefined.Value());
Im trying to parse a pipe delimited file and insert fields into a table. when i start the application nothing happens in my DB. My DB has 4 columns (account_name, command_name, and system_name, CreateDt). The file i am parsing has the date in the first row then extra data. The rows following i only need the first 3 fields in each the rest is extra data. the last row is the row count. i skipped the inserting date because for now but want to get back to it after at least able to insert the first 3 fields. I have little experience with parsing a file and storing data in a DB and have looked through jdbc examples to get to this point but im struggling and am sure there is a better way.
File Example
20200310|extra|extra|extra||
Mn1223|01192|windows|extra|extra|extra||
Sd1223|02390|linux|extra|extra|extra||
2
table format
account_name command_name system_name createDt
Mn1223 01192 windows 20200310
Sd1223 02390 linux 20200310
Code to parse and insert into DB
public List insertZygateData (List<ZygateEntity> parseData) throws Exception {
String filePath = "C:\\DEV\\Test_file.xlsx";
List<String> lines = Files.readAllLines(Paths.get(filePath));
// remove date and amount
lines.remove(0);
lines.remove(lines.size() - 1);
for (ZygateEntity zygateInfo : parseData){
new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
.addValue("command_name", zygateInfo.getCommandName())
.addValue("system_name", zygateInfo.getSystemName())
.getValues();
}
return lines.stream()
.map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());
}
public boolean cleantheTable() throws SQLException {
String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
"VALUES (:account_name,:command_name,:system_name)";
boolean truncated = false;
Statement stmt = null;
try {
String sqlTruncate = "truncate table Landing.midrange_xygate_load";
jdbcTemplate.execute(sqlTruncate);
truncated = true;
} catch (Exception e) {
e.printStackTrace();
truncated = false;
return truncated;
} finally {
if (stmt != null) {
jdbcTemplate.execute(sql);
stmt.close();
}
}
log.info("Clean the table return value :" + truncated);
return truncated;
}
}
Entity/Model
public ZygateEntity(String accountName, String commandName, String systemName){
this.accountName=accountName;
this.commandName=commandName;
this.systemName=systemName;
}
//getters and setters
#Override
public String toString() {
return "ZygateEntity [accountName=" + accountName + ", commandName=" + commandName + ", systemName=" + systemName + ", createDt=" + createDt +"]";
}
}
Taking a look at what you've provided, it seems you have a jumbled collection of bits of code, and while most of it is there, it's not all there and not quite all in the right order.
To get some kind of clarity, try to break down what it is you're doing into separate steps, and have a method that focuses on each step. In particular, you write
Im trying to parse a pipe delimited file and insert fields into a table
This naturally breaks down into two parts:
parsing the pipe-delimited file, and
inserting fields into a table.
For the first part, you seem to have most of the parts already in your insertZygateData method. In particular, this line reads all the lines of a file into a list:
List<String> lines = Files.readAllLines(Paths.get(filePath));
These lines then remove the first and last lines from the list of lines read:
// remove date and amount
lines.remove(0);
lines.remove(lines.size() - 1);
You then have some code that looks a bit out of place: this seems to be something to do with inserting into the database, but we haven't created our list of ZygateEntity objects as we haven't yet finished reading the file. Let's put this for loop to one side for the moment.
Finally, we take the list of lines we read, split them using pipes, create ZygateEntity objects from the parts and create a List of these objects, which we then return.
return lines.stream()
.map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());
Putting this lot together, we have a useful method that parses the file, completing the first part of the task:
private List<ZygateEntity> parseZygateData() throws IOException {
String filePath = "C:\\DEV\\Test_file.xlsx";
List<String> lines = Files.readAllLines(Paths.get(filePath));
// remove date and amount
lines.remove(0);
lines.remove(lines.size() - 1);
return lines.stream()
.map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());
}
(Of course, we could add a parameter for the file path to read, but in the interest of getting something working, it's OK to stick with the current hard-coded file path.)
So, we've got our list of ZygateEntity objects. How do we write a method to insert them into the database?
We can find a couple of the ingredients we need in your code sample. First, we need the SQL statement to insert the data. This is in your cleanThetable method:
String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
"VALUES (:account_name,:command_name,:system_name)";
We then have this loop:
for (ZygateEntity zygateInfo : parseData){
new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
.addValue("command_name", zygateInfo.getCommandName())
.addValue("system_name", zygateInfo.getSystemName())
.getValues();
}
This loop creates a MapSqlParameterSource out of each ZygateEntity object, and then converts it to a Map<String, Object> by calling the getValues() method. But then it does nothing with this value. Effectively you're creating these objects and getting rid of them again without doing anything with them. This isn't ideal.
A MapSqlParameterSource is used with a Spring NamedParameterJdbcTemplate. Your code mentions a jdbcTemplate, which appears to be a field within the class that parses data and inserts into the database, but you don't show the full code of this class. I'm going to have to assume it's a NamedParameterJdbcTemplate rather than a 'plain' JdbcTemplate.
A NamedParameterJdbcTemplate contains a method update that takes a SQL string and a SqlParameterSource. We have a SQL string, and we're creating MapSqlParameterSource objects, so we can use these to carry out the insert. There's not a lot of point in creating one of these MapSqlParameterSource objects only to convert it to a map, so let's remove the call to getValues().
So, we now have a method to insert the data into the database:
public void insertZygateData(List<ZygateEntity> parseData) {
String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
"VALUES (:account_name,:command_name,:system_name)";
for (ZygateEntity zygateInfo : parseData){
SqlParameterSource source = new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
.addValue("command_name", zygateInfo.getCommandName())
.addValue("system_name", zygateInfo.getSystemName());
jdbcTemplate.update(sql, source);
}
}
Finally, let's take a look at your cleanThetable method. As with the others, let's keep it focused on one task: it looks like at the moment you're trying to delete the data out of the table and then insert it in the same method, but let's have it just focus on deleting the data as we've now got a method to insert the data.
We can't immediately get rid of the String sql = ... line, because the finally block in your code uses it. If stmt is not null, then you attempt to run the INSERT statement and then close stmt.
However, stmt is never assigned any value other than null, so it remains null. stmt != null is therefore always false, so the INSERT statement never runs. Your finally block never does anything, so you would be best off removing it altogether. With your finally block gone, you can also get rid of your local variable stmt and the sql string, leaving us with a method whose focus is to truncate the table:
public boolean cleantheTable() throws SQLException {
boolean truncated = false;
try {
String sqlTruncate = "truncate table Landing.midrange_xygate_load";
jdbcTemplate.execute(sqlTruncate);
truncated = true;
} catch (Exception e) {
e.printStackTrace();
truncated = false;
return truncated;
}
log.info("Clean the table return value :" + truncated);
return truncated;
}
I'll leave it up to you to write the code that calls these methods. I wrote some code for this purpose, and it ran successfully and inserted into a database.
So, in summary, no data was being written to your database because you were never making a call to the database to insert any. In your insertZygateData method you were creating the parameter-source objects but not doing anything useful with them, and in your cleanThetable method, it looked like you were trying to insert data, but your line jdbcTemplate.execute(sql) that attempted to do this never ran. Even if stmt wasn't null, this line wouldn't work as you didn't pass the parameter values in anywhere: you would get an exception from the database as it would be expecting values for the parameters but you never gave it any.
Hopefully my explanation gives you a way of getting your code working and helps you understand why it wasn't.
I am writing Java Code to get data from SAP BAPI using Java Connector (JCo). This is my first time to make a connection to SAP using JCo. I was able to get the Tables available in the Data Source and also get one particular Table and Number of Columns using table_name.getNumColumns() which gives me the total count of columns. But when I do, table_name.getNumRows(), it says 0. Where as in my Data source, there are around 85 Rows. How can I get the rows from this table?
The code I have been using:
public class SapConnection {
public static void gettingTableData(JCoFunction function) {
JCoParameterList table_list = function.getTableParameterList();
JCoTable my_table = function.getTableParameterList().getTable("SOME_TABLE");
System.out.println("Field Count: "+my_table.getFieldCount());
// This is not working as Number of Rows is 0.
for(int i = 0; i<my_table.getNumRows(); i++, my_table.nextRow()) {
// get those rows and do something ..
}
System.out.println("Is Empty: "+my_table.isEmpty()); // returns True
System.out.println("Is First Row: "+my_table.isFirstRow()); // returns false
System.out.println("Next Row: "+my_table.nextRow()); // returns false
System.out.println("Num Rows: "+my_table.getNumRows()); // returning 0
}
public static void loadDataSourceAndGetData(JCoDestination dest) throws JCoException {
JCoRepository sapRepository = dest.getRepository();
JCoFunctionTemplate template =
sapRepository.getFunctionTemplate("DATA_SOURCE_NAME");
JCoFunction my_function = template.getFunction();
gettingTableData(my_function);
}
public static void main(String[] args) throws JCoException {
// get the Properties created for connection.
Properties pp = getJcoProperties();
PropertiesDestinationDataProvider pddp = new PropertiesDestinationDataProvider(pp);
Environment.registerDestinationDataProvider(pddp);
JCoDestination dest = getDestination();
try {
// Using JCo Context for stateful function calls to Start() and End()
JCoContext.begin(dest);
loadDataSourceAndGetData(dest);
JCoRepository sapRepository = dest.getRepository();
System.out.println(sapRepository.getMonitor().getLastAccessTimestamp());
} finally {
// end the connection.
JCoContext.end(dest);
}
}
}
If you would like to get some data from a SAP BAPI it would help a lot also to call this BAPI. The data doesn't materialize automatically in the JCo objects out of thin air.
In your code you do not execute any JCoFunction.
Set the mandatory import parameter values for this BAPI (if there are any), execute the BAPI (your JCoFunction object) and then you will get the export data from the SAP system in response which will then also add appropriate rows to the JCoTable object.
I'm trying to read text files .txt with more than 10.000 lines per file, splitting them and inserting the data in Access database using Java and UCanAccess. The problem is that it becomes slower and slower every time (as the database gets bigger).
Now after reading 7 text files and inserting them into database, it would take the project more than 20 minutes to read another file.
I tried to do just the reading and it works fine, so the problem is the actual inserting into database.
N.B: This is my first time using UCanAccess with Java because I found that the JDBC-ODBC Bridge is no longer available. Any suggestions for an alternative solution would also be appreciated.
If your current task is simply to import a large amount of data from text files straight into the database, and it does not require any sophisticated SQL manipulations, then you might consider using the Jackcess API directly. For example, to import a CSV file you could do something like this:
String csvFileSpec = "C:/Users/Gord/Desktop/BookData.csv";
String dbFileSpec = "C:/Users/Public/JackcessTest.accdb";
String tableName = "Book";
try (Database db = new DatabaseBuilder()
.setFile(new File(dbFileSpec))
.setAutoSync(false)
.open()) {
new ImportUtil.Builder(db, tableName)
.setDelimiter(",")
.setUseExistingTable(true)
.setHeader(false)
.importFile(new File(csvFileSpec));
// this is a try-with-resources block,
// so db.close() happens automatically
}
Or, if you need to manually parse each line of input, insert a row, and retrieve the AutoNumber value for the new row, then the code would be more like this:
String dbFileSpec = "C:/Users/Public/JackcessTest.accdb";
String tableName = "Book";
try (Database db = new DatabaseBuilder()
.setFile(new File(dbFileSpec))
.setAutoSync(false)
.open()) {
// sample data (e.g., from parsing of an input line)
String title = "So, Anyway";
String author = "Cleese, John";
Table tbl = db.getTable(tableName);
Object[] rowData = tbl.addRow(Column.AUTO_NUMBER, title, author);
int newId = (int)rowData[0]; // retrieve generated AutoNumber
System.out.printf("row inserted with ID = %d%n", newId);
// this is a try-with-resources block,
// so db.close() happens automatically
}
To update an existing row based on its primary key, the code would be
Table tbl = db.getTable(tableName);
Row row = CursorBuilder.findRowByPrimaryKey(tbl, 3); // i.e., ID = 3
if (row != null) {
// Note: column names are case-sensitive
row.put("Title", "The New Title For This Book");
tbl.updateRow(row);
}
Note that for maximum speed I used .setAutoSync(false) when opening the Database, but bear in mind that disabling AutoSync does increase the chance of leaving the Access database file in a damaged (and possibly unusable) state if the application terminates abnormally while performing the updates.
Also, if you need to use slq/ucanaccess, you have to call setAutocommit(false) on the connection at the begin, and do a commit each 200/300 record. The performances will improve drammatically (about 99%).
I'm using the neo4j 1.9.M01 version with the java-rest-binding 1.8.M07, and I have a problem with this code that aims to get a node from a neo4j database with the property "URL" that is "ARREL", using the Query language via rest. The problems seems to happens only inside a transaction, throwing an exception, but otherwise works well :
RestGraphDatabase graphDb = new RestGraphDatabase("http://localhost:7474/db/data");
RestCypherQueryEngine queryEngine = new RestCypherQueryEngine(graphDb.getRestAPI());
Node nodearrel = null;
Transaction tx0 = gds.beginTx();
try{
final String queryStringarrel = ("START n=node(*) WHERE n.URL =~{URL} RETURN n");
QueryResult<Map<String, Object>> retornar = queryEngine.query(queryStringarrel, MapUtil.map("URL","ARREL"));
for (Map<String,Object> row : retornar)
{
nodearrel = (Node)row.get("n");
System.out.println("Arrel: "+nodearrel.getProperty("URL")+" id : "+nodearrel.getId());
}
tx0.success();
}
(...)
But an exception happens: *exception tx0: Error reading as JSON ''
* every execution at the line that returns the QueryResult object.
I also have tried to do it with the ExecutionEngine (between a transaction):
ExecutionEngine engine = new ExecutionEngine( graphDb );
String ARREL = "ARREL";
ExecutionResult result = engine.execute("START n=node(*) WHERE n.URL =~{"+ARREL+"} RETURN n");
Iterator<Node> n_column = result.columnAs("n");
Node arrelat = (Node) n_column.next();
for ( Node node : IteratorUtil.asIterable( n_column ) )
(...)
But it also fails at the *n_column.next()* returning a null object that throws an exception.
The problem is that I need to use the transactions to optimize the queries due if not it take too much time processing all the queries that I need to do. Should I try to join several operations to the query, to avoid using the transactions?
try to add single quotes at:
START n=node(*) WHERE n.URL =~ '{URL}' RETURN n
Can you update your java-rest-binding to the latest version (1.8) ? In between we had a version that automatically applied REST-batch-operations to places with transaction semantics.
So the transactions you see are not real transactions but just recording your operations to be executed as batch-rest-operations on tx.success/finish
Execute the queries within the transaction, but only access the results after the tx is finished. Then your results will be there.
This is for instance useful to send many cypher queries in one go to the server and have the results available all in one go afterwards.
And yes #ulkas use parameters but not like that:
START n=node(*) WHERE n.URL =~ {URL} RETURN n
params: { "URL" : "http://your.url" }
No quotes neccessary when using params, just like SQL prepared statements.