I use NodeJS for insert 8.000.000 records into my orientdb database, but after about 2.000.000 insert records my app is stopped and show error "Java Heap".
Is there a way for release memory after every record inserted?
Ram usage:
-befor start app: 2.6g
-after insert 2milions records: 7.6g
My app.js (NodeJS):
var dbConn = [];
var dbNext = 0;
var dbMax = 25;
for (var i = 0; i <= dbMax; i++) {
var db = new ODatabase({
host: orientdb.host,
port: 2424,
username: 'root',
password: orientdb.password,
name: 'test',
});
dbConn.push(db);
}
//---------------------------------------------------
//Start loop
// record = {name: 'test'}
record["#class"] = "table";
var db = nextDB();
db.open().then(function () {
return db.record.create(record);
}).then(function (res) {
db.close().then(function () {
//----resume loop
});
}).error(function (err) {
//------
});
// end loop - iteration loop
//---------------------------------------------------
function nextDB() {
if (++dbNext >= dbMax) {
dbNext -= dbMax;
}
return dbConn[dbNext];
}
OrientJS wasn't efficient for insert massive data from SqlServer to OrientDB. I used ETL module for massive insert, that is fastest way and good idea for transpot massive data without increase memory more than 2GB.
I could transported 7.000 records per minute.
My ETL's config.json:
{
"config": {
log : "debug"
},
"extractor" : {
"jdbc": { "driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url": "jdbc:sqlserver://10.10.10.10;databaseName=My_DB;",
"userName": "sa",
"userPassword": "123",
"query": "select * from My_Table"
}
},
"transformers" : [
{ "vertex": { "class": "Company"} }
],
"loader" : {
"orientdb": {
"dbURL": "plocal:D:\DB\Orient_DB",
dbUser: "admin",
dbPassword: "admin",
"dbAutoCreate": true,
"tx": false,
"batchCommit": 1000,
"wal" : false,
"dbType": "graph"
}
}
}
From the documentation, for massive insertion you should declare your intention :
db.declareIntent( new OIntentMassiveInsert() );
// YOUR MASSIVE INSERTION
db.declareIntent( null );
But by now it seems not implemented in orientJS driver.
Another thing is that you should not open/close your database for each new record created. This is in general bad practise.
I do not have the node.js environment by now but something like this should do the trick:
db.open().then(function () {
// when available // db.declareIntent( new OIntentMassiveInsert() );
for (var i = 0; i < 8000000; i++) {
// create a new record
myRecord = { "#class" : "myClass", "attributePosition" : i };
db.record.create(myRecord);
}
// when available // db.declareIntent( null );
}).then(function () { db.close() });
Related
I want to check if a specific value of a key exists in the Realtime Database or not and perform some action based on it.
I have the following data:
"Orders" : {
"03403426747" : {
"17" : {
"State" : "(4) Canceled",
"address" : "yubt",
"date" : "Feb 28, 2022",
"discount" : "0",
"name" : "uk",
"phone" : "0311111111",
"time" : "15:33:58 PM",
"totalAmount" : "3778"
},
"18" : {
"State" : "(1) Approved",
"address" : "yubt",
"date" : "Feb 28, 2022",
"discount" : "120",
"name" : "uk",
"phone" : "03111111111",
"time" : "16:01:58 PM",
"totalAmount" : "7703"
}
}
}
I want to check If any order from these has has "State" value other than "(3) Completed" & "(4) Canceled".
if anyone's order has a value other than these, I want to remove that user from the list which contains users with pending orders.
and if at any time that user has new order or older order State changed I want to again add that user to the list.
I want to check If any order from these has has "State" value other than "(3) Completed" & "(4) Canceled".
There is no way you can query the Realtime Database using a negation. What you can do instead is to create a separate query for each "other" state and join the results on the client.
However, if you consider at some point in time to try using Cloud Firestore, then you should consider using not equal (!=) query or if it fits your needs the not-in query.
After a long time, my brain light burned and I came up with a solution.
Create a data class for OrderState
public class OrderState {
public static int ordersCount = 0;
public static boolean state = false;
public static void update(boolean state){
if (state){
ordersCount = ordersCount + 1;
OrderState.state = true;
}else
if (!state && ordersCount > 0){
ordersCount = ordersCount - 1;
if (ordersCount < 1) OrderState.state = false;
}
}
public static void reset(){
ordersCount = 0;
state = false;
}
}
On FirebaseRecyclerAdapter -> onBindViewHolder
//if order not canceled or completed. it will update OrderState,
//ordersCount + 1 and state to true
if (adminOrders.getState().equals(Prevalent.orderStateNew) ||
adminOrders.getState().equals(Prevalent.orderStateApproved) ||
adminOrders.getState().equals(Prevalent.orderStateShipped) ||
adminOrders.getState().equals(Prevalent.orderStateApproved)){
OrderState.update(true);
}
changeUserWithOrderState();
On changing state of order by admin
//if the order is not already cancelled or completed, reduce one order from OrderState as it will be readded automatically upon Recycler refresh.
if (!adminOrders.getState().equals(Prevalent.orderStateCanceled) &&
!adminOrders.getState().equals(Prevalent.orderStateCompleted)) OrderState.update(false);
Al last if the user does not has any order with states New, Approved, and Shipped
OrderState.orderCount = 0;
OrderState.state = false;
and upon updating the database it will set the state to false.
private void changeUserWithOrderState() {
DatabaseReference userWithOrder = FirebaseDatabase.getInstance().getReference()
.child(Prevalent.usersWithOrders)
.child(userPhoneKey);
HashMap<String, Object> map = new HashMap<>();
map.put(Prevalent.orderState, String.valueOf(OrderState.state));
userWithOrder.updateChildren(map).addOnCompleteListener(task -> {
//Changed state based upon OrderState.state value...
});
}
I'm creating an Android application wherein I have to keep count of YTD, MTD, and Daily record punched for an organization as well as individual users. I tried the approach where for every save of record, I have a counter collection where I save data like
ORG_ORGID_2020 (for YTD)
ORG_ORGID_202005 (for MTD)
ORG_ORGID_20200513 (for Daily data)
ORG_USER1_2020 (for YTD)
ORG_USER2_202005 (for MTD)
ORG_USER3_20200513 (for Daily data)
so that I don't have to read many documents while fetching reports. Now to minimize the reads I save properties in the above documents (org_ID, year (i.e. 2020), yearMonth (i.e.202005), and so on. I save above documents in the form of the counter object
public class Counter {
#DocumentId
private String id;
private long count;
private String dealerId;
private String userId;
private String year;
private String yearMonth;
private String yearMonthDate;
}
Not the issue arises when I have to update the counter. I tried using
private FieldValue count;
and was able to update the count properly using
Counter counter = new Counter();
counter.setCount(FieldValue.increment(1));
counter.setDealerId(intentDealer.getId());
counter.setYear(strFullYear);
batch.set(dealerYtdColRef, counter, SetOptions.merge());
but when I try to fetch the record, I get
java.lang.RuntimeException: No properties to serialize found on class com.google.firebase.firestore.FieldValue
if I change the field to
private long count;
I'm not getting as, how to update the counter. I have to set all fields too along with the counter. I tried using .update method too, but it gives an error when the document is not present and has to be created for the first time.
Hw can I properly manage the counters? I'm doing counter part from app-only instead of functions because I'm trying to get the app work in free firebase tier only.
The problem in your code is the following line of code:
counter.setCount(FieldValue.increment(1));
Your count property is defined in your Counter class to be of type long. When you are using the setCount() method to set its value, you should pass a long value as an argument but you actually don't. The following statement:
FieldValue.increment(1)
Return an object of type FieldValue and not a long, hence that error. To increment the value of your count property atomically by one, please use the following lines of code:
Map<String, Object> updateCount = new HashMap<>();
updateCount.put("count", FieldValue.increment(1));
yourDocRef.set(updateCount, SetOptions.merge());
Finally, as per suggestion from Alex, I use the map values, but at the same time, I moved my code to google. Please let me know if I have done the implementation wrong. It seems to work though
const functions = require('firebase-functions');
// The Firebase Admin SDK to access the Firebase Realtime Database.
const admin = require('firebase-admin');
admin.initializeApp();
// // Create and Deploy Your First Cloud Functions
// // https://firebase.google.com/docs/functions/write-firebase-functions
//
// exports.helloWorld = functions.https.onRequest((request, response) => {
// response.send("Hello from Firebase!");
// });
const db = admin.firestore();
// [START_EXCLUDE]
const settings = { timestampsInSnapshots: true };
db.settings(settings);
// [END_EXCLUDE]
// [START aggregate_function]
exports.aggregateEnquiries = functions.firestore
.document('enquiries/{id}')
.onWrite(async (change, context) => {
if (!change.before.exists) {
// New document Created : add one to count
var dealerId = change.after.data().dealerId;
var userId = change.after.data().assignedTo;
var date = change.after.data().createdDt.toDate();
var day = date.getDate();
var month = date.getMonth() + 1;
var year = date.getFullYear();
var yearMonth = String(year) + (month < 10 ? "0" + month : month);
var yearMonthDate = yearMonth + (day < 10 ? "0" + day : day);
try {
return await db.collection("dealers").doc(dealerId)
.get()
.then((doc) => {
if (doc !== null && doc.exists) {
const increment = admin.firestore.FieldValue.increment(1);
db.collection("enquiries_agg")
.doc("D_" + dealerId + "_" + year)
.set({ "count": increment }, { merge: true });
db.collection("enquiries_agg")
.doc("D_" + dealerId + "_" + monthYear)
.set({ "count": increment }, { merge: true });
db.collection("enquiries_agg")
.doc("U_" + userId + "_" + year)
.set({
"count": increment,
"dealerId": dealerId,
"userId": userId,
"reference": String(year)
}, { merge: true });
db.collection("enquiries_agg")
.doc("U_" + userId + "_" + yearMonth)
.set({
"count": increment,
"dealerId": dealerId,
"userId": userId,
"reference": String(yearMonth)
}, { merge: true });
db.collection("enquiries_agg")
.doc("U_" + userId + "_" + yearMonthDate)
.set({
"count": increment,
"dealerId": dealerId,
"userId": userId,
"reference": String(yearMonthDate)
}, { merge: true });
} else {
console.log("error in aggregare entries.");
}
return null;
});
}
catch (error) {
console.log("Error getting documents: ", error);
throw new Error("Profile doesn't exist : " + error);
}
} else if (change.before.exists && change.after.exists) {
// Updating existing document : Do nothing
} else if (!change.after.exists) {
// Deleting document : subtract one from count
}
return null;
});
I have a mongodb collection. The documents have two fields called rtd ( number of days, int value) and timestamp (long value). I need to get all the documents satisfy this condition using Criteria query
if a document is x
currentTimestamp - x.timestamp converted to days < x.rtd
try {
return mongoTemplate.find(query(criteria), PredictiveEntity.class).stream().filter(predictiveEntity ->
predictiveEntity.getRtd() >= TimeUnit.DAYS.convert(Instant.now().toEpochMilli() - predictiveEntity.getTimestamp(), TimeUnit.MILLISECONDS)
).collect(Collectors.toList());
} catch (Exception e) {
return null;
}
The following query can get you the expected output:
db.predictiveentry.find({
$expr:{
$lt:[
{
$toInt:{
$divide:[
{
$subtract:[new Date().getTime(), "$timestamp"]
},
86400000
]
}
},
"$rtd"
]
}
})
Since $expr is still not supported in Criteria, we need to follow a different route i.e parse the BSON query directly.
Query query = new BasicQuery("{ $expr:{ $lt:[ { $toInt:{ $divide:[ { $subtract:[new Date().getTime(), '$timestamp'] }, 86400000 ] } }, '$rtd' ] } }");
return mongoTemplate.find(query, PredictiveEntity.class).stream().collect(Collectors.toList());
I'm still learning Ionic and programming in general. I followed a link on the internet and I was able to create the white and read the necessary data, but I am not able to insert data in the created table. Can anyone help me with this?
I follow this tutorial: ionic-sqlite
My code:
getRegiao() { // Regiões //
return new Promise<Regiao[]>((resolve, reject) => {
let sql = "SELECT NOM_REGIAO, ID " +
"FROM TB_REGIAO "
this.executeQuery(sql).then(data => {
let regioes = [];
if (data != undefined)
data.forEach(function (row) {
let regiao: Regiao = { nom_regiao: row[0], id: row[1] }
regioes.push(regiao);
});
resolve(regioes);
}).catch(error => {
console.log(error);
});
});
}
addUser() {
let sql = "INSERT INTO TB_USUARIO (EMAIL) VALUES ('BLITCRANK#HOTMAIL.COM')";
// let sql = "SELECT EMAIL FROM TB_USUARIO";
this.executeQuery(sql);
}
executeQuery(sql: string) {
let db: any;
return new Promise<any>((resolve, reject) => {
let xhr = new XMLHttpRequest();
xhr.open('GET', this.dbName, true);
xhr.responseType = 'arraybuffer';
xhr.onload = (e) => {
let uInt8Array = new Uint8Array(xhr.response);
db = new SQL.Database(uInt8Array);
let contents = db.exec(sql);
console.log(contents);
if (contents.length > 0)
resolve(contents[0].values);
else
resolve("query executada sem retorno")
};
xhr.send();
});
}
Please use this plugin , this is the plugin that I am using.
Take note in using execute sql like this db.executeSql('create table danceMoves(name VARCHAR(32))', {})
But rather use this db.executeSql('create table danceMoves(name VARCHAR(32))', [])
I dont know why instead of using object '{}' they replace it as array '[]' I think they forgot to update the documentaion
I have a collection in mongodb - "text_failed" which has all the numbers on which I failed to send an SMS, the time they failed and some other information.
A document in this collection looks like this:
{
_id(ObjectId): xxxxxx2af8....
failTime(String): 2015-05-15 01:15:48
telNum(String): 95634xxxxx
//some other information
}
I need to fetch the top 500 numbers which failed the most in a month's duration. A number can occur any number of time during this month.(Eg: a number failed 143 times, other 46 etc.)
The problem I have is that during this duration the numbers failed crossed 7M. It's difficult to process this much information using the following code which doesn't use aggregation:
DBCollection collection = mongoDB.getCollection("text_failed");
BasicDBObject query = new BasicDBObject();
query.put("failTime", new BasicDBObject("$gt", "2015-05-15 00:00:00").append("$lt", "2015-06-15 00:00:00"));
BasicDBObject field = new BasicDBObject();
field.put("telNum", 1);
DBCursor cursor = collection.find(query, field);
HashMap<String, Integer> hm = new HashMap<String, Integer>();
//int count = 1;
System.out.println(cursor);
while(cursor.hasNext()) {
//System.out.println(count);
//count++;
DBObject object = cursor.next();
if(hm.containsKey(object.get("telNum").toString())) {
hm.put(object.get("telNum").toString(), hm.get(object.get("telNum").toString()) + 1);
}
else {
hm.put(object.get("telNum").toString(), 1);
}
}
This fetches 7M+ documents for me. I need only the top 500 numbers. The result should look something like this:
{
telNum: xxxxx54654 //the number which failed
count: 129 //number of times it failed
}
I used aggregation myself but didn't get the desired results. Can this be accomplished by aggregation? Or is there any other way more efficient in which I can do this?
You could try the following aggregation pipeline:
db.getCollection("text_failed").aggregate([
{
"$match": {
"failTime": { "$gt": "2015-05-01 00:00:00", "$lt": "2015-06-01 00:00:00" }
}
},
{
"$group": {
"_id": "$telNum",
"count": { "$sum": 1 }
}
},
{
"$sort": { "count": -1 }
},
{
"$limit": 500
}
])