I am trying to insert all the data at once using the following:
userModel.setServiceData(gson.toJson(sortedResponse).getBytes());
This is giving Out of Memory for the huge data.
So I am trying to add data in chunks to the postgres database using something like:
byte[] chunkedMessage = {};
for (Iterator<Message> iterator = sortedResponse.iterator(); iterator.hasNext();) {
Message message = (Message) iterator.next();
chunkedMessage = gson.toJson(message).getBytes();
userModel.setServiceData(chunkedMessage);
}
Obviously, this will end up storing the last chunk but not all chunks collectively.
I am using hibernate so thinking if there is any easy way with hibernate query or may be as easy as something in the above code.
Do not use arrays (byte[]), as it needs to allocate the memory for the complete data you want to insert into the database and switch to streams. Change the type of userModel.serviceData from byte[] to java.sql.Blob, which allows to create a new Blob instance from an InputStream or Reader:
Blob blob = Hibernate.getLobCreator(session).createBlob(stream, size);
Related
I have a requirement to insert blob value into one of the tables(DB type - Oracle). I am using Java to convert my image file into a byte array first and then to blob object by using javax.sql.rowset.serial.SerialBlob.
However, when I create a blob object using
Blob blob = new SerialBlob(byteArray);
The resulting blob object is of the form javax.sql.rowset.serial.SerialBlob#'some-hex-value'. When I use the blob object in my insert query, Oracle gives me: Invalid hex number error.
But when I try to insert using only the hex part and removing 'javax.sql.rowset.serial.SerialBlob' part from my blob object, the insert query works fine and the record gets inserted.
So is there any way to extract the hex value from the blob object that I am converting my image file into?
Thanks!
i have a from which accept image from user. i sent this form to the next jsp page to insert data in database. i take database field format blog. i do write get parameter line, now i want to convert it to blob type. how can i do this ?
String pic=request.getParameter("pic");
i did not want to use prepare statement to insert data in database
You can do it by getting byte[] content from String
byte[] byteData = yourString.getBytes("UTF-8");//Better to specify encoding
Blob blobData = dbConnection.createBlob();
blobData.setBytes(1, byteData);
Moreover, you can directly store String to BLOB column.
I am novice User to Couchbase, I am trying to insert the documents into the default bucket like below.
I found the following 2 ways to insert the json documents in the bucket:
1) Inserting by preparing JsonDocument and upsert into the bucket
StringBuilder strBuilder = new StringBuilder();
strBuilder.append("{'phone':{'y':{'phonePropertyList':{'dskFlag':'false','serialId':1000,'inputTray':{'LIST': {'e':[{'inTray':{'id':'1','name':'BypassTray','amount': {'unit':'sheets','state':'empty','typical':'0','capacity':'100'}");
String LDATA = strBuilder.toString();
Cluster cluster = CouchbaseCluster.create("localhost");
Bucket bucket = cluster.openBucket("default");
JsonObject deviceinfoObj = JsonObject.create().put("phoneinfo", LDATA);
bucket.upsert(JsonDocument.create("phone", deviceinfoObj));
2) Or by using like direct query like SQL
String query = "upsert into default(KEY, VALUE) values(LDATA)"
I am not able to find how to execute the above query like noram sql statement
Example:
Statement st = connection.createStatement();
ResultSet rs = st.executeQuery(query);
How to use N1QLQuery to insert the json document in the Couchbase Bucket.
I found the two ways to fetching the document.
i) Getting the document directly by using document Id
bucket.get("phone").content().get("phoneinfo")
ii) Getting the documents by using N!QueryResultSet
N1qlQueryResult result = bucket.query(N1qlQuery.simple("select * from default;"));
for (N1qlQueryRow row : result) {
System.out.println(row);
}
I got confused with the different approaches for inserting and fetching the documents from/to bucket in the Couchbase.
If I insert the documents by using 1st approach I need to prepare JsonObject with some key and value as a entire jsondocument.
So I think I better to insert the document using the second approach, So I will be able to fetch the documents using N1QLResultSet(2nd approach). but by using first approach I need to get the number of documents in the bucket and and then only I can loop through all the documents
Queries:
1) How to get the selective nested nodes from the document
2) In json document, Should I need to split key-value for each node ant then put in the JSONObject to prepare JSONDocument?
3) How to create a view for the bucket to do fast retrieval?
Why dont you create a simple java pojo and define it as a #Document. Then use CrudRepo to save and retrieve documents from couchbase.
this sample might help you:
https://github.com/maverickmicky/spring-couchbase-cache
I'm building a logging application that does the following:
gets JSON strings from many loggers continuously and saves them to a db
serves the collected data as a per logger bulk
my intention is to use a document based NoSQL storage to have the bulk structure right away. After some research I decided to go for MongoDB because of the following features:
- comprehensive functions to insert data into existing structures ($push, (capped) collection)
- automatic sharding with a key I choose (so I can shard on a per logger basis and therefore serve bulk data in no time - all data already being on the same db server)
The JSON I get from the loggers looks like this:
[
{"bdy":{
"cat":{"id":"36494h89h","toc":55,"boc":99},
"dataT":"2013-08-12T13:44:03Z","had":0,
"rng":23,"Iss":[{"id":10,"par":"dim, 10, dak"}]
},"hdr":{
"v":"0.2.7","N":2,"Id":"KBZD348940"}
}
]
The logger can send more than one element in the same array. I this example it is just one.
I started coding in Java with the mongo driver and the first problem I discovered was: I have to parse my with no doubt valid JSON to be able to save it in mongoDB. I learned that this is due to BSON being the native format of MongoDB. I would have liked to forward the JSON string to the db directly to save that extra execution time.
so what I do in a first Java test to save just this JSON string is:
String loggerMessage = "...the above JSON string...";
DBCollection coll = db.getCollection("logData");
DBObject message = (DBObject) JSON.parse(loggerMessage);
coll.insert(message);
the last line of this code causes the following exception:
Exception in thread "main" java.lang.IllegalArgumentException: BasicBSONList can only work with numeric keys, not: [_id]
at org.bson.types.BasicBSONList._getInt(BasicBSONList.java:161)
at org.bson.types.BasicBSONList._getInt(BasicBSONList.java:152)
at org.bson.types.BasicBSONList.get(BasicBSONList.java:104)
at com.mongodb.DBCollection.apply(DBCollection.java:767)
at com.mongodb.DBCollection.apply(DBCollection.java:756)
at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:220)
at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:204)
at com.mongodb.DBCollection.insert(DBCollection.java:76)
at com.mongodb.DBCollection.insert(DBCollection.java:60)
at com.mongodb.DBCollection.insert(DBCollection.java:105)
at mongomockup.MongoMockup.main(MongoMockup.java:65)
I tried to save this JSON via the mongo shell and it works perfectly.
How can I get this done in Java?
How could I maybe save the extra parsing?
What structure would you choose to save the data? Array of messages in the same document, collection of messages in single documents, ....
It didn't work because of the array. You need a BasicDBList to be able to save multiple messages. Here is my new solution that works perfectly:
BasicDBList data = (BasicDBList) JSON.parse(loggerMessage);
for(int i=0; i < data.size(); i++){
coll.insert((DBObject) data.get(i));
}
Not able to store large size String in CLOB
String which contains xml data . the size of String is greater than 19000.
I am able to store small string but getting null value in database for large string.
Please suggest the solution for the same.
I am using following code:
Clob myClob = Hibernate.createClob(updateStr);
Reader reader = myClob.getCharacterStream();
myServiceRequest.setRequestData(Hibernate.createClob(reader, (int)myClob.length()));
updateStr is a String which contains xml data and its size is very large. that i am not able to insert in DB.
RequestData is column(type CLOB) in database where i am inserting String .