As the title says, I need to store some big json files on a "blob" field on a oracle database. We've been using the "utl_raw.cast_to_raw" function, which seems to be the suggested way to do it, but we quickly hit the 2000 byte size cap.
I've been looking around for a solution, and the only one I've found points towards converting the Json to a byte array through a webservice (We are using Java instead of C#, but it shouldn't be an issue).
There's something else we can do to get our big jsons on the database without having to relay on a new development?
You should use a CLOB type column, it can store GB of text. It´s not as easy to do CRUD operations with it, but nothing you can´t handle
Related
We support an application that has some bad design.
This application stores data in a Cassandra cluster in a TEXT column and sometimes writes quite large Strings in this column and we get a WriteFailureException.
Cassandra has a limit on the write size (16mb by default: https://docs.datastax.com/en/dse/6.7/dse-admin/datastax_enterprise/config/configCassandra_yaml.html#configCassandra_yaml__max_mutation_size_in_kb) which is great.
We would like to notify the user that they are trying to write a large chunk of data in case such limit is reached.
As I understand there is no way to distinguish whether this exception occurred because of this limit or due to any other errors inside the Cassandra cluster.
It would be even better to check if the size of the date exceeds the limit before trying to write it in Cassandra.
Java String is UTF-16, Cassandra's TEXT is UTF-8, so my naive approach is to convert a String to UTF-8 and check it's size like that: s.getBytes(StandardCharsets.UTF_8).lenght()
However this seems quite expensive to convert a String to UTF-8 just to throw it away.
Is there a sane way to do it? How do people check if their data fits in Cassandra before writing it?
Java 8, Cassandra 3.11
The better way is to check the size not of the individual strings, but the size of the request, because it's also dependent on the protocol version. If you're using prepared statements, then you can bind values and then call requestSizeInBytes on the bound statement (for driver 3.x), like this (source code)
int stmtSize = boundStatement.requestSizeInBytes(protocolVersion, codecRegistry);
For driver 4.x it's the computeSizeInBytes function (doc)
But take into account that it's approximate size, but it could be quite good approximation anyway
I want to store multiple values (String, Int and Date) in a file via Java in Android Studio.
I don't have that much experience in that area, so I tried to google a bit, but I didn't get the solution, which I've been looking for. So, maybe you can recommend me something?
What I've tried so far:
Android offers a SharedPreferences feature, which allows a user to save a primitive value for a key. But I have multiple values for a key, so that won't work for me.
Another option is saving data on an external storage medium as file. As far as good. But I want to keep the filesize at minimum and load the file as fast as possible. That's the place, where I can't get ahead. If I directly save all values as simple text, I would need to parse the .txt file per hand to load the data which will take time for multiple entries.
Is there a possibility to save multiple entries with multiple values for a particular key in an efficient way?
No need to reinvent a bicycle. Most probably the best option for your case is using the databases. Look into Sqlite or Realm.
You don’t divulge enough details about your data structure or volume, so it is difficult to give a specific solution.
Generally speaking, you have these three choices.
Serialize a collection
I have multiple values for a key
You could use a Map with a List or Set as its value. This has been discussed countless times on Stack Overflow.
Then use Serialization to write and read to storage.
Text file
Write a text file.
Use Tab-delimited or CSV format if appropriate. I suggest using the Apache Commons CSV library for that.
Database
If you have much data, or concurrency issues with multiple threads, use a database such as the H2 Database Engine.
I am learning Inubit. I want to know, how may I store images in a database using the Inubit tool set?
The question is more than a year old. I guess you solved it by now.
For all others coming here, let me sketch out the typical way you'd do that.
0. (optional) Compress data.
Depending on the compression of the image (e.g. its GIF, PDF, uncompressed TIFF, etc. and not JPEG), you might want to compress it via a Compressor module first to reduce needed database space and increase overall performance on the next steps. Be sure to compress the binary data and not the base64-encoded string (see next step)!
1. Encode binary stream to base64.
Depending on where you get the image
data from, chances are that it already is base64 encoded. E.g. you
used a file connector to retrieve it from disk with the appropriate option checked or used a web service
connector. If you really have a binary data stream, convert it to
base64 using an encoder module (better self-documenting) or using a variable
assignment using the XPATH-function isxp:encode (more concise).
2. Save the encoded data via a database connector.
Well, the details
for doing this right are pretty much database specific. The cheap
trick that should work on any database, is storing the base64-string
simply as a string in a TEXT / CLOB column. This will waste about
three times as much space in the database as the original binary
data, since base64 is poorly packed. Doing it right would mean to
construct a forced SQL query in an XSLT that decodes the
base64-string to binary and stores it. Here is some reference
to how it can be done in Oracle.
Hope, this might be of some help.
Cheers,
Jörn
Jörn Willhöft
Willhöft IT-Beratung GmbH, Berlin, Germany
You do not store the image in the database, you only record the path to the image. The Image will be stored on the server.
Here is an example of how to store the path to the image : How to insert multiple images path to database
What is difference between serialization and database storage In java? Doesnt serialization actually mean storing data in a database on server?
Let's think of the database like a bowl.
If you want to keep stuff from going everywhere, you put it in the bowl.
Your stuff is the data you want to store. Right now it's out there, on the table, in a box.
So we're going to take the stuff out of that box. The problem is, the stuff in our box probably won't fit into the bowl. How do we fix that?
We need to change it into the type of object that will fit into our bowl. We need to serialize it.
Our serialized data will fit in the bowl now. So we take our serialized data and we pour it into the bowl, and we have the most important meal of the day.
In case this was all really complicated. Simplified: to serialize is to change, and a database is a place to store stuff. Often, you change stuff before you store it.
Serialization can be used to prepare an object for database storage - it is a the process of converting an object into a storable or transmittable format, such as a string or a stream of bytes.
We can't store a java object into most normal storage types as-is - but if we for instance serialize it into JSON we can store it. We can then retrieve the JSON at a later point from the storage and deserialize it to get back an object the same as our original object, given that the serialization and deserialization is properly implemented.
Of course, this doesn't have to entail database storage - having the object serialized into a JSON stream for instance also allows us to transmit it over the internet to be deserialized on another computer.
No. Not at all. Serialization in Java is an API which generates a storeable version of an object that you can later load back from disk (or wherever you store it) and make it back into an object with (hopefully!) the same state as it once had. There are alternatives to it such as Google Protobufs which are better for networked applications, but it is good enough for most simple uses.
Serialization is the process of converting a data structure into a form that can be persisted (saved on a hard drive) in any way. It can be binary, xml, plain text, html, ... usually the goal is to be able to deserialize, that is restore back the state of your data structure at the time it was persisted.
A database is just the place (and not the way) where you store your data.
I have a huge set of data of tables in Open Office 3.0 document format.
Table 1:
(x range)|(x1,y1) |(x2,y2)|(x3,x3)|(x4,y4)
(-20,90) |(-20,0) |(-5,1) |(5,1) |(10,0)
...
Like wise i have n number of tables.All of these tables are fuzzy set membership functions.In simple terms they are computational model's according to which i have to process the input data.There are many number of such tables with differing row size and column size 3/4 .These data's are not going to change once loaded.
Example:
When i get a value of x in between the range -20 to 90.I will apply the first rule(given above).Suppose that it is -1(which is in between value of -20 and -5).Then I have to find a corresponding value between 0 and 1.
My First question is how to extract all the data from the tables in document format so that i can use in my java program.I know a bit of python and I know python can be useful in such cases.But then how to use it in my Java program.
Secondly what would be the best data structure i should use in such a senario.
Note: I'm not using any database.So i would prefer to keep the tables either in xml or some other format so that i can load it easily to the program.I also thinking of making a suitable data structure and then serializing them so that I can load them whenever required instead of parsing a file and recreating the data structure.Please post your comments.
In order to parse an OpenOffice Document in Java (to extract data), you can use a dedicated API such as ODFDOM.
I think this solution is very complicated for what you need. A easier solution would be to extract manually the OpenOffice table, to put it in a format more friendly to parse in Java:
CSV
DataBase (MySQL, etc.)