I am trying to model a simple diagram editor that stores data into GAE's HRD datastore, and I am facing problems that I am wondering if you can give me some advices on. Basically, the editor's model are:
[Diagram] --contains--> [Shapes] -- contains--> [Text]
Initially, I put these into the same entity group, to make sure the data is consistent. By doing so, whenever I add new shapes or add text objects to existing shapes, they show up properly (as it's queried from datastore); however, I am having problem because the user could be adding many shapes quickly, and that leads to more than one update to the diagram object per second, which leads to write contention.
Alternatively, I could design like so:
[Diagram] [ Shape - contains diagramId] [ Text - contains shapeId]
which put them in different entity group, and when I create a new shape I just have to save the shape object itself. This solves the write contention problem, but the data is no longer consistent -- depending on how soon HRD commits the write, I might get stale data.
I tried combinations such as putting the diagrams in the cache, and retrieve from the HRD only if the cache doesn't contain the diagram; however, this is unpredictable as I can't be sure that the diagram would be in the cache (I expect many diagrams to be edited at the same time)...
What's the best practice on dealing with such issue? I am using Java and JDO, if that makes any difference...
Few notes:
HRD reads are strongly consistent if you use get and eventually consistent if you use query (as queries rely on indexes and those need time to be built). You can get around your problem by using get, if possible.
If Texts and Shapes are part of one Diagram and if you do not need to access them separatelly AND you do not need to search by Text's or Shape's properties, then you could just serialize Text and Shape inside a Diagram. I use Objectify and this is simply achieved with the #Serialize annotation on a field (don't know about JDO as I don't use it).
Related
I want to present a list of the names/basic attributes of some complex objects (i.e. they are comprised of multiple collections of other objects) in a recycler view, then get the full object on user selection. For example, the top level objects are "Play Scripts", and each contains a number of "Spoken Lines" spoken by one of the "Actors" associated with the Play Script.
I'm trying to use the Android Architecture components to do this and have (using Florian # codinginflow.com 's tutorials) successfully used Room to create a simplified Play_Script class, DAO and Repository. I've also created some basic REST web services in ASP.Net which can serve up data from a MySQL db.
It strikes me that the path that I am going down will perform poorly and use excessive network bandwidth getting lots of data that I won't use. I'm getting every Play Script (including its Spoken Lines etc) just so that I have the Play Script "Name" and "Description" attributes to populate the Recycler.
In the olden days, I'd just "SELECT ID, Name, Description FROM Play_Script" and once the user had made their choice, I'd use the ID as the key to get everything else that I needed. I suspect that I'm missing something fundamental in the design of my data entities but can't come up with any keywords that would let me search for examples of this common sort of task being done well (/at all).
Please can you help this SO noob with his 1st question?
Cheers,
Z
Update 15 May:
Though I haven't had a response, from what I've been reading in recent weeks (e.g. re Dependency Injection) I suspect that there is no blanket approach for this sort of thing in Android development. It appears that people generally either retrieve extensive data and then use what they require or else build multiple Web Service APIs to return sparse data that includes keys that the client can use to expand when required. So, for example you might make both a "plays_light" and a "plays_detail" Get API.
My solution has been exactly as my May update - i.e. to extend the web API and offer a number of similar calls that return varying granularities of information. It's not particularly elegant and I suspect there may be better ways but it works. In general, I'm finding that the user tends to need less detail in the parent entities and more as we get to individual children/grandchildren.
I do now realise why some apps are so slow though: It's easy to be lazy in the web service design and just return loads of data - only a fragment of which will be used by the client - and justify this by convincing yourself that single API will be universally applicable and thus easier for whoever picks up my code down the line to understand.
Again, it could be my inexperience but I find the local caching of relational data on the Android side retrieved through the API calls quite clunky - lots of storing foreign keys and then re-parsing json to get data into the SQLite tables. I'd hoped Dagger would have been more useful in simplifying this than it has turned out to be so far. I actually unravelled a whole load of Dagger-related code just to preserve my sanity. Not sure I was entirely successful!
Better answers are still very much welcome.
Z
Background: I have done a bit of looking into Caching in Spring and it seems like a great way to save time for common read operations. My code currently has a loop over a large number of items, where I am performing logic to see if certain other objects are connected in a way through common items. A way to think about this is similar to a shopping website's related items showing up when you view a certain item. The values I use to determine this are complex, but that is the basic idea.
On loading the item page there is a very long load time trying to compute and figure out which other items are related in some way as to display links to them. Instead of computing this list every time an item page loads, I have started "caching" items with a list of their recommended items. Many things in the system can trigger a need to recalculate these relations: adding/removing properties to items, adding/removing items, etc.
Problem: My "cache" is simply a singleton object containing a Map for items and their related objects. The process of iterating through every item in the system when any change to the cache is needed is very time consuming and process intensive. Java Caches don't seem to be the right answer due to constant changes to items. Is there any other design patterns that I am overlooking for this design? Caches seem to be close, but I am not sure if this problem fits into the mold of caching, due to it being a little more complex then a bulk amount of reads to a single item.
Are caches the way to go with this? If caching isn't the right solution, what is?
It seems that caches are not a solution for your problem, but they might help you in reaching a solution.
For example instead of caching the created items another approach is to cache information that rarely changes but is crucial to create the lists.
Spring function based caching (ie #Cachable) might come in handy, either for caching or invalidation.
The next level is to examine different types of caches (ie. redis) and what they offer in terms of algorithms, sorting and Pub/Sub.
Please deal with this naive question.
Objective is to create a UI and dynamically be able to change the basic workflows, add another option, add another steps etc.
As an example, one simple workflow could be as following:
What's the age of 'X'? TEXT_BOX
if(age>18) proceed to step 2.
else, go to next page (let's say, same processing happens again with different value of X).
What does 'X do?
a. Job
b. Business
Submit (Go to next 'X').
I wanted to keep this workflow in XML as complete tree (all branches of if/else-if/else) and pass it on UI for rendering. Some of the sub-trees will be populated as per action performed in previous step.
This way, small modifications or workflow changes will not require any code changes.
Other option is to use JSON and pass it directly (rather than converting XML to JSON and passing) but it will be loosely coupled and could be difficult to manage in future.
Is there any clearcut benefit I should think of before choosing any of them?
Will any of them provide any extra benefit in the problem I am trying to solve?
Thanks,
It is possible to mimic the structure that you plan to have in an XML, as a json string, without any hurdles as far as I think of. (If you think of any hurdles please point out, I might help you out with idea to handle that).
json will for sure save you lots of bandwith if the data you are planning to send is large. This will inturn also reduce the roundtrip time and make your application more responsive.
Let me describe the problem. A lot of suppliers send us data files in various formats (with various headers). We do not have any control on the data format (what columns the suppliers send us). Then this data needs to be converted to our standard transactions (this standard is constant and defined by us).
The challenge here is that we do not have any control on what columns suppliers send us in their files. The destination standard is constant. Now I have been asked to develop a framework through which the end users can define their own data transformation rules through UI. (say field A in destination transaction is equal to columnX+columnY or first 3 characters of columnZ from input file). There will be many such data transformation rules.
The goal is that the users should be able to add all these supplier files (and convert all their data to my company data from front end UI with minimum code change). Please suggest me some frameworks for this (preferably java based).
Worked in a similar field before. Not sure if I would trust customers/suppliers to use such a tool correctly and design 100% bulletproof transformations. Mapping columns is one thing, but how about formatting problems in dates, monetary values and the likes? You'd probably need to manually check their creations anyway or you'll end up with some really nasty data consistency issues. Errors caused by faulty data transformation are little beasts hiding in the dark and jumping at you when you need them the least.
If all you need is a relatively simple, graphical way to design data conversations, check out something like Talend Open Studio (just google it). It calls itself an ETL tool, but we used for all kinds of stuff.
I'm looking for good dynamic data storage mainly based on JAVA, or to have ability to be really easy used by JAVA.
Main problem in my project is in fast that our data structures will not remain stable, even more structures will be changed from time to time, so basically casual Relation Data Base will lost fight on that level, because drop and add new columns are pretty risky. Which means that some NOSQL or XML based or even file based storage will be usable there.
All inputs are coming from other resource which could be a SOAP callback, JASON call back, import from CSV file or manual input, based on that have to create entities and than to fill it with data.
Last thing on which I have to keep eye on is to bring unstructured, semi-structured and differently structured data in a unified form. Beside of this would be nice to have ability of maintenance of huge amount of data in accepted time duration.
Any ideas?
HyperSQL is open source and free.