Am here not asking for someone to write out an example, am looking for more input on ways to accomplish this, and why. (Guidance as such)
I have been trying and reading about Listview, But from what i understand, Listview and using search with it, would only work with a relatively small data set. (I am having a hard time getting to grips with it)
I am wanting to search a table (Across the network currently using jdbc:jtds) and the table can contain anything from 1,000 - 500,000 lines.
(An i know people are going to advise against it, but its more for internal application usage than public release)
The way i understand SearchView is its Android Implementation of the Ajax live search, and if so then it would need an array of data to filter through.
To pre load 500,000 would be impracticable, Am i correct in thinking this?
The method i was thinking was creating a new activity with multiple Textfields, and a query behind each one or multiples and performing a search based on the text that has been inputted and the user pressing a Submit / Search.
The draw back of this, is its more old school with back and forth process if a typo has been made, Where the SearchView would be interactive display results straight away from what the user starts to type in.
Could someone advise what may be the best method to do this? There appears to be way too many ways to search for data in java / android and am getting lost in the number of ways to do this.
(There is no Database being held locally, this is all purely running of a local network only.
I have currently been trying with Listadapter and SearchBinding)
Related
I want to present a list of the names/basic attributes of some complex objects (i.e. they are comprised of multiple collections of other objects) in a recycler view, then get the full object on user selection. For example, the top level objects are "Play Scripts", and each contains a number of "Spoken Lines" spoken by one of the "Actors" associated with the Play Script.
I'm trying to use the Android Architecture components to do this and have (using Florian # codinginflow.com 's tutorials) successfully used Room to create a simplified Play_Script class, DAO and Repository. I've also created some basic REST web services in ASP.Net which can serve up data from a MySQL db.
It strikes me that the path that I am going down will perform poorly and use excessive network bandwidth getting lots of data that I won't use. I'm getting every Play Script (including its Spoken Lines etc) just so that I have the Play Script "Name" and "Description" attributes to populate the Recycler.
In the olden days, I'd just "SELECT ID, Name, Description FROM Play_Script" and once the user had made their choice, I'd use the ID as the key to get everything else that I needed. I suspect that I'm missing something fundamental in the design of my data entities but can't come up with any keywords that would let me search for examples of this common sort of task being done well (/at all).
Please can you help this SO noob with his 1st question?
Cheers,
Z
Update 15 May:
Though I haven't had a response, from what I've been reading in recent weeks (e.g. re Dependency Injection) I suspect that there is no blanket approach for this sort of thing in Android development. It appears that people generally either retrieve extensive data and then use what they require or else build multiple Web Service APIs to return sparse data that includes keys that the client can use to expand when required. So, for example you might make both a "plays_light" and a "plays_detail" Get API.
My solution has been exactly as my May update - i.e. to extend the web API and offer a number of similar calls that return varying granularities of information. It's not particularly elegant and I suspect there may be better ways but it works. In general, I'm finding that the user tends to need less detail in the parent entities and more as we get to individual children/grandchildren.
I do now realise why some apps are so slow though: It's easy to be lazy in the web service design and just return loads of data - only a fragment of which will be used by the client - and justify this by convincing yourself that single API will be universally applicable and thus easier for whoever picks up my code down the line to understand.
Again, it could be my inexperience but I find the local caching of relational data on the Android side retrieved through the API calls quite clunky - lots of storing foreign keys and then re-parsing json to get data into the SQLite tables. I'd hoped Dagger would have been more useful in simplifying this than it has turned out to be so far. I actually unravelled a whole load of Dagger-related code just to preserve my sanity. Not sure I was entirely successful!
Better answers are still very much welcome.
Z
I am working on a small, toy application to expand my knowledge about Java JavaFx and Sql. I have a MySQL server in my local network, which I am able to communicate with and a simple Tableview can be populated, sorted ... etc. The data has only to be shown to the user, no editing. Everything nice and clean.
The Problems:
There are around 170 000 rows with 10 col., all chars, to display, which seems to be rather hard to do in reasonable time. The query is done during startup and it take around 1 1/2 min before I can see the table.
Also the memory footprint is enormous, the application without populated Tableview around 70 mb, with all the data it has 600-700 mb (the xml file which is used to populate the mysql is 70 mb in size ... ) !
Sorting is REALLY slow, I am using Stringproperty which should give a boost according to: JavaFx tableview sort is really slow how to improve sort speed as in java swing (If I understood that correctly) However the custom sort, I did not try so far.
My thoughts:
Similar to the application design for mobile, I think an adapter-pattern can fix these problems. Hence, I create an OberservableList with the correct size of elements, but only populate a limit of rows in the beginning. When I am scrolling done (scroll wheel) the List has to be updated with new elements in advance via sql-queries. This should give me a performance boost for the first Problem. Nice idea but what am I going to do if the user is going to scroll done via the scrollbar(click and drag down), then I would skip certain entries, but I need the information to give the user the feedback where to scroll to.
How could I fix this ?
For the sorting, I would use the sql sorting methods, so each sort will be performed on the sql server and a new OberservableList will be created. As before, only a certain amount of data would be loaded in the first query.
If this approach would also effect the memory footprint, I am not sure.
Your opinion:
Are my ideas reasonable and do-able in Java, JavaFx ?
I would love to hear your ideas about these problems.
Thank you.
I found out that JVx is capable of providing the lazy-loading function. This should do the trick
The problem is how to store (and search) a set of items a user likes and dislikes. Although each user may have 2-100 items in their set, the possible values for the items numbers in the tens of thousands (and is expanding).
Associated with each item is a value say from 10 (like) to 0 (neutral) to -10 (dislike).
So given a user with a particular set, how to find users with similar sets (say a percentage overlap on the intersection)? Ideally the set of matches could be reduced via a filter that includes only items with like/dislike values within a certain percentage.
I don't see how to use key/value or column-store for this, and walking relational table of items for each user would seem to consume too many resources. Making the sets into documents would seem to lose clarity.
The web app is in Java. I've searched ORMS, NoSQL, ElasticSearch and related tools and databases. Any suggestions?
Ok this seems like the actual storage isn’t the problem, but you want to make a suggestion system based on the likes/dislikes.
The point is that you can store things however you want, even in SQL, most SQL RDBMS will be good enough for your data store, but you can of course also use anything else you want. The point, is that no SQL solution (which I know of) will give you good results with this. The thing you are looking for is a suggestion system based on artificial intelligence, and the best one for distributed systems, where they have many libraries implemented, is Apache Mahout.
According to what I’ve learned about it so far, it can do what you need basically out of the box. I know that it’s based on Hadoop and Yarn but I’m not sure if you can import data from anywhere you want, or need to have it in HDFS.
Other option would be to implement a machine learning algorithm on your own, which would run only on one machine, but you just won’t get the results you want with a simple query in any sql system.
The reason you need machine learning algorithms and a query with some numbers won’t be enough in most of the cases, is the diversity of users you are facing… What if you have a user B which liked / disliked everything he has in common with user A the same way - but the coverage is only 15%. On the other hand you have user C which is pretty similar to A (while not at 100%, the directions are pretty much the same) and C has marked over 90% of the things, which A also marked. In this scenario C is much closer to A than B would be, but B has 100% coverage. There are many other scenarios where most simple percentages won’t be enough, and that’s why many companies which have suggestion systems (Amazon, Netflix, Spotify, …) use Apache Mahout and similar systems to get those done.
My applications requirement is to contact the webservice, get the xml, parse it and display it using a listfield. I am calling all this classes xmlhandler, objectmodel, displaying it using a lisfield from a class that extends mainscreen which is making my application slow.
Can anyone suggest me how to make it fast?
Is it a apt to popup a loading screen and start a thread for contacting the webservice, get the xml, parsing it and kill the thread, then populate the listscreen and display it?
suggestions of any kind is welcome!
Test the speed of every part of your program. What I usually use is System.nanoTime() and find the difference in time after every part of the program.
Find out which part is slow before you do anything else.
Otherwise, you'll waste a lot of your time on the wrong parts.
For doing this kind of timing work, I often will do internal logging into a StringBuilder, or maybe just into an ArrayList holding raw, unformatted data. After the test is over, I format and output the data. This minimizes the effect of the logging on the timings.
I can only gues so forgive me if I'm wrong - to me it seems more efficient to create the item of list field only when they're really viewed. So I'd try to keep in memory only the parsed strings and create only the UI items currently to be displayed, discard invisible. To make it more smooth you can you can extend it one or more pages before and after current page.
This way the number of displayed items is always constant. You may also add paging to the service layer to limit number of records trabsmitted at once.
So I have two possible solutions that I want to implement. Firstly I will state my problem. The task I have been assigned to requires me to go to a website called finra.org and do broker checks to see if the brokers in my excel sheet(which gives the name and company among other things) still work A, and if they do work do they still work for the company in the excel sheet. If they do move on to the next one, and if they don't delete them from the sheet. The issue lies in that I have 37k names to check. I calculated this and to do it individually, which is annoying and takes the whole day allows me to do a maximum of 1400 a day. That is on a productive day when I dont have other things to do. So I figured a better use of my time ( I am an intern) would be to write a program which (here are my two suggested solutions:)
1.) Automatically through minimal key strokes copies the data and pastes it into the search box on the page. Ill still have to click and search but at least I would eliminate copying and pasting and switching between screens which takes the majority of the time.
2.) Completely automate the process. I was thinking of copying the names into a text file and then somehow writing a program that takes each name and submits a query to this website which would then show me the result. Perhaps sends the result text to a text file and then i could just GREP the text file for the data that i need.
Any idea if any of this is possible?
Thanks,
Kevin
Definitely possible. I'm doing something similar with a database and an Excel spreadsheet of values using AutoHotKey to automate queries, Chrome console commands and Javascript bookmarklets to scrape data into the clipboard, and Ruby/Nokogiri with more complex and/or structured parsing tasks.
Either of your methods will work - if you have little programming background, I would suggest starting with AutoHotKey since it mimics keyboard and mouse commands, so the programming is much more straightforward and easier to understand. If you have some object-oriented programming skills, learning Ruby/Nokogiri might be your solution, depending on how FINRA's page is structued.