My application has 24 buttons to count different vehicle types and directions (the app will be used to count traffic). Currently, I'm saving a line to a .csv file each time a button is pressed. The csv file contains a timestamp, direction, etc.
I have to measure how many times every button was pressed in 15-minute intervals.
How should I store the counters for the buttons?
I just need to output how often every button (every button has a different tag for identification) was pressed in 15 minutes.
I was thinking about using a HashMap, which could just take the button's tag as key and the number of occurrences as value, like this:
HashMap<String, Integer> hm = new HashMap<String, Integer>();
Integer value = (Integer) hm.get(buttonTag);
if (value == null) {
hm.put(buttonTag, 1);
} else {
int nr_occ = value.intValue() + 1;
hm.put(buttonTag, nr_occ);
}
However, I don't need the total sums of button presses, but in 15 minute chunks. And as I'm already writing timestamps to the raw csv-file, I'm wondering how I should store these values.
Edit: I'm currently testing the HashMap, and it's working really well for counting, but I see two issues: first of all, grouping it into 15-minute intervals and secondly, the hashmap isn't sorted in any way. I'd need to output the values sorted by vehicle types and directions.
If your data is simple (i.e. you only need a key/value pair) then consider using SharedPreferences to store the button id and the time it was pressed.
This is a good way because it is extremely fast. You already put the info into a .csv file but to extract the data and traverse through it so that you can compare timestamps is too much overhead IMO.
When a button is pressed store your data in the .csv file then also store the key/value (id/timestamp) then you can iterate through that and do your compare and output whatever you need to output.
The other way (and probably even better) is to simply create and write to your .csv file, then dump it and use something else more robust to process that data as you will probably be doing this anyway.
EDIT: I see a lot of answers which are saying to use SQLite, statics, etc...
These are all inferior methods for what you are asking. Here is why...
SQLite is WAY too much for just a simple button id and timestamp. However if you think you might need to store more data in the future this may in fact be an option.
A static variable might be destroyed if the system happens to destroy the Activity.
If you want to be cool, just serialise the HashMap every time your activity gets destroyed (onDestroy) and load it back up when your activity gets created (onCreate). This is going to be the quickest and simplest way.
Where are you serializing your data to? To a file in Environment.getExternalStorage().
Oh and you might want to keep track of a timestamp to clear the HashMap every 15 minutes - you could put this in SharedPreferences.
I assume you already know how to do this but just in case: Java Serialization
Might be me but this sounds as an ideal situation for using a SQLite database. So you can easily get this interval thing from that by selecting on the timestamp.
Related
Codeless at the moment, just trying to figure it out to start my project. Say for example I'm creating a dare game. Every time a user clicks the dare button, it should randomly pick a dare from the dare file, either a txt or CSV and display it in a field.
Best way to accomplish this?
Choosing a line from a file, uniformly at random, can be done using the reservoir sampling technique. For each line in the file, choose it at a 1/N chance, where N is the number of lines read so far, including the line just read. The random line is then the last line chosen this way.
I would suggest you read all the contents of the file and save it into an object. After that generate a random number every time the user clicks a button. The simplest way is to store it into a list of Strings (List<String>) but if you have other data that needs to be stored, example you have the dare and the corresponding sanction if the user doesn't complete the dare then you can make an object and store it to the object. Example:
public class Dare {
String dare;
String sanction;
// add more attributes if needed
// constructors and getters and setters below
}
Then you could have a list of Dare (List<Dare>).
In order to make sure every item has an even chance of being picked, you need, at minimum, to know the number of items (lines, in your case) at the time you generate the random number.
There are three general options to achieve this:
Load all of your data from your data files first and store it all in some kind of list. Then, you will have the count and items available, or
Store information about the number of items at the beginning of the file, so you can at least read the item count and choose the line number. Then you'd have to read the correct line after choosing a number, or
Ensure every record (line) in the file has a fixed length, and calculate the number of lines (and offset to the start of each) based on the file size and known line length.
There's different situations that call for each option but in your case option 1 makes the most sense (and option 3 makes the least). I'll leave the details as an exercise to you but:
Load each line from the file, storing in a list.
Choose a random number n, in [0, length of list).
Then, you can readily access the nth item in the list.
As an alternative, if you'd like to make sure you go through the entire list and pick each item exactly once, another approach is:
Load each line from the file, storing in a list.
Randomly shuffle the items in the list.
Go through the randomly shuffled list, in order.
Here's some relevant resources:
http://www.cplusplus.com/reference/string/string/
http://www.cplusplus.com/reference/fstream/ifstream/
http://www.cplusplus.com/reference/istream/istream/getline/
http://www.cplusplus.com/reference/vector/vector/
http://www.cplusplus.com/reference/list/list/
http://www.cplusplus.com/reference/cstdlib/rand/
Bonus: http://www.cplusplus.com/reference/algorithm/random_shuffle/
I had a quick question. I'm working on a school project and I need to parse an extremely large text file. It's for a database class, so I need to get unique actor names from the file because actors will be a primary key in the mysql database. I've already written the parser and it works great, but at the time I forgot to remove the duplicates. So, I decided the easiest way would be to create an actors arraylist. (Using ArrayList ADT) Then use the contain() method to check if the actor name is in the arraylist before I print it to a new text file. If it is I do nothing, if it isn't I add it to the arraylist and print to the page. Now the program is running extremely slow. Before the arraylist, it took about 5 minutes. The old actor file was 180k without duplicates removed. Now its been running for 30 minutes and at 12k so far. (I'm expecting 100k-150k total this time.)
I left the size of the arraylist blank because I dont know how many actors are in the file, but at least 1-2 million. I was thinking of just putting 5 million in for its size and checking to see if it got them all after. (Simply check last arraylist index and if empty, it didnt run out of space.) Would this reduce time because the arraylist isnt redoubling constantly and recopying everything over? Is there another method which would be faster than this? I'm also concerned my computer might run out of memory before it completes. Any advice would be great.
(Also I did try running 'unique' command on the text file without success. The actor names print out 1 per line. (in one column) I was thinking maybe the command was wrong. How would you remove duplicates from a text file column in a windows or linux command prompt?) Thank you and sorry for the long post. I have a midterm tomorrow and starting to get stressed.
Use Set instead of List so you don't have to check if the collection contains the element. Set doesn't allow duplicates.
Cost of lookup using arrayList contains() gives you roughly O(n) performance.
Doing this a million times is what I think, killing your program.
Use a HashSet implementation of Set. It will afford you theoretically constant time lookup and will automatically remove duplicates for you.
-try using memory mapped file in java for faster access to the large file
-and instead of ArrayList use a HashMap collection where the key is the actor's name (or the hash code) this will improve a lot the speed since the look-up of a key in a HashMap is very fast
Alright so this problem has been breaking my brain all day today.
The Problem: I am currently receiving stock tick data at an extremely high rate through multicasts. I have already parsed this data and am receiving it in the following form.
-StockID: Int-64
-TimeStamp: Microseconds from Epoch
-Price: Int
-Quantity: Int
Hundreds of these packets of data are parsed every second. I am trying to reduce the computation on my storage end by packaging up this data into dictionaries/hashtables hashed by the stockID (key == stockID)(value == array of [timestamp, price, quantity] elements).
I also want each dictionary to represent timestamps within a 5min interval. When the incoming data's timestamps get past the 5min time interval, I want this new data to go into a new dictionary that represents the next time interval. Also, a special key will be hashed at key -1 telling what 5 particular minute interval per day does this dictionary belong to (so if you receive something at 12:32am, it should hash into the dictionary that has value 7 at key -1, since this represents the time interval of 12:30am to 12:35am for that particular day). Once the time passes, the dict that has its time expired can be sent off to the dataWrapper.
Now, you might be coming up with some ideas right about now. But here's a big constraint. The timestamps that are coming in Are not necessarily strictly increasing; however, if one waits about 10 seconds after an interval has ended then it can be safe to assume that every data coming in belongs to the current interval.
The reason I am doing all this complicated things is to reduce computation on the storage side of my application. With the setup above, my storage side thread can simply iterate over all of the key, value pairs within the dictionary and store them in the same location on the storage system without having to reopen files, reassign groups or change directories.
Good Luck! I will greatly appreciate ANY answers btw. :)
Preferred if you can send me something in python (that's what I'm doing the project in), but I can perfectly understand Java, C++, Ruby or PHP.
Summary
I am trying to put stock data into dictionaries that represent 5min intervals for each dictionary. The timestamp that comes with the data determines what particular dictionary it should be put in. This could be relatively easy except that timestamps are not strictly increasing as they come in, so dictionaries cannot be sent off to the datawrapper immediately once 5 mins has passed by the timestamps, since it isn't guaranteed to not receive any more data within 10 seconds, after this its okay to send it to the wrapper.
I just want any kind of ideas, algorithms, or partial implementations that could help me with the scheduling of this. How can we switch the current use of dictionaries within both timestamps (for the data) and actual time (the 10seconds buffer).
Clarification Edit
The 5 min window should be data driven (based upon timestamps), however the 10 second timeout appears to be clock time.
Perhaps I am missing something ....
Its appears you want to keep the data in 5 min buckets, but you can't be sure you have all the data for a bucket for up to 10 sec after it has rolled over.
This means for each instrument you need to keep the current bucket and the previous bucket. When its 10 seconds past the 5 min boundary you can publish/write out the old bucket.
i am currently creating a bus time table app. as of now it gets the current date and time.
I currently have an array of times the bus will be coming next. I want to compare the current time to that of those in the array and return the one closest... representing the next bus. However, there are different timetables for every stop, one for each direction. furthermore they change almost daily.
Is there a nicer data structure anyone can recommend for storing and recalling this data? My code is very messy so would like a better method of comparing,storing and returning data than arrays.
Thanks.
I would recommend using heap data structure (min-heap).
Each time the top most element will represent the nearest event, so you can remove it operate on it (computing the next event's time) and then insert it again.
it is easy and fast.
I'm developing a "funny quotes" app for android. I have over 1000 quotes which I want to use inside my app but I don't know whether I should use a database or text file. Please note that the app should not read the same sentence twice. and it has a previous/next button so i need to keep track of the previous quotes. please tell me which one is better and more optimized. Also, if you can please link me to a good tutorial about storing/reading the data.
thanks
Use a database. It's faster and more flexible than a text file. One day you will extend the app and then you will be glad you used a database. I recommend to, when you boot up the app, just select all the rows using the in-built random functionality of your database. 1000 rows should not take too long. Then just iterate through the resulting ArrayList (or whatever you choose to use) of strings you end up with - the first quote you show will be element 0 from that list, the second element element 1 from that list, and so on. If you use this approach, you won't need any other structure to keep track of used quotes - just use the iterator variable that you use for indexing the quote array.
fetchAllRows on this page seems to be what you want for getting the data.
If you choose not to keep too much in memory, you could keep just a list of quote IDs that have been used so far. The last element of that list would be the current quote, and the previous elements would be what the user should see when they press the back button.
If you will never read the same string twice the I will recommend you to not use String class as there objects are immutable and will stick in the string pool waiting to be reassigned to a reference, but that will never happen as you will never read the same string twice.
The use of DB's will over complicate things.
I suggest you to read a flat file in bytes and then translate them to StringBuider objects hence keeping it simple enough but still preventing intensive GC().
I hope it helps..
USing DB should be fine as I think you would not want all the data in memory. You can keep all the quotes in DB and keep a flag to keep track whether a quote was read or not (simply update it to true once read.)
This way you can choose from any of the quote which has the flag as false.
Have you considered CsvJdbc? You have the benefit of simple csv files with an easy upgrade path to a real database later when you have a significant number of records.
1k records is quite small and in my opinion not sufficient to merit a database.