Hi I am developing a game in Java for learning purposes and have a question regarding in-app handling of data
Say I have a bunch of xml files storing a variety of statistics for, weapons armours etc and I parse these files into my application as lists;
Note, this data will never change and is effectively "final", it is constant data that will be used by the game.
Now my question is, would you store this data (for use while the game is running) in a class (in my example here a singleton) within your application that you can access easily? So for example something like this (where the list has been read elsewhere by the XML parser)
public class WeaponData {
private List<Weapon> weaponData;
public static final WeaponData instance = new WeaponData(parseXml("weapons"));
private WeaponData(List<Weapon> data) {
weaponData = data;
}
public static WeaponData getInstance() {
return instance;
}
public List<Weapon> getWeaponData() {
return weaponData;
}
}
And allows me to use the data with
WeaponData.getInstance().getWeaponData();
If so, is this the way to go about it, or is there a better way?
Or, the only alternative I can think of, is to keep reading the XML whenever its needed (which seems unwise, paticularly given, this data retrieval may (for a commercial application atleast) be a network operation, even if in my case it is just a hdd read). Plus if anything else it would be repeated code.
Thanks for your time!
You should probably read it in once and save it, but don't store it as a singleton. Read the WeaponData during initialization and store an instance of it. If you don't like that you are welcome to use the pattern you suggested, it just might be awkward later.
If you ever get to where WeaponData might be updated while the app is running, you may want to re-read it, but not ever time.
I'm thinking that your approach will work, especially as you have mentioned that the data won't become large enough that it imposes on memory. Another concern would be the frequency in which you must read the data. There's a tradeoff between those two considerations, but it sounds like the small XML size warrants unmarshalling the XML into objects immediately.
Just for completeness, you should be using JAX-B to generate Java classes from your XML schema, rather than rolling your own parser.
If you fear that the XML's data may change, you might consider using the WatchService API to detect changes and re-parse the XML file.
Related
I'm developing a small web-app whose servlets periodically get access to a shared resource which is a simple text-file on the server side holding some lines of mutable data. Most of the time, servelts just read file for the data, but some servelts may also update it, adding new lines to the file or removing and replacing existing lines. Although file contents is not updated very often, there is still little chance for the data inconsistency and file corruption if two or more servlets decide to read and write to file at the same time.
The first goal is to make the file reading/writing safe. For this purpose, I've created a helper FileReaderWriter class providing some static methods for thread-safe file access. The read and write methods are coordinated by ReentrantReadWiteLock. The rule is quite simple: multiple threads may read from file at any time as far as no other thread is writing to it at the same time.
public class FileReaderWriter {
private static final ReentrantReadWriteLock rwLock = new ReentrantReadWriteLock();
public static List<String> read(Path path) {
List<String> list = new ArrayList<>();
rwLock.readLock().lock();
try {
list = Files.readAllLines(path);
} catch (IOException e) {
e.printStackTrace();
} finally {
rwLock.readLock().unlock();
}
return list;
}
public static void write(Path path, List<String> list) {
rwLock.writeLock().lock();
try {
Files.write(path, list);
} catch (IOException e) {
e.printStackTrace();
} finally {
rwLock.writeLock().unlock();
}
}
}
Then, every servelt may use the above method for file reading like this:
String dataDir = getServletContext().getInitParameter("data-directory");
Path filePath = Paths.get(dataDir, "test.txt");
ArrayList<String> list = FileReaderWriter.read(filePath);
Similarly, writing may be done with FileReaderWriter.write(filePath, list) method. Note: if some data needs to be replaced or removed (which means fetching the data form a file, processing it and writing updated data back to a file), then the whole code paths for this operation should be locked by rwLock.writeLock() for atomicity reasons.
Now, when access to a shared file seems to be safe (at least, I hope so), the next step is to make it fast. From the scalability perspective, reading a file at every user's request to the servlet doesn't sound reasonable. So, what I thought of is to read the contents of file into ArrayList (or other collection) only once during the context initialization time and then share this ArrayList (not the file) as a context-scoped data-holder attribute. Then a context-scoped attribute can be shared by servlets with the same locking mechanism as described above and the contents of the updated ArrayList may be independently stored back to the file on some regular basis.
Another solution (in order to avoid locking) would be to use CopyOnWriteArrayList (or some other collection from java.util.concurrent package) for holding a shared data and designate a single-threaded ExecutorService to dump its contents into a file when needed. I also heard of Java Memory-Mapped Files for mapping the entire file into internal memory, but not sure if such approach is appropriate for this particular situation.
So, could anybody, please, guide me thorough the most effective ways (maybe, suggesting some other alternatives) to solve the problem with a shared file access, provided that the writing to a file is quite infrequent and the contents of it is not expected to exceed a dozens of lines.
You don't explain your real problem, only your current attempt then, is difficult to provide a good solution.
Your approach has two serious problems:
Problem 1: concurrency
a shared resource which is a simple text-file on the server side
holding some lines of mutable data
90% of the solution to a problem is a good data structure. A mutable file it's not. Even popular database engines have important concurrency limitations (eg. SQLite), don't try to reinvent the wheel.
Problem 2: horizontal scalability
Even if he solves his local concurrency problems (eg. synchronous methods), you won't be able to deploy multiple instances (nodes/servers) of your application.
Solution 1: use the right tool for the job
You don't explain exactly the nature of your (data management) problem but probably any NoSQL database will do you good (reading about MongoDB can be a good starting point).
(Bad) solution 2: use FileLock
If for some reason you insist on doing what you indicate, use low level file locks using FileLock. You will only have to deal with partial file locks and even these can be distributed horizontally. You won't have to worry about synchronizing other resources either, as file-level locks will suffice.
(Limited) solution 3: in memory structure
If you don't need horizontal scalability, you can use a shared in memory structure like ConcurrentHashMap but you will lose the horizontal scalability and you could lose transactions if you do not persist the information before an application stop.
Conclusion
Although there are more exotic distributed data models, using a database for even a single table may be the best and simplest solution.
I'm a decent C++ programmer, good enough to do what I want. But I'm working on my first Android App (obviously not C++ related), and I'm having an issue where I'd like to translate what I know from C++ over to the XML/Java used in Android Studio.
Basically I have (in C++) an array of structures. And maybe I didn't do the perfect search, but I sure as heck tried to look around for the answer, but I didn't come up with anything.
How would I go about placing an array of structures inside the XML file and utilizing it in Java?
As a bit of a buffer, let me say that I'm not really looking for code, just verification that this is possible, and a method on how to go about it. I don't mind researching to learn what I want, but I haven't come up with anything. Like I said, I probably haven't googled it properly because I'm unsure of exactly how to ask it.
EDIT: So it appears that XML doesn't have a structure (or anything similar? not sure). But I can utilize a Java class with public variables. Now my question is more or less: What would be the best way to go about inserting all the information into the array/class/variables?
In C++ terms, I could neatly place all the info into a text file and then read from it, using a FOR loop to place all the info in the structures. Or, if I don't want to use an outside source/file, I could hardcode the information into each variable. Tedious, but it'd work. I'm not sure, in Android terms, if I could use the same method and pack in a text file with the app, and read from the file using a FOR loop to insert the information into the array/class/variables
class answerStruct
{
public String a;
public boolean status;
};
class questionStruct
{
public String q;
answerStruct[] answer = new answerStruct[4];
};
I'm not placing this here to brag at my super high tech program, but to give a visual, and frankly that's less I have to write out. This is the method I plan on going with. But, being Java, I'm open to possibly better options. My question still stands as far as inputting information into the variables. Hard code? or does Android/Java allow me to place a text file with my app, and read from it into the variables?
XML is just a markup language for tree-structured data, and imposes no restrictions on how you name or structure your tree nodes.
What I think that you're looking for is an XML Object Serialiser: a way to serialise your in-memory structure into XML for a more permanent storage, and then at a later run, deserialise it back into memory. There are many XML Serialisers for Java, each with an own proprietary XML format.
I've used Simple XML in the past, and found it easy and flexible.
I have to use already developed JAR in order to keep my code running. The JAR provides me with functionality of transforming file formats . My code looks something like this:
public class Transformer {
//some fields
//constructor
public List<MyFile> tranformFiles(List<MyFile> files){
JarClassUsed used = new JarClassUsed();
List<MyFile> data = new ArrayList<>();
foreach(MyFile file : files){
data.add(used.TransformFileFormat(file));
}
return data;
}
}
It's working fine, but still very slow. The problem is that the JAR I have to use is making a hidden/internal DB connection call when transforming files (reading predefined configurations). When count is around 100 is acceptable, but I have one case with more than 1000. And as already you guessed is quite problematic.
How can I keep this hidden connection Opened - still DB connections are expensive to create and Singleton or static sound like good solution. It's not smart to create it for each file that is being processed. Can I use reflection somehow here?
So we talked about it in the office... it seems the best course of action is to find a library that does the same formats, if thats not possible externalize the configuration and utilize a different library.
I know this does not answer your question in any way... and you probably already knew that... but unless you have the raw source and are allowed to change that jar, the "hidden" connection is something written into the library.
I really wish I had better advice, but I think the best course of action is to investigate other implementations.
If you can list the formats I would be more than happy to assist in this investigation in a different medium.
Is there a generally-accepted way to return a large list of objects using Java EE?
For example, if you had a database ResultSet that had millions of objects how would you return those objects to a (remote) client application?
Another example -- that is closer to what I'm actually doing -- would be to aggregate data from hundreds of sources, normalize it, and incrementally transfer it to a client system as a single "list".
Since all the data cannot fit in memory, I was thinking that a combination of a stateful SessionBean and some sort of custom Iterator that called back to the server would do the trick.
So, in other words, if I have an API like Iterator<Data> getData() then what's a good way to implement getData() and Iterator<Data>?
How have you successfully solved this problem in the past?
Definitely don't duplicate the entire DB into Java's memory. This makes no sense and only makes things unnecessarily slow and memory-hogging. Rather introduce pagination at database level. You should query only the data you actually need to display on the current page, like as Google does.
If you actually have a hard time in implementing this properly and/or figuring the SQL query for the specific database, then have a look at this answer. For JPA/Hibernate equivalent, have a look at this answer.
Update as per the comments (which actually changes the entire question subject...), here's a basic (pseudo) kickoff example:
List<Source> inputSources = createItSomehow();
Source outputSource = createItSomehow();
for (Source inputSource : inputSources) {
while (inputSource.next()) {
outputSource.write(inputSource.read());
}
}
This way you effectively end up with a single entry in Java's memory instead of the entire collection as in the following (inefficient) example:
List<Source> inputSources = createItSomehow();
List<Entry> entries = new ArrayList<Entry>();
for (Source inputSource : inputSources) {
while (inputSource.next()) {
entries.add(inputSource.read());
}
}
Source outputSource = createItSomehow();
for (Entry entry : entries) {
outputSource.write(entry);
}
Pagination is a good solution when working with a web based ui. sometimes, however, it is much more efficient to stream everything in one call. the rmiio library was written explicitly for this purpose, and is already known to work in a variety of app servers.
If your list is huge, you must assume that it can't fit in memory. Or at least that if your server need to handle that on many concurrent access then you have high risk of OutOfMemoryException.
So basically, what you do is paging and using batch reading. let say you load 1 thousand objects from your database, you send them to the client request response. And you loop until you have processed all objects. (See response from BalusC)
Problem is same on client side, and you'll likely to need to stream the data to the file system to prevent OutOfMemory errors.
Please also note : It is okay to load millions of object from a database as an administrative task : like for performing a backup, and export of some 'exceptional' case. But you should not use it as a request any user could do. It will be slow and drain server resources.
I have a hobby project, which is basically to maintain 'todo' tasks in the way I like.
One task can be described as:
public class TodoItem {
private String subject;
private Date dueBy;
private Date startBy;
private Priority priority;
private String category;
private Status status;
private String notes;
}
As you can imagine I would have 1000s of todo items at a given time.
What is the best strategy to store a
todo item? (currently on an XML file)
such that all the items are loaded
quickly up on app start up(the
application shows kind of a dashboard
of all the items at start up)?
What is the best way to design its
back-end so that it can be ported to
Android/or a J2ME based phone?
Currently this is done using Java
Swing. What should I concentrate on so
that it works efficiently on a device
where memory is limited?
The application throws open a form
to enter new todo task. For now, I
would like to save the newly added
task to my-todos.xml once the user
presses "save" button. What are the
common ways to append such a change
to an existing XML file?(note that I don't want to read the whole file again and then persist)
For storing: SQLite seems like a good solution for things such as searching and cross platform support. Android and many other devices support SQLite.
As with any programming question there are a lot of ways to do things. However, by specifying that you are intending to go to a phone, you list of considerations changes. Firstly you need to look at your intended phones to see what they support. Especially in terms of data storage.
Xml or some other flat file format will work fine if you don't have too much data and don't want to enable searching and other functions which will access the data in random ways.
But if you want to store larger amounts of data or do random access, you need to look into data storage techniques that are more database like. This is where you intended target platforms are likely to impose limits in terms of performance or storage limits.
The other alternative is that you design the application so that it's storage os decoupled from the core program. This means that you can apply different types of data storage, depending on whether it's a PC or phone, yet not have to recode everything else.
One option that comes to mind is an in-memory DB, which exists in various flavors. I've yet to use one of these, so I can't tell you about memory usage or platform constraints. Still, it's worth looking at.
Another option that comes to mind is to maintain a large collection of TodoItem objects, and write your own code to read from and persist this collection to the XML file. Essentially, build a class that contains the large Map (or whatever you decide to use) and have this class implement Externalizable.
Both of these options will allow you to read the XML file to its in-memory representation, search and alter the state, and eventually write the final state back to XML when the app goes down (or at fixed intervals, whatever you decide).
You might be able to use java.util.prefs.Preferences.