I have 2 requirements, both involving data-transfer.
In the first one I have an application 'A' which has data in its staging table of database, this data has to be transferred to another application 'B'. Here there are around 50k-60k records that need to be transferred daily. I want to know what is the best way to transfer this? should i still follow the old school method of creating a file and transferring it or go for web-API's or anything else? Please suggest anything in JAVA.
In the second one, i have a file that needs to be transferred from app 'A' to app 'B'. The file size is of around 10GB. what is the best way of achieving it? I had a look at IBM ESB, is there any other better alternate?
This is best done outside of any API because it will likely take quite a bit of time.
My suggestion is to export the data to csv, zip it up, transfer it to the other system via ftp for example.
On the second system you would need something which monitors the folder where you ftp stuff and this other thing would take care of getting the data into the second system. I have setup such a system using the Azure cloud for example using Webjobs and it worked very well.
You can also come up with something else which is appropriate to your specific case.
Related
I am developing an API in Java. It is basically a java servlet that returns content in json (application/json). Using a Tomcat server. One of the field in the response is supposed to be a link to a downloadable .txt file.
I wonder what is the best way to deliver this file:
Generating this file on every request seems to me killer, even having some cron to clean directories with files
Any way to give a temporary link only while that request for a period without saving to the file system?
Thank you.
If you say writing to the file system would kill your application, then I deduce from that that your IO performance is too weak for that, right? I mean, if you even would not have the storage capacity for that, then your infrastructure is not suitable for your application at all. I can only see four other ways for solving that problem (but maybe there are more, my list is not exclusive):
Store the text file in a database. The database should also store timeout information. Good if there are more than 1 application servers and there is a load balancer in front of them (but all application servers share the same database).
Store the text file in RAM, maybe using a cache library which does the cleanup tasks automatically for you - but be aware that a cache library will usually not guarantee a minimum storage time for each file.
Do not store the text file at all, but create it just when it is requested (no idea if that is possible in your application).
Do not provide a link to the text file, but directly include its content in the json answer (of course it would then be escaped as a JSon String), which means your server can directly forget about it when the answer has been sent, but the client _must_ download it without checking if it needs the file or not.
I need to allow for csv-file downloads on my page and I was going to try ngCsv (from Angular) but for browser support this seems fairly limited. I've seen quite a few examples of this being done with vanilla Javascript. And after a discussion with a colleague of "backend vs. frontend" I'm feeling more and more unsure of what to do.
Are there any true optimization/efficiency reasons why I should avoid doing this on the client side (assuming the files are no more than 100MB each download)?
Are there any true optimization/efficiency reasons why I should avoid
doing this on the client side (assuming the files are no more than
100MB each download)?
If the data on the .csv would be the same for each user, and only updated every now and then, I would suggest you have your server create / update a static .csv. It wouldn't be resource-intensive, and you wouldn't have to worry about browser compatibility / user resources.
If, however, the data you need to create a .csv for is different on a per-user basis, then you should consider creating the file client-side. If you can help it, you don't want your server having to dynamically generate 100MB .csv files each time a user clicks the link.
You could write a script that only generates the .csv client-side if the browser is not mobile and there is web-worker support. If either of those conditions are not met, you could fall back to having your server do it.
Ultimately, your answer is going to really depend on the requirements / context of this project. Try to cache the results where possible, and use common sense. Good luck :)
I consider myself an accomplished programmer, but I'm relatively new to Android App development. I'm creating an application that will store information into the SQLite database used in Android. What I'd like to be able to do is be able to take a query of that data and export it either as a file of some sort or just send it to another iteration of the same application on a different phone. Then be able to have that phone import the same information into its own database, seeing that the information should line up correctly as long as it keeps it.
Can anyone provide some good starting examples of how I would best go about this and/or tutorials on how to go about doing it? Right now I'm just not sure how to get started and I could use some help to push me in the right direction, so I'd really appreciate the help.
Thank you ahead of time to anyone who replies.
and export it either as a file
depends on what kind of data you have. You can write any kind of text-based data using a RandomAccessFile for example.
send it to another iteration of the same application on a different
phone
You will need to have your own backend to do so. You could identify the target device by using GCM
I wish to be able to keep a score that a player gets when playing my game from game to game and to when they close the game and re-open the scores are still saved. The only way I can think of is to do so using a text file, like I would of done in VB6. However, that then means that they can edit the text file? Or not? My score is stored in a "double" that can be accessed from any class and is being transferred around classes as it is, if that makes a difference.
Hope someone can suggest the best way to go about this.
If keeping the score secret and non editable is very important, I suggest you either store the score on of all players a secure server that only you control, or if that is outside the scope of your project, use an encryption method and also store the score as binary data (i.e. store your gamestate object, not the score itself) instead of a text file.
Any app. that has a GUI can be launched using Java Web Start & use the PersistenceService. Data in the persistence service is not easily accessible to the end user. Here is a small demo. of the persistence service.
As to how to store the data, If it is not absolutely vital to prevent the user from altering it, I would use a Properties object or XML/POJO.
If it is very important (e.g. gamers competing for a $10,000 prize), encrypt the values, then go with the remote server, encrypted (etc.).
You can encrypt the file using one way or another, so it will not be easily editable (and editing attempts may corrupt the score at all, consequently.) Here is a simple example of AES string encryption.
If you store the file on the local machine, obviously every user that have read/write permissions on that file could modify it.
I suggest you to follow one of these ways
Encrypt the file and decrpyt it on open
Save it onto a remote file onto a server
Use a DB
What have you tried? It seems to be some kind of homework for me.
You can read and write Files with Java. You can also do object-serialization or use an embedded database.
update:
I would suggest to store all information within a database at the server. There are many way to do this. The concrete implementation would depend in your backend.
I've been tasked with implementing large (2gb+) file uploads via a web browser. After evaluating various technologies, java applets seem to be the way forward (only one's which provide proper access to the local disk). I was wondering if anyone can recommend a 3rd party file upload app we can use as a base? requirements are
Decent UI, ideally we want something similar to facebooks photo uploader
Can handle large (2gb+) files
Resumable uploads
We beed the source to extend it to our needs (dont mind paying extra)
You're probably looking for JUpload.
Update: not sure if it has as nice of UI as you're hoping, but unless you want to build a custom solution like I have it's your best option.
Just a tip, maybe it is obvious, i don't know :P
It is nice to send the big file in chunks like 2mb, and on the server side you just append the bytes to the target file. The server knows what bytes it needs, and if a upload is aborted and continued later, the server can just send a message about from what bytes to start uploading the file again. Then we get resumability (is it a word? :P) and safety of large HTTP-uploads (since, in fact, we are sending many smuller uploads, and each upload is checked to be of the correct size on the server).
We wrote an implementation like this once with a Java-applet as the client and PHP on the server, I'll see if I can dig it out as a reference for you :p
Not really a solution : from experience you may bump into the following issues:
problems when uploading over HTTPs
problems uploading through proxies
Just wanted to make you aware of these two cases, for you to test when evaluating a solution.
Hope, you will get solutions for your prob over here.. http://jupload.sourceforge.net/