Develop Reference

Java Temporary file for Memory management

I have a huge data that has to be displayed in a page retrieved from database table. The table is regularly updated with new data and page has to display the latest data whenever refreshed. Since, the page is loaded with heavy loads of data it takes time to load (2-3minutes) and sometime it suffers from only displaying white page.
So, what I need to do is create a temporary file in server which stores this data from database and the page will extract info from this file. This file should be reloaded with new data in each 10-15 minutes so that till 15 mins page displays the old data but after 15 minutes if page is refreshed it shows the new data.
Hope my problem will be solved with the above idea but I am only limited to above theory. Please provide me solution how it can be done and any other better idea than the above. if any java code sample that would be great to solve my problem.
Thanks

Now you are propably using a lightweight webserver (guessing from the fact you are talking about pages loading white)
But if were using some java ee application server like glassfish, you could have a scheduled task running every 15minutes that loads the data and stores it into a file.
http://docs.oracle.com/javaee/6/tutorial/doc/bnboy.html
Maybe Quartz is the answer for you:
Scheduled task in a web application?
Or maybe you are not using a webserver/appserver at all and my answer completly invalid.

Processing Huge amount of Reporting Data

I have a reporting system where I need to combine data from 3 diffrent servers(systems) show it to users . This works fine as far as there few thousand records but as it increases system times out .
I was thinking to use another approach . When user runs result I will generate a request to generate report to database. A daemon running will pick it up and gather data and generate a report and will email to user a link to reporting data . I can do multiple things here .
1)insert gathered data to database in form where it can be simply displayed to users on page.
2)generate html and store it to database .
Although I think 1st option is best here are there any other suggestions ?

Several thoughts:
Consider using hadoop to make your report processing very scalable
Consider running the reports on an interval (e.g. every hour). When the user requests a report you can immediately show them the most recently generated report.
Definitely do not store html. Instead render it when the user makes the request. By the way, this also gives you the flexibility to show different versions of the reports to different users

Fetching data in small units instead of fetching at a time

I'm currently working on a Struts 2 application with Spring as backend with MySQL database.
Right now fetching 1000+ rows on page load.
I'm using tablesorter jquery plugin for displaying the results,sorting on click and pagination purpose.
But the page is taking lot of time to load and also it hangs in IE !!
Is there any easier way to do this like fetching required data from database as and when the user makes the selection like fetch first 10 rows,sorting data,etc.
Any ideas to make the application work faster are welcome !!

I couldn't tell from the docs if the plugin supported server-side paging or not, but that'd be one thing to consider. It was unclear if its paging plugin had any options for this.
Does the plugin's example with a thousand students crash under the same circumstance?

How to log impressions (and data) in Java for a javascript widget?

I have a javascript widget the loads JSON data from a Java webapp. I want to record impressions, and the ids of the data I return (5 or 10 longs).
I want to be able to process these and give a summary of how many impressions a person got on their widget, and record how many times each piece of data was seen.
I'm guessing it's not a great idea to store it all in my postgres database since it could probably be a lot of data (perhaps tens of impressions per second).
Right now I'm considering just writing it all to a log file and processing it every hour.
Is there a better way? Maybe with some kind of message system, or event system?

Writing to log and processing off-line should be ok.
You can program your logging system to create hourly log files, then process files that are not written-to any more. You can do background processing in separate thread/process at low priority.

I'm not sure what you are after, but maybe it's a good solution to do call to an web-analytics (.e.g. Google Analytics) tool from the javascript part of your widget?
You won't have access to your raw data, but it will scale wonderful and you'll have nice graphs.

How do I handle large resultsets in Crystal Reports using Java?

Problem Definition
I have a performance problem displaying large/huge resultset of data on a crystal report. The report takes about 4 minutes or more depending on the resultset size.
How do you handle large resultsets in Crystal Reports without a performance issue?
Environment
Crystal Reports XI
Apache WebSvr 2.X
Jboss 4.2.3
Struts
Java Reporting Component (JRC)
Crystal Report Viewer (CRV)
Firefox
Details
I use the CRXI thick client to build my report (.rpt) and then use it in my webapplication (webapp) under Jboss.
User specifies the filter criteria to generate a report (date range etc) and submits the request to the webapp. Webapp queries the database, gets a "resultset".
I initialize the JRC and CRV according to all the specifications and finally call the "processHttpRequest" method of Crystal Report Viewer to display the report on browser.
So.....
Request received to generate a report with a filter criteria
Query DB to get resultset
Initialize JRC and CRV
finally display the report by calling
reportViewer.processHttpRequest(request, response, request.getSession().getServletContext(), null);
The performance problem is within the last step. I put logs everywhere and noticed that database query doesnt take too long to return resultset. Everything processes pretty quickly till I call the processHttpRequest of CRV. This method just hangs for a long time before displaying the report on browser.
CRV runs pretty fast when the resultset is smaller, but for large resultset it takes a long long time.
I do have subreports and use Crystal report formulas on the reports. Some of them are used for grouping also. But I dont think Subreports is the real culprit here. Because I have some other reports that dont have any subreports, and they too get really slow displaying large resultsets.
Solutions?
So obviously I need a good solution to this generic problem of "How do you handle large resultsets in Crystal Reports?"
I have thought of some half baked ideas.
A) Use external pagination and fetch data only for the current page being displayed. But for this, CRXI must allow me to create my own buttons (previous, next, last), so I can control the click event and fetch data accordingly. I tried capturing events by registering event handler "addToolbarCommandEventListener" of CRV. But my listener gets invoked "after" processHttpRequest method completes, which doesnt help.
Some how I need to be able to control the UI by adding my own previous page, next page, last page buttons and controlling it's click events.
B) Have CRXI use a jquery type of functionality, to allow browser side page navigation. So maybe the first time it'll take 5 mins to display the report, but once it's displayed, user can go to any page without sending the request back to server.
C) Try using Crystal Reports 2008. I'm open to using this version, but I couldnt figureout if it has any features that can help me.
D) Use BIRT or JasperReports, since they apparently have the ability to handle large datasets, by caching a part of the dataset on disk etc...
E) Will using the Crystal Reports Servers like cache server, application server etc help in any way? I read a little on the Crystal Page Viewer, Interactive Viewer, Part Viewer etc....but I'm not sure if any of these things are going to solve the issue.
I'd appreciate it if someone can point me in the right direction.