Develop Reference

MediatorLiveData.getValue().isEmpty() returns false after delete directly

How can I check if a table is empty knowing that I'm not in the Activity with a guaranteed or synchronized returned value??
Hey guys, I'm new to architecture components.
I having a very strange problem with MediatorLiveData..
all I need to do is to delete all values in a Room Entity using background service "AlarmManager"
and postValue(null) to the MediatorLiveData,
then I need to call refreshTables() which check if there is no data in the table it calls a webservice and inserts the values.
everything works fine but when I watched the calling of refreshTables() after the deleting, sometimes the condition getValue().isEmpty() returns false!!
when this thing happens it leaves the table without data.
I believe there is a delay in MediatorLiveData observing.
Notes:
1- I'm calling refreshTables() in a thread (executor) after the deleting, of course, refreshTables() executes using its executor.
2- this fault happens almost every 20 times of working fine.
3- I tried to override onChange but an exception was thrown because it cannot cast LiveData to LifeCycleOwner
4- TableEntity is just a Room Entity contains fields about a restaurant table
//the constructor
private DataRepository(AppDatabase database
) {
this.mDatabase = database;
executor = new AppExecutors();
mObserableTables = new MediatorLiveData<>();
mObserableTables.addSource(mDatabase.tableDao().loadAllTables(),
tableEntities -> {
if (mDatabase.getDatabaseCreated().getValue() != null)
mObserableTables.postValue(tableEntities);
});
}
private MediatorLiveData<List<TableEntity>> mObserableTables;
public void refereshTables()
{
executor.networkIO().execute(()->
{
try {
if(mObserableTables.getValue()==null || mObserableTables.getValue().isEmpty()) {//<-the problem
List<TableEntity> tableEntities = new WebserviceCall().getTablesOnline();
mDatabase.tableDao().insertOrReplaceAllTables(tableEntities);
}
}catch (IOException e)
{
Log.d(this.getClass().getName(),e.getMessage());
}
});
}
public void refreshTablesForService()
{
executor.diskIO().execute(()->
{
int deletedRows=mDatabase.tableDao().deleteAllTables();
Log.d(this.getClass().getName()
,"deleted rows from tables= "+String.valueOf(deletedRows));
mObserableTables.postValue(null);
refereshTables();
});
}

answering my own question :)
after hours of deep digging, I discovered that nothing to do getValue().
the problem was in .postValue() method which is used UiThread to put the value in LiveData. this for sure requires adding a task to UI which may be delayed and checking the data before it finished this task leads to inconsistency.
even trying to load all data from the table was providing inconsistency information.
trying to removeSource() will do nothing about this problem too.
I managed to handle this issue by using "SELECT COUNT(*) FROM tables" query which gives me accurate information.
btw, adding "Thread.sleep(100)" before calling refereshTables(); fixes the problem but give a very bad user experience.
a fantastic explanation allowed me to figure this out is https://code.tutsplus.com/tutorials/android-architecture-component-livedata--cms-29317

Getting CPU 100 percent when I am trying to downloading CSV in Spring

I am getting CPU performance issue on server when I am trying to download CSV in my project, CPU goes 100% but SQL returns the response within 1 minute. In the CSV we are writing around 600K records for one user it is working fine but for concurrent users we are getting this issue.
Environment
Spring 4.2.5
Tomcat 7/8 (RAM 2GB Allocated)
MySQL 5.0.5
Java 1.7
Here is the Spring Controller code:-
#RequestMapping(value="csvData")
public void getCSVData(HttpServletRequest request,
HttpServletResponse response,
#RequestParam(value="param1", required=false) String param1,
#RequestParam(value="param2", required=false) String param2,
#RequestParam(value="param3", required=false) String param3) throws IOException{
List<Log> logs = service.getCSVData(param1,param2,param3);
response.setHeader("Content-type","application/csv");
response.setHeader("Content-disposition","inline; filename=logData.csv");
PrintWriter out = response.getWriter();
out.println("Field1,Field2,Field3,.......,Field16");
for(Log row: logs){
out.println(row.getField1()+","+row.getField2()+","+row.getField3()+"......"+row.getField16());
}
out.flush();
out.close();
}}
Persistance Code:- I am using spring JDBCTemplate
#Override
public List<Log> getCSVLog(String param1,String param2,String param3) {
String sql =SqlConstants.CSV_ACTIVITY.toString();
List<Log> csvLog = JdbcTemplate.query(sql, new Object[]{param1, param2, param3},
new RowMapper<Log>() {
#Override
public Log mapRow(ResultSet rs, int rowNum)
throws SQLException {
Log log = new Log();
log.getField1(rs.getInt("field1"));
log.getField2(rs.getString("field2"));
log.getField3(rs.getString("field3"));
.
.
.
log.getField16(rs.getString("field16"));
}
return log;
}
});
return csvLog;
}

I think you need to be specific on what you meant by "100% CPU usage" whether it's the Java process or MySQL server. As you have got 600K records, trying to load everything in to memory would easily end up in OutOfMemoryError. Given that this works for one user means that you've got enough heap space to process this number of records for just one user and symptoms surface when there are multiple users trying to use the same service.
First issue I can see in your posted code is that you try to load everything into one big list and the size of the list varies based on the content of the Log class. Using a list like this also means that you have to have enough memory to process JDBC result set and generate new list of Log instances. This can be a major problem with a growing number of users. This type of short-lived objects will cause frequent GC and once GC cannot keep up with the amount of garbage being collected it fails obviously. To solve this major issue my suggestion is to use ScrollableResultSet. Additionally you can make this result set read-only, for example below is code fragment for creating a scrollable result set. Take a look at the documentation for how to use it.
Statement st = conn.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE, ResultSet.CONCUR_READ_ONLY);
Above option is suitable if you're using pure JDBC or SpringJDBC template. If Hibernate is already used in your project you can still achieve the same this with the below code fragment. Again please check the documentation for more information and you have a different JPA provider.
StatelessSession session = sessionFactory.openStatelessSession();
Query query = session.createSQLQuery(queryStr).setCacheable(false).setFetchSize(Integer.MIN_VALUE).setReadOnly(true);
query.setParameter(query_param_key, query_paramter_value);
ScrollableResults resultSet = query.scroll(ScrollMode.FORWARD_ONLY);
This way you're not loading all the records to Java process in one go, instead you they're loaded on demand and will have small memory footprint at any given time. Note that JDBC connection will be open until you're done with processing the entire record set. This also means that your DB connection pool can be exhausted if many users are going to download CSV files from this endpoint. You need to take measures to overcome this problem (i.e use of an API manager to rate limit the calls to this endpoint, reading from a read-replica or whatever viable option).
My other suggestion is to stream data which you have already done, so that any records fetched from the DB are processed and sent to client before the next set of records are processed. Again I would suggest you to use a CSV library such as SuperCSV to handle this as these libraries are designed to handle a good load of data.
Please note that this answer may not exactly answer your question as you haven't provided necessary parts of your source such as how to retrieve data from DB but will give the right direction to solve this issue

Your problem in loading all data on application server from database at once, try to run query with limit and offset parameters (with mandatory order by), push loaded records to client and load next part of data with different offset. It help you decrease memory footprint and will not required keep connection to database open all the time. Of course, database will loaded a bit more, but maybe whole situation will better. Try different limit values, for example 5K-50K and monitor cpu usage - on both app server and database.
If you can allow keep many open connection to database #Bunti answer is very good.
http://dev.mysql.com/doc/refman/5.7/en/select.html

start and stop stream consumption in php script from my java frontend

i am trying to make a project where the user inputs a couple of brands and receives feedback of their mentioned times in twitter.. so far i ve made/found/tweaked the basic frontend in java, the php scripts for consuming the twitter streaming api plus a database to save the incoming tweets,, the last two in a xaamp server locally. Its working perfectly but i must run the scripts manually through my browser.
what i need is
php scripts to run in the backround (as their creator insists )
start and stop scripts at will to change and renew requested keywords
and i need this to be done from my frontend.
i thought of this..
since in the first place i send user inputs to the scripts through my database (java arrayOfBrands => db table => php requests and gets a column as array ) i could do smth like a flag in mysql database changable by my frontend and critical in the execution of my script.
then i looked at this question
Starting / Stopping php script running in background from browser
but the guy here proposes to do it as a cronjob... the other guy who made the scripts says about a thousand times in the instructions "DONT DO IT AS A CRONJOB" but as a procedure in the backround... and they both seem good and pro(Adam Green 140dev and Aziz Saleh 1763 fame here ) and i dont know which way to go..
here is "my" code
<?php
/**
* get_tweets.php
* Collect tweets from the Twitter streaming API
* This must be run as a continuous background process
* Latest copy of this code: http://140dev.com/free-twitter-api-source-code-library/
* #author Adam Green <140dev#gmail.com>
* #license GNU Public License
* #version BETA 0.30
*/
ini_set('display_errors', true);
require_once('140dev_config.php');
require_once('brands.php');
require_once('../libraries/phirehose/Phirehose.php');
require_once('../libraries/phirehose/OauthPhirehose.php');
class Consumer extends OauthPhirehose
{
// A database connection is established at launch and kept open permanently
public $oDB;
public function db_connect() {
require_once('db_lib.php');
$this->oDB = new db;
}
// This function is called automatically by the Phirehose class
// when a new tweet is received with the JSON data in $status
public function enqueueStatus($status) {
$tweet_object = json_decode($status);
// Ignore tweets without a properly formed tweet id value
if (!(isset($tweet_object->id_str))) { return;}
$tweet_id = $tweet_object->id_str;
// If there's a ", ', :, or ; in object elements, serialize() gets corrupted
// You should also use base64_encode() before saving this
$raw_tweet = base64_encode(serialize($tweet_object));
$field_values = 'raw_tweet = "' . $raw_tweet . '", ' .
'tweet_id = ' . $tweet_id;
$this->oDB->insert('json_cache',$field_values);
}
}
// Open a persistent connection to the Twitter streaming API
$stream = new Consumer(OAUTH_TOKEN, OAUTH_SECRET, Phirehose::METHOD_FILTER);
// Establish a MySQL database connection
$stream->db_connect();
// The keywords for tweet collection are entered here as an array
// More keywords can be added as array elements
// For example: array('recipe','food','cook','restaurant','great meal')
$stream->setTrack($mybrands);
// Start collecting tweets
// Automatically call enqueueStatus($status) with each tweet's JSON data
$stream->consume();
this is the script i need to run/not run
plz help me in any way possible either code or advice or just a tip are valuable for me.
thanks in advance.

Ideas on concurrent datastructure

I am not sure if i can put my question in the clearest fashion but i will try my best.
Lets say i am retrieving some information from a third party api. The retrieved information will be huge in size. To have a performance gain, instead of retrieving all the info in one go, i will be retrieving the info in a paged fashion (the api gives me that facility, basically an iterator). The return type is basically a list of objects.
My aim here is to process the information i have in hand(that includes comparing and storing in db and many other operations) while i get paged response on the request.
My question here to the expert community is , what data structure do you prefer in such case. Also does a framework like spring batch help you in getting performance gains in such cases.
I know the question is a bit vague, but i am looking for general ideas,tips and pointers.

In these cases, the data structure for me is java.util.concurrent.CompletionService.
For purposes of example, I'm going to assume a couple of additional constraints:
You want only one outstanding request to the remote server at a time
You want to process the results in order.
Here goes:
// a class that knows how to update the DB given a page of results
class DatabaseUpdater implements Callable { ... }
// a background thread to do the work
final CompletionService<Object> exec = new ExecutorCompletionService(
Executors.newSingleThreadExecutor());
// first call
List<Object> results = ThirdPartyAPI.getPage( ... );
// Start loading those results to DB on background thread
exec.submit(new DatabaseUpdater(results));
while( you need to ) {
// Another call to remote service
List<Object> results = ThirdPartyAPI.getPage( ... );
// wait for existing work to complete
exec.take();
// send more work to background thread
exec.submit(new DatabaseUpdater(results));
}
// wait for the last task to complete
exec.take();
This just a simple two-thread design. The first thread is responsible for getting data from the remote service and the second is responsible for writing to the database.
Any exceptions thrown by DatabaseUpdater will be propagated to the main thread when the result is taken (via exec.take()).
Good luck.

In terms of doing the actual parallelism, one very useful construct in Java is the ThreadPoolExecutor. A rough sketch of what that might look like is this:
public class YourApp {
class Processor implements Runnable {
Widget toProcess;
public Processor(Widget toProcess) {
this.toProcess = toProcess;
}
public void run() {
// commit the Widget to the DB, etc
}
}
public static void main(String[] args) {
ThreadPoolExecutor executor =
new ThreadPoolExecutor(1, 10, 30,
TimeUnit.SECONDS,
new LinkedBlockingDeque());
while(thereAreStillWidgets()) {
ArrayList<Widget> widgets = doExpensiveDatabaseCall();
for(Widget widget : widgets) {
Processor procesor = new Processor(widget);
executor.execute(processor);
}
}
}
}
But as I said in a comment: calls to an external API are expensive. It's very likely that the best strategy is to pull all the Widget objects down from the API in one call, and then process them in parallel once you've got them. Doing more API calls gives you the overhead of sending the data all the way from the server to you, every time -- it's probably best to pay that cost the fewest number of times that you can.
Also, keep in mind that if you're doing DB operations, it's possible that your DB doesn't allow for parallel writes, so you might get a slowdown there.

Java Multithreaded I/0 and communication problem

I am using java to create an application for network management. In this application I establish communication with network devices using SNMP4j library (for the snmp protocol). So, Im supposed to scan certain values of the network devices using this protocol and put the result into a file for caching. Up in some point I decided to make my application multi-threaded and assign a device to a thread. I created a class that implements the runnable interface and then scans for the values that I want for each device.
When i run this class alone it, works fine. but when I put multiple threads at the same time the output mess up, it prints additional or out of order output into the files. Now, i wonder if this problem is due to the I/O or due to the communication.
Here I'll put some of the code so that you can see what im doing and help me figure what's wrong.
public class DeviceScanner implements Runnable{
private final SNMPCommunicator comm;
private OutputStreamWriter out;
public DeviceScanner(String ip, OutputStream output) throws IOException {
this.device=ip;
this.comm = new SNMPV1Communicator(device);
oids=MIB2.ifTableHeaders;
out = new OutputStreamWriter(output);
}
#Override
public void run(){
//Here I use the communicator to request for desired data goes something like ...
String read=""
for (int j=0; j<num; j++){
read= comm.snmpGetNext(oids);
out.write(read);
this.updateHeaders(read);
}
out.flush();
//...
}
}
some of the expected ooutput would be something like:
1.3.6.1.2.1.1.1.0 = SmartSTACK ELS100-S24TX2M
1.3.6.1.2.1.1.2.0 = 1.3.6.1.4.1.52.3.9.1.10.7
1.3.6.1.2.1.1.3.0 = 26 days, 22:35:02.31
1.3.6.1.2.1.1.4.0 = admin
1.3.6.1.2.1.1.5.0 = els
1.3.6.1.2.1.1.6.0 = Computer Room
but instead i get something like (varies):
1.3.6.1.2.1.1.1.0 = SmartSTACK ELS100-S24TX2M
1.3.6.1.2.1.1.2.0 = 1.3.6.1.4.1.52.3.9.1.10.7
1.3.6.1.2.1.1.4.0 = admin
1.3.6.1.2.1.1.5.0 = els
1.3.6.1.2.1.1.3.0 = 26 days, 22:35:02.31
1.3.6.1.2.1.1.6.0 = Computer Room
1.3.6.1.2.1.1.1.0 = SmartSTACK ELS100-S24TX2M
1.3.6.1.2.1.1.2.0 = 1.3.6.1.4.1.52.3.9.1.10.7
*Currently I have one file per device scanner desired.
i get them from a list of ip , it looks like this. Im also using a little threadpool to keep a limited number of threads at the same time .
for (String s: ips){
output= new FileOutputStream(new File(path+s));
threadpool.add(new DeviceScanner(s, output));
}

I suspect SNMPV1Communicator(device) is not thread-safe. As I can see it's not a part of SNMP4j library.

Taking a wild guess at what's going on here, try putting everything inside a synchronized() block, like this:
synchronized (DeviceScanner.class)
{
for (int j=0; j<num; j++){
read= comm.snmpGetNext(oids);
out.write(read);
this.updateHeaders(read);
}
out.flush();
}
If this works, my guess is right and the reason for the problems you're seeing is that you have many OutputStreamWriters (one on each thread), all writing to a single OutputStream. Each OutputStreamWriter has its own buffer. When this buffer is full, it passes the data to the OutputStream. It's essentially random when each each OutputStreamWriter's buffer is full - it might well be in the middle of a line.
The synchronized block above means that only one thread at a time can be writing to that thread's OutputStreamWriter. The flush() at the end means that before leaving the synchronized block, the OutputStreamWriter's buffer should have been flushed to the underlying OutputStream.
Note that synchronizing in this way on the class object isn't what I'd consider best practice. You should probably be looking at using a single instance of some other kind of stream class - or something like a LinkedBlockingQueue, with all of the SNMP threads passing their data over to a single file-writing thread. I've added the synchronized as above because it was the only thing available to synchronize on within your pasted example code.

You've got multiple threads all using buffered output, and to the same file.
There's no guarantees as to when those threads will be scheduled to run ... the output will be fairly random ordered, dictated by the thread scheduling.