I'm developing a Javafx application, that synchronizes some data from two different databases.
In the call method I get all the data and store it in an ArrayList. Then I loop through the ArrayList and I try to get that same data from the second database.
If it exists I compare it for differences and if there are differences I update it. Otherwise, if it dosen't exist, I insert it via a DAO object method.
The problem is that sometimes the second database takes some time to provide the response so the process continues its execution and the new data will be compared with old data.
My question is, how can I stop the process until the data has all been fetched and then proceed to the synchronization logic?
#Override
protected Map call() throws Exception {
Map<String, Integer> m = new HashMap();
updateTitle( "getting the data ..." );
int i, updated = 0, inserted = 0;
// creating first database instance
DAOFactory db1Dao = DAOFactory.getInstance( "db1" );
//creating the first database dataObject instance
Db1EmployerDAO empDb1Dao = db1Dao.getDAODb1Employer();
// creating second database instance
DAOFactory db2Dao = DAOFactory.getInstance( "db2" );
//creating the second database dataObject instance
Db2EmployeurDAO empDb2Dao = db2Dao.getDAODb2Employer();
Employer emp;
// getting all the object
List< Employer > LEmpDb1 = empDb1Dao.getAll();
updateTitle( "Data proccessing ..." );
//for each data in the list
for( i = 1; i <= LEmpDb1.size(); i++ ){
if( isCancelled() )
break;
updateMessage( "Processing employer : "+ LEmpDb1.get( i-1 ).getNemploy() +" "+ LEmpDb1.get( i-1 ).getRaison() );
//trying to get the object from the second database which the
//process sometimes pass befor the result is getting which is my problem
emp = empDb2Dao.getEmployerByNo( LEmpDb1.get( i-1 ).getNemploy() );
if( emp != null){
if( !LEmpDb1.get( i-1 ).equals( emp ) )
if( empDb2Dao.update( LEmpDb1.get( i-1 ) ) ){
updated++;
LOG.log( "MAJ employeur : "+ LEmpDb1.get( i ).getNemploy()+" => "+LEmpDb1.get( i ).getDifferences( emp ) );
}
} else {
if( empDb2Dao.insert( LEmpDb1.get( i-1 ) ) )
inserted++;
}
updateProgress( i, LEmpDb1.size() );
}
m.put( "upd", updated );
m.put( "ins", inserted );
m.put( "all", LEmpDb1.size() );
return m;
}
The getEmployerByNo method
public synchronized Employer getEmployerByNo( String no_emp ) throws DAOException {
Employeur emp = null;
Connection con = null;
PreparedStatement stm = null;
ResultSet res = null;
try{
con = dao.getConnection();
stm = preparedRequestInitialisation( con, GET_BY_NO_SQL, no_emp );
res = stm.executeQuery();
if( res.next() ){
//map is a function that map the database resultset data with the object properties
emp = map( res );
LOG.info( "getting the employer : "+ no_emp );
}
} catch( SQLException e ){
throw new DAOException( e.getLocalizedMessage() );
} finally{
silentClose( res, stm, con );
}
return emp;
}
Look into using an ExecutorService and Future.get() as needed to wait for completion. See the documentation here and here. Here is a more-or-less complete example:
public class Application implements Runnable {
private final ExecutorService pool = Executors.newCachedThreadPool();
public void run() {
Dao firstDao = new DaoImpl();
Dao secondDao = new AnotherDaoImpl();
FetchAllTask fetchAll = new FetchAllTask(firstDao);
Future<?> fetchAllFuture = pool.submit(fetchAll);
try {
fetchAllFuture.get();
} catch (Exception e) {
// TODO handle
System.out.println("An exception occurred!");
e.printStackTrace();
}
ConcurrentSkipListSet<AnObject> items = fetchAll.getItems();
Iterator<AnObject> it = items.iterator();
while (it.hasNext()) {
// insert your cancellation logic here
// ...
AnObject daoObj = it.next();
FetchOneTask fetchOne = new FetchOneTask(secondDao, daoObj.getId());
Future<?> fetchOneFuture = pool.submit(fetchOne);
try {
fetchOneFuture.get();
AnObject anotherDaoObj = fetchOne.getAnObject();
if (anotherDaoObj == null) {
// the object retrievied by the first dao (first datasource)
// is not in the second; it needs to be inserted into the second
System.out.println(String.format("Inserting %s", daoObj));
secondDao.insert(daoObj);
} else {
System.out.println(String.format("Updating %s to %s", anotherDaoObj, daoObj));
secondDao.update(daoObj);
}
} catch (Exception e) {
System.out.println("An exception occurred!");
e.printStackTrace();
}
}
Set<AnObject> itemsInSecondDb = secondDao.fetchAll();
for (AnObject o : itemsInSecondDb) {
System.out.println(o);
}
pool.shutdown();
}
// ... invoke the app thread from somewhere else
}
Related
I'm working on a bot for supergroups.
How I can verify if member is admin?
my lib:org.telegram 4.2 https://core.telegram.org/bots/api
I tried with ChatMember, GetChatAdministrators and SendMessage methods but I have no idea how insert parameters because they don't ask them but they have only .get option (null respose). Only GetChatAdministrators permit a .set method for ChatID but it give error
GetChatAdministrators getadmin = new GetChatAdministrators().setChatId(ChatIDSupergroup);
ArrayList<ChatMember> s = null;
try {
s = getadmin.deserializeResponse(null); //Unable to deserialize response
} catch (TelegramApiRequestException e) {
e.printStackTrace();
}
ChatMember member = new ChatMember(); //There are only get options
String status=member.getStatus(); //null
I had a same problem but I couldn't find an answer from anywhere. Then I tried myself and finally today I found the following solution:
public List<ChatMember> getChatAdministrators(Long chatId){
List<ChatMember> chatAdministrators = Collections.emptyList();
try {
chatAdministrators = execute(new GetChatAdministrators(String.valueOf(chatId)));
} catch (TelegramApiException e) {
e.printStackTrace();
}
return chatAdministrators;
}
This method returns list of administrators of telegram groups, supergroups or channels.
function adminCheck( id, chat_id ) {
var bAdminCheck = false;
var name = "";
var aChatMember = getChatAdministrators( chat_id );
var contents = JSON.parse( aChatMember );
var i = 0;
while( !bAdminCheck && ( i < contents.result.length ) ) {
if( id == contents.result[i].user.id ) {
bAdminCheck = true;
}
i++;
}
return {
AdminCheck: bAdminCheck,
};
}
Available methods
source:
https://core.telegram.org/bots/api#available-methods
getChatAdministrators
Use this method to get a list of administrators in a chat. On success, returns an Array of ChatMember objects that contains information about all chat administrators except other bots. If the chat is a group or a supergroup and no administrators were appointed, only the creator will be returned.
I have a Watcher that updates my data structures when a change is heard. However, if the change is not instantaneous (i.e. if a large file is being copied from another file system, or a big part of the file is modified), the data-structure tries to update too early and throws an error.
How can I modify my code so that updateData() is called after only the last ENTRY_MODIFY is called, rather than after every single ENTRY_MODIFY.
private static boolean processWatcherEvents () {
WatchKey key;
try {
key = watcher.poll( 10, TimeUnit.MILLISECONDS );
} catch ( InterruptedException e ) {
return false;
}
Path directory = keys.get( key );
if ( directory == null ) {
return false;
}
for ( WatchEvent <?> event : key.pollEvents() ) {
WatchEvent.Kind eventKind = event.kind();
WatchEvent <Path> watchEvent = (WatchEvent<Path>)event;
Path child = directory.resolve( watchEvent.context() );
if ( eventKind == StandardWatchEventKinds.ENTRY_MODIFY ) {
//TODO: Wait until modifications are "finished" before taking these actions.
if ( Files.isDirectory( child ) ) {
updateData( child );
}
}
boolean valid = key.reset();
if ( !valid ) {
keys.remove( key );
}
}
return true;
}
As #TT suggested, you can do it pretty easily with file locks.
When you get an event, use a blocking method lock() on read and write access. Hence the operation is blocking, the code automatically waits until the write operation is finished.
FileChannel channel = new RandomAccessFile(file, "rw").getChannel();
try (channel) { // auto closable, uses channel.close() in finally block
channel.lock(); // wait until file modifications are finished
channel.read(...); // now you can safely read the file
}
However, this won't work between different JVM processes, because they don't share the same lock.
Is your problem can be solved by using timestamp.
Create a map for storing the timestamp to the map.
Map<Path, Long> fileTimeStamps;
For process event check last modified timestamp.
long oldFileModifiedTimeStamp = fileTimeStamps.get(filePath);
long newFileModifiedTimeStamp = filePath.toFile().lastModified();
if (newFileModifiedTimeStamp > oldFileModifiedTimeStamp)
{
fileTimeStamps.remove(filePath);
onEventOccurred();
fileTimeStamps.put(filePath, filePath.toFile().lastModified());
}
I ended up writing a thread that keeps a list of things I want updated and delays actually updating them until 80 milliseconds have passed. Whenever an ENTRY_MODIFY event happens, it resets the counter. I think this is a good solution, but there may be better?
#SuppressWarnings({ "rawtypes", "unchecked" })
private static boolean processWatcherEvents () {
WatchKey key;
try {
key = watcher.poll( 10, TimeUnit.MILLISECONDS );
} catch ( InterruptedException e ) {
return false;
}
Path directory = keys.get( key );
if ( directory == null ) {
return false;
}
for ( WatchEvent <?> event : key.pollEvents() ) {
WatchEvent.Kind eventKind = event.kind();
WatchEvent <Path> watchEvent = (WatchEvent<Path>)event;
Path child = directory.resolve( watchEvent.context() );
if ( eventKind == StandardWatchEventKinds.ENTRY_CREATE ) {
if ( Files.isDirectory( child ) ) {
loadMe.add( child );
} else {
loadMe.add( child.getParent() );
}
} else if ( eventKind == StandardWatchEventKinds.ENTRY_DELETE ) {
//Handled by removeMissingFiles(), can ignore.
} else if ( eventKind == StandardWatchEventKinds.ENTRY_MODIFY ) {
System.out.println( "Modified: " + child.toString() ); //TODO: DD
if ( Files.isDirectory( child ) ) {
modifiedFileDelayedUpdater.addUpdateItem( child );
} else {
modifiedFileDelayedUpdater.addUpdateItem( child );
}
} else if ( eventKind == StandardWatchEventKinds.OVERFLOW ) {
for ( Path path : musicSourcePaths ) {
updateMe.add( path );
}
}
boolean valid = key.reset();
if ( !valid ) {
keys.remove( key );
}
}
return true;
}
...
class UpdaterThread extends Thread {
public static final int DELAY_LENGTH_MS = 80;
public int counter = DELAY_LENGTH_MS;
Vector <Path> updateItems = new Vector <Path> ();
public void run() {
while ( true ) {
long sleepTime = 0;
try {
long startSleepTime = System.currentTimeMillis();
Thread.sleep ( 20 );
sleepTime = System.currentTimeMillis() - startSleepTime;
} catch ( InterruptedException e ) {} //TODO: Is this OK to do? Feels like a bad idea.
if ( counter > 0 ) {
counter -= sleepTime;
} else if ( updateItems.size() > 0 ) {
Vector <Path> copyUpdateItems = new Vector<Path> ( updateItems );
for ( Path path : copyUpdateItems ) {
Library.requestUpdate ( path );
updateItems.remove( path );
}
}
}
}
public void addUpdateItem ( Path path ) {
counter = DELAY_LENGTH_MS;
if ( !updateItems.contains( path ) ) {
updateItems.add ( path );
}
}
};
I'm trying to integrate calcite with Kafka, I refrenced CsvStreamableTable.
Each ConsumerRecord is convert to Object[] using the fowlloing code:
static class ArrayRowConverter extends RowConverter<Object[]> {
private List<Schema.Field> fields;
public ArrayRowConverter(List<Schema.Field> fields) {
this.fields = fields;
}
#Override
Object[] convertRow(ConsumerRecord<String, GenericRecord> consumerRecord) {
Object[] objects = new Object[fields.size()+1];
int i = 0 ;
objects[i++] = consumerRecord.timestamp();
for(Schema.Field field : this.fields) {
Object obj = consumerRecord.value().get(field.name());
if( obj instanceof Utf8 ){
objects[i ++] = obj.toString();
}else {
objects[i ++] = obj;
}
}
return objects;
}
}
Enumerator is implemented as following,one thread is constantly polling records from kafka and put them into a queue, getRecord() method poll from that queue:
public E current() {
return current;
}
public boolean moveNext() {
for(;;) {
if(cancelFlag.get()) {
return false;
}
ConsumerRecord<String, GenericRecord> record = getRecord();
if(record == null) {
try {
Thread.sleep(200L);
} catch (InterruptedException e) {
e.printStackTrace();
}
continue;
}
current = rowConvert.convertRow(record);
return true;
}
}
I tested SELECT STREAM * FROM Kafka.clicks, it works fine.
rowtime is the first column explicitly added,and the value is record Timestamp of Kafka.
But when I tried
SELECT STREAM FLOOR(rowtime TO HOUR)
AS rowtime,ip,COUNT(*) AS c FROM KAFKA.clicks GROUP BY FLOOR(rowtime TO HOUR), ip
It threw exception
java.sql.SQLException: Error while executing SQL "SELECT STREAM FLOOR(rowtime TO HOUR) AS rowtime,ip,COUNT(*) AS c FROM KAFKA.clicks GROUP BY FLOOR(rowtime TO HOUR), ip": From line 1, column 85 to line 1, column 119: Streaming aggregation requires at least one monotonic expression in GROUP BY clause
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
You need to declare that the "ROWTIME" column is monotonic. In MockCatalogReader, note how "ROWTIME" is declared monotonic in the "ORDERS" and "SHIPMENTS" streams. That’s why some queries in SqlValidatorTest.testStreamGroupBy() are valid and others are not. The key method relied up by the validator is SqlValidatorTable.getMonotonicity(String columnName).
This is the Code I was made. It shouldn't get and enter to NoResultException, but it doesn't as expected. There is an unused data. I try to print out, here is the output : "[ ]"
private void deleteButtonActionPerformed(java.awt.event.ActionEvent evt) {
try {
int row = tableDataRangka.getSelectedRow();
String idRangka = tableDataRangka.getValueAt(row, 0).toString();
System.out.println( relasiRumahkayuRangkaDAO.getRelasiByIdRangka(idRangka).toString() );
} catch (NoResultException nre) {
// Doing something..
} catch (Exception ex) {
Logger.getLogger(MasterDataProjectUI.class.getName()).log(Level.SEVERE, null, ex);
}
}
Here is the code of method "getRelasiByIdRangka" :
public List<RelasiRumahKayuRangka> getRelasiByIdRangka(String idRangka) throws Exception{
initEntityManager();
List<RelasiRumahKayuRangka> rrDrs = new ArrayList<>();
Query q = em.createNamedQuery("RelasiRumahKayuRangka.findByIdRangka");
q.setParameter("idRangka", idRangka);
rrDrs.addAll(q.getResultList());
closeEntityManager();
return rrDrs;
}
And this one is the JPA query, findByIdRangka :
#NamedQuery(name = "RelasiRumahKayuRangka.findByIdRangka", query = "SELECT r FROM RelasiRumahKayuRangka r WHERE r.relasiRumahKayuRangkaPK.idRangka = :idRangka"),
Do you guys know the solution, so the code can be catched by NoResultException ?
In your code. if the result set is empty then list will also be empty.
Make use of that situation and throw a desired exception.
In your getRelasiByIdRangka(String idRangka) add following changes.
public List<RelasiRumahKayuRangka>
getRelasiByIdRangka( String idRangka ) throws Exception {
initEntityManager();
// rest of the code here
// ...
// check if some results are found or not
if( rrDrs.isEmpty() ) { // or ( rrDrs.size() == 0 )
throw new NoResultException( "No results found" );
} // if empty
// this should be a filled in list
return rrDrs;
} // end of method
I am working on a project in which I have three tables in a different database with different schemas. So that means I have three different connection parameters for those three tables to connect using JDBC-
Let's suppose-
For Table1-
Username:- A
Password:- B
URL: C
Columns-
ID1 String
Account1 String
For Table2-
Username:- P
Password:- Q
URL:- R
Columns-
ID2 String
Account2 String
For Table3-
Username:- T
Password:- U
URL:- V
Columns-
ID3 String
Account3 String
And I am supposed to insert in all the three tables or any one of them using JDBC.
Below are the three use cases I have-
From the command prompt if suppose I am passing Table1 only, then I am suppose to insert only in Table1 columns by making connection to
Table1.
And if I am passing Table1, Table2 from the command prompt then I am suppose to insert in both Table1 and Table2 columns by making
connection to Table1 and Table2.
And if I am passing Table1, Table2 and Table3 then I am suppose to enter in all the three tables using there respective connection
parameter
I am not able to understand how to write code for the above particular scenario in such a cleaner way so that it can be extended in near future as well if I come up with four tables. I can have a one constant file which can store the SQL that needs to be executed for any of the three tables and some other constant thing as well.
public static void main(String[] args) {
}
class Task implements Runnable {
private Connection dbConnection = null;
private PreparedStatement preparedStatement = null;
public Task() {
}
#Override
public void run() {
dbConnection = getDbConnection();
//prepare the statement and execute it
}
}
private Connection getDBConnection() {
Connection dbConnection = null;
Class.forName(Constants.DRIVER_NAME);
dbConnection = DriverManager.getConnection( , , );
return dbConnection;
}
Can anyone provide some thoughts on this how should I proceed forward?
Note:-
Column in each table will differ a lot. Like in some tables, column can be 10 and in some other table, column can be 20.
Create databases.properties file with content like this:
# Table 1
table1.url: jdbc:mysql://localhost:3306/garden
table1.user: gardener
table1.password: shavel
table1.table: fruits
table1.column.id: fruitID
table1.column.color: fruitColor
table1.column.weight: fruitWeight
# ... More fruit columns here ...
# Table 2
table2.url: jdbc:mysql://otherhost:3306/forest
table2.user: forester
table2.password: axe
table2.table: trees
table2.column.id: treeID
table2.column.height: treeHeight
# ... More tree columns here ...
# ... More tables here ...
Then do something like this:
public static void main (String [] args)
{
Properties databasesProperties = new Properties ();
databasesProperties.load ("databases.properties");
for (String arg: args)
{
String url = databasesProperties.get (arg + ".url");
String user = databasesProperties.get (arg + ".user");
String password= databasesProperties.get (arg + ".password");
String table = databasesProperties.get (arg + ".table");
String columnPrefix = arg + ".column."
Map <String, String> columns = new HashMap <String, String> ();
for (String key: databasesProperties.stringPropertyNames ())
{
if (key.startsWith (columnPrefix))
columns.put (
key.substring (columnPrefix.length ()),
databasesProperties.get (key));
}
doInsert (url, user, password, table, columns);
}
}
Later you can always add more tables into your databases.properties file.
Save your Database properties in a class file DBPropery.java.
final class DBProperty
{
static String[] urls = {
"C",
"R",
"V"
}; //You can add more URLs here.
static String[] driver= {
"Driver1",
"Driver2",
"Driver3"
};//You can add more drivers string
static String[] table = {
"Table1",
"Table2",
"Table3"
};//You can add more table names here According to URLs mentioned in urls array.
static String[] user = {
"A",
"P",
"T"
};//You can add more user names here according to URls mentioned in urls array.
static String[] pwd = {
"B",
"Q",
"U"
};//You can add more Password here according to URls mentioned in urls array.
static String[] queries = {
"Query for Table1",
"Query for Table2",
"Query for Table3",
};//You can add more queries here for more tables according to URls mentioned in urls array.
static int[] columns ={
2,
2,
2
};//You can change the column numbers according to need . 0th index belongs to Table1 , 1 to table2....so on.
//If you add more tables , add corresponding columns count to next index.
static String[] columnValues ={
"1^John",
"34^Vicky",
"65^Ethen"
};//String at each index represents a row in corresponding table in table[] array. each column is seperated by delimiter "^".
}
Make all Changes in DBProperty.java file.
Then proceed with following class file
import java.sql.*;
import java.util.*;
class MultiTableInsert implements Runnable
{
Map<String,Integer> columnsInTable;
Map<String,String> tableDriver;
Map<String,String> rowForTable;
Map<String,String> queryForTable;
Map<String,String> urlForTable;
Map<String,String> userForTable;
Map<String,String> pwdForTable;
String[] tables ;
public MultiTableInsert(String... tables)//Loading all Database Settings here..
{
this.tables = tables;
columnsInTable = new LinkedHashMap<String,Integer>();
rowForTable = new LinkedHashMap<String,String>();
tableDriver = new LinkedHashMap<String,String>();
urlForTable = new LinkedHashMap<String,String>();
userForTable= new LinkedHashMap<String,String>();
pwdForTable = new LinkedHashMap<String,String>();
for (int i = 0 ; i < DBProperty.urls.length ; i++ )
{
try
{
tableDriver.put(DBProperty.table[i],DBProperty.driver[i]);
queryForTable.put(DBProperty.table[i],DBProperty.queries[i]);
columnsInTable.put(DBProperty.table[i],DBProperty.columns[i]);
rowForTable.put(DBProperty.table[i],DBProperty.columnValues[i]);
urlForTable.put(DBProperty.table[i],DBProperty.urls[i]);
userForTable.put(DBProperty.table[i],DBProperty.user[i]);
pwdForTable.put(DBProperty.table[i],DBProperty.pwd[i]);
}
catch (Exception ex)
{
ex.printStackTrace();
}
}
}
#Override
public void run()
{
insertIntoTable(tables);
}
private void insertIntoTable(String... tables)
{
for (String tble : tables )
{
Connection con = null;
PreparedStatement pStmt = null;
try
{
Class.forName(tableDriver.get(tble));
con = DriverManager.getConnection(urlForTable.get(tble),userForTable.get(tble),pwdForTable.get(tble));
pStmt = con.prepareStatement(queryForTable.get(tble));
int columns = columnsInTable.get(tble);
String sRow = rowForTable.get(tble);
StringTokenizer tokenizer = new StringTokenizer(sRow,"^");
for (int i = 0; i < columns ; i++)
{
pStmt.setString(i+1,(String)tokenizer.nextElement());
}
pStmt.execute();
}
catch (Exception ex)
{
ex.printStackTrace();
}
finally
{
try
{
con.close();
}catch (Exception ex){}
try
{
pStmt.close();
}catch (Exception ex){}
}
}
}
public static void main(String[] args)
{
int length = args.length;
int THREAD_COUNTS = 10;//Number of threads you want to start.
switch (length)
{
case 0:
System.out.println("Usage: javac MultiTableInsert Table1/Table2/Table3 <Table1/Table2/Table3> <Table1/Table2/Table3>");
System.exit(0);
case 1:
for (int i = 0 ; i < THREAD_COUNTS ; i++)
{
MultiTableInsert mti = new MultiTableInsert(args[0]);
Thread th = new Thread(mti,"Thread"+i);//Create New Thread
th.start(); //Start Thread
}
break;
case 2:
for (int i = 0 ; i < THREAD_COUNTS ; i++)
{
MultiTableInsert mti = new MultiTableInsert(args[0],args[1]);//Create New Thread
Thread th = new Thread(mti,"Thread"+i); //Start Thread
th.start();
}
break;
default:
for (int i = 0 ; i < THREAD_COUNTS ; i++)
{
MultiTableInsert mti = new MultiTableInsert(args[0],args[1],args[2]);//Create New Thread
Thread th = new Thread(mti,"Thread"+i); //Start Thread
th.start();
}
break;
}
}
}