I am relatively new to Cassandra, I am trying to retrieve data using prepared statement executed via an Executor Pool. Looks like the data I am receiving is not consistent.
I have this user_connections table, where user_id is the row key, friends_id list as a set column.
I have this another table, friends_info table, where friend_id is the row key and all other information as columns.
When am trying to retrieve friends list for user AA, I am retrieving the friends list BBB, CCC, DDD. Which is perfectly fine.
When am trying to retrieve BBB, CCC, DDD via an executor pool using prepared statement. Data is inconsistent. Some times all three records are BBB, Some times all three records are, Some times two records are BBB and one is CCC etc...
I have provided the methods and relevant classes that I am using, can you please help me with this. I know prepared statement is thread safe and expected to work as expected.
public Set<User> listUserConnections(String userId) {
Session session = client.getSession();
Set<String> listUserConnectionIds = listUserConnections(userId, session);
if (listUserConnectionIds.size() == 0)
return new HashSet<User>();
Set<User> listConnectionUserDetails = retrieveUserConnectionProfileInfo(
listUserConnectionIds, session);
return listConnectionUserDetails;
}
private Set<User> retrieveUserConnectionProfileInfo(Set<String> listUserConnectionIds,
Session session) {
Set<Callable<User>> callables = new HashSet<Callable<User>>();
for(String key:listUserConnectionIds){
logger.info("about to add callable" + key);
Callable<User> callable = new QueryTask(key);
callables.add(callable);
}
// invoke in parallel
List<Future<User>> futures;
Set<User> users = new HashSet<User>();
// TODO Revisit this
ExecutorService executorPool = Executors.newFixedThreadPool(100);
try {
futures = executorPool.invokeAll(callables);
for (Future<User> f : futures) {
User user = f.get();
users.add(user);
logger.info("User from future"+user.getUserId());
}
} catch (ExecutionException e) {
logger.error("Error in retrieving the stores in parallel ", e);
} catch (InterruptedException e) {
logger.error("Error in retrieving the stores in parallel as it was interrupted ", e);
} finally {
executorPool.shutdown();
}
return users;
}
//Executor Pool Class
class QueryTask implements Callable<User> {
private String userName;
// final PreparedStatement statement =
// client.getSession().prepare(QueryConstants.GET_ALL_STORE_BRANDS);
QueryTask(String name) {
this.userName = name;
}
#Override
public User call() throws Exception {
// -------------I am seeing the userName is correct------------- for example BBB
logger.info("inside call processing queries for " + userName);
//------------This is a prepared statement, userPreparedStatement.getbStUserInfo()
BoundStatement bStUserInfo = userPreparedStatement.getbStUserInfo();
bStUserInfo.bind(userName);
Session session = client.getSession();
ResultSet rs = session.execute(bStUserInfo);
User user = new User();
Iterator<Row> rowIterator = rs.iterator();
while (rowIterator.hasNext())
{
Row row = rowIterator.next();
//-------This user id is not right
logger.info("Inside the callable after retrieval"+row.getString(TableConstants.Users.COLUMN_NAME_USER_ID));
user.setUserId(row.getString(TableConstants.Users.COLUMN_NAME_USER_ID));
return user;
}
logger.info("outside the while loop");
return user;
}
}
Thank you so much #Ralf and Alex Popescu for getting back to me on this. Datastax had a documentation that dives deep on how the Aync calls work.
# Alex Popescu. Thanks I tried their aync calls and it worked fine for me.
Related
I have a simple program that is supposed to get a user from github's API and I want to count the times the method is called.
Here is my REST Controller with a GET method (that's the method to be counted):
#RestController
#RequestMapping("/api/v1")
public class UserController {
private UserService userService;
public UserController(UserService userService) {
this.userService = userService;
}
#GetMapping("/user/info/{login}")
public User getUser(#PathVariable String login) throws IOException {
userService.insertRecord(login);
return userService.fetchUser(login);
}
}
My program works (more or less) as it's supposed to, BUT .insertRecord() method does not work at all. It basically does nothing. The method SHOULD check if the database already has a user of the given login. If yes - then it should proceed to update the record and increment the REQUEST_COUNT number by 1. If no - then it should create a new record of a given login and REQUEST_COUNT as 1. The table in my database has only two column - LOGIN and REUQEST_COUNT.
But that method literally does nothing, the table in my database remains untouched.
public void insertRecord(String login) {
//this part checks if there is a login like that in the database already
String sql = "select * from calls where LOGIN = ?";
PreparedStatement preparedStatement;
ResultSet resultSet;
try {
preparedStatement = getConnection().prepareStatement(sql);
preparedStatement.setString(1, login);
resultSet = preparedStatement.executeQuery();
//if there's a result - I am trying to increment the value of REQUEST_COUNT column by 1.
if (resultSet.next()) {
String updateSql = "update calls set REQUEST_COUNT = REQUEST_COUNT + 1 where login = ?";
PreparedStatement updatePreparedStatement;
try {
updatePreparedStatement = getConnection().prepareStatement(updateSql);
updatePreparedStatement.setString(1, login);
} catch (SQLException e) {
logger.error("Could not insert a record into the database.");
e.printStackTrace();
}
//if there is no result - I want to insert a new record.
} else {
String insertSql = "insert into calls (LOGIN, REQUEST_COUNT) values (?, ?)";
try (final PreparedStatement insertPreparedStatement = getConnection().prepareStatement(insertSql)) {
insertPreparedStatement.setString(1, login);
insertPreparedStatement.setInt(2, 1);
} catch (SQLException e) {
logger.error("Could not insert a record into the database.");
e.printStackTrace();
}
}
} catch (SQLException e) {
logger.error("Operation failed.");
e.printStackTrace();
}
}
No Logger's messages are printed either, it's as if the method was simply completely ignored.
Please, help.
Because you are not calling updatePreparedStatement.executeUpdate() !
The sql will only take effetc after you call executeUpdate().
You can put a filter that will execute before/after the endpoint executed. And in that particular filter, you can also track which endpoint is executed and take appropriate action for any specific endpoint.
I try to fetch record from database using hibernate's scrollable result and with reference from this github project, i tried to send each record as chunk response.
Controller:
#Transactional(readOnly=true)
public Result fetchAll() {
try {
final Iterator<String> sourceIterator = Summary.fetchAll();
response().setHeader("Content-disposition", "attachment; filename=Summary.csv");
Source<String, ?> s = Source.from(() -> sourceIterator);
return ok().chunked(s.via(Flow.of(String.class).map(i -> ByteString.fromString(i+"\n")))).as(Http.MimeTypes.TEXT);
} catch (Exception e) {
return badRequest(e.getMessage());
}
}
Service:
public static Iterator<String> fetchAll() {
StatelessSession session = ((Session) JPA.em().getDelegate()).getSessionFactory().openStatelessSession();
org.hibernate.Query query = session.createQuery("select l.id from Summary l")
.setFetchSize(Integer.MIN_VALUE).setCacheable(false).setReadOnly(true);
ScrollableResults results = query.scroll(ScrollMode.FORWARD_ONLY);
return new models.ScrollableResultIterator<>(results, String.class);
}
Iterator:
public class ScrollableResultIterator<T> implements Iterator<T> {
private final ScrollableResults results;
private final Class<T> type;
public ScrollableResultIterator(ScrollableResults results, Class<T> type) {
this.results = results;
this.type = type;
}
#Override
public boolean hasNext() {
return results.next();
}
#Override
public T next() {
return type.cast(results.get(0));
}
}
For test purpose, i am having 1007 records in my table, whenever i call this end point, it always return only 503 records.
Enabled AKKA log level to DEBUG and tried it again, it logs the following line for 1007 times
2016-07-25 19:55:38 +0530 [DEBUG] from org.hibernate.loader.Loader in application-akka.actor.default-dispatcher-73 - Result row: From the log i confirm that it fetching all, but couldn't get where the remaining one got left.
I run the same query in my workbench and i export it to a file locally and compared it with the file generated by the end point, kept LHS record generated from end point and RHS file exported from Workbench.
First row matches, second and third didn't match. After that it got matches for alternate records until the end.
Please correct me, if am doing anything wrong and suggest me is this the correct approach for generating CSV for large db records.
For the sake of testing, i removed the CSV conversion logic in my above snippet.
// Controller code
// Prepare a chunked text stream
ExportAsChuncked eac = new ExportAsChuncked();
response().setHeader("Content-disposition","attachment; filename=results.csv");
Chunks<String> chunks = new StringChunks() {
// Called when the stream is ready
public void onReady(Chunks.Out<String> out) {
try {
eac.exportData(scrollableIterator, out);
}catch (XOException e) {
Logger.error(ERROR_WHILE_DOWNLOADING_RESPONSE, e);
}
out.close();
}
};
// Serves this stream with 200 OK
return ok(chunks);
// Export as chunk logic
class ExportAsChuncked {
void exportData(Iterator<String> data, Chunks.Out<String> out) {
while(data.hasNext()) {
out.write(data.next());
}
}
}
I have an aplication which create a number of query (update or insert) and then each query is executed.
The whole code is working fine but I've saw that my server IO latency is too much during this proccess.
The code execute a loop which is taking arround 1 minute.
Then what I wanted to do is write each query in memory instead to execute it, and then, once I have the whole list of query to execute, use "LOAD DATA LOCAL INFILE" from mysql, which will take less time.
My question is: How can I write all my query (String object) in a "File" or "any other container" in java to use it after the loop?.
#user3283548 This is my example code:
Class1:
import java.util.ArrayList;
public class Class1 {
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
ArrayList<String> Staff=new ArrayList<String>();
Staff.add("tom");
Staff.add("Laura");
Staff.add("Patricia");
for (int x = 0; x < Staff.size(); x++) {
System.out.println(Staff.get(x));
Class2 user = new Class2 (Staff.get(x));
user.checkUser();
}
}
}
Class2:
public class Class2 {
private String user;
public Class2(String user){
this.user=user;
}
public void checkUser() throws Exception{
if (user.equals("tom")){
String queryUser="update UsersT set userStatus='2' where UserName='"+user+"';";
Class3 updateUser = new Class3(queryUser);
updateUser.UpdateQuery();;
}else{
String queryUser="Insert into UsersT (UserName,userStatus)Values('"+user+"','1');";
Class3 updateUser = new Class3(queryUser);
updateUser.InsertQuery();
System.out.println(user+" is not ton doing new insert");
}
}
}
Class3:
public class Class3 {
public String Query;
public Class3(String Query){
this.Query = Query;
}
public void UpdateQuery() throws Exception{
/*// Accessing Driver From Jar File
Class.forName("com.mysql.jdbc.Driver");
//DB Connection
Connection con = DriverManager.getConnection("jdbc:mysql://localhost:3306/default","root","1234567");
String sql =Query;
PreparedStatement pst = con.prepareStatement(sql);*/
System.out.println(Query); //Just to test
//pst.execute();
}
public void InsertQuery() throws Exception{
/*// Accessing Driver From Jar File
Class.forName("com.mysql.jdbc.Driver");
//DB Connection
Connection con = DriverManager.getConnection("jdbc:mysql://localhost:3306/default","root","1234567");
String sql =Query;
PreparedStatement pst = con.prepareStatement(sql);*/
System.out.println(Query); //Just to test
//pst.execute();
}
}
Then, what I wanted to do is create an ArraList in Class1 and use it in Class3 to collect all the queries which has to be executed.
The idea is to execute the list of queries in one time, once the main process is finished, istead to do it for each element within in loop of the Class1. I wanted to do it, because I think it will be take less resource IO from the server HD
Your loop is probably too slow because you're building up Strings using String
I'd hazard a guess you're doing things like
String query = "SELECT * FROM " + variablea + " WHERE + variableb + " = " ...
If you're doing a lot of string concatenation then use StringBuilder as every time you change a string it is actually re-created which is expensive. Simply changing your code to use StringBuilder instead of string will probably cut your loop executed time to a couple of MS. Simply call .toString() method of StringBuilder obj to get the string.
Storing objects
If you want to store anything for later use you should store it in a Collection. If you want a a key-value relationship then use a Map (HashMap would suit you fine). If you just want the values use an List (ArrayList is most popular).
So for example if I wanted to store query strings for later use I would...
Construct the string using StringBuilder.
Put the string (by calling .toString() into a HashMap
Get the query string from the HashMap...
You should never store things on disk if you don't need them to be persistent over application restarts and even then I'd store them in a database not in a file.
Hope this helps.
Thanks
David
EDIT: UPDATE BASED ON YOU POSTING YOUR CODE:
OK this needs some major re-factoring!
I've kept it really simple because I don't have a lot of time to re-write comprehensively.
I've commented where I have made corrections.
Your major issue here is creating objects in loops. You should just create the object once as creating objects is expensive.
I've also corrected other coding issues and replaced the for loop as you shouldn't be writing it like that.I've also renamed the classes to something useful.
I've not tested this so you may need to do some work to get it to work. But this should be a lot faster.
OLD CLASS 1
import java.util.ArrayList;
import java.util.List;
public class StaffChecker {
public static void main(String[] args) throws Exception {
// Creating objects is expensive, you should do this as little as possible
StaffCheckBO staffCheckBO = new StaffCheckBO();
// variables should be Camel Cased and describe what they hold
// Never start with ArrayList start with List you should specific the interface on the left side.
List<String> staffList = new ArrayList<String>();
staffList.add("tom");
staffList.add("Laura");
staffList.add("Patricia");
// use a foreach loop not a (int x = 0 ... ) This is the preffered method.
for (String staffMember : staffList) {
// You now dont need to use .get() you can access the current variable using staffMember
System.out.println(staffMember);
// Do the work
staffCheckBO.checkUser(staffMember);
}
}
}
OLD CLASS 2
/**
* Probably not really any need for this class but I'll assume further business logic may follow.
*/
public class StaffCheckBO {
// Again only create our DAO once...CREATING OBJECTS IS EXPENSIVE.
private StaffDAO staffDAO = new StaffDAO();
public void checkUser(String staffMember) throws Exception{
boolean staffExists = staffDAO.checkStaffExists(staffMember);
if(staffExists) {
System.out.println(staffMember +" is not in database, doing new insert.");
staffDAO.insertStaff(staffMember);
} else {
System.out.println(staffMember +" has been found in the database, updating user.");
staffDAO.updateStaff(staffMember);
}
}
}
OLD CLASS 3
import java.sql.*;
/**
* You will need to do some work to get this class to work fully and this is obviously basic but its to give you an idea.
*/
public class StaffDAO {
public boolean checkStaffExists(String staffName) {
boolean staffExists = false;
try {
String query = "SELECT * FROM STAFF_TABLE WHERE STAFF_NAME = ?";
PreparedStatement preparedStatement = getDBConnection().prepareStatement(query);
// Load your variables into the string in order to be safe against injection attacks.
preparedStatement.setString(1, staffName);
ResultSet resultSet = preparedStatement.executeQuery();
// If a record has been found the staff member is in the database. This obviously doesn't account for multiple staff members
if(resultSet.next()) {
staffExists = true;
}
} catch (SQLException e) {
System.out.println("SQL Exception in getStaff: " + e.getMessage());
}
return staffExists;
}
// Method names should be camel cased
public void updateStaff(String staffName) throws Exception {
try {
String query = "YOUR QUERY";
PreparedStatement preparedStatement = getDBConnection().prepareStatement(query);
// Load your variables into the string in order to be safe against injection attacks.
preparedStatement.setString(1, staffName);
ResultSet resultSet = preparedStatement.executeQuery();
} catch (SQLException e) {
System.out.println("SQL Exception in getStaff: " + e.getMessage());
}
}
public void insertStaff(String staffName) throws Exception {
try {
String query = "YOUR QUERY";
PreparedStatement preparedStatement = getDBConnection().prepareStatement(query);
// Load your variables into the string in order to be safe against injection attacks.
preparedStatement.setString(1, staffName);
ResultSet resultSet = preparedStatement.executeQuery();
} catch (SQLException e) {
System.out.println("SQL Exception in getStaff: " + e.getMessage());
}
}
/**
* You need to abstract the connection logic away so you avoid code reuse.
*
* #return
*/
private Connection getDBConnection() {
Connection connection = null;
try {
Class.forName("com.mysql.jdbc.Driver");
connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/default", "root", "1234567");
} catch (ClassNotFoundException e) {
System.out.println("Could not find class. DB Connection could not be created: " + e.getMessage());
} catch (SQLException e) {
System.out.println("SQL Exception. " + e.getMessage());
}
return connection;
}
}
Is there a difference between:
Object o = new Object();
session.save(o)
and
session.save(new Object());
I ask because I am finding sometimes the objects are mixed up in my implementation. The discrepancy is found in production database. So it is hard to test.
Here is the edited code:
pubic class Product {
Logger logger = Logger.getRootLogger();
private String globalId;
public void service() {
this.gobalId = getVariable("GLOBALID"); //from Asterisk
Transaction tran = null;
try {
Session session = HibernateUtil.getCurrentSession();
int charge=0;
//set charge
Transaction tran = session.beginTransaction();
session.save(new Packet(globalId, charge));
tran.commit();
} catch (Exception e) {
if (tran != null) tran.rollback();
logger.error("Error", e);
} finally {
HibernateUtil.closeCurrentSession();
}
}
}
At run time, I see in the Packet table duplicate globalId rows with different values for the charge column. The globalId is supposed to be unique, even though it is not the primary key in the table. There could be a couple of explanations:
Asterisk is sending a wrong value for globalId in the getVariable() method
The service method needs to be synchrnozied (the Asterisk Java API says the Product class is implemented like a Servlet)
I don't see any errors in the logs. So no exceptions are thrown in the code.
Any help is appreciated.
Thanks
First time that I ran into this error I've surrounded my tx.commit() with a if condition but am not sure why I am still receiving this error.
Struts Problem Report
Struts has detected an unhandled exception:
Messages:
Transaction not successfully started
File: org/hibernate/engine/transaction/spi/AbstractTransactionImpl.java
Line number: 200
Stacktraces
org.hibernate.TransactionException: Transaction not successfully started
org.hibernate.engine.transaction.spi.AbstractTransactionImpl.rollback(AbstractTransactionImpl.java:200)
After a product has been selected by user, in my main function I will call two functions as following.
First function to retrieve the object of selected product.
Second function to check if selected user has the product therefore it returns true if client has the product otherwise returns false;
Function 1
....
Product pro = new Product();
final Session session = HibernateUtil.getSession();
try {
final Transaction tx = session.beginTransaction();
try {
pro = (Product) session.get(Product.class, id);
if (!tx.wasCommitted()) {
tx.commit();
}
} catch (Exception e) {
tx.rollback();
e.printStackTrace();
}
} finally {
HibernateUtil.closeSession();
}
.....
Function 2
.....
final Session session = HibernateUtil.getSession();
try {
final Transaction tx = session.beginTransaction();
try {
User user = (User) session.get(User.class, id);
if (!tx.wasCommitted()) {
tx.commit();
}
if(client.hasProduct(proId)){
return client.getProduct(proId);
}
return false;
} catch (Exception e) {
tx.rollback(); <<<Error is on this line
e.printStackTrace();
}
} finally {
HibernateUtil.closeSession();
}
....
Take a look at Transaction.isActive() method. You can wrap call to rollback() method with condition, checking whether transaction is still active. And the second, I'd prefer the following code:
final Session session = HibernateUtil.getSession();
try {
final Transaction tx = session.beginTransaction();
// do things
tx.commit();
} finally {
if (tx.isActive()) {
try {
tx.rollback();
} catch (Exception e) {
logger.log("Error rolling back transaction", e);
}
}
try {
session.close();
} catch (Exception e) {
logger.log("Error closing session", e);
}
}
Of course, code in the finally section better to wrap into public static method and just call it in every finally.
BTW, why are you doing something outside tranaction? I usually commit after all things get done, to achieve a better consistency and avoid LazyInitializationException.
One possibility is that the exception you are catching in the second functions is from the code after the commit(), so you end up trying to rollback a transaction that is already committed, which is not allowed.
You could try reorganizing your code to make sure that rollback is never called after commit. Maybe even something simple like reducing the scope of the inner try-catch:
final Session session = HibernateUtil.getSession();
try {
final Transaction tx = session.beginTransaction();
try {
User user = (User) session.get(User.class, id);
if (!tx.wasCommitted()) {
tx.commit();
}
} catch (Exception e) {
tx.rollback();
e.printStackTrace();
}
if(client.hasProduct(proId)){
return client.getProduct(proId);
}
return false;
} finally {
HibernateUtil.closeSession();
}
The error indicates the transaction wasn't started at the time tried to roll back - and the problem may be that you are trying to wrap a get, which does not alter the db state and does not leave behind garbage that needs to be committed or rolled back. Nothing changes when you perform select *.
In addition to this, you may want to extract this transaction handling into a common method that is independent of the work being done, so you don't have to write this over and over again, that leaves your code open for bugs. Basically, it seems like you are getting DB objects but then intermingling some business logic withing the same method. Perhaps consider doing something like below:
DB Handling Function
public static <T> T getDBObject( Class<T> clazz, Serializable id )
throws SQLException
{
Session session = null;
try
{
session = HibernateUtil.getSession();
return (T)session.get( clazz, id );
}
finally
{
if ( session != null )
{
session.close();
}
}
}
Now that you can pull object of the DB (note that they will be detached, but still valid), you can then perform work on the objects. I many not have captured exactly what you need to check, but it seems like it is something like:
Example Comparison Function
public boolean doesUserHaveProduct(Serializable userId, Serializable productId)
{
try
{
User user = getDBObject(User.class, userId);
Product product = getDBObject( Product.class, productId );
return user.hasProduct( product );
}
catch (SQLException e)
{
e.printStackTrace();
return false;
}
}