I am following RetwisJ tutorial available here. In this I don't think Redis transactions are implemented. For example, in the following function, if some exception occurs in between, the data will be left in an inconsistent state.
I want to know how a function like the following can be implemented in Spring Data Redis as a single transaction:
public String addUser(String name, String password) {
String uid = String.valueOf(userIdCounter.incrementAndGet());
// save user as hash
// uid -> user
BoundHashOperations<String, String, String> userOps = template.boundHashOps(KeyUtils.uid(uid));
userOps.put("name", name);
userOps.put("pass", password);
valueOps.set(KeyUtils.user(name), uid);
users.addFirst(name);
return addAuth(name);
}
Here userIdCounter, valueOps and users are initialized in the constructor. I have come across this in the documentation(section 4.8), but I can't figure out how to fit that into this function where some variables are initialized outside the function(please don't tell I have to initialize these variables in each and every function where I need transactions!).
PS: Also is there any #Transaction annotation or transaction manager available for Spring Data Redis?
UPDATE: I have tried using MULTI, EXEC. The code which I have written is for another project, but when its applied to this problem it'll be as follows:
public String addMyUser(String name, String password) {
String uid = String.valueOf(userIdCounter.incrementAndGet());
template.execute(new SessionCallback<Object>() {
#Override
public <K, V> Object execute(RedisOperations<K, V> operations)
throws DataAccessException {
operations.multi();
getUserOps(operations, KeyUtils.uid(uid)).put("name", name);
getUserOps(operations, KeyUtils.uid(uid)).put("pass", password);
getValueOps(operations).set(KeyUtils.user(name), uid);
getUserList(operations, KeyUtils.users()).leftPush(name);
operations.exec();
return null;
}
});
return addAuth(name);
}
private ValueOperations<String, String> getValueOps(RedisOperations operations) {
return operations.opsForValue();
}
private BoundHashOperations<String, String, String> getUserOps(RedisOperations operations, String key) {
return operations.boundHashOps(key);
}
private BoundListOperations<String, String> getUserList(RedisOperations operations, String key) {
return operations.boundListOps(key);
}
Please tell whether this way of using MULTI, EXEC is recommended or not.
Up to SD Redis 1.2 you will have to take care of tansaction handling yourself using TransactionSynchronisationManager
The snipplet above could then look something like this:
public String addUser(String name, String password) {
String uid = String.valueOf(userIdCounter.incrementAndGet());
// start the transaction
template.multi();
// register synchronisation
if(TransactionSynchronisationManager.isActualTransactionActive()) {
TransactionSynchronisationManager.registerSynchronisation(new TransactionSynchronizationAdapter()) {
#Override
public void afterCompletion(int status) {
switch(status) {
case STATUS_COMMITTED : template.exec(); break;
case STATUS_ROLLED_BACK : template.discard(); break;
default : template.discard();
}
}
}
}
BoundHashOperations<String, String, String> userOps = template.boundHashOps(KeyUtils.uid(uid));
userOps.put("name", name);
userOps.put("pass", password);
valueOps.set(KeyUtils.user(name), uid);
users.addFirst(name);
return addAuth(name);
}
Please note that once in multi, read operations will also be part of the transaction which means you'll likely not be able to read data from redis server.
The setup might differ form the above one as you could want to additionally call WATCH. Further on you'll also have to take care of multiple callbacks do not sending MULTI and/or EXEC more than once.
The upcoming 1.3 RELEASE of Spring Data Redis will ship with support for spring managed transactions in a way of taking care of MULTi|EXEC|DISCARD plus allowing read operations (on already existing keys) while transaction synchronization is active. You could already give the BUILD-SNAPSHOT a spin and turn this on by setting template.setEnableTransactionSupport(true).
By default, RedisTemplate does not participate in managed Spring
transactions. If you want RedisTemplate to make use of Redis
transaction when using #Transactional or TransactionTemplate, you need
to be explicitly enable transaction support for each RedisTemplate by
setting setEnableTransactionSupport(true). Enabling transaction
support binds RedisConnection to the current transaction backed by a
ThreadLocal. If the transaction finishes without errors, the Redis
transaction gets commited with EXEC, otherwise rolled back with
DISCARD. Redis transactions are batch-oriented. Commands issued during
an ongoing transaction are queued and only applied when committing the
transaction.
Spring Data Redis distinguishes between read-only and write commands
in an ongoing transaction. Read-only commands, such as KEYS, are piped
to a fresh (non-thread-bound) RedisConnection to allow reads. Write
commands are queued by RedisTemplate and applied upon commit.
https://docs.spring.io/spring-data/data-redis/docs/current/reference/html/#tx.spring
Related
Hoping someone else is having the same issue as me, or has other ideas.
I'm currently running Play 1.4.x (not by choice), but also working on upgrading to play 1.5.x, though I verified the same issue happens on both versions.
I created a simple Functional Test that loads data via fixtures
My fixture for loading test data is like so
data.yml
User(testUser):
name: blah
AccessToken(accessToken):
user: testUser
token: foo
Data(testData):
user: testUser
...
I've created a controller to do something with the data like this, that has middleware for authentication check. The routes file will map something like /foo to BasicController.test
public class BasicController extends Controller{
#Before
public void doAuth(){
String token = "foo"; // Get token somehow from header
AccessToken token = AccessToken.find("token = ?", token).first(); // returns null;
// do something with the token
if(token == null){
//return 401
}
//continue to test()
}
public void test(){
User user = //assured to be logged-in user
... // other stuff not important
}
}
Finally I have my functional test like so:
public class BasicControllerTest extends FunctionalTest{
#org.junit.Before
public void loadFixtures(){
Fixtures.loadModels("data.yml");
}
#Test
public void doTest(){
Http.Request request = newRequest()
request.headers.put(...); // Add auth token to header
Http.Response response = GET(request, "/foo");
assertIsOk(response);
}
}
Now, the problem I'm running into, is that I can verify the token is still visible in the headers, but running AccessToken token = AccessToken.find("token = ?", token).first(); returns null
I verified in the functional test, before calling the GET method that the accessToken and user were created successfully from loading the fixtures. I can see the data in my, H2 in-memory database, through plays new DBBrowser Plugin in 1.5.x. But for some reason the data is not returned in the controller method.
Things I've tried
Ensuring that the fixtures are loaded only once so there is no race condition where data is cleared while reading it.
Using multiple ways of querying the database via nativeQuery jpql/hql query language and through plays native way of querying data.
Testing on different versions of play
Any help would be very much appreciated!
This issue happens on functional tests, because JPA transactions must be encapsulated in a job to ensure that the result of the transaction is visible in your method. Otherwise, since the whole functional test is run inside a transaction, the result will only visible at the end of the test (see how to setup database/fixture for functional tests in playframework for a similar case).
So you may try this:
#Test
public void doTest() {
...
AccessToken token = new Job<AccessToken>() {
#Override
public User doJobWithResult() throws Exception {
return AccessToken.find("token = ?", tokenId).first();
}
}.now().get();
....
}
Hoping it works !
I think I had a similar issue, maybe this helps someone.
There is one transaction for the functional test and a different transaction for the controller. Changes made in the test will only become visible by any further transaction if those changes were committed.
One can achieve this by closing and re-opening the transaction in the functional test like so.
// Load / Persist date here
JPA.em().getTransaction().commit(); // commit and close the transaction
JPA.em().getTransaction().begin(); // reopen (if you need it)
Now the data should be returned in the controller method.
So your test would look like this:
public class BasicControllerTest extends FunctionalTest{
#org.junit.Before
public void loadFixtures(){
Fixtures.loadModels("data.yml");
JPA.em().getTransaction().commit();
// JPA.em().getTransaction().begin(); reopen (if you need it)
}
#Test
public void doTest(){
Http.Request request = newRequest()
request.headers.put(...); // Add auth token to header
Http.Response response = GET(request, "/foo");
assertIsOk(response);
}
}
I did never try this with fixtures. But i would assume they run in the same transaction.
I have a SpringBoot gradle project using apache flink to process datastream signals. When a new signal comes through the datastream, I would like to query look up (i.e. findById() ) it's details using an ID in a postgres database table which is already created in order to get additional information about the signal and enrich the data. I would like to avoid using spring dependencies to perform the lookup (i.e Autowire repository) and want to stick with flink implementation for the lookup.
Where can i specify how to add the postgres connection config information such as port, database, url, username, password etc... (for simplicity purposes can assume the postgres db is local in my machine). Is it as simple as adding the configuration to the application.properties file? if so how can i write the query method to look up the record in the postgres table when searching by non primary key value?
Some online sources are suggesting using this skeleton code but I am not sure how/id it fits my use case. (I have a EventEntity model created which contains all the params/columns from the table which i'm looking up).
like so
public class DatabaseMapper extends RichFlatMapFunction<String, EventEntity> {
// Declare DB connection & query statements
public void open(Configuration parameters) throws Exception {
//Initialize DB connection
//prepare query statements
}
#Override
public void flatMap(String value, Collector<EventEntity> out) throws Exception {
}
}
Your sample code is correct. You can set all your custom initialization and preparation code for PostgreSQL in open() method. Then you can use your pre-configured fields in your flatMap() function.
Here is one sample for Redis operations
I have used RichAsyncFunction here and I suggest you do the same as it is suggested as best practice. Read here for more: https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/operators/asyncio.html)
You can pass configuration parameteres in your constructor method and use it in your initialization process
public static class AsyncRedisOperations extends RichAsyncFunction<Object,Object> {
private JedisPool jedisPool;
private Configuration redisConf;
public AsyncRedisOperations(Configuration redisConf) {
this.redisConf = redisConf;
}
#Override
public void open(Configuration parameters) {
JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
jedisPoolConfig.setMaxTotal(this.redisConf.getInteger("pool", 8));
jedisPoolConfig.setMaxIdle(this.redisConf.getInteger("pool", 8));
jedisPoolConfig.setMaxWaitMillis(this.redisConf.getInteger("maxWait", 0));
JedisPool jedisPool = new JedisPool(jedisPoolConfig,
this.redisConf.getString("host", "192.168.10.10"),
this.redisConf.getInteger("port", 6379), 5000);
try {
this.jedisPool = jedisPool;
this.logger.info("Redis connected: " + jedisPool.getResource().isConnected());
} catch (Exception e) {
this.logger.error(BaseUtil.append("Exception while connecting Redis"));
}
}
#Override
public void asyncInvoke(Object in, ResultFuture<Object> out) {
try (Jedis jedis = this.jedisPool.getResource()) {
String key = jedis.get(key);
this.logger.info("Redis Key: " + key);
}
}
}
Now the project is using springmvc+ spring + mybatis + druid + postgresql
The users in the project correspond to the users in the database, so each time you run SQL, you switch the users with the (set role user) command and then perform the crud operations of the database.
My question:
Because there are many connections in the connection pool, the first step is to get the connection of the database, then switch users, and then perform the operation of business SQL on the database. But I don't know which part of the project this logic should be processed, because the connection of the connection pool and the execution of SQL are implemented by the underlying code. Do you have any good plans?
Can you provide me with a complete demo, such as the following operations:
Step 1, get the user's name from spring security (or shiro).
Step 2, Get the connection currently using the database from the connection pool.
Step 3, execute SQL (set role user) to switch roles.
Step 4, perform crud operation.
Step 5, Reset the database connection(reset role)
Here is a simple way to do what you need with the help of mybatis-spring.
Unless you already use mybatis-spring the first step would be to change the configuration of your project so that you obtain SqlSessionFactory using org.mybatis.spring.SqlSessionFactoryBean provided by mybatis-spring.
The next step is the implementation of setting/resetting the user role for the connection. In mybatis the connection lifecycle is controlled by the class implementing org.apache.ibatis.transaction.Transaction interface. The instance of this class is used by the query executor to get the connection.
In a nutshell you need to create your own implementation of this class and configure mybatis to use it.
Your implementation can be based on the SpringManagedTransaction from mybatis-spring and would look something like:
import org.springframework.security.core.Authentication;
class UserRoleAwareSpringManagedTransaction extends SpringManagedTransaction {
public UserRoleAwareSpringManagedTransaction(DataSource dataSource) {
super(dataSource);
}
#Override
public Connection getConnection() throws SQLException {
Connection connection = getCurrentConnection();
setUserRole(connection);
return connection;
}
private Connection getCurrentConnection() {
return super.getConnection();
}
#Override
public void close() throws SQLException {
resetUserRole(getCurrentConnection());
super.close();
}
private void setUserRole(Connection connection) {
Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
String username = authentication.getName();
Statement statement = connection.createStatement();
try {
// note that this direct usage of usernmae is a subject for SQL injection
// so you need to use the suggestion from
// https://stackoverflow.com/questions/2998597/switch-role-after-connecting-to-database
// about encoding of the username
statement.execute("set role '" + username + "'");
} finally {
statement.close();
}
}
private void resetUserRole(Connection connection) {
Statement statement = connection.createStatement();
try {
statement.execute("reset role");
} finally {
statement.close();
}
}
}
Now you need to configure mybatis to use you Transaction implementation. For this you need to implement TransactionFactory similar to org.mybatis.spring.transaction.SpringManagedTransactionFactory provided by mybatis-spring:
public class UserRoleAwareSpringManagedTransactionFactory implements TransactionFactory {
#Override
public Transaction newTransaction(DataSource dataSource, TransactionIsolationLevel level, boolean autoCommit) {
return new UserRoleAwareSpringManagedTransaction(dataSource);
}
#Override
public Transaction newTransaction(Connection conn) {
throw new UnsupportedOperationException("New Spring transactions require a DataSource");
}
#Override
public void setProperties(Properties props) {
}
}
And then define a bean of type UserRoleAwareSpringManagedTransactionFactory in your spring context and inject it into transactionFactory property of the SqlSessionFactoryBeen in your spring context.
Now every time mybatis obtains a Connection the implementation of Transaction will set the current spring security user to set the role.
Best practice is that database users are applications. Application users' access to particular data/resource should be controlled in the application. Applications should not rely on database to restrict data/resource access. Therefore, application users should not have different roles in database. An application should use only a single database user account.
Spring is manifestation of best practices. Therefore, Spring does not implement this functionality. If you want such functionality, you need to hack.
Referring to this, your best bet is to:
#Autowired JdbcTemplate jdbcTemplate;
// ...
public runPerUserSql() {
jdbcTemplate.execute("set role user 'user_1';");
jdbcTemplate.execute("SELECT 1;");
}
I still do not have much confidence in this. Unless you are writing a pgAdmin webapp for multiple users, you should re-consider your approach and design.
I'm trying to do some testing on a piece of legacy code and I have hit a bit of a wall. The code is part of a backend server for one of our angular web applications. The particular code I need to test is responsible for managing the creation of a sales rep account using data passed in from the client. In addition to saving the new rep to our mongo db, the code also has to handle saving to an external sql db. The work flow looks something like this:
Receive 'put' request from client.
Create a new 'rep' object with the passed in data.
Save the rep to a mongo db.
Call a Groovy class that will do an 'insert' on a remote db and return the id of the remote record.
Save the remote id into the mongo data.
Normally, I would use Mockito to mock out the sql connection, but I couldn't get that to work for this case. My next thought was to try mocking the Groovy class instead. I don't actually care about the internals of the groovy method, I just need an Id back. So far that has not worked either. The reason I suspect is that the groovy method is getting called from a protected method inside my service class. I don't have any control over the signature of this method, it is an override from another library.
Is there anything I can do to be able to test this code without having to set up an actual connection to the Sql db?
Web Service:
#Override
protected void beforeInsert() {
super.beforeInsert();
final Rep weakRep = this;
dwPhase1 = injector.getInstance(DwPhase1.class);
return dwPhase1.insertRepDetail(weakRep);
}
Mocking code:
DwPhase1 dwGroovy = Mockito.mock(DwPhase1.class);
Mockito.when(dwGroovy.insertRepDetail(Mockito.any(Rep.class))).then(new Answer<Object>() {
#Override
public Object answer(InvocationOnMock invocation) throws Throwable {
// TODO Auto-generated method stub
return "hello";
}
});
}
Groovy snippet:
class DwPhase1 extends SQL{
public Number insertRepDetail(Rep r){
String insert="""insert into REP_DETAIL (some values)""";
List<List<Object>> rows =null;
sql.withTransaction {
if(update!=null){
sql.executeUpdate(update);
}
rows = sql.executeInsert(params,insert.toString());
}
Number dwId = null;
if(rows!=null && !rows.isEmpty()){
List columns=rows.get(0);
if(columns!=null && !columns.isEmpty()){
dwId = columns.get(0);
}
}
return dwId;
}
}
Basically I have two services each one of them handle methods for each persistent object that I have in my project, these services hold some method which the endpoint(Google) will call to perform something.
I'm using Google Could Endpoints + Mysql Cloud + Hibernate.
Two POs
#Entity
public class Device {
...
}
#Entity
public class User {
...
}
The services for each one of POs
public class DeviceService {
Device getDevice(Long devId){
return new Dao().getById(devId, Device.class);
}
void allocateDevice(Long userId){
User u = new UserService().getUser(userId);
... do stuff
}
}
public class UserService {
User getUser(Long userId){
return new Dao().getById(userId, User.class);
}
}
The endpoint for each one
public class DeviceEndpoint {
#ApiMethod(
name = "device.get",
path = "device/{devId}",
httpMethod = ApiMethod.HttpMethod.GET
)
Device getDevice(Long devId){
MyEntityManager em = new MyEntityManager();
try {
em.getTransaction().begin();
new DeviceService().getDevice(devId);
em.getTransaction().commit();
}finally {
em.cleanup(); //custom method to rollback also
}
return device;
}
#ApiMethod(
name = "device.allocate",
path = "device/{userId}/allocate",
httpMethod = ApiMethod.HttpMethod.GET
)
void allocateDevice(Long deviceId){
MyEntityManager em = new MyEntityManager();
try {
em.getTransaction().begin();
new DeviceService().allocateDevice(userId);
em.getTransaction().commit();
}finally {
em.cleanup(); //custom method to rollback also
}
}
}
I would like to know where I put the database transaction logic(begin,commit,rollback).
Dao layer
Firstly I had inserted into Dao class, but every query/insert/update I had to open and close the connection and when I had to use more than one CRUD I did several open/close connections and it had been expensive and delayed.
Example: In one endpoint request I want to obtain some object from db and update. Two operations and two open/close connections.
Endpoint layer(as example)
Secondly I put the logic to open/close on endpoint methods(as example above), but they said(my work colleagues) it isn't a good pattern, begin and commit transactions in this layer isn't a good idea, then they suggested to do the third option.
Service layer
Put that logic(begin/commit/rollback) into Service layer, in each method, I tried but, some methods call another and that last also open and close the connection, so when the second method return, the transaction came closed.
Please, let me know case missing some important info.
Typically This type of action is performed in the Service Layer as this layer is there to provide logic to operate on the data sent to and from the DAO layer - that being said you could bundle these together into the same module.
The comment "I tried but, some methods call another and that last also open and close the connection, so when the second method return, the transaction came closed." Is interesting; I am not sure how you are managing your connections; but you may want/need to revisit if your connections are being closed before transactions are completed - you may want to look at Hibernates HibernateTransactionManager
Where should "#Transactional" be place Service Layer or DAO