I'm trying to test legacy Java application, without an ability to re-factor its code at the moment. All I need to do is to understand what SQL requests it is sending through JDBC and when. All these requests I'd like to protocol to a plain text file for future review.
I need to develop a custom JDBC driver which will fool the application, and behave like a normal driver (will accept request, return some values, fail on certain conditions, etc). A mock structure, as usual...
The question is - do you know any existing frameworks for this task? Ideally I would like to have an ability to configure my driver's behavior through, say, XML file.
p6spy wraps an existing jdbc connection, and allows you to e.g. see what goes back and forth.
See http://www.mkyong.com/hibernate/how-to-display-hibernate-sql-parameter-values-solution/ for instructions.
If you want to do unit tests, not an integration tests, than
you can use a very basic and simple approach, using Mockito only, like this:
public class JDBCLowLevelTest {
private TestedClass tested;
private Connection connection;
private static Driver driver;
#BeforeClass
public static void setUpClass() throws Exception {
// (Optional) Print DriverManager logs to system out
DriverManager.setLogWriter(new PrintWriter((System.out)));
// (Optional) Sometimes you need to get rid of a driver (e.g JDBC-ODBC Bridge)
Driver configuredDriver = DriverManager.getDriver("jdbc:odbc:url");
System.out.println("De-registering the configured driver: " + configuredDriver);
DriverManager.deregisterDriver(configuredDriver);
// Register the mocked driver
driver = mock(Driver.class);
System.out.println("Registering the mock driver: " + driver);
DriverManager.registerDriver(driver);
}
#AfterClass
public static void tearDown() throws Exception {
// Let's cleanup the global state
System.out.println("De-registering the mock driver: " + driver);
DriverManager.deregisterDriver(driver);
}
#Before
public void setUp() throws Exception {
// given
tested = new TestedClass();
connection = mock(Connection.class);
given(driver.acceptsURL(anyString())).willReturn(true);
given(driver.connect(anyString(), Matchers.<Properties>any()))
.willReturn(connection);
given(connection.prepareCall(anyString())).willReturn(statement);
}
}
Than you can test various scenarios, like in any other Mockito test e.g.
#Test
public void shouldHandleDoubleException() throws Exception {
// given
SomeData someData = new SomeData();
given(connection.prepareCall(anyString()))
.willThrow(new SQLException("Prepare call"));
willThrow(new SQLException("Close exception")).given(connection).close();
// when
SomeResponse response = testClass.someMethod(someData);
// then
assertThat(response, is(SOME_ERROR));
}
Related
Often when I consider a new library or technology, I create a small POC or test program to get a feel for it. So I did with gRPC-Spring-Boot-Starter. A simple example code is posted below my question text.
This sample has been extended in complexity and eventually, the library has found its way into production code. So far it has survived many runs under moderate load.
Note that, naturally, the production service is not client to itself. But the production gRPC service is in fact client to other gRPC services.
Now I was thinking to write some kind of between-unit-and-integration test where I spin up local instances (starting with a single one) of those other gRPC services (pulling data from some static local resource, for example). Basically, this test code looks very much like the one posted below my question.
However - as soon as we poll for results in forEachRemaining(), the test ends up hanging: I suspect a deadlock in ClientCalls#waitAndDrain (io.grpc:grpc-stub).
The funny thing is - this does not happen if the client were created "manually", i.e. without utilizing the third-party Spring extension:
ManagedChannel channel = ManagedChannelBuilder.forTarget("localhost:9091")
.defaultLoadBalancingPolicy("round_robin")
.usePlaintext()
.build();
StockStaticDataRequestServiceBlockingStub stub = StockStaticDataRequestServiceGrpc.newBlockingStub(channel);
I use Spring Boot 2.6.3, gRPC-Spring-Boot-Starter 2.13.1, gRPC 1.44.0, proto 3.19.2 and netty 4.1.73, for what it is worth.
Now I wonder if someone here encountered similar issues or can give me some pointers while I am trying to figure out the inner workings of gRPC more.
Added sample project on GH.
The main branch contains the - maybe dubious - test setup I chose in the beginning, branches are some refinements, like using #Abhijit Sarkar's grpc-test library. Tests are green so far.
grpc:
client:
stocks:
address: 'static://localhost:9091'
enableKeepAlive: false
negotiationType: plaintext
server:
port: 9092
#SpringBootTest
class TestGrpc {
#GrpcClient("stocks")
private StockStaticDataRequestServiceBlockingStub stub;
#BeforeAll
public static void setUp() throws Exception {
final Server server = ServerBuilder
.forPort(9091)
.addService(new StockStaticDataRequestTestService())
.build();
server.start();
final Thread serverThread = new Thread(() -> {
try {
server.awaitTermination();
} catch (final InterruptedException e) {
Thread.currentThread().interrupt();
}
});
serverThread.setDaemon(false);
serverThread.start();
}
#Test
void testClient() {
StockStaticManyDataRequest request = StockStaticManyDataRequest.newBuilder()
.addAllTickerSymbols(List.of("AAPL"))
.build();
stub.getManyStockStatics(request).forEachRemaining(security -> {
LOG.info("security={}", security);
});
}
}
public class StockStaticDataRequestTestService extends StockStaticDataRequestServiceImplBase {
#Override
public void getManyStockStatics(StockStaticManyDataRequest request, StreamObserver<Security> responseObserver) {
responseObserver.onNext(Security.newBuilder()
.setSecurity("TEST-MANY")
.build());
responseObserver.onNext(Security.newBuilder()
.setSecurity("TEST-MORE")
.build());
responseObserver.onCompleted();
}
}
message Security {
string tickerSymbol = 1;
string security = 2;
}
message StockStaticManyDataRequest {
repeated string tickerSymbols = 1;
}
service StockStaticDataRequestService {
rpc getManyStockStatics(StockStaticManyDataRequest) returns (stream Security) {}
}
I think what the problem might be is that you should not be starting the server at all. There are some grpc-spring-boot-starter annotations that should be added to a test configuration class that will start / stop the server. See details here.
https://yidongnan.github.io/grpc-spring-boot-starter/en/server/testing.html#integration-tests
I also tried to make what you have work, but the server once started really won't shutdown. This makes the next test suite that runs fail due to port conflicts when it tries to start.
Here's my test class.
#Slf4j
#SpringBootTest
#ActiveProfiles("test")
#SpringJUnitConfig(classes = { ServiceIntegrationTestConfiguration.class })
#DirtiesContext
class TestGprc {
#GrpcClient("stocks")
private StockStaticDataRequestServiceBlockingStub stub;
/**
* #throws java.lang.Exception
*/
#BeforeAll
static void setUpBeforeClass() throws Exception {
log.info("setUpBeforeClass");
}
/**
* #throws java.lang.Exception
*/
#AfterAll
static void tearDownAfterClass() throws Exception {
log.info("tearDownAfterClass");
}
/**
* #throws java.lang.Exception
*/
#BeforeEach
void setUp() throws Exception {
}
/**
* #throws java.lang.Exception
*/
#AfterEach
void tearDown() throws Exception {
}
#Test
#DirtiesContext
void testClient() {
StockStaticManyDataRequest request = StockStaticManyDataRequest.newBuilder()
.addAllTickerSymbols(List.of("AAPL"))
.build();
stub.getManyStockStatics(request).forEachRemaining(security -> {
log.info("security={}", security);
});
}
}
Here's the configuration project.
#Configuration
#ImportAutoConfiguration({ GrpcServerAutoConfiguration.class, // Create required server beans
GrpcServerFactoryAutoConfiguration.class, // Select server implementation
GrpcClientAutoConfiguration.class,
GrpcStarterApplication.class})
public class ServiceIntegrationTestConfiguration {
// add mock beans here of needed.
}
My overrides for the properties. see application-test.yaml
grpc:
client:
stocks:
address: in-process:test
enableKeepAlive:
negotiationType:
server:
inProcessName: test
port: -1
I posted the entire maven project here:
https://github.com/aerobiotic/grpc-spring-starter
Simply clone it and mvn clean install :-)
As far as your dead-lock goes in your production code:
make sure you are calling onCompleted
check your catch blocks and make sure onError is being called and that there is logging happening.
It's possible that starting the server and not getting it shutdown is affecting something. Perhaps test code is connecting to a server from a previous test.
I have a SpringBoot gradle project using apache flink to process datastream signals. When a new signal comes through the datastream, I would like to query look up (i.e. findById() ) it's details using an ID in a postgres database table which is already created in order to get additional information about the signal and enrich the data. I would like to avoid using spring dependencies to perform the lookup (i.e Autowire repository) and want to stick with flink implementation for the lookup.
Where can i specify how to add the postgres connection config information such as port, database, url, username, password etc... (for simplicity purposes can assume the postgres db is local in my machine). Is it as simple as adding the configuration to the application.properties file? if so how can i write the query method to look up the record in the postgres table when searching by non primary key value?
Some online sources are suggesting using this skeleton code but I am not sure how/id it fits my use case. (I have a EventEntity model created which contains all the params/columns from the table which i'm looking up).
like so
public class DatabaseMapper extends RichFlatMapFunction<String, EventEntity> {
// Declare DB connection & query statements
public void open(Configuration parameters) throws Exception {
//Initialize DB connection
//prepare query statements
}
#Override
public void flatMap(String value, Collector<EventEntity> out) throws Exception {
}
}
Your sample code is correct. You can set all your custom initialization and preparation code for PostgreSQL in open() method. Then you can use your pre-configured fields in your flatMap() function.
Here is one sample for Redis operations
I have used RichAsyncFunction here and I suggest you do the same as it is suggested as best practice. Read here for more: https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/operators/asyncio.html)
You can pass configuration parameteres in your constructor method and use it in your initialization process
public static class AsyncRedisOperations extends RichAsyncFunction<Object,Object> {
private JedisPool jedisPool;
private Configuration redisConf;
public AsyncRedisOperations(Configuration redisConf) {
this.redisConf = redisConf;
}
#Override
public void open(Configuration parameters) {
JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
jedisPoolConfig.setMaxTotal(this.redisConf.getInteger("pool", 8));
jedisPoolConfig.setMaxIdle(this.redisConf.getInteger("pool", 8));
jedisPoolConfig.setMaxWaitMillis(this.redisConf.getInteger("maxWait", 0));
JedisPool jedisPool = new JedisPool(jedisPoolConfig,
this.redisConf.getString("host", "192.168.10.10"),
this.redisConf.getInteger("port", 6379), 5000);
try {
this.jedisPool = jedisPool;
this.logger.info("Redis connected: " + jedisPool.getResource().isConnected());
} catch (Exception e) {
this.logger.error(BaseUtil.append("Exception while connecting Redis"));
}
}
#Override
public void asyncInvoke(Object in, ResultFuture<Object> out) {
try (Jedis jedis = this.jedisPool.getResource()) {
String key = jedis.get(key);
this.logger.info("Redis Key: " + key);
}
}
}
We are developing a new desktop application in JavaFx wherein for offline storage we are using SQLite and for orm we are using ormlite.
I want to implement DB connection pooling wherein a fixed number of connections should be set at the start and should be used, released and reused as required. Also, it would be good if we can make use of "readonly" and "writeonly" connections appropriately to maximize performance.
This is what we have written so far.
public class DAO {
private static JdbcPooledConnectionSource connectionSource;
private static DAO instance = null;
private DAO() throws SQLException {
try {
final File path = SystemUtils.getDatabaseFile();
final String DATABASE_URL = Constants.DATABASE_URL + path.getAbsolutePath();
Class.forName(Constants.DATABASE_DRIVER);
connectionSource = new JdbcPooledConnectionSource(DATABASE_URL);
//connectionSource.setMaxConnectionAgeMillis(5 * 60 * 1000);
connectionSource.setCheckConnectionsEveryMillis(5000);
connectionSource.setMaxConnectionsFree(5);
connectionSource.initialize();
init();
} catch (ClassNotFoundException cnfe) {
cnfe.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
}
}
private void init() throws ClassNotFoundException, SQLException {
TableUtils.createTableIfNotExists(connectionSource, Customer.class);
TableUtils.createTableIfNotExists(connectionSource, Address.class);
TableUtils.createTableIfNotExists(connectionSource, Location.class);
TableUtils.createTableIfNotExists(connectionSource, City.class);
TableUtils.createTableIfNotExists(connectionSource, Area.class);
TableUtils.createTableIfNotExists(connectionSource, Category.class);
TableUtils.createTableIfNotExists(connectionSource, Product.class);
TableUtils.createTableIfNotExists(connectionSource, AddonCategory.class);
TableUtils.createTableIfNotExists(connectionSource, ProductAddon.class);
}
public synchronized <D extends Dao<T, ?>, T> D getDao(Class<T> cls) throws SQLException {
Dao<T, ?> dao = DaoManager.createDao(connectionSource, cls);
D daoImpl = (D) dao;
return daoImpl;
}
public synchronized static DAO getInstance() throws SQLException {
if (instance == null) instance = new DAO();
return instance;
}
}
The problem here is everytime we are creating table (TableUtils.createTableIfNotExists) the pooled connection source is making a new connection and not reusing the one earlier used/created.
Not finding enough code examples on Internet on how to correctly use JdbcPooledConnectionSource.
JdbcPooledConnectionSource correct usage
Which SQLite driver are you using? The Xerial driver has the Sqlite code actually compiled into the Jar. This means that you really aren't "connecting" to another database just making calls to the database directly even though it is fronted by a JDBC interface.
This means that you really don't need a JdbcPooledConnectionSource. The pooled connections really only help when you are making connections over the network to a database server. If you look at your application, you should see file-descriptors (in /prod/#/fd if in Linux) which show open FDs to the database but not to sockets.
The problem here is everytime we are creating table (TableUtils.createTableIfNotExists) the pooled connection source is making a new connection and not reusing the one earlier used/created.
Hrm. I have some good coverage in unit tests around the pooled connection source. I'm surprised to hear that it isn't reusing connections. Can you get me a unit test to demonstrate it?
I want to override properties defined in application.properties in tests, but #TestPropertySource only allows to provide predefined values.
What I need is to start a server on a random port N, then pass this port to spring-boot application. The port has to be ephemeral to allow running multiple tests on the same host at the same time.
I don't mean the embedded http server (jetty), but some different server that is started at the beginning of the test (e.g. zookeeper) and the application being tested has to connect to it.
What's the best way to achieve this?
(here's a similar question, but answers do not mention a solution for ephemeral ports - Override default Spring-Boot application.properties settings in Junit Test)
As of Spring Framework 5.2.5 and Spring Boot 2.2.6 you can use Dynamic Properties in tests:
#DynamicPropertySource
static void dynamicProperties(DynamicPropertyRegistry registry) {
registry.add("property.name", "value");
}
Thanks to the changes made in Spring Framework 5.2.5, the use of #ContextConfiguration and the ApplicationContextInitializer can be replaced with a static #DynamicPropertySource method that serves the same purpose.
#SpringBootTest
#Testcontainers
class SomeSprintTest {
#Container
static LocalStackContainer localStack =
new LocalStackContainer().withServices(LocalStackContainer.Service.S3);
#DynamicPropertySource
static void initialize(DynamicPropertyRegistry registry) {
AwsClientBuilder.EndpointConfiguration endpointConfiguration =
localStack.getEndpointConfiguration(LocalStackContainer.Service.S3);
registry.add("cloud.aws.s3.default-endpoint", endpointConfiguration::getServiceEndpoint);
}
}
You could override the value of the port property in the #BeforeClass like this:
#BeforeClass
public static void beforeClass() {
System.setProperty("zookeeper.port", getRandomPort());
}
The "clean" solution is to use an ApplicationContextInitializer.
See this answer to a similar question.
See also this github issue asking a similar question.
To summarize the above mentioned posts using a real-world example that's been sanitized to protect copyright holders (I have a REST endpoint which uses an #Autowired DataSource which needs to use the dynamic properties to know which port the in-memory MySQL database is using):
Your test must declare the initializer (see the #ContextConfiguration line below).
// standard spring-boot test stuff
#RunWith(SpringRunner.class)
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
#ActiveProfiles("local")
#ContextConfiguration(
classes = Application.class,
// declare the initializer to use
initializers = SpringTestDatabaseInitializer.class)
// use random management port as well so we don't conflict with other running tests
#TestPropertySource(properties = {"management.port=0"})
public class SomeSprintTest {
#LocalServerPort
private int randomLocalPort;
#Value("${local.management.port}")
private int randomManagementPort;
#Test
public void testThatDoesSomethingUseful() {
// now ping your service that talks to the dynamic resource
}
}
Your initializer needs to add the dynamic properties to your environment. Don't forget to add a shutdown hook for any cleanup that needs to run. Following is an example that sets up an in-memory database using a custom DatabaseObject class.
public class SpringTestDatabaseInitializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
private static final int INITIAL_PORT = 0; // bind to an ephemeral port
private static final String DB_USERNAME = "username";
private static final String DB_PASSWORD = "password-to-use";
private static final String DB_SCHEMA_NAME = "default-schema";
#Override
public void initialize(ConfigurableApplicationContext applicationContext) {
DatabaseObject databaseObject = new InMemoryDatabaseObject(INITIAL_PORT, DB_USERNAME, DB_PASSWORD, DB_SCHEMA_NAME);
registerShutdownHook(databaseObject);
int databasePort = startDatabase(databaseObject);
addDatabasePropertiesToEnvironment(applicationContext, databasePort);
}
private static void addDatabasePropertiesToEnvironment(ConfigurableApplicationContext applicationContext, int databasePort) {
String url = String.format("jdbc:mysql://localhost:%s/%s", databasePort, DB_SCHEMA_NAME);
System.out.println("Adding db props to environment for url: " + url);
TestPropertySourceUtils.addInlinedPropertiesToEnvironment(
applicationContext,
"db.port=" + databasePort,
"db.schema=" + DB_SCHEMA_NAME,
"db.url=" + url,
"db.username=" + DB_USERNAME,
"db.password=" + DB_PASSWORD);
}
private static int startDatabase(DatabaseObject database) {
try {
database.start();
return database.getBoundPort();
} catch (Exception e) {
throw new IllegalStateException("Failed to start database", e);
}
}
private static void registerShutdownHook(DatabaseObject databaseObject) {
Runnable shutdownTask = () -> {
try {
int boundPort = databaseObject.getBoundPort();
System.out.println("Shutting down database at port: " + boundPort);
databaseObject.stop();
} catch (Exception e) {
// nothing to do here
}
};
Thread shutdownThread = new Thread(shutdownTask, "Database Shutdown Thread");
Runtime.getRuntime().addShutdownHook(shutdownThread);
}
}
When I look at the logs, it shows that for both of my tests that use this initializer class, they use the same object (the initialize method only gets called once, as does the shutdown hook). So it starts up a database, and leaves it running until both tests finish, then shuts the database down.
I am developping a Swing application that needs to communicate with a distant HTTP server. That application can be potentially used behind proxies.
I must then :
- detect automatically network proxy (potentially several on the same network)
- let the user manually enter a proxy configuration.
I want to write an integration test to validate thoses aspects, without having to install proxies on every CI machines and every developper machine.
How I see things :
integration test (with junit) start an "embedded" proxy (#BeforeClass) and a somewhat dummy http server
my tests (#Test)
test that this proxy can be detected automatically and open a connection to my dummy http server and successfully retrieve datas from it
manually set the proxy and perform the same test as above
I have heard about the "littleProxy" component but didn"t tried it yet.
Can anyone shed some advice / help / guidance regarding the best approach to solve my problem ?
I would consider whether you are testing the right thing. You don't need to test proxy servers or Java's network classes.
Consider this utility type for reading data from a URL:
public final class Network {
public interface UrlOpener {
public InputStream open(URL url) throws IOException;
}
private static UrlOpener urlOpener = new UrlOpener() {
public InputStream open(URL url) throws IOException {
return url.openStream();
}
};
public static InputStream openUrl(URL url) throws IOException {
return urlOpener.open(url);
}
public static void setUrlOpener(UrlOpener urlOpener) {
Network.urlOpener = urlOpener;
}
}
This can be used as an abstraction layer between your code and Java's network I/O:
public class SomeType {
public void processData(URL url) throws IOException {
InputStream input = Network.openUrl(url);
// process the data
}
}
Your tests use this layer to mock out the data:
#Before public void setup() throws IOException {
final URL mockUrl = this.getClass().getResource("/foo/bar.txt");
Network.UrlOpener opener = Mockito.mock(Network.UrlOpener.class);
Answer<InputStream> substituteUrl = new Answer<InputStream>() {
public InputStream answer(InvocationOnMock invocation) throws Throwable {
return mockUrl.openStream();
}
};
Mockito.when(opener.open(Mockito.any(URL.class))).then(substituteUrl);
Network.setUrlOpener(opener);
}
#Test public void testSomething() throws IOException {
SomeType something = new SomeType();
something.processData(new URL("http://example.com"));
}
This saves any mucking around with firewalls etc.
In order for this approach to work you would want to have confidence in a set of recorded transactions from real servers to use in your tests.
This approach can be complemented with a dedicated machine running more comprehensive integration tests.