I am developping a Swing application that needs to communicate with a distant HTTP server. That application can be potentially used behind proxies.
I must then :
- detect automatically network proxy (potentially several on the same network)
- let the user manually enter a proxy configuration.
I want to write an integration test to validate thoses aspects, without having to install proxies on every CI machines and every developper machine.
How I see things :
integration test (with junit) start an "embedded" proxy (#BeforeClass) and a somewhat dummy http server
my tests (#Test)
test that this proxy can be detected automatically and open a connection to my dummy http server and successfully retrieve datas from it
manually set the proxy and perform the same test as above
I have heard about the "littleProxy" component but didn"t tried it yet.
Can anyone shed some advice / help / guidance regarding the best approach to solve my problem ?
I would consider whether you are testing the right thing. You don't need to test proxy servers or Java's network classes.
Consider this utility type for reading data from a URL:
public final class Network {
public interface UrlOpener {
public InputStream open(URL url) throws IOException;
}
private static UrlOpener urlOpener = new UrlOpener() {
public InputStream open(URL url) throws IOException {
return url.openStream();
}
};
public static InputStream openUrl(URL url) throws IOException {
return urlOpener.open(url);
}
public static void setUrlOpener(UrlOpener urlOpener) {
Network.urlOpener = urlOpener;
}
}
This can be used as an abstraction layer between your code and Java's network I/O:
public class SomeType {
public void processData(URL url) throws IOException {
InputStream input = Network.openUrl(url);
// process the data
}
}
Your tests use this layer to mock out the data:
#Before public void setup() throws IOException {
final URL mockUrl = this.getClass().getResource("/foo/bar.txt");
Network.UrlOpener opener = Mockito.mock(Network.UrlOpener.class);
Answer<InputStream> substituteUrl = new Answer<InputStream>() {
public InputStream answer(InvocationOnMock invocation) throws Throwable {
return mockUrl.openStream();
}
};
Mockito.when(opener.open(Mockito.any(URL.class))).then(substituteUrl);
Network.setUrlOpener(opener);
}
#Test public void testSomething() throws IOException {
SomeType something = new SomeType();
something.processData(new URL("http://example.com"));
}
This saves any mucking around with firewalls etc.
In order for this approach to work you would want to have confidence in a set of recorded transactions from real servers to use in your tests.
This approach can be complemented with a dedicated machine running more comprehensive integration tests.
Related
Often when I consider a new library or technology, I create a small POC or test program to get a feel for it. So I did with gRPC-Spring-Boot-Starter. A simple example code is posted below my question text.
This sample has been extended in complexity and eventually, the library has found its way into production code. So far it has survived many runs under moderate load.
Note that, naturally, the production service is not client to itself. But the production gRPC service is in fact client to other gRPC services.
Now I was thinking to write some kind of between-unit-and-integration test where I spin up local instances (starting with a single one) of those other gRPC services (pulling data from some static local resource, for example). Basically, this test code looks very much like the one posted below my question.
However - as soon as we poll for results in forEachRemaining(), the test ends up hanging: I suspect a deadlock in ClientCalls#waitAndDrain (io.grpc:grpc-stub).
The funny thing is - this does not happen if the client were created "manually", i.e. without utilizing the third-party Spring extension:
ManagedChannel channel = ManagedChannelBuilder.forTarget("localhost:9091")
.defaultLoadBalancingPolicy("round_robin")
.usePlaintext()
.build();
StockStaticDataRequestServiceBlockingStub stub = StockStaticDataRequestServiceGrpc.newBlockingStub(channel);
I use Spring Boot 2.6.3, gRPC-Spring-Boot-Starter 2.13.1, gRPC 1.44.0, proto 3.19.2 and netty 4.1.73, for what it is worth.
Now I wonder if someone here encountered similar issues or can give me some pointers while I am trying to figure out the inner workings of gRPC more.
Added sample project on GH.
The main branch contains the - maybe dubious - test setup I chose in the beginning, branches are some refinements, like using #Abhijit Sarkar's grpc-test library. Tests are green so far.
grpc:
client:
stocks:
address: 'static://localhost:9091'
enableKeepAlive: false
negotiationType: plaintext
server:
port: 9092
#SpringBootTest
class TestGrpc {
#GrpcClient("stocks")
private StockStaticDataRequestServiceBlockingStub stub;
#BeforeAll
public static void setUp() throws Exception {
final Server server = ServerBuilder
.forPort(9091)
.addService(new StockStaticDataRequestTestService())
.build();
server.start();
final Thread serverThread = new Thread(() -> {
try {
server.awaitTermination();
} catch (final InterruptedException e) {
Thread.currentThread().interrupt();
}
});
serverThread.setDaemon(false);
serverThread.start();
}
#Test
void testClient() {
StockStaticManyDataRequest request = StockStaticManyDataRequest.newBuilder()
.addAllTickerSymbols(List.of("AAPL"))
.build();
stub.getManyStockStatics(request).forEachRemaining(security -> {
LOG.info("security={}", security);
});
}
}
public class StockStaticDataRequestTestService extends StockStaticDataRequestServiceImplBase {
#Override
public void getManyStockStatics(StockStaticManyDataRequest request, StreamObserver<Security> responseObserver) {
responseObserver.onNext(Security.newBuilder()
.setSecurity("TEST-MANY")
.build());
responseObserver.onNext(Security.newBuilder()
.setSecurity("TEST-MORE")
.build());
responseObserver.onCompleted();
}
}
message Security {
string tickerSymbol = 1;
string security = 2;
}
message StockStaticManyDataRequest {
repeated string tickerSymbols = 1;
}
service StockStaticDataRequestService {
rpc getManyStockStatics(StockStaticManyDataRequest) returns (stream Security) {}
}
I think what the problem might be is that you should not be starting the server at all. There are some grpc-spring-boot-starter annotations that should be added to a test configuration class that will start / stop the server. See details here.
https://yidongnan.github.io/grpc-spring-boot-starter/en/server/testing.html#integration-tests
I also tried to make what you have work, but the server once started really won't shutdown. This makes the next test suite that runs fail due to port conflicts when it tries to start.
Here's my test class.
#Slf4j
#SpringBootTest
#ActiveProfiles("test")
#SpringJUnitConfig(classes = { ServiceIntegrationTestConfiguration.class })
#DirtiesContext
class TestGprc {
#GrpcClient("stocks")
private StockStaticDataRequestServiceBlockingStub stub;
/**
* #throws java.lang.Exception
*/
#BeforeAll
static void setUpBeforeClass() throws Exception {
log.info("setUpBeforeClass");
}
/**
* #throws java.lang.Exception
*/
#AfterAll
static void tearDownAfterClass() throws Exception {
log.info("tearDownAfterClass");
}
/**
* #throws java.lang.Exception
*/
#BeforeEach
void setUp() throws Exception {
}
/**
* #throws java.lang.Exception
*/
#AfterEach
void tearDown() throws Exception {
}
#Test
#DirtiesContext
void testClient() {
StockStaticManyDataRequest request = StockStaticManyDataRequest.newBuilder()
.addAllTickerSymbols(List.of("AAPL"))
.build();
stub.getManyStockStatics(request).forEachRemaining(security -> {
log.info("security={}", security);
});
}
}
Here's the configuration project.
#Configuration
#ImportAutoConfiguration({ GrpcServerAutoConfiguration.class, // Create required server beans
GrpcServerFactoryAutoConfiguration.class, // Select server implementation
GrpcClientAutoConfiguration.class,
GrpcStarterApplication.class})
public class ServiceIntegrationTestConfiguration {
// add mock beans here of needed.
}
My overrides for the properties. see application-test.yaml
grpc:
client:
stocks:
address: in-process:test
enableKeepAlive:
negotiationType:
server:
inProcessName: test
port: -1
I posted the entire maven project here:
https://github.com/aerobiotic/grpc-spring-starter
Simply clone it and mvn clean install :-)
As far as your dead-lock goes in your production code:
make sure you are calling onCompleted
check your catch blocks and make sure onError is being called and that there is logging happening.
It's possible that starting the server and not getting it shutdown is affecting something. Perhaps test code is connecting to a server from a previous test.
I've read CDI 2.0 specification (JSR 365) and found out the existence of the #Observes(during=AFTER_SUCCESS) annotation, but it actually requires a custom event to be defined in order to work.
This is what i've got:
//simple """transactional""" file system manager using command pattern
#Transactional(value = Transactional.TxType.REQUIRED)
#TransactionScoped
#Stateful
public class TransactionalFileSystemManager implements SessionSynchronization {
private final Deque<Command> commands = new ArrayDeque<>();
public void createFile(InputStream content, Path path, String name) throws IOException {
CreateFile command = CreateFile.execute(content, path, name);
commands.addLast(command);
}
public void deleteFile(Path path) throws IOException {
DeleteFile command = DeleteFile.execute(path);
commands.addLast(command);
}
private void commit() throws IOException{
for(Command c : commands){
c.confirm();
}
}
private void rollback() throws IOException{
Iterator<Command> it = commands.descendingIterator();
while (it.hasNext()) {
Command c = it.next();
c.undo();
}
}
#Override
public void afterBegin() throws EJBException{
}
#Override
public void beforeCompletion() throws EJBException{
}
#Override
public void afterCompletion(boolean commitSucceeded) throws EJBException{
if(commitSucceeded){
try {
commit();
} catch (IOException e) {
throw new EJBException(e);
}
}
else {
try {
rollback();
} catch (IOException e) {
throw new EJBException(e);
}
}
}
}
However, I want to adopt a CDI-only solution so I need to remove anything EJB related (including the SessionSynchronization interface). How can i achieve the same result using CDI?
First the facts: the authoritative source for this topic is the Java Transaction API (JTA) specification. Search for it online, I got this.
Then the bad news: In order to truly participate in a JTA transaction, you either have to implement a connector according to the Java Connector Architecture (JCA) specification or a XAResource according to JTA. Never done any of them, I am afraid both are going to be hard. Nevertheless, if you search, you may find an existing implementation of a File System Connector.
Your code above will never accomplish true 2-phase commit because, if your code fails, the transaction is already committed, so the application state is inconsistent. Or, there is a small time window when the real transaction is committed but the file system change have not beed executed, again the state is inconsistent.
Some workarounds I can think of, none of which solves the consistency problem:
Persist the File System commands in a database. This ensures that they are enqueued transactionally. A scheduled job wakes up and actually tries to execute the queued FS commands.
Register a Synchronization with the current Transaction, fire an appropriate event from there. Your TransactionalFileSystemManager observes this event, no during attribute needed I guess.
Here is what I would like to achieve:
On one hand, I have an Oracle database. On the other hand, a "simple" Java application (let's call it "App").
And in the middle, an embedded ApacheDS in Java. The idea is to access that database through the embedded LDAP server.
At the moment, I'm able to connect "App" to the embedded LDAP Server, send parameters to it and execute some sql in the Oracle database.
But the problem is that I can't get the result back to "App".
Apparently, I should use my own "SearchHandler", but I can't figure out how to do it.
I hope my explanations are clear enough. If not, I can try to give more details.
server.setSearchHandler(new LdapRequestHandler<InternalSearchRequest>() {
#Override
public void handle(LdapSession ls, InternalSearchRequest t) throws Exception {
//Getting data from Oracle database
System.out.println(dataFromDatabase);
}
});
A little bit late, but you are on the right path.
I am doing more or less the same (but with BindRequestHandler), using ApacheDS as a LDAP proxy. I'm doing it using version 2.0.0-M23.
I extended it like this:
public class LoggerBindRequestHandler extends BindRequestHandler {
private static final Logger LOGGER = LoggerFactory.getLogger(LoggerBindRequestHandler.class);
#Override
public void handle(LdapSession session, BindRequest request) throws Exception {
LOGGER.debug(session.toString());
LOGGER.debug(request.toString());
super.handle(session, request);
}
#Override
public void handleSaslAuth(LdapSession session, BindRequest request) throws Exception {
LOGGER.debug(session.toString());
LOGGER.debug(request.toString());
super.handleSaslAuth(session, request);
}
#Override
public void handleSimpleAuth(LdapSession session, BindRequest request) throws Exception {
LOGGER.debug(session.toString());
LOGGER.debug(request.toString());
super.handleSimpleAuth(session, request);
}
}
Then I set the ldap server
ldapServer.setBindHandlers(new LoggerBindRequestHandler(), new LoggerBindResponseHandler());
And that's it.
As far I noticed, you are missing the ResponseHandler. This is why you are able to send commands to oracle, but do not send a response.
A bit of background, I'm trying to create a URL Stream Handler so I can keep track of how many connections I have active on my webview in my javafx application. Essentially, I'm running an AngularJs app in the WebView, and I'd like to know when it's finished. I can't touch the web site code, so adding a js notifier is not on the table. So, no matter what I put together, the setup always errors with 'protocol doesn't support input.' I've tried to override 'getDoInput' with a method that only returns false, but I still get the error. Any ideas?
Here is something close to what I'm doing:
public class MyUrlStreamHandlerFactory implements URLStreamHandlerFactory {
public URLStreamHandler createURLStreamHandler(String protocol) {
if (protocol.equalsIgnoreCase("http") || protocol.equalsIgnoreCase("https")) {
return new URLStreamHandler() {
#Override
protected URLConnection openConnection(URL url) throws IOException {
return new HttpURLConnection(url) {
#Override
public void connect() throws IOException {
}
#Override
public void disconnect() {
}
#Override
public boolean usingProxy() {
return false;
}
#Override
public boolean getDoInput() {
return false;
}
};
}
};
}
return null;
}
}
I'm installing it with:
URL.setURLStreamHandlerFactory(new MyUrlStreamHandlerFactory());
I understand what you're trying to accomplish however, I think this is the wrong way to go about it.
From: Java Network Programming by Elliotte Rusty Harold
Only abstract URLConnection classes are present in the java.net package. The concrete subclasses are hidden inside the sun.net package hierarchy. It is rare to instantiate URLConnection objects directly in your source code; instead, the runtime environment creates these objects as needed, depending on the protocol in use. The class (which is unknown at compile time) is then instantiated using the forName() and newInstance() methods of the java.lang.Class class.
For example, the connect() method of sun.net.www.protocol.http.HttpURLConnection creates a sun.net.www.http.HttpClient object, which is responsible for connecting to the server.
So unless you want to write your own http protocol handler and an HttpClient, I would suggest exploring other avenues.
Other Things
The only method, that I could find, that throws an UnknownServiceException with the message being "protocol doesn't support input" is:
java.net.URLConnection#getInputStream
/**
* Returns an input stream that reads from this open connection.
*
* #return an input stream that reads from this open connection.
* #exception IOException if an I/O error occurs while
* creating the input stream.
* #exception UnknownServiceException if the protocol does not support
* input.
*/
public InputStream getInputStream() throws IOException {
throw new UnknownServiceException("protocol doesn't support input");
}
Overriding getDoInput
You should not override getDoInput to only return false. Instead you should use setDoInput(false). However, you don't want to set doInput to false. You always want to read something, for instance the response code.
I'm trying to test legacy Java application, without an ability to re-factor its code at the moment. All I need to do is to understand what SQL requests it is sending through JDBC and when. All these requests I'd like to protocol to a plain text file for future review.
I need to develop a custom JDBC driver which will fool the application, and behave like a normal driver (will accept request, return some values, fail on certain conditions, etc). A mock structure, as usual...
The question is - do you know any existing frameworks for this task? Ideally I would like to have an ability to configure my driver's behavior through, say, XML file.
p6spy wraps an existing jdbc connection, and allows you to e.g. see what goes back and forth.
See http://www.mkyong.com/hibernate/how-to-display-hibernate-sql-parameter-values-solution/ for instructions.
If you want to do unit tests, not an integration tests, than
you can use a very basic and simple approach, using Mockito only, like this:
public class JDBCLowLevelTest {
private TestedClass tested;
private Connection connection;
private static Driver driver;
#BeforeClass
public static void setUpClass() throws Exception {
// (Optional) Print DriverManager logs to system out
DriverManager.setLogWriter(new PrintWriter((System.out)));
// (Optional) Sometimes you need to get rid of a driver (e.g JDBC-ODBC Bridge)
Driver configuredDriver = DriverManager.getDriver("jdbc:odbc:url");
System.out.println("De-registering the configured driver: " + configuredDriver);
DriverManager.deregisterDriver(configuredDriver);
// Register the mocked driver
driver = mock(Driver.class);
System.out.println("Registering the mock driver: " + driver);
DriverManager.registerDriver(driver);
}
#AfterClass
public static void tearDown() throws Exception {
// Let's cleanup the global state
System.out.println("De-registering the mock driver: " + driver);
DriverManager.deregisterDriver(driver);
}
#Before
public void setUp() throws Exception {
// given
tested = new TestedClass();
connection = mock(Connection.class);
given(driver.acceptsURL(anyString())).willReturn(true);
given(driver.connect(anyString(), Matchers.<Properties>any()))
.willReturn(connection);
given(connection.prepareCall(anyString())).willReturn(statement);
}
}
Than you can test various scenarios, like in any other Mockito test e.g.
#Test
public void shouldHandleDoubleException() throws Exception {
// given
SomeData someData = new SomeData();
given(connection.prepareCall(anyString()))
.willThrow(new SQLException("Prepare call"));
willThrow(new SQLException("Close exception")).given(connection).close();
// when
SomeResponse response = testClass.someMethod(someData);
// then
assertThat(response, is(SOME_ERROR));
}