I'm having trouble writing unit tests for a method that overwrites a file to a S3 bucket. The method grabs the original metadata of the file, and then overwrites the file with a new modified version and the same original metadata.
What I want the test to do is verify the inner methods like getObjectMetadata and putObject are called correctly with the right parameters
Here is the method:
public void upload(File file, String account, String bucketName) {
String key = "fakekey";
ObjectMetadata objMData = client.getObjectMetadata(bucketName, key).clone();
try {
// cloning metadata so that overwritten file has same metadata as original file
client.putObject(new PutObjectRequest(bucketName, key, file).withMetadata(objMData));
} catch(AmazonClientException e) {
e.printStackTrace();
}
Here is my test method:
#Mock
private AmazonS3 client = new AmazonS3Client();
public void testUpload() {
S3Uploader uploader = new S3Uploader(client);
File testFile = new File("file.txt");
String filename = "file.txt";
String bucketname = "buckettest";
String account = "account";
String key = account+filename;
ObjectMetadata objMetadata = Mockito.mock(ObjectMetadata.class);
when(client.getObjectMetadata(bucketname, key).clone()).thenReturn(objectMetadata);
// can I make this line do nothing? doNothing()??
doNothing.when(client.putObject(Matchers.eq(new PutObjectRequest(bucketName, key, file).withMetadata(objMData))));
uploader.upload(aFile, anAccount, bucketName);
// how do I verify that methods were called correctly??
// what can I assert here?
}
I'm getting a NullPointerException at the line in my test
when(client.getObjectMetadata(bucketname, key).clone()).thenReturn(objectMetadata);
I'm not even able to reach the method call. Honestly, what I'm pretty much asking is, how do I verify that this upload() method is correct?
The method you showed in your question uses a client instance to talk to S3. The client instance in the class to which this method belongs is either injected (at construction time, for example) or created (via a factory, perhaps). Assuming it is injected when the containing class is created then your test case might look like this:
#Test
public void testSomething() {
AmazonS3 client = Mockito.mock(AmazonS3.class);
S3Uploader uploader = new S3Uploader(client);
String bucketName = "aBucketName";
// ensures that the getObjectMetadata call fails thereby throwing the exception which your method catches
Mockito.when(client.getObjectMetadata(Matchers.eq(bucketName), Matchers.eq("fakekey")).thenThrow(new AmazonServiceException());
uploader.uploadToS3(aFile, anAccount, bucketName);
// at this stage you would typically assert that the response
// from the upload invocation is valid but as things stand
// upload() swallows the exception so there's nothing to assert against
}
#Test
public void testSomethingElse() {
AmazonS3 client = Mockito.mock(AmazonS3.class);
S3Uploader uploader = new S3Uploader(client);
String bucketName = "aBucketName";
String key = "fakekey";
File aFile = ...;
ObjectMetadata objMData = ...;
// ensures that the getObjectMetadata call succeeds thereby allowing the call to continue to the subsequent putObject invocation
Mockito.when(client.getObjectMetadata(eq(bucketName), eq(key)).thenReturn(objMData);
// ensures that the putObject call fails thereby throwing the exception which your method catches
Mockito.when(client.putObject(Matchers.eq(new PutObjectRequest(bucketName, key, file).withMetadata(objMData)).thenThrow(new AmazonServiceException());
uploader.uploadToS3(aFile, anAccount, bucketName);
// at this stage you would typically assert that the response
// from the upload invocation is valid but as things stand
// upload() swallows the exception so there's nothing to assert against
}
The above code uses Mockito to mock the AmazonS3 client, this allows you to tweak the behaviour of your client instance such that your test invocations go down the 'throw exception' paths.
On a side note the catch clauses look a little odd since AmazonS3.putObject and AmazonS3.getObjectMetadata are both declared to throw AmazonServiceException and AmazonServiceException extends AmazonClientException.
I would suggest you to use this project https://github.com/findify/s3mock.
Create a mock of S3 bucket, and then you can test what happens when the bucket you look for exist or not.
Related
In Short: Using AmazonS3Client to connect to a local instance of MinIO results in a UnknownHostException thrown because the url is resolved to http://{bucket_name}.localhost:port.
Detailed description of the problem:
I'm creating an integration test for a Java service that uses AmazonS3Client lib to retrieve content from S3. I'm using MinIO inside a test container to perform the role of Amazon S3, as follows:
#Container
static final GenericContainer<?> minioContainer = new GenericContainer<>("minio/minio:latest")
.withCommand("server /data")
.withEnv(
Map.of(
"MINIO_ACCESS_KEY", AWS_ACCESS_KEY.getValue(),
"MINIO_SECRET_KEY", AWS_SECRET_KEY.getValue()
)
)
.withExposedPorts(MINIO_PORT)
.waitingFor(new HttpWaitStrategy()
.forPath("/minio/health/ready")
.forPort(MINIO_PORT)
.withStartupTimeout(Duration.ofSeconds(10)));
and then I export its url dynamically (because test containers are deployed at a random port) using something like this:
String.format("http://%s:%s", minioContainer.getHost(), minioContainer.getFirstMappedPort())
which in turn results in a url like this:
http://localhost:54123
The problem I encountered during the runtime of my test lies within the actual implementation of AmazonS3Client.getObject(String,String) - when creating the request it performs the following validation (class S3RequestEndpointResolver, method resolveRequestEndpoint):
...
if (shouldUseVirtualAddressing(endpoint)) {
request.setEndpoint(convertToVirtualHostEndpoint(endpoint, bucketName));
request.setResourcePath(SdkHttpUtils.urlEncode(getHostStyleResourcePath(), true));
} else {
request.setEndpoint(endpoint);
request.setResourcePath(SdkHttpUtils.urlEncode(getPathStyleResourcePath(), true));
}
}
private boolean shouldUseVirtualAddressing(final URI endpoint) {
return !isPathStyleAccess && BucketNameUtils.isDNSBucketName(bucketName)
&& !isValidIpV4Address(endpoint.getHost());
}
This in turn returns true for the url http://localhost:54123 and as a result this method
private static URI convertToVirtualHostEndpoint(URI endpoint, String bucketName) {
try {
return new URI(String.format("%s://%s.%s", endpoint.getScheme(), bucketName, endpoint.getAuthority()));
} catch (URISyntaxException e) {
throw new IllegalArgumentException("Invalid bucket name: " + bucketName, e);
}
}
concatenates the name of the bucket to the host resulting in: http://mybucket.localhost:54123 and this ultimately results in a UnknownHostException to be thrown. I can work around this by setting the host to 0.0.0.0 instead of localhost, but this is hardly a solution.
Therefore I was wondering if i) this a bug/limitation in AmazonS3Client?; ii) I'm the one who is missing something, e.g. poor configuration ?
Thank you for your time
I was able to find a solution. Looking at the method used by the resolver:
private boolean shouldUseVirtualAddressing(final URI endpoint) {
return !isPathStyleAccess && BucketNameUtils.isDNSBucketName(bucketName)
&& !isValidIpV4Address(endpoint.getHost());
}
which was returning true and leading the flow to the wrong concatenation I found that we can set the first variable isPathStyleAccess when building the client. In my case, I created a bean in my test configuration to override the main one:
#Bean
#Primary
public AmazonS3 amazonS3() {
return AmazonS3Client.builder()
.withPathStyleAccessEnabled(true) //HERE
.withCredentials(new AWSStaticCredentialsProvider(
new BasicAWSCredentials(AWS_ACCESS_KEY.getValue(), AWS_SECRET_KEY.getValue())
))
.withEndpointConfiguration(
new AwsClientBuilder.EndpointConfiguration(s3Endpoint, region)
)
.build();
}
For the SDK V2, the solution was pretty similar:
S3AsyncClient s3 = S3AsyncClient.builder()
.forcePathStyle(true) // adding this one
.endpointOverride(new URI(s3Endpoint))
.credentialsProvider(() -> AwsBasicCredentials.create(s3Properties.getAccessKey(), s3Properties.getSecretKey()))
.build()
/*
this method will create the required manifest file in compatible format such that
quicksight can import data from specified s3 bucket
*/
private JSONObject CreateManifestFileJSONObject(JSONObject ManifestFile){
JSONArray URIPrefixArray= new JSONArray();
URIPrefixArray.put(PrefixLocation);
JSONObject URIPrefixJSONObject= new JSONObject();
URIPrefixJSONObject.put("URIPrefixes",URIPrefixArray);
JSONArray FileLocationsArray= new JSONArray();
FileLocationsArray.put(URIPrefixJSONObject);
JSONObject globalUploadSettings= new JSONObject();
globalUploadSettings.put("format","JSON");
ManifestFile.put("globalUploadSettings",globalUploadSettings);
ManifestFile.put("fileLocations",FileLocationsArray);
return(ManifestFile);
}
/*
this method will upload the ManifestFile to same S3 Bucket in which data files is stored
*/
private void UploadManifestFileJSONObjectToS3(JSONObject ManifestFile){
try {
AmazonS3 S3Client = new Utility().SetUpS3Client();
byte[] fileContentBytes = (ManifestFile.toString()).getBytes();
InputStream fileInputStream = new ByteArrayInputStream(fileContentBytes);
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setContentLength(fileContentBytes.length);
S3Client.putObject(new PutObjectRequest(Bucket_Name, ManifestFileName, fileInputStream, objectMetadata).withCannedAcl(CannedAccessControlList.PublicRead));
}
catch(Exception exception){
exception.printStackTrace();
}
}
public void handler() {
System.out.println("inside the manifest file");
try {
JSONObject ManifestFile = new JSONObject();
ManifestFile = CreateManifestFileJSONObject(ManifestFile);
UploadManifestFileJSONObjectToS3(ManifestFile);
}
catch(Exception exception){
exception.printStackTrace();
}
I want to test handler method but handlor method is calling private methods so I do not know how to write the test class for this code.
i want to write unit test for this class please help
this is the test class I am able to create up to this point but it will not surely mock s3 behaviour
#Test
public void handler() {
ManifestFileHandler manifestFileHandler=new ManifestFileHandler();
manifestFileHandler.handler();
}
You can test private methods with the help of PowerMock provides utilities which uses reflection to do certain things.
check the below example,
https://examples.javacodegeeks.com/core-java/mockito/mockito-mock-private-method-example-with-powermock/
There are two schools of thought on unit testing private functions. The first is that you make them public (or protected or package accessible) and test them as you would a public function. The second is that if they are private they are part of the encapsulated implementation detail and you only need to test them through the public functions.
My personal view is that complicated private functions are often a sign that you are breaking the single responsibility principle and it's likely you should have logic in private functions that should be split into a separate class that can then be tested through its public methods.
With respect to the code you've posted you have a larger problem than how to test the private functions: your class depends on other classes that you don't have control over. You have no way of mocking the behaviour of those classes to test various scenarios or to verify that they have been called correctly. I suspect it is this problem that is really behind your question.
As an example, I would suggest you inject a S3Client into your class rather than create it internally through new Utility().SetUpS3Client(). That way you can mock its behaviour and verify it is called correctly by your code. Attempting to do that with the real version of this class will be challenging.
So using this model, your code might look something like:
public class ManifestFileHandler {
private final S3Client client;
public ManifestFileHandler(S3Client client) {
this.client = client;
}
private void upload(JSONObject manifestFile) {
...
client.putObject(...);
}
public void handleManifest() {
...
upload(manifestFile);
...
}
}
And your test code (using mockito):
#Test
void testManifestUpload() {
S3Client client = mock(S3Client.class);
ManifestFileHandler handler = new ManifestFileHandler(client);
handler.handleManifest();
verify(client).putObject(expectedObject);
}
If you need to capture the argument passed to putObject and assert various aspects of it then that is possible with most mocking tools (including mockito) but is beyond the scope of your question.
How do you mock file reading/writing via JUnit?
Here is my scenario
MyHandler.java
public abstract class MyHandler {
private String path = //..path/to/file/here
public synchronized void writeToFile(String infoText) {
// Some processing
// Writing to File Here
File file = FileUtils.getFile(filepath);
file.createNewFile();
// file can't be written, throw FileWriteException
if (file.canWrite()) {
FileUtils.writeByteArrayToFile(file, infoText.getBytes(Charsets.UTF_8));
} else {
throw new FileWriteException();
}
}
public String readFromFile() {
// Reading from File here
String infoText = "";
File file = new File(path);
// file can't be read, throw FileReadException
if (file.canRead()) {
infoText = FileUtils.readFileToString(file, Charsets.UTF_8);
} else {
throw FileReadException();
}
return infoText
}
}
MyHandlerTest.java
#RunWith(PowerMockRunner.class)
#PrepareForTest({
MyHandler.class
})
public class MyHandlerTest {
private static MyHandler handler = null;
// Some Initialization for JUnit (i.e #Before, #BeforeClass, #After, etc)
#Test(expected = FileWriteException.class)
public void writeFileTest() throws Exception {
handler.writeToFile("Test Write!");
}
#Test(expected = FileReadException.class)
public void readFileTest() throws Exception {
handler.readFromFile();
}
}
Given above source, Scenario when file is not writable (write permission not allowed) is OK, However, when i try to do scenario wherein file is not readable (read permission not allowed). It always read the file, i have already tried to modify the file permission on the test code via below
File f = new File("..path/to/file/here");
f.setReadable(false);
However, I did some reading, setReadable() always returns false (failed) when run on Windows machine.
Is there a way to modify the file permission of the target file programmatically in relation to JUnit?
Note
Target source code to test cannot be modified, meaning
Myhandler.class is a legacy code which is not to be modified.
Instead of relying on the operating system file permissions, use PowerMock to mock FileUtils.getFile(...) and make it return an instance of File (e.g. anonymous sub class) that returns a specific value for canWrite()/canRead().
Mocking static methods with Mockito
Since Mockito cannot mock static methods, use a File factory instead (or refactor your FileUtils to be a factory), then you can mock it and return a mocked File instance as well, where you can also mock any File methods you want.
So instead of FileUtils.getFile(filepath) you will now have something like FileFactory.getInstance().getFile(filepath) for example, where you can mock getFile(String) method easily.
In jUnit there's a handy rule for scenarios like yours.
public class MyHandlerTest {
#Rule
// creates a temp folder that will be removed after each test
public org.junit.rules.TemporaryFolder folder = new org.junit.rules.TemporaryFolder();
private MyHandler handler;
#Before
public void setUp() throws Exception {
File file = folder.newFile("myFile.txt");
// do whatever you need with it - fill with test content and so on.
handler = new MyHandler(file.getAbsolutePath()); // use the real thing
}
// Test whatever behaviour you need with a real file and predefined dataset.
}
I have a unit test (simplified version below) that tests a Netty handler.
I create an EmbeddedChannel with an instance of the handler.
The caller writes a string to the channel
The handler receives the string, reverses and writes it back.
The caller reads the return values from the channel and verifies it is the reverse of the sent string.
This works perfectly. However, I need to verify the number of invocations on the channel, so I created a spy of the channel but mocked no methods, since I don't want to change the behavior of the class, just count invocations.
Now the test fails. 2 of the assertions succeed. They are a test to make sure the handler was called, and a test to verify the number of times a method of the channel was called. However, the final read response is always null when the spy is used.
I was under the impression that a solitary spy with no other mocking would not affect the behavior of the spied object, but obviously it does. The [nonPower] Mockito docs indicate the objects are copied which might cause this issue, but the PowerMockito docs are not as specific.
I am using Netty 4.1.6.Final and Powermock 1.5.6.
UPDATE: I managed to get the test working but it's a bit of wonky workaround. See the new method testSpiedEmbeddedChannel2. The workaround is that I create a non-spied channel (ecx), then a the spied channel (ec) using ecx. I issued the write on ec, and the read using ecx. This means if I try to verify methods used in the read, they will not be counted.
Here's the code with the successful and failing tests.
#RunWith(PowerMockRunner.class)
#PowerMockIgnore({"javax.management.*"})
#PrepareForTest(EmbeddedChannel.class)
public class TestEmbeddedChannel {
class EchoHandler extends ChannelDuplexHandler {
final AtomicInteger reads = new AtomicInteger(0);
#Override
public void channelRead(final ChannelHandlerContext ctx, final Object msg) throws Exception {
reads.incrementAndGet();
final String value = (String)msg;
final String response = new StringBuilder(value).reverse().toString();
ctx.channel().writeAndFlush(response);
}
}
#Test
public void testEmbeddedChannel() { // PASSES
final EchoHandler handler = new EchoHandler();
final EmbeddedChannel ec = new EmbeddedChannel(handler);
ec.writeInbound("Hello World");
final String response = ec.readOutbound();
Assert.assertEquals(1, handler.reads.get());
Assert.assertEquals("dlroW olleH", response);
}
#Test
public void testSpiedEmbeddedChannel() { // FAILS
final EchoHandler handler = new EchoHandler();
final EmbeddedChannel ec = spy(new EmbeddedChannel(handler));
ec.writeInbound("Hello World");
final String response = ec.readOutbound();
verify(ec, times(2)).isOpen(); // OK
Assert.assertEquals(1, handler.reads.get()); // OK
Assert.assertEquals("dlroW olleH", response); // FAILS
}
#Test
public void testSpiedEmbeddedChannel2() { // PASSES
final EchoHandler handler = new EchoHandler();
final EmbeddedChannel ecx = new EmbeddedChannel(handler);
final EmbeddedChannel ec = spy(ecx);
ec.writeInbound("Hello World");
final String response = ecx.readOutbound(); // Read using non-spied channel
verify(ec, times(2)).isOpen();
Assert.assertEquals(1, handler.reads.get());
Assert.assertEquals("dlroW olleH", response);
}
}
Thanks for any guidance here.
This is my utility class to mock the service
public class MockService {
public static void bootUpMockServices() throws IOException {
String orderServiceSpecification = readFile("mappings/orderServicesSpecifications.json", Charset.defaultCharset());
String singleOrder = readFile("mappings/singleOrder.json", Charset.defaultCharset());
WireMock.stubFor(WireMock.get(WireMock.urlEqualTo("/orders"))
.willReturn(WireMock.aResponse()
.withStatus(200)
.withBody(orderServiceSpecification)));
WireMock.stubFor(WireMock.get(WireMock.urlEqualTo("/orders/1"))
.willReturn(WireMock.aResponse()
.withStatus(200)
.withBody(singleOrder)));
}
public static String readFile(String path, Charset encoding)
throws IOException {
byte[] encoded = Files.readAllBytes(Paths.get(path));
return new String(encoded, encoding);
}
}
As you can see I'm mocking a GET call /orders (with all the orders) and responding with the body with all the orders kept in a json file.
I'm also calling a single order by GET call /orders/1. I'm responding it with an JSON object in a file. But I want it to be dynamic. Like when I hit it with orders/30 then, I should be dynamically fetch order with id=30 and render it.
Currently, if you want dynamic behaviour of the kind you described you'll need to write a ResponseDefinitionTransformer and register it with the WireMockServer or WireMockRule on construction.
This is documented here: http://wiremock.org/docs/extending-wiremock/#transforming-responses
Example of a transformer implementation here:
https://github.com/tomakehurst/wiremock/blob/master/src/test/java/com/github/tomakehurst/wiremock/ResponseDefinitionTransformerAcceptanceTest.java#L208-L222
What you're trying to do could be done pretty straightforwardly with a stub mapping matching on a URL regex something like /orders/(\d+) and a transformer that parses out the number part then modifies the bodyFileName on the ResponseDefinition.