Develop Reference

Getting a specific version of an image with Jib (Maven, Docker, testcontainers)

I'm trying to understand a comment that a colleague made. We're using testcontainers to create a fixture:
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.utility.DockerImageName;
public class SalesforceFixture extends GenericContainer<SalesforceFixture> {
private static final String APPLICATION_NAME = "salesforce-emulator";
public SalesforceFixture() {
// super(ImageResolver.resolve(APPLICATION_NAME));
super(DockerImageName.parse("gcr.io/ad-selfserve/salesforce-emulator:latest"));
...
}
...
The commented code is what it used to be. The next line is my colleague's suggestion. And on that line he commented:
This is the part I don't know. The [ImageResolver] gets the specific version of the emulator, rather than the latest. You need a docker-info file for that though, which jib doesn't automatically generate (but I think it can).
This is what I know or have figured so far:
SalesforceFixture is a class that will be used by other projects to write tests. It spins up a container in Docker, running a service that emulates the real service's API. It's like a local version of the service that behaves enough like the real thing that if one writes code and tests using the fixture, it should work the same in production. (This is where my knowledge ends.)
I looked into ImageResolver—it seems to be a class we wrote that searches a filesystem for something:
public static String resolve(String applicationName, File... roots) {
Stream<File> searchPaths = Arrays.stream(roots).flatMap((value) -> {
return Stream.of(new File(value, "../" + applicationName), new File(value, applicationName));
});
Optional<File> buildFile = searchPaths.flatMap((searchFile) -> {
if (searchFile.exists()) {
File imageFile = new File(searchFile + File.separator + "/target/docker/image-name");
if (imageFile.exists()) {
return Stream.of(imageFile);
}
}
return Stream.empty();
}).findAny();
InputStream build = (InputStream)buildFile.map(ImageResolver::fileStream).orElseGet(() -> {
return searchClasspath(applicationName);
});
if (build != null) {
try {
return IOUtils.toString(build, Charset.defaultCharset()).trim();
} catch (IOException var6) {
throw new RuntimeException("An exception has occurred while reading build file", var6);
}
} else {
throw new RuntimeException("Could not resolve target image for application: " + applicationName);
}
}
But I'm confused. What filesystem? Like, what is the present working directory? My local computer, wherever I ran the Java program from? Or is this from within some container? (I don't think so.) Or maybe the directory structure inside a .jar file? Or somewhere in gcr.io?
What does he mean about a "specific version number" vs. "latest"? I mean, when I build this project, whatever it built is all I have. Isn't that equivalent to "latest"? In what case would an older version of an image be present? (That's what made me think of gcr.io.)
Or, does he mean, that in the project using this project's image, one will not be able to specify a version via Maven/pom.xml—it will always spin up the latest.
Sorry this is long, just trying to "show my work." Any hints welcome. I'll keep looking.

I can't comment on specifics of your own internal implementations, but ImageResolver seems to work on your local filesystem, e.g. it looks into your target/ directory and also touches the classpath. I can imagine this code was just written for resolving an actual image name (not an image), since it also returns a String.
Regarding latest, using a latest tag for a Docker image is generally considered an anti-pattern, so likely your colleague is commenting about this. Here is a random article from the web explaining some of the issues with latest tag:
https://vsupalov.com/docker-latest-tag/
Besides, I don't understand why you ask these questions which are very specific to your project here on SO rather than asking your colleague.

Multiple java version usage in pipeline jobs

We have upgraded the ant jar files to 51 which needs to detect the java version11 but our old applications are still using the old jar file50 so we need to detect the java versions according to the pull request id. here we give PR id as parameter.
stage("abc"){
steps{
script{
if (PR_ID == '100')
tools{
jdk 'java11.0.5'
}
else
tools{
jdk 'java8'
fi
}
i am trying this way but getting error like java.lang.NoSuchMethodError: No such DSL method 'tools' found among steps [ArtifactoryGradleBuild, MavenDescriptorStep, addInteractivePromotion, archive, artifactoryDistributeBuild, artifactoryDownload, artifactoryEditProps.
Kindly help me on how can we use if else condition or any other condition to detect the java version according to the PR id.

A tools block is only allowed inside a pipeline block or stage block, not inside a script block.
What I would suggest is creating two different stages and using when to select the one to execute.
stage("abc-java11") {
when {
expression {
return PR_ID == '100'
}
}
tools {
jdk 'java11.0.5'
}
steps {
// Your steps
}
}
stage("abc-java8") {
when {
expression {
return PR_ID != '100'
}
}
tools {
jdk 'java8'
}
steps {
// Your steps
}
}

Maven project crashing with no stacktrace or log messages with Selenium dependency

I'm sure I'm doing something wrong here with dependency management, but can't quite figure it out.
My Maven project "A" is dependent on project "B" (a gradle managed project). Both "A" and "B" also have a dependency on Selenium for UI automation. "B" is a collection of some selenium processing libraries. The selenium webDriver object is initialized in project B and returned to A, like this:
Some class in project A:
public WebDriver myDriver;
myDriver = B.initializeWebDriver(myDriver, various selenium related parameters...);
Then "A" can utilize the initialized myDriver object in a testNG test.
The place where it crashes in "B" with no stack trace is:
initializeWebDriver method in project B:
public WebDriver initializeWebDriver(WebDriver webDriver, ...) {
webDriver = new RemoteWebDriver(new URL(url), cap); // <-- crashes here
return webDriver
}
I would have thought that "A" and "B" would simply need to have a dependency on "selenium-java". That's all "B" has as a selenium dependency, and can create the driver and use it in various unit tests wholly contained in project B. If project A has a dependency on selenium-java, and I run a test from A, then Java immediately ends the testNG test when it reaches the line of code in "B" that instantiates the remoteWebDriver. I have a try-catch in both A and B, and nothing ever happens. No stack trace. No log message. Just immediately dies inside the test method in A, not jumping to a 'catch', but jumping to the 'finally' and ending.
What makes me pretty sure it's a dependency management problem, is that if I change the dependencies on "A" to selenium-api and selenium-chrome-driver and selenium-remote-driver, it works fine. But we shouldn't need to do that -- selenium-java should be fine, and selenium-java contains those other objects (selenium-api, etc) anyway! I'm guessing it's some weird CLASSDEF problem and I'm not doing dependency management correctly. I've also never seen java just blatantly give up with no stack trace or errors like this, unless it's some really weird classdef thing.
Any ideas?
**
[Edit: additional code provided as requested]
From B:
public WebDriver getWebDriver(WebDriver webDriver,...) {
try {
System.out.println("inside B try");
webDriver = new RemoteWebDriver(new URL(hubUrl), cap);
System.out.println("B instantiated webdriver");
} catch (Exception e) {
System.out.println("Caught in B");
e.printStackTrace();
}
return webDriver;
}
From class in A:
try {
System.out.println("inside A try");
webDriver = B.getWebDriver(webDriver);
System.out.println("webdriver A successful");
} catch (Exception e) {
System.out.println("caught in A");
e.printStackTrace();
} finally {
closeBrowser(webDriver);
}
Output from this is:
inside A try
inside B try
then output from the closeBrowser routine from the 'finally' in A.
No evidence of any exceptions or anything caught. As soon as it hits the line in B to create the remoteWebDriver, it just dies somehow with no output, and goes to the 'finally' in A.
Edit: Fixed the problem, I had the gradle project 'B' ensure that it was including the selenium dependency in the .jar it was exporting for project B:
from sourceSets.main.output
from sourceSets.main.allJava
... and in Maven project A, I defined the dependency on selenium-java (removed all the other various selenium jars like selenium-api, etc as this works now) as scope:provided.. So it's ensuring that the selenium-java dependency is in fact truly acquired from project 'B'.
But I don't understand why I couldn't have A and B both be dependent on selenium-java. They were specifying the exact same version, and I looked at the dependency resolution and they both were truly using the same version, as opposed to using a different version due to some conflict of some kind.
[edit]
The order of the dependency listed in project A's POM file is what made it work, not the gradle change or the 'provided' change. Simply listing selenium-java first in the pom before project B.

I don't know why you are not getting a stacktrace. Perhaps you are terminating the program in your closeBrowser method? That would short-circuit the error and would explain why you are not getting uncaught exception stacktrace.
Anyhow, for debug purpose, try the following:
You are catching Exception, so Errors are ignored.
Change your try-catch to catch Throwable. This will ensure you catch any possible errors.
try{
...
} catch (Throwable e) {
System.out.println("caught in A");
e.printStackTrace();
}

JaCoCo branch coverage try with resources

I have a method that I am trying to unit-test. I cannot post the actual code, but it looks like this:
public int getTotal() throws MyException {
int total = 0;
try (ExternalResource externalResource = ExternalService.getResource()) {
try (OtherExternal otherResource = externalResource.getOtherResource()) {
if (someCondition) {
total = otherResource.getTotal();
}
}
}
}
JaCoCo is telling me that I am missing 4/8 branches on each of the try-with-resource blocks. I am testing that someCondition is true and someCondition is false, and JaCoCo shows that block completely covered.
I read this question, and I understand from the accepted answer that the issue is in how the byte code is generated.
I would like to be able to better understand how to identify the various branches that are generated, and then I can make a better judgement on wether to test them or not (are they unreachable, etc).

Per the change history in version 0.8.2:
Branches and instructions generated by javac 11 for try-with-resources statement are filtered out
I've tested this out locally using openjdk java8, and my try-with-resources now reports 100% branch coverage (even though the IOException is never thrown in my tests).
While it is good to test this behavior out, there are times when you can't easily reproduce such exceptions. For instance, in a method that just returns an open port:
public int getOpenPort() {
try (ServerSocket boundSocket = new ServerSocket(0)) {
return boundSocket.getLocalPort();
}
}
I know of no simple way to force this code to throw IOException without adding a bunch of confusing and unnecessarily complicated code, just to pass a branch coverage check. Luckily, the new (v0.8.2) jacoco library gives this method 100% coverage with a single test just calling Assert.assertNotEquals(0, portChecker.getOpenPort());.

You have to test every Exception and every condition. But JaCoCo sometimes failed to identify correctly what is covered or not.

FileNotFound (Access is denied) Exception on java.io

Why do I get this error when I run this program? This occurs after random iterations. Usually after the 8000th iteration.
public static void main(String[] args)
{
FileWriter writer = null;
try
{
for(int i = 0; i < 10000; i++)
{
File file = new File("C:\\Users\\varun.achar\\Desktop\\TODO.txt");
if(file.exists())
{
System.out.println("File exists");
}
writer = new FileWriter(file, true);
writer.write(i);
System.out.println(i);
writer.close();
if(!file.delete())
{
System.out.println("unable to delete");
}
//Thread.sleep(10);
//writer = null;
//System.gc();
}
}
catch(IOException e)
{
e.printStackTrace();
}
finally
{
if(writer != null)
{
try
{
writer.close();
}
catch(IOException e)
{
e.printStackTrace();
}
}
}
}
After the exception occurs, the file isn't present. That means the it is deleting, but FIleWriter tries to acquire the lock before that, even though it isn't a multi threaded program. Is it because the Windows isn't deleting the file fast enough, and hence the FileWriter doesn't get a lock? If so, then file.delete() method returns before windows actually deletes it?
How do i resolve it, since i'm getting a similar issue during load testing my application.
EDIT 1: Stacktrace:
java.io.FileNotFoundException: C:\Users\varun.achar\Desktop\TODO.txt (Access is denied)
at java.io.FileOutputStream.openAppend(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:192)
at java.io.FileOutputStream.<init>(FileOutputStream.java:116)
at java.io.FileWriter.<init>(FileWriter.java:61)
EDIT 2 : Added file.exists() and file.delete conditions in the program. and the new stacktrace:
7452
java.io.FileNotFoundException: C:\Users\varun.achar\Desktop\TODO.txt (Access is denied)
at java.io.FileOutputStream.openAppend(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:192)
at java.io.FileWriter.<init>(FileWriter.java:90)
at com.TestClass.main(TestClass.java:25)
EDIT 3 Thread dump
TestClass [Java Application]
com.TestClass at localhost:57843
Thread [main] (Suspended (exception FileNotFoundException))
FileOutputStream.<init>(File, boolean) line: 192
FileWriter.<init>(File, boolean) line: 90
TestClass.main(String[]) line: 24
C:\Users\varun.achar\Documents\Softwares\Java JDK\JDK 6.26\jdk\jre\bin\javaw.exe (09-Nov-2011 11:57:34 PM)
EDIT 4 : Program runs successfully on different machine with same OS. Now how do i ensure that the app with run successfully in the machine it is deployed in?

On any OS you can have only a certain number of open files/threads at a stretch. You seem to be hitting your OS limit. Try setting file to null inside the loop.

If I understand your stack trace correctly, the exception is coming when trying to create a new FileWriter. It's impossible to know the reason without investigating a bit further.
Return values may tell something. Especially, check what File.delete() returns.
Before trying to create new FileWriter, check what File.exists() returns.
If the previous delete() returns true and the exists() right after it also returns true, in a single-threaded program, then it's indeed something weird.
Edit: so it seems that deletion was successful and the file didn't exist after that. That how it's supposed to work, of course, so it's weird why FileWriter throws the exception. One more thought, try checking File.getParentFile().canWrite(). That is, do your permissions to write to the directory somehow disappear.
Edit 2:
Don't get the error on a different machine with the same OS. Now how do i make sure that this error won't come in the app where it'll be deployed?
So far you have one machine that works incorrectly and one that works correctly. Maybe you could try it on even more machines. It's possible that the first machine is somehow broken and that causes errors. It's amazing how often digital computers and their programs (I mean the OS and Java, not necessarily your program) can be just a "little bit broken" so that they work almost perfectly almost all of the time, but fail randomly with some specific hardware & use case - usually under heavy load - similar to how incorrect multi-threaded programs can behave. It doesn't have to be your fault to be your problem :-)
Frankly, the only way to make sure that errors won't come up in machine X is to run the program on machine X. Unusual stuff such as creating and deleting the same file 8000 times in rapid succession is prone to errors, even though it "should" work. Computers, operating systems and APIs are not perfect. The more unusual stuff you do, the more often the imperfections will realize themselves, because unusual usage is generally less thoroughly tested than everyday operations.

I have had the same issue, a java program (single threaded) that opens, deleted then re-opens the same file continuously.
On some windows systems we get the same issue as reported here, on Linux, Solaris, and various other windows systems it works fine.
Traceing the program with SysInternals Process Monitor (now MS) its clear the delete is done first, at the OS level, and clear the subsequent open fails with PENDING DELETE status.
So there seems to be some slight delay at the OS/NTFS/Disk level before the file is actually deleted, and that seems to be the cause of the random failure in our case.
As a workaround, I changed the .delete() call to instead just write over the top of it new FileWriter(file) and that seems to be working.
The problem did not occur on all systems, one specific model that would always fail, all be it after not fixed number of loops, was a Windows 7 / Dell Lattitude E6420 with WD Smartdrive, whereas my Windows 7 / Dell precision M4600 (with solid state drive) or T3400 with Linux I have never had the issue.
Cheers - Mark

It may be a long shot, but, can you try to work with a file that is NOT directly sitting on the Desktop. Instead of:
"C:\\Users\\varun.achar\\Desktop\\TODO.txt"
Try:
"C:\\Users\\varun.achar\\SomeOtherDirectory\\TODO.txt"
OS may be killing you here with all the Desktop hooks...
EDIT based on the comments:
Are there any scheduled jobs running on the "bad" machine?
Instead of debugging the environment, do you have a sys admin to do that?
Does this work on a clean Windows install? [95% chance it will]
Since the root cause seems to be environment, instead of solving a Windows configuration problem, would you be able to move forward with other tasks, and leave it to someone who keeps the list of discrepancies between the systems?

Can you conditionally try to write to the file ?
Using file.exists and then writing to it, so you can potentially avoid any other issues. Hard to say from this exception.
http://download.oracle.com/javase/6/docs/api/java/io/File.html#exists()
Could you also post a thread dump at that point, just to debug it further.
Please flush the writer, before writing again.

These are the scenerios you should handle before deleting a file http://www.java2s.com/Code/Java/File-Input-Output/DeletefileusingJavaIOAPI.htm
at least check for return value in your program.

Thanks folks for help me out but this is how it got resolved finally.
public static void main(String[] args)
{
FileWriter writer = null;
try
{
for(int i = 0; i < 10000; i++)
{
File file = new File("C:\\tenant-system-data\\abc.txt");
if(!file.getParentFile().canWrite())
{
System.out.println("parent file error");
}
if(file.exists())
{
System.out.println("File exists");
}
int count = 0;
while(count++ < 5)
{
try
{
file.createNewFile();
break;
}
catch(IOException e)
{
try
{
Thread.sleep(100);
}
catch(InterruptedException e1)
{
e1.printStackTrace();
}
}
}
writer = new FileWriter(file, true);
writer.write(i);
System.out.println(i);
writer.close();
if(!file.delete())
{
System.out.println("unable to delete");
}
//Thread.sleep(10);
//writer = null;
//System.gc();
}
}
catch(IOException e)
{
e.printStackTrace();
}
finally
{
if(writer != null)
{
try
{
writer.close();
}
catch(IOException e)
{
e.printStackTrace();
}
}
}
}

I just had the same problem (FileWriter & Access Denied).
My guess for the reason: Windows 7 had put a lock on the file because a preview of the file content was shown in an Explorer window (the file was selected in the window).
Solution: I de-selected the file in the Explorer window. And the IOException was gone.

You have delete permission in the directory but not create permission.