Develop Reference

Write a Java program that downloads the first 100 comics of the webcomic XKCD. Be sure to use https:// for all URLS

This is what I have so far, and I am having trouble downloading 1-100 comics starting at https://xkcd.com/1/ and I know I am supposed to be going to the source code for the website. However, I cant seem to figure out how to get all the first 100 comics into my designated file I set it to save to. For example, I want https://xkcd.com/1/(view-source:https://xkcd.com/1/), https://xkcd.com/2/(view-source:https://xkcd.com/2/), and all the way up to comic 100. I know the img src is at line 50, but once again I don't know how to approach it.
public static void main(String[] args) {
URL imgURL = null;
for (int web = 1; web <= 100; web++) {
try {
imgURL = new URL("https://imgs.xkcd.com/comics/barrel_cropped_(1).jpg");
InputStream stream = imgURL.openStream();
Files.copy(stream, Paths.get("file/WebComics" + web + ".png"));
System.out.println("Done!");
} catch (Exception e) {
e.printStackTrace();
System.out.println("Error!");
}
}
}
}

Add jsoup library jar to your project, and then try this:
static void do_page(int id) throws IOException {
Document doc = Jsoup.connect("https://xkcd.com/" + id).get();
Elements imgs = doc.select("#comic img");
for (Element e: imgs) {
System.out.println(e.attr("src"));
}
}
Then call the do_page function in a loop:
for (int i = 1; i <= 100; i++) {
do_page(i);
}
Now, instead of printing it, you can use JSoup again to probably download the images like you see fit.

Selenium Safari "Unknown command"

I'm working on Safari Browser and i got a problem.
"Unknown command: {"id":"5qhlf8uni92m","name":"mouseMoveTo","parameters":{"yoffset":25,"xoffset":10}}
(WARNING: The server did not provide any stacktrace information)"
How can i deal with this ?
NOTE: In my scenario, f book shows a notification pop-up and i can't select any element because when pop-up showed up, black screen appeared and i have to click anywhere to enable elements. That's why i used this code;
WebElement knownElement = null;
Actions builder = new Actions(driver);
builder.moveToElement(knownElement, 10, 25).click().build().perform();
In my opinion, it cause this problem. How can i change this code to fit in Safari ?

Please Refer this link : https://ynot408.wordpress.com/2011/09/22/drag-and-drop-using-selenium-webdriver/
OR :
public boolean onMouseOver(WebElement element){
boolean result = false;
try{
String mouseOverScript = "if(document.createEvent){
var evObj = document.createEvent('MouseEvents');evObj.initEvent('mouseover',
true, false); arguments[0].dispatchEvent(evObj);
} else if(document.createEventObject){
arguments[0].fireEvent('onmouseover');}";
JavascriptExecutor js = (JavascriptExecutor) driver;
js.executeScript(mouseOverScript, element);
result = true;
}catch (Exception e){
e.printStackTrace();
result = false;
}
return result;
}

How to wait for download to finish using Webdriver

Is there any way to wait for a download to finish in WebDriver?
Basically, I want to verify that downloaded file getting stored correctly inside hard drive and to verify that, need to wait till download finishes. Please help if anyone aware of such a scenario earlier.

Not a direct answer to your question but I hope it will help.
Pool the file on your hard drive (compute a MD5 of it). Once it is correct, you can go further with your test. Fail your tets if the file is not correct after a timeout.

Poll the configured download directory for absence of partial file.
Example code for Chrome below:
File destinationDir = new File("blah");
Map<String, Object> prefs = new HashMap<>();
prefs.put("download.default_directory", destinationDir.getAbsolutePath());
DesiredCapabilities desiredCapabilities = DesiredCapabilities.chrome();
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("prefs", prefs);
desiredCapabilities.setCapability(ChromeOptions.CAPABILITY, options);
WebDriver webDriver = new ChromeDriver(desiredCapabilities);
// Initiate download...
do {
Thread.sleep(5000L);
} while(!org.apache.commons.io.FileUtils.listFiles(destinationDir, new String[]{"crdownload"}, false).isEmpty());

As answered on Wait for Download to finish in selenium webdriver JAVA
private void waitForFileDownload(int totalTimeoutInMillis, String expectedFileName) throws IOException {
FluentWait<WebDriver> wait = new FluentWait(this.funcDriver.driver)
.withTimeout(totalTimeoutInMillis, TimeUnit.MILLISECONDS)
.pollingEvery(200, TimeUnit.MILLISECONDS);
File fileToCheck = getDownloadsDirectory()
.resolve(expectedFileName)
.toFile();
wait.until((WebDriver wd) -> fileToCheck.exists());
}
public synchronized Path getDownloadsDirectory(){
if(downloadsDirectory == null){
try {
downloadsDirectory = Files.createTempDirectory("selleniumdownloads_");
} catch (IOException ex) {
throw new RuntimeException("Failed to create temporary downloads directory");
}
}
return downloadsDirectory;
}
Then you can use a library like this to do the actual file handling to see the file is stored correctly (that could mean comparing file size, Md5 hashes or even checking the content of the file which Tika can actually do as well).
public void fileChecker(){
waitForFileDownload(20000,"filename_here");
File file = downloadsDirectory.resolve(expectedFileName).toFile();
AutoDetectParser parser = new AutoDetectParser();
parser.setParsers(new HashMap<MediaType, Parser>());
Metadata metadata = new Metadata();
metadata.add(TikaMetadataKeys.RESOURCE_NAME_KEY, file.getName());
try (InputStream stream = new FileInputStream(file)) {
parser.parse(stream, (ContentHandler) new DefaultHandler(), metadata, new ParseContext());
}
String actualHash = metadata.get(HttpHeaders.CONTENT_MD5);
assertTrue("There was a hash mismatch for file xyz:",actualHash.equals("expectedHash"));
}

I am your the bellow code in Python + Firefox:
browser.get('about:downloads') #Open the download page.
# WAit all icons change from "X" (cancel download).
WebDriverWait(browser, URL_LOAD_TIMEOUT * 40).until_not(
EC.presence_of_element_located((By.CLASS_NAME, 'downloadIconCancel')))

For small files, I currently either use an implied wait or wait for the JS callback that my file has downloaded before moving on. The code below was posted on SO by another individual, I can't find the post right away, so I won't take credit for it.
public static void WaitForPageToLoad(IWebDriver driver)
{
TimeSpan timeout = new TimeSpan(0, 0, 2400);
WebDriverWait wait = new WebDriverWait(driver, timeout);
IJavaScriptExecutor javascript = driver as IJavaScriptExecutor;
if (javascript == null)
throw new ArgumentException("driver", "Driver must support javascript execution");
wait.Until((d) =>
{
try
{
string readyState = javascript.ExecuteScript("if (document.readyState) return document.readyState;").ToString();
return readyState.ToLower() == "complete";
}
catch (InvalidOperationException e)
{
//Window is no longer available
return e.Message.ToLower().Contains("unable to get browser");
}
catch (WebDriverException e)
{
//Browser is no longer available
return e.Message.ToLower().Contains("unable to connect");
}
catch (Exception)
{
return false;
}
});
}
It should wait for your file to finish if it is small. Unfortunately, I haven't tested this on larger files ( > 5MB )

Compare image to actual screen

I'd like to make my Java program compare the actual screen with a picture (screenshot).
I don't know if it's possible, but I have seen it in Jitbit (a macro recorder) and I would like to implement it myself. (Maybe with that example you understand what I mean).
Thanks
----edit-----
In other words, is it possible to check if an image is showing in? To find and compare that pixels in the screen?

You may try aShot: documentation link
1) aShot can ignore areas you mark with special color.
2) aShot can provide image which display difference between images.
private void compareTowImages(BufferedImage expectedImage, BufferedImage actualImage) {
ImageDiffer imageDiffer = new ImageDiffer();
ImageDiff diff = imageDiffer
.withDiffMarkupPolicy(new PointsMarkupPolicy()
.withDiffColor(Color.YELLOW))
.withIgnoredColor(Color.MAGENTA)
.makeDiff(expectedImage, actualImage);
// areImagesDifferent will be true if images are different, false - images the same
boolean areImagesDifferent = diff.hasDiff();
if (areImagesDifferent) {
// code in case of failure
} else {
// Code in case of success
}
}
To save image with differences:
private void saveImage(BufferedImage image, String imageName) {
// Path where you are going to save image
String outputFilePath = String.format("target/%s.png", imageName);
File outputFile = new File(outputFilePath);
try {
ImageIO.write(image, "png", outputFile);
} catch (IOException e) {
// Some code in case of failure
}
}

You can do this in two steps:
Create a screenshot using awt.Robot
BufferedImage image = new Robot().createScreenCapture(new Rctangle(Toolkit.getDefaultToolkit().getScreenSize()));
ImageIO.write(image, "png", new File("/screenshot.png"));
Compare the screenshots using something like that: How to check if two images are similar or not using openCV in java?

Have a look at Sikuli project. Their automation engine is based on image comparison.
I guess, internally they are still using OpenCV for calculating image similarity, but there are plenty of OpenCV Java bindings like this, which allow to do so from Java.
Project source code is located here: https://github.com/sikuli/sikuli

Ok then, so I found an answer after a few days.
This method takes the screenshot:
public static void takeScreenshot() {
try {
BufferedImage image = new Robot().createScreenCapture(new Rectangle(490,490,30,30));
/* this two first parameters are the initial X and Y coordinates. And the last ones are the increment of each axis*/
ImageIO.write(image, "png", new File("C:\\Example\\Folder\\capture.png"));
} catch (IOException e) {
e.printStackTrace();
} catch (HeadlessException e) {
e.printStackTrace();
} catch (AWTException e) {
e.printStackTrace();
}
}
And this other one will compare the images
public static String compareImage() throws Exception {
// savedImage is the image we want to look for in the new screenshot.
// Both must have the same width and height
String c1 = "savedImage";
String c2 = "capture";
BufferedInputStream in = new BufferedInputStream(new FileInputStream(c1
+ ".png"));
BufferedInputStream in1 = new BufferedInputStream(new FileInputStream(
c2 + ".png"));
int i, j;
int k = 1;
while (((i = in.read()) != -1) && ((j = in1.read()) != -1)) {
if (i != j) {
k = 0;
break;
}
}
in.close();
in1.close();
if (k == 1) {
System.out.println("Ok...");
return "Ok";
} else {
System.out.println("Fail ...");
return "Fail";
}
}

Using HtmlUnit to click on multiple images on same page that have same source attribute but different onclick events

I'm using HtmlUnit to get data from a webpage. The web page has multiple images that look the same and have the same src value but each one has a different onclick event that directs it to a new page depending on which image was clicked and it is this new page that I need to save the data from. I need to loop through the images and click each one to get the results of the onclick event. At the moment the code I have loops through the images but each file contains the output from the onclick of the first image.
Can someone point out where I am missing something please? My code is as follows:
customer in the file name is a variable that I have declared earlier in the code and i changes for each loop so that each file has a different name.
DomNodeList<DomElement> iterable2 = page.getElementsByTagName("img");
Iterator<DomElement> i3 = iterable2.iterator();
int i = 0;
while(i3.hasNext())
{
HtmlElement element1 = null;
DomElement anElement = i3.next();
if(anElement instanceof HtmlImage)
{
HtmlImage input = (HtmlImage) anElement;
if(input.getSrcAttribute().equalsIgnoreCase("customise.gif") )
{
element1 = input;
page2 = element1.click();
webClient.waitForBackgroundJavaScript(30000);
String result = page2.asText();
try {
BufferedWriter out = new BufferedWriter(new FileWriter("Filepath//"+customer+i+".txt"));
out.write(result);
out.close();
}
catch (IOException e)
{
System.out.println("Exception ");
}
i++;
}
}
}

I got this working by using a solution posted for a similar problem on HtmlUnit site which can be found here http://htmlunit.10904.n7.nabble.com/Problem-in-clicking-multiple-javaScript-links-on-a-page-td22682.html
Using the solution in the final posting on the thread in the link my code is now:
page = link3.click();
// Added the following line of code
Object page1Script = page.getEnclosingWindow().getScriptObject();
HtmlPage page2 = null;
Iterable<DomElement> iterable2 = page.getElementsByTagName("img");
Iterator<DomElement> i3 = iterable2.iterator();
int i = 0;
while(i3.hasNext())
{
// Added the following two lines of code
page.getEnclosingWindow().setEnclosedPage(page);
page.getEnclosingWindow().setScriptObject(page1Script);
HtmlElement element1 = null;
page2 = null;
DomElement anElement = i3.next();
if(anElement instanceof HtmlImage)
{
HtmlImage input = (HtmlImage) anElement;
if(input.getSrcAttribute().equalsIgnoreCase("customize.gif") )
{
element1 = input;
page2 = element1.click();
webClient.waitForBackgroundJavaScript(30000);
String result = page2.asText();
try {
BufferedWriter out = new BufferedWriter(new FileWriter("Filepath\\"+customer+i+".txt"));
out.write(result);
out.close();
}
catch (IOException e)
{
System.out.println("Exception ");
}
i++;
}
}
}