Develop Reference

How to get text from nested span using Jsoup?

I'm trying to get the text in the span
using this code below. However the output is behaving as if the nested spans don't exist
Elements tags = document.select("div[id=tags]");
for (Element tag:tags){
Elements child_tags = tag.getElementsByTag("class");
String key = tag.html();
System.out.println(key); //only as a test
for (Element child_tag:child_tags){
System.out.println("\t" + child_tag.text());
}
My output is
<hr />Tags:
<span id="category"></span>
<span id="voteSelector" class="initially_hidden"> <br /> </span>

Assuming you are trying the code on https://chesstempo.com/chess-problems/15 and the data you want is shown in the below image
Now, Using Jsoup you will get the data whatever is rendered as a source code in the browser,for confirmation you can press CTRL+U in browser which will open up a new window where the actual contents which Jsoup will get will be displayed. Now coming to your questions the part which you are trying to retrieve itself is not present in the browser source code check that by pressing CTRL+U.
If the contents are rendered using JAVASCRIPT those will not be visible to JSOUP and hence you have to use something else which will run the javascript and provide you the details.
JSoup does not run Javascript and is not a browser.
EDIT
There is a turnaround using SELENIUM. Below is the working code to get the exact source code of the url and the required data which you are looking for:
import java.io.IOException;
import java.io.PrintWriter;
import org.json.simple.parser.ParseException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
public class JsoupDummy {
public static void main(String[] args) throws IOException, ParseException {
System.setProperty("webdriver.gecko.driver", "D:\\thirdPartyApis\\geckodriver-v0.19.1-win32\\geckodriver.exe");
WebDriver driver = new FirefoxDriver();
try {
driver.get("https://chesstempo.com/chess-problems/15");
Document doc = Jsoup.parse(driver.getPageSource());
Elements elements = doc.select("span.ct-active-tag");
for (Element element:elements){
System.out.println(element.html());
}
} catch (Exception e) {
e.printStackTrace();
} finally {
/*write.flush();
write.close();*/
driver.quit();
}
}
}
You need selenium web driver Selenium Web Driver which simulates the browser behaviour and allows you to render the html content written by scripts as well.

Elements child_tags = tag.getElementsByTag("class");
With this line you are trying to get an element with tag class i.e <class>...</class>, which dose not exist. Change that line to:
Elements child_tags = tag.getElementsByClass("tag");
to get elements by attribute value of class = tag or to:
Elements child_tags = tag.getElementsByTag("span");
to get elements by tag name = span.

Unable to create or switch to new tab in Selenium Webdriver with JAVA on MAC

I've tried a different options just to open new tab but all of them lead to the same result : sending char "t" to google search field.
The goal in my real test is to switch between tabs in browser, but I am unable even open new one.
Very simple test
public class LoginPhp2 {
#Test
public void testGoogle() {
WebDriver driver = new SafariDriver();
driver.get("https://www.google.com");
//driver.findElement(By.cssSelector("body")).sendKeys(Keys.COMMAND + "t");
Actions action= new Actions(driver);
action.keyDown(Keys.COMMAND).sendKeys("t").build().perform();
//action.keyDown(Keys.COMMAND).sendKeys("t").keyUp(Keys.COMMAND).build().perform();
}
}

You may use javascript to open a new tab.
JavascriptExecutor js = (JavascriptExecutor) driver;
js.executeScript("window.open();");

There are different ways of opening new tabs in windows and Mac. Actual keys to send depend on your OS, for example, Mac uses COMMAND + t, instead of CONTROL + t.
You can open new tab in Mac using:
driver.findElement(By.cssSelector("body")).sendKeys(Keys.COMMAND +"t");
After this you can switch to any of the opened tabs:
ArrayList<String> tabs = new ArrayList<String> (driver.getWindowHandles());
driver.switchTo().window(tabs.get(0));

I have followed the approach suggested earlier:
JavascriptExecutor js = (JavascriptExecutor) driver;
js.executeScript("window.open();");
This is opening new tab in the same window. I have tried it on my MAC machine

How to run multiple browsers instance with Webdriver, Java and Junit/TestNG?

I have a test ready to be executed but it takes a long time to finish. In this test I'm feeding in csv data, so basically the whole test will run 56 times. I was wondering if there's anyway I could use multiple browser instance and divide the workload to four instance. It will save me some time. I tried to use TestNG's ThreadPoolSize but it's not doing what I want it to. It's using the same data for four instances of firefox. I want each browser to have it's own unique data. Please check my code and let me know what I'm missing. I really appriciate every one's help.
public class StudentPageTest {
WebDriver driver;
DesiredCapabilities capability;
WebElement element;
WebDriverWait wait;
private String baseURL;
#BeforeTest
public void setUp() throws MalformedURLException{
//capability = DesiredCapabilities.firefox();
//driver = new FirefoxDriver();
//wait = new WebDriverWait(driver, 120);
//driver.manage().deleteAllCookies();
baseURL = "http://somewebsite.com";
}
#SuppressWarnings("resource")
#Test(threadPoolSize = 4)
public void StudentPortalTest() throws InterruptedException, IOException{
driver = new FirefoxDriver();
wait = new WebDriverWait(driver, 120);
driver.manage().deleteAllCookies();
String studentId = "studentID.csv";
BufferedReader br = null;
String line = "";
String cvsSplitBy = ",";
br = new BufferedReader(new FileReader(studentId));
while ((line = br.readLine()) != null) {
String[] student_id = line.split(cvsSplitBy);
//Logging in Student Portal---------------------------------------------------------------------------------------------------------------|
for (int i = 0; i < student_id.length; i++) {
driver.get(baseURL+student_id[i]);
driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
driver.findElement(By.cssSelector(".logo>img")).isDisplayed();
driver.findElement(By.cssSelector("#UserName")).sendKeys("SecretUserName");
driver.findElement(By.cssSelector("#Password")).sendKeys("EvenMoreSecretPassword");
driver.findElement(By.cssSelector(".submitBtn")).click();
driver.manage().timeouts().implicitlyWait(120, TimeUnit.SECONDS);
Thread.sleep(4000);
...............and the test goes on below...................
}
#AfterTest
public void tearDown(){
driver.quit();
}
}

You can let the Selenium grid do the task of distribution. Use TestNG to run your cases in parallel using dataprovider. Use a dataprovider to read your csv and pass one data to one #Test.
Set dataprovider thread count in your suite xml and make sure you set #DataProvider(parallel=true)
Going to your code, replace your call of instantiating a firefoxdriver to remotewebdriver to use the grid. Make sure your driver is not at class level but is a threadlocal : threadlocal<driver>.

I was trying many different ways to achieve the same thing you are trying to do. After a lot of trial and error I finally came across a solution that worked for me.
You need Selenium Grid You can then set up a few nodes to run your different browsers from
You need TestNG like you have been using but instead of trying to use threads, you will use an .xml file to pass in parameters to your test which will run the one test over and over in parallel.
If you read this document carefully and follow the steps, you should be able to achieve the same result

#Test(threadPoolSize = 4, invocationCount = 4)

Selenium moveByOffset doesn't do anything

I'm running latest selenium 2.41 with Firefox 28.0 on Linux Xubuntu 13.10
I'm trying to get FirefoxDriver to move the mouse over the page (in my test, I've used the wired webpage, that has a lot of hover-activated menus), but the moveByOffset is not doing anything noticeable to the mouse, at all:
package org.openqa.mytest;
import java.util.List;
import java.io.File;
import java.lang.*;
import org.openqa.selenium.By;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.*;
import org.openqa.selenium.firefox.*;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.interactions.*;
import org.apache.commons.io.FileUtils;
public class Example {
public static void main(String[] args) throws Exception {
// The Firefox driver supports javascript
FirefoxProfile profile = new FirefoxProfile();
profile.setEnableNativeEvents(true);
WebDriver driver = new FirefoxDriver(profile);
// Go to the Google Suggest home page
driver.get("http://www.wired.com");
File scrFile = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
// now save the screenshto to a file some place
FileUtils.copyFile(scrFile, new File("./screenshot.png"));
Actions builder = new Actions(driver);
Action moveM = builder.moveByOffset(40, 40).build();
moveM.perform();
Action click = builder.click().build();
click.perform();
//click.release();
Action moveM2 = builder.moveByOffset(50, 50).build();
moveM2.perform();
Action click2 = builder.click().build();
click2.perform();
//click2.release();
Action moveM3 = builder.moveByOffset(150, 540).build();
moveM3.perform();
for( int i=0; i < 1000; i++)
{
moveM = builder.moveByOffset(200, 200).build();
moveM.perform();
Thread.sleep(500);
moveM = builder.moveByOffset(-200, -200).build();
moveM.perform();
Thread.sleep(500);
}
//Action click3 = builder.click().build();
//click3.perform();
//click3.release();
scrFile = ((TakesScreenshot)driver).getScreenshotAs(OutputType.FILE);
// now save the screenshto to a file some place
FileUtils.copyFile(scrFile, new File("./screenshot2.png"));
driver.quit();
}
}
I'm expecting the mouse the move over the different elements and trigger all the hover actions, but nothing is happening

The method moveByOffset of class Actions is or has been broken. See Selenium WebDriver Bug 3578
(The error is described some lines more down in this bug document).
A project member (barancev) claims that this error should have been fixed with Selenium version 2.42.
Nevertheless I found the same error in version 2.44 running on openSUSE 12.3 with Firefox 33.0. moveToElement works, moveToOffset doesn't.

I struggled as well getting drag and drop working.
It seems as if selenium has problems if the dragtarget is not visible, thus scrolling is requiered.
Anyway, that's the (Java) code that works. Note that I call "release()" without an argument - neither the dropable Element nor the dragable Element as argument worked for me. As well as "moveToElement(dropable)" didnt work for me, that's why I calculated the offset manually.
public void dragAndDrop(WebElement dragable, WebElement dropable,
int dropableOffsetX, int dropableOffsetY) {
Actions builder = new Actions(driver);
int offsetX = dropable.getLocation().x + dropableOffsetX
- dragable.getLocation().x;
int offsetY = dropable.getLocation().y + dropableOffsetY
- dragable.getLocation().y;
builder.clickAndHold(dragable).moveByOffset(offsetX, offsetY).release()
.perform();
}

i was also struggling with this and the solution that worked for me is below we have to add 1 to either X or Y co-ordinate.
Looks like (x,y) takes us to the edge of the element where its not clickable
Below worked for me
WebElement elm = drv.findElement(By.name(str));
Point pt = elm.getLocation();
int NumberX=pt.getX();
int NumberY=pt.getY();
Actions act= new Actions(drv);
act.moveByOffset(NumberX+1, NumberY).click().build().perform();
you could even try adding +1 to y coordinate that also works
act.moveByOffset(NumberX+1, NumberY).click().build().perform();

Please try using moveToElement. It should work.
Actions action = new Actions(webdriver);
WebElement we = webdriver.findElement(By.xpath("<XPATH HERE>"));
action.moveToElement(we).moveToElement(webdriver.findElement(By.xpath("/expression-here"))).click().build().perform();

i suggest that if your browser is not perform movetoelement and move to offset then you have put wrong offset of element
for find offset you use Cordinates plugin in chrome

Selenium takes lots of time to get dynamic page of given URL

I am doing a Project in Java.
In this project I have to work with DOM.
For that I first load a dynamic page of any given URL, by using Selenium.
Then I parse them using Jsoup.
I want to get the dynamic page source code of given URL
Code snapshot:
public static void main(String[] args) throws IOException {
// Selenium
WebDriver driver = new FirefoxDriver();
driver.get("ANY URL HERE");
String html_content = driver.getPageSource();
driver.close();
// Jsoup makes DOM here by parsing HTML content
Document doc = Jsoup.parse(html_content);
// OPERATIONS USING DOM TREE
}
But the problem is, Selenium takes around 95% of the whole processing time, that is undesirable.
Selenium first opens Firefox, then loads the given page, then gets the dynamic page source code.
Can you tell me how I can reduce the time taken by Selenium, by replacing this tool with another efficient tool. Any other advice would also be welcome.
Edit NO. 1
There is some code given on this link.
FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("general.useragent.override", "some UA string");
WebDriver driver = new FirefoxDriver(profile);
But what is second line here, I didn't understand. As Documentation is also very poor of selenium.
Edit No. 2
System.out.println("Fetching %s..." + url1);
System.out.println("Fetching %s..." + url2);
WebDriver driver = new FirefoxDriver(createFirefoxProfile());
driver.get("url1");
String hml1 = driver.getPageSource();
driver.get("url2");
String hml2 = driver.getPageSource();
driver.close();
Document doc1 = Jsoup.parse(hml1);
Document doc2 = Jsoup.parse(hml2);

Try this:
public static void main(String[] args) throws IOException {
// Selenium
WebDriver driver = new FirefoxDriver(createFirefoxProfile());
driver.get("ANY URL HERE");
String html_content = driver.getPageSource();
driver.close();
// Jsoup makes DOM here by parsing HTML content
// OPERATIONS USING DOM TREE
}
private static FirefoxProfile createFirefoxProfile() {
File profileDir = new File("/tmp/firefox-profile-dir");
if (profileDir.exists())
return new FirefoxProfile(profileDir);
FirefoxProfile firefoxProfile = new FirefoxProfile();
File dir = firefoxProfile.layoutOnDisk();
try {
profileDir.mkdirs();
FileUtils.copyDirectory(dir, profileDir);
} catch (IOException e) {
e.printStackTrace();
}
return firefoxProfile;
}
The createFireFoxProfile() method creates a profile if one doesn't exist. It uses if a profile already exists. So selenium doesn't need to create the profile-dir structure each and every time.

if you are sure, confident about your code, you can go with phantomjs. it is a headless browser and will get your results with quick hits. FF will take time to execute.