I have a two page application:
/login
/profile
I want to get .har file page /profile.
When i go to the page /login, the cookie is created with a key=connect.sid and value = "example value". This cookie is not yet active.
I added the cookies with active connect.sid.
WebDriver webDriver = getDriver();
webDriver.get(LOGIN_PAGE);
webDriver.manage().addCookie(connectsSId);
it does not work because after the load page, /login crated a new cookies.
i also tried this code:
WebDriver webDriver = getDriver();
webDriver.get(PROFILE_PAGE);
webDriver.manage().deleteAllCookies();
webDriver.manage().addCookie(connectsSId);
and this does not work. cookies were added but it seems too late.
WebDriver webDriver = getDriver();
LoginPage loginPage = new LoginPage(getDriver());
LandingPage landingPage = loginPage.login();
landingPage.openProfilePage();
This code created a .har file for the page /login.
for some reason, the file is created only after the first call to the page. I can not solve this problem.
Use PhantomJS with BrowserMobProxy. PhantomJS helps us for JavaScript enables pages. The following code works for HTTPS web addresses, too.
Place 'phantomjs.exe' in C drive and you get the 'HAR-Information.har' file in C drive itself.
Make sure you DO NOT put a ' / ' at the end of the url, like
driver.get("https://www.google.co.in/")
It should be
driver.get("https://www.google.co.in");
Otherwise, it won't work.
package makemyhar;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import net.lightbody.bmp.BrowserMobProxy;
import net.lightbody.bmp.BrowserMobProxyServer;
import net.lightbody.bmp.core.har.Har;
import net.lightbody.bmp.proxy.CaptureType;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.phantomjs.PhantomJSDriver;
import org.openqa.selenium.phantomjs.PhantomJSDriverService;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
public class MakeMyHAR {
public static void main(String[] args) throws IOException, InterruptedException {
//BrowserMobProxy
BrowserMobProxy server = new BrowserMobProxyServer();
server.start(0);
server.setHarCaptureTypes(CaptureType.getAllContentCaptureTypes());
server.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
server.newHar("Google");
//PHANTOMJS_CLI_ARGS
ArrayList<String> cliArgsCap = new ArrayList<>();
cliArgsCap.add("--proxy=localhost:"+server.getPort());
cliArgsCap.add("--ignore-ssl-errors=yes");
//DesiredCapabilities
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
capabilities.setCapability(CapabilityType.SUPPORTS_JAVASCRIPT, true);
capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, cliArgsCap);
capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,"C:\\phantomjs.exe");
//WebDriver
WebDriver driver = new PhantomJSDriver(capabilities);
driver.get("https://www.google.co.in");
//HAR
Har har = server.getHar();
FileOutputStream fos = new FileOutputStream("C:\\HAR-Information.har");
har.writeTo(fos);
server.stop();
driver.close();
}
}
Set preferences in your Selenium code:
profile.setPreference("devtools.netmonitor.har.enableAutoExportToFile", true);
profile.setPreference("devtools.netmonitor.har.defaultLogDir", String.valueOf(dir));
profile.setPreference("devtools.netmonitor.har.defaultFileName", "network-log-file-%Y-%m-%d-%H-%M-%S");
and open console:
Actions keyAction = new Actions(driver);
keyAction.keyDown(Keys.LEFT_CONTROL).keyDown(Keys.LEFT_SHIFT).sendKeys("q").keyUp(Keys.LEFT_CONTROL).keyUp(Keys.LEFT_SHIFT).perform();
You can use browsermob proxy to capture all the request and response data
See here
I have tried as well to get the har file using a proxy like browsermob proxy
I did a lot of research because the file which I've received was always empty.
What I did was to enable the browser performance log.
Note this will work only with chrome driver.
This is my driver class (in python)
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium import webdriver
from lib.config import config
class Driver:
global performance_log
capabilities = DesiredCapabilities.CHROME
capabilities['loggingPrefs'] = {'performance': 'ALL'}
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument("--headless")
mobile_emulation = {"deviceName": "Nexus 5"}
if config.Env().is_mobile():
chrome_options.add_experimental_option(
"mobileEmulation", mobile_emulation)
else:
pass
chrome_options.add_experimental_option(
'perfLoggingPrefs', {"enablePage": True})
def __init__(self):
self.instance = webdriver.Chrome(
executable_path='/usr/local/bin/chromedriver', options=self.chrome_options)
def navigate(self, url):
if isinstance(url, str):
self.instance.get(url)
self.performance_log = self.instance.get_log('performance')
else:
raise TypeError("URL must be a string.")
The amount of information which is found the in output is huge so you'll have to filter the raw data and get the network received and send objects only.
import json
import secrets
def digest_log_data(performance_log):
# write all raw data in a file
with open('data.json', 'w', encoding='utf-8') as outfile:
json.dump(performance_log, outfile)
# open the file and real it with encoding='utf-8'
with open('data.json', encoding='utf-8') as data_file:
data = json.loads(data_file.read())
return data
def digest_raw_data(data, mongo_object={}):
for idx, val in enumerate(data):
data_object = json.loads(data[idx]['message'])
if (data_object['message']['method'] == 'Network.responseReceived') or (data_object['message']['method'] == 'Network.requestWillBeSent'):
mongo_object[secrets.token_hex(30)] = data_object
else:
pass
We choose to push this data into a mongo db which will be analyse later by an etl and pushed into a redshift database to create statistics .
I hope is what you are looking for.
The way Im running the script is :
import codecs
from pprint import pprint
import urllib
from lib import mongo_client
from lib.test_data import test_data as data
from jsonpath_ng.ext import parse
from IPython import embed
from lib.output_data import process_output_data as output_data
from lib.config import config
from lib import driver
browser = driver.Driver()
# get the list of urls which we need to navigate
urls = data.url_list()
for url in urls:
browser.navigate(config.Env().base_url() + url)
print('Visiting ' + url)
# get performance log
performance_log = browser.performance_log
# digest the performace log
data = output_data.digest_log_data(performance_log)
# initiate an empty dict
mongo_object = {}
# prepare the data for the mongo document
output_data.digest_raw_data(data, mongo_object)
# load data into the mongo db
mongo_client.populate_mongo(mongo_object)
browser.instance.quit()
My main source was this one which I've adjusted it to my needs.
https://www.reddit.com/r/Python/comments/97m9iq/headless_browsers_export_to_har/
Thanks
You may do it by the simplest way Selenide + Java + JS
import java.nio.file.Files and java.nio.file.Paths in you class
Then create function:
public static void getHar() {
open("http://you-task.com");
String scriptGetInfo = "performance.setResourceTimingBufferSize(1000000);" +
"return performance.getEntriesByType('resource').map(JSON.stringify).join('\\n')";
String har = executeJavaScript(scriptGetInfo);
Files.write(Paths.get("log.har"), har.getBytes());
}
It saves you log.har in the root of you project.
Just call this function in the place you want to save har-file
Related
I'm trying to use selenium dev tools java API, and for multiple API methods I'm getting java.util.concurrent.TimeoutException.
For example I'm trying to use Network.clearBrowserCache, which should work accroding to chromedriver docs: https://chromedevtools.github.io/devtools-protocol/tot/Network/
I'm calling clearBrowserCache using following code:
chromeDriver.getDevTools().send(Network.clearBrowserCache())
It fails, but at the same time if I use other devTools commands like this:
chromeDriver.getDevTools().send(Browser.getVersion())
It returns data properly.
Chrome version is: 85.0.4183.39
Chromedriver version is: 85.0.4183.87
Selenium-java version is: 4.0.0-alpha-6
Try calling createSession before calling clearBrowserCache.
Using your setup, this works:
chromeDriver.getDevTools().createSession();
chromeDriver.getDevTools().send(Network.clearBrowserCache())
and this produces java.util.concurrent.TimeoutException:
chromeDriver.getDevTools().send(Network.clearBrowserCache())
You can verify that the browser cache has been cleared with this snippet:
ChromeDriver driver = new ChromeDriver();
driver.get("https://refreshyourcache.com/en/cache-test/");
Thread.sleep(2000);
driver.getDevTools().createSession();
driver.getDevTools().send(Network.clearBrowserCache());
driver.get("https://refreshyourcache.com/en/cache-test/");
Thread.sleep(5000);
If you run the code, the pages displayed in the test browser will show these images:
If you commment out the line driver.getDevTools().send(Network.clearBrowserCache()); then you get a different result:
Using Selenium 4.0.0-alpha-6, Chrome v85 and ChromeDriver v85.0 through google-chrome-devtools you must be able to use getVersion() method as follows:
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.browser.Browser;
public class BrowserGetVersion {
public static void main(String[] args) {
System.setProperty("webdriver.chrome.driver","C:\\WebDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("--start-maximized");
ChromeDriver driver = new ChromeDriver(options);
DevTools devTools = driver.getDevTools();
devTools.createSession();
devTools.send(Browser.getVersion());
}
}
Similarly, using the clearBrowserCache() method you should be able to clear the browser cache using the following code block:
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.network.Network;
public class ClearChromeCache {
public static void main(String[] args) {
System.setProperty("webdriver.chrome.driver","C:\\WebDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("--start-maximized");
ChromeDriver driver = new ChromeDriver(options);
DevTools devTools = driver.getDevTools();
devTools.createSession();
devTools.send(Network.clearBrowserCache());
driver.get("https://www.google.com/");
}
}
Additional Consideration
Additionally, you can also use setCacheDisabled(true) to completely disable the cache as follows:
Code Block:
import java.util.Collections;
import java.util.Optional;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.network.Network;
import org.testng.Assert;
import org.testng.annotations.Test;
public class testngBasic {
#Test
public void foo() {
System.setProperty("webdriver.chrome.driver","C:\\WebDrivers\\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.addArguments("--start-maximized");
options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
options.setExperimentalOption("useAutomationExtension", false);
ChromeDriver driver = new ChromeDriver(options);
DevTools devTools = driver.getDevTools();
devTools.createSession();
devTools.send(Network.clearBrowserCache());
devTools.send(Network.enable(Optional.empty(), Optional.empty(), Optional.of(100000000)));
devTools.send(Network.setCacheDisabled(true));
devTools.addListener(Network.responseReceived(), responseReceived -> Assert.assertEquals(false, responseReceived.getResponse().getFromDiskCache()));
driver.get("https://www.google.com/");
}
}
This usecase
Possibly your code have nothing to do with java.util.concurrent.TimeoutException error and the real issue is either with the:
jdk version
guava version
Solution
Ensure that:
JDK is upgraded to current levels JDK 8u252.
guava is upgraded to guava-29.0-jre.
Outro
Disable cache in Selenium Chrome Driver
It worked fine
public void testCdt {
final ChromeLauncher launcher = new ChromeLauncher();
final ChromeService chromeService = launcher.launch(false);
final ChromeTab tab = chromeService.createTab();
final ChromeDevToolsService devToolsService = chromeService.createDevToolsService(tab);
final Page page = devToolsService.getPage();
Network network = devToolsService.getNetwork();
// Clear browser cached
network.clearBrowserCache();
// Log requests with onRequestWillBeSent event handler.
network.onRequestWillBeSent(
event ->
System.out.printf(
"request: %s %s%s",
event.getRequest().getMethod(),
event.getRequest().getUrl(),
System.lineSeparator()));
network.onLoadingFinished(
event -> {
chromeService.closeTab(tab);
launcher.close();
});
network.enable();
page.navigate("http://github.com");
devToolsService.waitUntilClosed();
}
When writing a basic search test for a job website in Selenium Java, I ma having problems when trying to accept the cookie warning displayed on the site.
The site has 2 cookie notifications, a middle layer and top layer banner that sit on each other.
I would be grateful for any suggestions (I'm new to Selenium Java!) that would allow me to get past this somewhat irritating (but minor) issue.
This is the code I am using to no avail (both cookie banners remains in place):
I have attempted the xpath approach detailed below
import java.util.Arrays;
import java.util.Set;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.Cookie;
import org.openqa.selenium.Keys;
import java.util.concurrent.TimeUnit;
importnet.bytebuddy.agent.builder.AgentBuilder.RedefinitionStrategy.DiscoveryStrategy.Explicit;
//These are being imported from the Selenium package supplied via Project Level Build Path>External Libraries
public class Demo4SeleniumTypeAndClickXPathExperis {
public static void main(String[] args) {
System.setProperty("webdriverchrome.driver", "C:\\chromedriver.exe");
WebDriver driver=new ChromeDriver();
driver.get("https://www.experis.co.uk/");//Browser URL
driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
//here is the offending item that seems to achieve no result
driver.findElement(By.xpath("//button[#title='Accept Cookies']")).submit();
driver.findElement(By.xpath("//*[#id=\"query\"]")).sendKeys("test or tester or qa");
driver.findElement(By.xpath("//*[#id=\"search\"]/span/div/div[1]/input")).clear();
driver.findElement(By.xpath("//*[#id=\"search\"]/span/div/div[1]/input")).sendKeys("Bristol");
driver.findElement(By.xpath("//*[#id=\"search\"]/span/div/div[1]/input")).sendKeys(Keys.RETURN);
driver.findElement(By.xpath("//*[#id=\"search\"]/span/div/div[1]/input")).submit();
driver.findElement(By.xpath("//*[#id=\"search\"]/span/div/div[1]/input")).submit();
//driver.close();
}
private static Object navigate() {
// TODO Auto-generated method stub
return null;
I am expecting to be able to accept the cookie banners and clear them from the screen
Use below code to accept both the cookies alert.
I've tested the code on the you URL you provided in your code.
By cookies_accept = By.xpath("//*[#title='Accept Cookies']");
By cookies_gotIt = By.xpath("//a[text()='Got it!']");
WebDriverWait wait = new WebDriverWait(driver, 10);
wait.until(ExpectedConditions.elementToBeClickable(cookies_accept)).click();
wait.until(ExpectedConditions.invisibilityOfElementLocated(cookies_accept));
wait.until(ExpectedConditions.elementToBeClickable(cookies_gotIt)).click();
To get a more specific answer, I recommend posting the HTML page source which includes HTML for the cookie accept button.
In my experience with accepting cookies, you might have to treat the popup as an alert:
driver.switchTo().alert().accept;
package com.selenium_abcd;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
public class cookiesDelete {
public static void main(String[] args) {
System.setProperty("webdriver.chrome.driver", "C:\\selenium\\chromedriver.exe");
WebDriver dr = new ChromeDriver();
dr.manage().deleteAllCookies();
dr.manage().deleteCookieNamed("Cookie name");
}
}
I need to implement a function that make a screenshot for a web page in a java backend project. I found some methods like using a headless browser is a good way but none of them performances perfectly (like jbrowser and ashot) for a long page or with too many images. I found that firefox has a function can make a screenshot for me. I wonder is there any java API for this function in headless mode? Or is there any other way to get a better screenshot performance? Thanks a lot
Here is my code to get a screenshot
package screenshot;
import com.machinepublishers.jbrowserdriver.JBrowserDriver;
import com.machinepublishers.jbrowserdriver.Settings;
import com.machinepublishers.jbrowserdriver.Timezone;
import org.openqa.selenium.Dimension;
import org.openqa.selenium.OutputType;
import ru.yandex.qatools.ashot.AShot;
import ru.yandex.qatools.ashot.Screenshot;
import ru.yandex.qatools.ashot.shooting.ShootingStrategies;
import javax.imageio.ImageIO;
import java.io.*;
public class JbrowserTest {
public String chekUrl(String str){
if (str.startsWith("http://") || str.startsWith("https://")) {
return str;
}
return str;
}
public static void main(String[] args) throws UnsupportedEncodingException {
// You can optionally pass a Settings object here,
// constructed using Settings.Builder
JBrowserDriver driver = new JBrowserDriver(Settings.builder().
timezone(Timezone.ASIA_SHANGHAI).screen(new Dimension(1920,1080)).build());
String url3 = "http://www.google.com";
// This will block for the page load and any
// associated AJAX requests
driver.get(url3);
driver.manage().window().maximize();
// You can get status code unlike other Selenium drivers.
// It blocks for AJAX requests and page loads after clicks
// and keyboard events.
System.out.println(driver.getStatusCode());
// Returns the page source in its current state, including
// any DOM updates that occurred after page load
String string2 = new String(driver.getPageSource().getBytes("utf-8"),"gb2312");
System.out.println(string2);
Screenshot screenshot2 = new AShot().shootingStrategy(ShootingStrategies.viewportPasting(100))
.takeScreenshot(driver);
try {
ImageIO.write(screenshot2.getImage(), "PNG",
new File("/Users/*******/Desktop/test2.png"));
byte[] screenshot = driver.getScreenshotAs(OutputType.BYTES);
System.out.println("the bytes" + screenshot.length);
String filePath = "/Users/*******/Desktop/test.png";
File file = new File(filePath);
FileOutputStream fw = new FileOutputStream(file);
fw.write(screenshot);
fw.close();
} catch (Exception ex) {
System.out.println("error" + ex);
}
// Close the browser. Allows this thread to terminate.
driver.quit();
}
}
You don't exactly specify your "performance requirement". An easy way to take screenshots by utilizing selenium and chrome drivers:
private void loadWebpage(){
//Init driver
ChromeOptions options = new ChromeOptions();
options.setHeadless(true);
options.addArguments("--window-size=1200x600", "--log-level=3");
WebDriver driver = new ChromeDriver(options);
//Load your website & wait until loaded with webdriver wait
takeScreenShot(driver, new File("outputFile.png"))
}
public static void takeScreenshot(WebDriver driver, File screenshotFile) throws
IOException {
File scrFile = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
Files.copy(scrFile.toPath(), screenshotFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
}
I made a working test locally with embedded Browsermob proxy server. Nothing new, but still here is the sample code.
_server = new BrowserMobProxyServer()
_server.start();
Proxy proxy = ClientUtil.createSeleniumProxy(_server);
ChromeOptions options = new ChromeOptions();
options.setCapability("proxy", proxy);
_driver = new ChromeDriver(options);
Now we're looking into options to integrate such tests into our CI pipeline and to execute these tests in the cloud (Browserstack/Sauce Labs). I'm trying to figure out what the setup will look like in this case. Right now my understanding is that the code (which sets up the proxy and actually contains the tests) will run on our server. This means that the embedded proxy will also run on our server which is not necessarily accessible from the outside. So the questions are:
Will I have to switch to a standalone browsermob proxy and make it accessible?
If yes, then is there any actual code sample of using the standalone proxy from code? (This options doesn't look particularly appealing since we'll have to write boilerplate code to wrap the REST API)
If no, then am I correct assuming the remote Selenium Webdriver will connect to the website under test through the newly set up embedded proxy by means of tunnelling (Sauce Connect and alike)?
What is the best practice of using Browsermob with CI server with cloud-based testing platforms?
If the test/webdriver instance will be running on a remote machine (browserstack or sauce) in your case, it is essential that the proxy generated by your proxy server should be authenticated on the remote machine to intercept traffic. I had a similar requirement and I set it up using a standalone BrowserMob instance. Below is a working sample code for browserstack with their local testing binary:
This will need the following Dependencies:
<dependency>
<groupId>com.browserstack</groupId>
<artifactId>browserstack-local-java</artifactId>
<version>1.0.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.5</version>
<scope>test</scope>
</dependency>
Code snippet:
import com.browserstack.local.Local;
import net.lightbody.bmp.BrowserMobProxy;
import net.lightbody.bmp.BrowserMobProxyServer;
import net.lightbody.bmp.client.ClientUtil;
import net.lightbody.bmp.core.har.Har;
import net.lightbody.bmp.proxy.CaptureType;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.ie.InternetExplorerDriver;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import java.io.File;
import java.net.URL;
import java.util.HashMap;
public class InterceptProxy {
public static final String USERNAME = <BrowserStack Username>;
public static final String AUTOMATE_KEY = <BrowserStack Key>;
public static final String URL = "https://" + USERNAME + ":" + AUTOMATE_KEY + "#hub-cloud.browserstack.com/wd/hub";
public static void main(String[] args) throws Exception {
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.start(0);
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
Local browserStackLocal = new Local();
HashMap<String, String> browserStackLocalArgs = new HashMap<String, String>();
browserStackLocalArgs.put("key", AUTOMATE_KEY);
browserStackLocalArgs.put("forcelocal", "true");
browserStackLocalArgs.put("forceproxy","true");
browserStackLocalArgs.put("force","true");
browserStackLocalArgs.put("v", "true");
String host=seleniumProxy.getHttpProxy().substring(0,seleniumProxy.getHttpProxy().indexOf(":"));
String port=seleniumProxy.getHttpProxy().substring(seleniumProxy.getHttpProxy().indexOf(":")+1,seleniumProxy.getHttpProxy().length());
browserStackLocalArgs.put("-local-proxy-host", host);
browserStackLocalArgs.put("-local-proxy-port", port);
browserStackLocal.start(browserStackLocalArgs);
DesiredCapabilities caps = new DesiredCapabilities();
caps.setCapability("browser", "Chrome");
caps.setCapability("browser_version", "62.0");
caps.setCapability("os", "Windows");
//caps.setCapability(CapabilityType.PROXY, seleniumProxy);
caps.setCapability("os_version", "10");
caps.setCapability("browserstack.local",true);
WebDriver driver = new RemoteWebDriver(new URL(URL), caps);
driver.manage().deleteAllCookies();
driver.manage().window().maximize();
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
// create a new HAR with the label "yahoo.com"
proxy.newHar("yahoo.com");
// open yahoo.com
driver.get("http://yahoo.com");
// get the HAR data
Har har = proxy.getHar();
//Writing Har to file
har.writeTo(new File("/Users/MyUser/Desktop/HAR.txt"));
driver.quit();
browserStackLocal.stop();
proxy.stop();
}
}
I need to extract coinmarket cap volume (ex: Market Cap: $306,020,249,332) from top of page with Java, please see picture attached.
I have used jsoup library in Java Eclipse but didn't extract volume. Jsoup extract only other attributes. Probably problem is from a java script library.
Also I have used html unit without success:
import java.io.IOException;
import java.util.List;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlAnchor;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
public class Testss {
public static void main(String\[\] args) throws IOException {
String url = "https://coinmarketcap.com/faq/";
WebClient client = new WebClient();
HtmlPage page = client.getPage(url);
List<?> anchors = page.getByXPath("//div\[#class='col-sm-6 text-center'\]//a");
for (Object obj : anchors) {
HtmlAnchor a = (HtmlAnchor) obj;
System.out.println(a.getTextContent().trim());
}
}
}
How can I extract volume from this site with Java?
Thanks!
Check the network tab findout the exact request which is fetching the data, In your case its https://files.coinmarketcap.com/generated/stats/global.json
Also the request URL is the below one
So, Fetching the main URL will not give you what you require, For that you have to fetch the data from the request URL directly and parse it using any JSON library. SimpleJSON I can suggest in one of those.
The JSON data which you will get after hitting the url.
{
"bitcoin_percentage_of_market_cap": 55.95083004655126,
"active_cryptocurrencies": 1324,
"total_volume_usd": 21503093761,
"active_markets": 7009,
"total_market_cap_by_available_supply_usd": 301100436864
}