i am trying to parse data from a webpage which is structured in a way like, 20 records per sheet
i am able to get the first 20 records and then i have to move to the second sheet, the second sheet link is as shown below
2
the href element has some javascript that loads the next 20 records. when it does, the url remains same.
the Javascript which is called
var theForm = document.forms['aspnetForm'];
if (!theForm) {
theForm = document.aspnetForm;
}
function __doPostBack(eventTarget, eventArgument) {
if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
theForm.__EVENTTARGET.value = eventTarget;
theForm.__EVENTARGUMENT.value = eventArgument;
theForm.submit();
}
}
how can i handle this part in jsoup? is it even possible?
thanks in advance.
Related
I need to write a selenium with java code where I need to perform all image font and size check but there is a pagination
where the default page set is 50 what if I need to perform font and size check in each page. I have attached my code but it will
check only first page.
Scenario:
1.On Parent page only 50 links are displayed I need to click on each record and perform font/size check in same page
I need to again click on next page and perform same font/size check after completing it should navigate to parent page and
again click on other records and vice versa but again we have pagination on parent page as well.
List<WebElement> list=driver.findElements(By.xpath("//a[#class='primary-cell-text link']"));
System.out.println(list.size());
//Here Pagination code is required ???
ArrayList<String> hrefs = new ArrayList<String>(); //List for storing all href values
for (WebElement var : list) {
System.out.println(var.getText()); // fetch the text present between the anchor tags
System.out.println(var.getAttribute("href"));
hrefs.add(var.getAttribute("href"));
System.out.println("*************************************");
}
//Navigating to each link
int i=0;
for (String href : hrefs) {
driver.navigate().to(href);
System.out.println((++i)+": navigated to URL with href: "+href);
Thread.sleep(3000); // To check if the navigation is done properly.
System.out.println("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++");
Thread.sleep(5000);
//Here Pagination code is required ????
//Below code is to perform action
List<WebElement> block=driver.findElements(By.xpath("//*[text()='High' or text()='Medium' or text()='Low']/preceding-sibling::div"));
for(int b=0;b<block.size();b++) {
System.out.println(block.get(b).getCssValue("height"));
System.out.println(block.get(b).getCssValue("width"));
String h=block.get(b).getCssValue("height");
String w=block.get(b).getCssValue("width");
if(h.equals("16px") && w.equals("16px")) {
System.out.println("Height/Width is Matching ");
}
else {
System.out.println("height/Width not matching");
}
}
Thread.sleep(3000);
//High Color Check
List<WebElement> high=driver.findElements(By.xpath("//*[text()='High']/preceding-sibling::div"));
for(int h=0;h<high.size();h++) {
WebElement hvar=driver.findElement(By.xpath(("(//*[text()='High']/preceding-sibling::div)["+(h + 1)+"]")));
String highColor=hvar.getCssValue("background-color");
System.out.println(highColor);
String hexHighcolor=Color.fromString(highColor).asHex();
System.out.println(hexHighcolor);
if(hexHighcolor.equals("#e11900")) {
System.out.println(": High color is matching i.e:#e11900");
}
else {
System.out.println(": Low color is not matching :#e11900");
}
}
You have to add logic for going to the next page like in web pages, we can either scroll down further or click on next page to perform pagination.
So for all pages that you have got, you can call a separate function to check your css values onto them.
I have this web page https://rrtp.comed.com/pricing-table-today/ and from that I need to get the information about Time (Hour Ending) and Day-Ahead Hourly Price column alone. I tried with the following code,
Document doc = Jsoup.connect("https://rrtp.comed.com/pricing-table-today/").get();
for (Element table : doc.select("table.prices three-col")) {
for (Element row : table.select("tr")) {
Elements tds = row.select("td");
if (tds.size() > 2) {
System.out.println(tds.get(0).text() + ":" + tds.get(1).text());
}
}
}
but unfortunately I am unable to get the data I need.
Is there something wrong in the code..? or This page can't be crawled...?
Need some help
As I said in comment:
You should hit https://rrtp.comed.com/rrtp/ServletFeed?type=pricingtabledual&date=20150717 because it's source from which data is loaded on the page you have pointed to.
Data under this link is not a valid html document (and this is why it's not working for you), but you can easily make it "quite" right.
All you have to do is first get the response and add <table>..</table> tags around it, then it's enough to parse it as html document.
Connection.Response response = Jsoup.connect("https://rrtp.comed.com/rrtp/ServletFeed?type=pricingtabledual&date=20150717").execute();
Document doc = Jsoup.parse("<table>" + response.body() + "</table>");
for (Element element : doc.select("tr")) {
System.out.println(element.html());
}
I have a form that runs a java agent on the WebQueryOpen event. This agent pulls data from a DB2 database and then puts them into the computed text fields I have placed on the form and are displayed whenever I open the form in the browser. This is working for me. However, when I try to use RichTextFields I get a ClassCastException error. No document is actually saved, I just open the form in the browser using this domino URL - https://company.com/database.nsf/sampleform?OpenForm
Sample code of simple text field - Displayed with w/o problems
Document sampledoc = agentContext.getDocumentContext();
String samplestr = "sample data from db2";
sampledoc.replaceItemValue("sampletextfield", samplestr);
When I tried using rich text field
Document sampledoc = agentContext.getDocumentContext();
String samplestr = "sample data from db2";
RichTextItem rtsample = (RichTextItem)sampledoc.getFirstItem('samplerichtextfield');
rtsample.appendText(samplestr); // ClassCastException error
Basically, I wanted to use rich text field so that it could accommodate more characters in case I pull a very long string data.
Screenshot of the field (As you can see it's a RichText)
The problem is that you're trying to access a regular Item as a RichTextItem.
The RichTextItem are special fields that are created with its own method just like this:
RichTextItem rtsample = (RichTextItem)sampledoc.createRichTextItem('samplerichtextfield');
It's different to the regular Items that can be created with a simple sampledoc.replaceItemValue(etc).
So, if you want to know if a item is RichTextItem and if it does not exist, create it, you can do this:
RichTextItem rti = null;
Item item = doc.getFirstItem("somefield");
if (item != null) {
if (item instanceof RichTextItem) {
//Yay!
rti = (RichTextItem) item;
} else {
//:-(
}
} else {
rti = doc.createRichTextItem("somefield");
//etc.
}
I have a window that displays my tweets in a label.
My tweets come from my FB page statuses and if i have put a pic or write more than 140 characters then i get a link in tweet to the actuall post.
I wonder if there is any way to get the label text to split so i can point the link into an url to open in webview
This is how far i have got:
var win = Ti.UI.currentWindow;
win.showNavBar();
var desc = Ti.UI.createLabel({
text: win.data,
font:{
fontSize:'20dp',
fontWeight:'bold'
},
height:'300dp',
left:'5dp',
top:'10dp',
color:'#111',
touchEnabled:true
});
win.add(desc);
desc.addEventListener('click',function(e){
var v = desc.text;
if(v.indexOf('http') != -1){
// open new window with webview
var tubeWindow = Ti.UI.createWindow({
modal: true,
barColor: '#050505',
backgroundColor: '#050505'
});
var linkview = Ti.UI.createWebView({
url: e.v,
barColor: '#050505',
backgroundColor: '#050505'
});
// Create a button to close the modal window
var close_modal = Titanium.UI.createButton({title:'Stäng'});
tubeWindow.rightNavButton = close_modal;
// Handle close_modal event
close_modal.addEventListener('click', function() {
tubeWindow.close();
});
tubeWindow.add(linkview);
tubeWindow.open({
modalTransitionStyle: Ti.UI.iPhone.MODAL_TRANSITION_STYLE_FLIP_HORIZONTAL,
});
}
});
win.open();
What i´ve been told i need to split the win.data to get the link. (win.data is the tweet)
now i just have: url: e.v, i need to get the link out
Any ideas on how this can work?
Thanx
//R
I did a similar thing a while ago. pull down the tweet(s) run the text through a regular expression to pull out a URL.
What I did was put each tweet in a tableview row, and set the tableview row to hasChild=true if the regular expression returned anything, then onClick of a tableView row, if hasChild == true open a webview with the given URL (stored in the row).
A regualr expression like the one here should work:
http://www.geekzilla.co.uk/view2D3B0109-C1B2-4B4E-BFFD-E8088CBC85FD.htm
So something like:
str= <<tweet text>>;
re= <<URL expression>>;
check=str.match(re);
now check contains either null or a url.
If I have a simple button:
<ice:panelGroup>
<ice:commandButton value="foobar"
action="#{fileManager.openNoFlashVisiblePopup}" />
</ice:panelGroup>
Is it possible to trigger the action openNoFlashVisiblePopup using just javascript? I know that there IceFaces has a JavaScript bridge but I don't know see a simple way to do just this.
i need to do this because I have a chunk of JavaScript that detects Flash and I need to show a IceFaces popup.
One way is to get the button element by ID and call its click() function.
document.getElementById('clientId').click();
You only need to give the form and button a fixed id so that you can use the generated HTML ID as clientId in Javascript code.
I know I'm a little late in seeing this, but the correct way to handle this (minus perhaps the overly exhuberant error checking) is:
// There's a <div> that looks like: <div class="portletfaces-bridge-body" id="A8660">.
// We'll find it and pull out the value of the ID to build elementId like: A8660:wtfForm:editeventparent
var div = null;
var divCollection = document.getElementsByTagName("div");
for (var i=0; i<divCollection.length; i++) {
if(divCollection[i].getAttribute("class") == "portletfaces-bridge-body") {
div = divCollection[i];
break;
}
}
if (div == null){
alert("could not find div portletfaces-bridge-body.");
return;
}
// Pull the id out of divInnerText.
var id = div.getAttribute("id");
if (id == null){
alert("id was null");
}
// prepare initializes fields to null so rendered cannot begin until both itemId and parentId are set.
var prepare = document.getElementById(id + ":wtfForm:editeventprepare");
if (prepare == null){
alert("editeventprepare element was not found.");
return;
}
prepare.click();