I have a code in which I traverse table rows and columns, and I'd like to add it's values to a list.
It takes me a significant amount of time.
So I added a time measurement, and I noticed that for some reason the time increases from row to row.
I cannot understand why.
Can you advise please?
private void buildTableDataMap() {
WebElement table = chromeWebDriver.findElement(By.id("table-type-1"));
List<WebElement> rows = table.findElements(By.tagName("tr"));
theMap.getInstance().clear();
String item;
for (WebElement row : rows) {
ArrayList<String> values = new ArrayList<>();
List<WebElement> tds = row.findElements(By.tagName("td"));
if(tds.size() > 0){
WebElement last = tds.get(tds.size() - 1);
long time = System.currentTimeMillis();
values.addAll(tds.stream().map(e->e.getText()).collect(Collectors.toList()));
System.out.println(System.currentTimeMillis() - time);
//remove redundant last entry:
values.remove(tds.size() - 1);
callSomeFunc(values, last);
item = tds.get(TABLE_COLUMNS.NAME_COL.getNumVal()).getText();
item = item.replaceAll("[^.\\- /'&A-Za-z0-9]", "").trim();//remove redundant chars
theMap.getInstance().getMap().put(item, values);
}
}
}
Guys, I continued researching.
First of all, Florent's kind answer did not help me because, at lease as I understand, It returned me an array list of strings which I had to parse, and I don't like this kind of solution too much...
So I nailed the problem in finding that the e.getText() call was increasing in time from call to call!!!
I also tried e.getAttribute("innerText") instead but no change.
Can't understand why. Any idea to solve?
WebElement last = null;
for (WebElement e : tds){
last = e;
long tm1 = 0, tm2 = 0;
if(Settings.verboseYN) {
tm1 = System.currentTimeMillis();
}
s = e.getText(); //This action increases in time!!!
if(Settings.verboseYN) {
tm2 = System.currentTimeMillis();
}
values.add(s); //a 0 ms action!!!
if(Settings.verboseYN) {
System.out.println("e.getText()) took " + (tm2 - tm1) + " ms...");
}
}
That is an graph of the time getText took...
08-May-18
Another source of growing execution time is this one:
void func(WebElement anchorsElement){
List<WebElement> anchors = anchorsElement.findElements(By.tagName("a"));
for (WebElement a : anchors) {
if (a.getAttribute("class").indexOf("a") > 0)
values.add("A");
else if (a.getAttribute("class").indexOf("b") > 0)
values.add("B");
else if (a.getAttribute("class").indexOf("c") > 0)
values.add("C");
}
}
Every functions has 5 iterations only, but still each call to the function increases its execution time.
Is there a solution for this one as well?
Calling the driver is an expensive operation. To significantly reduce the execution time, use a JavaScript injection with executeScript to read the whole table in a single call. Then process/filter the data on the client side with Java.
public ArrayList<?> readTable(WebElement table)
{
final String JS_READ_CELLS =
"var table = arguments[0]; " +
"return map(table.querySelectorAll('tr'), readRow); " +
"function readRow(row) { return map(row.querySelectorAll('td'), readCell) }; " +
"function readCell(cell) { return cell.innerText }; " +
"function map(items, fn) { return Array.prototype.map.call(items, fn) }; " ;
WebDriver driver = ((RemoteWebElement)table).getWrappedDriver();
Object result = ((JavascriptExecutor)driver).executeScript(JS_READ_CELLS, table);
return (ArrayList<?>)result;
}
The problem you are facing is because of the way Selenium works by design. Let's look at how a JavaScript get's executed or a operation is performed
tds.get(TABLE_COLUMNS.NAME_COL.getNumVal()).getText();
You have a collection of objects. Each object is assigned a unique ID on the browser side by the selenium driver
So when you do a getText() below is what happens
Your code -> HTTP Request -> Browser Driver -> Browser ->
|
<---------------------------------------------
Now if you have a table of 400rx10c then it accounts to 4000 HTTP calls, even if one call takes 10ms, we are looking at a 40000ms~=40sec, which is a decent delay to read a table
So what you want to do is to get all the data in single go by executing a javascript which give you 2d array back. It is quite simple, I found a code on below site
http://cwestblog.com/2016/08/21/javascript-snippet-convert-html-table-to-2d-array/
function tableToArray(tbl, opt_cellValueGetter) {
opt_cellValueGetter = opt_cellValueGetter || function(td) { return td.textContent || td.innerText; };
var twoD = [];
for (var rowCount = tbl.rows.length, rowIndex = 0; rowIndex < rowCount; rowIndex++) {
twoD.push([]);
}
for (var rowIndex = 0, tr; rowIndex < rowCount; rowIndex++) {
var tr = tbl.rows[rowIndex];
for (var colIndex = 0, colCount = tr.cells.length, offset = 0; colIndex < colCount; colIndex++) {
var td = tr.cells[colIndex], text = opt_cellValueGetter(td, colIndex, rowIndex, tbl);
while (twoD[rowIndex].hasOwnProperty(colIndex + offset)) {
offset++;
}
for (var i = 0, colSpan = parseInt(td.colSpan, 10) || 1; i < colSpan; i++) {
for (var j = 0, rowSpan = parseInt(td.rowSpan, 10) || 1; j < rowSpan; j++) {
twoD[rowIndex + j][colIndex + offset + i] = text;
}
}
}
}
return twoD;
}
I assume you store the above script in a SCRIPT variable and then you can run it like below
WebDriver driver = ((RemoteWebElement)table).getWrappedDriver();
Object result = ((JavascriptExecutor)driver).executeScript(SCRIPT + "\n return tableToArray(arguments[0]);" , table);
This will get you a 2D array of the data and you can then process it the way you like it
Related
I am automating a web application using Katalon studio with Selenium/Java. In a web table where we have to keep only first 15 records and rest of the records should be deleted. Each page contains 25 rows of data. The following is the code I have written but when I execute it deletes only 25 records and it stops. The code must delete the rows till it reaches 15, e.g. from 200 records to 15 records.
public void deleteRows() {
int rows_count;
int Row;
WebElement UsersTable;
WebElement UsersTable2;
List<WebElement> rows_UsersTable;
List<WebElement> cols_UserTable;
int cols_count;
int rowsNeeded = 15;
WebDriverWait wait;
List<WebElement> rows_UsersTable2;
UsersTable = driver.findElement(By.xpath("//table/tbody"));
rows_UsersTable = UsersTable.findElements(By.tagName("tr"));
rows_count = rows_UsersTable.size();
println ("Total rows is " +rows_count)
cols_UserTable = UsersTable.findElements(By.xpath("//table/tbody/tr[1]/td"));
cols_count = cols_UserTable.size();
println ("Total columns is " +cols_count)
if (rows_count >= rowsNeeded) {
int j = 1;
int newRowCount;
while (j <= rows_count) {
WebElement deleteIcon = driver.findElement(By.xpath("(//table/tbody/tr/td)[last()]/span"));
String text = deleteIcon.getText();
deleteIcon.click();
Thread.sleep(2000);
WebElement confirmBtn = driver.findElement(By.xpath("//button[text()='Confirm']"));
confirmBtn.click();
Thread.sleep(3000);
println ("Successfully deleted " + j);
UsersTable2 = driver.findElement(By.xpath("//table/tbody"));
rows_UsersTable2 = UsersTable.findElements(By.tagName("tr"));
newRowCount = rows_UsersTable2.size();
println ("The new row count is " + newRowCount)
j++;
}
}
}
I know there is something I have missed, please help.
Thanks
I have the following code, that allows me to remove a row from the right Jtable with a click. It works fine for all the rows, except when there is only one row remaining. BTW, sorry for most names being in portuguese, its my native language. Here are the images showing before and after i click the final row in the table. It updates the total, but the row remains. For every other case, it works perfectly.
Screenshot:
private void jtbSelecionadosMouseClicked(java.awt.event.MouseEvent evt)
{
int x = jtbSelecionados.rowAtPoint(evt.getPoint());
if (x >= 0)
{
String nomeProduto = (String)jtbSelecionados.getModel().getValueAt(x, 0);
for (int i = 0; i < itensVenda.size();i++)
{
if (itensVenda.get(i).getNomeProduto().equals(nomeProduto))
{
if(itensVenda.get(i).getQtd() > 1)
{
valorTotal -= (itensVenda.get(i).getPreco() / itensVenda.get(i).getQtd());
double precototal = itensVenda.get(i).getPreco();
double unit = precototal / itensVenda.get(i).getQtd();
System.out.println("Unidade: "+unit+"\nTotal: "+precototal);
itensVenda.get(i).setPreco(itensVenda.get(i).getPreco() - (itensVenda.get(i).getPreco() / itensVenda.get(i).getQtd()));
itensVenda.get(i).setQtd(itensVenda.get(i).getQtd() - 1);
recarregarTabela();
}
else if(itensVenda.get(i).getQtd() <= 1)
{
valorTotal -= itensVenda.get(i).getPreco() / itensVenda.get(i).getQtd();
itensVenda.remove(i);
recarregarTabela();
}
}
}
}
function that resets the table with new information:
private void recarregarTabela()
{
if (itensVenda.size() == 0)
{
dtm.getDataVector().removeAllElements();
dtm.setRowCount(0);
lblTotal.setText("Total: R$" + String.valueOf(valorTotal));
}
else
{
dtm.getDataVector().removeAllElements();
dtm.setRowCount(0);
for (Item item : itensVenda)
{
Object[] vetor = new Object[3];
vetor[0] = item.getNomeProduto();
vetor[1] = item.getQtd();
vetor[2] = String.format("%.2f", item.getPreco());
System.out.println(item.getPreco());
dtm.addRow(vetor);
}
lblTotal.setText("Total: R$" + String.valueOf(valorTotal));
}
}
You dont have to rebuild whole model everytime a single row is deleted. As you already have index of clicked or selected row you can just remove it from model using removeRow(index) method. I suspect that dtm is a DefaultTableModel so just call dtm.removeRow(index) everytime you need to remove row from table
Okay so for my project in my CSC330 class I am supposed to use threads to quickly find the answer to queries in a massive data set. Each item in the array is a user and the string is a collection of sites they visited on a website identified by number.
Example (the array is String users[])
users[1] = "1 4 5 7" users[2] = "1 2 7 17 10" users[3] = "6"
The queries are:
are there more than ___ users who looked at X
What percent of users looked at X
Are there more users who looked at X than Y
How many users viewed X ____ number of times
What percent of users looked at X more than Y
there are about a million users in this array and we had to solve the following queries without threads on a smaller scale version of the data for testing. All of mine work. Now I need to switch to the massive text file and I need to use threads to increase the speed. I'm wondering where it would be beneficial and how I could implement these threads. I will supply my code for solving each query. I'm thinking that I could use multiple threads to go through parts of the array at the same time but I'm not sure how to execute this.
public boolean query1(String num, String pageName){
if(num == null){
return false;
}
else
{
int userCount = 0;
int pageNum = convert(pageName);
System.out.println(pageNum);
String pageNumString = Integer.toString(pageNum);
System.out.println(pageNumString);
for(int i = 0; i < users.length; i++ )
{
for(String entry : users[i].split(" "))
{
if(entry.equals(pageNumString))
{
userCount++;
break;
}
}
}
if(userCount > Integer.parseInt(num)){
return false;
}
else{
return true;
}
}
}
public float query2(String pageName){
int userCount = 0;
int pageNum = convert(pageName);
String pageNumString = Integer.toString(pageNum);
for(int i = 0; i < users.length; i++ )
{
for(String entry : users[i].split(" "))
{
if(entry.equals(pageNumString))
{
userCount++;
break;
}
}
}
float percentage = (userCount*100.0f)/users.length;
return percentage;
}
public boolean query3(String pageName, String pageName2){
int userCount1 = 0;
int userCount2 = 0;
String pageNumString = Integer.toString(convert(pageName));
String pageNumString2 = Integer.toString(convert(pageName2));
for(int i = 0; i < users.length; i++ )
{
for(String entry : users[i].split(" "))
{
if(entry.equals(pageNumString))
{
userCount1++;
break;
}
}
for(String entry : users[i].split(" "))
{
if(entry.equals(pageNumString2))
{
userCount2++;
break;
}
}
}
return userCount1 > userCount2;
}
public int query4(String pageName, int numTimes){
int userCount = 0;
String pageNumString = Integer.toString(convert(pageName));
for(int i = 0; i < users.length; i++ )//runs through each user
{ int pageCount = 0;
for(String entry : users[i].split(" "))// runs through each user's pages
{
if(entry.equals(pageNumString))
{
pageCount++; // each time page is found page count increments 1
}
} // once done running through user's pages
if(pageCount == numTimes){ // check if the number of pages is equal to the number given
userCount++; // if so increment userCount
}
}
return userCount;
}
public float query5(String pageName, String pageName2){
int viewedMore = 0;
int userCount1 = 0;
int userCount2 = 0;
String pageNumString = Integer.toString(convert(pageName));
String pageNumString2 = Integer.toString(convert(pageName2));
for(int i = 0; i < users.length; i++ )
{
for(String entry : users[i].split(" ")){
userCount1 = 0;
userCount2 = 0;
if(entry.equals(pageNumString))
{
userCount1++;
break;
}
}
for(String entry : users[i].split(" "))
{
if(entry.equals(pageNumString2))
{
userCount2++;
break;
}
}
if(userCount1 > userCount2){
viewedMore++;
}
}
return viewedMore*100.0f/users.length;
}
At the very least, in query3 and query5 you can spawn off threads for each of the two inner for-loops; there's no reason they must be run sequentially. And for any of the query functions, you can certainly split the array into sections. Since your data is growing large, that approach will most likely be faster than iterating over the data with the main thread.
I would suggest providing the threads contiguous segments (e.g. index 0 to N-1; N to N+N-1 etc). This previous StackOverflow answer provides good reasoning why such an approach is most efficient. Once you get something working, you can play around with the number of threads to optimize.
edit to address your comment below
One approach would be to implement the strategy pattern, such that each of your Query are interchangeable across a client, where an executeQuery() call is the interface method. Think of having a client's call look something like
interface Query {
boolean executeQuery();
}
// client code...
Client client = new Client(...);
client.setQuery(new Query3(String num, String pageNum));
client.query(); // calls query.executeQuery();
Within the concrete Query classes, you can define individual behaviors of what the threads would do. This is a rough example:
public Query3 implements Query {
Query3(String pageNum`, String pageNum2) {
this.pageNum1=pageNum1;
this.pageNum2=pageNum2;
}
boolean executeQuery() {
for(int i = 0; i < users.length; i++ ) {
SearchThread first = new SearchThread(pageNum1);
SearchThread second = new SearchThread(pageNum2);
first.run();
second.run();
}
first.join();
second.join();
return first.userCount > second.userCount;
}
SearchThread extends Thread {
String pageNumString;
int userCount;
SearchThread(String pageNumString) {
this.pageNumString=pageNumString;
}
public void run() {
// do your search over this segment of the array, adding up userCounts
}
Here is another StackOverflow question that describes how to run multiple threads over a single array, with some boilerplate code to use.
I got a really annoying problem with my realtime graph. It works really pretty with MpAndroidChart library, but I guess I'm running out of memory. Because my realtime lineChart-graph become slower the longer it's running, until it get stuck. I didn't find any solution that works.
This is my linecharts screenshot
This is my code to enter values into charts,
( test01 - test04 are possible solution from other posts, I tried )
private void addEntry(double[][] pts) {
// System.out.println("######### addEntry #########");
if (mLineChartObj != null) {
int anzLineChartObj = mLineChartObj.size();
// System.out.println("######### Anzahl LineChartObj: " + anzLineChartObj);
for (int i = 0; i < anzLineChartObj; i++) {
LineChartObject ob = mLineChartObj.get(i);
LineData data = ob.lineChart.getData();
if (data != null) {
data.addXValue((ob.set).getEntryCount() + "");
data.addEntry(new Entry((float) pts[ob.rootNodePos][0], (ob.set).getEntryCount()), 0);
// data.addEntry(new Entry((float) (Math.random() * 9) + 0.45f, (ob.set).getEntryCount()), 0);
}
// °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
if (data.getXValCount() > 1400) {
System.out.println("X-ValueCount: " + ob.lineChart.getXValCount() + " EntryValueCount: " + ob.set.getEntryCount() + " ###############################");
// Test 01 °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
// for (int j = 0; j < 400; j++) {
// data.removeXValue(0);
// ob.set.removeEntry(0);
// }
// Test 02 °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
// runOnUiThread(new Runnable() {
// #Override
// public void run() {
// ob.lineChart.clearValues();
// }
// });
// Test 03 °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
// for (int j = 400; j > 0; j--) {
// data.removeXValue(j + 999);
// ob.set.removeEntry(j + 999);
// }
// Test 04 °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
// data.removeXValue(0);
// ob.set.removeEntry(0);
System.out.println("X-ValueCount: " + ob.lineChart.getXValCount() + " EntryValueCount: " + ob.set.getEntryCount() + " ###############################");
}
// °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
// notify chart data have changed
ob.lineChart.notifyDataSetChanged();
// limit number of visible entries
float range = 200f;
// ob.lineChart.setVisibleXRange(0f, range);
ob.lineChart.setVisibleXRangeMaximum(range);
// scroll the last entry
ob.lineChart.moveViewToX(data.getXValCount() - (range + 1f));
}
}
}
If I remove x-values from data or entries from set, several strange things happens. Sometimes the app gets crash, or the line of the charts begin to repeat itself, or sometimes I get more than one line per chart. I'm looking forward of a solution to remove old values from the chart.
EDIT 2: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The output of my linechart is like the following picture...It moves like a realtime graph, until xValues an amount of reach 1400, now my programm begins to remove entries and my linechart get stuck. After a while, the line begin to dissapear from the left side. At picture 1 it get stuck, at the next three pictures you can see how it dissapears.
I've been stuck with exactly the same problem today as I'm writing an app that visualizes sensor data. I just came up with the following makeshift solution that works for me:
// (following code snippet is from an event listener)
final ArrayList<Float> values = event.getValues();
final LineData data = mChart.getData();
// add data entry
if (mIndex < MAX_PLOT_SIZE) {
maxValue = 0f;
data.addXValue("" + mIndex);
for (int i = 0; i < values.size(); i++) {
final Float val = values.get(i);
final LineDataSet set = data.getDataSetByIndex(i);
set.addEntry(new Entry(val, mIndex));
}
mIndex++;
} else {
// shift values down
for (int i = 0; i < values.size(); i++) {
final LineDataSet set = data.getDataSetByIndex(i);
final List<Entry> yVals = set.getYVals();
final int size = yVals.size();
for (int j = 0; j < size - 1; j++) {
yVals.get(j).setVal(yVals.get(j + 1).getVal());
}
// add new value
final Float val = values.get(i);
yVals.get(size - 1).setVal(val);
}
}
It's admittedly a hack and there should be an easier way to do it, but at least it gets the job done. I guess you could try and adjust the xVal-Strings in the same manner ...
I am trying to read the object through xml using JAXB and updating some info the saving back.
here is the read and update code-
dbf = DocumentBuilderFactory.newInstance();
dbuilder = dbf.newDocumentBuilder();
document = dbuilder.parse(file);
jc = JAXBContext.newInstance(OutletWisePlanningListContainer.class);
Binder<Node> binder = jc.createBinder();
owpLContainer = (OutletWisePlanningListContainer)
binder.unmarshal(document);
owpLContList = owpLContainer.getOwpList();
then - updating the objects
for (OutletWisePlanningList owpl1 : owpLContList) {
owpl = owpl1.getOwpList();
owpList = new OutletWisePlanningList();
owpList = owpl1;
skuList.add(owpl1.getSkuId());
for (i = 1; i < sku; i++) {
if (owpl1.getSkuId().trim().equals(request.getParameter("skuId" + i).trim())) {
owpl1.getSkuId();
ArrayList idList = new ArrayList();
int j = 1, cnt = 1;
al1.clear();
int perf = Integer.parseInt(request.getParameter("hdnPerf" + i));
for (OutletWisePlanning owps : owpl) {
owp = new OutletWisePlanning();
for (j = 1; j < perf; j++) { //used only when Planned outlets is changed
if (owp.getMarketIntId().trim().equals(request.getParameter("UID" + i + '-' + j).trim()) && !request.getParameter("txtTARGET_SETvolume" + i + '-' + j).trim().equals("0")) {
//some updation in owp object
al1.add(owp);
}
}
if (request.getParameter("selPlanUnplanned").equals("0")) { //used when Unplanned in selected
al1.add(owp);
}
}
owpList.setOwpList(al1);
owpList.setSkuId(owpl1.getSkuId());
}
}
owpList1.add(owpList);
Iterator itr = owpList1.iterator();
while (itr.hasNext()) {
OutletWisePlanningList op1 = (OutletWisePlanningList) itr.next();
for (OutletWisePlanning op2 : op1.getOwpList()) {
System.out.println("Party id in the owpList1" + op2.getPartyId());
}
}
}
al.addAll(owpList1);
And the xml from where i am reading data and updating the same-
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<OutletWisePlanningListContainer>
<OutletWisePlanningList skuId="4214">
<OutletWisePlanning partyId="14560">
<marketIntId>14002</marketIntId>
<perfAllocId>818</perfAllocId>
<relPartyName>Aangan (Indian North)</relPartyName>
<addrLine1>Bhikaji Cama Place</addrLine1>
<locationName>-NA-</locationName>
<planQty>30</planQty>
<planCpc>10</planCpc>
<agreedQty>0</agreedQty>
<agreedCpc>0</agreedCpc>
<executedQty>0</executedQty>
<executedCpc>0</executedCpc>
<actualQty>0</actualQty>
<actualCpc>0</actualCpc>
<claimedQty>0</claimedQty>
<claimedCpc>0</claimedCpc>
<genCpc>0</genCpc>
<eligibility></eligibility>
<eligibleItem>0</eligibleItem>
</OutletWisePlanning>
</OutletWisePlanningList>
<OutletWisePlanningList skuId="4215">
<OutletWisePlanning partyId="14554">
<marketIntId>14105</marketIntId>
<perfAllocId>819</perfAllocId>
<relPartyName>Dhaba (Indian North)</relPartyName>
<addrLine1>Aurangzeb Road</addrLine1>
<locationName>Aurangzeb Road</locationName>
<planQty>44</planQty>
<planCpc>10</planCpc>
<agreedQty>0</agreedQty>
<agreedCpc>0</agreedCpc>
<executedQty>0</executedQty>
<executedCpc>0</executedCpc>
<actualQty>0</actualQty>
<actualCpc>0</actualCpc>
<claimedQty>0</claimedQty>
<claimedCpc>0</claimedCpc>
<genCpc>0</genCpc>
<eligibility></eligibility>
<eligibleItem>0</eligibleItem>
</OutletWisePlanning>
</OutletWisePlanningList>
I am getting the xml with different skuId(i.e. ok) but partyId = 14560 is replacing by partyId = 14554 with its whole data.
Please help
This code has lot of unnecessary initializations and assignments. You should have to clear all those.
However to answer your question, my best guess is that the issue is with this for loop.
for (OutletWisePlanning owps : owpl) {
owp = new OutletWisePlanning();
for (j = 1; j < perf; j++) { //used only when Planned outlets is changed
if (owp.getMarketIntId().trim().equals(request.getParameter("UID" + i + '-' + j).trim()) && !request.getParameter("txtTARGET_SETvolume" + i + '-' + j).trim().equals("0")) {
//some updation in owp object
al1.add(owp);
}
}
if (request.getParameter("selPlanUnplanned").equals("0")) { //used when Unplanned in selected
al1.add(owp);
}
}
First you are initializing the owp inside the for-each loop. owp = new OutletWisePlanning(); This is never been assigned any values still you are using it on the if condition. Here either the code is incomplete or you might have some initializations inside the constructor. Anyway within the inner for loop the "SAME OBJECT REFERENCE" gets modified and added to the list al1 when the if conditions(both inside and outside the loop) satisfies. So no matter how many times you add the object to this list all will have the same value.