I'm developing an android application which takes its information from a site that use JavaScript.
I want to run one of the java script function through my android app.
in example: this site http://www.bgu.co.il/tremp.aspx has "Next page" in the bottom (in hebrew) that javascript function do. i want to get to the next page through my app.
How can i send the site the command "move to next page" or activate button's onClick event?
EDIT: I'm taking information from this site, using an XML parser (sax parser), I want to get to the next page of this site in order to parse it either. I hope I made myself clear now
You really need to explain a little more fully...
Are you opening that page and parsing it in your code, have you embedded a WebView or are you just creating an Intent which opens that page in the user's preferred web-browser?
If the latter, then you definately cannot do what you're suggesting.
If you're using a WebView I'm pretty sure you still can't access the DOM of the page in the way you want to
That means your solution is to load and parse the webpage in your code - extract the 'next' page and then do with that whatever you wish...
Check out the Using JavaScript in WebView section in the Android Developer Guide
OK - now we know you're parsing the page I can try to answer your question!
You can't "click" something on a parsed page because you have no Javascript engine - it's just an XML document - a collection of nodes made-up of text.
In this case you need to work-out how the paging system actually works - and looking at the page in something like Firebug makes that quite simple to do.
The paging system works through a FORM POST mechanism - when you click 'Next' or a page number like '2' it simply loads the same page from the server, but with POST variables set.
The key variable is "__EVENTTARGET" - it tells ASP which page to load (in Hebrew of course).
The snag is that you'll need to pass all the other POST variables from this page too - that's every INPUT with it's associated value. I notice there's one called "_EVENTVALIDATION" which appears to exist to stop people getting around this too-easily by just passing "_EVENTTARGET".
A tool like Fiddler would be useful here too - it will show what is POSTed to the server when you click NEXT - all you have to do is replicate that in your call to load the page again and you have the next page - and repeat that with each page until there's nothing left to load...
Alternatively, you could switch to using a WebView - loading the page into that would mean it does have a Javascript engine and you could automate the 'clicking' (although I don't know the specifics of how to do that, the other answers allude that it's possible?) ?
Related
It kinda says it all in the title, I'm really new to programming so its probably very simple. But I thought I should ask for help anyway just in case I don't figure it out.
All help is appreciated!
You probably want to automate browser interactions. There's multiple ways you can do this when it comes to forms, but here's 2 ways to start you off.
You can use the Requests library to send POST requests to the server. To do this you would want to use the browser inspection tools to examine the POST request that is sent when you submit your form. You then can create a program to recreate that. This is a headless approach, meaning there is no browser involved that you can physically see your programming interacting with.
link: https://requests.readthedocs.io/en/master/
Method number 2 involves writing a program that uses a library that physically interacts with a browser. For example automates your mouse movements, page scrolls, and key presses into selected inputs on the page (in your case form input fields). One of the most popular libraries to accomplish this is Selenium. To use selenium you run an instance of your browser (Firefox and Google Chrome are supported and well documented) and then you would write code to automate visiting the forms page, selecting each form field, typing the data in the fields, and then submitting the form. To figure out how to access each area of the website form, you would want to use your browser inspection tools (Firefox browser inspection tools are better than Chrome's in my opinion) and you can figure out what each field is referred to by in the html that is used to build the page... for example upon inspecting the Name field of the form, you may find something like <form><input "id=name-field>Type your name here </input></form> in the html. You would then use a Selenium method such as driver.find_element_by_id('name-field') to access the element. You could set that element to a variable like this: name = driver.find_element_by_id('name-field') and then use a line like this name.send_keys("Billy Bob") to have Selenium type "Billy Bob" into the name field.
To have a button be clicked you simply map the button to a variable like this button = driver.find_element_by_id('button-id') and then you would do this button.click(). Note that "driver" in the above example refers to the instance of the web browser that you are automating and is created at the start of the program.
link: https://selenium-python.readthedocs.io/index.html
Method #2 is probably the route you want to take as a beginner. I hope that helps get you started.
I want to build a Java application through which a user can request a web page, which loads in inside the Java application (not in a browser).
After the page loads, the user can select whichever elements they like on the page, and I want to track which elements they click through the DOM.
I.E. the user clicks the image of a product, then I want to get that particular element of the DOM, with all the attributes such as src, class, id, etc.
I'd like to know if any frameworks get close to doing anything like this. Especially the clicked elements, since the web page I am pretty sure there are quite a few ways to load in inside a Java application.
I don't know if this is what you have in mind, but you can try to take a look at HtmlUnit http://htmlunit.sourceforge.net/gettingStarted.html . It is said that it is used for unit tests, but it can be used also for other purposes; one of the descriptions I found is:
"A java GUI-Less browser, which allows high-level manipulation of web pages, such as filling forms and clicking links; just getPage(url), find a hyperlink, click() and you have all the HTML, JavaScript, and Ajax are automatically processed."
This API returns a whole HTML table. I'm searching how to add this table (as is) into my UI but I've never seen such API throwing HTM table.Browsing Internet for an answer is not giving me any hope either.
Is it possible to put it into a webview? or any other UI object? My application sends a word to the API, and I'm getting the table in return.
I'd appreciate some code example.
You can certainly just show that exact same page in a WebView. If you want to parse the table and display only certain information, there is a library call JSOUP that is available which makes it very convenient to parse HTML.
It looks like you don't mind displaying the whole thing in a WebView - if that is acceptable, then you just load the page into a WebView widget. WebView will take care of rendering the page exactly as you see it in a browser. You only have to tell it what to load.
You parse the output like you would any other web request. If you wanted to include the table in your own webpage, you could. Or you could parse the response for the specific info you need.
Don't think of it as an API, think of it as a URL you're requesting and now you need to do something with the contents. That might help with your Googling. You're essentially doing page scraping.
I am trying to scrape a website, using Web Client, i am able to get the data on the first page and parse it, but I do not know how to read the data on the second page, the website is calling a java script to navigate to the second page. Can anyone suggest me how do I get the data from the next pages?
Thanks in advance
The problem you're going to have is while you (a person) can read the JavaScript in the first page and see it is navigating to another page, having the computer do this is going to be hard.
If you could identify the block of code performing the navigation, you would then need to execute it in such a way that allowed your program to extract the URL. This again is going to be very specific to the structure of the JavaScript and would require a person to identify this.
In short, I think you're dead in the water with this one, though it serves as a good example of why the Unobtrusive JavaScript concept is so important.
This framework integrates HtmlUnit with its headless javascript enabled browser to fully support scriping multiple pages in the same WebClient session: https://github.com/subes/invesdwin-webproxy
I need to scrape French court cases for a project, but I can't figure out how to get Java to navigate the Court's search engine.
Here's the search page I need to manipulate. I want to start scraping the results page, but I can't get to that page from Java with just the URL. I need some way to have Java order the server to execute a search based on my date parameters (01/01/2003 - 30/06/2003), and then I can run the show by simply manipulating the URL I'm connecting to.
Any Suggestions?
First make sure the terms of service for the site allow this.
I would httpclient posts to send the data and get the results. See the form on the page, figure out which variables you need to emulate and submit them with httpclient. You should get back the results you are looking for. Also this page has lots of javascript, so you need to figure out what it is doing, maybe its never submitting the form but making ajax calls to update the page, but maybe you can get the same results.
You can always install something like "fiddler" and watch the http traffic the page is sending and then emulate that using httpclient.