Folks,
I was looking for Java capability of reading thru the network tab.
I had a task of hitting a URL in the browser, reading and validate some tags which are fired on page load.
I Know to fetch the View Source Data of the page, but not sure on getting network tab data.
example:
Opening a URL in chrome browser and watching Network tab in the developer tool, i see bunch of Get requests(tags,js etc) getting fired & need to capture this data.
I wanted to know which is the best way to achieve this and great if any code snippet is available.
Cheers
You can use HttpClient to work on api testing. There are many tutorials to work on api testing using Java. For starter you can read about Apache Http client.
Related
I'm looking for a java-framework which enables me to easily communicate with a website.
What I'd like to do is for example:
log into a website
open various pages
read information
submit information into forms
send ajax-requests
read ajax-response
What I'm not looking for is a browser automation plugin like selenium. I'm trying to have my application directly communicate with the website.
That's the general outline. If you can think of a better solution for the following problem, I'm more than willing to follow your advice (:
We're working with an webapplication with an gruesome GUI. Unfortunatley we've got no means to tinker with said application or request changes to it. What I'd ike to do is to build is a client which logs into said application, fetches the data and displays them in a more appropriate manner with additional information based on that data while also providing tools to process this data and submit it back to that web-application.
Thanks in advance.
Selenium does come for JAVA. You can download it from here. http://www.seleniumhq.org/download/
Here is a tutorial:
https://www.airpair.com/selenium/posts/selenium-tutorial-with-java
How Selenium web driver works
Selenium web driver (firefox web driver) will open a web browser(firefox) for you and you can actually see what's going on. The choice of opening a browser window may not be the requirement for you. Then you can use:
HTMLUnitWebDriver
PhantomJSDriverService
Take a look at
http://hc.apache.org/httpcomponents-client-ga/quickstart.html
Its not a framework but a library but should provide you the needed methods to interact with your web application
I want to write a client application for a site (e.g. to read smth from site, add some comments, likes etc). I haven't got access to site sources and there isn't any API for work with it. So at my Android application I decided to parse this site (it has static pages) using : JSOUP Library
And using this library I'm going to write unofficial, but API for my purposes to work with this site, and then use it in my Android application.
Can somebody tell me, is this good practice or there are better ways to do? Is this good idea at all to parse site in Android device.
As I wrote in comment - in general building your own application on top of the third party www service is not a good idea. If you want to do it anyway you have 2 options:
Use jSoup (or any other html parser if exists) and parse third party content on the device
Set up some middleware server to parse content and serve it in some more convenient way.
The second option has a little advantages - you can fix application without forcing users to update it and probably you'll save a bit of device's bandwidth. Of course disadvantage is that you have to pay for server.
General problem with applications like that is that every single change with layout, skin, server configuration can cause your application to stop working, as well as parsing html needs much more work that just connect to existing API.
More over - publishing your application can cause some legal issues (copyright) and is against Google Play's policy:
Do not post an app where the primary functionality is to: Drive
affiliate traffic to a website or Provide a webview of a website not
owned or administered by you (unless you have permission from the
website owner/administrator to do so)
We a have jboss server running and have a basic web setup using Spring but now I would like to stream video into the browser. I am trying to use xuggle but then I read on their website that you can't put it into an Applet or use it with java webstart, so I'm assuming that means I can stream to a browser, is this assumption correct? If so does anyone else know any libraries or how I should go about doing this? If you need any more info or have questions I'll be happy to answer although I'm very new to streaming video and relatively knew to Spring.
UPDATE: So I'm able to generate a video using xuggle, and then I can embed that video in my html code... but I can't generate and stream at the same time. Does anyone have any ideas how to make xuggle push date out to my tomcat server?
You may want to look at Red5 media server.
Since you are doing jsp, just have your code write out the HTML 5 video element. That will provide basic video functionality. This will work in all HTML 5 compliant browsers as well as all mobile devices.
I'm having an issue with a servlet that delivers multimedia data. I'm trying to use it from mobile safari on the iPhone and I get the "server not configured correctly" error. I server the same multi-media file using a web share on my Mac (Apache) and it works fine. I start looking at HTTP heades to see if the servlet is messing up a header. I notice that Apache sets an ETag as well as an Accept-Range. Then I Google around and learn that the iPhone works with progressive downloads. Long story short, I think I need to enable progressive downloads in my servleet and I think the ETag and Accept-Range headers are part of it. I've never done this before so I figured I'd ask here if anyone could point me in the right direction. Do I need to implement the entire progressive download stuff in my servlet? Should I use a Filter? Where should I start?
Take a look at this article. It goes over creating a file servlet that supports resuming the download, caching and gzipping the content.
I need to screen scrape some data from a website, because it isn't available via their web service. When I've needed to do this previously, I've written the Java code myself using Apache's HTTP client library to make the relevant HTTP calls to download the data. I figured out the relevant calls I needed to make by clicking through the relevant screens in a browser while using the Charles web proxy to log the corresponding HTTP calls.
As you can imagine this is a fairly tedious process, and I'm wodering if there's a tool that can actually generate the Java code that corresponds to a browser session. I expect the generated code wouldn't be as pretty as code written manually, but I could always tidy it up afterwards. Does anyone know if such a tool exists? Selenium is one possibility I'm aware of, though I'm not sure if it supports this exact use case.
Thanks,
Don
I would also add +1 for HtmlUnit since its functionality is very powerful: if you are needing behaviour 'as though a real browser was scraping and using the page' that's definitely the best option available. HtmlUnit executes (if you want it to) the Javascript in the page.
It currently has full featured support for all the main Javascript libraries and will execute JS code using them. Corresponding with that you can get handles to the Javascript objects in page programmatically within your test.
If however the scope of what you are trying to do is less, more along the lines of reading some of the HTML elements and where you dont much care about Javascript, then using NekoHTML should suffice. Its similar to JDom giving programmatic - rather than XPath - access to the tree. You would probably need to use Apache's HttpClient to retrieve pages.
The manageability.org blog has an entry which lists a whole bunch of web page scraping tools for Java. However, I do not seem to be able to reach it right now, but I did find a text only representation in Google's cache here.
You should take a look at HtmlUnit - it was designed for testing websites but works great for screen scraping and navigating through multiple pages. It takes care of cookies and other session-related stuff.
I would say I personally like to use HtmlUnit and Selenium as my 2 favorite tools for Screen Scraping.
A tool called The Grinder allows you to script a session to a site by going through its proxy. The output is Python (runnable in Jython).