I need to extract data from a Java web application. To be specific I am looking to extract real time stock data from yahoo market tracker. Can anyone please suggest any method?
I'm not sure you can extract the data from Yahoo Market Tracker. Even if you can, you might not be allowed to - I can't see any obvious terms & conditions/licensing. I think (although I could be wrong, anyone got better info?) that you'll need to pay to get access to an API providing near realtime market data.
There is a HTTP-based Yahoo Stock Quote API you could use to get prices, described here. Very simple, returns a comma-separated list of attributes for one or more stock symbols, for example:
http://finance.yahoo.com/d/quotes.csv?s=MSFT&f=snd1l1yr
It might not be realtime enough, but it might be the best you can do for free.
You can use glorious HTTP protocol to do that. Use any language you are comfortable with (Java, C#, VB.NET, python, ruby, php) and crawl the website you are trying to get information from.
I need to extract data from a Java web
application
From your standpoint, the fact that it is a Java Webapp or a PHP-one or static html pages doesn't change anything. It is not because Java is backing the webapp that suddenly you get a "Java-way" to extract the info.
Now in some cases there are APIs provided allowing you to interact with the data present on the website: but once again the fact that the Webapp is a Java one or not bears no importance.
Related
I've got a client who uses Quickbooks Online for accounts, and he wants to be able to read data from it programmatically. We're using Clojure, so any solution in Java will work, or http gets etc can be made directly if necessary.
They've got what appears to be a nice RESTFUL interface to their stuff, and a java library for accessing it, but I can't make head or tail of their documentation: https://developer.intuit.com/docs?redirectid=accounting, which all seems to be about webapps and OAuth and other stuff.
All I want to be able to do is get, say, a single customer record.
Can anyone point me to the simplest possible Hello World type program, in any language? (Preferably Java or something easy to read like python)
I'd imagine that what I'm looking for would look something like:
import quickbooksapi
username='fluffy'
password='doom'
cus=quickbooksapi.get_customer(username,password,id=4)
print(cus)
or something?
Or have I just got the wrong end of some gigantic stick here?
This looks the best available documentation:
https://developer.intuit.com/hub/blog/2016/04/25/quick-start-to-quickbooks-online-rest-api-with-oauth1-0
It's a recent blog post by a Quickbooks developer, showing how to get Oauth keys and then use curl to access the REST API.
It seems that you have to pretend to make a SaaS app in order to get one bit of gubbins and then there's another thing where you can get the rest of the gubbins.
After that you can use curl, putting all the gubbins in the headers. (The postman extension for chrome that he uses can generate curl commands and equivalents in many other languages)
It works exactly as advertised (6th August 2016).
That's enough. I can take it from there.
I'm not sure where to ask this so please suggest if I should move this to another stackexchange site
What technologie(s) are sites such as www.jsfiddle.net using in order to store data in the cloud ? Specifically the ability to store a code snippet and then retrieve it later via a URL. Is there a public generic API available for this functionality ?
I've researched cloud offerings such as https://developers.google.com/appengine/docs/java/googlestorage/ but since jsfiddle is free I doubt it is using such a service ?
Via the jsFiddle about page:
jsFiddle’s hosting is kindly provided by DigitalOcean.
There is no such thing as a free lunch: someone has to pay for storage. In this case, it's paid for by a third party, not by jsFiddle developers.
As to what they are using to store their data, it's unclear (though you can ask the developers: their names and websites are listed on that page), but the choice will basically be one of the following:
run a distributed file system on VMs rented from a cloud provider, or
use a managed solution, such as Google Cloud Storage that you pointed to, with an API
Either way, you will incur some costs.
I've been programming in java for a little while and I've found no real way to even come close to this goal. My googling has been pretty fruitless as well.
I'm looking for a way to essentially download current weather (or other but weather is a good start I suppose) and save the current temp / humidity / dewpoint / next day forecast for those numbers into an array of strings
I have no idea where to start, but I figure that this will be a good place to start learning how to use java to fetch.
Thanks!
How would you approach this task in other language?
In the case of weather you would probably look for some API exposed by the site you're trying to get the weather from.
Here come some clues:
1. If you want to just issue an HTTP request, get a result (kind-of ajax style) and parse the web page you can use java.net package or if you want a (much more powerful) thirdparty lib, use Apache HTTP Client.
2. If you're looking for API exposed via WebServices (which I believe is a better approach here) then they're language agnostic, so you just turn to web services (SOAP/Rest) in Java just like in any other language.
I know, the answer is a little bit 'common', so please clarify 'how' are you planning to solve this issue (even in any other language)...
Hope, this helps
A good source for weather information is METAR. There is also a Java library jweather available which should encapsulate all network/protocol/api issues to a limited set of methods to retrieve the required weather information
I like to access some data from web pages that are arranged like a catalog/shop from an android app.
For a concrete example: This is the URL for Amazons listing on Mark Twains books:
http://www.amazon.com/s/ref=nb_sb_noss/180-5768314-5501168?url=search-alias%3Daps&field-keywords=mark+tain&x=0&y=0#/ref=nb_sb_noss_1?url=search-alias%3Daps&field-keywords=mark+twain&rh=i%3Aaps%2Ck%3Amark+twain
1) If I have the above URL how do I obtain e.g.
the number of entries and
for each entry the line with the title (and maybe the image)? Which probably includes how to iterate through all the follow-up pages and access each entry.
What is the best (correct + compatible + efficient) way to do this?
I got the impression that jquery might be of use. But so far my knowledge of HTML and Javascript is just about basic.
2) How to query for the URL for all of Mark Twains books?
3) Any suggested readings for this and similar kind of topics?
Thanks for your time and have a good day!
Thomas
You would be very well advised to not "screen scrape" other web sites. Besides being difficult to maintain (as the web site changes, etc.) - this will actually be against the terms of use / service (TOS) for many web sites.
Instead, see if the desired web sites offer a web service that you can use. These will return data in a much more consumable format, such as JSON or XML. You'll usually also get your own developer key (to track requests against), as well as other possible features that you wouldn't get if going directly against the HTML.
Amazon, in particular, certainly offers this. See https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html for details. (Don't be confused by the naming of "advertising".)
I'm writing a Java program, and I want a function that, given a string, returns the number of Google hits a search formed from that query returns. How can I do this? (Bonus points for the same answer but with Bing instead.)
For instance, googleHits("Has anyone really been far even as decided to use even go want to do look more like?") would return 131,000,000. (or however many there are.)
Related: How can I programmatically access the "did you mean" suggestion? (eg searching "teh circuz" returns "did you mean the circus?")
found it: http://code.google.com/apis/ajaxsearch/documentation/#fonje
The Google Terms of Service say this:
5.3 You agree not to access (or attempt to access) any of the Services
by any means other than through the
interface that is provided by Google,
unless you have been specifically
allowed to do so in a separate
agreement with Google. You
specifically agree not to access (or
attempt to access) any of the Services
through any automated means (including
use of scripts or web crawlers) and
shall ensure that you comply with the
instructions set out in any robots.txt
file present on the Services.
Google has ways of making life unpleasant for you / your company if you violate the Terms of Service ...
UPDATE: The second sentence is about the way that you use Google's services ... including their published APIs. It is not entirely clear from the wording what is allowed and what is forbidden; literally speaking "any automated means" is very broad. However a Java app that performed Google searches, screen-scraped the results and repackaged them to provide some value added service would (IMO) be a violation of the TOS. And using Google's published APIs to do the same thing would (IMO) also be a violation.
But that's my opinion, not Google's. And it is the Google opinion that matters. If anyone is thinking of doing something like this, they should contact Google and check that what they are proposing is OK.
The point is that Google is not going to assist people to subvert their search business model. Anyone who thinks they can get away with it based on some clever interpretation of the TOS is going to get burned.
for the first part of the answer, try read the t-o-s; for the "did you mean" part, see: http://norvig.com/spell-correct.html
You may be able to do it "legally" using the Google Java Client Library. I don't know for sure, but they may have some methods similar to what you're looking for, and you won't be violating their TOS.
Google Data APIs Library
You can legally access the Google AJAX Feed API through its RESTful interface:
http://code.google.com/apis/ajaxfeeds/documentation/#fonje
Bing still has a developer program where you can call against their API in a JSON/XML or SOAP matter:
http://www.bing.com/developers