Develop Reference

Java - Extracting plaintext from web-page source code (getting massive quantities of lyrics from website)

O community, I'm in the process of writing the pseudocode for an application that extracts song lyrics from a remote host (web-server, not my own) by reading the page's source code.
This is assuming that:
Lyrics are being displayed in plaintext
Portion of source code containing lyrics is readable by Java front-end application
I'm not looking for source code to answer the question, but what is the technical term used for querying a remote webpage for plaintext content?
If I can determine the webpage naming scheme, I could set the pointer of the URL object to the appropriate webpage, right? The only limitations would be irregular capitalization, and would only be effective if the plaintext was found in EXACTLY the same place.
Do you have any suggestions?
I was thinking something like this for "Buck 65", singing "I look good"
URL url = new URL(http://www.elyrics.net/read/b/buck-65-lyrics/i-look-good-lyrics.html);
I could substitute "buck-65-lyrics" & "i-look-good-lyrics" to reflect user input?
Input re-directed to PostgreSQL table
Current objective:
User will request name of {song, artist, album}, Java front-end will query remote webpage
Full source code (containing plaintext) will be extracted with Java front-end
Lyrics will be extracted from source code (somehow)
If song is not currently indexed by PostgreSQL server, will be added to table.
Operations will be made on the plaintext to suit the objectives of the program
I'm only looking for direction. If I'm headed completely in the wrong direction, please let me know. This is only for the pseudocode. I'm not looking for answers, or hand-outs, I need assistance in determining what I need to do. Are there external libraries for extracting plaintext that you know of? What technical names are there for what I'm trying to accomplish?
Thanks, Tyler

This approach is referred to as screen or data scraping. Note that employing it often breaks the target service's terms of service. Usually, this is not a robust approach, which is why API-like services with guarantees about how they operate are preferable.
Your approach sounds like it will work for the most part, but a few things to keep in mind.
If the web service you're interacting with requires a very precise URL scheme, you should not feed your user-provided data directly into it, since it is likely to be muddied by missing words, abbreviations, or misspellings. You might be better off doing some sort of search, first, and using that search's best result.
Reading HTML data is more complicated than you think. Use an existing library like jsoup to assist you.

The technical term to extract content from a site is web scraping, you can google that. There are a lot of online libraries, for java there is jsoup. Though its easy to write your own regex.
1st thing I would do i use curl and get the content from the site just for testing, this will give you a fair idea of what to do.

You will have to use a HTML parser. One of the most popular is jsoup.
Take care abut the legal aspect fo what you you do ;)

Rapid Miner 101

I'm back with a question. I'm playing with Rapid Miner for automatic text classification and cant get it work. I'm getting an error that says, "no example set in the example, offending operator Performance ". Any idea what that is referring to ?

In RapidMiner you have to use the converter components before using it as example sets. So, if you have an output as 'doc', for example, you have to use the component 'Documents to Data' in order to link it to the next input 'exa'. That´s all!

Could you provide more details about your RapidMiner text mining process?
Without more context, your question is difficult to answer.
For more help with RapidMiner, you may want to check out the RapidMiner user forum: http://forum.rapid-i.com/
At RapidMiner Resources, you can find RapidMiner tutorial videos about how to text mining with RapidMiner:
http://rapidminerresources.com/index.php?page=text-mining-3
Rapid-I also offers a 90 minutes text mining webinar. You can find it at the Rapid-I web page under "services" and "training" or in the web shop.
I hope these links help you to get started with automatic text classification with RapidMiner. If you provide more details about your RapidMiner text mining process, I may also be able to directly answer your question.

If it says that there is no Example Set, then the issue is probably with your original data. Can you post an image of your process?
For instance, make sure that you have connected the initial input to your operator - what two operators does the error occur at?
One thought: the example set in text mining is usually your document collection, but if you are really using documents (PDF, Word) then your format will be Documents (Doc), and you may need to transform your documents to data (Documents to Data operator). Then you should have an Example Set that you can feed into your Performance operator.
Hope this helps - as the earlier comment said, without knowing the process, it is hard to tell exactly where the error is.

Java implementation for LDPC codes

Is there any open source Java implementation for LDPC (Low Density Parity Check) codes, I found only MATLAB codes.
My scenario is I will take text file and divide into block and I will delete some data in text file, and by using LDPC codes I need to recover data from text files.
Thanks.

I haven't tried this but the code here should get you started
http://www.cs.utoronto.ca/~radford/ftp/LDPC-2006-02-08/install.html
http://www.cs.utoronto.ca/~radford/ftp/LDPC-2006-02-08/examples.html
It's in C though. Might be easy to port. Or not.

I'd suggest looking into ways of calling matlab functions in java. I know there are a couple. Also why LDPC? While its one of the best FEC, it involves lots of matrix manipulation if I recall correctly. This is stuff much better suited for mat[rix]lab. The right tool for the right job...

There are also these two pure Java implementations:
https://github.com/a4a881d4/ldpc-java
https://github.com/pierroweb/LDPC-correcting-codes
I haven't tested them and would appreciate feedback from anyone else that has.
There's also a Java wrapper around a C++ library: http://cpham.perso.univ-pau.fr/MULTICAST/Java_wrapper_for_LDPC.html
Not the most promising results, but something to start from, at the very least.

How to webscrape scholar.google.com in Java?

I want to write a Java func grabTopResults(String f) such that grabTopResults("automata theory") returns me a list of the top 100 cited papers on scholar.google.com for "automata theory".
Does anyone have suggestions for what libraries will make my life easy?
Thanks!

As I'm sure Google can afford the bandwidth, I'll ignore the question of whether this is immoral/illegal/prohibited by Google's T&C
First thing you need to do is figure out what HTTP request (or requests) you need to issue in order to obtain the page with the data you need. Once you've figured this out, use HttpClient to issue the same request from Java code. The previous link shows example code that explains how to do this.
Once you've downloaded the content of the relevant page, you'll need to use a HTML parser to extract the data you're interested in. The Jericho parser suggested by peperg is a good choice.
If the Google police come knocking, you've never heard of me, OK?

I use http://jericho.htmlparser.net/docs/index.html . Google Scholar doesn't have API ( http://code.google.com/p/google-ajax-apis/issues/detail?id=109 ). Of course it is not allowed by Google (read terms of use. Automatic requestr are forbidden).

Below is a bit of example code which gets the titles on the first page using the open source product TestPlan. It is a standalone product, but if you really need it I could help you integrated it into your Java code (it is written in Java itself).
GotoURL http://scholar.google.com/
SubmitForm with
%Params:q% automate theory
end
set %Items% as response //div[#class='gs_r']
foreach %Item% in %Items%
set %Title% as selectIn %Item% h3
Notice %Title%
end
This produces output like the below (my IP is Germany, thus a german response). Obviously you could format it however you like, or write it to a file; this is just a rough test.
00000000-00 GOTOURL http://scholar.google.com/
00000001-00 SUBMITFORM default
00000002-00 NOTICE [ZITATION] Stochastic complexity in statistical inquiry theory
00000003-00 NOTICE AUTOMATED THEORY FORMATION IN MATHEMATICS1
00000004-00 NOTICE Constraint generation via automated theory formation
00000005-00 NOTICE [BUCH] Automated theorem proving: after 25 years
00000006-00 NOTICE [BUCH] Introduction to the Theory of Computation
00000007-00 NOTICE [ZITATION] Computer-controlled systems: theory and design
00000008-00 NOTICE [BUCH] … , randomness & incompleteness: papers on algorithmic information theory
00000009-00 NOTICE [BUCH] Automatic control systems
00000010-00 NOTICE [BUCH] VLSI physical design automation: theory and practice
00000011-00 NOTICE Singular Control Systems.

mapping data in properties files

I have the following data:
User System SubSystem
user1 System1 SubSystem1
user2 System1 SubSystem2
user3 N/A N/A
and i need to be able to determine the system/subsystem tuple from the user. I must be able to add users at any time without rebuilding and redeploying the system.
I know the database would be the best option here but I cannot use a database table.
I currently have it mapped using a hash map but I don't want it to be hard-coded. I was thinking about using a properties file but I can't visualize how I would implement it. Anyone else have any suggestion?
Not that it matters but I'm using JAVA, on weblogic 10.3.

You could do this using a HashMap (as you do now) and store it using XStream.
XStream allows you to serialise/deserialise Java objects to/from readable/editable XML. You can then write this to (say) a filesystem, and the result is editable by hand.
The downside is that it's a serialisation in XML of a Java object, so not as immediately obvious as a properties file to edit. However it's still very readable, and easily understood by anyone remotely technical. Whether this is an appropriate solution depends on the audience of this file.

Sounds like something you could very well use YAML for..
SnakeYAML looks to be a workable Java implementation.

I would go for something as simple as :
user1 = userValue
user1.system = systemValue
user1.system.subsystem= subsystemValue
user2 = userValue
user2.system = systemValue
user2.system.subsystem= subsystemValue
user(id) is used as "primary" key in your properties, and a very simple concatenation of your fields to store your table values.
I use this very often : trust me, it's much more powerfull than it may appear :)

For this project i've gone with the solution proposed by Olivier. Some project contrainst (legacy of the project) prevent me for going with a probably better solution of using XStream.
Thx for the feed back guys