Develop Reference

How can I efficiently extract text from bunch for web pages without extra information

I have list of webpages around 1 million, I want to efficiently just extract text from those pages. Currently I am using BeautifulSoup library in python to get text from HTML and using request command to get html of a webpage. This approach extract some extra information in addition to the text like if any javascript is listed in body.
Could you please suggest me any suitable and efficient way to do the task. I looked at scrapy but it looks like it crawls specific website. Can we pass it list of specific webpages to get information from ?
Thank you in advance.

Yes, you can use Scrapy to crawl a set of URLs in a generic fashion.
You simply need to set them on the start_urls list attribute of your spider, or reimplement the start_requests spider method to yield requests from any data source, and then implement your parse callback to perform the generic content extraction you want.
You can use html-text to extract text from them, and regular Scrapy selectors to extract additional data like the one you mention.

In scrapy you can set up your own parser. E.g. Beautiful soup. This parser you can call from your parse method.
To extract text from generic pages I traverse the body only, exclude comments etc and some tags like script, style, etc:
for snippet in soup.find('body').descendants:
if isinstance(snippet, bs4.element.NavigableString) \
and not isinstance(snippet, EXCLUDED_STRING_TYPES)\
and snippet.parent.name not in EXCLUDED_TAGS:
snippet = re.sub(UNICODE_WHITESPACES, ' ', snippet)
snippet = snippet.strip()
if snippet != '':
snippets.append(snippet)
with
EXCLUDED_STRING_TYPES = (bs4.Comment, bs4.CData, bs4.ProcessingInstruction, bs4.Declaration)
EXCLUDED_TAGS = ['script', 'noscript', 'style', 'pre', 'code']
UNICODE_WHITESPACES = re.compile(u'[\t\n\x0b\x0c\r\x1c\x1d\x1e\x1f \x85\xa0\u1680\u2000\u2001\u2002\u2003\u2004'
u'\u2005\u2006\u2007\u2008\u2009\u200a\u2028\u2029\u202f\u205f\u3000]+')

How to replace a query string in an Apache Velocity template?

In my web application I'm trying to prevent users from inserting JavaScript in the freeText parameter when they're running a search.
To do this, I've written code in the header Velocity file to check whether the query string contains a parameter called freeText, and if so, use the replace method to replace the characters within the parameter value. However, when you load the page, it still displays the original query string - I'm unsure on how to replace the original query string with my new one which has the replaced characters.
This is my code:
#set($freeTextParameter = "$request.getParameter('freeText')")
freeTextParameter: $freeTextParameter
#if($freeTextParameter)
##Do the replacement:
#set($replacedQueryString = "$freeTextParameter.replace('confirm','replaced')")
replacedQueryString after doing the replace: $replacedQueryString
The query string now: $request.getQueryString()
The freeText parameter now: $request.getParameter('freeText')
#end
In the code above, the replacedQueryString variable has changed as expected (ie the replacement has been carried out as expected), but the $request.getQueryString() and $request.getParameter('freeText') are still the same as before, as if the replacement had never happened.
Seeing as there is a request.getParameter method which works fine for getting the parameters, I assumed there would be a request.setParameter method to do the same thing in reverse, but there isn't.

The Java String is an immutable object, which means that the replace() method will return an altered string, without changing the original one.
Since the parameters map given by the HttpServletRequest object cannot be modified, this approach doesn't work well if your templates rely on $request.getParameter('freeText').
Instead, if you rely on VelocityTools, then you can rather rely on $params.freeText in your templates. Then, you can tune your WEB-INF/tools.xml file to make this parameters map alterable:
<?xml version="1.0">
<tools>
<toolbox scope="request">
<tool key="params" readOnly="false"/>
...
</toolbox>
...
</tools>
(Version 2.0+ of the tools is required).
Then, in your header, you can do:
#set($params.freeText = params.freeText.replace('confirm','replaced'))

I managed to fix the issue myself - it turned out that there was another file (which gets called on every page) in which the $!request.getParameter('freeText')" variable is used. I have updated that file so that it uses the new $!replacedQueryString variable (ie the one with the JavaScript stripped out) instead of the existing "$!request.getParameter('freeText')" variable. This now prevents the JavaScript from being executed on every page.
So, this is the final working code in the header Velocity file:
#set($freeTextParameter = "$!m.request.httpRequest.getParameter('freeText')")
#if($freeTextParameter)
#set($replacedQueryString = "$freeTextParameter.replace('confirm','').replace('<','').replace('>','').replace('(','').replace(')','').replace(';','').replace('/','').replace('\"','').replace('&','').replace('+','').replace('script','').replace('prompt','').replace('*','').replace('.','')")
#end

Velocity template localization

I'm working on a task that's using Velocity to generate an email from a template. One of my requirements is that the email message be localized, based on a Locale submitted by the user. I've got this working for the most part, using Velocity's ResourceTool to pull in a MessageBundle. The other requirement is that I can only have one template - I don't want one template per language or locale because this is difficult to maintain if changes are needed.
As a part of a MessageBundle, I can have what's referred to as "compound messages", which are messages that have variables that are replaced when the message is evaluated. For example:
dear.name=Dear {0}:
My question is this: how do I get Velocity to replace the {0} value with a name that I pass in?
It's not acceptable to simply modify my Velocity template so that I simply replace the name at the template level - in some languages, the greeting word "Dear" comes after the name, so this would yield an incorrect output.

Found the answer: in my velocity template, I can use the Velocity directive:
#evaluate(${msg.dear.name})
And in my message bundle, my message looks like:
dear.name=Dear $name
If I set the name in the Velocity context, it gets replaced as I expect.

constraints not being fully incorporated Play 2.1

Was wondering why the constraints created for a form are not included in the input tag directly when created through the form helper?
Explanation (using Play 2.1):
Model:
public class Account {
#MaxLength(5)
private String id = "";
...
...
view:
#form(action = routes.Application.addAccount()) {
#inputText(accountForm("id"), '_label -> "Enter your id:")
}
renders automatically in html as:
Enter your id:
Maximum length: 5
Should it not render like this (actually constraining the form text field):
Enter your id:
Maximum length: 5
How can I get code that will automatically include constraints such as these in the form? It's just that I do not really think it is a good idea to have a maxlength defined in the form model and a separate one defined in the view.
Thanks

If I've understood you correctly, it sounds like you're looking to implement one of these features:
Highlight an input text field that is overlength before form submission
Clipping text in an input field so that it does not go overlength
Play's HTML templating engine doesn't natively provide this kind of client-side instant form validation. This functionality needs to be implemented via JavaScript, and JavaScript generation is not really a concern for Play.
If you want to progressively enhance your form and provide client-side validation, you'll have to write the JavaScript yourself. Of course there are libraries that you can use to help you with this task. For example, if you are already using jQuery you can use its validation plugin.
As you've mentioned in your question, it would be better to have a maximum length limit declared in one place only, rather than duplicated in your client-side JavaScript code and your server-side Java code. As a suggestion, you could keep the limit declared in Java code, but introduce a new action in your controller tier that returns a JSON response containing this limit. This action could then be called via AJAX when loading your form page.
EDIT
Didn't know about the maxlength attribute, thanks Saad. If you feed in your maximum length limit as an input parameter to your template, you can populate an input element's maxlength attribute as follows:
#(accountForm: Form[Account], maxLength: Int)
...
#form(action = routes.Application.addAccount()) {
...
#inputText(
field = accountForm("id"),
args = '_label -> "Enter your id:", 'maxlength -> maxLength
)
...
}
...
There may be a more elegant way to pass maxLength into your HTML template (e.g use the HTTP context map, or have it as a public field on your Account form object). The above code snippet just demonstrates how to correctly generate the input text field once you can access it in the template.

Velocity - How to avoid ParseErrorException when using jQuery?

I'm trying to add a jQuery post to some JavaScript on a web page. The entire page is built up of several Velocity templates. Everything has been fine until I've tried to add the jQuery post, now I get:
org.apache.velocity.exception.ParseErrorException: Encountered "," at line 282, column 24 of /WEB-INF/velocity/www/comments.vm
Was expecting one of:
"(" ...
<RPAREN> ...
<ESCAPE_DIRECTIVE> ...
~~~snip~~~
Line 282 is $.post(... and column 24 appears to be the first "," character. Initially I had the JSON on this line, but I moved it up (to the var myJSONObject ... line)as I thought the error related to invalid JSON (tabs at the start of the line gave a misleading column number).
var myJSONObject = {"body": "", "action": "postcomment", "submitted": "true", "ajax": "true"};
myJSONObject.body = $("body").val();
$.post("$!{articleurl}", myJSONObject, function(result){
btn.textContent='Comment sent successfully.';
});
Minor Update
I changed the following lines:
var url = "$articleurl";
$.post(url, myJSONObject, function(result){
~~~snip~~~
The parse exception still focuses on the first ",". I'm assuming the issue is that Velocity thinks it should be able to resolve $.post - when in fact, it's jQuery. I've used jQuery in other Velocity VM templates without any problem. Is there a way to get Velocity to ignore certain lines / statements when parsing?
Update 2
I found this link about escaping references in Velocity, but it does not resolve my issue. Adding a "\" before $.post gives me the exact same error, but the column is one extra, because of the character added at the start of the line.

You can wrap your javascript with #[[ ... ]]# which tells Velocity to not parse the enclosed block (new in Velocity 1.7)
#[[
<script>
...
</script>
]]#

Ok, there appears to be two solutions for this:
First, with jQuery we can just avoid using the global alias $ and instead use the jQuery object directly:
jQuery.post(url, myJSONObject, function(result){
~~~snip~~~
In my case, the above works great. But I suspect in other scenarios (non-jQuery) this may not be possible. In which case, we can 'hide' our character within a valid Velocity reference like this:
#set( $D = '$' )
${D}
Source: http://velocity.apache.org/engine/devel/user-guide.html#escapinginvalidvtlreferences
I'd still like to know why the backslash escape didn't work, but the above will at least get me moving again. :)

I think this is a bug in version 1.6.x, because it works fine in 1.7(If it did not, please tell me, I test it many times..), according to the reference, the $ takes effect only when it is followed by a-zA-Z. I want to try do debug what happened really, but the translation code is generated by Java CC tool, it is too hard to recognize the logic...

you must create a js file with your javascript code
and import your js file into your vm code

I couldn't get it to work with any of the other fixes like escaping "$" in velocity unfortunately. I got it working by loading an external js-file with the jQuery instead of writing jQuery directly in velocity. Worked out for me at least, hope it helps someone :)
/björn