Develop Reference

Redisearch query with "begin with" instead of "contains"

I am trying to understand on how to perform queries in Redisearch strictly with "begins with" and I keep getting "contains".
For example if I have fields with values like 'football', 'myfootball', 'greenfootball' and would provide a search term like this:
> FT.SEARCH myIdx #myfield:foot*
I want just to get 'football' but I keep getting other fields that contain the word instead of beginning with that word.
Is there a way to avoid this?
I was trying to use VERBATIM and things like #myfield:^foot* but nothing.
I am using JRedisearch as a client but eventually I had to enter the DB and perform these queries manually in order to figure out what's happening. That being said, is this possible to do with this client at the moment?
Thanks
EDIT
A sample of my index setup:
Client client = new Client(INDEX_NAME, url, PORT);
Schema sc = new Schema().addSortableTextField("url", 1.0); // using this field for query
client.dropIndex(true);
client.createIndex(sc, Client.IndexOptions.Default());
return client;
Sample document:
id: // random uuid
urlPath: myfootbal
application: web
market: Europe

After checking the RDB provided I see that when searching foot* you are not getting myfootbal. The replies look like this: /dot-com/plp/football/x/index.html. You are getting those replies because this url is tokenized, and '/' is one of the tokenize chars. If you do not want those urls to be tokenized you need to declare them as TAGS and not as TEXT. This way the entire url will be indexed as is and when search for foot* it will not appear in the results.
For more information about TAGS see the FT.CREATE documentation: https://oss.redislabs.com/redisearch/Commands.html

How can I efficiently extract text from bunch for web pages without extra information

I have list of webpages around 1 million, I want to efficiently just extract text from those pages. Currently I am using BeautifulSoup library in python to get text from HTML and using request command to get html of a webpage. This approach extract some extra information in addition to the text like if any javascript is listed in body.
Could you please suggest me any suitable and efficient way to do the task. I looked at scrapy but it looks like it crawls specific website. Can we pass it list of specific webpages to get information from ?
Thank you in advance.

Yes, you can use Scrapy to crawl a set of URLs in a generic fashion.
You simply need to set them on the start_urls list attribute of your spider, or reimplement the start_requests spider method to yield requests from any data source, and then implement your parse callback to perform the generic content extraction you want.
You can use html-text to extract text from them, and regular Scrapy selectors to extract additional data like the one you mention.

In scrapy you can set up your own parser. E.g. Beautiful soup. This parser you can call from your parse method.
To extract text from generic pages I traverse the body only, exclude comments etc and some tags like script, style, etc:
for snippet in soup.find('body').descendants:
if isinstance(snippet, bs4.element.NavigableString) \
and not isinstance(snippet, EXCLUDED_STRING_TYPES)\
and snippet.parent.name not in EXCLUDED_TAGS:
snippet = re.sub(UNICODE_WHITESPACES, ' ', snippet)
snippet = snippet.strip()
if snippet != '':
snippets.append(snippet)
with
EXCLUDED_STRING_TYPES = (bs4.Comment, bs4.CData, bs4.ProcessingInstruction, bs4.Declaration)
EXCLUDED_TAGS = ['script', 'noscript', 'style', 'pre', 'code']
UNICODE_WHITESPACES = re.compile(u'[\t\n\x0b\x0c\r\x1c\x1d\x1e\x1f \x85\xa0\u1680\u2000\u2001\u2002\u2003\u2004'
u'\u2005\u2006\u2007\u2008\u2009\u200a\u2028\u2029\u202f\u205f\u3000]+')

REST services PATCH API example

I am trying to call the REST Webservices PATCH API, here is My JSON payload
[
{ "op":"replace", "path":"/values/Timestamp","value":"2016-10-28T15:25:43.511Z"},
{ "op":"replace", "path":"/values/Flag", "value":true },
{ "op":"replace", "path":"/values/Flow", "value":"Flow A"},
{"op":"replace", "path":"/values/Interests", "value":[ "Sports", "Book Reading" ] }
]
JSON Value attribute has different values with different data types. and I want to prepare Entity object(Java) and convert it into JSON and call REST end point.
now
I am not very sure
which is the best suitable data type I can choose for values attribute
I have referred following links but I didn't get enough details
Android REST API using PATCH method
https://www.rfc-editor.org/rfc/rfc5789#section-2.1
http://blog.earaya.com/blog/2013/05/30/the-right-way-to-do-rest-updates/
http://williamdurand.fr/2014/02/14/please-do-not-patch-like-an-idiot/
but I didn't get enough details.
any suggestion on this is really appriciated

Got the java object from the client and created another Java class with below properties and set the values
opn - string
path - String
value - Object
added above java objects to array list then used the GSON library to convert it into the array of JSON objects which will be accepted by patch api.
and please note the content type is application/json-patch+json

creating array in javascript from array object pass in by request.setAttribute in servlet

i'm trying to create a dynamic photo gallery which retrieve the photo's location from mySQL. Store the location to a photo object under the name 'private String location;'
There will be an ArrayList to hold all the different photos. After, the servlet will forward to a jsp page
request.setAttribute("list", list);
request.getRequestDispatcher("car.jsp").forward(request, response);
i have a java script for the photo gallery that takes in an array of, ["path_to_image", "optional_link", "optional_linktarget", "optional_textdescription"].
imagearray: [
["path_to_image", "optional_link", "optional_linktarget", "optional_textdescription"],
["a.jpg", "www.a.com", "", ""]
],
I would like to retrieve the location from the object in the list passed in from the servlet and convert it into the imagearray for my photo gallery to work.
I'm quite new to javascript and i've been looking around for similar example or tutorial but i couldn't find any relevant ones. Please help me out, thank you so much for your time.

what i get from your question is photo is an object of a class and location is a member variable of that class.
request.setAttribute("list", list);
request.getRequestDispatcher("car.jsp").forward(request, response);
is this list is a Arraylist of photo object or location member variable.
also you are setting attribute in java and you want that list to hold by javascript.
then in that case you can check JSON for holding your java object and to convert into javascript object.
you will get your string in JSON similar to
{imagearray:[{"path_to_image":"path_to_image","optional_link":"optional_link","optional_linktarget":"optional_linktarget","optional_textdescription"}]}

What you want to do can be simply achieved by the following sequence:
Get results from a database.
Create JSON object.
Set that object as request attribute.
Assign JSON to a JavaScript variable.
Now, let's carry on doing that list.
Get results from database
You should have a method of type getPhotoList() that returns List<Photo>. I suppose that your Photo class has the fields you'd like to export to JavaScript. In the end, you'll have List<Photo> photos initialized.
Create a JSON object
You can of course do that on your own, but a much better idea is to employ a specialized library that converts a java object to a JSON object. For example, you could use Gson library, which is a known library for that type of conversions. In the end, you'll have a JSON object, by calling String photosJSON = new Gson().toJson(photos);.
Set the JSON as a request attribute and perform a forward
Standard operation here.
request.setAttribute("photos", photosJSON);
request.getRequestDispatcher("car.jsp").forward(request, response);
Assign JSON to a JavaScript variable
In your JSP code, within a <script> block, have the following line:
var photosJS = JSON.parse(${photos});
Finally, you'll have a JS variable photosJS with a list you got from the database.

Workaround on Scraping HTML by diving into js source code

I learn about jSoup recently and would like to dive more into it. However, I have met obstacle handling webpages with javascript (I have no knowledge in js, yet :/).
I have read that htmlunit would be the correct tool to perform webbrowser actions, but I figured out that I would need no knowledge in js if I can find out the JSON object obtained in the webpage using the javascript.
For example, this page:
among the source files, one of them is tooltips.js. In this file, variable rgNeededFeeds is generated and called in method LoadHeropediaData(), which is the method to generate the whole URL link for getting the json object.
URL = URL + 'jsfeed/heropediadata?feeds='+strFeeds+'&v=3633666222511362823&l=english';
I could not get my mind on what is actually strFeeds. I have tried various combinations but it doesn't work (it returned an empty array...). Or, my guess is totally off?
What I actually need is the data it displays on top when you click on one of the "items". The info in the "hover" would do too, but it lack the "recepi" info. And I'm presuming that by getting the json object from the full URL above, well, basically all data infos should be in that json.
Anyways, this is only based on what I understand from staring at those source files for hours. Do correct me if I'm wrong. (I'm in Java by the way)
**p/s: I would also like to take this opportunity to express my thanks to Balusc, he has been everywhere when I have doubts on jSoup. :>*

strFeeds is nothing but one of these two strings : itemdata or abilitydata
You can find this in tooltips.js at line 38-45
var rgNeededFeeds = [];
$.each( [ 'item', 'ability' ],
function( i, ttType ){
icons = GetIconCollection( ttType );
if ( icons.length ){
rgNeededFeeds.push( ttType+'data' );
//..............
}
}
)
ttType is the value of an iteration over the array [ 'item', 'ability' ] which concatenated with the string data is pushed into the array rgNeededFeeds
The function LoadHeropediaData is called at the end of the function above with rgNeededFeeds as parameter :
LoadHeropediaData( rgNeededFeeds );
Aside note : If you begin to start scraping websites, learning javascript will be MANDATORY.
NOTE : you're right, the JSON contains all the information needed...