Develop Reference

jsoup - Not able to fetch a specific website

I'm using latest jsoup (1.13.1) in latest Eclipse IDE for Java Developers (includes Incubating components)
Version: 2020-09 (4.17.0)
Build id: 20200910-1200.
I'm trying to parse a very specific website, but with no success.
After I execute these lines:
doc = Jsoup.connect("http://pokehb.pw/%D7%A2%D7%95%D7%A0%D7%94/21/%D7%A4%D7%A8%D7%A7/43").get();
doc.select("title").forEach(System.out::println);
Nothing gets printed.
It's not just the , any element or property of the page is not available.
Yes, the URL is weird, but this is the one I need, I can browse it fine in Chrome.
I also know this is now due to the Hebrew in the website, since other Hebrew sites works ok.
For example, using this URL seems fine: https://context.reverso.net/translation/hebrew-english/%D7%9C%D7%9B%D7%AA%D7%95%D7%91%D7%AA+url
Any hint on what can be done?

What I can tell you is there's a "laravel_session" in the cookies. This suggests you'll need a more capable technology than JSoup. Try HtmlUnit instead and it might work.

What I ended up doing is using this command:
doc = Jsoup.parse(driver.getPageSource());
Which brought all of the page's source into the doc.
From there it was a simple use of getElementsByClass and getElementsByTag.
Hope this helps someone, and thanks Rob for trying to answer.

how to fetch data from webpage in java or python

I Am Working on project and in a specific part i need to search for some information and get the results of this search from the internet so how the data can be fetched from the web page to use it ?

Use Selenium - Here's a nice tutorial https://www.guru99.com/selenium-tutorial.html
Selenium is a java library, that supports all major browsers. The download and documentation links can be found on its website - https://www.seleniumhq.org/docs/

You can do that with python library beautifulsoup.

Parse a page (partly generated by JavaScript) by using Selenium

I've got a problem: I want to parse a page (e.g. this one) to collect information about the offered apps and save these information into a database.
Moreover I am using crawler4j for visiting every (available) page. But the problem - as I can see - is, that crawler4j needs links to follow in the source code.
But in this case the hrefs are generated by some JavaScript code so that crawler4j does not get new links to visit / pages to crawl.
So my idea was to use Selenium so that I can inspect several Elements like in a real Browser like Chrome or Firefox (I'm quite new with this).
But, to be honest, I don't know how to get the "generated" HTML instead of the source code.
Can anybody help me?

To inspect elements, you do not need the Selenium IDE, just use Firefox with the Firebug extension. Also, with the developer tools add on you can view a page's source and also the generated source (this is mainly for PHP).
Crawler4J can not handle javascript like this. It is better left for another more advanced crawling library. See this response here:
Web Crawling (Ajax/JavaScript enabled pages) using java

Jsoup1.7.2 isn't working google app engine 1.7.5

I'm working on a project where my application is hosted on Google App Engine and uses Jsoup html parsing library. In my application I'm using TaskQueues with the default queueu, the only task in that queue is to connect to a URL and start parsing the page. No errors or warnings appear in the log files, it just exits as it doesn't see the line of Jsoup parsing the document. Here is a snippet of my code:
log.warning("Before connection");
Document doc = Jsoup.connect("http://en.wikipedia.org/").get();
log.warning("After connection");
The TaskQueue works fine, I've tested and I'm 100% sure that there is no problem with it.
I've tried to manually connect to the webpage and download it then pass it to Jsoup and start parsing it there, the connection worked fine and the webpage was downloaded successfully, but yet Jsoup wasn't able to do anything.
My biggest problem is that there no errors and no warnings in the log file; so I don't know what it is going on.

The problem was that I was using Jsoup 1.7.2 which is clearly not very compatible with Google App Engine. I switched back to Jsoup 1.7.1 and the problem was fixed.

App engine restricts a number of of classes, I would assume that either the Jsoup.connect(url) or the .parse method depends on one of those restricted classes and throws an exception.
To eliminate the chance of the Jsoup.connect causing issues then I'd suggest you use App Engine URL Fetch to get a String of the page at the URL and then use:
Document doc = Jsoup.parse(htmlString);
However if there are issues with the parse then you really need to get errors/logging working and there's not a lot of info here yet to be able to suggest something. Try to put the problematic code in a try-catch block and see if you can catch an exception.
Additionally try a later version of GAE SDK (1.8.1 is the current one). I've previously had a conflict with GAE SDK's checkRestricted method that interfered with Jsoup, so that may be the case with 1.7.5.

Create a search box for an HTML page in a .war file

I have web project which included JSP pages in HTML pages. It's basically a product ordering website. When I run its .war file on my local machine, it has a search box on every page where you can search locally on the website. I cant get the search box to work for my .war. I've looked up for the code to do so on the web, and it says to enter the website's URL in the search value but I don't have a URL its basically a project!
The URL I'm using to run the project is [http://localhost:8080/myProjectWar/].

You want apply search on your own application. Very easy and suitable solution is just apply google custom search on your application.
Convert your search box to google custom search box. Entire searching and displaying the results and links will be take care by google.
Register your application with google custom search engine and get your credentials.
Try below link for registering .
Google Custom Search