Develop Reference

Request Body Shorter than Sent by Client - HttpServer Java

I create Java Application using HttpServer as bellow:
public class Application
{
public static void main(String args[])
{
HttpServer httpPaymentServer;
httpPaymentServer = HttpServer.create(new InetSocketAddress(Config.portPayment), 0);
httpPaymentServer.createContext("/json", new Payment("json"));
}
public class Payment implements HttpHandler
{
public Payment(String dataType)
{
}
public void handle(HttpExchange httpExchange) throws IOException
{
String body = "";
if(httpExchange.getRequestMethod().equalsIgnoreCase("POST"))
{
try
{
Headers requestHeaders = httpExchange.getRequestHeaders();
Set<Map.Entry<String, List<String>>> entries = requestHeaders.entrySet();
int contentLength = Integer.parseInt(requestHeaders.getFirst("Content-length"));
InputStream inputStream = httpExchange.getRequestBody();
byte[] postData = new byte[contentLength];
int length = inputStream.read(postData, 0, contentLength);
if(length < contentLength)
{
}
else
{
String fullBody = new String(postData);
Map<String, String> query = Utility.splitQuery(fullBody);
body = query.getOrDefault("data", "").toString();
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
}
}
On my server (Centos 7), on the first request, it is no problem. But on next request, not all of the request body can be read.
But on my PC (Windows 10) no problem.
What is the problem.

For your InputStream you call read only once - it may not return all the data. That data may even be not received at that time.
Instead you should call read in a loop until you get all the bytes (when you reach end of stream read returns -1). Or use one of the approaches suggested here How to read / convert an InputStream into a String in Java?

Thank you. This work for me
public void handle(HttpExchange httpExchange) throws IOException
{
String body = "";
if(httpExchange.getRequestMethod().equalsIgnoreCase("POST"))
{
try
{
Headers requestHeaders = httpExchange.getRequestHeaders();
Set<Map.Entry<String, List<String>>> entries = requestHeaders.entrySet();
int contentLength = Integer.parseInt(requestHeaders.getFirst("Content-length"));
InputStream inputStream = httpExchange.getRequestBody();
int j;
String fullBody = "";
for(j = 0; j < contentLength; j++)
{
byte b = (byte) httpExchange.getRequestBody().read();
fullBody += String.format("%c", b);
}
Map<String, String> query = Utility.splitQuery(fullBody);
body = query.getOrDefault("data", "").toString();
}
catch (Exception e)
{
e.printStackTrace();
}
}
}

HtmlViewService in OBIEE Web Service API is not rendering report (it's only showing spinning loader)

I have a Java application that leverages the OBIEE Web Service API to consume data from the BI Server. I am able to XMLViewService and the WebCatalogService just fine, but I can't quite get the HtmlViewService to properly render a report in the Java app. The report just shows the spinning loader, but never actually renders the report. I'm pretty sure it has to do with the fact that the Java app and the BI Server are on different domains. This is what the API documentation says:
In situations where Oracle BI Web Services and the third-party Web server do not belong to the same Domain Name Service (DNS) domain, users may get JavaScript errors related to browser security constraints for cross-domain scripting. To avoid these issues, use the setBridge() method to modify callback URLs to point to the third-party Web server. Be aware that a Web component executed by the third-party Web server to re-route requests to Oracle BI Web Services is not provided. This function would need to be fulfilled by the third-party application.
Several years ago, I did this same type of integration using .NET/C# and ran in the the same issue because the .NET app and the BI Server were on different domains. As a result, I had to create an HTTP Handler (.ashx file) as well as use the setBridge() method to to solve the issue.
The challenge that I'm having is that I can't find a servlet bridge example for Java. And I'm not too confident in porting the .NET/.ASHX code to a Java servlet/bridge. Does anyone have any code examples or direction they could provide to point me in the right direction? Here's a snippet of code to show you what I'm doing to pull back the report data:
// define report path
ReportRef reportRef = new ReportRef();
reportRef.setReportPath(reportFolder + "/" + reportName);
// set page params
StartPageParams pageParams = new StartPageParams();
pageParams.setDontUseHttpCookies(true);
// set report params
String pageId = htmlService.startPage(pageParams, sawSessionId);
String reportId = pageId + reportName;
htmlService.addReportToPage(pageId, reportId, reportRef, null, null, null, sawSessionId);
// get report html
StringBuffer reportHtml = new StringBuffer();
reportHtml.append(htmlService.getHtmlForReport(pageId, reportId, sawSessionId));
// return html
return reportHtml.toString();
This is the error that is coming back in the browser:
XMLHttpRequest cannot load http://myobiserver.com/analytics/saw.dll?ajaxGo. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://myjavaapp.com' is therefore not allowed access.
Per requested, here is my .NET/.ASHX bridge:
using System.Collections.Specialized;
using System.Net;
using System.Text;
using System.Web;
using System;
using System.Collections;
using System.Configuration;
using System.Collections.Specialized;
using System.Web;
using System.Text;
using System.Net;
using System.IO;
using System.Diagnostics;
/*
This is a ASP.NET handler that handles communication
between the SharePoint site and OracleBI.
It will be deployed to:
C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\LAYOUTS\OracleBI
*/
public class OracleBridge: IHttpHandler
{
public bool IsReusable {get{return true;}}
public OracleBridge()
{
}
string getServer()
{
string strServer = "http://<enter-domain>/analytics/saw.dll";
int index = strServer.LastIndexOf("/");//split off saw.dll
if (index >=0)
return strServer.Substring(0,index+1);
else
return strServer;
}
public void ProcessRequest(HttpContext context)
{
HttpWebRequest req = forwardRequest(context);
forwardResponse(context,req);
}
private HttpWebRequest forwardRequest(HttpContext context)
{
string strURL = makeURL(context);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(strURL);
req.Method = context.Request.HttpMethod;
NameValueCollection headers = context.Request.Headers;
req.Accept = headers.Get("Accept");
req.Expect = headers.Get("Expect");
req.ContentType = headers.Get("Content-Type");
string strModifiedSince = headers.Get("If-Modified-Since");
if (strModifiedSince != null && strModifiedSince.Length != 0)
req.IfModifiedSince = DateTime.Parse(strModifiedSince);
req.Referer = headers.Get("Referer");
req.UserAgent = headers.Get("User-Agent");
if (!req.Method.Equals("GET"))
{
CopyStreams(context.Request.InputStream,req.GetRequestStream());
}
return req;
}
private void forwardResponse(HttpContext context, HttpWebRequest req)
{
HttpWebResponse resp =null;
try
{
resp = (HttpWebResponse)req.GetResponse();
}
catch(WebException e)
{
resp = (HttpWebResponse)e.Response;
}
context.Response.StatusCode = (int)resp.StatusCode;
for (int i = 0; i < resp.Cookies.Count; i++)
{
Cookie c = resp.Cookies[i];
HttpCookie hc = new HttpCookie(c.Name, c.Value);
hc.Path = c.Path;
hc.Domain = getServer();
context.Response.Cookies.Add(hc);
}
context.Response.ContentType = resp.ContentType;
CopyStreams(resp.GetResponseStream(), context.Response.OutputStream);
}
private string makeURL(HttpContext context)
{
string strQuery = context.Request.Url.Query;
string[] arrParams = strQuery.Split('?','&');
StringBuilder resultingParams = new StringBuilder();
string strURL=null;
foreach(string strParam in arrParams )
{
string[] arrNameValue = strParam.Split('=');
if (!arrNameValue[0].Equals("RedirectURL"))
{
if (strParam.Length != 0)
{
if (resultingParams.Length != 0)
resultingParams.Append("&");
resultingParams.Append(strParam);
}
}
else if (arrNameValue.Length >1)
strURL = HttpUtility.UrlDecode(arrNameValue[1]);
}
if (strURL ==null)
throw new Exception("Invalid URL format. requestURL parameter is missing");
String sAppendChar = strURL.Contains("?") ? "&" : "?";
if (strURL.StartsWith("http:") || strURL.StartsWith("https:"))
{
String tmpURL = strURL + sAppendChar + resultingParams.ToString();
return tmpURL;
}
else
{
String tmpURL = getServer() + strURL + sAppendChar + resultingParams.ToString();
return tmpURL;
}
}
private void CopyStreams(Stream inStr,Stream outStr)
{
byte[] buf = new byte[4096];
try
{
do
{
int iRead = inStr.Read(buf,0,4096);
if (iRead == 0)
break;
outStr.Write(buf,0,iRead);
}
while (true);
}
finally
{
outStr.Close();
}
}
}

Using the BridgeServlet from link (http://pastebin.com/NibVnBLb) posted in previous answer did not work for us. In our web portal, embedding Oracle BI dashboard using BridgeServlet above, redirected us to OBI login page and console log showed incorrect resources(js/css) web links (local URL instead of OBIEE URL's).
Instead we used this class (with some minor adjustments) https://gist.github.com/rafaeltuelho/9376341#file-obieehttpservletbridge-java.
Tested with Java 11, Oracle Business Intelligence 12.2.1.4.0, WSDL v12 of OBIEE (http://OBIEE-server:port/analytics/saw.dll/wsdl/v12), with SSO disabled.
Here it is the BridgeServlet class:
package com.abs.bi;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.util.Enumeration;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
import javax.servlet.ServletException;
import javax.servlet.http.Cookie;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
/**
* Servlet implementation class OBIEEBridge
*/
public class BridgeServlet extends HttpServlet {
private static final long serialVersionUID = 1L;
/**
* #see HttpServlet#HttpServlet()
*/
public BridgeServlet() {
super();
}
/**
* #see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse
* response)
*/
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
try {
this.processRequest(request, response);
} catch (Exception e) {
throw new ServletException(e);
}
}
/**
* #see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse
* response)
*/
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
try {
this.processRequest(request, response);
} catch (Exception e) {
throw new ServletException(e);
}
}
protected void processRequest(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
HttpURLConnection urlCon = forwardRequest(request);
forwardResponse(response, urlCon);
}
#SuppressWarnings("unchecked")
private String decodeURL(HttpServletRequest request) {
StringBuffer bufURL = new StringBuffer("");
Map<String, String[]> params = request.getParameterMap();
String[] arrURL = params.get("RedirectURL");
String strURL = arrURL == null || arrURL.length == 0 ? null : arrURL[0];
bufURL.append(strURL);
int nQIndex = strURL.lastIndexOf('?');
if (params != null && !params.isEmpty()) {
bufURL.append(((nQIndex >= 0) ? "&" : "?"));
Set<String> keys = params.keySet();
Iterator<String> it = keys.iterator();
while (it.hasNext()) {
try {
String strKey = it.next();
if (strKey.equalsIgnoreCase("RedirectURL")) {
continue;
}
String strEncodedKey = URLEncoder.encode(strKey, "UTF-8");
String[] paramValues = params.get(strKey);
for (String paramValue : paramValues) {
bufURL.append(strEncodedKey);
bufURL.append("=");
bufURL.append(URLEncoder.encode(paramValue, "UTF-8"));
bufURL.append("&");
}
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
}
bufURL.deleteCharAt(bufURL.length() - 1);
}
return bufURL.toString();
}
#SuppressWarnings("unchecked")
private HttpURLConnection forwardRequest(HttpServletRequest request) throws IOException {
String strURL = decodeURL(request);
String[] arrURL = strURL.split("&", 2);
String baseURL = arrURL[0];
URL url = new URL(baseURL);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
String strMethod = request.getMethod();
con.setRequestMethod(strMethod);
Enumeration<String> en = request.getHeaderNames();
String strHeader;
while (en.hasMoreElements()) {
strHeader = en.nextElement();
String strHeaderValue = request.getHeader(strHeader);
con.setRequestProperty(strHeader, strHeaderValue);
}
// is not a HTTP GET request
if (strMethod.compareTo("GET") != 0) {
con.setDoOutput(true);
con.setDoInput(true);
con.setUseCaches(false);
DataOutputStream forwardStream = new DataOutputStream(con.getOutputStream());
try {
String urlParameters = arrURL[1];
forwardStream.writeBytes(urlParameters);
forwardStream.flush();
} finally {
forwardStream.close();
}
}
return con;
}
private void forwardResponse(HttpServletResponse response, HttpURLConnection con) throws IOException {
int nContentLen = -1;
String strKey;
String strValue;
try {
response.setStatus(con.getResponseCode());
for (int i = 1; true; ++i) {
strKey = con.getHeaderFieldKey(i);
strValue = con.getHeaderField(i);
if (strKey == null) {
break;
}
if (strKey.equals("Content-Length")) {
nContentLen = Integer.parseInt(con.getHeaderField(i));
continue;
}
if (strKey.equalsIgnoreCase("Connection") || strKey.equalsIgnoreCase("Server")
|| strKey.equalsIgnoreCase("Transfer-Encoding") || strKey.equalsIgnoreCase("Content-Length")) {
continue; // skip certain headers
}
if (strKey.equalsIgnoreCase("Set-Cookie")) {
String[] cookieStr1 = strValue.split(";");
String[] cookieStr2 = cookieStr1[0].split("=");
// String[] cookieStr3 = cookieStr1[1].split("=");
/*
* Change the Set-Cookie HTTP Header to remove the 'path' attribute. Thus the
* browser can accept the ORA_BIPS_NQID cookie from Oracle BI Server
*/
Cookie c = new Cookie(cookieStr2[0], cookieStr2[1]);
c.setPath("/");
response.addCookie(c);
} else {
response.setHeader(strKey, strValue);
}
}
copyStreams(con.getInputStream(), response.getOutputStream(), nContentLen);
} finally {
response.getOutputStream().close();
con.getInputStream().close();
}
}
private void copyStreams(InputStream inputStream, OutputStream forwardStream, int nContentLen) throws IOException {
byte[] buf = new byte[1024];
int nCount = 0;
int nBytesToRead = 1024;
int nTotalCount = 0;
do {
if (nContentLen != -1)
nBytesToRead = nContentLen - nTotalCount > 1024 ? 1024 : nContentLen - nTotalCount;
if (nBytesToRead == 0)
break;
// try to read some bytes from src stream
nCount = inputStream.read(buf, 0, nBytesToRead);
if (nCount < 0)
break;
nTotalCount += nCount;
// try to write some bytes in target stream
forwardStream.write(buf, 0, nCount);
} while (true);
}
}
AbsServiceUtils which contains SAWSessionService and HtmlViewService web service calls to Oracle BI server:
package com.abs.bi;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.jsp.JspWriter;
import oracle.bi.web.soap.AuthResult;
import oracle.bi.web.soap.HtmlViewService;
import oracle.bi.web.soap.HtmlViewServiceSoap;
import oracle.bi.web.soap.ReportHTMLLinksMode;
import oracle.bi.web.soap.ReportHTMLOptions;
import oracle.bi.web.soap.ReportRef;
import oracle.bi.web.soap.SAWLocale;
import oracle.bi.web.soap.SAWSessionParameters;
import oracle.bi.web.soap.SAWSessionService;
import oracle.bi.web.soap.SAWSessionServiceSoap;
import oracle.bi.web.soap.StartPageParams;
public class AbsServiceUtils {
private AbsServiceUtils() {
}
public static URL buildWsdlUrl() throws MalformedURLException {
return new URL(IAbsService.BASEWSDLURL);
}
public static void writeBiContent(HttpServletRequest request, JspWriter out, String biReport) throws IOException {
String userAgent = request.getHeader("User-Agent");
Locale userLocale = request.getLocale();
String bridgeServletContextPath = request.getContextPath() + "/bridgeservlet";
String reportHtml = writeBiContent(biReport, userAgent, userLocale, bridgeServletContextPath);
if (out != null) {
out.println(reportHtml);
}
}
public static String writeBiContent(String biReport, String userAgent, Locale userLocale,
String bridgeServletContextPath) throws MalformedURLException {
HtmlViewService htmlViewService = new HtmlViewService(buildWsdlUrl());
HtmlViewServiceSoap htmlClient = htmlViewService.getHtmlViewService();
SAWSessionService sAWSessionService = new SAWSessionService(buildWsdlUrl());
SAWSessionServiceSoap myPort = sAWSessionService.getSAWSessionServiceSoap();
SAWSessionParameters sessionparams = new SAWSessionParameters();
sessionparams.setUserAgent(userAgent);
SAWLocale sawlocale = new SAWLocale();
sawlocale.setLanguage(userLocale.getLanguage());
sawlocale.setCountry(userLocale.getCountry());
sessionparams.setLocale(sawlocale);
sessionparams.setAsyncLogon(false);
AuthResult result = myPort.logonex(IAbsService.BIUSERNAME, IAbsService.BIPASSWORD, sessionparams);
String sessionID = result.getSessionID();
List<String> keepAliveSessionList = new ArrayList<>(1);
keepAliveSessionList.add(sessionID);
myPort.keepAlive(keepAliveSessionList);
StartPageParams spparams = new StartPageParams();
spparams.setDontUseHttpCookies(true);
String pageID = htmlClient.startPage(spparams, sessionID);
/**
* This method will set the path to the servlet which will act like a bridge to
* retrieve all the OBIEE resources like the javascript, CSS and the report.
*/
if (bridgeServletContextPath != null) {
htmlClient.setBridge(bridgeServletContextPath, sessionID);
}
ReportHTMLOptions htmlOptions = new ReportHTMLOptions();
htmlOptions.setEnableDelayLoading(false);
htmlOptions.setLinkMode(ReportHTMLLinksMode.IN_PLACE.value());
ReportRef reportref = new ReportRef();
reportref.setReportPath(IAbsService.BIROOTPATH + biReport);
StartPageParams startpageparams = new StartPageParams();
startpageparams.setDontUseHttpCookies(false);
htmlClient.addReportToPage(pageID, biReport.replace(" ", ""), reportref, null, null, htmlOptions, sessionID);
String reportHtml = htmlClient.getHeadersHtml(pageID, sessionID);
reportHtml = reportHtml + htmlClient.getHtmlForReport(pageID, biReport.replace(" ", ""), sessionID);
reportHtml = reportHtml + htmlClient.getCommonBodyHtml(pageID, sessionID);
return reportHtml;
}
}
IAbsService:
package com.abs.bi;
public interface IAbsService {
public static final String BASEWSDLURL = "http://<OracleBIServer:port>/analytics/saw.dll/wsdl/v12";
public static final String BIUSERNAME = "USER";
public static final String BIPASSWORD = "PASS";
public static final String BIROOTPATH = "/shared/sharedfolder/";
public static final String BIREPORTNAME = "report";
}

Use the link http://pastebin.com/NibVnBLb to check out the bridge code for java. Hope this might be helful.

crawler for gwt application taking too much time

i have a gwt application that i need to optimize for seo ( crawl the content for google), and i have been trying many solutions wich are not meeting our needs (it's taking us a big amount of time to return the html page), the trials are:
I tried to use htmlUnit as headless browser to crawl the page on demand, it takes about 15 second to get the html content (when auditing this timing, it results that 80% of this timing is taken by a loop that waits for background javascript "while (waitForBackgroundJavaScript > 0 && loopCount < _maxLoopChecks) ")
A technic that consists on crawling the page prior to google request, then giving the saved snapshot when google is asking for it (but this solution is definitely not convenient because the content changes very frequently and google may consider this as a "CLOACKING")
Any suggestion?
the code used to crawl:
public class CrawlFilter implements Filter {
private class SyncAllAjaxController extends NicelyResynchronizingAjaxController {
private static final long serialVersionUID = 1L;
#Override
public boolean processSynchron(HtmlPage page, WebRequest request, boolean async) {
return true;
}
}
private final Logger log = Logger.getLogger(CrawlFilter.class.getName());
/**
* Special URL token that gets passed from the crawler to the servlet
* filter. This token is used in case there are already existing query
* parameters.
*/
private static final String ESCAPED_FRAGMENT_FORMAT1 = "_escaped_fragment_=";
private static final int ESCAPED_FRAGMENT_LENGTH1 = ESCAPED_FRAGMENT_FORMAT1.length();
/**
* Special URL token that gets passed from the crawler to the servlet
* filter. This token is used in case there are not already existing query
* parameters.
*/
private static final String ESCAPED_FRAGMENT_FORMAT2 = "&" + ESCAPED_FRAGMENT_FORMAT1;
private static final int ESCAPED_FRAGMENT_LENGTH2 = ESCAPED_FRAGMENT_FORMAT2.length();
private static final long _pumpEventLoopTimeoutMillis = 30000;
private static final long _jsTimeoutMillis = 1000;
private static final long _pageWaitMillis = 200;
private static final int _maxLoopChecks = 2;
private WebClient webClient;
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain filterChain) throws IOException, ServletException {
// Grab the request uri and query strings.
final HttpServletRequest httpRequest = (HttpServletRequest) request;
final String requestURI = httpRequest.getRequestURI();
final String queryString = httpRequest.getQueryString();
final HttpServletResponse httpResponse = (HttpServletResponse) response;
if ((queryString != null) && (queryString.contains(ESCAPED_FRAGMENT_FORMAT1))) {
final int port = httpRequest.getServerPort();
final String urlStringWithHashFragment = requestURI + rewriteQueryString(queryString);
final String scheme = httpRequest.getScheme();
final URL urlWithHashFragment = new URL(scheme, "127.0.0.1", port, urlStringWithHashFragment);
final WebRequest webRequest = new WebRequest(urlWithHashFragment);
log.fine("Crawl filter encountered escaped fragment, will open: " + webRequest.toString());
httpResponse.setContentType("text/html;charset=UTF-8");
final PrintWriter out = httpResponse.getWriter();
out.println(renderPage(webRequest));
out.flush();
out.close();
log.fine("HtmlUnit completed webClient.getPage(webRequest) where webRequest = " + webRequest.toString());
} else {
filterChain.doFilter(request, response);
}
}
#Override
public void destroy() {
if (webClient != null) {
webClient.closeAllWindows();
}
}
#Override
public void init(FilterConfig config) throws ServletException {
}
private StringBuilder renderPage(WebRequest webRequest) throws IOException {
webClient = new WebClient(BrowserVersion.FIREFOX_17);
webClient.getCache().clear();
webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setRedirectEnabled(false);
webClient.setAjaxController(new SyncAllAjaxController());
webClient.setCssErrorHandler(new SilentCssErrorHandler());
final HtmlPage page = webClient.getPage(webRequest);
webClient.getJavaScriptEngine().pumpEventLoop(_pumpEventLoopTimeoutMillis);
int waitForBackgroundJavaScript = webClient.waitForBackgroundJavaScript(_jsTimeoutMillis);
int loopCount = 0;
while (waitForBackgroundJavaScript > 0 && loopCount < _maxLoopChecks) {
++loopCount;
waitForBackgroundJavaScript = webClient.waitForBackgroundJavaScript(_jsTimeoutMillis);
if (waitForBackgroundJavaScript == 0) {
log.fine("HtmlUnit exits background javascript at loop counter " + loopCount);
break;
}
synchronized (page) {
log.fine("HtmlUnit waits for background javascript at loop counter " + loopCount);
try {
page.wait(_pageWaitMillis);
} catch (InterruptedException e) {
log.log(Level.SEVERE, "HtmlUnit ERROR on page.wait at loop counter " + loopCount, e);
}
}
}
webClient.getAjaxController().processSynchron(page, webRequest, false);
if (webClient.getJavaScriptEngine().isScriptRunning()) {
log.warning("HtmlUnit webClient.getJavaScriptEngine().shutdownJavaScriptExecutor()");
webClient.getJavaScriptEngine().shutdownJavaScriptExecutor();
}
final String staticSnapshotHtml = page.asXml();
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append("<hr />\n");
stringBuilder.append("<center><h3>This is a non-interactive snapshot for crawlers. Follow <a href=\"");
stringBuilder.append(webRequest.getUrl() + "\">this link</a> for the interactive application.<br></h3></center>");
stringBuilder.append("<hr />");
stringBuilder.append(staticSnapshotHtml);
return stringBuilder;
}
/**
* Maps from the query string that contains _escaped_fragment_ to one that
* doesn't, but is instead followed by a hash fragment. It also unescapes any
* characters that were escaped by the crawler. If the query string does not
* contain _escaped_fragment_, it is not modified.
*
* #param queryString
* #return A modified query string followed by a hash fragment if applicable.
* The non-modified query string otherwise.
* #throws UnsupportedEncodingException
*/
private static String rewriteQueryString(String queryString) throws UnsupportedEncodingException {
int index = queryString.indexOf(ESCAPED_FRAGMENT_FORMAT2);
int length = ESCAPED_FRAGMENT_LENGTH2;
if (index == -1) {
index = queryString.indexOf(ESCAPED_FRAGMENT_FORMAT1);
length = ESCAPED_FRAGMENT_LENGTH1;
}
if (index != -1) {
StringBuilder queryStringSb = new StringBuilder();
if (index > 0) {
queryStringSb.append("?");
queryStringSb.append(queryString.substring(0, index));
}
queryStringSb.append("#!");
queryStringSb.append(URLDecoder.decode(queryString.substring(index
+ length, queryString.length()), "UTF-8"));
return queryStringSb.toString();
}
return queryString;
}
}

I suggest having HtmlUnit generate the static html offline. You control the update frequency.
Then, have your servlet filter intercepting the crawler request return the already generated static html.

How to serve static content using suns simple httpserver

I'm using jersey's HttpServerFactory to create a simple embedded HttpServer that hosts a couple of rest services. We just needed something small quick and lightweight. I need to host a small static html page inside the same server instance. Is there a simple way to add a static handler to the server? Is there a pre-defined handler I can use? It seems like a pretty common task, I'd hate to re-write code for it if it already exists.
server = HttpServerFactory.create(url);
server.setExecutor(Executors.newCachedThreadPool());
server.createContext("/staticcontent", new HttpHandler() {
#Override
public void handle(HttpExchange arg0) throws IOException {
//What goes here?
}
});
server.start();

Here is a safe version. You may want to add a couple of MIME types, depending on which ones are common (or use another method if your platform has that).
package de.phihag.miniticker;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.HashMap;
import java.util.Map;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
public class StaticFileHandler implements HttpHandler {
private static final Map<String,String> MIME_MAP = new HashMap<>();
static {
MIME_MAP.put("appcache", "text/cache-manifest");
MIME_MAP.put("css", "text/css");
MIME_MAP.put("gif", "image/gif");
MIME_MAP.put("html", "text/html");
MIME_MAP.put("js", "application/javascript");
MIME_MAP.put("json", "application/json");
MIME_MAP.put("jpg", "image/jpeg");
MIME_MAP.put("jpeg", "image/jpeg");
MIME_MAP.put("mp4", "video/mp4");
MIME_MAP.put("pdf", "application/pdf");
MIME_MAP.put("png", "image/png");
MIME_MAP.put("svg", "image/svg+xml");
MIME_MAP.put("xlsm", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
MIME_MAP.put("xml", "application/xml");
MIME_MAP.put("zip", "application/zip");
MIME_MAP.put("md", "text/plain");
MIME_MAP.put("txt", "text/plain");
MIME_MAP.put("php", "text/plain");
};
private String filesystemRoot;
private String urlPrefix;
private String directoryIndex;
/**
* #param urlPrefix The prefix of all URLs.
* This is the first argument to createContext. Must start and end in a slash.
* #param filesystemRoot The root directory in the filesystem.
* Only files under this directory will be served to the client.
* For instance "./staticfiles".
* #param directoryIndex File to show when a directory is requested, e.g. "index.html".
*/
public StaticFileHandler(String urlPrefix, String filesystemRoot, String directoryIndex) {
if (!urlPrefix.startsWith("/")) {
throw new RuntimeException("pathPrefix does not start with a slash");
}
if (!urlPrefix.endsWith("/")) {
throw new RuntimeException("pathPrefix does not end with a slash");
}
this.urlPrefix = urlPrefix;
assert filesystemRoot.endsWith("/");
try {
this.filesystemRoot = new File(filesystemRoot).getCanonicalPath();
} catch (IOException e) {
throw new RuntimeException(e);
}
this.directoryIndex = directoryIndex;
}
/**
* Create and register a new static file handler.
* #param hs The HTTP server where the file handler will be registered.
* #param path The path in the URL prefixed to all requests, such as "/static/"
* #param filesystemRoot The filesystem location.
* For instance "/var/www/mystaticfiles/".
* A request to "/static/x/y.html" will be served from the filesystem file "/var/www/mystaticfiles/x/y.html"
* #param directoryIndex File to show when a directory is requested, e.g. "index.html".
*/
public static void create(HttpServer hs, String path, String filesystemRoot, String directoryIndex) {
StaticFileHandler sfh = new StaticFileHandler(path, filesystemRoot, directoryIndex);
hs.createContext(path, sfh);
}
public void handle(HttpExchange he) throws IOException {
String method = he.getRequestMethod();
if (! ("HEAD".equals(method) || "GET".equals(method))) {
sendError(he, 501, "Unsupported HTTP method");
return;
}
String wholeUrlPath = he.getRequestURI().getPath();
if (wholeUrlPath.endsWith("/")) {
wholeUrlPath += directoryIndex;
}
if (! wholeUrlPath.startsWith(urlPrefix)) {
throw new RuntimeException("Path is not in prefix - incorrect routing?");
}
String urlPath = wholeUrlPath.substring(urlPrefix.length());
File f = new File(filesystemRoot, urlPath);
File canonicalFile;
try {
canonicalFile = f.getCanonicalFile();
} catch (IOException e) {
// This may be more benign (i.e. not an attack, just a 403),
// but we don't want the attacker to be able to discern the difference.
reportPathTraversal(he);
return;
}
String canonicalPath = canonicalFile.getPath();
if (! canonicalPath.startsWith(filesystemRoot)) {
reportPathTraversal(he);
return;
}
FileInputStream fis;
try {
fis = new FileInputStream(canonicalFile);
} catch (FileNotFoundException e) {
// The file may also be forbidden to us instead of missing, but we're leaking less information this way
sendError(he, 404, "File not found");
return;
}
String mimeType = lookupMime(urlPath);
he.getResponseHeaders().set("Content-Type", mimeType);
if ("GET".equals(method)) {
he.sendResponseHeaders(200, canonicalFile.length());
OutputStream os = he.getResponseBody();
copyStream(fis, os);
os.close();
} else {
assert("HEAD".equals(method));
he.sendResponseHeaders(200, -1);
}
fis.close();
}
private void copyStream(InputStream is, OutputStream os) throws IOException {
byte[] buf = new byte[4096];
int n;
while ((n = is.read(buf)) >= 0) {
os.write(buf, 0, n);
}
}
private void sendError(HttpExchange he, int rCode, String description) throws IOException {
String message = "HTTP error " + rCode + ": " + description;
byte[] messageBytes = message.getBytes("UTF-8");
he.getResponseHeaders().set("Content-Type", "text/plain; charset=utf-8");
he.sendResponseHeaders(rCode, messageBytes.length);
OutputStream os = he.getResponseBody();
os.write(messageBytes);
os.close();
}
// This is one function to avoid giving away where we failed
private void reportPathTraversal(HttpExchange he) throws IOException {
sendError(he, 400, "Path traversal attempt detected");
}
private static String getExt(String path) {
int slashIndex = path.lastIndexOf('/');
String basename = (slashIndex < 0) ? path : path.substring(slashIndex + 1);
int dotIndex = basename.lastIndexOf('.');
if (dotIndex >= 0) {
return basename.substring(dotIndex + 1);
} else {
return "";
}
}
private static String lookupMime(String path) {
String ext = getExt(path).toLowerCase();
return MIME_MAP.getOrDefault(ext, "application/octet-stream");
}
}

This will do the trick, though it does allow anyone to walk the tree by requesting ../../../
You can change ./wwwroot to any valid java filepath.
static class MyHandler implements HttpHandler {
public void handle(HttpExchange t) throws IOException {
String root = "./wwwroot";
URI uri = t.getRequestURI();
System.out.println("looking for: "+ root + uri.getPath());
String path = uri.getPath();
File file = new File(root + path).getCanonicalFile();
if (!file.isFile()) {
// Object does not exist or is not a file: reject with 404 error.
String response = "404 (Not Found)\n";
t.sendResponseHeaders(404, response.length());
OutputStream os = t.getResponseBody();
os.write(response.getBytes());
os.close();
} else {
// Object exists and is a file: accept with response code 200.
String mime = "text/html";
if(path.substring(path.length()-3).equals(".js")) mime = "application/javascript";
if(path.substring(path.length()-3).equals("css")) mime = "text/css";
Headers h = t.getResponseHeaders();
h.set("Content-Type", mime);
t.sendResponseHeaders(200, 0);
OutputStream os = t.getResponseBody();
FileInputStream fs = new FileInputStream(file);
final byte[] buffer = new byte[0x10000];
int count = 0;
while ((count = fs.read(buffer)) >= 0) {
os.write(buffer,0,count);
}
fs.close();
os.close();
}
}
}

using dbpedia spotlight in java or scala

Does anyone know where to find a little how to on using dbpedia spotlight in java or scala? Or could anyone explain how it's done? I can't find any information on this...

The DBpedia Spotlight wiki pages would be a good place to start.
And I believe the installation page has listed the most popular ways (using a jar, or set up a web service) to use the application.
It includes instructions on using the Java/Scala API with your own installation, or calling the Web Service.
There are some additional data needed to be downloaded to run your own server for full service, good time to make a coffee for yourself.

you need download dbpedia spotlight (jar file) after that u can use next two classes ( author pablomendes ) i only make some change .
public class db extends AnnotationClient {
//private final static String API_URL = "http://jodaiber.dyndns.org:2222/";
private static String API_URL = "http://spotlight.dbpedia.org:80/";
private static double CONFIDENCE = 0.0;
private static int SUPPORT = 0;
private static String powered_by ="non";
private static String spotter ="CoOccurrenceBasedSelector";//"LingPipeSpotter"=Annotate all spots
//AtLeastOneNounSelector"=No verbs and adjs.
//"CoOccurrenceBasedSelector" =No 'common words'
//"NESpotter"=Only Per.,Org.,Loc.
private static String disambiguator ="Default";//Default ;Occurrences=Occurrence-centric;Document=Document-centric
private static String showScores ="yes";
#SuppressWarnings("static-access")
public void configiration(double CONFIDENCE,int SUPPORT,
String powered_by,String spotter,String disambiguator,String showScores){
this.CONFIDENCE=CONFIDENCE;
this.SUPPORT=SUPPORT;
this.powered_by=powered_by;
this.spotter=spotter;
this.disambiguator=disambiguator;
this.showScores=showScores;
}
public List<DBpediaResource> extract(Text text) throws AnnotationException {
LOG.info("Querying API.");
String spotlightResponse;
try {
String Query=API_URL + "rest/annotate/?" +
"confidence=" + CONFIDENCE
+ "&support=" + SUPPORT
+ "&spotter=" + spotter
+ "&disambiguator=" + disambiguator
+ "&showScores=" + showScores
+ "&powered_by=" + powered_by
+ "&text=" + URLEncoder.encode(text.text(), "utf-8");
LOG.info(Query);
GetMethod getMethod = new GetMethod(Query);
getMethod.addRequestHeader(new Header("Accept", "application/json"));
spotlightResponse = request(getMethod);
} catch (UnsupportedEncodingException e) {
throw new AnnotationException("Could not encode text.", e);
}
assert spotlightResponse != null;
JSONObject resultJSON = null;
JSONArray entities = null;
try {
resultJSON = new JSONObject(spotlightResponse);
entities = resultJSON.getJSONArray("Resources");
} catch (JSONException e) {
//throw new AnnotationException("Received invalid response from DBpedia Spotlight API.");
}
LinkedList<DBpediaResource> resources = new LinkedList<DBpediaResource>();
if(entities!=null)
for(int i = 0; i < entities.length(); i++) {
try {
JSONObject entity = entities.getJSONObject(i);
resources.add(
new DBpediaResource(entity.getString("#URI"),
Integer.parseInt(entity.getString("#support"))));
} catch (JSONException e) {
LOG.error("JSON exception "+e);
}
}
return resources;
}
}
second class
/**
* #author pablomendes
*/
public abstract class AnnotationClient {
public Logger LOG = Logger.getLogger(this.getClass());
private List<String> RES = new ArrayList<String>();
// Create an instance of HttpClient.
private static HttpClient client = new HttpClient();
public List<String> getResu(){
return RES;
}
public String request(HttpMethod method) throws AnnotationException {
String response = null;
// Provide custom retry handler is necessary
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
try {
// Execute the method.
int statusCode = client.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
LOG.error("Method failed: " + method.getStatusLine());
}
// Read the response body.
byte[] responseBody = method.getResponseBody(); //TODO Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
// Deal with the response.
// Use caution: ensure correct character encoding and is not binary data
response = new String(responseBody);
} catch (HttpException e) {
LOG.error("Fatal protocol violation: " + e.getMessage());
throw new AnnotationException("Protocol error executing HTTP request.",e);
} catch (IOException e) {
LOG.error("Fatal transport error: " + e.getMessage());
LOG.error(method.getQueryString());
throw new AnnotationException("Transport error executing HTTP request.",e);
} finally {
// Release the connection.
method.releaseConnection();
}
return response;
}
protected static String readFileAsString(String filePath) throws java.io.IOException{
return readFileAsString(new File(filePath));
}
protected static String readFileAsString(File file) throws IOException {
byte[] buffer = new byte[(int) file.length()];
#SuppressWarnings("resource")
BufferedInputStream f = new BufferedInputStream(new FileInputStream(file));
f.read(buffer);
return new String(buffer);
}
static abstract class LineParser {
public abstract String parse(String s) throws ParseException;
static class ManualDatasetLineParser extends LineParser {
public String parse(String s) throws ParseException {
return s.trim();
}
}
static class OccTSVLineParser extends LineParser {
public String parse(String s) throws ParseException {
String result = s;
try {
result = s.trim().split("\t")[3];
} catch (ArrayIndexOutOfBoundsException e) {
throw new ParseException(e.getMessage(), 3);
}
return result;
}
}
}
public void saveExtractedEntitiesSet(String Question, LineParser parser, int restartFrom) throws Exception {
String text = Question;
int i=0;
//int correct =0 ; int error = 0;int sum = 0;
for (String snippet: text.split("\n")) {
String s = parser.parse(snippet);
if (s!= null && !s.equals("")) {
i++;
if (i<restartFrom) continue;
List<DBpediaResource> entities = new ArrayList<DBpediaResource>();
try {
entities = extract(new Text(snippet.replaceAll("\\s+"," ")));
System.out.println(entities.get(0).getFullUri());
} catch (AnnotationException e) {
// error++;
LOG.error(e);
e.printStackTrace();
}
for (DBpediaResource e: entities) {
RES.add(e.uri());
}
}
}
}
public abstract List<DBpediaResource> extract(Text text) throws AnnotationException;
public void evaluate(String Question) throws Exception {
evaluateManual(Question,0);
}
public void evaluateManual(String Question, int restartFrom) throws Exception {
saveExtractedEntitiesSet(Question,new LineParser.ManualDatasetLineParser(), restartFrom);
}
}
main()
public static void main(String[] args) throws Exception {
String Question ="Is the Amazon river longer than the Nile River?";
db c = new db ();
c.configiration(0.0, 0, "non", "CoOccurrenceBasedSelector", "Default", "yes");
System.out.println("resource : "+c.getResu());
}

I just add one little fix for your answer.
Your code is running, if you add the evaluate method call:
public static void main(String[] args) throws Exception {
String question = "Is the Amazon river longer than the Nile River?";
db c = new db ();
c.configiration(0.0, 0, "non", "CoOccurrenceBasedSelector", "Default", "yes");
c.evaluate(question);
System.out.println("resource : "+c.getResu());
}
Lamine

In the request method of the second class (AnnotationClient) in Adel's answer, the author Pablo Mendes hasn't finished
TODO Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
which is an annoying warning that needs to be removed by replacing
byte[] responseBody = method.getResponseBody(); //TODO Going to buffer response body of large or unknown size. Using getResponseBodyAsStream instead is recommended.
// Deal with the response.
// Use caution: ensure correct character encoding and is not binary data
response = new String(responseBody);
with
Reader in = new InputStreamReader(method.getResponseBodyAsStream(), "UTF-8");
StringWriter writer = new StringWriter();
org.apache.commons.io.IOUtils.copy(in, writer);
response = writer.toString();