i have a gwt application that i need to optimize for seo ( crawl the content for google), and i have been trying many solutions wich are not meeting our needs (it's taking us a big amount of time to return the html page), the trials are:
I tried to use htmlUnit as headless browser to crawl the page on demand, it takes about 15 second to get the html content (when auditing this timing, it results that 80% of this timing is taken by a loop that waits for background javascript "while (waitForBackgroundJavaScript > 0 && loopCount < _maxLoopChecks) ")
A technic that consists on crawling the page prior to google request, then giving the saved snapshot when google is asking for it (but this solution is definitely not convenient because the content changes very frequently and google may consider this as a "CLOACKING")
Any suggestion?
the code used to crawl:
public class CrawlFilter implements Filter {
private class SyncAllAjaxController extends NicelyResynchronizingAjaxController {
private static final long serialVersionUID = 1L;
#Override
public boolean processSynchron(HtmlPage page, WebRequest request, boolean async) {
return true;
}
}
private final Logger log = Logger.getLogger(CrawlFilter.class.getName());
/**
* Special URL token that gets passed from the crawler to the servlet
* filter. This token is used in case there are already existing query
* parameters.
*/
private static final String ESCAPED_FRAGMENT_FORMAT1 = "_escaped_fragment_=";
private static final int ESCAPED_FRAGMENT_LENGTH1 = ESCAPED_FRAGMENT_FORMAT1.length();
/**
* Special URL token that gets passed from the crawler to the servlet
* filter. This token is used in case there are not already existing query
* parameters.
*/
private static final String ESCAPED_FRAGMENT_FORMAT2 = "&" + ESCAPED_FRAGMENT_FORMAT1;
private static final int ESCAPED_FRAGMENT_LENGTH2 = ESCAPED_FRAGMENT_FORMAT2.length();
private static final long _pumpEventLoopTimeoutMillis = 30000;
private static final long _jsTimeoutMillis = 1000;
private static final long _pageWaitMillis = 200;
private static final int _maxLoopChecks = 2;
private WebClient webClient;
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain filterChain) throws IOException, ServletException {
// Grab the request uri and query strings.
final HttpServletRequest httpRequest = (HttpServletRequest) request;
final String requestURI = httpRequest.getRequestURI();
final String queryString = httpRequest.getQueryString();
final HttpServletResponse httpResponse = (HttpServletResponse) response;
if ((queryString != null) && (queryString.contains(ESCAPED_FRAGMENT_FORMAT1))) {
final int port = httpRequest.getServerPort();
final String urlStringWithHashFragment = requestURI + rewriteQueryString(queryString);
final String scheme = httpRequest.getScheme();
final URL urlWithHashFragment = new URL(scheme, "127.0.0.1", port, urlStringWithHashFragment);
final WebRequest webRequest = new WebRequest(urlWithHashFragment);
log.fine("Crawl filter encountered escaped fragment, will open: " + webRequest.toString());
httpResponse.setContentType("text/html;charset=UTF-8");
final PrintWriter out = httpResponse.getWriter();
out.println(renderPage(webRequest));
out.flush();
out.close();
log.fine("HtmlUnit completed webClient.getPage(webRequest) where webRequest = " + webRequest.toString());
} else {
filterChain.doFilter(request, response);
}
}
#Override
public void destroy() {
if (webClient != null) {
webClient.closeAllWindows();
}
}
#Override
public void init(FilterConfig config) throws ServletException {
}
private StringBuilder renderPage(WebRequest webRequest) throws IOException {
webClient = new WebClient(BrowserVersion.FIREFOX_17);
webClient.getCache().clear();
webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setRedirectEnabled(false);
webClient.setAjaxController(new SyncAllAjaxController());
webClient.setCssErrorHandler(new SilentCssErrorHandler());
final HtmlPage page = webClient.getPage(webRequest);
webClient.getJavaScriptEngine().pumpEventLoop(_pumpEventLoopTimeoutMillis);
int waitForBackgroundJavaScript = webClient.waitForBackgroundJavaScript(_jsTimeoutMillis);
int loopCount = 0;
while (waitForBackgroundJavaScript > 0 && loopCount < _maxLoopChecks) {
++loopCount;
waitForBackgroundJavaScript = webClient.waitForBackgroundJavaScript(_jsTimeoutMillis);
if (waitForBackgroundJavaScript == 0) {
log.fine("HtmlUnit exits background javascript at loop counter " + loopCount);
break;
}
synchronized (page) {
log.fine("HtmlUnit waits for background javascript at loop counter " + loopCount);
try {
page.wait(_pageWaitMillis);
} catch (InterruptedException e) {
log.log(Level.SEVERE, "HtmlUnit ERROR on page.wait at loop counter " + loopCount, e);
}
}
}
webClient.getAjaxController().processSynchron(page, webRequest, false);
if (webClient.getJavaScriptEngine().isScriptRunning()) {
log.warning("HtmlUnit webClient.getJavaScriptEngine().shutdownJavaScriptExecutor()");
webClient.getJavaScriptEngine().shutdownJavaScriptExecutor();
}
final String staticSnapshotHtml = page.asXml();
StringBuilder stringBuilder = new StringBuilder();
stringBuilder.append("<hr />\n");
stringBuilder.append("<center><h3>This is a non-interactive snapshot for crawlers. Follow <a href=\"");
stringBuilder.append(webRequest.getUrl() + "\">this link</a> for the interactive application.<br></h3></center>");
stringBuilder.append("<hr />");
stringBuilder.append(staticSnapshotHtml);
return stringBuilder;
}
/**
* Maps from the query string that contains _escaped_fragment_ to one that
* doesn't, but is instead followed by a hash fragment. It also unescapes any
* characters that were escaped by the crawler. If the query string does not
* contain _escaped_fragment_, it is not modified.
*
* #param queryString
* #return A modified query string followed by a hash fragment if applicable.
* The non-modified query string otherwise.
* #throws UnsupportedEncodingException
*/
private static String rewriteQueryString(String queryString) throws UnsupportedEncodingException {
int index = queryString.indexOf(ESCAPED_FRAGMENT_FORMAT2);
int length = ESCAPED_FRAGMENT_LENGTH2;
if (index == -1) {
index = queryString.indexOf(ESCAPED_FRAGMENT_FORMAT1);
length = ESCAPED_FRAGMENT_LENGTH1;
}
if (index != -1) {
StringBuilder queryStringSb = new StringBuilder();
if (index > 0) {
queryStringSb.append("?");
queryStringSb.append(queryString.substring(0, index));
}
queryStringSb.append("#!");
queryStringSb.append(URLDecoder.decode(queryString.substring(index
+ length, queryString.length()), "UTF-8"));
return queryStringSb.toString();
}
return queryString;
}
}
I suggest having HtmlUnit generate the static html offline. You control the update frequency.
Then, have your servlet filter intercepting the crawler request return the already generated static html.
Related
I have an apache wicket panel for google recaptcha where I integrate wicket with google recaptcha and everything is working as it should, but I need to add an error message or feedbackpanel I am not quite good with wicket, so please advise me what is the best solution for my case?
public class ReCaptchaPanel extends Panel {
private static final long serialVersionUID = -8346261172610929107L;
private static final Logger log = LoggerFactory.getLogger(ReCaptchaPanel.class);
final String publicKey = "6LdrIgceAAAAADWF-gIz_OHJg_5gLu7IVw11hTnt";
final String secretKey = "6LdrIgceAAAAAE9qxHViBj3Sl8uyqUVdluugPRTq";
final String scriptClassName = "data-sitekey";
final String recCaptchaClassName = "g-recaptcha";
String currentLanguage = getLocale().getLanguage();
String scriptValue = "https://www.google.com/recaptcha/api.js?hl=" + currentLanguage;
/* in the following link you can find the language codes
* https://developers.google.com/recaptcha/docs/language
* */
public ReCaptchaPanel(String id) {
super(id);
final IFeedbackMessageFilter feedbackFilter = new ContainerFeedbackMessageFilter(this);
final FeedbackPanel feedbackPanel = new FeedbackPanel("feedback", feedbackFilter);
feedbackPanel.setOutputMarkupId(true);
add(feedbackPanel);
add(new Label("reCaptchaComponent", "").add(new SimpleAttributeModifier("class", recCaptchaClassName))
.add(new SimpleAttributeModifier(scriptClassName, publicKey)));
add(new Label("reCaptchaScript", "").add(new SimpleAttributeModifier("src", scriptValue)));
}
public ReCaptchaPanel(String id, IModel model) {
super(id, model);
}
/**
* Validates Google reCAPTCHA V2 .
*
* #param secretKey Secret key (key given for communication between arena and Google)
* #param response reCAPTCHA response from client side (g-recaptcha-response).
* #return true if validation successful, false otherwise.
*/
public synchronized boolean isCaptchaValid(String secretKey, String response) {
try {
String url = "https://www.google.com/recaptcha/api/siteverify",
params = "secret=" + secretKey + "&response=" + response;
HttpURLConnection http = (HttpURLConnection) new URL(url).openConnection();
http.setDoOutput(true);
http.setRequestMethod("POST");
http.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded; charset=UTF-8");
OutputStream out = http.getOutputStream();
out.write(params.getBytes(StandardCharsets.UTF_8));
out.flush();
out.close();
InputStream res = http.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(res, StandardCharsets.UTF_8));
StringBuilder sb = new StringBuilder();
int cp;
while ((cp = rd.read()) != -1) {
sb.append((char) cp);
}
JSONObject json = new JSONObject(sb.toString());
res.close();
log.info("Google ReCaptcha has been verified");
return json.getBoolean("success");
} catch (Exception e) {
e.printStackTrace();
}
log.error(" Google ReCaptcha failed");
return false;
}
public synchronized boolean isCaptchaValid() {
HttpServletRequest httpServletRequest = ApplicationBase.getHttpServletRequest();
assert httpServletRequest != null;
String response = httpServletRequest.getParameter("g-recaptcha-
response");
return isCaptchaValid(secretKey, response);
}
}
I use this panel in another wicket component ( form) ,
class FormEmail extends Form {
.........
final ReCaptchaPanel reCaptchaPanel = new ReCaptchaPanel("reCaptchaPanel") {
/**
* #see org.apache.wicket.Component#isVisible()
*/
#Override
public boolean isVisible() {
SessionBase session = (SessionBase) getSession();
ItemListPanelConfigParams configParams = new ItemListPanelConfigParams(session);
return configParams.isShowCaptcha();
}
};
add(reCaptchaPanel);
Button buttonSend = new Button("buttonSend") {
#Override
public void onSubmit() {
onSendCallback(emailRecipient, emailReplyTo, comment);
setResponsePage(callbackPage);
}
};
what i need to achieve is that add a message when the verify method in code above is false; I mean an error message, anyone knows how.
I am trying to upload a file into a server through okhttp, I am using a form so using the MultipartBuilder.
OkHttpClient client = new OkHttpClient();
MediaType OCTET_STREAM = MediaType.parse("application/octet-stream");
RequestBody body = new MultipartBuilder()
.type(MultipartBuilder.FORM)
.addPart(Headers.of("Content-Disposition",
"form-data; name=\"breadcrumb\"; filename=\"myfile.bin\"",
"Content-Transfer-Encoding", "binary"),
RequestBody.create(OCTET_STREAM, file))
.build();
Request request = new Request.Builder()
.header("Authorization", cred)
.url(url)
.post(body)
.build();
Response response = client.newCall(request).execute();
But when tracing what goes on the wire:
POST /breadcrumb/ HTTP/1.1
Authorization: Basic aHl6OmhvbGExMjM=
Content-Type: multipart/form-data; boundary=9bc835d6-24b8-42c4-ae8d-5bc89b3fe68f
Transfer-Encoding: chunked
Host: myurl:8000
Connection: Keep-Alive
Accept-Encoding: gzip
10e
--9bc835d6-24b8-42c4-ae8d-5bc89b3fe68f
Content-Disposition: form-data; name="breadcrumb"; filename="myfile.bin"
Content-Transfer-Encoding: binary
Content-Type: application/octet-stream
Content-Length: 21
some file content in binary
--9bc835d6-24b8-42c4-ae8d-5bc89b3fe68f--
0
When I use the traditional Apache http builders, it looks similar, but I don't see the strange characters at the beginning and at the end (10e, 0). Any ideas?
Thanks for your help.
You didn't specified exact Content-Length header, so OkHttpClient started to use chunked transfer encoding.
In HTTP protocol receiver must always know exact length of content(for purpose of allocating memory or other resources) before content will be realy send to server. There is two ways to send it - whole length of content in Content-Length header or by using chucked encoding if content-length can't be calculated at start of request.
This line:
10e
is just says that after that line client will send part of some data with length 0x10e (270) bytes.
The current MultipartBuilder implementation does not support setting a fixed Content-Length. One option would be implementing a FixedMultipartBuilder, which does its magic in the public long contentLength() method. Instead of returning -1L it now computes the length, e.g.:
import com.squareup.okhttp.Headers;
import com.squareup.okhttp.MediaType;
import com.squareup.okhttp.MultipartBuilder;
import com.squareup.okhttp.RequestBody;
import com.squareup.okhttp.internal.Util;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
import okio.BufferedSink;
import okio.ByteString;
/**
* Fluent API to build <a href="http://www.ietf.org/rfc/rfc2387.txt">RFC
* 2387</a>-compliant request bodies.
*/
public final class FixedMultipartBuilder {
private static final byte[] COLONSPACE = { ':', ' ' };
private static final byte[] CRLF = { '\r', '\n' };
private static final byte[] DASHDASH = { '-', '-' };
private final ByteString boundary;
private MediaType type = MultipartBuilder.MIXED;
// Parallel lists of nullable headers and non-null bodies.
private final List<Headers> partHeaders = new ArrayList<>();
private final List<RequestBody> partBodies = new ArrayList<>();
/** Creates a new multipart builder that uses a random boundary token. */
public FixedMultipartBuilder() {
this(UUID.randomUUID().toString());
}
/**
* Creates a new multipart builder that uses {#code boundary} to separate
* parts. Prefer the no-argument constructor to defend against injection
* attacks.
*/
public FixedMultipartBuilder(String boundary) {
this.boundary = ByteString.encodeUtf8(boundary);
}
/**
* Set the MIME type. Expected values for {#code type} are
* {#link com.squareup.okhttp.MultipartBuilder#MIXED} (the default),
* {#link com.squareup.okhttp.MultipartBuilder#ALTERNATIVE},
* {#link com.squareup.okhttp.MultipartBuilder#DIGEST},
* {#link com.squareup.okhttp.MultipartBuilder#PARALLEL} and
* {#link com.squareup.okhttp.MultipartBuilder#FORM}.
*/
public FixedMultipartBuilder type(MediaType type) {
if (type == null) {
throw new NullPointerException("type == null");
}
if (!type.type().equals("multipart")) {
throw new IllegalArgumentException("multipart != " + type);
}
this.type = type;
return this;
}
/** Add a part to the body. */
public FixedMultipartBuilder addPart(RequestBody body) {
return addPart(null, body);
}
/** Add a part to the body. */
public FixedMultipartBuilder addPart(Headers headers, RequestBody body) {
if (body == null) {
throw new NullPointerException("body == null");
}
if (headers != null && headers.get("Content-Type") != null) {
throw new IllegalArgumentException("Unexpected header: Content-Type");
}
if (headers != null && headers.get("Content-Length") != null) {
throw new IllegalArgumentException("Unexpected header: Content-Length");
}
partHeaders.add(headers);
partBodies.add(body);
return this;
}
/**
* Appends a quoted-string to a StringBuilder.
*
* <p>RFC 2388 is rather vague about how one should escape special characters
* in form-data parameters, and as it turns out Firefox and Chrome actually
* do rather different things, and both say in their comments that they're
* not really sure what the right approach is. We go with Chrome's behavior
* (which also experimentally seems to match what IE does), but if you
* actually want to have a good chance of things working, please avoid
* double-quotes, newlines, percent signs, and the like in your field names.
*/
private static StringBuilder appendQuotedString(StringBuilder target, String key) {
target.append('"');
for (int i = 0, len = key.length(); i < len; i++) {
char ch = key.charAt(i);
switch (ch) {
case '\n':
target.append("%0A");
break;
case '\r':
target.append("%0D");
break;
case '"':
target.append("%22");
break;
default:
target.append(ch);
break;
}
}
target.append('"');
return target;
}
/** Add a form data part to the body. */
public FixedMultipartBuilder addFormDataPart(String name, String value) {
return addFormDataPart(name, null, RequestBody.create(null, value));
}
/** Add a form data part to the body. */
public FixedMultipartBuilder addFormDataPart(String name, String filename, RequestBody value) {
if (name == null) {
throw new NullPointerException("name == null");
}
StringBuilder disposition = new StringBuilder("form-data; name=");
appendQuotedString(disposition, name);
if (filename != null) {
disposition.append("; filename=");
appendQuotedString(disposition, filename);
}
return addPart(Headers.of("Content-Disposition", disposition.toString()), value);
}
/** Assemble the specified parts into a request body. */
public RequestBody build() {
if (partHeaders.isEmpty()) {
throw new IllegalStateException("Multipart body must have at least one part.");
}
return new MultipartRequestBody(type, boundary, partHeaders, partBodies);
}
private static final class MultipartRequestBody extends RequestBody {
private final ByteString boundary;
private final MediaType contentType;
private final List<Headers> partHeaders;
private final List<RequestBody> partBodies;
public MultipartRequestBody(MediaType type, ByteString boundary, List<Headers> partHeaders,
List<RequestBody> partBodies) {
if (type == null) throw new NullPointerException("type == null");
this.boundary = boundary;
this.contentType = MediaType.parse(type + "; boundary=" + boundary.utf8());
this.partHeaders = Util.immutableList(partHeaders);
this.partBodies = Util.immutableList(partBodies);
}
#Override public MediaType contentType() {
return contentType;
}
private long contentLengthForPart(Headers headers, RequestBody body) throws IOException {
// Check if the body has an contentLength != -1, otherwise cancel!
long bodyContentLength = body.contentLength();
if(bodyContentLength < 0L) {
return -1L;
}
long length = 0;
length += DASHDASH.length + boundary.size() + CRLF.length;
if (headers != null) {
for (int h = 0, headerCount = headers.size(); h < headerCount; h++) {
length += headers.name(h).getBytes().length
+ COLONSPACE.length
+ headers.value(h).getBytes().length
+ CRLF.length;
}
}
MediaType contentType = body.contentType();
if (contentType != null) {
length += "Content-Type: ".getBytes().length
+ contentType.toString().getBytes().length
+ CRLF.length;
}
length += CRLF.length;
length += bodyContentLength;
length += CRLF.length;
return length;
}
#Override public long contentLength() throws IOException {
long length = 0;
for (int p = 0, partCount = partHeaders.size(); p < partCount; p++) {
long contentPartLength = contentLengthForPart(partHeaders.get(p), partBodies.get(p));
if(contentPartLength < 0) {
// Too bad, can't get contentPartLength!
return -1L;
}
length += contentPartLength;
}
length += DASHDASH.length + boundary.size() + DASHDASH.length + CRLF.length;
return length;
}
#Override public void writeTo(BufferedSink sink) throws IOException {
for (int p = 0, partCount = partHeaders.size(); p < partCount; p++) {
Headers headers = partHeaders.get(p);
RequestBody body = partBodies.get(p);
sink.write(DASHDASH);
sink.write(boundary);
sink.write(CRLF);
if (headers != null) {
for (int h = 0, headerCount = headers.size(); h < headerCount; h++) {
sink.writeUtf8(headers.name(h))
.write(COLONSPACE)
.writeUtf8(headers.value(h))
.write(CRLF);
}
}
MediaType contentType = body.contentType();
if (contentType != null) {
sink.writeUtf8("Content-Type: ")
.writeUtf8(contentType.toString())
.write(CRLF);
}
// Skipping the Content-Length for individual parts
sink.write(CRLF);
partBodies.get(p).writeTo(sink);
sink.write(CRLF);
}
sink.write(DASHDASH);
sink.write(boundary);
sink.write(DASHDASH);
sink.write(CRLF);
}
}
}
Sorry for the long listing... Sadly the MultipartBuilder is final so we have to copy most of the source instead of simply extend it.
I have a Java application that leverages the OBIEE Web Service API to consume data from the BI Server. I am able to XMLViewService and the WebCatalogService just fine, but I can't quite get the HtmlViewService to properly render a report in the Java app. The report just shows the spinning loader, but never actually renders the report. I'm pretty sure it has to do with the fact that the Java app and the BI Server are on different domains. This is what the API documentation says:
In situations where Oracle BI Web Services and the third-party Web server do not belong to the same Domain Name Service (DNS) domain, users may get JavaScript errors related to browser security constraints for cross-domain scripting. To avoid these issues, use the setBridge() method to modify callback URLs to point to the third-party Web server. Be aware that a Web component executed by the third-party Web server to re-route requests to Oracle BI Web Services is not provided. This function would need to be fulfilled by the third-party application.
Several years ago, I did this same type of integration using .NET/C# and ran in the the same issue because the .NET app and the BI Server were on different domains. As a result, I had to create an HTTP Handler (.ashx file) as well as use the setBridge() method to to solve the issue.
The challenge that I'm having is that I can't find a servlet bridge example for Java. And I'm not too confident in porting the .NET/.ASHX code to a Java servlet/bridge. Does anyone have any code examples or direction they could provide to point me in the right direction? Here's a snippet of code to show you what I'm doing to pull back the report data:
// define report path
ReportRef reportRef = new ReportRef();
reportRef.setReportPath(reportFolder + "/" + reportName);
// set page params
StartPageParams pageParams = new StartPageParams();
pageParams.setDontUseHttpCookies(true);
// set report params
String pageId = htmlService.startPage(pageParams, sawSessionId);
String reportId = pageId + reportName;
htmlService.addReportToPage(pageId, reportId, reportRef, null, null, null, sawSessionId);
// get report html
StringBuffer reportHtml = new StringBuffer();
reportHtml.append(htmlService.getHtmlForReport(pageId, reportId, sawSessionId));
// return html
return reportHtml.toString();
This is the error that is coming back in the browser:
XMLHttpRequest cannot load http://myobiserver.com/analytics/saw.dll?ajaxGo. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://myjavaapp.com' is therefore not allowed access.
Per requested, here is my .NET/.ASHX bridge:
using System.Collections.Specialized;
using System.Net;
using System.Text;
using System.Web;
using System;
using System.Collections;
using System.Configuration;
using System.Collections.Specialized;
using System.Web;
using System.Text;
using System.Net;
using System.IO;
using System.Diagnostics;
/*
This is a ASP.NET handler that handles communication
between the SharePoint site and OracleBI.
It will be deployed to:
C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\LAYOUTS\OracleBI
*/
public class OracleBridge: IHttpHandler
{
public bool IsReusable {get{return true;}}
public OracleBridge()
{
}
string getServer()
{
string strServer = "http://<enter-domain>/analytics/saw.dll";
int index = strServer.LastIndexOf("/");//split off saw.dll
if (index >=0)
return strServer.Substring(0,index+1);
else
return strServer;
}
public void ProcessRequest(HttpContext context)
{
HttpWebRequest req = forwardRequest(context);
forwardResponse(context,req);
}
private HttpWebRequest forwardRequest(HttpContext context)
{
string strURL = makeURL(context);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(strURL);
req.Method = context.Request.HttpMethod;
NameValueCollection headers = context.Request.Headers;
req.Accept = headers.Get("Accept");
req.Expect = headers.Get("Expect");
req.ContentType = headers.Get("Content-Type");
string strModifiedSince = headers.Get("If-Modified-Since");
if (strModifiedSince != null && strModifiedSince.Length != 0)
req.IfModifiedSince = DateTime.Parse(strModifiedSince);
req.Referer = headers.Get("Referer");
req.UserAgent = headers.Get("User-Agent");
if (!req.Method.Equals("GET"))
{
CopyStreams(context.Request.InputStream,req.GetRequestStream());
}
return req;
}
private void forwardResponse(HttpContext context, HttpWebRequest req)
{
HttpWebResponse resp =null;
try
{
resp = (HttpWebResponse)req.GetResponse();
}
catch(WebException e)
{
resp = (HttpWebResponse)e.Response;
}
context.Response.StatusCode = (int)resp.StatusCode;
for (int i = 0; i < resp.Cookies.Count; i++)
{
Cookie c = resp.Cookies[i];
HttpCookie hc = new HttpCookie(c.Name, c.Value);
hc.Path = c.Path;
hc.Domain = getServer();
context.Response.Cookies.Add(hc);
}
context.Response.ContentType = resp.ContentType;
CopyStreams(resp.GetResponseStream(), context.Response.OutputStream);
}
private string makeURL(HttpContext context)
{
string strQuery = context.Request.Url.Query;
string[] arrParams = strQuery.Split('?','&');
StringBuilder resultingParams = new StringBuilder();
string strURL=null;
foreach(string strParam in arrParams )
{
string[] arrNameValue = strParam.Split('=');
if (!arrNameValue[0].Equals("RedirectURL"))
{
if (strParam.Length != 0)
{
if (resultingParams.Length != 0)
resultingParams.Append("&");
resultingParams.Append(strParam);
}
}
else if (arrNameValue.Length >1)
strURL = HttpUtility.UrlDecode(arrNameValue[1]);
}
if (strURL ==null)
throw new Exception("Invalid URL format. requestURL parameter is missing");
String sAppendChar = strURL.Contains("?") ? "&" : "?";
if (strURL.StartsWith("http:") || strURL.StartsWith("https:"))
{
String tmpURL = strURL + sAppendChar + resultingParams.ToString();
return tmpURL;
}
else
{
String tmpURL = getServer() + strURL + sAppendChar + resultingParams.ToString();
return tmpURL;
}
}
private void CopyStreams(Stream inStr,Stream outStr)
{
byte[] buf = new byte[4096];
try
{
do
{
int iRead = inStr.Read(buf,0,4096);
if (iRead == 0)
break;
outStr.Write(buf,0,iRead);
}
while (true);
}
finally
{
outStr.Close();
}
}
}
Using the BridgeServlet from link (http://pastebin.com/NibVnBLb) posted in previous answer did not work for us. In our web portal, embedding Oracle BI dashboard using BridgeServlet above, redirected us to OBI login page and console log showed incorrect resources(js/css) web links (local URL instead of OBIEE URL's).
Instead we used this class (with some minor adjustments) https://gist.github.com/rafaeltuelho/9376341#file-obieehttpservletbridge-java.
Tested with Java 11, Oracle Business Intelligence 12.2.1.4.0, WSDL v12 of OBIEE (http://OBIEE-server:port/analytics/saw.dll/wsdl/v12), with SSO disabled.
Here it is the BridgeServlet class:
package com.abs.bi;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLEncoder;
import java.util.Enumeration;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
import javax.servlet.ServletException;
import javax.servlet.http.Cookie;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
/**
* Servlet implementation class OBIEEBridge
*/
public class BridgeServlet extends HttpServlet {
private static final long serialVersionUID = 1L;
/**
* #see HttpServlet#HttpServlet()
*/
public BridgeServlet() {
super();
}
/**
* #see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse
* response)
*/
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
try {
this.processRequest(request, response);
} catch (Exception e) {
throw new ServletException(e);
}
}
/**
* #see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse
* response)
*/
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
try {
this.processRequest(request, response);
} catch (Exception e) {
throw new ServletException(e);
}
}
protected void processRequest(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
HttpURLConnection urlCon = forwardRequest(request);
forwardResponse(response, urlCon);
}
#SuppressWarnings("unchecked")
private String decodeURL(HttpServletRequest request) {
StringBuffer bufURL = new StringBuffer("");
Map<String, String[]> params = request.getParameterMap();
String[] arrURL = params.get("RedirectURL");
String strURL = arrURL == null || arrURL.length == 0 ? null : arrURL[0];
bufURL.append(strURL);
int nQIndex = strURL.lastIndexOf('?');
if (params != null && !params.isEmpty()) {
bufURL.append(((nQIndex >= 0) ? "&" : "?"));
Set<String> keys = params.keySet();
Iterator<String> it = keys.iterator();
while (it.hasNext()) {
try {
String strKey = it.next();
if (strKey.equalsIgnoreCase("RedirectURL")) {
continue;
}
String strEncodedKey = URLEncoder.encode(strKey, "UTF-8");
String[] paramValues = params.get(strKey);
for (String paramValue : paramValues) {
bufURL.append(strEncodedKey);
bufURL.append("=");
bufURL.append(URLEncoder.encode(paramValue, "UTF-8"));
bufURL.append("&");
}
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
}
bufURL.deleteCharAt(bufURL.length() - 1);
}
return bufURL.toString();
}
#SuppressWarnings("unchecked")
private HttpURLConnection forwardRequest(HttpServletRequest request) throws IOException {
String strURL = decodeURL(request);
String[] arrURL = strURL.split("&", 2);
String baseURL = arrURL[0];
URL url = new URL(baseURL);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
String strMethod = request.getMethod();
con.setRequestMethod(strMethod);
Enumeration<String> en = request.getHeaderNames();
String strHeader;
while (en.hasMoreElements()) {
strHeader = en.nextElement();
String strHeaderValue = request.getHeader(strHeader);
con.setRequestProperty(strHeader, strHeaderValue);
}
// is not a HTTP GET request
if (strMethod.compareTo("GET") != 0) {
con.setDoOutput(true);
con.setDoInput(true);
con.setUseCaches(false);
DataOutputStream forwardStream = new DataOutputStream(con.getOutputStream());
try {
String urlParameters = arrURL[1];
forwardStream.writeBytes(urlParameters);
forwardStream.flush();
} finally {
forwardStream.close();
}
}
return con;
}
private void forwardResponse(HttpServletResponse response, HttpURLConnection con) throws IOException {
int nContentLen = -1;
String strKey;
String strValue;
try {
response.setStatus(con.getResponseCode());
for (int i = 1; true; ++i) {
strKey = con.getHeaderFieldKey(i);
strValue = con.getHeaderField(i);
if (strKey == null) {
break;
}
if (strKey.equals("Content-Length")) {
nContentLen = Integer.parseInt(con.getHeaderField(i));
continue;
}
if (strKey.equalsIgnoreCase("Connection") || strKey.equalsIgnoreCase("Server")
|| strKey.equalsIgnoreCase("Transfer-Encoding") || strKey.equalsIgnoreCase("Content-Length")) {
continue; // skip certain headers
}
if (strKey.equalsIgnoreCase("Set-Cookie")) {
String[] cookieStr1 = strValue.split(";");
String[] cookieStr2 = cookieStr1[0].split("=");
// String[] cookieStr3 = cookieStr1[1].split("=");
/*
* Change the Set-Cookie HTTP Header to remove the 'path' attribute. Thus the
* browser can accept the ORA_BIPS_NQID cookie from Oracle BI Server
*/
Cookie c = new Cookie(cookieStr2[0], cookieStr2[1]);
c.setPath("/");
response.addCookie(c);
} else {
response.setHeader(strKey, strValue);
}
}
copyStreams(con.getInputStream(), response.getOutputStream(), nContentLen);
} finally {
response.getOutputStream().close();
con.getInputStream().close();
}
}
private void copyStreams(InputStream inputStream, OutputStream forwardStream, int nContentLen) throws IOException {
byte[] buf = new byte[1024];
int nCount = 0;
int nBytesToRead = 1024;
int nTotalCount = 0;
do {
if (nContentLen != -1)
nBytesToRead = nContentLen - nTotalCount > 1024 ? 1024 : nContentLen - nTotalCount;
if (nBytesToRead == 0)
break;
// try to read some bytes from src stream
nCount = inputStream.read(buf, 0, nBytesToRead);
if (nCount < 0)
break;
nTotalCount += nCount;
// try to write some bytes in target stream
forwardStream.write(buf, 0, nCount);
} while (true);
}
}
AbsServiceUtils which contains SAWSessionService and HtmlViewService web service calls to Oracle BI server:
package com.abs.bi;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.jsp.JspWriter;
import oracle.bi.web.soap.AuthResult;
import oracle.bi.web.soap.HtmlViewService;
import oracle.bi.web.soap.HtmlViewServiceSoap;
import oracle.bi.web.soap.ReportHTMLLinksMode;
import oracle.bi.web.soap.ReportHTMLOptions;
import oracle.bi.web.soap.ReportRef;
import oracle.bi.web.soap.SAWLocale;
import oracle.bi.web.soap.SAWSessionParameters;
import oracle.bi.web.soap.SAWSessionService;
import oracle.bi.web.soap.SAWSessionServiceSoap;
import oracle.bi.web.soap.StartPageParams;
public class AbsServiceUtils {
private AbsServiceUtils() {
}
public static URL buildWsdlUrl() throws MalformedURLException {
return new URL(IAbsService.BASEWSDLURL);
}
public static void writeBiContent(HttpServletRequest request, JspWriter out, String biReport) throws IOException {
String userAgent = request.getHeader("User-Agent");
Locale userLocale = request.getLocale();
String bridgeServletContextPath = request.getContextPath() + "/bridgeservlet";
String reportHtml = writeBiContent(biReport, userAgent, userLocale, bridgeServletContextPath);
if (out != null) {
out.println(reportHtml);
}
}
public static String writeBiContent(String biReport, String userAgent, Locale userLocale,
String bridgeServletContextPath) throws MalformedURLException {
HtmlViewService htmlViewService = new HtmlViewService(buildWsdlUrl());
HtmlViewServiceSoap htmlClient = htmlViewService.getHtmlViewService();
SAWSessionService sAWSessionService = new SAWSessionService(buildWsdlUrl());
SAWSessionServiceSoap myPort = sAWSessionService.getSAWSessionServiceSoap();
SAWSessionParameters sessionparams = new SAWSessionParameters();
sessionparams.setUserAgent(userAgent);
SAWLocale sawlocale = new SAWLocale();
sawlocale.setLanguage(userLocale.getLanguage());
sawlocale.setCountry(userLocale.getCountry());
sessionparams.setLocale(sawlocale);
sessionparams.setAsyncLogon(false);
AuthResult result = myPort.logonex(IAbsService.BIUSERNAME, IAbsService.BIPASSWORD, sessionparams);
String sessionID = result.getSessionID();
List<String> keepAliveSessionList = new ArrayList<>(1);
keepAliveSessionList.add(sessionID);
myPort.keepAlive(keepAliveSessionList);
StartPageParams spparams = new StartPageParams();
spparams.setDontUseHttpCookies(true);
String pageID = htmlClient.startPage(spparams, sessionID);
/**
* This method will set the path to the servlet which will act like a bridge to
* retrieve all the OBIEE resources like the javascript, CSS and the report.
*/
if (bridgeServletContextPath != null) {
htmlClient.setBridge(bridgeServletContextPath, sessionID);
}
ReportHTMLOptions htmlOptions = new ReportHTMLOptions();
htmlOptions.setEnableDelayLoading(false);
htmlOptions.setLinkMode(ReportHTMLLinksMode.IN_PLACE.value());
ReportRef reportref = new ReportRef();
reportref.setReportPath(IAbsService.BIROOTPATH + biReport);
StartPageParams startpageparams = new StartPageParams();
startpageparams.setDontUseHttpCookies(false);
htmlClient.addReportToPage(pageID, biReport.replace(" ", ""), reportref, null, null, htmlOptions, sessionID);
String reportHtml = htmlClient.getHeadersHtml(pageID, sessionID);
reportHtml = reportHtml + htmlClient.getHtmlForReport(pageID, biReport.replace(" ", ""), sessionID);
reportHtml = reportHtml + htmlClient.getCommonBodyHtml(pageID, sessionID);
return reportHtml;
}
}
IAbsService:
package com.abs.bi;
public interface IAbsService {
public static final String BASEWSDLURL = "http://<OracleBIServer:port>/analytics/saw.dll/wsdl/v12";
public static final String BIUSERNAME = "USER";
public static final String BIPASSWORD = "PASS";
public static final String BIROOTPATH = "/shared/sharedfolder/";
public static final String BIREPORTNAME = "report";
}
Use the link http://pastebin.com/NibVnBLb to check out the bridge code for java. Hope this might be helful.
For some reason I'm getting null pointer exception. It's downloading the image here and logcat points me to call
public Result call(final String method, final String apiKey, final String... params) {
return call(method, apiKey, map(params));
}
/**
* Performs the web-service call. If the <code>session</code> parameter is
* <code>non-null</code> then an authenticated call is made. If it's
* <code>null</code> then an unauthenticated call is made.<br/>
* The <code>apiKey</code> parameter is always required, even when a valid
* session is passed to this method.
*
* #param method The method to call
* #param apiKey A Last.fm API key
* #param params Parameters
* #param session A Session instance or <code>null</code>
* #return the result of the operation
*/
public Result call(final String method, final String apiKey, Map<String, String> params) {
params = new WeakHashMap<String, String>(params);
InputStream inputStream = null;
// no entry in cache, load from web
if (inputStream == null) {
// fill parameter map with apiKey and session info
params.put(PARAM_API_KEY, apiKey);
try {
final HttpURLConnection urlConnection = openPostConnection(method, params);
inputStream = getInputStreamFromConnection(urlConnection);
if (inputStream == null) {
lastResult = Result.createHttpErrorResult(urlConnection.getResponseCode(),
urlConnection.getResponseMessage());
return lastResult;
}
} catch (final IOException ignored) {
}
}
try {
final Result result = createResultFromInputStream(inputStream);
lastResult = result;
return result;
} catch (final IOException ignored) {
} catch (final SAXException ignored) {
}
return null;
}
It finally cracks at the line "new InputSource(new InputStreamReader(inputStream, "UTF-8")));".
/**
* #param inputStream
* #return
* #throws SAXException
* #throws IOException
*/
private Result createResultFromInputStream(final InputStream inputStream) throws SAXException,
IOException {
final Document document = newDocumentBuilder().parse(
new InputSource(new InputStreamReader(inputStream, "UTF-8")));
final Element root = document.getDocumentElement(); // lfm element
final String statusString = root.getAttribute("status");
final Status status = "ok".equals(statusString) ? Status.OK : Status.FAILED;
if (status == Status.FAILED) {
final Element errorElement = (Element)root.getElementsByTagName("error").item(0);
final int errorCode = Integer.parseInt(errorElement.getAttribute("code"));
final String message = errorElement.getTextContent();
return Result.createRestErrorResult(errorCode, message);
} else {
return Result.createOkResult(document);
}
}
Any ideas? I have no idea what might be wrong. If sufficient info is provided then let me know - I'll get what you need. I'm a beginner. :)
Maybe I do not understand the servlet lifecycle very well, but this is what i want:
I want to display a page generated by a servlet let's say servlet: paginaAmd.
On this page I want to display a list of images stored in folder on web server.
The address of url of the images is something like:
/img/80-80-1-1-1-1-1-1-1-1-1
where /img/* is my servlet for displaying images.
All works well if I want to display one image at a time in browser.
But when I want to put all the list at once, the images are not displayed correctly. Sometimes are not displayed at all, sometimes are displayed in wrong position (the position does not alter in time), and sometimes are displayed only some images.
I suspect that somehow not all the doGet() methods are catched.
Can someone give me some advice?
Here are the servlet code witch is implemented by the tutorial here: http://balusc.blogspot.fr/2007/04/imageservlet.html
#WebServlet(name = "ImgDisplay", urlPatterns = {"/img/*"})
public class ImgDisplay extends HttpServlet
{
private SessionFactory sessionfactory = new AnnotationConfiguration().configure().buildSessionFactory();
private Query query;
private String mesajEroare = "";
private HttpServletRequest _request;
private HttpServletResponse _response;
private int width = 0;
private int height = 0;
private int idImagine = 0;
private int format = 0;
private String titluArticol = "";
private String numeImagine = "";
private boolean imgValida = false;
private int DEFAULT_BUFFER_SIZE = 1024 * 100;
String fileUploadPath = "";
#Override
public void init() throws ServletException {
}
#Override
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{
this._request = request;
this._response = response;
this.SetVariabile();
if(imgValida)
{
String nImagine = this.GetImageFromDisk();
this.DisplayImage(nImagine);
}
}
private void SetVariabile()
{
String reqUrl = _request.getRequestURL().toString();
String aUrl[] = reqUrl.split("/");
String urlImg = aUrl[aUrl.length - 1];
aUrl = urlImg.split("-");
try
{
this.width = Integer.parseInt(aUrl[0]);
this.height = Integer.parseInt(aUrl[1]);
this.idImagine = Integer.parseInt(aUrl[2]);
this.format = Integer.parseInt(aUrl[3]);
this.numeImagine = aUrl[aUrl.length - 1];
this.imgValida = true;
}
catch(Exception e)
{
this.imgValida = false;
}
}
private String GetImageFromDisk()
{
String nImagine;
//preiau imaginea
PaginiImagini pa = new PaginiImagini();
Session session;
try
{
session = sessionfactory.openSession();
session.beginTransaction();
query = session.getNamedQuery("PaginiImagini.findByImagineID");
query.setInteger("imagineID", this.idImagine);
pa = (PaginiImagini) query.uniqueResult();
session.getTransaction().commit();
session.close();
}
catch( Exception e )
{
this.mesajEroare = "Nu pot citi din baza de date!";
}
// citesc imagine de pe disk
ServletContext sctx = getServletContext();
this.fileUploadPath = sctx.getInitParameter("file-upload-path");
String pathImagine = this.fileUploadPath + "/" + Setari.pathImaginiMici;
if(this.width > Setari.wImagineMica || this.height > Setari.hImagineMica)
{
pathImagine = this.fileUploadPath + "/" + Setari.pathImaginiMari;
}
nImagine = pathImagine + "/" + pa.getNumeImaginePeDisc();
return nImagine;
}
private void DisplayImage(String imageToRead) throws FileNotFoundException, IOException
{
File image = new File(imageToRead);
String contentType = getServletContext().getMimeType(image.getName());
_response.setContentType(contentType);
_response.setHeader("Content-Length", String.valueOf(image.length()));
_response.setHeader("Content-Disposition", "inline; filename=\"" + image.getName() + "\"");
_response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
_response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
_response.setDateHeader("Expires", 0); // Proxies.
// Prepare streams.
BufferedInputStream input = null;
BufferedOutputStream output = null;
try
{
// Open streams.
input = new BufferedInputStream(new FileInputStream(image), DEFAULT_BUFFER_SIZE);
output = new BufferedOutputStream(_response.getOutputStream(), DEFAULT_BUFFER_SIZE);
// Write file contents to response.
byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
int length;
while ((length = input.read(buffer)) > 0)
{
output.write(buffer, 0, length);
}
}
finally
{
// Gently close streams.
close(output);
close(input);
}
}
/**
*
* #param resource
*/
private static void close(Closeable resource)
{
if (resource != null)
{
try
{
resource.close();
}
catch (IOException e)
{
// Do your thing with the exception. Print it, log it or mail
// it.
//e.printStackTrace();
}
}
}
}
You have serious concurrency issues in your servlet. A single instance of the servlet is used to serve all the requests to this servlet. So a servlet should be stateless. But the first thing you're doing is
this._request = request;
this._response = response;
This means that if two concurrent requests are made to the servlet, you might have the first one set these two instance variables, then the second one resetting the same instance variables. The first image would thus be sent as a response to the second request, and nothing would be sent as a response to the first request. And this is only one of the strange things that could happen. You could also have exceptions and inconsistent data.
Don't store the request and response (and any other state) in instance variables. Pass them from method to method. I've not analyzed the whole code, but the only instance field that you should have in the servlet is sessionFactory field.