I am new to HttpClient and I'm trying to get the prices of players in the fifa ultimate team web app.
Using Developer Tools, i managed to find the Post Method yielding the prices of the player.
Searching for a specific player gives the following URL request.
https://utas.s2.fut.ea.com/ut/game/fifa15/transfermarket?maskedDefId=20801&start=0&num=16&type=player
When I'm viewing the response of the method, I can see all the player prices currently on the market in Developer Tools.
So I tried to code it in Java to get those prices, but my output is nothing like the response given in Developer Tools.
This is my code:
HttpClient client = new DefaultHttpClient();
String url = "https://utas.s2.fut.ea.com/ut/game/fifa15/transfermarket?maskedDefId=20801&start=0&num=16&type=player";
HttpPost loginPost = new HttpPost(url);
loginPost.setHeader("Host", "utas.s2.fut.ea.com");
loginPost.setHeader("Connection", "keep-alive");
loginPost.setHeader("Content-Lenght", "1");
loginPost.setHeader("Cache-Control", "max-age=0");
loginPost.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
loginPost.setHeader("Origin", "https://www.easports.com");
loginPost.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36");
loginPost.setHeader("Content-Type", "application/json");
loginPost.setHeader("Accept", "application/json");
loginPost.setHeader("X-UT-SID", "23c19a93-32ac-4f52-9aab-21c6be112ede");
loginPost.setHeader("X-UT-Embed-Error", "true");
loginPost.setHeader("X-UT-PHISHING-TOKEN", "8649502695209937021");
loginPost.setHeader("X-HTTP-Method-Override", "GET");
loginPost.setHeader("Referer", "https://www.easports.com/iframe/fut15/bundles/futweb/web/flash/FifaUltimateTeam.swf?cl=144469");
loginPost.setHeader("Accept-Encoding", "gzip,deflate");
loginPost.setHeader("Accept-Language", "nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4");
loginPost.setHeader("Cookie", "CEM-Rememberlogin=mRLLf4Ua9QJAz1J1va8yrhwlh9jFEaLsccY1Q3Ja7wREWecJnF3H348jrRpTnr3qXnhX-YKV6TH3HWgEPqgExDUP2vMFeiwkAYSkhyMuJUyHF_Fsou-fiv7Ur2XVYpqVo1KEP_dO0lLaJ5zTV1I8paWera57lPKmWxY5KzOdnRTGBbpbBv2bJOVPlGLpUFWmbOdMxElcTQga7S2-nM3icZc0JQQ2gmtDCKJ48VHSTl3x3BEuT3ltfNvFEt8elFIf; Path=/");
List <NameValuePair> nvps = new ArrayList <NameValuePair>();
nvps.add(new BasicNameValuePair("maskedDefId", "20801"));
nvps.add(new BasicNameValuePair("num", "16"));
nvps.add(new BasicNameValuePair("start", "0"));
nvps.add(new BasicNameValuePair("type", "player"));
loginPost.setEntity(new UrlEncodedFormEntity(nvps));
HttpResponse response = client.execute(loginPost);
HttpEntity entity = response.getEntity();
String content = EntityUtils.toString(entity);
System.out.println(content);
This is the response Developer Tools is showing and what I need:
{"auctionInfo":[{"itemData":{"id":164674417976,"timestamp":1412181858,"itemType":"player","pile":5,"resourceId":1929400641,"rating":93,"leagueId":0, etc}]}}
This is my response:
?[][][][][][][][]?V?M-.NLOU??+???Q*JM,??S?RJ?(?,JMQ([]?[]G[]Et???S??L[][]k[]??[]?6[][][]
where [] stands for the unknown character (square)
So my question is: How do I adjust the code such that I will get the response of the requested URL?
Any help would be appreciated
Edit:
All I had to do was to delete some 'setted' headers:
Those are the 'setted' headers I still need:
loginPost.setHeader("Host", "utas.s2.fut.ea.com");
loginPost.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36");
loginPost.setHeader("X-UT-SID", "23c19a93-32ac-4f52-9aab-21c6be112ede");
loginPost.setHeader("X-UT-Embed-Error", "true");
loginPost.setHeader("X-UT-PHISHING-TOKEN", "8649502695209937021");
loginPost.setHeader("X-HTTP-Method-Override", "GET");
But here, the values of "X-UT-SID" and "X-UT-PHISHING-TOKEN" are not correct, they are not the actual values (since I copied them from a previous session). How should I fix my code in order to get the actual value?
Thanks in advance
Is there some documentation? Maybe you have to set manually the encoding, like:
loginPost.setHeader("Content-Type", "application/json; charset=utf-8");
maybe change "utf-8" with other charset (like 'iso-8859-1')
Related
I want to make a API call using graphql:
URL url = new URL("https://www.some_web_site.com/voyager/api/graphql?variables=(jobCardPrefetchQuery:(jobUseCase:JOB_DETAILS,prefetchJobPostingCardUrns:List(urn%3Ali%3Afsd_jobPostingCard%3A%283381613144%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283309215638%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283390173915%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283384739773%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283349746057%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283399364227%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283346758701%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283339724174%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283394131711%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283376993869%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283367203416%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283391072254%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283378145979%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283386882674%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283363291070%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283379552483%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283384379850%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283384189666%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283357674221%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%282921934527%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283360994137%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283400914209%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283322816290%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283389955425%2CJOB_DETAILS%29,urn%3Ali%3Afsd_jobPostingCard%3A%283385984792%2CJOB_DETAILS%29)))&&queryId=voyagerJobsDashJobCards.a2332c3024c06e104d995060b03b3e43");
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
httpConn.setRequestMethod("GET");
httpConn.setRequestProperty("authority", "www.some_web_site.com");
httpConn.setRequestProperty("accept", "application/vnd.some_web_site.normalized+json+2.1");
httpConn.setRequestProperty("accept-language", "en-US,en;q=0.9");
httpConn.setRequestProperty("cookie", "bcookie=\"v=2&6fc50d25-9d9c-4c9c-8e15-cb85a0d0487b\"; bscookie=\"v=1&20210323132219381954c4-ad82-44b3-8d2c-22095b879042AQGEhxQJj6yW11ERtduVNxs5rtVSAMBp\"; li_rm=AQGGLfZ06r63ZgAAAXsWmDpJhcTdHaPCFAvu2x_4WmdidiGZZgHI2VCkG9Vcm15E52dFwfTRZjD7-y6fFfZ3os-ouUFUUMAP9zUFRXtpppob6Jsk-5jywhWJ; _ga=GA1.2.978357381.1628171751; timezone=Europe/Sofia; aam_uuid=25646872215935551520582305049506167860; _gcl_au=1.1.798106900.1628171779; li_theme=system; li_alerts=e30=; __ssid=4461b759-d6fd-43a8-9aa6-6170e1b52f81; li_theme_set=app; VID=V_2022_01_19_10_1493; G_ENABLED_IDPS=google; li_gc=MTsyMTsxNjQ4NDI4MjQ1OzI7MDIxb+aBjuH+4tvdvwmua4v9UqyRyzc5sIUB38QjcHnEqQY=; visit=v=1&M; liap=true; JSESSIONID=\"ajax:3454879913095470865\"; s_fid=1B1937DEEC2C6D37-2F033B1588EC1876; mbox=PC#830b22e2c1204684880490b80b06cb60.34_0#1682989647|session#6f6b357e1c5b44d1ae3c653b5c83f989#1667439507; s_ips=937; AnalyticsSyncHistory=AQKuwzqgAo50uAAAAYUIpWEC9kS13rXinz-XQWIXmd4Fq4Glmch0GzNnqIVIJlRwZMhL95WX5GCfYDJE2sCHMw; lms_ads=AQGr99ghK-2XugAAAYUIpWHGhGJwOoz_n5sH3AoFmuwV4_X1HGg0oD9yS7BdMBufU7J1lFVWiLUsMINxUT9QnPJh0dCuicrl; lms_analytics=AQGr99ghK-2XugAAAYUIpWHGhGJwOoz_n5sH3AoFmuwV4_X1HGg0oD9yS7BdMBufU7J1lFVWiLUsMINxUT9QnPJh0dCuicrl; li_at=AQEDARqyax8BofFVAAABhQuvuGkAAAGFL7w8aU0Aq95MvKsqbsob115un5v2PCrAkDC55M2vjdQiOXIekRn__kFf4G3e8KbfOR4xHerEh33Y4o77t9HPCfG6eKAICbmbiWB6UzSjxy5ieOuLy_I05Zgm; lang=v=2&lang=en-us; AMCVS_14215E3D5995C57C0A495C55%40AdobeOrg=1; lil-lang=en_US; s_cc=true; gpv_pn=www.linkedin.com%2Flearning%2Fjava-ee-servlets-and-javaserver-pages-jsp; s_tp=3337; s_plt=4.10; s_pltp=www.linkedin.com%2Flearning%2Fjava-ee-servlets-and-javaserver-pages-jsp; s_ppv=www.linkedin.com%2Flearning%2Fjava-ee-servlets-and-javaserver-pages-jsp%2C67%2C28%2C2237%2C2%2C3; s_tslv=1671032611869; AMCV_14215E3D5995C57C0A495C55%40AdobeOrg=-637568504%7CMCIDTS%7C19340%7CMCMID%7C25469860276152368910638363702655569919%7CMCAAMLH-1671655210%7C6%7CMCAAMB-1671655210%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1671057610s%7CNONE%7CMCCIDH%7C-173138152%7CvVersion%7C5.1.1; UserMatchHistory=AQIkSvhSMBMwDAAAAYUSkHzxlcAhQiwkqVW-fnoKFql17EjfqYUVpnT9u_X3_wgEKxbgMP0NKPdWcqy4JyBytaa8r7aBuSeZ52xEM5ViomuA8YbDshzpGmWcjbySw4ifRsckhAY1PeMzxtWWCv3cO84T4Aii8VdzlvIdnF3YkuO8zxtroIYtVJoi1tzxAaDMhRy3xKoUoFQruT7B0iyg61cyguKbBgpUPS_DNWjviIET8kbC-ZdINHtUfodXMxtYmz6a2YOYivvIE2Qfwfy97wA8YCMB-rWKObet_sk; lidc=\"b=VB47:s=V:r=V:a=V:p=V:g=3532:u=864:x=1:i=1671053737:t=1671134164:v=2:sig=AQF22ngB_c7HxFphw3OcQBCH-Z1vEk3\"; li_mc=MTsy33sxNjcxMDU0MDYwOzI7MDIxWvtOdIZ8/F4IRqSFMX7wJYOWOToT2pAAzOP8rEr5pyA=; sdsc=22%3A1%2C1671054185039%7EJAPP%2C0Z6uuB8klhAtG9A5iEG19Pz7UGyA%3D");
httpConn.setRequestProperty("csrf-token", "ajax:3454129913095170865");
httpConn.setRequestProperty("referer", "https://www.some_web_site.com/jobs/search/?currentJobId=1332613144&geoId=103829153&keywords=java&location=Norway&refresh=true&start=950");
httpConn.setRequestProperty("sec-ch-ua", "\"Not?A_Brand\";v=\"8\", \"Chromium\";v=\"108\", \"Google Chrome\";v=\"108\"");
httpConn.setRequestProperty("sec-ch-ua-mobile", "?0");
httpConn.setRequestProperty("sec-ch-ua-platform", "\"Windows\"");
httpConn.setRequestProperty("sec-fetch-dest", "empty");
httpConn.setRequestProperty("sec-fetch-mode", "cors");
httpConn.setRequestProperty("sec-fetch-site", "same-origin");
httpConn.setRequestProperty("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36");
httpConn.setRequestProperty("x-li-lang", "en_US");
httpConn.setRequestProperty("x-li-page-instance", "urn:li:page:d_flagship3_search_srp_jobs;HsFe0k8FR5a+hftD/cGHXA==");
httpConn.setRequestProperty("x-li-track", "{\"clientVersion\":\"1.11.5479\",\"mpVersion\":\"1.11.5479\",\"osName\":\"web\",\"timezoneOffset\":2,\"timezone\":\"Europe/Sofia\",\"deviceFormFactor\":\"DESKTOP\",\"mpName\":\"voyager-web\",\"displayDensity\":1,\"displayWidth\":1920,\"displayHeight\":1080}");
httpConn.setRequestProperty("x-restli-protocol-version", "2.0.0");
InputStream responseStream = httpConn.getResponseCode() / 100 == 2
? httpConn.getInputStream()
: httpConn.getErrorStream();
Scanner s = new Scanner(responseStream).useDelimiter("\\A");
String response = s.hasNext() ? s.next() : "";
System.out.println(response);
But for some reason I get unreadable response when the call is made using the above Java code:
xlIjpudWxsLCJncm91cE5hbWUiOm51bGwsImh5cGVybGlua09wZW5FeHRlcm5hbGx5IjpudWxsLCJsaXN0U3R5bGUiOm51bGwsInByb2ZpbGVGdWxsTmFtZSI6bnVsbCwic3RyaW5nRmllbGRSZWZlcmVuY2UiOm51bGwsImxlYXJuaW5nQ291cnNlTmFtZSI6bnVsbCwicHJvZmlsZU1lbnRpb24iOm51bGwsInN0eWxlIjoiTElTVF9JVEVNIiwic2Nob29sTmFtZSI6bnVsbCwiaGFzaHRhZyI6bnVsbH0sIiRyZWNpcGVUeXBlcyI6WyJjb20ubGlua2VkaW4uYTg2ZWMzYTdhYjUwYzJjODAxZTAzOTVhODZmZjE5YzAiXSwiJHR5cGUiOiJjb20ubGlua2VkaW4udm95YWdlci5kYXNoLmNvbW1vbi50ZXh0LlRleHRBdHRyaWJ1dGUifSx7InN0YXJ0IjoxMzQzLCJsZW5ndGgiOjMwNSwiZGV0YWlsRGF0YSI6eyJqb2JQb3N0aW5nTmFtZSI6bnVsbCwiaHlwZXJsaW5rIjpudWxsLCJwcm9maWxlRmFtaWxpYXJOYW1lIjpudWxsLCJjb2xvciI6bnVsbCwiY29tcGFueU5hbWUiOm51bGwsImljb24iOm51bGwsImVwb2NoIjpudWxsLCJzeXN0ZW1JbWFnZSI6bnVsbCwibGlzdEl0ZW1TdHlsZSI6bnVsbCwiZ3JvdXBOYW1lIjpudWxsLCJoeXBlcmxpbmtPcGVuRXh0ZXJuYWxseSI6bnVsbCwibGlzdFN0eWxlIjpudWxsLCJwcm9maWxlRnVsbE5hbWUiOm51bGwsInN0cmluZ0ZpZWxkUmVmZXJlbmNl
When I make the same call in curl format using Postman I get a readable JSON response. Do you know why in IntelliJ I get this unreadable response? It should be JSON? How can I solve the problem?
EDIT:
This is caused by IntelliJ bug: https://youtrack.jetbrains.com/issue/KTIJ-22158/JUnit5-test-prints-a-cryptic-AssertionError-message-in-the-IDE?s=JUnit5-test-prints-a-cryptic-AssertionError-message-in-the-IDE
I am going to transform java code to python code.
but login POST request is not working via python code.
// Java code
String inputData = “{\"lang\" : \"ko\", \"loginName\" : \"kkstar123\", \"password\" : \"123123123\"}”;
String strUrl = “http://10.110.120.80/management/user/login.json”;
StringBuffer sb = new StringBuffer();
HttpURLConnection conn = (HttpURLConnection) new URL(strUrl).openConnection();
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("Accept", "application/json");
conn.setRequestProperty("Content-Type", "application/json; charset=\"UTF-8\"");
OutputStream out = conn.getOutputStream(); // if remove OutputStream, it return 404 error
out.write(inputData.getBytes());
out.close();
System.out.println(conn.getResponseCode());
InputStreamReader in = new InputStreamReader((InputStream)conn.getContent());
BufferedReader br = new BufferedReader(in);
String line;
while ((line = br.readLine()) != null)
sb.append(line).append("\n");
System.out.println(sb.toString());
Above works fine, but.. below python code return 404 error
// python code
import requests
from bs4 import BeautifulSoup
header = {
'Accept': 'application/json',
'Content-Type': 'application/json; charset="UTF-8"',
'User-Agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Mobile Safari/537.36'
}
payload = {
"lang": "ko",
"loginName": "kkstar123",
"password": "123123123"
}
loginURL = "http://10.110.120.80/management/user/login.json"
with requests.Session() as s:
login_req = s.post(loginURL, data = payload, headers = header)
print(login_req)
print(html)
What is the problem? plz help me to resolve this issue. (if i use selenium, it works fine! but i want get request and response quickly! so i am trying to use requests api)
I should't use 'data' param for POST request. after change this to json, it works. i think i add the Content-Type into header with JSON, so i should use json param.
Have you checked if the request sent in the browser involves any cookies?
If there are cookies present, you'll need to convert those cookies using SimpleCookie and add them to request's headers.
Also, create a Session object from requests as it will be better.
I've gotten an auth code from doing a "GET" request and signing in, and I'm trying to use that code to get an auth token using an http "POST", but I always get response code 400. My code is as follows:
// Create HTTP Objects
HttpClient httpclient = HttpClients.createDefault();
HttpPost httpPost = new HttpPost("https://www.googleapis.com/oauth2/v4/token");
// Add Header
httpPost.addHeader("Content-Type", "application/x-www-form-urlencoded");
// Create Parameters
List<NameValuePair> params = new ArrayList<>();
params.add(new BasicNameValuePair("code", code));
params.add(new BasicNameValuePair("client_id", client_id));
params.add(new BasicNameValuePair("client_secret", client_secret));
params.add(new BasicNameValuePair("redirect_uri", redirect_uri));
params.add(new BasicNameValuePair("grant_type", "authorization_code"));
// Add Parameters and Excecute
httpPost.setEntity(new UrlEncodedFormEntity(params, "UTF-8"));
HttpResponse response = httpclient.execute(httpPost);
It seems to match perfectly the documentation they provide here:
https://developers.google.com/identity/protocols/OAuth2WebServer
Any ideas why this is happening?
Alright, I figured it out, it was as simple as it took long enough to debug my code for the auth code from Google to expire.
I got a new code from Google and it worked.
I am trying to get the html source of
https://www.coinbet24.com/en/odds/football/algeria/ligue-1
In general, I have done this tons of times, and never had a problem, yet this specific website is giving me a hard time.
No matter what I try, I get a response with populated head, but an empty body.
The only time that it works and I actually get the full response, is if I manually set the Cookie in the request header to be equal to the Cookie of my actual browser.
I tried automating this process by first getting the connection headers and setting the Cookie through those, but once again, I am getting a blank body.
This is how I get the Cookie, then set it for the request. I also tried with Apache HttpClient. Same result.
URL url = new URL(urlStr);
URLConnection connection = url.openConnection();
connection.addRequestProperty("User-Agent",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36");
Map<String, List<String>> headers = connection.getHeaderFields();
connection = url.openConnection();
String cookie =
headers.get("Set-Cookie").get(0).split(";")[0] + "; " + headers.get("Set-Cookie").get(1).split(";" + "")[0];
System.out.println("cookie = " + cookie);
connection.addRequestProperty("User-Agent",
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36");
connection.addRequestProperty("Cookie", cookie);
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuilder sb = new StringBuilder();
String str;
while ((str = br.readLine()) != null) {
sb.append(str);
}
return sb.toString();
Any help is appreciated. Thanks in advance.
I am trying to get the html of MyAnimeList.net (specifically this page: http://myanimelist.net/anime.php?q=toradora!), I am using a method that has worked for me before on different websites but doesn't work for me here.
The Method I use:
public String getWebsiteSourceCode(String sURL){
try{
URL url = new URL(sURL);
URLConnection urlConn= url.openConnection();
//NEW LINE
urlConn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.102 Safari/537.36");
BufferedReader in = new BufferedReader(new InputStreamReader(
urlConn.getInputStream(), "UTF-8"));
String inputLine;
StringBuilder a = new StringBuilder();
while ((inputLine = in.readLine()) != null)
a.append(inputLine);
in.close();
return a.toString();
}catch(Exception e){
e.printStackTrace();
return "null";
}
}
What I get:
<html><head><META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"></head><iframe src="/_Incapsula_Resource?CWUDNSAI=9&incident_id=124000930038292057-125560654487356886&edet=12&cinfo=464f095fc75381e904000000" frameborder=0 width="100%" height="100%" marginheight="0px" marginwidth="0px">Request unsuccessful. Incapsula incident ID: 124000930038292057-125560654487356886</iframe></html>
What I should be getting: The html code of the webpage (I can get it in google chrome by right click + view page source, and it is completely different from what I get from my method).
From what I get, it says something about ROBOTS, so I assume the website has cookies or something to track whether I am using a browser or a bot... What I want to know is whether or not it is possible to bypass this, and how would I go upon doing so? Thanks for your help :) (Preferably in Java, since that is what I am using)
EDIT: tried adding this line:
urlConn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.102 Safari/537.36");
but I get the same error...
try
URL url = new URL(sURL);
URLConnection urlConn= url.openConnection();
c.setRequestProperty("User-Agent", "....");
For an idea of what to put in ... have a look at http://www.whatsmyuseragent.com/
Using your code I get
<html><head><META NAME="robots" CONTENT="noindex,nofollow"><script>(function(){f
unction getSessionCookies(){cookieArray=new Array();var cName=/^\s?incap_ses_/;v
ar c=document.cookie.split(";");for(var i=0;i<c.length;i++){key=c[i].substr(0,c[
i].indexOf("="));value=c[i].substr(c[i].indexOf("=")+1,c[i].length);if(cName.tes
t(key)){cookieArray[cookieArray.length]=value}}return cookieArray}function setIn
capCookie(vArray){try{cookies=getSessionCookies();digests=new Array(cookies.leng
th);for(var i=0;i<cookies.length;i++){digests[i]=simpleDigest((vArray)+cookies[i
])}res=vArray+",digest="+(digests.join())}catch(e){res=vArray+",digest="+(encode
URIComponent(e.toString()))}createCookie("___utmvc",res,20)}function simpleDiges
t(mystr){var res=0;for(var i=0;i<mystr.length;i++){res+=mystr.charCodeAt(i)}retu
rn res}function createCookie(name,value,seconds){if(seconds){var date=new Date()
;date.setTime(date.getTime()+(seconds*1000));var expires="; expires="+date.toGMT
String()}else{var expires=""}document.cookie=name+"="+value+expires+"; path=/"}f
unction test(o){var res="";var vArray=new Array();for(test in o){switch(o[test])
{case"exists":try{vArray[vArray.length]=encodeURIComponent(test+"="+typeof(eval(
test)))}catch(e){vArray[vArray.length]=encodeURIComponent(test+"="+e)}break;case
"value":try{vArray[vArray.length]=encodeURIComponent(test+"="+eval(test).toStrin
g())}catch(e){vArray[vArray.length]=encodeURIComponent(test+"="+e)}break;case"pl
ugins":try{p=navigator.plugins;pres="";for(a in p){pres+=(p[a]["description"]+"
").substring(0,20)}vArray[vArray.length]=encodeURIComponent("plugins="+pres)}cat
ch(e){vArray[vArray.length]=encodeURIComponent("plugins="+e)}break;case"plugin":
try{a=navigator.plugins;for(i in a){f=a[i]["filename"].split(".");if(f.length==2
){vArray[vArray.length]=encodeURIComponent("plugin="+f[1]);break}}}catch(e){vArr
ay[vArray.length]=encodeURIComponent("plugin="+e)}break}}vArray=vArray.join();re
turn vArray}var o={navigator:"exists","navigator.vendor":"value",opera:"exists",
ActiveXObject:"exists","navigator.appName":"value",platform:"plugin",webkitURL:"
exists","navigator.plugins.length==0":"value"};try{setIncapCookie(test(o));docum
ent.createElement("img").src="/_Incapsula_Resource?SWKMTFSR=1&e="+Math.random()}
catch(e){img=document.createElement("img");img.src="/_Incapsula_Resource?SWKMTFS
R=1&e="+e}})();</script><script>(function() { var z="";var b="7472797B7661722078
68723B76617220743D6E6577204461746528292E67657454696D6528293B76617220737461747573
3D227374617274223B7661722074696D696E673D6E65772041727261792833293B77696E646F772E
6F6E756E6C6F61643D66756E6374696F6E28297B74696D696E675B325D3D22723A222B286E657720
4461746528292E67657454696D6528292D74293B646F63756D656E742E637265617465456C656D65
6E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C
555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E742873746174
75732B222028222B74696D696E672E6A6F696E28292B222922297D3B69662877696E646F772E584D
4C4874747052657175657374297B7868723D6E657720584D4C48747470526571756573747D656C73
657B7868723D6E657720416374697665584F626A65637428224D6963726F736F66742E584D4C4854
545022297D7868722E6F6E726561647973746174656368616E67653D66756E6374696F6E28297B73
7769746368287868722E72656164795374617465297B6361736520303A7374617475733D6E657720
4461746528292E67657454696D6528292D742B223A2072657175657374206E6F7420696E69746961
6C697A656420223B627265616B3B6361736520313A7374617475733D6E6577204461746528292E67
657454696D6528292D742B223A2073657276657220636F6E6E656374696F6E2065737461626C6973
686564223B627265616B3B6361736520323A7374617475733D6E6577204461746528292E67657454
696D6528292D742B223A2072657175657374207265636569766564223B627265616B3B6361736520
333A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2070726F6365
7373696E672072657175657374223B627265616B3B6361736520343A7374617475733D22636F6D70
6C657465223B74696D696E675B315D3D22633A222B286E6577204461746528292E67657454696D65
28292D74293B6966287868722E7374617475733D3D323030297B706172656E742E6C6F636174696F
6E2E72656C6F616428297D627265616B7D7D3B74696D696E675B305D3D22733A222B286E65772044
61746528292E67657454696D6528292D74293B7868722E6F70656E2822474554222C222F5F496E63
617073756C615F5265736F757263653F535748414E45444C3D353238343936313938333732343733
393534322C3239343135383533343939333730393439362C31313138373735393633303935393534
323637302C3339333836222C66616C7365293B7868722E73656E64286E756C6C297D636174636828
63297B7374617475732B3D6E6577204461746528292E67657454696D6528292D742B2220696E6361
705F6578633A20222B633B646F63756D656E742E637265617465456C656D656E742822696D672229
2E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D36372674
3D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74
696D696E672E6A6F696E28292B222922297D3B";for (var i=0;i<b.length;i+=2){z=z+parseI
nt(b.substring(i, i+2), 16)+",";}z = z.substring(0,z.length-1); eval(eval('Strin
g.fromCharCode('+z+')'));})();</script></head><body><iframe style="display:none;
visibility:hidden;" src="http://my.incapsula.com/public/ga/jsTest.html" id="gaIf
rame"></iframe></body></html>