Develop Reference

Add HTTP header to response in java socket server

I'm not sure where or what command to use to add the HTTP header to the response from the server.
import java.io.*;
import java.net.*;
import com.sun.net.httpserver.*;
public class Response {
private static final int BUFFER_SIZE = 9999;
Request request;
BufferedOutputStream output;
//constructor para el output
public Response(BufferedOutputStream output){
this.output = output;
}
//Set del request
public void setRequest(Request request){
this.request = request;
}
public void sendResource() throws IOException{
File file = new File(Java_Server.Web_dir,request.getUri());
byte [] bytearray = new byte[(int) file.length()];
FileInputStream file_out = null;
if(file.exists())
file_out = new FileInputStream(file);
else{
String errorMessage = "HTTP/1.1 404 File Not Found\r\n" +
"Content-Type: text/html\r\n" +
"Content-Length: 23\r\n" +
"\r\n" +
"<h1>File Not Found</h1>";
output.write(errorMessage.getBytes());
}
BufferedInputStream bis = new BufferedInputStream(file_out);
try{
bis.read(bytearray,0,bytearray.length);
output.write(bytearray,0 , bytearray.length);
output.flush();
output.close();
return;
}catch (IOException e){
e.printStackTrace();
}
}
The contents is deliver to the browser but without the HTTP header and if a image is send for example, the browser doesn't show the image, it shows byte for byte.

The preferred way to do is, is to implement a Servlet and run it in a Servlet Container. Then you call the method setHeader on the HttpServletResponse object:
public class ExampleServlet extends HttpServlet {
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws IOException, ServletException {
response.setHeader("X-Whatever-Header-Name-You-Want", "Value");
}
}

httpsurlconnection posts string wrongly in java application

In my java application I used a Httpsurlconnection to post some string data to the server. When I test this code on android, it works perfectly. However, in a java application it does not work. Client java application is as follows:
public static void main(String[] args) {
disableSslVerification();
new HttpsClient().testIt();
}
private void testIt() {
String https_url = "https://XXX.XX.XXX.XXX:XXXX/XXXXX/TestServlet";
URL url;
try {
url = new URL(https_url);
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
print_content(con, "test");
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
private void print_content(HttpsURLConnection connection, String data) {
if (connection != null) {
try {
connection.setConnectTimeout(6000);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
Charset cSet = Charset.forName("UTF-8");
byte bytes[] = data.getBytes(cSet);
connection.setRequestProperty("Content-Length", ""
+ Integer.toString(bytes.length));
connection.setRequestProperty("Content-Language", "tr");
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(
connection.getOutputStream());
wr.write(bytes);
wr.flush();
wr.close();
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is, cSet));
String line;
StringBuilder response = new StringBuilder();
while ((line = rd.readLine()) != null) {
response.append(line);
response.append(System.getProperty("line.separator"));
}
rd.close();
System.out.println(response.toString());
} catch (Exception e) {
System.out.println(e.getMessage());
} finally {
if (connection != null) {
connection.disconnect();
}
}
}
}
And the servlet is as follows:
#Override
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
String s = getHTML(request);
try {
out.print("received data:");
out.print(s);
} finally {
out.close();
}
}
private String getHTML(HttpServletRequest request) throws IOException {
int n = request.getContentLength();
if (n < 1) {
return "";
}
byte bytes[] = new byte[n];
request.getInputStream().read(bytes);
return new String(bytes, "UTF-8");
}
When I run this application, servlet's response is:
received data:t☐☐☐
Always only the first character is correctly send to the servlet. The same code works perfect on android. Can anyone help me please? Thanks...

I can't see an obvious problem with your code that would cause this.
Can anyone help me please?
I suggest that you take a methodical approach to investigating the problem. Use a packet sniffer to check what is actually being sent over the wire. Check that the actual headers in the request and response are correct. Check that the request and response bodies are really properly encoded UTF-8 ...
What you find in your investigation / evidence gathering will help you figure out where the problem (or problems) are occurring ... and that will allow you to home in on the part(s) of your code that is/are responsible.

request.getInputStream().read(bytes);
You might need to do this read in a loop. At the very least, check how many bytes have been read. The array appears to be empty except for the first char.
Reads some number of bytes from the input stream and stores them into
the buffer array b. The number of bytes actually read is returned as
an integer. This method blocks until input data is available, end of
file is detected, or an exception is thrown.

InputStream being lost

I have a class titled: ServiceCaller.java
This class contains a method used to call web services:
public static Response callService(String strURL, String Token, int timeout, Boolean isPostMethod) {
String error = "";
int statusCode = HttpStatus.SC_INTERNAL_SERVER_ERROR;
HttpURLConnection urlConnection = null;
try
{
URL url = new URL(strURL);
// Allow non trusted ssl certificates
if(strURL.startsWith("https"))
{
TrustManagerManipulator.allowAllSSL();
}
urlConnection = (HttpURLConnection) url.openConnection();
if (isPostMethod) {
urlConnection.setRequestMethod("POST");
}
else {
urlConnection.setRequestMethod("GET");
}
// Allow Inputs
urlConnection.setDoInput(true);
// Allow Outputs
urlConnection.setDoOutput(true);
// Don't use a cached copy.
urlConnection.setUseCaches(false);
urlConnection.setRequestProperty("Connection", "Keep-Alive");
urlConnection.setRequestProperty("Content-Type", "application/json");
urlConnection.setRequestProperty("Token", Helpers.getUTF8Encode(Token));
urlConnection.setConnectTimeout(timeout);
DataOutputStream dos = new DataOutputStream(urlConnection.getOutputStream());
dos.flush();
dos.close();
statusCode = urlConnection.getResponseCode();
Response r = new Response(statusCode, urlConnection.getInputStream(), "No Exception");
return r;
} catch (Exception ex) {
error = ex.getMessage();
if (error != null && !error.equals("") && error.contains("401"))
statusCode = HttpStatus.SC_UNAUTHORIZED;
} finally {
urlConnection.disconnect();
}
return new Response(statusCode, null, error);
}
Here's the Response class:
public static class Response
{
private int statusCode;
private InputStream responseStream;
private String exception;
public int getStatusCode() {
return statusCode;
}
public InputStream getResponseStream() {
return responseStream;
}
public String getExceptionError() {
return exception;
}
public Response(int code, InputStream stream, String strException)
{
this.statusCode = code;
this.responseStream = stream;
this.exception = strException;
}
}
This is the Test class that I use to test the function in ServiceCaller:
public class TestDemo {
private static final String EncriptionKey = "keyValueToUse";
public static void main(String[] args) {
try {
String strURL = "http://...";
String strURL2 = "http://...";
String Token = "iTcakW5...";
int timeout = 120000;
Boolean isPostMethod = true;
ServiceCaller.Response resp = ServiceCaller.CallService(strURL2, Token, timeout, isPostMethod);
InputStream inputStream = resp.getResponseStream();
StringWriter writer = new StringWriter();
IOUtils.copy(inputStream, writer);
String resultJSON = writer.toString();
System.out.println("Status Code: " + resp.getStatusCode());
System.out.println("JSON String:\n" + resultJSON);
System.out.println("Exception: " + resp.getExceptionError());
} catch (Exception e) {
e.printStackTrace();
}
}
}
Here's the Output of executing hte previous code:
Status Code: 200
JSON String:
Exception: No Exception
Here's the problem, the InputString that is returned in the Test class appears to be empty because the conversion to string returns an empty string BUT if I do the same code to convert the InputString inside the CallService function then the conversion is successful, also note that the Status Code and Exception (strings) are being returned correctly.

public static Response CallService(String strURL, String Token, int timeout, Boolean isPostMethod) {
HttpURLConnection urlConnection = ...
...
new Response(statusCode, urlConnection.getInputStream(), "No Exception");
}
This code missing in the ... is probably the most important part. I guess you are closing the HttpURLConnection before returning back to the caller. How you do this can vary:
You simply close it before the return
try-catch-finally: You are closing it in the finally block.
you are using a try-with-resource construction as introduced in Java 7. The HttpURLConnection might be getting closed automatically. This is more unlikely since HttpURLConnection does not implement AutoClosable.

I've solved by first getting the InputStream from HttpURLConnection, then converting it to a byte array, then putting that byte array into a ByteArrayInputStream
byte[] bytes = IOUtils.toByteArray(urlConnection.getInputStream());
ByteArrayInputStream byteStream = new ByteArrayInputStream(bytes);
return new Response(statusCode, byteStream, "");
According to the documentation a ByteArrayInputStream:
public ByteArrayInputStream(byte[] buf) Creates a ByteArrayInputStream
so that it uses buf as its buffer array. The buffer array is not
copied. The initial value of pos is 0 and the initial value of count
is the length of buf. Parameters: buf - the input buffer.

The problem is that you are already consuming the InputStream in your CallService method
statusCode = urlConnection.getResponseCode();
Response resp = new Response(statusCode, urlConnection.getInputStream(), "");
InputStream inputStream = resp.getResponseStream();
StringWriter writer = new StringWriter();
IOUtils.copy(inputStream, writer); // consuming the stream
String resultJSON = writer.toString(); // you never use this, so why is it here?
So when you try to read from it again in main() there are no bytes left.
You can only read bytes from it once.
This doesn't throw any exceptions because IOUtils simply calls InputStream#read(...) which returns -1 if the EOF has been reached.
Note that Java naming convention states that method names should start with a lowercase character.

Java HTTPS Client [duplicate]

I'm trying to find Java's equivalent to Groovy's:
String content = "http://www.google.com".toURL().getText();
I want to read content from a URL into string. I don't want to pollute my code with buffered streams and loops for such a simple task. I looked into apache's HttpClient but I also don't see a one or two line implementation.

Now that some time has passed since the original answer was accepted, there's a better approach:
String out = new Scanner(new URL("http://www.google.com").openStream(), "UTF-8").useDelimiter("\\A").next();
If you want a slightly fuller implementation, which is not a single line, do this:
public static String readStringFromURL(String requestURL) throws IOException
{
try (Scanner scanner = new Scanner(new URL(requestURL).openStream(),
StandardCharsets.UTF_8.toString()))
{
scanner.useDelimiter("\\A");
return scanner.hasNext() ? scanner.next() : "";
}
}

This answer refers to an older version of Java. You may want to look at ccleve's answer.
Here is the traditional way to do this:
import java.net.*;
import java.io.*;
public class URLConnectionReader {
public static String getText(String url) throws Exception {
URL website = new URL(url);
URLConnection connection = website.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
connection.getInputStream()));
StringBuilder response = new StringBuilder();
String inputLine;
while ((inputLine = in.readLine()) != null)
response.append(inputLine);
in.close();
return response.toString();
}
public static void main(String[] args) throws Exception {
String content = URLConnectionReader.getText(args[0]);
System.out.println(content);
}
}
As #extraneon has suggested, ioutils allows you to do this in a very eloquent way that's still in the Java spirit:
InputStream in = new URL( "http://jakarta.apache.org" ).openStream();
try {
System.out.println( IOUtils.toString( in ) );
} finally {
IOUtils.closeQuietly(in);
}

Or just use Apache Commons IOUtils.toString(URL url), or the variant that also accepts an encoding parameter.

There's an even better way as of Java 9:
URL u = new URL("http://www.example.com/");
try (InputStream in = u.openStream()) {
return new String(in.readAllBytes(), StandardCharsets.UTF_8);
}
Like the original groovy example, this assumes that the content is UTF-8 encoded. (If you need something more clever than that, you need to create a URLConnection and use it to figure out the encoding.)

Now that more time has passed, here's a way to do it in Java 8:
URLConnection conn = url.openConnection();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8))) {
pageText = reader.lines().collect(Collectors.joining("\n"));
}

Additional example using Guava:
URL xmlData = ...
String data = Resources.toString(xmlData, Charsets.UTF_8);

Java 11+:
URI uri = URI.create("http://www.google.com");
HttpRequest request = HttpRequest.newBuilder(uri).build();
String content = HttpClient.newHttpClient().send(request, BodyHandlers.ofString()).body();

If you have the input stream (see Joe's answer) also consider ioutils.toString( inputstream ).
http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html#toString(java.io.InputStream)

The following works with Java 7/8, secure urls, and shows how to add a cookie to your request as well. Note this is mostly a direct copy of this other great answer on this page, but added the cookie example, and clarification in that it works with secure urls as well ;-)
If you need to connect to a server with an invalid certificate or self signed certificate, this will throw security errors unless you import the certificate. If you need this functionality, you could consider the approach detailed in this answer to this related question on StackOverflow.
Example
String result = getUrlAsString("https://www.google.com");
System.out.println(result);
outputs
<!doctype html><html itemscope="" .... etc
Code
import java.net.URL;
import java.net.URLConnection;
import java.io.BufferedReader;
import java.io.InputStreamReader;
public static String getUrlAsString(String url)
{
try
{
URL urlObj = new URL(url);
URLConnection con = urlObj.openConnection();
con.setDoOutput(true); // we want the response
con.setRequestProperty("Cookie", "myCookie=test123");
con.connect();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
StringBuilder response = new StringBuilder();
String inputLine;
String newLine = System.getProperty("line.separator");
while ((inputLine = in.readLine()) != null)
{
response.append(inputLine + newLine);
}
in.close();
return response.toString();
}
catch (Exception e)
{
throw new RuntimeException(e);
}
}

Here's Jeanne's lovely answer, but wrapped in a tidy function for muppets like me:
private static String getUrl(String aUrl) throws MalformedURLException, IOException
{
String urlData = "";
URL urlObj = new URL(aUrl);
URLConnection conn = urlObj.openConnection();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8)))
{
urlData = reader.lines().collect(Collectors.joining("\n"));
}
return urlData;
}

URL to String in pure Java
Example call to get payload from http get call
String str = getStringFromUrl("YourUrl");
Implementation
You can use the method described in this answer, on How to read URL to an InputStream and combine it with this answer on How to read InputStream to String.
The outcome will be something like
public String getStringFromUrl(URL url) throws IOException {
return inputStreamToString(urlToInputStream(url,null));
}
public String inputStreamToString(InputStream inputStream) throws IOException {
try(ByteArrayOutputStream result = new ByteArrayOutputStream()) {
byte[] buffer = new byte[1024];
int length;
while ((length = inputStream.read(buffer)) != -1) {
result.write(buffer, 0, length);
}
return result.toString(UTF_8);
}
}
private InputStream urlToInputStream(URL url, Map<String, String> args) {
HttpURLConnection con = null;
InputStream inputStream = null;
try {
con = (HttpURLConnection) url.openConnection();
con.setConnectTimeout(15000);
con.setReadTimeout(15000);
if (args != null) {
for (Entry<String, String> e : args.entrySet()) {
con.setRequestProperty(e.getKey(), e.getValue());
}
}
con.connect();
int responseCode = con.getResponseCode();
/* By default the connection will follow redirects. The following
* block is only entered if the implementation of HttpURLConnection
* does not perform the redirect. The exact behavior depends to
* the actual implementation (e.g. sun.net).
* !!! Attention: This block allows the connection to
* switch protocols (e.g. HTTP to HTTPS), which is <b>not</b>
* default behavior. See: https://stackoverflow.com/questions/1884230
* for more info!!!
*/
if (responseCode < 400 && responseCode > 299) {
String redirectUrl = con.getHeaderField("Location");
try {
URL newUrl = new URL(redirectUrl);
return urlToInputStream(newUrl, args);
} catch (MalformedURLException e) {
URL newUrl = new URL(url.getProtocol() + "://" + url.getHost() + redirectUrl);
return urlToInputStream(newUrl, args);
}
}
/*!!!!!*/
inputStream = con.getInputStream();
return inputStream;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
Pros
It is pure java
It can be easily enhanced by adding different headers as a map (instead of passing a null object, like the example above does), authentication, etc.
Handling of protocol switches is supported

login and download a webpage with java and storing cookies

I'm having trouble logging into my schools moodle webpage and downloading the source code,
so far i am able to receive the login page it never actually logs in,
any help would be greatly appreciated i have been stuck with this problem for a couple of weeks now.
The code below is a not my own but a modified version of multiple examples that i have found on the web.
import java.net.*;
import java.io.*;
import java.util.*;
public class LoginByHttpPost
{
private static final String POST_CONTENT_TYPE = "application/x-www-form-urlencoded";
private static final String LOGIN_USER_NAME = "myusername";
private static final String LOGIN_PASSWORD = "mypassword";
private static final String LOGIN_DOMAIN = "students.ltu.edu.au";
private static final String TARGET_URL = "https://www.latrobe.edu.au/lms/login/";
private String page ="";
public static void main (String args[])
{
LoginByHttpPost httpUrlBasicAuthentication = new LoginByHttpPost();
httpUrlBasicAuthentication.httpPostLogin();
}
public void httpPostLogin ()
{
try
{
String urlEncodedContent = preparePostContent(LOGIN_USER_NAME, LOGIN_PASSWORD, LOGIN_DOMAIN);
HttpURLConnection urlConnection = doHttpPost(TARGET_URL, urlEncodedContent);
page = readResponse(urlConnection);
System.out.println("Successfully made the HTPP POST.");
System.out.println("Recevied response is: '/n" + page + "'");
}
catch(IOException ioException)
{
System.out.println("Problems encounterd.");
}
}
private String preparePostContent(String loginUserName, String loginPassword, String loginDomain) throws UnsupportedEncodingException
{
String encodedLoginUserName = URLEncoder.encode(loginUserName, "UTF-8");
String encodedLoginPassword = URLEncoder.encode(loginPassword, "UTF-8");
String encodedLoginDomain = URLEncoder.encode(loginDomain, "UTF-8");
String content = URLEncoder.encode("username=", "UTF-8") + encodedLoginUserName
+ URLEncoder.encode("&password=", "UTF-8") + encodedLoginPassword
+ URLEncoder.encode("&domain=", "UTF-8") + encodedLoginDomain
+ URLEncoder.encode("&Login=", "UTF-8") + URLEncoder.encode("Login", "UTF-8");
return content;
}
public HttpURLConnection doHttpPost(String targetUrl, String content) throws IOException
{
DataOutputStream dataOutputStream = null;
HttpURLConnection conn = null;
String cookieFirst = null;
String cookieValue = null;
String totalCookie = "";
try
{
CookieManager manager = new CookieManager();
manager.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(manager);
URL url = new URL(targetUrl);
conn = (HttpURLConnection)url.openConnection();
conn.getContent();
CookieStore cookiejar = manager.getCookieStore();
List<HttpCookie> cookiesList = cookiejar.getCookies();
for(HttpCookie cookiel: cookiesList)
{
totalCookie += cookiel+"; ";
}
totalCookie = totalCookie.substring(0, totalCookie.length()-1);
System.out.println("Total Cookie: " + totalCookie);
}
catch(Exception e)
{
System.out.println("Something went wrong");
}
HttpURLConnection urlConnection = null;
try{
URL url = new URL(targetUrl);
urlConnection = (HttpURLConnection)url.openConnection();
urlConnection.setDoInput(true);
urlConnection.setDoOutput(true);
urlConnection.setUseCaches(true);
urlConnection.setRequestProperty("Content-Type", POST_CONTENT_TYPE);
urlConnection.setRequestProperty("Content-Length", Integer.toString(content.length()));
urlConnection.setInstanceFollowRedirects(true);
urlConnection.setRequestMethod("POST");
urlConnection.setRequestProperty("Cookie", totalCookie);
urlConnection.connect();
dataOutputStream = new DataOutputStream(urlConnection.getOutputStream());
dataOutputStream.writeBytes(content);
dataOutputStream.flush();
dataOutputStream.close();
}
catch(IOException ioException)
{
System.out.println("I/O problems while trying to do a HTTP post.");
ioException.printStackTrace();
if (dataOutputStream != null)
{
try
{
dataOutputStream.close();
}
catch(Throwable ignore)
{
}
}
if (urlConnection != null)
{
urlConnection.disconnect();
}
throw ioException;
}
return urlConnection;
}
private String readResponse(HttpURLConnection urlConnection) throws IOException
{
BufferedReader bufferedReader = null;
try
{
bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String responeLine;
StringBuilder response = new StringBuilder();
while ((responeLine = bufferedReader.readLine()) != null)
{
response.append(responeLine + "\n");
}
return response.toString();
}
catch(IOException ioException)
{
System.out.println("Problems while reading the response");
ioException.printStackTrace();
throw ioException;
}
finally
{
if (bufferedReader != null)
{
try
{
bufferedReader.close();
}
catch(Throwable ignore)
{
}
}
}
}
}

To access this web page and log in, you're using a web browser and not a sequance of telnet commands, because it's much easier, right? Then, as a programmer, do the same and use a programmatic web browser rather than a sequence of low-level actions using cookies and URL connections. It will also be much easier.
HtmlUnit is such a programmatic web browser. The end of its Getting started page shows an example of loading a web page and submitting a form. HtmlUnit will handle the submission, cookie handling, encoding, etc. for you.