Develop Reference

Avoiding duplicate code when the unique part is within a loop/try-catch

I have a class that has two methods that have a lot of duplicate code but the bit that's unique is in the middle of the whole thing. From my research I think I should be doing the "Execute around method" pattern but I can't find a resource that I can follow as they all seem to use code I can't replicate.
I have two methods, apiPost and apiGet, which I've pasted below. I've wrapped the unique parts of these methods with comments showing where the unique section starts and ends:
/**
* Class that handles authorising the connection and handles posting and getting data
*
* #version %I%, %G%
* #since 1.0
*/
public class CallHandler {
private static PropertyLoader props = PropertyLoader.getInstance();
final static int MAX = props.getPropertyAsInteger(props.MAX_REQUESTS);
private final Logger log = LoggerFactory.getLogger(CallHandler.class);
private final static String POST = "POST";
private final static String GET = "GET";
/**
* Makes a POST call to the API URL provided and returns the JSON response as a string
* http://stackoverflow.com/questions/15570656/how-to-send-request-payload-to-rest-api-in-java
*
* #param urlString the API URL to send the data to, as a string
* #param payload the serialised JSON payload string
* #return and value returned as a JSON string, ready to be deserialised
*/
public String apiPost(String urlString, String payload) {
boolean keepGoing = true;
int tries = 0;
String line;
StringBuilder jsonString = new StringBuilder();
log.debug("Making API Call: {}", urlString);
while (keepGoing && tries < MAX) {
tries++;
try {
URL url = new URL(urlString);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
// UNIQUE CODE START
prepareConnection(connection, POST);
OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
// UNIQUE CODE END
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
while ((line = br.readLine()) != null) {
jsonString.append(line);
}
br.close();
connection.disconnect();
keepGoing = false;
} catch (Exception e) {
log.warn("Try #{}. Error posting: {}", tries, e.getMessage());
log.warn("Pausing for 1 second then trying again...");
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException f) {
log.warn("Sleeping has been interrupted: {}", f.getMessage());
}
}
}
return jsonString.toString();
}
/**
* Makes a GET call to the API URL provided and returns the JSON response as a string
* http://stackoverflow.com/questions/2793150/using-java-net-urlconnection-to-fire-and-handle-http-requests
*
* #param urlString the API URL to request the data from, as a string
* #return the json response as a string, ready to be deserialised
*/
public String apiGet(String urlString) {
boolean keepGoing = true;
int tries = 0;
String line;
StringBuilder jsonString = new StringBuilder();
log.debug("Making API Call: {}", urlString);
while (keepGoing && tries < MAX) {
tries++;
try {
URL url = new URL(urlString);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
// UNIQUE CODE START
prepareConnection(connection, GET);
connection.connect();
// UNIQUE CODE END
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
while ((line = br.readLine()) != null) {
jsonString.append(line);
}
br.close();
connection.disconnect();
keepGoing = false;
} catch (Exception e) {
log.warn("Try #{}. Error getting from API: {}", tries, e.getMessage());
log.warn("Pausing for 1 second then trying again...");
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException f) {
log.warn("Sleeping has been interrupted: {}", f.getMessage());
}
}
}
return jsonString.toString();
}
/**
* Prepares the HTTP Url connection depending on whether this is a POST or GET call
*
* #param connection the connection to prepare
* #param method whether the call is a POST or GET call
*/
private void prepareConnection(HttpURLConnection connection, String method) {
String charset = "UTF-8";
try {
connection.setRequestMethod(method);
if (method.equals(GET)) {
connection.setRequestProperty("Accept-Charset", charset);
} else if (method.equals(POST)) {
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "application/json; charset=" + charset);
}
connection.setRequestProperty("Accept", "application/json");
connection.setRequestProperty("Authorization", "Bearer " + apiKey);
} catch (Exception e) {
log.error("Error preparing HTTP URL connection: {}", e.getMessage());
throw new RuntimeException(e.getMessage());
}
}
Can I use the "Execute around method" pattern to save on code duplication here? If so could someone help me figure out how to refactor this code to make use of it. If this is the wrong way to go about it could someone suggest a smart alternative?

It can be done by extracting "unique" code into special worker. More specifically for example, you can use lambda expressions:
public String apiPost(String urlString, String payload) {
return commonMethod(urlString, payload, (connection) -> {
// UNIQUE CODE START
prepareConnection(connection, POST);
OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
// UNIQUE CODE END
});
}
interface ConnectionWorker {
void run(HttpURLConnection connection) throws IOException;
}
public String commonMethod(String urlString, String payload, ConnectionWorker worker) {
boolean keepGoing = true;
int tries = 0;
String line;
StringBuilder jsonString = new StringBuilder();
log.debug("Making API Call: {}", urlString);
while (keepGoing && tries < MAX) {
tries++;
try {
URL url = new URL(urlString);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
worker.run(connection);
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
while ((line = br.readLine()) != null) {
jsonString.append(line);
}
br.close();
connection.disconnect();
keepGoing = false;
} catch (Exception e) {
log.warn("Try #{}. Error posting: {}", tries, e.getMessage());
log.warn("Pausing for 1 second then trying again...");
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException f) {
log.warn("Sleeping has been interrupted: {}", f.getMessage());
}
}
}
return jsonString.toString();
}
UPDATE: In case if you can not use java 8 and lambda, you can always switch to creating anonymous class:
return commonMethod(urlString, payload, new ConnectionWorker() {
#Override
public void run(HttpURLConnection connection) throws IOException {
// UNIQUE CODE START
CallHandler.this.prepareConnection(connection, POST);
OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
// UNIQUE CODE END
}
});

httpsurlconnection posts string wrongly in java application

In my java application I used a Httpsurlconnection to post some string data to the server. When I test this code on android, it works perfectly. However, in a java application it does not work. Client java application is as follows:
public static void main(String[] args) {
disableSslVerification();
new HttpsClient().testIt();
}
private void testIt() {
String https_url = "https://XXX.XX.XXX.XXX:XXXX/XXXXX/TestServlet";
URL url;
try {
url = new URL(https_url);
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
print_content(con, "test");
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
private void print_content(HttpsURLConnection connection, String data) {
if (connection != null) {
try {
connection.setConnectTimeout(6000);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
Charset cSet = Charset.forName("UTF-8");
byte bytes[] = data.getBytes(cSet);
connection.setRequestProperty("Content-Length", ""
+ Integer.toString(bytes.length));
connection.setRequestProperty("Content-Language", "tr");
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(
connection.getOutputStream());
wr.write(bytes);
wr.flush();
wr.close();
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is, cSet));
String line;
StringBuilder response = new StringBuilder();
while ((line = rd.readLine()) != null) {
response.append(line);
response.append(System.getProperty("line.separator"));
}
rd.close();
System.out.println(response.toString());
} catch (Exception e) {
System.out.println(e.getMessage());
} finally {
if (connection != null) {
connection.disconnect();
}
}
}
}
And the servlet is as follows:
#Override
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
String s = getHTML(request);
try {
out.print("received data:");
out.print(s);
} finally {
out.close();
}
}
private String getHTML(HttpServletRequest request) throws IOException {
int n = request.getContentLength();
if (n < 1) {
return "";
}
byte bytes[] = new byte[n];
request.getInputStream().read(bytes);
return new String(bytes, "UTF-8");
}
When I run this application, servlet's response is:
received data:t☐☐☐
Always only the first character is correctly send to the servlet. The same code works perfect on android. Can anyone help me please? Thanks...

I can't see an obvious problem with your code that would cause this.
Can anyone help me please?
I suggest that you take a methodical approach to investigating the problem. Use a packet sniffer to check what is actually being sent over the wire. Check that the actual headers in the request and response are correct. Check that the request and response bodies are really properly encoded UTF-8 ...
What you find in your investigation / evidence gathering will help you figure out where the problem (or problems) are occurring ... and that will allow you to home in on the part(s) of your code that is/are responsible.

request.getInputStream().read(bytes);
You might need to do this read in a loop. At the very least, check how many bytes have been read. The array appears to be empty except for the first char.
Reads some number of bytes from the input stream and stores them into
the buffer array b. The number of bytes actually read is returned as
an integer. This method blocks until input data is available, end of
file is detected, or an exception is thrown.

Java HTTPS Client [duplicate]

I'm trying to find Java's equivalent to Groovy's:
String content = "http://www.google.com".toURL().getText();
I want to read content from a URL into string. I don't want to pollute my code with buffered streams and loops for such a simple task. I looked into apache's HttpClient but I also don't see a one or two line implementation.

Now that some time has passed since the original answer was accepted, there's a better approach:
String out = new Scanner(new URL("http://www.google.com").openStream(), "UTF-8").useDelimiter("\\A").next();
If you want a slightly fuller implementation, which is not a single line, do this:
public static String readStringFromURL(String requestURL) throws IOException
{
try (Scanner scanner = new Scanner(new URL(requestURL).openStream(),
StandardCharsets.UTF_8.toString()))
{
scanner.useDelimiter("\\A");
return scanner.hasNext() ? scanner.next() : "";
}
}

This answer refers to an older version of Java. You may want to look at ccleve's answer.
Here is the traditional way to do this:
import java.net.*;
import java.io.*;
public class URLConnectionReader {
public static String getText(String url) throws Exception {
URL website = new URL(url);
URLConnection connection = website.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
connection.getInputStream()));
StringBuilder response = new StringBuilder();
String inputLine;
while ((inputLine = in.readLine()) != null)
response.append(inputLine);
in.close();
return response.toString();
}
public static void main(String[] args) throws Exception {
String content = URLConnectionReader.getText(args[0]);
System.out.println(content);
}
}
As #extraneon has suggested, ioutils allows you to do this in a very eloquent way that's still in the Java spirit:
InputStream in = new URL( "http://jakarta.apache.org" ).openStream();
try {
System.out.println( IOUtils.toString( in ) );
} finally {
IOUtils.closeQuietly(in);
}

Or just use Apache Commons IOUtils.toString(URL url), or the variant that also accepts an encoding parameter.

There's an even better way as of Java 9:
URL u = new URL("http://www.example.com/");
try (InputStream in = u.openStream()) {
return new String(in.readAllBytes(), StandardCharsets.UTF_8);
}
Like the original groovy example, this assumes that the content is UTF-8 encoded. (If you need something more clever than that, you need to create a URLConnection and use it to figure out the encoding.)

Now that more time has passed, here's a way to do it in Java 8:
URLConnection conn = url.openConnection();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8))) {
pageText = reader.lines().collect(Collectors.joining("\n"));
}

Additional example using Guava:
URL xmlData = ...
String data = Resources.toString(xmlData, Charsets.UTF_8);

Java 11+:
URI uri = URI.create("http://www.google.com");
HttpRequest request = HttpRequest.newBuilder(uri).build();
String content = HttpClient.newHttpClient().send(request, BodyHandlers.ofString()).body();

If you have the input stream (see Joe's answer) also consider ioutils.toString( inputstream ).
http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html#toString(java.io.InputStream)

The following works with Java 7/8, secure urls, and shows how to add a cookie to your request as well. Note this is mostly a direct copy of this other great answer on this page, but added the cookie example, and clarification in that it works with secure urls as well ;-)
If you need to connect to a server with an invalid certificate or self signed certificate, this will throw security errors unless you import the certificate. If you need this functionality, you could consider the approach detailed in this answer to this related question on StackOverflow.
Example
String result = getUrlAsString("https://www.google.com");
System.out.println(result);
outputs
<!doctype html><html itemscope="" .... etc
Code
import java.net.URL;
import java.net.URLConnection;
import java.io.BufferedReader;
import java.io.InputStreamReader;
public static String getUrlAsString(String url)
{
try
{
URL urlObj = new URL(url);
URLConnection con = urlObj.openConnection();
con.setDoOutput(true); // we want the response
con.setRequestProperty("Cookie", "myCookie=test123");
con.connect();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
StringBuilder response = new StringBuilder();
String inputLine;
String newLine = System.getProperty("line.separator");
while ((inputLine = in.readLine()) != null)
{
response.append(inputLine + newLine);
}
in.close();
return response.toString();
}
catch (Exception e)
{
throw new RuntimeException(e);
}
}

Here's Jeanne's lovely answer, but wrapped in a tidy function for muppets like me:
private static String getUrl(String aUrl) throws MalformedURLException, IOException
{
String urlData = "";
URL urlObj = new URL(aUrl);
URLConnection conn = urlObj.openConnection();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8)))
{
urlData = reader.lines().collect(Collectors.joining("\n"));
}
return urlData;
}

URL to String in pure Java
Example call to get payload from http get call
String str = getStringFromUrl("YourUrl");
Implementation
You can use the method described in this answer, on How to read URL to an InputStream and combine it with this answer on How to read InputStream to String.
The outcome will be something like
public String getStringFromUrl(URL url) throws IOException {
return inputStreamToString(urlToInputStream(url,null));
}
public String inputStreamToString(InputStream inputStream) throws IOException {
try(ByteArrayOutputStream result = new ByteArrayOutputStream()) {
byte[] buffer = new byte[1024];
int length;
while ((length = inputStream.read(buffer)) != -1) {
result.write(buffer, 0, length);
}
return result.toString(UTF_8);
}
}
private InputStream urlToInputStream(URL url, Map<String, String> args) {
HttpURLConnection con = null;
InputStream inputStream = null;
try {
con = (HttpURLConnection) url.openConnection();
con.setConnectTimeout(15000);
con.setReadTimeout(15000);
if (args != null) {
for (Entry<String, String> e : args.entrySet()) {
con.setRequestProperty(e.getKey(), e.getValue());
}
}
con.connect();
int responseCode = con.getResponseCode();
/* By default the connection will follow redirects. The following
* block is only entered if the implementation of HttpURLConnection
* does not perform the redirect. The exact behavior depends to
* the actual implementation (e.g. sun.net).
* !!! Attention: This block allows the connection to
* switch protocols (e.g. HTTP to HTTPS), which is <b>not</b>
* default behavior. See: https://stackoverflow.com/questions/1884230
* for more info!!!
*/
if (responseCode < 400 && responseCode > 299) {
String redirectUrl = con.getHeaderField("Location");
try {
URL newUrl = new URL(redirectUrl);
return urlToInputStream(newUrl, args);
} catch (MalformedURLException e) {
URL newUrl = new URL(url.getProtocol() + "://" + url.getHost() + redirectUrl);
return urlToInputStream(newUrl, args);
}
}
/*!!!!!*/
inputStream = con.getInputStream();
return inputStream;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
Pros
It is pure java
It can be easily enhanced by adding different headers as a map (instead of passing a null object, like the example above does), authentication, etc.
Handling of protocol switches is supported

HttpURLConnection - returned format

I'm using HttpUrlConnection to consume REST services. When I do GET, like the one I show below, I don't want to get the information that is returned char by char. I want to get the result in the format that it is returned. Here, for example, I want to get a boolean, and not something like System.out.print((char) ch);. How can I receive it?
I know that I can parse a String into a Boolean type, but if I receive another data type?
public class SensorGetDoorStatus {
public static void main(String[] args) throws Exception
{
HttpURLConnection urlConnection = null;
try {
String webPage = "http://localhost:8080/LULServices/webresources/services.sensors/doorstatus";
String name = "xxxx";
String password = "xxxx";
Authenticator myAuth = new Authenticator()
{
final String USERNAME = "xxxx";
final String PASSWORD = "xxxxx";
#Override
protected PasswordAuthentication getPasswordAuthentication()
{
return new PasswordAuthentication(USERNAME, PASSWORD.toCharArray());
}
};
Authenticator.setDefault(myAuth);
String authString = name + ":" + password;
byte[] authEncBytes = Base64.encodeBase64(authString.getBytes());
String authStringEnc = new String(authEncBytes);
URL urlToRequest = new URL(webPage);
urlConnection = (HttpURLConnection) urlToRequest.openConnection();
urlConnection.setRequestProperty("Authorization", "Basic " + authStringEnc);
System.out.println("Authorization : Basic " + authStringEnc);
urlConnection.setRequestMethod("GET");
urlConnection.setRequestProperty("Accept", "application/json");
urlConnection.setReadTimeout(15*1000);
urlConnection.connect();
InputStream inStrm = urlConnection.getInputStream();
int ch;
while (((ch = inStrm.read()) != -1))
System.out.print((char) ch);
inStrm.close();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
System.out.println("Failure processing URL");
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
finally {
if (urlConnection != null) {
urlConnection.disconnect();
}
}
}
}

I'd play around with BufferedReader: it allows to read an InputStream line by line, and you might be able to parse your data that way.
If your data is in a predefined format like XML or JSON, you'd better use a library to parse your response data, though.

You might want to look at using DataInputStream. You can use the method readBoolean method of this class to read a boolean.
The DataInputStream and DataOutputStream also provides you with methods to write and read specific data types for example int, float, long etc
You should have written writeBoolean of class DataOutputStream on the other end from where you are sending data.
The code will look like follwoing:
InputStream inStrm = urlConnection.getInputStream();
DataInputStream doi = new DataInputStream(inStrm);
boolean bol = doi.readBoolean();
doi.close();
inStrm.close();

Error reading xml from doPost HttpServletRequest

I am trying to read the XML file being posted using http doPost method. While parsing using SAXParser it throws an Exception:
Content is not allowed in prolog.
The doPost code is:
protected void doPost(HttpServletRequest request, HttpServletResponse response)
{
ServletInputStream httpIn = request.getInputStream();
byte[] httpInData = new byte[request.getContentLength()];
StringBuffer readBuffer = new StringBuffer();
int retVal = -1;
while ((retVal = httpIn.read(httpInData)) != -1)
{
for (int i=0; i<retVal; i++)
{
readBuffer.append(Character.toString((char)httpInData[i]));
}
}
System.out.println("XML Received" + readBuffer);
try
{
SAXParser parser = SAXParserFactory.newInstance().newSAXParser();
ByteArrayInputStream inputStream = new ByteArrayInputStream(
readBuffer.toString().getBytes("UTF-8"));
final XmlParser xmlParser = new XmlParser();
parser.parse(inputStream, xmlParser);
}
catch (Exception e)
{
System.out.println("Exception parsing the xml request" + e);
}
}
This is the JUnit I am testing with:
public static void main(String args[])
{
StringBuffer buffer = new StringBuffer();
buffer.append("<?xml version=\"1.0\" encoding=\"UTF-8\"?>");
buffer.append("<person>");
buffer.append("<name>abc</name>");
buffer.append("<age>25</age>");
buffer.append("</person>");
try
{
urlParameters = URLEncoder.encode(buffer.toString(), "UTF-8");
}
catch (Exception e1)
{
e1.printStackTrace();
}
String targetURL = "http://localhost:8888/TestService";
URL url;
HttpURLConnection connection = null;
try
{
//Create connection
url = new URL(targetURL);
connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type", "application/xml");
connection.setRequestProperty("Content-Length", "" +
Integer.toString(urlParameters.getBytes("UTF-8").length));
connection.setRequestProperty("Content-Language", "en-US");
connection.setUseCaches (false);
connection.setDoInput(true);
connection.setDoOutput(true);
//Send request
DataOutputStream wr = new DataOutputStream (
connection.getOutputStream ());
wr.writeBytes (urlParameters);
wr.flush ();
wr.close ();
}
catch (Exception e)
{
e.printStackTrace();
}
The XML output in the servlet that I am getting is something like this:
XML Received %3C%3Fxml+version%3D%221.0%22+encoding%3D%22UTF-8%22%3F%3E%3Cperson%3E%
So this is throwing an exception in SAXparser:
What am I doing wrong? Am I sending the XML in wrong way or reading it wrong way?

You assume
httpInData[i]
is a char, while it is a byte. Your content being UTF-8, that makes a big difference. Use a Reader instead.
Then, you are URLEncoding your XML, which is useless, as it is a POST data. Don't encode it, simply send the data.
replace
urlParameters = URLEncoder.encode(buffer.toString(), "UTF-8");
by
urlParameters = buffer.toString();
Also, the name urlParameter is poorly chosen, as this is a single post body, doesn't go in the url, and isn't really a parameter.