I'm trying to find Java's equivalent to Groovy's:
String content = "http://www.google.com".toURL().getText();
I want to read content from a URL into string. I don't want to pollute my code with buffered streams and loops for such a simple task. I looked into apache's HttpClient but I also don't see a one or two line implementation.
Now that some time has passed since the original answer was accepted, there's a better approach:
String out = new Scanner(new URL("http://www.google.com").openStream(), "UTF-8").useDelimiter("\\A").next();
If you want a slightly fuller implementation, which is not a single line, do this:
public static String readStringFromURL(String requestURL) throws IOException
{
try (Scanner scanner = new Scanner(new URL(requestURL).openStream(),
StandardCharsets.UTF_8.toString()))
{
scanner.useDelimiter("\\A");
return scanner.hasNext() ? scanner.next() : "";
}
}
This answer refers to an older version of Java. You may want to look at ccleve's answer.
Here is the traditional way to do this:
import java.net.*;
import java.io.*;
public class URLConnectionReader {
public static String getText(String url) throws Exception {
URL website = new URL(url);
URLConnection connection = website.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
connection.getInputStream()));
StringBuilder response = new StringBuilder();
String inputLine;
while ((inputLine = in.readLine()) != null)
response.append(inputLine);
in.close();
return response.toString();
}
public static void main(String[] args) throws Exception {
String content = URLConnectionReader.getText(args[0]);
System.out.println(content);
}
}
As #extraneon has suggested, ioutils allows you to do this in a very eloquent way that's still in the Java spirit:
InputStream in = new URL( "http://jakarta.apache.org" ).openStream();
try {
System.out.println( IOUtils.toString( in ) );
} finally {
IOUtils.closeQuietly(in);
}
Or just use Apache Commons IOUtils.toString(URL url), or the variant that also accepts an encoding parameter.
There's an even better way as of Java 9:
URL u = new URL("http://www.example.com/");
try (InputStream in = u.openStream()) {
return new String(in.readAllBytes(), StandardCharsets.UTF_8);
}
Like the original groovy example, this assumes that the content is UTF-8 encoded. (If you need something more clever than that, you need to create a URLConnection and use it to figure out the encoding.)
Now that more time has passed, here's a way to do it in Java 8:
URLConnection conn = url.openConnection();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8))) {
pageText = reader.lines().collect(Collectors.joining("\n"));
}
Additional example using Guava:
URL xmlData = ...
String data = Resources.toString(xmlData, Charsets.UTF_8);
Java 11+:
URI uri = URI.create("http://www.google.com");
HttpRequest request = HttpRequest.newBuilder(uri).build();
String content = HttpClient.newHttpClient().send(request, BodyHandlers.ofString()).body();
If you have the input stream (see Joe's answer) also consider ioutils.toString( inputstream ).
http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html#toString(java.io.InputStream)
The following works with Java 7/8, secure urls, and shows how to add a cookie to your request as well. Note this is mostly a direct copy of this other great answer on this page, but added the cookie example, and clarification in that it works with secure urls as well ;-)
If you need to connect to a server with an invalid certificate or self signed certificate, this will throw security errors unless you import the certificate. If you need this functionality, you could consider the approach detailed in this answer to this related question on StackOverflow.
Example
String result = getUrlAsString("https://www.google.com");
System.out.println(result);
outputs
<!doctype html><html itemscope="" .... etc
Code
import java.net.URL;
import java.net.URLConnection;
import java.io.BufferedReader;
import java.io.InputStreamReader;
public static String getUrlAsString(String url)
{
try
{
URL urlObj = new URL(url);
URLConnection con = urlObj.openConnection();
con.setDoOutput(true); // we want the response
con.setRequestProperty("Cookie", "myCookie=test123");
con.connect();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
StringBuilder response = new StringBuilder();
String inputLine;
String newLine = System.getProperty("line.separator");
while ((inputLine = in.readLine()) != null)
{
response.append(inputLine + newLine);
}
in.close();
return response.toString();
}
catch (Exception e)
{
throw new RuntimeException(e);
}
}
Here's Jeanne's lovely answer, but wrapped in a tidy function for muppets like me:
private static String getUrl(String aUrl) throws MalformedURLException, IOException
{
String urlData = "";
URL urlObj = new URL(aUrl);
URLConnection conn = urlObj.openConnection();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8)))
{
urlData = reader.lines().collect(Collectors.joining("\n"));
}
return urlData;
}
URL to String in pure Java
Example call to get payload from http get call
String str = getStringFromUrl("YourUrl");
Implementation
You can use the method described in this answer, on How to read URL to an InputStream and combine it with this answer on How to read InputStream to String.
The outcome will be something like
public String getStringFromUrl(URL url) throws IOException {
return inputStreamToString(urlToInputStream(url,null));
}
public String inputStreamToString(InputStream inputStream) throws IOException {
try(ByteArrayOutputStream result = new ByteArrayOutputStream()) {
byte[] buffer = new byte[1024];
int length;
while ((length = inputStream.read(buffer)) != -1) {
result.write(buffer, 0, length);
}
return result.toString(UTF_8);
}
}
private InputStream urlToInputStream(URL url, Map<String, String> args) {
HttpURLConnection con = null;
InputStream inputStream = null;
try {
con = (HttpURLConnection) url.openConnection();
con.setConnectTimeout(15000);
con.setReadTimeout(15000);
if (args != null) {
for (Entry<String, String> e : args.entrySet()) {
con.setRequestProperty(e.getKey(), e.getValue());
}
}
con.connect();
int responseCode = con.getResponseCode();
/* By default the connection will follow redirects. The following
* block is only entered if the implementation of HttpURLConnection
* does not perform the redirect. The exact behavior depends to
* the actual implementation (e.g. sun.net).
* !!! Attention: This block allows the connection to
* switch protocols (e.g. HTTP to HTTPS), which is <b>not</b>
* default behavior. See: https://stackoverflow.com/questions/1884230
* for more info!!!
*/
if (responseCode < 400 && responseCode > 299) {
String redirectUrl = con.getHeaderField("Location");
try {
URL newUrl = new URL(redirectUrl);
return urlToInputStream(newUrl, args);
} catch (MalformedURLException e) {
URL newUrl = new URL(url.getProtocol() + "://" + url.getHost() + redirectUrl);
return urlToInputStream(newUrl, args);
}
}
/*!!!!!*/
inputStream = con.getInputStream();
return inputStream;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
Pros
It is pure java
It can be easily enhanced by adding different headers as a map (instead of passing a null object, like the example above does), authentication, etc.
Handling of protocol switches is supported
Related
I need to create a hastebin paste in java, but I don't know how.
I tried this
public static String paste(String content) throws MalformedURLException, IOException {
URL url = new URL("https://hasteb.in/documents");
URLConnection con = url.openConnection();
HttpURLConnection http = (HttpURLConnection) con;
http.addRequestProperty("data", content);
http.setRequestMethod("POST");
http.setDoOutput(true);
InputStream in = new BufferedInputStream(http.getInputStream());
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder entirePage = new StringBuilder();
String inputLine;
while ((inputLine = reader.readLine()) != null) {
entirePage.append(inputLine);
}
reader.close();
if (!(entirePage.toString().contains("\"key\":\""))) {
return "UNKNOWN";
}
return "https://hasteb.in/"+entirePage.toString().split("\"key\":\"")[1].split("\",")[0];
}
But it does not work.
Error is Caused by: java.lang.IllegalArgumentException: Illegal character(s) in message header value
Any help?
To do a POST request using HttpURLConnection you need to write the request body using the provided OutputStream functionality rather than setting a HTTP header named "data":
try (OutputStream out = http.getOutputStream()) {
out.write(content.getBytes());
out.flush();
}
This needs to happen BEFORE you start reading from the InputStream (and I would suggest using try-with-resources for that as well).
I am new to spring``REST api design. Somehow I have created a api (producer). Now I want to consume this same API. I have never consumed API before. How can I do so ? I have tried using HttpURLConnection but couldn't do it. Below is my code for producer api. Now I want to call this API using java code.
Rest Api (producer)
#PostMapping(value="register")
public ResponseEntity<CoreResponseHandler> doRegister(#RequestPart String json,#RequestParam(required=false) MultipartFile file) throws JsonParseException, JsonMappingException, IOException {
try {
String imgPath = env.getProperty("image_path");
String outputDir = imgPath;
RegEntity reg = new ObjectMapper().readValue(json, RegEntity.class);
String result = validateInsertRegBean(reg);
if(!result.equalsIgnoreCase("success")) {
return new ResponseEntity<CoreResponseHandler>(new SuccessResponseBeanRefined(HttpStatus.BAD_REQUEST, ResponseStatusEnum.FAILED, ApplicationResponse.Failed,result),HttpStatus.BAD_REQUEST);
}
String imagePath=null;
System.out.println("=================================");
if(file!=null && !file.isEmpty()) {
System.out.println("file NOT nOT NOT EmpTY");
String username=reg.getUsername();
File userFile = new File(outputDir+username);
if(!userFile.exists()) {
userFile.mkdir();
}
try {
file.transferTo(new File(outputDir+username+File.separator+ file.getOriginalFilename()));
System.out.println("*************** "+file.getOriginalFilename());
} catch (IOException e) {
return new ResponseEntity<CoreResponseHandler>(new SuccessResponseBeanRefined(HttpStatus.BAD_REQUEST, ResponseStatusEnum.FAILED, ApplicationResponse.Failed,"image unable to upload"),HttpStatus.BAD_REQUEST);
}
File ff[] = userFile.listFiles();
for(File f:ff) {
if(f.getName().contains("thumbnail"))
f.delete();
}
Thumbnails.of(new File(outputDir+username+File.separator).listFiles())
.size(100, 100)
.outputFormat("jpg")
.toFiles(Rename.SUFFIX_HYPHEN_THUMBNAIL);
imagePath=outputDir+username+File.separator+file.getOriginalFilename();
}
Date dt = new Date();
long lngDt = dt.getTime();
Ofuser ofuser = new Ofuser();
ofuser.setBlockedNum(null);
ofuser.setEmail(reg.getEmail());
ofuser.setEncryptedPassword(reg.getEncryptedPassword());
ofuser.setName(reg.getName());
ofuser.setStatus(reg.getStatus());
ofuser.setUsername(reg.getUsername());
ofuser.setVcardResize(null);
ofuser.setImage(null);
ofuser.setPlainPassword(null);
ofuser.setCreationDate(lngDt+"");
ofuser.setModificationDate(lngDt+"");
ofuser.setImagePath(imagePath);
Ofuser tempuser = ofuserService.save(ofuser);
IdMaster idMaster0 = new IdMaster();
idMaster0.setUsername(tempuser.getUsername());
idMaster0.setUserId(lngDt+"");
IdMaster tempidMaster = idMasterService.save(idMaster0);
if(tempuser!=null && tempidMaster!=null) {
System.out.println("################## "+tempuser.getUsername());
return new ResponseEntity<CoreResponseHandler>(
new SuccessResponseBean(HttpStatus.OK, ResponseStatusEnum.SUCCESSFUL, ApplicationResponse.SUCCESSFUL),
HttpStatus.OK);
}
else {
return new ResponseEntity<CoreResponseHandler>(
new SuccessResponseBean(HttpStatus.BAD_REQUEST, ResponseStatusEnum.FAILED, ApplicationResponse.Failed),
HttpStatus.BAD_REQUEST);
}
}catch(Exception ex) {
ex.printStackTrace();
return new ResponseEntity<CoreResponseHandler>(new SuccessResponseBeanRefined(HttpStatus.BAD_REQUEST, ResponseStatusEnum.FAILED, ApplicationResponse.Failed,"something went wrong!!"),HttpStatus.BAD_REQUEST);
}
}
postman screenshot
On postman its working fine. Now I want to consume this very api programatically. how can I do this.
Basically I need to call this method using some code:
#PostMapping(value="register")
public ResponseEntity<CoreResponseHandler> doRegister(#RequestPart String json,#RequestParam(required=false) MultipartFile file){}
don't have much experience in doing so..please guide.
============================================================================
Ok, after much digging in google I made following code but still not able to call rest api. My code:
package com.google.reg.utils;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import org.json.JSONException;
import org.json.JSONObject;
public class Test {
public static void main(String[] args) {
try {
URL url = new URL("http://localhost:8012/register");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Type", "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW charset=utf-8");
conn.setDoOutput(true);
conn.setDoInput(true);
JSONObject jsonObject = buidJsonObject();
setPostRequestContent(conn, jsonObject);
conn.connect();
System.out.println(conn.getResponseCode());
if(conn.getResponseCode()==200){
BufferedReader reader= new BufferedReader(new InputStreamReader(conn.getInputStream()));
StringBuilder stringBuilder = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null)
{
stringBuilder.append(line + "\n");
}
System.out.println(stringBuilder+" <<<<<<<<<<<<<<<<<<<<<<<<<" );
jsonObject=new JSONObject(stringBuilder.toString());
}
else{
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getErrorStream()));
StringBuilder stringBuilder = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null)
{
stringBuilder.append(line + "\n");
}
System.out.println(stringBuilder+" <<<<<<<<<<<<<<<<<<<<<<<<<" );
System.out.println("ERRORERRORERRORERRORERRORERRORERRORERRORERROR"+conn.getResponseCode());
}
}catch(Exception ex) {
ex.printStackTrace();
}
}
private static JSONObject buidJsonObject() throws JSONException {
JSONObject jsonObject = new JSONObject();
jsonObject.accumulate("username", "129898912");
jsonObject.accumulate("encryptedPassword", "encpass");
jsonObject.accumulate("name", "fish");
jsonObject.accumulate("email", "fish#fish.com");
jsonObject.accumulate("status", "active");
return jsonObject;
}
private static void setPostRequestContent(HttpURLConnection conn,
JSONObject jsonObject) throws IOException {
OutputStream os = conn.getOutputStream();
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(os, "UTF-8"));
writer.write(jsonObject.toString());
writer.flush();
writer.close();
os.close();
}
}
but still no success. Below is the error I get in console:
400
<html><body><h1>Whitelabel Error Page</h1><p>This application has no explicit mapping for /error, so you are seeing this as a fallback.</p><div id='created'>Wed Sep 26 12:28:17 UTC 2018</div><div>There was an unexpected error (type=Bad Request, status=400).</div><div>Required request part 'json' is not present</div></body></html>
<<<<<<<<<<<<<<<<<<<<<<<<<
ERRORERRORERRORERRORERRORERRORERRORERRORERROR400
Since you are using Spring, you can use RestTemplate which is again a spring library.
Please refer rest template
This is actually one client, but there are more. Its up to you do decide which one you should use and which suites best to you.
As #Damith said, rest template is a good library, and it's embedded on the springboot dependency tree. But also there are some others library, such as OkHttp or JAX-RX which can help you. My favorite one is OkHttp3, Good Luck.
I have a command line:
curl -u username:passwd -k "https://myserver/a/b/c"
It works and I will get right information when I call it under Ubuntu or Cygwin.
Now I am willing to accomplish it in Java. So I have Java code like that:
public class Test {
public static String auth = "username:passwd";
public static String url = "https:/myserver/a/b/c";
public static void main(String[] args) {
try {
URL url = new URL(url);
final byte[] authBytes = auth.getBytes(StandardCharsets.UTF_8);
String encoding = java.util.Base64.getEncoder().encodeToString(authBytes);
HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setRequestProperty("Authorization", encoding);
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
But It seems not work and always return HTTP response code 411.
Is there something wrong with the java code?
Thanks in advanced.
HTTP response code 411 means that "The server refuses to accept the request without a defined Content- Length."
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
HttpsUrlConnection should be able to do that for you. Check out setFixedLengthStreamingMode(). I think you will also need to setDoOutput().
I have a http end point which serves xml responses to my queries. I tried to fire the http requests like this -
import java.io.*;
import java.net.*;
public class c {
public String getHTML(String urlToRead) {
URL url;
HttpURLConnection conn;
BufferedReader rd;
String line;
String result = "";
try {
url = new URL(urlToRead);
conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
// how to store the response as an xml file on disc
rd.close();
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
return result;
}
public static void main(String args[])
{
c c = new c();
System.out.println(c.getHTML(args[0]));
}
}
Now, the endpoint sends xml response which i want to store as a xml file on disc. how can i do that. can someone help?
I have not tried it, but maybe you could start from something like this:
String dest = "c:\\filename.xml";
// this line substitutes your rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
InputStream is = conn.getInputStream();
// these others goes in your "how to store..." comment
byte[] bytes = IOUtils.toByteArray(is);
FileOutputStream fos = new FileOutputStream(dest);
fos.write(bytes);
fos.flush();
fos.close();
In my java application I used a Httpsurlconnection to post some string data to the server. When I test this code on android, it works perfectly. However, in a java application it does not work. Client java application is as follows:
public static void main(String[] args) {
disableSslVerification();
new HttpsClient().testIt();
}
private void testIt() {
String https_url = "https://XXX.XX.XXX.XXX:XXXX/XXXXX/TestServlet";
URL url;
try {
url = new URL(https_url);
HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
print_content(con, "test");
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
private void print_content(HttpsURLConnection connection, String data) {
if (connection != null) {
try {
connection.setConnectTimeout(6000);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
Charset cSet = Charset.forName("UTF-8");
byte bytes[] = data.getBytes(cSet);
connection.setRequestProperty("Content-Length", ""
+ Integer.toString(bytes.length));
connection.setRequestProperty("Content-Language", "tr");
connection.setRequestProperty("Accept-Charset", "UTF-8");
connection.setUseCaches(false);
connection.setDoInput(true);
connection.setDoOutput(true);
DataOutputStream wr = new DataOutputStream(
connection.getOutputStream());
wr.write(bytes);
wr.flush();
wr.close();
InputStream is = connection.getInputStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is, cSet));
String line;
StringBuilder response = new StringBuilder();
while ((line = rd.readLine()) != null) {
response.append(line);
response.append(System.getProperty("line.separator"));
}
rd.close();
System.out.println(response.toString());
} catch (Exception e) {
System.out.println(e.getMessage());
} finally {
if (connection != null) {
connection.disconnect();
}
}
}
}
And the servlet is as follows:
#Override
protected void doPost(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
response.setContentType("text/html;charset=UTF-8");
PrintWriter out = response.getWriter();
String s = getHTML(request);
try {
out.print("received data:");
out.print(s);
} finally {
out.close();
}
}
private String getHTML(HttpServletRequest request) throws IOException {
int n = request.getContentLength();
if (n < 1) {
return "";
}
byte bytes[] = new byte[n];
request.getInputStream().read(bytes);
return new String(bytes, "UTF-8");
}
When I run this application, servlet's response is:
received data:tâââ
Always only the first character is correctly send to the servlet. The same code works perfect on android. Can anyone help me please? Thanks...
I can't see an obvious problem with your code that would cause this.
Can anyone help me please?
I suggest that you take a methodical approach to investigating the problem. Use a packet sniffer to check what is actually being sent over the wire. Check that the actual headers in the request and response are correct. Check that the request and response bodies are really properly encoded UTF-8 ...
What you find in your investigation / evidence gathering will help you figure out where the problem (or problems) are occurring ... and that will allow you to home in on the part(s) of your code that is/are responsible.
request.getInputStream().read(bytes);
You might need to do this read in a loop. At the very least, check how many bytes have been read. The array appears to be empty except for the first char.
Reads some number of bytes from the input stream and stores them into
the buffer array b. The number of bytes actually read is returned as
an integer. This method blocks until input data is available, end of
file is detected, or an exception is thrown.