Before writing something like "why don't you use Java HTTP client such as apache, etc", I need you to know that the reason is SSL. I wish I could, they are very convenient, but I can't.
None of the available HTTP clients support GOST cipher suite, and I get handshake exception all the time. The ones which do support the suite, doesn't support SNI (they are also proprietary) - I'm returned with a wrong cert and get handshake exception over and over again.
The only solution was to configure openssl (with gost engine) and curl and finally execute the command with Java.
Having said that, I wrote a simple snippet for executing a command and getting input stream response:
public static InputStream executeCurlCommand(String finalCurlCommand) throws IOException
{
return Runtime.getRuntime().exec(finalCurlCommand).getInputStream();
}
Additionally, I can convert the returned IS to a string like that:
public static String convertResponseToString(InputStream isToConvertToString) throws IOException
{
StringWriter writer = new StringWriter();
IOUtils.copy(isToConvertToString, writer, "UTF-8");
return writer.toString();
}
However, I can't see a pattern according to which I could get a good response or a desired response header:
Here's what I mean
After executing a command (with -i flag), there might be lots and lots of information like in the screen below:
At first, I thought that I could just split it with '\n', but the thing is that a required response's header or a response itself may not satisfy the criteria (prettified JSON or long redirect URL break the rule).
Also, the static line GOST engine already loaded is a bit annoying (but I hope that I'll be able to get rid of it and nothing unrelated info like that will emerge)
I do believe that there's a pattern which I can use.
For now I can only do that:
public static String getLocationRedirectHeaderValue(String curlResponse)
{
String locationHeaderValue = curlResponse.substring(curlResponse.indexOf("Location: "));
locationHeaderValue = locationHeaderValue.substring(0, locationHeaderValue.indexOf("\n")).replace("Location: ", "");
return locationHeaderValue;
}
Which is not nice, obviosuly
Thanks in advance.
Instead of reading the whole result as a single string you might want to consider reading it line by line using a scanner.
Then keep a few status variables around. The main task would be to separate header from body. In the body you might have a payload you want to treat differently (e.g. use GSON to make a JSON object).
The nice thing: Header and Body are separated by an empty line. So your code would be along these lines:
boolean inHeader = true;
StringBuilder b = new StringBuilder;
String lastLine = "";
// Technically you would need Multimap
Map<String,String> headers = new HashMap<>();
Scanner scanner = new Scanner(yourInputStream);
while scanner.hasNextLine() {
String line = scanner.nextLine();
if (line.length() == 0) {
inHeader = false;
} else {
if (inHeader) {
// if line starts with space it is
// continuation of previous header
treatHeader(line, lastLine);
} else {
b.append(line);
b.appen(System.lineSeparator());
}
}
}
String body = b.toString();
I'm trying to send a string from a java (android) app to a node.js server.
But one character disappears somewhere in the middle and I can't really figure out why.
To send I use a HttpUrlConnection (conn) and send the string like this:
try {
OutputStream os = conn.getOutputStream();
os.write(json.getBytes());
os.close();
} catch (Exception e) {
e.printStackTrace();
}
Here is the base64 encoded string when sent, and string when received:
khVGUBH2kNAR5PPRy7v5dO5iz48Rc7benYARu78\/9wY=\n
khVGUBH2kNAR5PPRy7v5dO5iz48Rc7benYARu78/9wY=\n
so one backslash has be removed.
In node I use this:
exports.getString = function(req, res) {
var string = req.body.thestring;
}
which outputs the later of the two strings.
var express = require('express'),
http = require('http'),
stylus = require('stylus'),
nib = require('nib');
var app = express();
app.configure(function () {
app.use(express.logger('dev'));
//app.use(express.bodyParser());
app.use(express.json());
app.use(express.urlencoded());
app.use(app.router);
}
Any ideas of how I can get the missing character?
The missing backslash character is most probably disappearing in node.js side.
As per the chosen answer on the following question:
Two part question on JavaScript forward slash
As far as JS is concerned / and \ / are identical inside a string
So maybe a fix from Java's would solve your problem by using String's replaceAll method to replace all occurrences of \/ with \\/:
os.write(json.replaceAll("\\/", "\\\\/").getBytes());
Note that replaceAll returns the new string and doesn't change the original string.
Making the base64 encoding url safe solved my problem.
I am writing something like an image proxy where I receive URLs from my site front-end, and I download images , re-size them, and return smaller images for the front end and client to download from the "proxy".
This means I need to take care of all-sorts of url patterns, this is why I chose to decode the given url and than encode it using URIUtils.decode:
private String fixUrl(String fromUrl) throws URIException {
fromUrl = URIUtil.decode(fromUrl);
fromUrl = URIUtil.encodeQuery(fromUrl);
return fromUrl;
}
This should help me take care of urls that are already encoded.
My problem is that some of the urls are double encoded, and from what I saw, URIUtils.decode, performs recursive decode and this means that in cases of double encoded urls I will get a bad url that does not work.
Is there a simple way to decode only once?
I'd try to check that URL still contains character %. If it does not contain any % it is not encoded and you can stop your decoding procedure.
Easiest option I know is to use the built-in
java.net.URLDecoder.decode
In order to decode automatically only once.
If like me you have the case that sometimes URL's are double / triple encoded - you can use this recursive function in order to decode again and again until there are no "%" or "+" :
private static String completeDecode(String url) {
if(url.contains("%") || url.contains("+"))
{
try
{
return(completeDecode(java.net.URLDecoder.decode(url, "UTF-8")));
}
catch (UnsupportedEncodingException e)
{
e.printStackTrace();
}
}
return url;
}
Cheers
how to check protocol is present in URL , if not present need to append it.
is there any class to achieve this in java?
eg: String URL = www.google.com
need to get http://www.google.com
Just use String.startsWith("http://") to check this.
public String ensure_has_protocol(final String a_url)
{
if (!a_url.startsWith("http://"))
{
return "http://" + a_url;
}
return a_url;
}
EDIT:
An alternative would use a java.net.URL instance, whose constructor would throw an java.net.MalformedURLException if the URL did not contain a (legal) protocol (or was invalid for any other reason):
public URL make_url(final String a_url) throws MalformedURLException
{
try
{
return new URL(a_url);
}
catch (final MalformedURLException e)
{
}
return new URL("http://" + a_url);
}
You can use URL.toString() to obtain string representation of the URL. This is an improvement on the startsWith() approach as it guarantees that return URL is valid.
Let's say you have String url = www.google.com. String class methods would be enough for the goal of checking protocol identifiers. For example, url.startsWith("https://") would check whether a specific string is starting with the given protocol name.
However, are these controls enough for validation?
I think they aren't enough. First of all, you should define a list of valid protocol identifiers, e.g. a String array like {"http", "ftp", "https", ...}. Then you can parse your input String with regex ("://") and test your URL header whether it belongs to the list of valid protocol identifiers. And domain name validation methods are beyond this question, you can/should handle it with different techniques as well.
Just for completeness, I would do something like the following:
import com.google.common.base.Strings;
private static boolean isUrlHttps(String url){
if(Strings.isNullOrEmpty(url))
return false;
return url.toLowerCase().startsWith("https://");
}
I am trying to get a java.net.URI object from a String. The string has some characters which will need to be replaced by their percentage escape sequences. But when I use URLEncoder to encode the String with UTF-8 encoding, even the / are replaced with their escape sequences.
How can I get a valid encoded URL from a String object?
http://www.google.com?q=a b gives http%3A%2F%2www.google.com... whereas I want the output to be http://www.google.com?q=a%20b
Can someone please tell me how to achieve this.
I am trying to do this in an Android app. So I have access to a limited number of libraries.
You might try: org.apache.commons.httpclient.util.URIUtil.encodeQuery in Apache commons-httpclient project
Like this (see URIUtil):
URIUtil.encodeQuery("http://www.google.com?q=a b")
will become:
http://www.google.com?q=a%20b
You can of course do it yourself, but URI parsing can get pretty messy...
Android has always had the Uri class as part of the SDK:
http://developer.android.com/reference/android/net/Uri.html
You can simply do something like:
String requestURL = String.format("http://www.example.com/?a=%s&b=%s", Uri.encode("foo bar"), Uri.encode("100% fubar'd"));
I'm going to add one suggestion here aimed at Android users. You can do this which avoids having to get any external libraries. Also, all the search/replace characters solutions suggested in some of the answers above are perilous and should be avoided.
Give this a try:
String urlStr = "http://abc.dev.domain.com/0007AC/ads/800x480 15sec h.264.mp4";
URL url = new URL(urlStr);
URI uri = new URI(url.getProtocol(), url.getUserInfo(), url.getHost(), url.getPort(), url.getPath(), url.getQuery(), url.getRef());
url = uri.toURL();
You can see that in this particular URL, I need to have those spaces encoded so that I can use it for a request.
This takes advantage of a couple features available to you in Android classes. First, the URL class can break a url into its proper components so there is no need for you to do any string search/replace work. Secondly, this approach takes advantage of the URI class feature of properly escaping components when you construct a URI via components rather than from a single string.
The beauty of this approach is that you can take any valid url string and have it work without needing any special knowledge of it yourself.
Even if this is an old post with an already accepted answer, I post my alternative answer because it works well for the present issue and it seems nobody mentioned this method.
With the java.net.URI library:
URI uri = URI.create(URLString);
And if you want a URL-formatted string corresponding to it:
String validURLString = uri.toASCIIString();
Unlike many other methods (e.g. java.net.URLEncoder) this one replaces only unsafe ASCII characters (like ç, é...).
In the above example, if URLString is the following String:
"http://www.domain.com/façon+word"
the resulting validURLString will be:
"http://www.domain.com/fa%C3%A7on+word"
which is a well-formatted URL.
If you don't like libraries, how about this?
Note that you should not use this function on the whole URL, instead you should use this on the components...e.g. just the "a b" component, as you build up the URL - otherwise the computer won't know what characters are supposed to have a special meaning and which ones are supposed to have a literal meaning.
/** Converts a string into something you can safely insert into a URL. */
public static String encodeURIcomponent(String s)
{
StringBuilder o = new StringBuilder();
for (char ch : s.toCharArray()) {
if (isUnsafe(ch)) {
o.append('%');
o.append(toHex(ch / 16));
o.append(toHex(ch % 16));
}
else o.append(ch);
}
return o.toString();
}
private static char toHex(int ch)
{
return (char)(ch < 10 ? '0' + ch : 'A' + ch - 10);
}
private static boolean isUnsafe(char ch)
{
if (ch > 128 || ch < 0)
return true;
return " %$&+,/:;=?#<>#%".indexOf(ch) >= 0;
}
You can use the multi-argument constructors of the URI class. From the URI javadoc:
The multi-argument constructors quote illegal characters as required by the components in which they appear. The percent character ('%') is always quoted by these constructors. Any other characters are preserved.
So if you use
URI uri = new URI("http", "www.google.com?q=a b");
Then you get http:www.google.com?q=a%20b which isn't quite right, but it's a little closer.
If you know that your string will not have URL fragments (e.g. http://example.com/page#anchor), then you can use the following code to get what you want:
String s = "http://www.google.com?q=a b";
String[] parts = s.split(":",2);
URI uri = new URI(parts[0], parts[1], null);
To be safe, you should scan the string for # characters, but this should get you started.
I had similar problems for one of my projects to create a URI object from a string. I couldn't find any clean solution either. Here's what I came up with :
public static URI encodeURL(String url) throws MalformedURLException, URISyntaxException
{
URI uriFormatted = null;
URL urlLink = new URL(url);
uriFormatted = new URI("http", urlLink.getHost(), urlLink.getPath(), urlLink.getQuery(), urlLink.getRef());
return uriFormatted;
}
You can use the following URI constructor instead to specify a port if needed:
URI uri = new URI(scheme, userInfo, host, port, path, query, fragment);
Well I tried using
String converted = URLDecoder.decode("toconvert","UTF-8");
I hope this is what you were actually looking for?
The java.net blog had a class the other day that might have done what you want (but it is down right now so I cannot check).
This code here could probably be modified to do what you want:
http://svn.apache.org/repos/asf/incubator/shindig/trunk/java/common/src/main/java/org/apache/shindig/common/uri/UriBuilder.java
Here is the one I was thinking of from java.net: https://urlencodedquerystring.dev.java.net/
Or perhaps you could use this class:
http://developer.android.com/reference/java/net/URLEncoder.html
Which is present in Android since API level 1.
Annoyingly however, it treats spaces specially (replacing them with + instead of %20). To get round this we simply use this fragment:
URLEncoder.encode(value, "UTF-8").replace("+", "%20");
I ended up using the httpclient-4.3.6:
import org.apache.http.client.utils.URIBuilder;
public static void main (String [] args) {
URIBuilder uri = new URIBuilder();
uri.setScheme("http")
.setHost("www.example.com")
.setPath("/somepage.php")
.setParameter("username", "Hello Günter")
.setParameter("p1", "parameter 1");
System.out.println(uri.toString());
}
Output will be:
http://www.example.com/somepage.php?username=Hello+G%C3%BCnter&p1=paramter+1