I'm trying to get my head around the OAuth2 Java library that Google provides.
I have everything I need to make the request to Google's token endpoint manually using Springs built-in WebClient. However, this is very verbose and feels like re-inventing the wheel. It got me thinking that there must be a way to get this data using the classes provided by the library. Right?
Currently I am using the com.google.auth.oauth2.UserAuthorizer class to build up a request for the exchange of information.
val userCredentials: UserCredentials = UserAuthorizer.newBuilder()
.setClientId(googleOauthConfig.clientId)
.setTokenStore(tokenStore)
.setScopes(googleOauthConfig.scopes)
.setTokenServerUri(URI.create("https://oauth2.googleapis.com/token"))
.setCallbackUri(redirectUri)
.build()
.getCredentialsFromCode(authorizationCode, redirectUri)
The internals of getCredentialsFromCode() parses the response and it contains all the tokens. Including the id_token but, it gets discarded when constructing the UserCredentials object further down.
return UserCredentials.newBuilder()
.setClientId(clientId.getClientId())
.setClientSecret(clientId.getClientSecret())
.setRefreshToken(refreshToken)
.setAccessToken(accessToken)
.setHttpTransportFactory(transportFactory)
.setTokenServerUri(tokenServerUri)
.build(); // no mention of id_token
Regardless, I want to get this value so I can know basic information about the user such as their name, birthday and email address from a single request.
There does exist a method called idTokenWithAudience() which returns a Google ID Token from the refresh token response. If I call this, I get a token back that doesn't contain all the data that was available in the identically named id_token mentioned earlier making it a no-go either.
You can use IdTokenCredentials to access the ID token like so:
var credentials = UserCredentials.newBuilder()
.setClientId("...")
.setClientSecret("...")
.setRefreshToken("...")
.build()
.createScoped("openid email");
var idToken = IdTokenCredentials
.newBuilder()
.setIdTokenProvider((IdTokenProvider)credentials)
.build();
idToken.refresh();
System.out.println(idToken.getIdToken().getTokenValue());
I have a server using Kotlin 1.5, JDK 11, http4k v4.12, and I've got the Twilio Java SDK v8.19, hosted using Google Cloud Run.
I've created a predicate using Twilio's Java SDK RequestValidator.
import com.twilio.security.RequestValidator
import mu.KotlinLogging
import org.http4k.core.Filter
import org.http4k.core.HttpHandler
import org.http4k.core.Method
import org.http4k.core.Response
import org.http4k.core.Status
import org.http4k.core.body.form
import org.http4k.core.queries
import org.http4k.core.then
import org.http4k.core.toParametersMap
import org.http4k.filter.RequestPredicate
import org.http4k.filter.ServerFilters
import org.http4k.lens.Header
private val twilioAuthHeaderLens = Header.optional("X-Twilio-Signature")
/** Twilio's helper [RequestValidator]. */
private val twilioValidator = RequestValidator("my-auth-token")
/**
* Use the Twilio helper validator, [RequestValidator]
*/
val twilioAuthPredicate: RequestPredicate = { request ->
when (val requestSignature: String? = twilioAuthHeaderLens(request)) {
null -> {
logger.debug { "Request has no Twilio request header valid" }
false
}
else -> {
val uri: String = request.uri.toString()
val paramMap: Map<String, String?> = request.form().toMap()
logger.info { "Validating request with uri: $uri, paramMap: $paramMap, signature: $requestSignature" }
val isTwilioSignatureValid = twilioValidator.validate(uri, paramMap, requestSignature)
logger.info { "Request Twilio valid: $isTwilioSignatureValid" }
isTwilioSignatureValid
}
}
}
This works using the example Twilio provide, as demonstrated with this Kotest unit test.
(the test and the example code are mismatched - but OperatorAuth is a class that applies the twilioAuthPredicate, and ApplicationProperties fetches the Twilio auth key from a .env file.)
test("demo https://www.twilio.com/docs/usage/security") {
val twilioApiKey = "12345"
val appProps = ApplicationProperties(
TWILIO_API_AUTH_TOKEN(twilioApiKey, TEST_ENV)
)
// system-under-test
val handler: HttpHandler = OperatorAuth(appProps).then { Response(OK) }
// construct a GET request: https://mycompany.com/myapp.php?foo=1&bar=2
val urlProto = "https"
val urlBase = "mycompany.com"
val requestSignature = "0/KCTR6DLpKmkAf8muzZqo1nDgQ="
val request = Request(Method.GET, "$urlProto://$urlBase/myapp.php")
.query("foo", "1")
.query("bar", "2")
.form("CallSid", "CA1234567890ABCDE")
.form("Caller", "+12349013030")
.form("Digits", "1234")
.form("From", "+12349013030")
.form("To", "+18005551212")
.header("X-Twilio-Signature", requestSignature)
.header("X-Forwarded-Proto", urlProto)
.header("Host", urlBase)
val response = handler(request)
response shouldHaveStatus OK
}
However, aside from this simple example, no other requests work, either when creating a unit test, or when live. All Twilio requests fail validation and my server returns 401. The information in the Twilio website is completely opaque. It's incredibly frustrating. It doesn't tell me how it's calculated the hash so I can't tell what is going wrong.
Warning 15003
Message Got HTTP 401 response to https://my-gcr-server.run.app/twilio
Here is an example test using real values gathered from a log (although I have edited the identifiers).
test("real request") {
val appProps = ApplicationProperties() // this loads the Twilio Auth Key from my environment variables
val handler: HttpHandler = OperatorAuth(appProps).then { Response(OK) }
// construct a GET request
val urlProto = "https"
val urlBase = "my-gcr-server.run.app"
val requestSignature = "GATG2313LSuCYRbPASD4axJ26XyTk="
val request = Request(Method.GET, "$urlProto://$urlBase/voicemail/transcript")
.query("ApplicationSid", "AP1234567890abcdefg")
.query("ApiVersion", "2010-04-01")
.query("Called", "")
.query("Caller", "client:Anonymous")
.query("CallStatus", "ringing")
.query("CallSid", "CA1234567890abcdefg")
.query("From", "client:Anonymous")
.query("To", "")
.query("Direction", "inbound")
.query("AccountSid", "AC1234567890abcdefg")
// note, changing these variables to be form parameters doesn't affect the result, Twilio's validator still says the request is invalid.
.header("X-Twilio-Signature", requestSignature)
.header("I-Twilio-Idempotency-Token", "337aaaa-1111-2222-3333-ffffb5333")
.header("Content-Type", "text/html")
.header("User-Agent: ", "TwilioProxy/1.1")
.header("X-Forwarded-Proto", urlProto)
.header("Host", urlBase)
val response = handler(request)
response shouldHaveStatus OK // this fails, Status: expected:<200 OK> but was:<401 Unauthorized>
}
Sometimes validation fails because of Google Cloud. I had previously hosted my server on Google Cloud Functions until I discovered there's an issue where GCF silently omits part of the URI https://github.com/GoogleCloudPlatform/functions-framework-java/issues/90
There is also an issue where if a request is 'modified', for example if I set a Twilio callback URL to include a query param, e.g. https://my-gcr-server.app.run/twilio/callback?type=recording, then the Twilio signature ignores this parameter, but when validating the auth it's impossible to know which parameters Twilio is ignoring. The same is true if the headers are altered.
Is there a working method of validating that a request originates from Twilio? Or an alternative validation solution?
Update
I've just found that Twilio's RequestValidator is really under-tested, there's only one example RequestValidatorTest
Twilio developer evangelist here.
The documentation describes how the signature is created and that may show up some differences in the way you are testing. On your server, the algorithm to check the signature is:
Take the full URL of the request URL you specify for your phone number or app, from the protocol (https...) through the end of the query string (everything after the ?).
If the request is a POST, sort all of the POST parameters alphabetically (using Unix-style case-sensitive sorting order).
Iterate through the sorted list of POST parameters, and append the variable name and value (with no delimiters) to the end of the URL string.
Sign the resulting string with HMAC-SHA1 using your AuthToken as the key (remember, your AuthToken's case matters!).
Base64 encode the resulting hash value.
Compare your hash to ours, submitted in the X-Twilio-Signature header. If they match, then you're good to go.
You are using GET requests, so you can discard steps 2 and 3.
There are some things I can see from this algorithm that might cause differences in the way you are testing the validator.
Your error from testing in real life used the URL https://my-gcr-server.run.app/twilio, but your test script from a real request uses https://my-gcr-server.run.app/voicemail/transcript. The URL matters in the generation of the signature.
Your test also adds query parameters to the request, but it's hard to know what the order of those params will be. The order of the query params in the URL should be the exact same as the URL Twilio made the request to.
On the other hand, if the original Twilio request was a POST request, then those parameters should be added as form parameters, as the algorithm takes the form parameters, sorts them and appends them to the URL, with no delimiters.
You said:
There is also an issue where if a request is 'modified', for example if I set a Twilio callback URL to include a query param, e.g. https://my-gcr-server.app.run/twilio/callback?type=recording, then the Twilio signature ignores this parameter, but when validating the auth it's impossible to know which parameters Twilio is ignoring. The same is true if the headers are altered.
This is not true, a query param is part of the URL, as I've said above. Twilio does not ignore parameters, it deals with them according to the algorithm described above. As for headers, aside from the X-Twilio-Signature which is used to test the signature against, they do not come into play.
Having said all that, I am not sure why a real life request would fail the validator as it should be handling all of the things I have discussed above. You can inspect the code used to validate a request and get a signature.
In your code:
val uri: String = request.uri.toString()
val paramMap: Map<String, String?> = request.form().toMap()
logger.info { "Validating request with uri: $uri, paramMap: $paramMap, signature: $requestSignature" }
val isTwilioSignatureValid = twilioValidator.validate(uri, paramMap, requestSignature)
logger.info { "Request Twilio valid: $isTwilioSignatureValid" }
isTwilioSignatureValid
Can you guarantee that the uri is indeed the original URL that Twilio made the request to, not a URL that has been parsed into parts and put back together with the query parameters in a different order? In a GET request, does request.form().toMap() return an empty Map?
Sorry this isn't a full answer, I'm not much of a Java/Kotlin developer. I'm hoping this gives you a good idea of what to look into though.
I was having trouble because I was testing with ngrok to route the request to my local server while developing. I had was I was running the algorithm (see above in philnash 's answer as per the Twilio docs )
However while I was setting the callback to the https ngrok enpoint which Twilio used to calculate the signature, the actual request that cam to me was the http endpoint, ngrok forwards the https to http on the free account.
so I was testing a http endpoint but Twilio was calculating against the https endpoint.
when I told Twilio to callback on the http endpoint there was no ngrok misdirection and the signature matched!
Also I notice from the "validate a request: and "get a signature" links above in philnash's answer that the code tried both with a port (eg 443 or 80) in the URL and without and accepts either signature as a match.
I am currently using Identity Server 4 which is .Net Core based to issue JWT tokens. I have a .Net Core web api that has middleware in order for the JWT to be validated in the Startup.cs:
services.AddAuthentication("Bearer")
.AddJwtBearer("Bearer", options =>
{
options.Authority = "http://localhost:5005";
options.Audience = "api1";
});
As we can see, its not asking for much except the location of the token server and who the api is which is `api1. Obviously, its doing some more complex things under the hood.
I found a Java based equivalent to the middleware above, which validates a JWT:
String token = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXUyJ9.eyJpc3MiOiJhdXRoMCJ9.AbIJTDMFc7yUa5MhvcP03nJPyCPzZtQcGEp-zWfOkEE";
RSAPublicKey publicKey = //Get the key instance
RSAPrivateKey privateKey = //Get the key instance
try {
Algorithm algorithm = Algorithm.RSA256(publicKey, privateKey);
JWTVerifier verifier = JWT.require(algorithm)
.withIssuer("auth0")
.build(); //Reusable verifier instance
DecodedJWT jwt = verifier.verify(token);
} catch (JWTVerificationException exception){
//Invalid signature/claims
}
This is from HERE as was recommended by the jwt.io site.
Basically I want to be able to implement something in Java that is doing the same thing as the .Net Core code above, but its clearly asking for things like a Public and Private key that I have no idea how to provide at this point, as the JWT is coming through the header of the request.
Under the hood, the Microsoft JWT middleware is going to IdentityServer's discovery endpoint and loading in configuration such as the issuer and JWKS (public keys). The discovery document is always hosted on /.well-known/openid-configuration.
To validate the token, you will at least need the public keys from the JWKS. In the past I've loaded it it using the jwks-rsa library: https://www.scottbrady91.com/Kotlin/JSON-Web-Token-Verification-in-Ktor-using-Kotlin-and-Java-JWT
When validating the access token, as a minimum, you'll also need to check the token's audience (is the token intended for you) and if it has expired.
I have succeed using openID and OAuth separately, but I can't make them work together.
Am I doing something incorrect:
String userSuppliedString = "https://www.google.com/accounts/o8/id";
ConsumerManager manager = new ConsumerManager();
String returnToUrl = "http://example.com:8080/app-test-1.0-SNAPSHOT/GAuthorize";
List<DiscoveryInformation> discoveries = manager.discover(userSuppliedString);
DiscoveryInformation discovered = manager.associate(discoveries);
AuthRequest authReq = manager.authenticate(discovered, returnToUrl);
session.put("openID-discoveries", discovered);
FetchRequest fetch = FetchRequest.createFetchRequest();
fetch.addAttribute("email","http://schema.openid.net/contact/email",true);
fetch.addAttribute("oauth", "http://specs.openid.net/extensions/oauth/1.0",true);
fetch.addAttribute("consumer","example.com" ,true);
fetch.addAttribute("scope","http://www.google.com/calendar/feeds/" ,true);
authReq.addExtension(fetch);
destinationUrl = authReq.getDestinationUrl(true);
then destinationUrl is
https://www.google.com/accounts/o8/ud?openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.return_to=http%3A%2F%2Fexample.com%3A8080%2FgoogleTest%2Fauthorize&openid.realm=http%3A%2F%2Fexample.com%3A8080%2FgoogleTest%2Fauthorize&openid.assoc_handle=AMlYA9WVkS_oVNWtczp3zr3sS8lxR4DlnDS0fe-zMIhmepQsByLqvGnc8qeJwypiRQAuQvdw&openid.mode=checkid_setup&openid.ns.ext1=http%3A%2F%2Fopenid.net%2Fsrv%2Fax%2F1.0&openid.ext1.mode=fetch_request&openid.ext1.type.email=http%3A%2F%2Fschema.openid.net%2Fcontact%2Femail&openid.ext1.type.oauth=http%3A%2F%2Fspecs.openid.net%2Fextensions%2Foauth%2F1.0&openid.ext1.type.consumer=example.com&openid.ext1.type.scope=http%3A%2F%2Fwww.google.com%2Fcalendar%2Ffeeds%2F&openid.ext1.required=email%2Coauth%2Cconsumer%2Cscope"
but in the response from google request_token is missing
http://example.com:8080/googleTest/authorize?openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.mode=id_res&openid.op_endpoint=https%3A%2F%2Fwww.google.com%2Faccounts%2Fo8%2Fud&openid.response_nonce=2011-11-29T17%3A38%3A39ZEU2iBVXr_zQG5Q&openid.return_to=http%3A%2F%2Fexample.com%3A8080%2FgoogleTest%2Fauthorize&openid.assoc_handle=AMlYA9WVkS_oVNWtczp3zr3sS8lxR4DlnDS0fe-zMIhmepQsByLqvGnc8qeJwypiRQAuQvdw&openid.signed=op_endpoint%2Cclaimed_id%2Cidentity%2Creturn_to%2Cresponse_nonce%2Cassoc_handle%2Cns.ext1%2Cext1.mode%2Cext1.type.email%2Cext1.value.email&openid.sig=5jUnS1jT16hIDCAjv%2BwAL1jopo6YHgfZ3nUUgFpeXlw%3D&openid.identity=https%3A%2F%2Fwww.google.com%2Faccounts%2Fo8%2Fid%3Fid%3DAItOawk8YPjBcnQrqXW8tzK3aFVop63E7q-JrCE&openid.claimed_id=https%3A%2F%2Fwww.google.com%2Faccounts%2Fo8%2Fid%3Fid%3DAItOawk8YPjBcnQrqXW8tzK3aFVop63E7q-JrCE&openid.ns.ext1=http%3A%2F%2Fopenid.net%2Fsrv%2Fax%2F1.0&openid.ext1.mode=fetch_response&openid.ext1.type.email=http%3A%2F%2Fschema.openid.net%2Fcontact%2Femail&openid.ext1.value.email=example%40gmail.com
why?
In the above code, you have added OAuth extension parameters with the Attribute Exchange extension parameters. But since OAuth and Attribute Exchange are different extensions, therefore you have to create a different extension message for OAuth parameters and then add it to Authentication request message.
But since there is no mechanism to add OAuth parameters to the Authentication message, therefore you'll have to create such a mechanism. You can get information about it in the following link
http://code.google.com/p/openid4java/wiki/ExtensionHowTo
You can then use the code provided in the following link to hard code this mechanism
http://code.google.com/p/openid4java/issues/detail?id=110&q=oauth
We're trying to connect with another company's custom API which uses two-legged OAuth to authenticate a request and send us a response.
At the moment the code we have is sending a request but it's not being authenticated at the other end and so sending a UNAUTHORISED response.
The steps we have investigated so far:
Connected to the remote site using an OAuth implementation in python using the same credentials.
Asked the other company to compare our OAuth request with another that succeeds to see if there is a anything missing in ours.
After the second point above, the only difference between our request and another working request is that the oauth_token parameter is present in our request and not in others. Furthermore, he said they have an oauth_body_hash_value in most of their requests but that's not present in ours - although they do get working requests without it.
Is there a way to remove the oauth_token parameter in Scribe? Alternatively, is the oauth_body_hash_value always needed? Can a request work without?
I've included the code below, I am completely new to OAuth so please feel free to tell me if there's something else that's wrong.
Note that the TestAPI.class extends DefaultAPI10a and just returns "" for all three required methods.
public class TestImporter {
private static final String REQ_URL = "http://test.com/";
private static final String KEY = "KEY";
private static final String SECRET = "SECRET";
// test variables
private static final String VAR1 = "Test123";
public static void main(String[] args) {
OAuthService service = new ServiceBuilder()
.provider(TestAPI.class)
.apiKey(KEY)
.apiSecret(SECRET)
.build();
Token token = new Token("", "");
OAuthRequest request = new OAuthRequest(Verb.GET, REQ_URL + VAR1 + "/");
service.signRequest(token, request);
Response response = request.send();
System.out.println(response.getBody());
}
}
Regarding your own answer seems that what you want to do is put the signature in the querystring and not use the Authorization header.
This, though valid is not recommended. Anyway if you really need to do it, there's a way of creating the OAuthService to "sign" in the querystring:
ServiceBuilder()
.provider(TestAPI.class)
.apiKey(KEY)
.apiSecret(SECRET)
.signatureType(SignatureType.QueryString)
.build();
Assuming their implementation is not broken, it should not matter that you have 'extra' OAuth headers included. Having said that, the oauth_token header is not optional (I assume you are communicating using OAuth 1.0). This header should contain the access token for the User. In your example you show this token as being blank, which is quite odd!
Now assuming for some reason that it is valid to send blank 'usernames' to this third party system, you will want to make sure that your OAuth signature is matching on both sides (yours and the other companies). Use a protocol sniffer to capture the value of the oauth_signature header, then ask your third party to verify that they generate a signature which is the same. If not then you probably have a signature hashing problem.
It turns out that when we thought we were sending a fully formed HTTP GET request, we weren't.
The library was adding all of the information to the header (where we were getting our information from), but not adding any oauth information to the request Url. I can only assume it's to do with us using two-legged authorisation (hence the empty Token).
By copying the map of oAuthParameters into queryStringParameters, it then allowed the Url to be formed correctly.