Develop Reference

Java Custom Google Analytics 4 Server-Side Event User-IP

In my current Java project, it's easy to track server-side user events in the "old" Google Analytics Universal Project with simple REST calls to Google Analytics. So that location tracking was working, i could override the server ip with the user ip, according to the parameter "&uip=1.2.3.4" (https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters?hl=de#uip).
As upgrading to GA4 is recommended, I was able to change all the REST parameters in my project and show my events in the new dashboard, except for the user location. I can't find any information about such a parameter. I tried using still "uip" but now all my requests are located to the country of my server.
Unfortunately it's not possible to track the event client side, because my project is a simple REST API, returning only JSON data.
Does anyone have an idea, if there's such a parameter like "uip" for ga4 or if this isn't possible anymore?
In the following way I setup my parameters:
private String getQueryParameters(MeasurementEvent event) {
StringBuilder body = new StringBuilder();
body.append("?v=").append(version);
body.append("&tid=").append(trackingId);
body.append("&cid=").append(event.getClientId());
body.append("&en=").append(eventName);
body.append("&aip=1");
if (StringUtils.hasText(event.getAction())) {
body.append("&ep.useraction=").append(event.getAction());
}
if (StringUtils.hasText(event.getCategory())) {
body.append("&ep.awsregion=").append(event.getCategory());
}
if (StringUtils.hasText(event.getLabel())) {
body.append("&ep.softwarename=").append(event.getLabel());
}
if (StringUtils.hasText(event.getRemoteAddress())) {
body.append("&uip=").append(event.getRemoteAddress());
}
if (StringUtils.hasText(event.getUrl())) {
body.append("&dl=").append(event.getUrl());
}
return body.toString();
}

Apache Camel AWS S3: credential expiration and temporary credentials

Apache Camel interfaces well with AWS S3, but I have found a scenario in which it is not built correctly for. Going over all of the Camel examples I have seen online, I have never seen anyone use the recommended, industry standard, AWS temporary credentials on non-local environments. Using static credentials that live for ~6 months is a security issue as well as a manual burden (to refresh) and realistically shouldn't be used anywhere except for local environments.
Given a custom, s3 client setup, Camel can take temporary credentials, however, a Camel route pointed to AWS S3 will experience an expiration at some point. Camel is not smart enough to know this and will continue to try to poll a S3 bucket without throwing any exceptions or timeout errors indefinitely.
I have tried to add a timeout configuration to my endpoint like so:
aws-s3://" + incomingAWSBucket + "?" + "amazonS3Client=#amazonS3Client&timeout=4000
Can anyone explain how to interface Camel with AWS temporary credentials or throw an exception if AWS credentials expire (given the aforementioned setup)?
Thanks for the help!
UPDATE:
I pushed a feature to Apache Camel to handle the issue above:
https://github.com/apache/camel/blob/master/components/camel-aws-s3/src/main/docs/aws-s3-component.adoc#use-useiamcredentials-with-the-s3-component

The answer to this question is dense enough for a tutorial if others want it. For now, I will copy and paste it to the correct forums and threads to get the word out:
Without complaining too much, I'd just like to say that for how powerful Camel is, its documentation and example base is really lacking for production scenarios in the AWS world... Sigh... Thats a mouthful and probably a stretch for any open source lib.
I figured out how to solve the credential problem by referencing the official camel-s3 documentation to first see how to create an advanced S3 configuration (relying on the aws sdk itself -- you can see a bare bones example there -- it makes the s3 client manually).
After I figured this out, I went out to the aws sdk documentation on IAM credentials to figure out how this could work on an EC2 instance since I am able to build the client itself. In the aforementioned docs, there are a few bare bones examples as well. Upon testing testing with the examples listed, I found that the credential refresh (the sole purpose of this question) was not working. It could get credentials at first, but it was not refreshing them during my tests after they were manually expired.
Lastly, I figured out that you can specify a provider chain that can handle the refreshing of the credentials on its own. The aws documentation that explains this is here.
In the end, I still need to have static credentials for my local camel setups that poll aws s3 buckets, however, my remote environments that live on ec2s can access them with temporary credentials that refresh themselves flawlessly. WOWSA! :)
To do this, I simply made a factory that uses a local camel setup for my local development and remote camel setup that relies on the temporary IAM credentials. This saves me the security concern and the work on needing to manually refresh credentials for all remote environments!
I will not explain how to create a factory or how my local & remote configurations are set up entirely, but I will include my code sample of the AmazonS3ClientBuilder that creates an S3 Client for remote setups.
AmazonS3ClientBuilder.standard()
.withCredentials(new InstanceProfileCredentialsProvider(false))
.withRegion(Regions.US_WEST_2)
.build();
If there is a desire on how I got this to work, I can provide an example project that shows the entire process.
By request, here are my local and remote implementations of the s3 client:
Local:
public class LocalAWSS3ClientManagerImpl implements AWSS3ClientManager {
private static Logger logger = LoggerFactory.getLogger(LocalAWSS3ClientManagerImpl.class);
private PriorityCodeSourcesRoutesProperties priorityCodeSourcesRoutesProperties;
private SimpleRegistry registry = new SimpleRegistry();
private CamelContext camelContext;
public LocalAWSS3ClientManagerImpl(PriorityCodeSourcesRoutesProperties priorityCodeSourcesRoutesProperties) {
this.priorityCodeSourcesRoutesProperties = priorityCodeSourcesRoutesProperties;
registry.put("amazonS3Client", getS3Client());
camelContext = new DefaultCamelContext(registry);
logger.info("Creating an AWS S3 manager for a local instance (you should not see this on AWS EC2s).");
}
private AmazonS3 getS3Client() {
try {
String awsBucketAccessKey = priorityCodeSourcesRoutesProperties.getAwsBucketAccessKey();
String awsBucketSecretKey = priorityCodeSourcesRoutesProperties.getAwsBucketSecretKey();
AWSCredentials awsCredentials = new BasicAWSCredentials(awsBucketAccessKey, awsBucketSecretKey);
return AmazonS3ClientBuilder.standard().withCredentials(
new AWSStaticCredentialsProvider(awsCredentials)).build();
} catch (RuntimeException ex) {
logger.error("Could not create AWS S3 client with the given credentials from the local config.");
}
return null;
}
public Endpoint getIncomingAWSEndpoint(final String incomingAWSBucket, final String region,
final String fileNameToSaveAndDownload) {
return camelContext.getEndpoint(
"aws-s3://" + incomingAWSBucket + "?" + "amazonS3Client=#amazonS3Client"
+ "&region=" + region + "&deleteAfterRead=false" + "&prefix=" + fileNameToSaveAndDownload);
}
public Endpoint getOutgoingLocalEndpoint(final String outgoingEndpointDirectory,
final String fileNameToSaveAndDownload) {
return camelContext.getEndpoint(
"file://" + outgoingEndpointDirectory + "?" + "fileName="
+ fileNameToSaveAndDownload + "&readLock=markerFile");
}
}
Remote:
public class RemoteAWSS3ClientManagerImpl implements AWSS3ClientManager {
private static Logger logger = LoggerFactory.getLogger(RemoteAWSS3ClientManagerImpl.class);
private PriorityCodeSourcesRoutesProperties priorityCodeSourcesRoutesProperties;
private SimpleRegistry registry = new SimpleRegistry();
private CamelContext camelContext;
public RemoteAWSS3ClientManagerImpl(PriorityCodeSourcesRoutesProperties priorityCodeSourcesRoutesProperties) {
this.priorityCodeSourcesRoutesProperties = priorityCodeSourcesRoutesProperties;
registry.put("amazonS3Client", getS3Client());
camelContext = new DefaultCamelContext(registry);
logger.info("Creating an AWS S3 client for a remote instance (normal for ec2s).");
}
private AmazonS3 getS3Client() {
try {
logger.info("Attempting to create an AWS S3 client with IAM role's temporary credentials.");
return AmazonS3ClientBuilder.standard()
.withCredentials(new InstanceProfileCredentialsProvider(false))
.withRegion(Regions.US_WEST_2)
.build();
} catch (RuntimeException ex) {
logger.error("Could not create AWS S3 client with the given credentials from the instance. "
+ "The default credential chain was used to create the AWS S3 client. "
+ ex.toString());
}
return null;
}
public Endpoint getIncomingAWSEndpoint(final String incomingAWSBucket, final String region,
final String fileNameToSaveAndDownload) {
return camelContext.getEndpoint(
"aws-s3://" + incomingAWSBucket + "?" + "amazonS3Client=#amazonS3Client"
+ "&region=" + region + "&deleteAfterRead=false" + "&prefix=" + fileNameToSaveAndDownload);
}
public Endpoint getOutgoingLocalEndpoint(final String outgoingEndpointDirectory,
final String fileNameToSaveAndDownload) {
return camelContext.getEndpoint(
"file://" + outgoingEndpointDirectory + "?" + "fileName="
+ fileNameToSaveAndDownload + "&readLock=markerFile");
}
}

AmazonClientException: Unable To Load Credentials from any Provider in the Chain

My mule application writes json record to a kinesis stream. I use KPL producer library. When run locally, it picks AWS credentials from .aws/credentials and writes record to kinesis successfully.
However, when I deploy my application to Cloudhub, it throws AmazonClientException, obviously due to not having access to any of directories that DefaultAWSCredentialsProviderChain class supports. (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html)
This is how I attach credentials and it looks locally in .aws/credentials:
config.setCredentialsProvider( new
DefaultAWSCredentialsProviderChain());
I couldn't figure out a way to provide credentials explicitly using my-app.properies file.
Then I tried to create a separate configuration file with getters/setters. set access key and private key as private and then impement a getter:
public AWSCredentialsProvider getCredentials() {
if(accessKey == null || secretKey == null) {
return new DefaultAWSCredentialsProviderChain();
}
return new StaticCredentialsProvider(new BasicAWSCredentials(getAccessKey(), getSecretKey()));
}
}
This was intended to be used instead of DefaultAWSCredentialsProviderChain class this way---
config.setCredentialsProvider(new AWSConfig().getCredentials());
Still throws the same error when deployed.
The following repo states that it is possible to provide explicit credentials. I need help to figure out how because I can't find a proper documentation / example.
https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/src/com/amazonaws/services/kinesis/producer/sample/SampleProducer.java

I have Faced the same issue so, I got this solution I hope this will work for you also.
#Value("${s3_accessKey}")
private String s3_accessKey;
#Value("${s3_secretKey}")
private String s3_secretKey;
//this above value I am taking from Application.properties file
BasicAWSCredentials creds = new BasicAWSCredentials(s3_accessKey,
s3_secretKey);
AmazonS3 s3Client = AmazonS3ClientBuilder.standard().
withCredentials(new AWSStaticCredentialsProvider(creds))
.withRegion(Regions.US_EAST_2)
.build();

Vertx HttpClient getNow not working

I have problem with vertx HttpClient.
Here's code which shows that tests GET using vertx and plain java.
Vertx vertx = Vertx.vertx();
HttpClientOptions options = new HttpClientOptions()
.setTrustAll(true)
.setSsl(false)
.setDefaultPort(80)
.setProtocolVersion(HttpVersion.HTTP_1_1)
.setLogActivity(true);
HttpClient client = vertx.createHttpClient(options);
client.getNow("google.com", "/", response -> {
System.out.println("Received response with status code " + response.statusCode());
});
System.out.println(getHTML("http://google.com"));
Where getHTML() is from here: How do I do a HTTP GET in Java?
This is my output:
<!doctype html><html... etc <- correct output from plain java
Feb 08, 2017 11:31:21 AM io.vertx.core.http.impl.HttpClientRequestImpl
SEVERE: java.net.UnknownHostException: failed to resolve 'google.com'. Exceeded max queries per resolve 3
But vertx can't connect. What's wrong here? I'm not using any proxy.

For reference: a solution, as described in this question and in tsegismont's comment here, is to set the flag vertx.disableDnsResolver to true:
-Dvertx.disableDnsResolver=true
in order to fall back to the JVM DNS resolver as explained here:
sometimes it can be desirable to use the JVM built-in resolver, the JVM system property -Dvertx.disableDnsResolver=true activates this behavior
I observed this DNS resolution issue with a redis client in a kubernetes environment.

I had this issue, what caused it for me was stale DNS servers being picked up by the Java runtime, i.e. servers registered for a network the machine was no longer connected to. The issue is first in the Sun JNDI implementation, it also exists in Netty which uses JNDI to bootstrap its list of name servers on most platforms, then finally shows up in VertX.
I think a good place to fix this would be in the Netty layer where the set of default DNS servers is bootstrapped. I have raised a ticket with the Netty project so we'll see if they agree with me! Here is the Netty ticket
In the mean time a fairly basic workaround is to filter the default DNS servers detected by Netty, based on whether they are reachable or not. Here is a code Sample in Kotlin to apply before constructing the main VertX instance.
// The default set of name servers provided by JNDI can contain stale entries
// This default set is picked up by Netty and in turn by VertX
// To work around this, we filter for only reachable name servers on startup
val nameServers = DefaultDnsServerAddressStreamProvider.defaultAddressList()
val reachableNameServers = nameServers.stream()
.filter {ns -> ns.address.isReachable(NS_REACHABLE_TIMEOUT)}
.map {ns -> ns.address.hostAddress}
.collect(Collectors.toList())
if (reachableNameServers.size == 0)
throw StartupException("There are no reachable name servers available")
val opts = VertxOptions()
opts.addressResolverOptions.servers = reachableNameServers
// The primary Vertx instance
val vertx = Vertx.vertx(opts)
A little more detail in case it is helpful. I have a company machine, which at some point was connected to the company network by a physical cable. Details of the company's internal name servers were set up by DHCP on the physical interface. Using the wireless interface at home, DNS for the wireless interface gets set to my home DNS while the config for the physical interface is not updated. This is fine since that device is not active, ipconfig /all does not show the internal company DNS servers. However, looking in the registry they are still there:
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces
They get picked up by the JNDI mechanism, which feeds Netty and in turn VertX. Since they are not reachable from my home location, DNS resolution fails. I can imagine this home/office situation is not unique to me! I don't know whether something similar could occur with multiple virtual interfaces on containers or VMs, it could be worth looking at if you are having problems.

Here is the sample code which works for me.
public class TemplVerticle extends HttpVerticle {
public static void main(String[] args) {
Vertx vertx = Vertx.vertx();
// Create the web client and enable SSL/TLS with a trust store
WebClient client = WebClient.create(vertx,
new WebClientOptions()
.setSsl(true)
.setTrustAll(true)
.setDefaultPort(443)
.setKeepAlive(true)
.setDefaultHost("www.w3schools.com")
);
client.get("www.w3schools.com")
.as(BodyCodec.string())
.send(ar -> {
if (ar.succeeded()) {
HttpResponse<String> response = ar.result();
System.out.println("Got HTTP response body");
System.out.println(response.body().toString());
} else {
ar.cause().printStackTrace();
}
});
}
}

Try using web client instead of httpclient, here you have an example (with rx):
private val client: WebClient = WebClient.create(vertx, WebClientOptions()
.setSsl(true)
.setTrustAll(true)
.setDefaultPort(443)
.setKeepAlive(true)
)
open fun <T> get(uri: String, marshaller: Class<T>): Single<T> {
return client.getAbs(host + uri).rxSend()
.map { extractJson(it, uri, marshaller) }
}
Another option is to use getAbs.

Batching operations in Google Apps Admin Java API

I have written a Java app that synchronises Google Groups on our Google Apps for Education domain (similar in function to Google Apps School Directory Sync, but customised for some of our specific needs).
The synchronisation works, but it is slow because it is performing each task individually. I know that there are API interfaces for batching operations, but I can't find any examples of how this is implemented with the Java API.
The code I'm using looks similar to this (authentication and other setup is taken care of elsewhere):
try
{
Member m = new Member ();
m.setEmail (member);
m.setRole ("MEMBER");
service.members ().insert (group, m).execute ();
}
catch (Exception e)
{
// ERROR handling
}
Instead of executing these operations one-by-one, I would like to batch them instead. Can anyone tell me how?

Look here: Batch Java API
For example:
BatchRequest batch = new BatchRequest(httpTransport, httpRequestInitializer);
batch.setBatchUrl(new GenericUrl(/*your customized batch URL goes here*/));
batch.queue(httpRequest1, dataClass, errorClass, callback);
batch.queue(httpRequest2, dataClass, errorClass, callback);
batch.execute();
Remember, that:
The body of each part is itself a complete HTTP request, with its own
verb, URL, headers, and body. The HTTP request must only contain the
path portion of the URL; full URLs are not allowed in batch requests.
UPDATE
Look also here, how to build batch with Google Batch API:
https://github.com/google/google-api-java-client
UPDATE 2
Try something like this:
// Create the Storage service object
Storage storage = new Storage(httpTransport, jsonFactory, credential);
// Create a new batch request
BatchRequest batch = storage.batch();
// Add some requests to the batch request
storage.objectAccessControls().insert("bucket-name", "object-key1",
new ObjectAccessControl().setEntity("user-123423423").setRole("READER"))
.queue(batch, callback);
storage.objectAccessControls().insert("bucket-name", "object-key2",
new ObjectAccessControl().setEntity("user-guy#example.com").setRole("READER"))
.queue(batch, callback);
storage.objectAccessControls().insert("bucket-name", "object-key3",
new ObjectAccessControl().setEntity("group-foo#googlegroups.com").setRole("OWNER"))
.queue(batch, callback);
// Execute the batch request. The individual callbacks will be called when requests finish.
batch.execute();
From here: Batch request with Google Storage Json Api (JAVA)