Is there a way to view/dump DNS cached used by java.net api?
Here is a script to print the positive and negative DNS address cache.
import java.lang.reflect.Field;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;
public class DNSCache {
public static void main(String[] args) throws Exception {
InetAddress.getByName("stackoverflow.com");
InetAddress.getByName("www.google.com");
InetAddress.getByName("www.yahoo.com");
InetAddress.getByName("www.example.com");
try {
InetAddress.getByName("nowhere.example.com");
} catch (UnknownHostException e) {
}
String addressCache = "addressCache";
System.out.println(addressCache);
printDNSCache(addressCache);
String negativeCache = "negativeCache";
System.out.println(negativeCache);
printDNSCache(negativeCache);
}
private static void printDNSCache(String cacheName) throws Exception {
Class<InetAddress> klass = InetAddress.class;
Field acf = klass.getDeclaredField(cacheName);
acf.setAccessible(true);
Object addressCache = acf.get(null);
Class cacheKlass = addressCache.getClass();
Field cf = cacheKlass.getDeclaredField("cache");
cf.setAccessible(true);
Map<String, Object> cache = (Map<String, Object>) cf.get(addressCache);
for (Map.Entry<String, Object> hi : cache.entrySet()) {
Object cacheEntry = hi.getValue();
Class cacheEntryKlass = cacheEntry.getClass();
Field expf = cacheEntryKlass.getDeclaredField("expiration");
expf.setAccessible(true);
long expires = (Long) expf.get(cacheEntry);
Field af = cacheEntryKlass.getDeclaredField("address");
af.setAccessible(true);
InetAddress[] addresses = (InetAddress[]) af.get(cacheEntry);
List<String> ads = new ArrayList<String>(addresses.length);
for (InetAddress address : addresses) {
ads.add(address.getHostAddress());
}
System.out.println(hi.getKey() + " "+new Date(expires) +" " +ads);
}
}
}
The java.net.InetAddress uses caching of successful and unsuccessful host name resolutions.
From its javadoc:
The InetAddress class has a cache to
store successful as well as
unsuccessful host name resolutions.
By default, when a security manager is
installed, in order to protect against
DNS spoofing attacks, the result of
positive host name resolutions are
cached forever. When a security
manager is not installed, the default
behavior is to cache entries for a
finite (implementation dependent)
period of time. The result of
unsuccessful host name resolution is
cached for a very short period of time
(10 seconds) to improve performance.
If the default behavior is not
desired, then a Java security property
can be set to a different Time-to-live
(TTL) value for positive caching.
Likewise, a system admin can configure
a different negative caching TTL value
when needed.
Two Java security properties control
the TTL values used for positive and
negative host name resolution caching:
networkaddress.cache.ttl
Indicates the caching policy for
successful name lookups from the name
service. The value is specified as as
integer to indicate the number of
seconds to cache the successful
lookup. The default setting is to
cache for an implementation specific
period of time.
A value of -1 indicates "cache
forever".
networkaddress.cache.negative.ttl (default: 10)
Indicates the caching
policy for un-successful name lookups
from the name service. The value is
specified as as integer to indicate
the number of seconds to cache the
failure for un-successful lookups.
A value of 0 indicates "never cache".
A value of -1 indicates "cache
forever".
If what you have in mind is dumping the caches (of type java.net.InetAddress$Cache) used by java.net.InetAddress , they are internal implementation details and thus private:
/*
* Cached addresses - our own litle nis, not!
*/
private static Cache addressCache = new Cache(Cache.Type.Positive);
private static Cache negativeCache = new Cache(Cache.Type.Negative);
So I doubt you'll find anything doing this out of the box and guess that you'll have to play with reflection to achieve your goal.
Above answer does not work in Java 8 anymore.
Here a slight adaption:
import java.lang.reflect.Field;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;
public class DNSCache {
public static void main(String[] args) throws Exception {
InetAddress.getByName("stackoverflow.com");
InetAddress.getByName("www.google.com");
InetAddress.getByName("www.yahoo.com");
InetAddress.getByName("www.example.com");
try {
InetAddress.getByName("nowhere.example.com");
} catch (UnknownHostException e) {
}
String addressCache = "addressCache";
System.out.println(addressCache);
printDNSCache(addressCache);
String negativeCache = "negativeCache";
System.out.println(negativeCache);
printDNSCache(negativeCache);
}
private static void printDNSCache(String cacheName) throws Exception {
Class<InetAddress> klass = InetAddress.class;
Field acf = klass.getDeclaredField(cacheName);
acf.setAccessible(true);
Object addressCache = acf.get(null);
Class cacheKlass = addressCache.getClass();
Field cf = cacheKlass.getDeclaredField("cache");
cf.setAccessible(true);
Map<String, Object> cache = (Map<String, Object>) cf.get(addressCache);
for (Map.Entry<String, Object> hi : cache.entrySet()) {
Object cacheEntry = hi.getValue();
Class cacheEntryKlass = cacheEntry.getClass();
Field expf = cacheEntryKlass.getDeclaredField("expiration");
expf.setAccessible(true);
long expires = (Long) expf.get(cacheEntry);
Field af = cacheEntryKlass.getDeclaredField("addresses");
af.setAccessible(true);
InetAddress[] addresses = (InetAddress[]) af.get(cacheEntry);
List<String> ads = new ArrayList<String>(addresses.length);
for (InetAddress address : addresses) {
ads.add(address.getHostAddress());
}
System.out.println(hi.getKey() + " expires in "
+ Instant.now().until(Instant.ofEpochMilli(expires), ChronoUnit.SECONDS) + " seconds " + ads);
}
}
}
The above answer does not work with Java 11. In Java 11, both positive and negative cache entries can be retrieved using the 'cache' instance variable.
Here are new adaptations:
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.lang.reflect.Field;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
public class DnsCacheFetcher {
static long startTimeinNano = System.nanoTime();
public static void main(String[] args) throws Exception {
System.out.println("SecurityManager: " + System.getSecurityManager());
InetAddress.getByName("stackoverflow.com");
InetAddress.getByName("www.google.com");
InetAddress.getByName("www.yahoo.com");
InetAddress.getByName("www.ankit.com");
try {
InetAddress.getByName("nowhere.example.com");
} catch (UnknownHostException e) {
System.out.println("Unknown host: " + e);
}
String addressCache = "cache";
System.out.println(">>>>" + addressCache);
printDNSCache(addressCache);
/*
* String negativeCache = "negativeCache"; System.out.println(">>>>" +
* negativeCache); printDNSCache(negativeCache);
*/
}
private static void printDNSCache(String cacheName) throws Exception {
Class<InetAddress> klass = InetAddress.class;
Field[] fields = klass.getDeclaredFields();
/*
* for (Field field : fields) { System.out.println(field.getName()); }
*/
Field acf = klass.getDeclaredField(cacheName);
acf.setAccessible(true);
Object addressCache = acf.get(null);
Class cacheKlass = addressCache.getClass();
Map<String, Object> cache = (Map<String, Object>) acf.get(addressCache);
for (Map.Entry<String, Object> hi : cache.entrySet()) {
/* System.out.println("Fetching cache for: " + hi.getKey()); */
Object cacheEntry = hi.getValue();
Class cacheEntryKlass = cacheEntry.getClass();
Field expf = cacheEntryKlass.getDeclaredField("expiryTime");
expf.setAccessible(true);
long expires = (Long) expf.get(cacheEntry);
Field af = cacheEntryKlass.getDeclaredField("inetAddresses");
af.setAccessible(true);
InetAddress[] addresses = (InetAddress[]) af.get(cacheEntry);
List<String> ads = null;
if (addresses != null) {
ads = new ArrayList<String>(addresses.length);
for (InetAddress address : addresses) {
ads.add(address.getHostAddress());
}
}
/*
* System.out.println(hi.getKey() + " expires in " +
* (Instant.now().until(Instant.ofEpochMilli(expires), ChronoUnit.SECONDS)) +
* " seconds. inetAddresses: " + ads);
*/
/*
* System.nanoTime() + 1000_000_000L * cachePolicy : this how java 11 set
* expiryTime
*/
System.out.println(hi.getKey() + " expires in approx " + (expires - startTimeinNano) / 1000_000_000L
+ " seconds. inetAddresses: " + ads);
}
}}
https://github.com/alibaba/java-dns-cache-manipulator
A simple 0-dependency thread-safe Java™ lib for setting/viewing dns programmatically without touching host file, make unit/integration test portable; and a tool for setting/viewing dns of running JVM process.
This lib/tool read and set java dns cache by reflection, with concerns:
compatibility with different java version(support Java 6/8/11/17).
dns cache implementation in java.net.InetAddress is different in different java version.
thread-safety
support IPv6
Related
I am using spring data Redis 2.5.7 to consume the stream messages from Redis, this is my consumer code looks like:
package com.dolphin.soa.post.common.mq;
import com.alibaba.fastjson.JSON;
import com.dolphin.soa.post.contract.request.ArticleRequest;
import com.dolphin.soa.post.model.entity.SubRelation;
import com.dolphin.soa.post.service.IArticleService;
import com.dolphin.soa.post.service.ISubRelationService;
import lombok.extern.slf4j.Slf4j;
import misc.enumn.user.SubStatus;
import org.apache.commons.collections4.CollectionUtils;
import org.apache.commons.collections4.MapUtils;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.data.redis.connection.stream.MapRecord;
import org.springframework.data.redis.core.DefaultTypedTuple;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.ZSetOperations;
import org.springframework.data.redis.stream.StreamListener;
import org.springframework.stereotype.Component;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
/**
* #author dolphin
*/
#Component
#Slf4j
public class StreamMessageListener implements StreamListener<String, MapRecord<String, String, String>> {
#Value("${dolphin.redis.stream.group}")
private String groupName;
#Value("${dolphin.redis.user.sub.article.key}")
private String subArticleKey;
private final StringRedisTemplate stringRedisTemplate;
private final RedisTemplate<String, Object> articleRedisTemplate;
private final RedisTemplate<String, Long> redisLongTemplate;
private final ISubRelationService subRelationService;
private final IArticleService articleService;
public StreamMessageListener(StringRedisTemplate stringRedisTemplate,
#Qualifier("redisObjectTemplate") RedisTemplate<String, Object> articleRedisTemplate,
ISubRelationService subRelationService,
#Qualifier("redisLongTemplate") RedisTemplate<String, Long> redisLongTemplate,
IArticleService articleService) {
this.stringRedisTemplate = stringRedisTemplate;
this.articleRedisTemplate = articleRedisTemplate;
this.subRelationService = subRelationService;
this.redisLongTemplate = redisLongTemplate;
this.articleService = articleService;
}
#Override
public void onMessage(MapRecord<String, String, String> message) {
try {
Map<String, String> body = message.getValue();
log.debug("receive message from redis:" + JSON.toJSONString(body));
handleArticle(body);
this.stringRedisTemplate.opsForStream().acknowledge(groupName, message);
} catch (Exception e) {
log.error("handle redis stream message error", e);
}
}
private void handleArticle(Map<String, String> body) {
Long channelId = MapUtils.getLongValue(body, "sub_source_id", -1L);
Long articleId = MapUtils.getLongValue(body, "id", -1L);
if (channelId <= 0L || articleId <= 0L) {
log.error("id incorrect", body);
return;
}
ArticleRequest articleRequest = new ArticleRequest();
articleRequest.setChannelId(channelId);
articleRequest.setSubStatus(SubStatus.SUB);
// TODO: may be page should avoid memory overflow, this is dangerous when the subscribe user increase
List<SubRelation> relations = subRelationService.list(articleRequest);
if (CollectionUtils.isEmpty(relations)) {
return;
}
relations.forEach(item -> {
var articleIdsSet = new HashSet<ZSetOperations.TypedTuple<Long>>();
ZSetOperations.TypedTuple<Long> singleId = new DefaultTypedTuple<>(articleId, Double.valueOf(articleId));
articleIdsSet.add(singleId);
/**
* because now we only have less than 2GB memory
* the redis stream only pass the article id and channel id not the full article
* at this procedure has a extern stop 1 query from database
*/
articleService.getArticleFromCache(Arrays.asList(articleId));
String userSubCacheKey = subArticleKey + item.getUserId();
Boolean isKeyExists = redisLongTemplate.hasKey(userSubCacheKey);
if (isKeyExists) {
/**
* only the user is active recently
* the redis will cache the user subscribe list then we push the newest article
* to the subscribe user one by one(may be we should make the operation less)
* when the channel subscribe user increment
* this may become a performance neck bottle
*/
redisLongTemplate.opsForZSet().add(userSubCacheKey, articleIdsSet);
}
});
}
}
I am facing a problem that the Redis stream have message but the spring data Redis consumer did not consume it. when I am using this command to check the stream message:
> XINFO STREAM pydolphin:stream:article
length
45
radix-tree-keys
1
radix-tree-nodes
2
last-generated-id
1652101310592-0
groups
1
first-entry
1652083221122-0
id
2288687
sub_source_id
4817
last-entry
1652101310592-0
id
2288731
sub_source_id
4792
it shows that there have some message but the consumer did not consume it. why did this happen? what should I do to fix this problem?Sometimes it will consume one or more message but not all the message will be consumed. It always have many message in the queue. I have already tried to use this command to check the pending message:
XPENDING pydolphin:stream:article pydolphin:stream:group:article - + 20
it only return 1 pending message. but the stream queue have 40+ message.
I'm getting the following exception when trying to read a file to AWS S3 Error:
Unable to load AWS credentials from any provider in the chain.
I have generated a public bucket and also generated an AWS IAM role with full S3 bucket access and textract access.
I am trying to read an image from S3 bucket and run AWS Textract service.
Java code:
package com.textract;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectInputStream;
import software.amazon.awssdk.core.SdkBytes;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.textract.TextractClient;
import software.amazon.awssdk.services.textract.model.*;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
public class App {
public static Map<String, String> getRelationships(Map<String, Block> blockMap, Map<String, Block> keyMap,Map<String, Block> valueMap) {
Map<String, String> result = new LinkedHashMap<>();
for(Map.Entry<String, Block> itr : keyMap.entrySet()) {
Block valueBlock = findValue(itr.getValue(), valueMap);
String key = getText(itr.getValue(), blockMap);
String value = getText(valueBlock, blockMap);
result.put(key, value);
}
return result;
}
public static Block findValue(Block keyBlock, Map<String, Block> valueMap) {
Block b = null;
for(Relationship relationship : keyBlock.relationships()) {
if(relationship.type().toString().equals("VALUE")) {
for(String id : relationship.ids()) {
b = valueMap.get(id);
}
}
}
return b;
}
public static String getText(Block result, Map<String, Block> blockMap) {
StringBuilder stringBuilder = new StringBuilder();
for(Relationship relationship : result.relationships()) {
if(relationship.type().toString().equals("CHILD")) {
for(String id : relationship.ids()) {
Block b = blockMap.get(id);
if(b.blockTypeAsString().equals("WORD")) {
stringBuilder.append(b.text()).append(" ");
}
}
}
}
return stringBuilder.toString();
}
public static void main(String[] args) {
BasicAWSCredentials creds = new BasicAWSCredentials("<Secret Key>", "<Access Key>");
AmazonS3 s3client = AmazonS3Client.builder()
.withRegion("ap-southeast-1")
.withCredentials(new AWSStaticCredentialsProvider(creds))
.build();
// AmazonS3 s3client = AmazonS3ClientBuilder.standard().build();
S3Object s3Object = s3client.getObject("<S3 Bucket>", "image.jpg");
S3ObjectInputStream s3ObjectInputStream = s3Object.getObjectContent();
SdkBytes bytes = SdkBytes.fromInputStream(s3ObjectInputStream);
Document doc = Document.builder().bytes(bytes).build();
List<FeatureType> list = new ArrayList<>();
list.add(FeatureType.FORMS);
AnalyzeDocumentRequest request = AnalyzeDocumentRequest.builder().featureTypes(list).document(doc).build();
TextractClient textractClient = TextractClient.builder().region(Region.US_WEST_2).build();
AnalyzeDocumentResponse response = textractClient.analyzeDocument(request);
List<Block> blocks = response.blocks();
Map<String, Block> blockMap = new LinkedHashMap<>();
Map<String, Block> keyMap = new LinkedHashMap<>();
Map<String, Block> valueMap = new LinkedHashMap<>();
for (Block b : blocks) {
String block_id = b.id();
blockMap.put(block_id, b);
if(b.blockTypeAsString().equals("KEY_VALUE_SET")) {
for(EntityType entityType : b.entityTypes()) {
if(entityType.toString().equals("KEY")) {
keyMap.put(block_id, b);
} else {
valueMap.put(block_id, b);
}
}
}
}
System.out.println(getRelationships(blockMap, keyMap, valueMap));
textractClient.close();
}
}
Console error:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.]
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:112)
at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:104)
at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:76)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:97)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.textract.DefaultTextractClient.analyzeDocument(DefaultTextractClient.java:215)
at com.textract.App.main(App.java:83)
You are only passing the credentials to the S3 client. You aren't passing them to the Textract client. If you are creating the credentials provider manually like that, you will need to pass it to each client instance that is created.
I highly suggest using the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to provide credentials, instead of setting them directly in your code like that, as it will be much more flexible when you run this code in other environments, and will allow the AWS SDK to pick up the credentials automatically without the need for you to create a credentials provider at all.
I finally got it working with the below code with the below article:
Calling Amazon Textract Synchronous Operations
String document="input.png";
String bucket="bucket";
AmazonTextract client = AmazonTextractClientBuilder.defaultClient();
DetectDocumentTextRequest request = new DetectDocumentTextRequest()
.withDocument(new Document()
.withS3Object(new S3Object()
.withName(document)
.withBucket(bucket)));
DetectDocumentTextResult result = client.detectDocumentText(request);
Working Code
package aws.cloud.work;
import java.io.IOException;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.textract.AmazonTextract;
import com.amazonaws.services.textract.AmazonTextractClientBuilder;
import com.amazonaws.services.textract.model.DetectDocumentTextRequest;
import com.amazonaws.services.textract.model.DetectDocumentTextResult;
import com.amazonaws.services.textract.model.Document;
import com.amazonaws.services.textract.model.S3Object;
public class TextractOriginalMaster3 {
static AmazonTextractClientBuilder clientBuilder = AmazonTextractClientBuilder.standard().withRegion(Regions.AP_SOUTHEAST_1);
public static void main(String[] args) throws IOException {
//Set AWS Credentials to use Textract
clientBuilder.setCredentials(new AWSStaticCredentialsProvider(new
BasicAWSCredentials("Accesskey", "Secretkey")));
//**Getting document from S3 Bucket Path
String document = "image.png";
String bucket = "Bucket Name";
//Calling AWS Textract Client
AmazonTextract client = clientBuilder.build();
DetectDocumentTextRequest request = new DetectDocumentTextRequest()
.withDocument(new Document()
.withS3Object(new S3Object()
.withName(document)
.withBucket(bucket)));
DetectDocumentTextResult result = client.detectDocumentText(request);
System.out.println(result);
result.getBlocks().forEach(block -> {
if (block.getBlockType().equals("LINE"))
System.out.println("text is " + block.getText() + " confidence is " + block.getConfidence());
});
}
}
To answer this, we can load the credentials using StaticCredentialsProvider
Please refer the below code.
TextractClient textractClient = TextractClient.builder().region(Region.of(region)).credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials
.create(accessKey,secretKey))).build();
I am trying to create a simple application where the app will consume Kafka message do some cql transform and publish to Kafka and below is the code:
JAVA: 1.8
Flink: 1.13
Scala: 2.11
flink-siddhi: 2.11-0.2.2-SNAPSHOT
I am using library: https://github.com/haoch/flink-siddhi
input json to Kafka:
{
"awsS3":{
"ResourceType":"aws.S3",
"Details":{
"Name":"crossplane-test",
"CreationDate":"2020-08-17T11:28:05+00:00"
},
"AccessBlock":{
"PublicAccessBlockConfiguration":{
"BlockPublicAcls":true,
"IgnorePublicAcls":true,
"BlockPublicPolicy":true,
"RestrictPublicBuckets":true
}
},
"Location":{
"LocationConstraint":"us-west-2"
}
}
}
main class:
public class S3SidhiApp {
public static void main(String[] args) {
internalStreamSiddhiApp.start();
//kafkaStreamApp.start();
}
}
App class:
package flinksidhi.app;
import com.google.gson.JsonObject;
import flinksidhi.event.s3.source.S3EventSource;
import io.siddhi.core.SiddhiManager;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.core.fs.FileSystem;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer;
import org.apache.flink.streaming.siddhi.SiddhiCEP;
import org.json.JSONObject;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import static flinksidhi.app.connector.Consumers.createInputMessageConsumer;
import static flinksidhi.app.connector.Producer.*;
public class internalStreamSiddhiApp {
private static final String inputTopic = "EVENT_STREAM_INPUT";
private static final String outputTopic = "EVENT_STREAM_OUTPUT";
private static final String consumerGroup = "EVENT_STREAM1";
private static final String kafkaAddress = "localhost:9092";
private static final String zkAddress = "localhost:2181";
private static final String S3_CQL1 = "from inputStream select * insert into temp";
private static final String S3_CQL = "from inputStream select json:toObject(awsS3) as obj insert into temp;" +
"from temp select json:getString(obj,'$.awsS3.ResourceType') as affected_resource_type," +
"json:getString(obj,'$.awsS3.Details.Name') as affected_resource_name," +
"json:getString(obj,'$.awsS3.Encryption.ServerSideEncryptionConfiguration') as encryption," +
"json:getString(obj,'$.awsS3.Encryption.ServerSideEncryptionConfiguration.Rules[0].ApplyServerSideEncryptionByDefault.SSEAlgorithm') as algorithm insert into temp2; " +
"from temp2 select affected_resource_name,affected_resource_type, " +
"ifThenElse(encryption == ' ','Fail','Pass') as state," +
"ifThenElse(encryption != ' ' and algorithm == 'aws:kms','None','Critical') as severity insert into outputStream";
public static void start(){
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//DataStream<String> inputS = env.addSource(new S3EventSource());
//Flink kafka stream consumer
FlinkKafkaConsumer<String> flinkKafkaConsumer =
createInputMessageConsumer(inputTopic, kafkaAddress,zkAddress, consumerGroup);
//Add Data stream source -- flink consumer
DataStream<String> inputS = env.addSource(flinkKafkaConsumer);
SiddhiCEP cep = SiddhiCEP.getSiddhiEnvironment(env);
cep.registerExtension("json:toObject", io.siddhi.extension.execution.json.function.ToJSONObjectFunctionExtension.class);
cep.registerExtension( "json:getString", io.siddhi.extension.execution.json.function.GetStringJSONFunctionExtension.class);
cep.registerStream("inputStream", inputS, "awsS3");
inputS.print();
System.out.println(cep.getDataStreamSchemas());
//json needs extension jars to present during runtime.
DataStream<Map<String,Object>> output = cep
.from("inputStream")
.cql(S3_CQL1)
.returnAsMap("temp");
//Flink kafka stream Producer
FlinkKafkaProducer<Map<String, Object>> flinkKafkaProducer =
createMapProducer(env,outputTopic, kafkaAddress);
//Add Data stream sink -- flink producer
output.addSink(flinkKafkaProducer);
output.print();
try {
env.execute();
} catch (Exception e) {
e.printStackTrace();
}
}
}
Consumer class:
package flinksidhi.app.connector;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import org.json.JSONObject;
import java.util.Properties;
public class Consumers {
public static FlinkKafkaConsumer<String> createInputMessageConsumer(String topic, String kafkaAddress, String zookeeprAddr, String kafkaGroup ) {
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", kafkaAddress);
properties.setProperty("zookeeper.connect", zookeeprAddr);
properties.setProperty("group.id",kafkaGroup);
FlinkKafkaConsumer<String> consumer = new FlinkKafkaConsumer<String>(
topic,new SimpleStringSchema(),properties);
return consumer;
}
}
Producer class:
package flinksidhi.app.connector;
import flinksidhi.app.util.ConvertJavaMapToJson;
import org.apache.flink.api.common.serialization.SerializationSchema;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer;
import org.apache.flink.streaming.util.serialization.KeyedSerializationSchema;
import org.json.JSONObject;
import java.util.Map;
public class Producer {
public static FlinkKafkaProducer<Tuple2> createStringProducer(StreamExecutionEnvironment env, String topic, String kafkaAddress) {
return new FlinkKafkaProducer<Tuple2>(kafkaAddress, topic, new AverageSerializer());
}
public static FlinkKafkaProducer<Map<String,Object>> createMapProducer(StreamExecutionEnvironment env, String topic, String kafkaAddress) {
return new FlinkKafkaProducer<Map<String,Object>>(kafkaAddress, topic, new SerializationSchema<Map<String, Object>>() {
#Override
public void open(InitializationContext context) throws Exception {
}
#Override
public byte[] serialize(Map<String, Object> stringObjectMap) {
String json = ConvertJavaMapToJson.convert(stringObjectMap);
return json.getBytes();
}
});
}
}
I have tried many things but the code where the CQL is invoked is never called and doesn't even give any error not sure where is it going wrong.
The same thing if I do creating an internal stream source and use the same input json to return as string it works.
Initial guess: if you are using event time, are you sure you have defined watermarks correctly? As stated in the docs:
(...) an incoming element is initially put in a buffer where elements are sorted in ascending order based on their timestamp, and when a watermark arrives, all the elements in this buffer with timestamps smaller than that of the watermark are processed (...)
If this doesn't help, I would suggest to decompose/simplify the job to a bare minimum, for example just a source operator and some naive sink printing/logging elements. And if that works, start adding back operators one by one. You could also start by simplifying your CEP pattern as much as possible.
First of all thanks a lot #Piotr Nowojski , just because of your small pointer which no matter how many times I pondered over about event time , it did not came in my mind. So yes while debugging the two cases:
With internal datasource , where it was processing successfully, while debugging the flow , I identified that it was processing a watermark after it was processing the data, but it did not catch me that it was somehow managing the event time of the data implicitly.
With kafka as a datasource , while I was debugging I could very clearly see that it was not processing any watermark in the flow, but it did not occur to me that , it is happening because of the event time and watermark not handled properly.
Just adding a single line of code in the application code which I understood from below Flink code snippet:
#deprecated In Flink 1.12 the default stream time characteristic has been changed to {#link
* TimeCharacteristic#EventTime}, thus you don't need to call this method for enabling
* event-time support anymore. Explicitly using processing-time windows and timers works in
* event-time mode. If you need to disable watermarks, please use {#link
* ExecutionConfig#setAutoWatermarkInterval(long)}. If you are using {#link
* TimeCharacteristic#IngestionTime}, please manually set an appropriate {#link
* WatermarkStrategy}. If you are using generic "time window" operations (for example {#link
* org.apache.flink.streaming.api.datastream.KeyedStream#timeWindow(org.apache.flink.streaming.api.windowing.time.Time)}
* that change behaviour based on the time characteristic, please use equivalent operations
* that explicitly specify processing time or event time.
*/
I got to know that by default flink considers event time and for that watermark needs to be handled properly which I didn't so I added below link for setting the time characteristics of the flink execution environment:
env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime);
and kaboom ... it started working , while this is deprecated and needs some other configuration, but thanks a lot , it was a great pointer and helped me a lot and I solved the issue..
Thanks again #Piotr Nowojski
Hi I am trying to create a exception logs of my java application code in AWS cloudwatch for that I have used CloudWatchLogsClient to put my events to it but i am getting a below Error
DEBUG software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain - Unable to load credentials from SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
at software.amazon.awssdk.auth.credentials.internal.SystemSettingsCredentialsProvider.resolveCredentials(SystemSettingsCredentialsProvider.java:58)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:91)
at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:52)
at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:100)
at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:71)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68)
at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:68)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.cloudwatchlogs.DefaultCloudWatchLogsClient.describeLogStreams(DefaultCloudWatchLogsClient.java:1168)
at com.WorkingwithS3.WorkingwithS3.PutLogEvents.main(PutLogEvents.java:58)
Here is my code sample
package com.WorkingwithS3.WorkingwithS3;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.cloudwatchlogs.CloudWatchLogsClient;
import software.amazon.awssdk.services.cloudwatchlogs.CloudWatchLogsClientBuilder;
import software.amazon.awssdk.services.cloudwatchlogs.model.DescribeLogStreamsRequest;
import software.amazon.awssdk.services.cloudwatchlogs.model.DescribeLogStreamsResponse;
import software.amazon.awssdk.services.cloudwatchlogs.model.InputLogEvent;
import software.amazon.awssdk.services.cloudwatchlogs.model.PutLogEventsRequest;
import java.util.Arrays;
// snippet-end:[cloudwatch.java2.put_log_events.import]
/**
* Puts a sample CloudWatch log event
*/
public class PutLogEvents {
public static void main(String[] args) {
BasicAWSCredentials creds = new BasicAWSCredentials("xxxx",
"xxxxx");
// BasicAWSCredentials creds = new BasicAWSCredentials("xxxxxxxx",
// "xxxx");
String regionId = "xxx";
String logGroupName = "xxxx";
String streamName = "xxxxx";
// snippet-start:[cloudwatch.java2.put_log_events.main]
CloudWatchLogsClient logsClient = CloudWatchLogsClient.builder().region(Region.of(regionId)).build();
// A sequence token is required to put a log event in an existing stream.
// Look up the stream to find its sequence token.
// First describe all streams in the log group.
DescribeLogStreamsRequest logStreamRequest = DescribeLogStreamsRequest.builder()
.logGroupName(logGroupName)
.logStreamNamePrefix(streamName)
.build();
DescribeLogStreamsResponse describeLogStreamsResponse = logsClient.describeLogStreams(logStreamRequest);
// Assume that a single stream is returned since a specific stream name was specified in the previous request.
String sequenceToken = describeLogStreamsResponse.logStreams().get(0).uploadSequenceToken();
// Build an input log message to put to CloudWatch.
InputLogEvent inputLogEvent = InputLogEvent.builder()
.message("{ \"key1\": \"value1\", \"key2\": \"value2\" }")
.timestamp(System.currentTimeMillis())
.build();
// Specify the request parameters.
PutLogEventsRequest putLogEventsRequest = PutLogEventsRequest.builder()
.logEvents(Arrays.asList(inputLogEvent))
.logGroupName(logGroupName)
.logStreamName(streamName)
// Sequence token is required so that the log can be written to the
// latest location in the stream.
.sequenceToken(sequenceToken)
.build();
logsClient.putLogEvents(putLogEventsRequest);
// snippet-end:[cloudwatch.java2.put_log_events.main]
System.out.println("Successfully put CloudWatch log event");
}
}
Could anyone please guide how to specify the credentials for CloudWatchLogsClient?
Thanks in advance
From the trace we can see the sdk client in this instance CloudWatchLogsClient.builder() is failing to find the credentials and hence failing to build.
The client will look for the credentials in the following defaults locations
For a lot of reasons its good to set up your code to read the credentials from environment variables.
This follows for many reasons.
AWS encourage the use of environment variables for credentials.
The increasing need to run your application in some kind of a container cluster like Kubernetes for example.
Often in a containerized environment access to the file system can be problematic.
In many container tools like docker-compose its trivial to pass environment variables to the container.
In the link defaults locations it specifies the options precisely how to supply the credentials for the CloudWatchLogsClient.builder() operation and for the reasons above suggest you adopt the environment variables solution and you can test that they are set correctly by using `
Map<String, String> mapOfEnvironmentVariables = System.getenv();
to retrieve them.
Below code working fine i am able to write the exception in cloudwatch using CloudWatchLogsClient just for reference i have attached code
package com.example.DynamoDB;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.bind.annotation.ExceptionHandler;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.cloudwatchlogs.CloudWatchLogsClient;
import software.amazon.awssdk.services.cloudwatchlogs.model.DescribeLogStreamsRequest;
import software.amazon.awssdk.services.cloudwatchlogs.model.DescribeLogStreamsResponse;
import software.amazon.awssdk.services.cloudwatchlogs.model.InputLogEvent;
import software.amazon.awssdk.services.cloudwatchlogs.model.PutLogEventsRequest;
import java.util.Arrays;
#ControllerAdvice
public class ExceptionControllerAdvice {
#ExceptionHandler(Exception.class)
public ResponseEntity<ErrorResponse> exceptionHandler(Exception ex) {
ErrorResponse error = new ErrorResponse();
error.setErrorCode(HttpStatus.INTERNAL_SERVER_ERROR.value());
error.setMessage(ex.getMessage());
error.setController(ex.getStackTrace()[0].getClassName());
error.setService(ex.getStackTrace()[0].getClassName());
error.setTimestamp(System.currentTimeMillis());
PutLogEvents(error);
return new ResponseEntity<ErrorResponse>(error, HttpStatus.OK);
}
public static void PutLogEvents(ErrorResponse Er)
{
String regionId = "us-east-1";
String logGroupName = "xxxxxxxxx";
String logStreamName = "xxxxxxxxx";
CloudWatchLogsClient logsClient = CloudWatchLogsClient.builder().region(Region.of(regionId)).build();
// A sequence token is required to put a log event in an existing stream.
// Look up the stream to find its sequence token.
String sequenceToken = getNextSequenceToken(logsClient, logGroupName, logStreamName);
// Build a JSON log using the EmbeddedMetricFormat.
String message = "[{" +
" \"Timestamp\": " + Er.getTimestamp() + "," +
" \"ErrorCode\": " + Er.getErrorCode() + "," +
" \"ControllerName\": " + Er.getErrorCode() + "," +
" \"ServiceName\": " + Er.getErrorCode() + "," +
" \"ErrorMsg\": " + Er.getErrorCode() + "" +
"}]";
InputLogEvent inputLogEvent = InputLogEvent.builder()
.message(message)
.timestamp(Er.getTimestamp())
.build();
// Specify the request parameters.
PutLogEventsRequest putLogEventsRequest = PutLogEventsRequest.builder()
.logEvents(Arrays.asList(inputLogEvent))
.logGroupName(logGroupName)
.logStreamName(logStreamName)
// Sequence token is required so that the log can be written to the
// latest location in the stream.
.sequenceToken(sequenceToken)
.build();
logsClient.putLogEvents(putLogEventsRequest);
}
private static String getNextSequenceToken(CloudWatchLogsClient logsClient, String logGroupName, String logStreamName) {
DescribeLogStreamsRequest logStreamRequest = DescribeLogStreamsRequest.builder()
.logGroupName(logGroupName)
.logStreamNamePrefix(logStreamName)
.build();
DescribeLogStreamsResponse describeLogStreamsResponse = logsClient.describeLogStreams(logStreamRequest);
// Assume that a single stream is returned since a specific stream name was
// specified in the previous request.
return describeLogStreamsResponse.logStreams().get(0).uploadSequenceToken();
}
}
import java.io.IOException;
import java.lang.ref.SoftReference;
import java.net.URI;
import java.security.cert.CRLException;
import java.security.cert.CertificateException;
import java.security.cert.X509CRL;
import java.security.cert.X509Certificate;
import java.util.Collections;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import javax.naming.NamingException;
import org.joda.time.DateTime;
import org.kp.oppr.esb.logger.Logger;
import org.springframework.beans.factory.annotation.Autowired;
public class CachedCrlRepository {
private static final Logger LOGGER = new Logger("CachedCrlRepository");
private final Map<URI, SoftReference<X509CRL>> crlCache = Collections
.synchronizedMap(new HashMap<URI, SoftReference<X509CRL>>());;
private static int DEFAULT_CACHE_AGING_HOURS;
#Autowired
private DgtlSgntrValidator validator;
#Autowired
private CrlRepository crlRepository;
public X509CRL findCrl(URI crlUri, X509Certificate issuerCertificate,
Date validationDate) throws DigitalValdiationException,
CertificateException, CRLException, IOException, NamingException {
SoftReference<X509CRL> crlRef = this.crlCache.get(crlUri);
if (null == crlRef) {
LOGGER.info("Key CRL URI : " + crlUri + " not found in the cache " );
return refreshCrl(crlUri, issuerCertificate, validationDate);
}
X509CRL crl = crlRef.get();
if (null == crl) {
LOGGER.info("CRL Entry garbage collected: " + crlUri);
return refreshCrl(crlUri, issuerCertificate, validationDate);
}
if (validationDate.after(crl.getNextUpdate())) {
LOGGER.info("CRL URI no longer valid: " + crlUri);
LOGGER.info("CRL validation date: " + validationDate + " is after CRL next update date: " + crl.getNextUpdate());
return refreshCrl(crlUri, issuerCertificate, validationDate);
}
Date thisUpdate = crl.getThisUpdate();
LOGGER.info("This update " + thisUpdate);
/*
* The PKI the nextUpdate CRL extension indicates 7 days. The
* actual CRL refresh rate is every 3 hours. So it's a bit dangerous to
* only base the CRL cache refresh strategy on the nextUpdate field as
* indicated by the CRL.
*/
DateTime cacheMaturityDateTime = new DateTime(thisUpdate)
.plusHours(DEFAULT_CACHE_AGING_HOURS);
LOGGER.info("Cache maturity Date Time " + cacheMaturityDateTime);
if (validationDate.after(cacheMaturityDateTime.toDate())) {
LOGGER.info("Validation date: " + validationDate + " is after cache maturity date: " + cacheMaturityDateTime.toDate());
return refreshCrl(crlUri, issuerCertificate, validationDate);
}
LOGGER.info("using cached CRL: " + crlUri);
return crl;
}
public static int getDEFAULT_CACHE_AGING_HOURS() {
return DEFAULT_CACHE_AGING_HOURS;
}
public static void setDEFAULT_CACHE_AGING_HOURS(int dEFAULT_CACHE_AGING_HOURS) {
DEFAULT_CACHE_AGING_HOURS = dEFAULT_CACHE_AGING_HOURS;
}
private X509CRL refreshCrl(URI crlUri, X509Certificate issuerCertificate,
Date validationDate) throws DigitalValdiationException,
CertificateException, CRLException, IOException, NamingException {
X509CRL crl = crlRepository.downloadCRL(crlUri.toString());
this.crlCache.put(crlUri, new SoftReference<X509CRL>(crl));
return crl;
}
}
I have this class CachedCrlrepository that stores CRL list from particular provider. I want to know if my implementation is thread safe or I am missing something over here. The cache is for a web service, so it is multi-threaded.
My doubt in for this particular method
private X509CRL refreshCrl(URI crlUri, X509Certificate issuerCertificate,
Date validationDate) throws DigitalValdiationException,
CertificateException, CRLException, IOException, NamingException {
X509CRL crl = crlRepository.downloadCRL(crlUri.toString());
this.crlCache.put(crlUri, new SoftReference<X509CRL>(crl));
return crl;
}
I think this particular line need to be synchronized
this.crlCache.put(crlUri, new SoftReference<X509CRL>(crl));
synchronized(this)
{
this.crlCache.put(crlUri, new SoftReference<X509CRL>(crl));
}
Another issue which I see is that after a GC is run the cache still have that entry in the memory. It never execute these line of code
if (null == crl) {
LOGGER.info("CRL Entry garbage collected: " + crlUri);
return refreshCrl(crlUri, issuerCertificate, validationDate);
}
Generally you should not use a synchronized Map in cases where you are expecting heavy traffic and high concurrent access on your object which in this case is crlCache. For each and every read or write threads will wait behind another and in heavy load, your thread count will go high and eventually your server will crash. You can look into ConcurrentHashMap. which is designed to work efficiently in such scenarios.
Your second point:
synchronized(this)
{
this.crlCache.put(crlUri, new SoftReference<X509CRL>(crl));
}
is not at all required with current code as put method is already synchronized.
For minimal changes replace
private final Map<URI, SoftReference<X509CRL>> crlCache = Collections
.synchronizedMap(new HashMap<URI, SoftReference<X509CRL>>());;
with
private final ConcurrentHashMap<URI, SoftReference<X509CRL>> crlCache = new ConcurrentHashMap<URI, SoftReference<X509CRL>>();
Finally, using SoftReference is good but there are better options. Guava from google is a very robust and efficient cache builder.