kafka.api.OffsetRequest - unable to retrieve results - java

Am working through the task of determining ConsumerLag and need to retrieve the current Producer Offset as follows:
PartitionOffsetRequestInfo partitionOffsetRequestInfo =
new PartitionOffsetRequestInfo(OffsetRequest.LatestTime(), 100);
List<TopicAndPartition> partitions = new ArrayList<>();
for(int i = 0; i < partitionMetadataList.size(); i++) {
TopicAndPartition topicAndPartition = new TopicAndPartition(topic, i);
partitions.add(topicAndPartition);
tuple2List.add(new Tuple2<>(topicAndPartition, partitionOffsetRequestInfo));
}
Tuple2<TopicAndPartition, PartitionOffsetRequestInfo>[] tuple2Array =
tuple2List.parallelStream().toArray(Tuple2[]::new);
WrappedArray<Tuple2<TopicAndPartition, PartitionOffsetRequestInfo>> wrappedArray =
Predef.wrapRefArray(tuple2Array);
scala.collection.immutable.Map<TopicAndPartition, PartitionOffsetRequestInfo> offsetRequestInfoMap =
(scala.collection.immutable.Map<TopicAndPartition, PartitionOffsetRequestInfo>)
scala.Predef$.MODULE$.Map().apply(wrappedArray);
OffsetRequest offsetRequest = new OffsetRequest(offsetRequestInfoMap, (short)0,
0, OffsetRequest.DefaultClientId(), Request.OrdinaryConsumerId());
Looking at the OffsetResponse am presented with an array of UnknownTopicOrPartitionException offsets. If I pass in (short)1 for the versionId (as I have for calls to OffsetFetchResponse) I get a NetworkReceive.readFromReadableChannel exception when I try to retrieve the results.
Questions:
a. Is there a better way to get current producer offset?
b. Why doesn't the OffsetRequest call work with VersionId = 1?
Edit:
Note that Im able to use this channel to retrieve ConsumerOffset so I know it works.
I can retrieve the values using the cmdline:
kafka-consumer-groups --bootstrap-server hostname:9092 --describe --new-consumer --group test_consumer
Edit:
Attempted to reuse sample scala (rewritten as Java) code:
KafkaConsumer<String, String> kafkaConsumer = getConsumer();
List<org.apache.kafka.common.TopicPartition>topicAndPartitions = new ArrayList<>();
org.apache.kafka.common.TopicPartition topicAndPartition = new org.apache.kafka.common.TopicPartition("my_topic", 0);
topicAndPartitions.add(topicAndPartition);
kafkaConsumer.assign(topicAndPartitions);
kafkaConsumer.seekToEnd(topicAndPartitions);
long lPos = kafkaConsumer.position(topicAndPartition);
Got the same exception (NetworkReceive.readFromReadableChannel) when it got to the .position() call.

Based on the source code, current version for OffsetRequest is 0 not 1. Besides the source code does not return Version-1 response containing timestamp information as doc says. So it might be a doc bug.

If you've come to this point here's a working solution:
private void getOffsets(String topic, String group) {
KafkaConsumer<String, String> kafkaConsumer = getConsumer(topic, group);
List<PartitionInfo> partitionInfos = kafkaConsumer.partitionsFor(topic);
List<org.apache.kafka.common.TopicPartition>topicAndPartitions = new ArrayList<>();
for(int i = 0; i < partitionInfos.size(); i++) {
org.apache.kafka.common.TopicPartition topicAndPartition = new org.apache.kafka.common.TopicPartition(topic, i);
topicAndPartitions.add(topicAndPartition);
}
List<Long>startList = new ArrayList<>();
List<Long>endList = new ArrayList<>();
kafkaConsumer.assign(topicAndPartitions);
for(int i = 0; i < partitionInfos.size(); i++) {
OffsetAndMetadata offsetAndMetadata = kafkaConsumer.committed(topicAndPartitions.get(i));
if(offsetAndMetadata != null) {
startList.add(offsetAndMetadata.offset());
}
}
// did we find any active partitions?
if(startList.size() == 0) {
LOGGER.info("topic:group not found: {}:{}", topic, group);
return;
}
kafkaConsumer.seekToEnd(topicAndPartitions);
for(int i = 0; i < partitionInfos.size(); i++) {
endList.add(i, kafkaConsumer.position(topicAndPartitions.get(i)));
}
LOGGER.debug("startlist.size: {} endlist.size: {} partitions: {}", startList.size(), endList.size(), partitionInfos.size());
long sumLag = 0;
for(int i = 0; i < partitionInfos.size(); i++) {
long lStart = startList.get(i);
long lEnd = endList.get(i);
sumLag += (lEnd - lStart);
/*
* At this point Im sending the info to data dog.
* The 'sum' value is nice to have.
*/
LOGGER.debug("partition: {} start: {} end: {} lag: {}", i, lStart, lEnd, (lEnd - lStart));
}
kafkaConsumer.poll(100);
topicAndPartitions.clear();
kafkaConsumer.assign(topicAndPartitions)
}

Related

Is there a way to dynamically create a counter if such "month" exists?

I would like to enquire or get some reference as to how can I dynamically create a counter for each month if the exists ? Currently, I am retrieving the dates from a CSV file and store it in an ArrayList, from there I am comparing the dates to check whether if such month exists. If the month exists then "counter++". Afterwards, store the counter in a hashmap. I understand my code currently is an inefficient way of coding. How could I make it better ?
CODE
public HashMap<String, Integer> getDataPoint() {
//My function code
HashMap<String, Integer> numberOfPost = new HashMap<String, Integer>();
int janCounter = 0;
int febCounter = 0;
int marCounter = 0;
int aprCounter = 0;
int mayCounter = 0;
int juneCounter = 0;
int julyCounter = 0;
int augCounter = 0;
int septCounter = 0;
int octCounter = 0;
int novCounter = 0;
int decCounter = 0;
String pattern = "MMM";
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
OpenCsvReader reader = new OpenCsvReader();
ArrayList <STPost> STArray = reader.loadST("file_path");
Iterator STitr = STArray.iterator();
while (STitr.hasNext()) {
STPost St = (STPost) STitr.next();
Date retrievedate = St.getTime();
String strDate = sdf.format(retrievedate);
if(strDate.equals("Jan")) {
janCounter++;
}
else if (strDate.equals("Feb")) {
febCounter++;
}
else if (strDate.equals("Mar")) {
marCounter++;
}
else if (strDate.equals("Apr")) {
aprCounter++;
}
else if (strDate.equals("May")) {
mayCounter++;
}
else if (strDate.equals("June")) {
juneCounter++;
}
else if (strDate.equals("July")) {
julyCounter++;
}
else if (strDate.equals("Aug")) {
augCounter++;
}
else if (strDate.equals("Sept")) {
septCounter++;
}
else if (strDate.equals("Oct")) {
octCounter++;
}
else if (strDate.equals("Nov")) {
novCounter++;
}
else if (strDate.equals("Dec")) {
decCounter++;
}
numberOfPost.put("January", janCounter);
numberOfPost.put("Feburary", febCounter);
numberOfPost.put("March", marCounter);
numberOfPost.put("April", aprCounter);
numberOfPost.put("May", mayCounter);
numberOfPost.put("June", juneCounter);
numberOfPost.put("July", julyCounter);
numberOfPost.put("August", augCounter);
numberOfPost.put("September", septCounter);
numberOfPost.put("October", octCounter);
numberOfPost.put("November", novCounter);
numberOfPost.put("December", decCounter);
}
return numberOfPost
}
You can create an array of months and check if value exists there using indexOf method.
String months = "JanFebMarAprMayJunJulAugSepOctNovDec";
Integer idx = months.indexOf(strDate);
Thereafter you can use SimpleDateFormat("MMMM") pattern to put and get it into your map.
if(idx > -1) {
String longDate = new SimpleDateFormat("MMMM").format(retrievedate);
Integer current = numberOfPost.get(longDate);
if (current == null) {
current = 1;
} else {
current += 1;
}
numberOfPost.put(longDate, current);
}
Thereafter, you can use map iterator to display content of map.
You already have a good start. Using a the hash map will make the code much tidier.
You can replace all those if statements and put statements with the code below:
if (!numberOfPosts.containsKey(strDate)) {
numberOfPosts.put(strDate, 0);
}
numberOfPosts.put(strDate, numberOfPosts.get(strDate) + 1);
The if statement will create a dictionary entry if there is not one with the key of strDate. The value of the entry is set to 0.
numberOfPosts.put(strDate, numberOfPosts.get(strDate) + 1)
The line above increments by 1 the dictionary entry with the key of strDate.

Linked List Optimization

I'm working on this program that emulates restriction enzymes and DNA splicing. I'm using DnaSequenceNode[s] as linked list nodes.
I have a problem with one of the function in my code, cutSplice() is supposed to create a new DnaStrand that is a clone of the current DnaStrand, but with every instance of enzyme replaced by splicee.
For example, if the LinkedDnaStrand is instantiated with "TTGATCC", and
cutSplice("GAT", "TTAAGG") is called, then the linked list should become something like (previous pointers not shown):
first -> "TT" -> "TTAAGG" -> "CC" -> null
My function works. However, my method cutSplice() takes more than 80 seconds to splice 200 DNAs. I'm supposed to bring that 80 seconds to 2 seconds.
This is all my code for the class : LinkedDnaStrand.java
And here's the code for the method cutSplice()
public DnaStrand cutSplice(String enzyme, String splicee) {
DnaStrand newStrand = null;
String original_Dna = this.toString();
String new_Dna = original_Dna.replaceAll(enzyme, splicee);
String[] splicee_split = new_Dna.split(splicee); // splits the new DNA string DnaStrand
newStrand = null;
int i = 0;
if (original_Dna.startsWith(enzyme)) {
newStrand = new LinkedDnaStrand(splicee);
} else {
newStrand = new LinkedDnaStrand(splicee_split[0]);
newStrand.append(splicee);
}
for (i = 1; i < splicee_split.length - 1; i++) {
String node = splicee_split[i];
newStrand.append(node);
newStrand.append(splicee);
}
newStrand.append(splicee_split[splicee_split.length - 1]);
if (original_Dna.endsWith(enzyme)) {
newStrand.append(splicee);
}
return newStrand;
}
Does anybody see anything that could make a critical difference on the time this function takes to process 200 DNAs sample?
Well, it is comfortable to use the string methods, but you are losing time in converting to the string, back to sequence, and (as pointed out in the previous comments) with the regex based string functions.
It will certainly consume less time to operate on the linked list directly, although this will require you to implement the replacement algorithm yourself:
#Override
public LinkedDnaStrand cutSplice(String enzyme, String splicee)
{
LinkedDnaStrand strand = new LinkedDnaStrand();
DnaSequenceNode end = null;
DnaSequenceNode begin = top;
int pos = 0;
DnaSequenceNode tmpStart, tmpEnd;
for (DnaSequenceNode current = top; current != null; current = current.next)
{
if(current.value != enzyme.charAt(pos))
{
tmpStart = tmpEnd = new DnaSequenceNode(begin.value);
for (DnaSequenceNode n = begin.next; n != current.next; n = n.next)
{
DnaSequenceNode c = new DnaSequenceNode(n.value);
tmpEnd.next = c;
c.previous = tmpEnd;
tmpEnd = c;
}
}
else if(++pos == enzyme.length())
{
tmpStart = tmpEnd = new DnaSequenceNode(splicee.charAt(0));
for (int i = 1; i < splicee.length(); ++i)
{
DnaSequenceNode c = new DnaSequenceNode(splicee.charAt(i));
tmpEnd.next = c;
c.previous = tmpEnd;
tmpEnd = c;
}
}
else
{
continue;
}
if(end == null)
{
strand.top = end = tmpStart;
}
else
{
end.next = tmpStart;
tmpStart.previous = end;
}
end = tmpEnd;
begin = current.next;
pos = 0;
}
return strand;
}
I do not claim that there is not any opportunity to further optimize, but this should be a lot faster as the original version. I tested it successfully with the example you gave, if you yet find a bug, feel free to fix it yourself...
Note 1: I did explicitely create a new sequence from the string (instead of using the constructor) to get the end of the sequence without having to iterate over it again.
Note 2: I assumed existing a constructor DnaSequenceNode(char value) and DnaSequenceNode having a member public char value. You might have to adjust the code appropriately if any of these assumptions fails.

i get ConcurrentModificationException when adding an item [duplicate]

This question already has answers here:
ConcurrentModificationException despite using synchronized
(3 answers)
Closed 8 years ago.
im trying to save the manager id of distinct managers from collabs to managersId but i get an exeption "ConcurrentModificationException"
public void fillTree() throws SystemException, PortalException {
TreeNode nodeParent;
TreeNode nodeFils;
Set<Long> managersId = new HashSet<Long>();
UserVO user = new UserVO();
collabs = CollabLocalServiceUtil.getCollabs(-1, -1);
Iterator<Long> iter = managersId.iterator();
long id;
for (int i = 0; i < collabs.size(); i++) {
id = collabs.get(i).getManagerId();
synchronized (managersId) {
managersId.add((Long) id);
System.out.println(id);
}
}
while (iter.hasNext()) {
id = iter.next();//throw exeption
user = getUserById(id);
nodeParent = new DefaultTreeNode(user.getFullName(), root);
for (int j = 0; j < collabs.size(); j++) {
if (collabs.get(j).getManagerId() == user.getUserId()) {
nodeFils = new DefaultTreeNode(getUserById(
collabs.get(j).getUserId()).getFullName(),
nodeParent);
}
}
}
}
im using liferay portal
Edit based on question update.
In this new code the issue is with your iterator. You initialized the iterator and then modified the collection and then trying to use the dirty iterator. That is the cause of concurrent modification exception.
So the fix for this issue is simple. Just move Iterator<Long> iter = managersId.iterator(); after for loop. Try
public void fillTree() throws SystemException, PortalException {
TreeNode nodeParent;
TreeNode nodeFils;
Set<Long> managersId = new HashSet<Long>();
UserVO user = new UserVO();
collabs = CollabLocalServiceUtil.getCollabs(-1, -1);
long id;
for (int i = 0; i < collabs.size(); i++) {
id = collabs.get(i).getManagerId();
synchronized (managersId) {
managersId.add((Long) id);
System.out.println(id);
}
}
Iterator<Long> iter = managersId.iterator(); // Getting the new iterator with latest value.
while (iter.hasNext()) {
id = iter.next();//Now this wont throw exeption
user = getUserById(id);
nodeParent = new DefaultTreeNode(user.getFullName(), root);
for (int j = 0; j < collabs.size(); j++) {
if (collabs.get(j).getManagerId() == user.getUserId()) {
nodeFils = new DefaultTreeNode(getUserById(
collabs.get(j).getUserId()).getFullName(),
nodeParent);
}
}
}
}
OLD ANSWER
Firstly from your logic I think you are trying to get unique manager id as list. In this case you could use a Set.
As for your current problem if its executing in a multi thread environment you could use synchronized block like
List<Long> managersId = new ArrayList<Long>();
collabs = CollabLocalServiceUtil.getCollabs(-1, -1);
long id;
for (int i = 0; i < collabs.size(); i++) {
id = collabs.get(i).getManagerId();
synchronized (managersId) {
if (!managersId.contains(id)) {
managersId.add((Long) id);
}
}
}
Or else you could use java.util.concurrent.CopyOnWriteArrayList for concurrent list operation.
List<Long> managersId = new CopyOnWriteArrayList<Long>();
Also as a third option you can make a normal list synchronized by collections class
Collections.synchronizedList(managersId);

How to get array data with CATIA using com4j API to get point coordinate?

I try to retreive information from catia usig com4j. Some methods require to pass an array in argument to retreive information but the array is never populated. In this example is to get coordinate from a point in catia.
The declaration of the method generated by com4j
public interface Point extends com4j.catia.HybridShape {
...
void getCoordinates(java.lang.Object[] oCoordinates);
...
}
my code to get the coordinate
public static void testGetPointCoordinates() {
String catiafile="E:\\test.CATPart";
Application app =null;
app = COM4J.createInstance(Application.class, "CATIA.Application");
Documents documents = app.documents();
Document oDocument = (Document) documents.open(new Holder<>(catiaFile.getAbsolutePath()));
PartDocument partDocument = oDocument.queryInterface(PartDocument.class);
Part part = partDocument.part();
Factory HSFactory = part.hybridShapeFactory();
HybridShapeFactory HSF = HSFactory.queryInterface(HybridShapeFactory.class);
HybridBodies hbodies = part.hybridBodies();
int n = hbodies.count();
for (int i = 1; i <= n; i++) {
HybridBody hbody = null;
hbody = hbodies.item(i);
int nbitems = hbody.hybridShapes().count();
for (int j = 1; j <= nbitems; j++) {
String name = hbody.hybridShapes().item(j).name();
System.out.println("name=" + name);
//Object tab[]=new Object[3];
if (name.startsWith("Point.12")) {//true point
HybridShape hs = hbody.hybridShapes().item(j);
Reference reference = part.createReferenceFromObject(hs);
HybridShapePointCoord p3 = hs.queryInterface(HybridShapePointCoord.class);
//works
System.out.println("point name = " + p3.name());
System.out.println(p3.y().value());//display 50.0
//doesn't work
Variant tab[] = new Variant[3];
for (int k = 0; k < 3; k++) {
Variant variant = new Variant();
variant.set(k);
tab[k] = variant;
}
p3.getCoordinates(tab);
System.out.println(tab[1].getParseableString()); //display 1 (value not modified)
//doesn't work
tab = new Variant[3];
for (int k = 0; k < 3; k++) {
Variant variant = new Variant(Variant.Type.VT_EMPTY);//tested with VT_R4 VT_R8,...
tab[k] = variant;
}
System.out.println(tab[1].getJavaCode()); //display null (Don't know how to print VT_EMPTY as an Java literal)
//doesn't work with this other solution but ok in VBA
tab = new Variant[3];
//Variant v = new Variant(Type.VT_EMPTY);
tab[0] = new Variant(Variant.Type.VT_EMPTY);
tab[1] = new Variant(Variant.Type.VT_EMPTY);
tab[2] = new Variant(Variant.Type.VT_EMPTY);
HybridShapePointExplicit point = HSF.addNewPointDatum(reference);
point.getCoordinates(tab);
System.out.println(tab[1].doubleValue() + " " + tab[2].toString()); //display 0
//doesn't work
//crash JVM
// tab = new Variant[3];
// p3.getCoordinates(tab);
break;
}
}
}
}
the code below works in VBA with CATIA
Dim P1
Dim coordonnees(2)
Dim coordonnees2(100, 3)
coordonnees(0) = 0
coordonnees(1) = 0
coordonnees(2) = 0
Set P1 = HSF.AddNewPointDatum(hbody.HybridShapes.Item(i))
'fonction to get coordinates
P1.GetCoordinates coordonnees
'set name and coordinates
coordonnees2(Y, 0) = hbody.HybridShapes.Item(i).name
coordonnees2(Y, 1) = Round(coordonnees(0), 3)
coordonnees2(Y, 2) = Round(coordonnees(1), 3)
coordonnees2(Y, 3) = Round(coordonnees(2), 3)
I'm not familiar with com4j, but Point.GetCoordinates is a restricted method:
.
The solution to this problem in VB is you need to create a new variant and set it equal to p3. Then, call GetCoordinates from the variant's context. Intellisense will not work well on the variant, but the call to GetCoordinates will work.
In VB the code would look something like this:
...
Dim p3 As Variant
Dim arrayOfCoord(2) As Variant
Set p3 = hybridShapePointCoord1 'set variant = strongly typed point object
p3.GetCoordinates arrayOfCoord
...
In your case, non-VB, your code might look like this
...
Object p3Obj; //used in vb:Variant p3Var;
p3Obj = p3;
p3Obj.GetCoordinates(tab);//Intellisense will not work, but the call to this method should
...
Let me know if this works.

Jfreechart: how to get the individual items in a BoxAndWhiskerChart

I am generating a box and whisker chart with one item per category.I also want to generate a report with the mean, median and all the values per item in the BoxPlot. So, after i create the dataset, defaultboxandwhiskercategorydataset based on the categoryType, I call the method convertReportData to fetch each item in the defaultboxandwhiskercategorydataset and save the mean, median etc into another data object later for report generation. But it just prints only one category. Could anyone please help me to figure out what is wrong?
My boxplot
Code:
public static BoxAndWhiskerCategoryDataset createDataset() {
startTime = inputData.getItimeFrom();
endTime = inputData.getItimeTo();
List<String> categorylist = new ArrayList<>();
categorylist.add("Distance 0-20");
categorylist.add("Distance 20-40");
categorylist.add("Distance 40-60");
categorylist.add("Distance 60-80");
categorylist.add("Distance 80-100");
categorylist.add("Distance >100");
Map<String, List<Double>> map = new HashMap<String, List<Double>>();
map = addDistance(values_list);
DefaultBoxAndWhiskerCategoryDataset defaultboxandwhiskercategorydataset = new DefaultBoxAndWhiskerCategoryDataset();
for (String categoryType : categorylist) {
map.remove(null);
for (Map.Entry<String, List<Double>> entry : map.entrySet()) {
if (entry.getKey().equalsIgnoreCase(categoryType)) {
defaultboxandwhiskercategorydataset.add(entry.getValue(),
categoryType, " ");
}
}
}
convertReportData(defaultboxandwhiskercategorydataset, categorylist);
return defaultboxandwhiskercategorydataset;
}
private static void convertReportData(DefaultBoxAndWhiskerCategoryDataset boxandwhiskercategorydataset, List<String> latencyTypelist) {
report = new HashMap<>();
for (int i = 0; i < boxandwhiskercategorydataset.getColumnKeys().size(); i++) {
BoxAndWhiskerItem item = boxandwhiskercategorydataset.getItem(i, 0);
ReportData data = new ReportData();
data.setMean(item.getMean());
data.setMedian(item.getMedian());
data.setQ1(item.getQ1());
data.setQ3(item.getQ3());
data.setMaxOutlier(item.getMaxOutlier());
data.setMaxRegularNumber(item.getMaxRegularValue());
data.setMinOutlier(item.getMinOutlier());
data.setMinRegularNumber(item.getMinRegularValue());
data.setOutliers(item.getOutliers());
report.put(boxandwhiskercategorydataset.getRowKey(i).toString(),
data);
}
}
The problem is with
for (int i = 0; i < boxandwhiskercategorydataset.getColumnKeys().size(); i++) {
you are using getColumnKeys whereas you have only one Column. It should have been,
for (int i = 0; i < boxandwhiskercategorydataset.getRowKeys().size(); i++) {

Categories

Resources