Develop Reference

Can produce to Kafka but cannot consume

I'm using the Kafka JDK client ver 0.10.2.1 . I am able to produce simple messages to Kafka for a "heartbeat" test, but I cannot consume a message from that same topic using the sdk. I am able to consume that message when I go into the Kafka CLI, so I have confirmed the message is there. Here's the function I'm using to consume from my Kafka server, with the props - I pass the message I produced to the topic only after I have indeed confirmed the produce() was succesful, I can post that function later if requested:
private def consumeFromKafka(topic: String, expectedMessage: String): Boolean = {
val props: Properties = initProps("consumer")
val consumer = new KafkaConsumer[String, String](props)
consumer.subscribe(List(topic).asJava)
var readExpectedRecord = false
try {
val records = {
val firstPollRecs = consumer.poll(MAX_POLLTIME_MS)
// increase timeout and try again if nothing comes back the first time in case system is busy
if (firstPollRecs.count() == 0) firstPollRecs else {
logger.info("KafkaHeartBeat: First poll had 0 records- trying again - doubling timeout to "
+ (MAX_POLLTIME_MS * 2)/1000 + " sec.")
consumer.poll(MAX_POLLTIME_MS * 2)
}
}
records.forEach(rec => {
if (rec.value() == expectedMessage) readExpectedRecord = true
})
} catch {
case e: Throwable => //log error
} finally {
consumer.close()
}
readExpectedRecord
}
private def initProps(propsType: String): Properties = {
val prop = new Properties()
prop.put("bootstrap.servers", kafkaServer + ":" + kafkaPort)
propsType match {
case "producer" => {
prop.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
prop.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")
prop.put("acks", "1")
prop.put("producer.type", "sync")
prop.put("retries", "3")
prop.put("linger.ms", "5")
}
case "consumer" => {
prop.put("group.id", groupId)
prop.put("enable.auto.commit", "false")
prop.put("auto.commit.interval.ms", "1000")
prop.put("session.timeout.ms", "30000")
prop.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
prop.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
prop.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
// poll just once, should only be one record for the heartbeat
prop.put("max.poll.records", "1")
}
}
prop
}
Now when I run the code, here's what it outputs in the console:
13:04:21 - Discovered coordinator serverName:9092 (id: 2147483647
rack: null) for group 0b8947e1-eb68-4af3-ac7b-be3f7c02e76e. 13:04:23
INFO o.a.k.c.c.i.ConsumerCoordinator - Revoking previously assigned
partitions [] for group 0b8947e1-eb68-4af3-ac7b-be3f7c02e76e 13:04:24
INFO o.a.k.c.c.i.AbstractCoordinator - (Re-)joining group
0b8947e1-eb68-4af3-ac7b-be3f7c02e76e 13:04:25 INFO
o.a.k.c.c.i.AbstractCoordinator - Successfully joined group
0b8947e1-eb68-4af3-ac7b-be3f7c02e76e with generation 1 13:04:26 INFO
o.a.k.c.c.i.ConsumerCoordinator - Setting newly assigned partitions
[HeartBeat_Topic.Service_5.2018-08-03.13_04_10.377-0] for group
0b8947e1-eb68-4af3-ac7b-be3f7c02e76e 13:04:27 INFO
c.p.p.l.util.KafkaHeartBeatUtil - KafkaHeartBeat: First poll had 0
records- trying again - doubling timeout to 60 sec.
And then nothing else, no errors thrown -so no records are polled. Does anyone have any idea what's preventing the 'consume' from happening? The subscriber seems to be successful, as I'm able to successfully call the listTopics and list partions no problem.

Your code has a bug. It seems your line:
if (firstPollRecs.count() == 0)
Should say this instead
if (firstPollRecs.count() > 0)
Otherwise, you're passing in an empty firstPollRecs, and then iterating over that, which obviously returns nothing.

Kafka Topic creation from Java API [duplicate]

This question already has answers here:
How to create a Topic in Kafka through Java
(6 answers)
Closed 1 year ago.
I am trying to create a Kafka topic using Java API, but getting LEADER is NOT AVAILABLE.
Code:
int partition = 0;
ZkClient zkClient = null;
try {
String zookeeperHosts = "localhost:2181"; // If multiple zookeeper then -> String zookeeperHosts = "192.168.20.1:2181,192.168.20.2:2181";
int sessionTimeOutInMs = 15 * 1000; // 15 secs
int connectionTimeOutInMs = 10 * 1000; // 10 secs
zkClient = new ZkClient(zookeeperHosts, sessionTimeOutInMs, connectionTimeOutInMs, ZKStringSerializer$.MODULE$);
String topicName = "mdmTopic5";
int noOfPartitions = 2;
int noOfReplication = 1;
Properties topicConfiguration = new Properties();
AdminUtils.createTopic(zkClient, topicName, noOfPartitions, noOfReplication, topicConfiguration);
} catch (Exception ex) {
ex.printStackTrace();
} finally {
if (zkClient != null) {
zkClient.close();
}
}
Error:
[2017-10-19 12:14:42,263] WARN Error while fetching metadata with correlation id 1 : {mdmTopic5=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-10-19 12:14:42,370] WARN Error while fetching metadata with correlation id 3 : {mdmTopic5=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-10-19 12:14:42,479] WARN Error while fetching metadata with correlation id 4 : {mdmTopic5=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
Does Kafka 0.11.0.1 supports AdminUtils.??? Please let me know how to create topic in this version.
Thanks in Advance.

Since Kafka 0.11 there is a proper Admin API for creating (and deleting) topics and I'd recommend to use it instead of directly connecting to Zookeeper.
See AdminClient.createTopics(): http://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/admin/AdminClient.html#createTopics(java.util.Collection)

Generally LEADER NOT AVAILABLE points to network issues rather than issues with your code.
Try:
telnet host port to see if you can connect to all required hosts/ports from your machine.
However, the latest approach is to use the BOOTSTRAP_SERVERS while creating topics.
A working version of topic creation code using scala would be as follows:
Import the required kafka-clients using sbt.
// https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients
libraryDependencies += Seq("org.apache.kafka" % "kafka-clients" % "2.1.1")
The code for topic creation in scala:
import java.util.Arrays
import java.util.Properties
import org.apache.kafka.clients.admin.NewTopic
import org.apache.kafka.clients.admin.{AdminClient, AdminClientConfig}
class CreateKafkaTopic {
def create(): Unit = {
val config = new Properties()
config.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "192.30.1.5:9092")
val localKafkaAdmin = AdminClient.create(config)
val partitions = 3
val replication = 1.toShort
val topic = new NewTopic("integration-02", partitions, replication)
val topics = Arrays.asList(topic)
val topicStatus = localKafkaAdmin.createTopics(topics).values()
//topicStatus.values()
println(topicStatus.keySet())
}
}
Hope it helps.

Access FreePastry program that is behind NAT

I'm trying to connect to my program that uses FreePastry behind a NAT but getting no where. mIP is my public IP, mBootport and mBindport is 50001. I have forworded this ports in my router to my computer stil it does not work. I disabled the firewall yet nothing. I disconnected the router and connect directly to the internet and stil it does not work. The only time it does work in on my local network. So something most be wrong in either the code of the config file but i can not see what is wrong.
Environment env = new Environment();
InetSocketAddress bootaddress = new InetSocketAddress(mIP, mBootport);
NodeIdFactory nidFactory = new RandomNodeIdFactory(env);
PastryNodeFactory factory = new SocketPastryNodeFactory(nidFactory, mBindport, env);
for (int curNode = 0; curNode < mNumNodes; curNode++) {
PastryNode node = factory.newNode();
NetworkHandler app = new NetworkHandler(node, mLog);
apps.add(app);
node.boot(bootaddress);
synchronized(node) {
while(!node.isReady() && !node.joinFailed()) {
node.wait(500);
if (node.joinFailed()) {
throw new IOException("Could not join the FreePastry ring. Reason:"+node.joinFailedReason());
}
}
}
System.out.println("Finished creating new node: " + node);
mLog.append("Finished creating new node: " + node + "\n");
}
Iterator<NetworkHandler> i = apps.iterator();
NetworkHandler app = (NetworkHandler) i.next();
app.subscribe();
public class NetworkHandler implements ScribeClient, Application {
int seqNum = 0;
CancellableTask publishTask;
Scribe myScribe;
Topic myTopic;
JTextArea mLog;
protected Endpoint endpoint;
public NetworkHandler(Node node, JTextArea log) {
this.endpoint = node.buildEndpoint(this, "myinstance");
mLog = log;
myScribe = new ScribeImpl(node,"myScribeInstance");
myTopic = new Topic(new PastryIdFactory(node.getEnvironment()), "example topic");
System.out.println("myTopic = "+myTopic);
mLog.append("myTopic = "+myTopic + "\n");
endpoint.register();
}
public void subscribe() {
myScribe.subscribe(myTopic, this);
}
}
freepastry.params
# this file holds the default values for pastry and it's applications
# you do not need to modify the default.params file to override these values
# instead you can use your own params file to set values to override the
# defaults. You can specify this file by constructing your
# rice.environment.Environment() with the filename you wish to use
# typically, you will want to be able to pass this file name from the command
# line
# max number of handles stored per routing table entry
pastry_rtMax = 1
pastry_rtBaseBitLength = 4
# leafset size
pastry_lSetSize = 24
# maintenance frequencies
pastry_leafSetMaintFreq = 60
pastry_routeSetMaintFreq = 900
# drop the message if pastry is not ready
pastry_messageDispatch_bufferIfNotReady = false
# number of messages to buffer while an app hasn't yet been registered
pastry_messageDispatch_bufferSize = 32
# FP 2.1 uses the new transport layer
transport_wire_datagram_receive_buffer_size = 131072
transport_wire_datagram_send_buffer_size = 65536
transport_epoch_max_num_addresses = 2
transport_sr_max_num_hops = 5
# proximity neighbor selection
transport_use_pns = true
# number of rows in the routing table to consider during PNS
# valid values are ALL, or a number
pns_num_rows_to_use = 10
# commonapi testing parameters
# direct or socket
commonapi_testing_exit_on_failure = true
commonapi_testing_protocol = direct
commonapi_testing_startPort = 5009
commonapi_testing_num_nodes = 10
# set this to specify the bootstrap node
#commonapi_testing_bootstrap = localhost:5009
# random number generator's seed, "CLOCK" uses the current clock time
random_seed = CLOCK
# sphere, euclidean or gt-itm
direct_simulator_topology = sphere
# -1 starts the simulation with the current time
direct_simulator_start_time = -1
#pastry_direct_use_own_random = true
#pastry_periodic_leafset_protocol_use_own_random = true
pastry_direct_gtitm_matrix_file=GNPINPUT
# the number of stubs in your network
pastry_direct_gtitm_max_overlay_size=1000
# the number of virtual nodes at each stub: this allows you to simulate multiple "LANs" and allows cheeper scaling
pastry_direct_gtitm_nodes_per_stub=1
# the factor to multiply your file by to reach millis. Set this to 0.001 if your file is in microseconds. Set this to 1000 if your file is in seconds.
pastry_direct_gtitm_delay_factor=1.0
#millis of the maximum network delay for the generated network topologies
pastry_direct_max_diameter=200
pastry_direct_min_delay=2
#setting this to false will use the old protocols which are about 200 times as fast, but may cause routing inconsistency in a real network. Probably won't in a simulator because it will never be incorrect about liveness
pastry_direct_guarantee_consistency=true
# rice.pastry.socket parameters
# tells the factory you intend to use multiple nodes
# this causes the logger to prepend all entries with the nodeid
pastry_factory_multipleNodes = true
pastry_factory_selectorPerNode = false
pastry_factory_processorPerNode = false
# number of bootstap nodehandles to fetch in parallel
pastry_factory_bootsInParallel = 1
# the maximum size of a message
pastry_socket_reader_selector_deserialization_max_size = 1000000
# the maximum number of outgoing messages to queue when a socket is slower than the number of messages you are queuing
pastry_socket_writer_max_queue_length = 30
pastry_socket_writer_max_msg_size = 20480
pastry_socket_repeater_buffer_size = 65536
pastry_socket_pingmanager_smallPings=true
pastry_socket_pingmanager_datagram_receive_buffer_size = 131072
pastry_socket_pingmanager_datagram_send_buffer_size = 65536
# the time before it will retry a route that was already found dead
pastry_socket_srm_check_dead_throttle = 300000
pastry_socket_srm_proximity_timeout = 3600000
pastry_socket_srm_ping_throttle = 30000
pastry_socket_srm_default_rto = 3000
pastry_socket_srm_rto_ubound = 10000
pastry_socket_srm_rto_lbound = 50
pastry_socket_srm_gain_h = 0.25
pastry_socket_srm_gain_g = 0.125
pastry_socket_scm_max_open_sockets = 300
pastry_socket_scm_max_open_source_routes = 30
# the maximum number of source routes to attempt, setting this to 0 will
# effectively eliminate source route attempts
# setting higher than the leafset does no good, it will be bounded by the leafset
# a larger number tries more source routes, which could give you a more accurate
# determination, however, is more likely to lead to congestion collapse
pastry_socket_srm_num_source_route_attempts = 8
pastry_socket_scm_socket_buffer_size = 32768
# this parameter is multiplied by the exponential backoff when doing a liveness check so the first will be 800, then 1600, then 3200 etc...
pastry_socket_scm_ping_delay = 800
# adds some fuzziness to the pings to help prevent congestion collapse, so this will make the ping be advanced or delayed by this factor
pastry_socket_scm_ping_jitter = 0.1
# how many pings until we call the node faulty
pastry_socket_scm_num_ping_tries = 5
pastry_socket_scm_write_wait_time = 30000
pastry_socket_scm_backoff_initial = 250
pastry_socket_scm_backoff_limit = 5
pastry_socket_pingmanager_testSourceRouting = false
pastry_socket_increment_port_after_construction = true
# if you want to allow connection to 127.0.0.1, set this to true
pastry_socket_allow_loopback = false
# these params will be used if the computer attempts to bind to the loopback address, they will open a socket to this address/port to identify which network adapter to bind to
pastry_socket_known_network_address = yahoo.com
pastry_socket_known_network_address_port = 80
pastry_socket_use_own_random = true
pastry_socket_random_seed = clock
# force the node to be a seed node
rice_socket_seed = false
# the parameter simulates some nodes being firewalled, base on rendezvous_test_num_firewalled
rendezvous_test_firewall = false
# probabilistic fraction of firewalled nodes
rendezvous_test_num_firewalled = 0.3
# don't firewall the first node, useful for testing
rendezvous_test_makes_bootstrap = false
# FP 2.1 uses the new transport layer
transport_wire_datagram_receive_buffer_size = 131072
transport_wire_datagram_send_buffer_size = 65536
# NAT/UPnP settings
nat_network_prefixes = 127.0.0.1;10.;192.168.
# Enable and set this if you have already set up port forwarding and know the external address
#external_address = 123.45.67.89:1234
#enable this if you set up port forwarding (on the same port), but you don't
#know the external address and you don't have UPnP enabled
#this is useful for a firwall w/o UPnP support, and your IP address isn't static
probe_for_external_address = true
# values how to probe
pastry_proxy_connectivity_timeout = 15000
pastry_proxy_connectivity_tries = 3
# possible values: always, never, prefix (prefix is if the localAddress matches any of the nat_network_prefixes
# whether to search for a nat using UPnP (default: prefix)
nat_search_policy = prefix
# whether to verify connectivity (default: boot)
firewall_test_policy = never
# policy for setting port forwarding the state of the firewall if there is already a conflicting rule: overwrite, fail (throw exception), change (use different port)
# you may want to set this to overwrite or fail on the bootstrap nodes, but most freepastry applications can run on any available port, so the default is change
nat_state_policy = change
# the name of the application in the firewall, set this if you want your application to have a more specific name
nat_app_name = freepastry
# how long to wait for responses from the firewall, in millis
nat_discovery_timeout = 5000
# how many searches to try to find a free firewall port
nat_find_port_max_tries = 10
# uncomment this to use UPnP NAT port forwarding, you need to include in the classpath: commons-jxpath-1.1.jar:commons-logging.jar:sbbi-upnplib-xxx.jar
nat_handler_class = rice.pastry.socket.nat.sbbi.SBBINatHandler
# hairpinning:
# default "prefix" requires more bandwidth if you are behind a NAT. It enables multiple IP
# addresses in the NodeHandle if you are behind a NAT. These are usually the internet routable address,
# and the LAN address (usually 192.168.x.x)
# you can set this to never if any of the following conditions hold:
# a) you are the only FreePastry node behind this address
# b) you firewall supports hairpinning see
# http://scm.sipfoundry.org/rep/ietf-drafts/behave/draft-ietf-behave-nat-udp-03.html#rfc.section.6
nat_nodehandle_multiaddress = prefix
# if we are not scheduled for time on cpu in this time, we setReady(false)
# otherwise there could be message inconsistency, because
# neighbors may believe us to be dead. Note that it is critical
# to consider the amount of time it takes the transport layer to find a
# node faulty before setting this parameter, this parameter should be
# less than the minimum time required to find a node faulty
pastry_protocol_consistentJoin_max_time_to_be_scheduled = 15000
# in case messages are dropped or something, how often it will retry to
# send the consistent join message, to get verification from the entire
# leafset
pastry_protocol_consistentJoin_retry_interval = 30000
# parameter to control how long dead nodes are retained in the "failed set" in
# CJP (see ConsistentJoinProtocol ctor) (15 minutes)
pastry_protocol_consistentJoin_failedRetentionTime = 900000
# how often to cleanup the failed set (5 mins) (see ConsistentJoinProtocol ctor)
pastry_protocol_consistentJoin_cleanup_interval = 300000
# the maximum number of entries to send in the failed set, only sends the most
recent detected failures (see ConsistentJoinProtocol ctor)
pastry_protocol_consistentJoin_maxFailedToSend = 20
# how often we send/expect to be sent updates
pastry_protocol_periodicLeafSet_ping_neighbor_period = 20000
pastry_protocol_periodicLeafSet_lease_period = 30000
# what the grace period is to receive a periodic update, before checking
# liveness
pastry_protocol_periodicLeafSet_request_lease_throttle = 10000
# how many entries are kept in the partition handler's table
partition_handler_max_history_size=20
# how long entries in the partition handler's table are kept
# 90 minutes
partition_handler_max_history_age=5400000
# what fraction of the time a bootstrap host is checked
partition_handler_bootstrap_check_rate=0.05
# how often to run the partition handler
# 5 minutes
partition_handler_check_interval=300000
# the version number of the RouteMessage to transmit (it can receive anything that it knows how to)
# this is useful if you need to migrate an older ring
# you can change this value in realtime, so, you can start at 0 and issue a command to update it to 1
pastry_protocol_router_routeMsgVersion = 1
# should usually be equal to the pastry_rtBaseBitLength
p2p_splitStream_stripeBaseBitLength = 4
p2p_splitStream_policy_default_maximum_children = 24
p2p_splitStream_stripe_max_failed_subscription = 5
p2p_splitStream_stripe_max_failed_subscription_retry_delay = 1000
#multiring
p2p_multiring_base = 2
#past
p2p_past_messageTimeout = 30000
p2p_past_successfulInsertThreshold = 0.5
#replication
# fetch delay is the delay between fetching successive keys
p2p_replication_manager_fetch_delay = 500
# the timeout delay is how long we take before we time out fetching a key
p2p_replication_manager_timeout_delay = 20000
# this is the number of keys to delete when we detect a change in the replica set
p2p_replication_manager_num_delete_at_once = 100
# this is how often replication will wake up and do maintainence; 10 mins
p2p_replication_maintenance_interval = 600000
# the maximum number of keys replication will try to exchange in a maintainence message
p2p_replication_max_keys_in_message = 1000
#scribe
p2p_scribe_maintenance_interval = 180000
#time for a subscribe fail to be thrown (in millis)
p2p_scribe_message_timeout = 15000
#util
p2p_util_encryptedOutputStream_buffer = 32678
#aggregation
p2p_aggregation_logStatistics = true
p2p_aggregation_flushDelayAfterJoin = 30000
#5 MINS
p2p_aggregation_flushStressInterval = 300000
#5 MINS
p2p_aggregation_flushInterval = 300000
#1024*1024
p2p_aggregation_maxAggregateSize = 1048576
p2p_aggregation_maxObjectsInAggregate = 25
p2p_aggregation_maxAggregatesPerRun = 2
p2p_aggregation_addMissingAfterRefresh = true
p2p_aggregation_maxReaggregationPerRefresh = 100
p2p_aggregation_nominalReferenceCount = 2
p2p_aggregation_maxPointersPerAggregate = 100
#14 DAYS
p2p_aggregation_pointerArrayLifetime = 1209600000
#1 DAY
p2p_aggregation_aggregateGracePeriod = 86400000
#15 MINS
p2p_aggregation_aggrRefreshInterval = 900000
p2p_aggregation_aggrRefreshDelayAfterJoin = 70000
#3 DAYS
p2p_aggregation_expirationRenewThreshold = 259200000
p2p_aggregation_monitorEnabled = false
#15 MINS
p2p_aggregation_monitorRefreshInterval = 900000
#5 MINS
p2p_aggregation_consolidationDelayAfterJoin = 300000
#15 MINS
p2p_aggregation_consolidationInterval = 900000
#14 DAYS
p2p_aggregation_consolidationThreshold = 1209600000
p2p_aggregation_consolidationMinObjectsInAggregate = 20
p2p_aggregation_consolidationMinComponentsAlive = 0.8
p2p_aggregation_reconstructionMaxConcurrentLookups = 10
p2p_aggregation_aggregateLogEnabled = true
#1 HOUR
p2p_aggregation_statsGranularity = 3600000
#3 WEEKS
p2p_aggregation_statsRange = 1814400000
p2p_aggregation_statsInterval = 60000
p2p_aggregation_jitterRange = 0.1
# glacier
p2p_glacier_logStatistics = true
p2p_glacier_faultInjectionEnabled = false
p2p_glacier_insertTimeout = 30000
p2p_glacier_minFragmentsAfterInsert = 3.0
p2p_glacier_refreshTimeout = 30000
p2p_glacier_expireNeighborsDelayAfterJoin = 30000
#5 MINS
p2p_glacier_expireNeighborsInterval = 300000
#5 DAYS
p2p_glacier_neighborTimeout = 432000000
p2p_glacier_syncDelayAfterJoin = 30000
#5 MINS
p2p_glacier_syncMinRemainingLifetime = 300000
#insertTimeout
p2p_glacier_syncMinQuietTime = 30000
p2p_glacier_syncBloomFilterNumHashes = 3
p2p_glacier_syncBloomFilterBitsPerKey = 4
p2p_glacier_syncPartnersPerTrial = 1
#1 HOUR
p2p_glacier_syncInterval = 3600000
#3 MINUTES
p2p_glacier_syncRetryInterval = 180000
p2p_glacier_syncMaxFragments = 100
p2p_glacier_fragmentRequestMaxAttempts = 0
p2p_glacier_fragmentRequestTimeoutDefault = 10000
p2p_glacier_fragmentRequestTimeoutMin = 10000
p2p_glacier_fragmentRequestTimeoutMax = 60000
p2p_glacier_fragmentRequestTimeoutDecrement = 1000
p2p_glacier_manifestRequestTimeout = 10000
p2p_glacier_manifestRequestInitialBurst = 3
p2p_glacier_manifestRequestRetryBurst = 5
p2p_glacier_manifestAggregationFactor = 5
#3 MINUTES
p2p_glacier_overallRestoreTimeout = 180000
p2p_glacier_handoffDelayAfterJoin = 45000
#4 MINUTES
p2p_glacier_handoffInterval = 240000
p2p_glacier_handoffMaxFragments = 10
#10 MINUTES
p2p_glacier_garbageCollectionInterval = 600000
p2p_glacier_garbageCollectionMaxFragmentsPerRun = 100
#10 MINUTES
p2p_glacier_localScanInterval = 600000
p2p_glacier_localScanMaxFragmentsPerRun = 20
p2p_glacier_restoreMaxRequestFactor = 4.0
p2p_glacier_restoreMaxBoosts = 2
p2p_glacier_rateLimitedCheckInterval = 30000
p2p_glacier_rateLimitedRequestsPerSecond = 3
p2p_glacier_enableBulkRefresh = true
p2p_glacier_bulkRefreshProbeInterval = 3000
p2p_glacier_bulkRefreshMaxProbeFactor = 3.0
p2p_glacier_bulkRefreshManifestInterval = 30000
p2p_glacier_bulkRefreshManifestAggregationFactor = 20
p2p_glacier_bulkRefreshPatchAggregationFactor = 50
#3 MINUTES
p2p_glacier_bulkRefreshPatchInterval = 180000
p2p_glacier_bulkRefreshPatchRetries = 2
p2p_glacier_bucketTokensPerSecond = 100000
p2p_glacier_bucketMaxBurstSize = 200000
p2p_glacier_jitterRange = 0.1
#1 MINUTE
p2p_glacier_statisticsReportInterval = 60000
p2p_glacier_maxActiveRestores = 3
#transport layer testing params
org.mpisws.p2p.testing.transportlayer.replay.Recorder_printlog = true
# logging
#default log level
loglevel = WARNING
#example of enabling logging on the endpoint:
#rice.p2p.scribe#ScribeRegrTest-endpoint_loglevel = INFO
logging_packageOnly = true
logging_date_format = yyyyMMdd.HHmmss.SSS
logging_enable=true
# 24 hours
log_rotate_interval = 86400000
# the name of the active log file, and the filename prefix of rotated log
log_rotate_filename = freepastry.log
# the format of the date for the rotating log
log_rotating_date_format = yyyyMMdd.HHmmss.SSS
# true will tell the environment to ues the FileLogManager
environment_logToFile = false
# the prefix for the log files (otherwise will be named after the nodeId)
fileLogManager_filePrefix =
# the suffix for the log files
fileLogManager_fileSuffix = .log
# wether to keep the line prefix (declaring the node id) for each line of the log
fileLogManager_keepLinePrefix = false
fileLogManager_multipleFiles = true
fileLogManager_defaultFileName = main
# false = append true = overwrite
fileLogManager_overwrite_existing_log_file = false
# the amount of time the LookupService tutorial app will wait before timing out
# in milliseconds, default is 30 seconds
lookup_service.timeout = 30000
# how long to wait before the first retry
lookup_service.firstTimeout = 500
Edit: Comfirmed with wireshark that the message indeed reach the computer freepastry just don't accept the connection.

Not sure what you mean by "not work". To test the connectivity between your client and your server (sit behind NAT), you just need do something like "telnet mIP mBindport" on your client side, assuming you have a telnet utility (default on Linux and Mac, you can install one, like nc ("netcat") on your windows).
If the port forwarding is set up correctly, you should see something like the following when the TCP connection is set up with your server.
Connected to localhost.
Escape character is '^]'.
Once the TCP session sets up correctly, you can stop the "telnet" program and use your real client (in java) to talk to your server, it should work fine.
If the TCP session didn't set up, you may want to check on the server side. Use either a wireshark or tcpdump to capture packets with filter "tcp port 50001", and run the telnet command above to check if there is a TCP packet come in.
If nothing show up in wireshark or tcpdump, then your firewall (like portforwarding) is not set up correctly.
If the TCP packet does show up in wireshark or tcpdump, then your server program may be at fault. Check the IP address it binds to using the command (linux):
netstat -antp | grep 50001
(on windows, the command is slightly different).
Typically it should bind to IP address 0.0.0.0 (all ip), if it doesn't, you should check whether the IP it binds to has connectivity/route to the outside world (outside the NAT).
Good luck.

I would try to set your IP as your local for the computer Free Pastry is running on. It sounds like the computer is getting the information but Free Pastry is looking for it on a different address. If you set your mIP to be local, I think it would work. This would be if it is behind the router/NAT.
Port forwarding forwards packets from your public IP on port 50001 to your internal computer IP on whatever port you set, normally the same 50001. If you set your program to listen on the public IP, it doesn't have access to it so it will not accept any packets/messages. Set to listen on the computers IP, or 0.0.0.0/localhost, it should accept any packets/messages on that port.

java BerkeleyDB remove couple of records

How can I remove couple of records in one transaction?
Configs:
EnvironmentConfig myEnvConfig = new EnvironmentConfig();
StoreConfig storeConfig = new StoreConfig();
myEnvConfig.setReadOnly(readOnly);
storeConfig.setReadOnly(readOnly);
// If the environment is opened for write, then we want to be
// able to create the environment and entity store if
// they do not exist.
myEnvConfig.setAllowCreate(!readOnly);
storeConfig.setAllowCreate(!readOnly);
// Allow transactions if we are writing to the store.
myEnvConfig.setTransactional(!readOnly);
storeConfig.setTransactional(!readOnly);
// Open the environment and entity store
bklEnv = new Environment(envHome, myEnvConfig);
//bklEnv.openDatabase(null, envHome.getAbsolutePath(), myEnvConfig);
bklstore = new EntityStore(bklEnv, entryStore, storeConfig);
Clear old data in cyclically. Here we are clearing data by getting firstKey() from index:
public void clearOldDBData(Integer maxCount) throws DatabaseException {
TransactionConfig config = new TransactionConfijg();
config.setReadUncommitted(true);
Transaction txn = berkeleyDbEnv.getBklEnv().beginTransaction(null, config);
txn.setTxnTimeout(1000);
Long keyV = null;
try{
PrimaryIndex<Long,MemoryBTB> memoryBTBIndex =
berkeleyDbEnv.getBklstore().getPrimaryIndex(Long.class, MemoryBTB.class);
if(!memoryBTBIndex.sortedMap().isEmpty() && memoryBTBIndex.sortedMap().keySet().size() > maxCount){
for(int i = 0; i < memoryBTBIndex.sortedMap().keySet().size() - maxCount; i++){
log.trace(BERKELEYDB_CLEAR_DATA);
System.out.println("**************************************************");
PrimaryIndex<Long,MemoryBTB> memoryBTBIndexInternal =
berkeleyDbEnv.getBklstore().getPrimaryIndex(Long.class, MemoryBTB.class);
memoryBTBIndexInternal.delete(txn, memoryBTBIndexInternal.sortedMap().firstKey());
}
}
txn.commit();
System.out.println("+++++++++++++++++++++++++++++++++++++++++++++++++++");
}catch(DatabaseException dbe){
// one more time deleting
try {
Thread.sleep(100);
dataAccessor.getMemoryBTB().delete(txn, keyV);
txn.commit();
}catch(DatabaseException dbeInternal){
log.trace(String.format(TXN_ABORT, dbeInternal.getMessage()));
txn.abort();
} catch (InterruptedException e) {
e.printStackTrace();
throw dbe;
}
}
}
Stacktrace:
[12/12 10:35:20] - TRACE - BerkeleyRepository - Berkeley DB clear data
**************************************************
[12/12 10:35:20] - TRACE - BerkeleyRepository - Berkeley DB clear data
**************************************************
[12/12 10:35:21] - TRACE - MemService - Berkeley DB JSON produce error: (JE 3.3.75) Lock expired. Locker 7752330 -1_Thread-295_ThreadLocker: waited for lock on database=persist#MemoryEntityStore#com.company.memcheck.persists.MemoryBTB LockAddr:1554328 node=333 type=READ grant=WAIT_NEW timeoutMillis=500 startTime=1386862520718 endTime=1386862521218
Owners: [<LockInfo locker="31510392 17395_Thread-295_Txn" type="WRITE"/>]
Waiters: []
com.sleepycat.util.RuntimeExceptionWrapper: (JE 3.3.75) Lock expired. Locker 7752330 -1_Thread-295_ThreadLocker: waited for lock on database=persist#MemoryEntityStore#com.com pany.memcheck.persists.MemoryBTB LockAddr:1554328 node=333 type=READ grant=WAIT_NEW timeoutMillis=500 startTime=1386862520718 endTime=1386862521218
Owners: [<LockInfo locker="31510392 17395_Thread-295_Txn" type="WRITE"/>]
Waiters: []
at com.sleepycat.collections.StoredContainer.convertException(StoredContainer.java:466)
at com.sleepycat.collections.StoredSortedMap.getFirstOrLastKey(StoredSortedMap.java:216)
at com.sleepycat.collections.StoredSortedMap.firstKey(StoredSortedMap.java:185)
at com.company.memcheck.repository.BerkeleyRepositoryImpl.clearOldDBData(BerkeleyRepositoryImpl.java:142)
at com.company.memcheck.service.MemServiceImpl.removeOldData(MemServiceImpl.java:305)
at com.company.memcheck.service.MemServiceImpl.access$3(MemServiceImpl.java:299)
at com.company.memcheck.service.MemServiceImpl$2.run(MemServiceImpl.java:129)
at java.lang.Thread.run(Thread.java:662)
So as we can see only one entry in a loop where "*******" indicating that it was successful but the others failed.
Do I use cursor for this one?

Operation timed out using CouchbaseClient

I am getting Timeout exceptions even though there is not much load on the Couchbase server.
net.spy.memcached.OperationTimeoutException: Timeout waiting for value
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1003)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1018)
at com.eos.cache.CacheClient.get(CacheClient.java:280)
at com.eos.cache.GenericCacheAccessObject.get(GenericCacheAccessObject.java:55)
...
...
Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: /192.168.4.12:11210
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:157)
at net.spy.memcached.internal.GetFuture.get(GetFuture.java:62)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:997)
...30 more
This is how I am creating the client.
List<URI> uris = new ArrayList<URI>();
String[] serverTokens = getServers().split(" ");
for (int index = 0; index < serverTokens.length; index++) {
uris.add(new URI(serverTokens[index]));
}
CouchbaseConnectionFactoryBuilder ccfb = new CouchbaseConnectionFactoryBuilder();
ccfb.setProtocol(Protocol.BINARY);
ccfb.setOpTimeout(10000); // wait up to 10 seconds for an operation to
// succeed
ccfb.setOpQueueMaxBlockTime(5000); // wait up to 5 seconds when trying
// to enqueue an operation
ccfb.setMaxReconnectDelay(1500);
CouchbaseConnectionFactory cf = ccfb.buildCouchbaseConnection(uris, bucket, "");
CouchbaseClient client = new CouchbaseClient(cf);
I am maintaining a pool of persistent clients in our web server. And we are not even touching the max conn limit which has been set to 15 only.
Pls help me guys in solving this.