I need to assign different values of memory for each new worker. So I tried changing memory for each bolt and spout. I am currently using a custom scheduler also. Here is my approach to the problem.
MY CODE:
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new EmailSpout(), 1).addConfiguration("node", "zoo1").setMemoryLoad(512.0);
builder.setBolt("increment1", new IncrementBolt(), PARALLELISM).shuffleGrouping("spout").addConfiguration("node", "zoo2").setMemoryLoad(2048.0);
builder.setBolt("increment2", new IncrementBolt(), PARALLELISM).shuffleGrouping("increment1").addConfiguration("node", "zoo3").setMemoryLoad(2048.0);
builder.setBolt("increment3", new IncrementBolt(), PARALLELISM).shuffleGrouping("increment2").addConfiguration("node", "zoo4").setMemoryLoad(2048.0);
builder.setBolt("output", new OutputBolt(), 1).globalGrouping("increment2").addConfiguration("node", "zoo1").setMemoryLoad(512.0);
Config conf = new Config();
conf.setDebug(false);
conf.setNumWorkers(4);
StormSubmitter.submitTopologyWithProgressBar("Microbenchmark", conf, builder.createTopology());
MY STORM.YAML:
storm.zookeeper.servers:
- "zoo1"
storm.zookeeper.port: 2181
nimbus.seeds: ["zoo1"]
storm.local.dir: "/home/ubuntu/eranga/storm-data"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
- 6704
storm.scheduler: "org.apache.storm.scheduler.NodeBasedCustomScheduler"
supervisor.scheduler.meta:
node: "zoo4"
worker.profiler.enabled: true
worker.profiler.childopts: "-XX:+UnlockCommercialFeatures -XX:+FlightRecorder"
worker.profiler.command: "flight.bash"
worker.heartbeat.frequency.secs: 1
worker.childopts: "-Xmx2048m -Xms2048m -Djava.net.preferIPv4Stack=true -Dorg.xml.sax.driver=com.sun.org.apache.xerces.internal.parsers.SAXParser -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl"
When I submit the topology I get the following error.
ERROR:
Exception in thread "main" java.lang.IllegalArgumentException: Topology will not be able to be successfully scheduled: Config TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB=768.0 < 2048.0 (Largest memory requirement of a component in the topology). Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount
at org.apache.storm.StormSubmitter.validateTopologyWorkerMaxHeapSizeMBConfigs(StormSubmitter.java:496)
Any suggestions?
Try using this.
import org.apache.storm.Config;
public class TopologyExecuter{
for(List<StormTopology> StormTopologyObject : StormTopologyObjects ){
Config topologyConf = new Config();
topologyConf.put(Config.TOPOLOGY_WORKER_CHILDOPTS,"-Xmx512m -Xms256m");
StormSubmitter.submitTopology("topology name", topologyConf, StormTopologyObject);
}
}
Did you try following the advice from the error message?
Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount
Try adding this to storm.yaml:
topology.worker.max.heap.size.mb=2048.0
Related
I am trying to make it so I can redeploy a JBoss 7.1.0 cluster with a WAR that has apache ignite.
I am starting the cache like this:
System.setProperty("IGNITE_UPDATE_NOTIFIER", "false");
igniteConfiguration = new IgniteConfiguration();
int failureDetectionTimeout = Integer.parseInt(getProperty("IGNITE_TCP_DISCOVERY_FAILURE_DETECTION_TIMEOUT", "60000"));
igniteConfiguration.setFailureDetectionTimeout(failureDetectionTimeout);
String igniteVmIps = getProperty("IGNITE_VM_IPS");
List<String> addresses = Arrays.asList("127.0.0.1:47500");
if (StringUtils.isNotBlank(igniteVmIps)) {
addresses = Arrays.asList(igniteVmIps.split(","));
}
int networkTimeout = Integer.parseInt(getProperty("IGNITE_TCP_DISCOVERY_NETWORK_TIMEOUT", "60000"));
boolean failureDetectionTimeoutEnabled = Boolean.parseBoolean(getProperty("IGNITE_TCP_DISCOVERY_FAILURE_DETECTION_TIMEOUT_ENABLED", "true"));
int tcpDiscoveryLocalPort = Integer.parseInt(getProperty("IGNITE_TCP_DISCOVERY_LOCAL_PORT", "47500"));
int tcpDiscoveryLocalPortRange = Integer.parseInt(getProperty("IGNITE_TCP_DISCOVERY_LOCAL_PORT_RANGE", "0"));
TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
tcpDiscoverySpi.setLocalPort(tcpDiscoveryLocalPort);
tcpDiscoverySpi.setLocalPortRange(tcpDiscoveryLocalPortRange);
tcpDiscoverySpi.setNetworkTimeout(networkTimeout);
tcpDiscoverySpi.failureDetectionTimeoutEnabled(failureDetectionTimeoutEnabled);
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(addresses);
tcpDiscoverySpi.setIpFinder(ipFinder);
igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);
Ignite ignite = Ignition.start(igniteConfiguration);
ignite.cluster().active(true);
Then I am stopping the cache when the application undeploys:
ignite.close();
When I try to redeploy, I get the following error during initialization.
org.apache.ignite.spi.IgniteSpiException: Failed to marshal custom event: StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=org.apache.ignite.internal.cluster.ClusterGroupAdapter$CachesFilter#7385a997, clsName=null, depInfo=null, hnd=org.apache.ignite.internal.GridEventConsumeHandler#2aec6952, bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, deserEx=null, routineId=bbe16e8e-2820-4ba0-a958-d5f644498ba2]
If I full restart the server, starts up fine.
Am I missing some magic in the shutdown process?
I see what I did wrong, and it was code I omitted from the ticket.
ignite.events(ignite.cluster().forCacheNodes(cacheConfig.getKey())).remoteListen(locLsnr, rmtLsnr,
EVT_CACHE_OBJECT_PUT, EVT_CACHE_OBJECT_READ, EVT_CACHE_OBJECT_REMOVED);
When it was trying to register this code twice, it was causing that strange error.
I put a try-catch ignore around it for now and things seem to be ok.
I hope I'm able to explain this!
I'm starting a dockerized java spring boot application that will connect to a single dockerized Kafka instance.
To do this I have setup a link in the docker-compose file that will allow the application to connect to the kafka docker, named kafka-cluster on port 9092.
When I start the both containers I get an error in the java application saying that it's unable to connect to KafkaAdmin:
[AdminClient clientId=adminclient-2] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
But it's trying to connect to localhost/127.0.0.1.
Further up in the logs, I can see it started a connection to KafkaAdmin twice:
first:
2020-03-25 13:53:15.515 INFO 7 --- [ main] o.a.k.clients.admin.AdminClientConfig : AdminClientConfig values:
bootstrap.servers = [kafka-cluster:9092]
client.dns.lookup = default
client.id =
connections.max.idle.ms = 300000
<more properties>
and then again straight after (but on localhost):
2020-03-25 13:53:15.780 INFO 7 --- [ main] o.a.k.clients.admin.AdminClientConfig : AdminClientConfig values:
bootstrap.servers = [localhost:9092]
client.dns.lookup = default
client.id =
connections.max.idle.ms = 300000
<more properties>
This is the config:
#Bean
public KafkaAdmin kafkaAdmin() {
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
return new KafkaAdmin(configs);
}
#Bean
public NewTopic sysCcukCdcAssetsCreate() {
return new NewTopic(newPanelTopic, 1, (short) 1);
}
#Bean
public NewTopic sysCcukCdcAssetsUpdate() {
return new NewTopic(updatedPanelTopic, 1, (short) 1);
}
where bootstrapServers = kafka-cluster:9092
I can't see why it seems like KafkaAdmin has 2 sets of configs but it seems like it is causing the error.
Any guidance or suggestions massively appreciated :)
So! This was a great rubber duck.
Turns out the spring.kafka.bootstrap.servers property hadn't been set in the properties file and it defaults to localhost setting this to kafka-cluster:9092 fixed it :)
I am new in Apache ignite. I am trying to populate cache and read from cache. I have created 2 java projects one populates Apache ignite caches and the other one print cached data however printing cache project gives error.
here is the code that I use to populate caches
public void run(String... arg0) throws Exception
{
try (Ignite ignite = Ignition.start("ignite.xml"))
{
int iteration=0;
while(true)
{
iteration++;
IgniteCache<Object, Object> cache = ignite.getOrCreateCache("test cache " + iteration);
System.out.println(""+100);
System.out.println("Caching started for iteration " + iteration);
printMemory();
for (int i = 0; i < 100; i++)
{
cache.put(i, new CacheObject(i, "Cached integer " + i));
System.out.println(i);
Thread.sleep(100);
}
//cache.destroy();
System.out.println("**************************************"+cache.size());
}
}
}
this is the code that I use to print cached data
Ignition.setClientMode(true);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setPeerClassLoadingEnabled(true);
TcpDiscoveryMulticastIpFinder discoveryMulticastIpFinder = new TcpDiscoveryMulticastIpFinder();
Set<String> set = new HashSet<>();
set.add("serverhost:47500..47509");
discoveryMulticastIpFinder.setAddresses(set);
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
discoverySpi.setIpFinder(discoveryMulticastIpFinder);
cfg.setDiscoverySpi(discoverySpi);
cfg.setPeerClassLoadingEnabled(true);
cfg.setIncludeEventTypes(EVTS_CACHE);
Ignite ignite = Ignition.start(cfg);
System.out.println("***************************************************\n"+ignite.cacheNames()+"\n****************************");
CacheConfiguration<String, BinaryObject> cacheConfiguration = new CacheConfiguration<>(CACHE_NAME);
IgniteCache<String, BinaryObject> cache = ignite.getOrCreateCache(cacheConfiguration).withKeepBinary();
this 2 peace of code in different project so while 1 first project populating cache, I am trying to reach the cache from another project
the error below is occurred when I am trying to reach cached data
Error that returns from the code that reads cache and prints it
Aug 01, 2018 9:25:25 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager]
class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:258)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:660)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1505)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:917)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1688)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:534)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:515)
at org.apache.ignite.Ignition.start(Ignition.java:322)
at test.App.main(App.java:76)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Local node's marshaller differs from remote node's marshaller (to make sure all nodes in topology have identical marshaller, configure marshaller explicitly in configuration) [locMarshaller=org.apache.ignite.internal.binary.BinaryMarshaller, rmtMarshaller=org.apache.ignite.marshaller.optimized.OptimizedMarshaller, locNodeAddrs=[192.168.1.71/0:0:0:0:0:0:0:1%lo, /127.0.0.1, /192.168.1.71], locPort=0, rmtNodeAddr=[192.168.1.71/0:0:0:0:0:0:0:1%lo, /127.0.0.1, /192.168.1.71], locNodeId=b41f0d09-5a7f-424b-b3b5-420a5e1acdf6, rmtNodeId=ff436f20-5d4b-477e-aade-837d59b1eaa7]
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1647)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1460)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Local node's marshaller differs from remote node's marshaller (to make sure all nodes in topology have identical marshaller, configure marshaller explicitly in configuration) [locMarshaller=org.apache.ignite.internal.binary.BinaryMarshaller, rmtMarshaller=org.apache.ignite.marshaller.optimized.OptimizedMarshaller
Looks like in ignite.xml you've explicitly set marshaller. Please check <property name="marshaller"> in xml config file. You should have the same marshaller configured for all nodes in the cluster, or they won't be able to communicate.
I am working with java in a maven project. I was using couchbase 2.3.1 but in trying to resolve this issue I rolled back to 2.2.8 to no avail.
The issue I get is that while I do get date through to my couchbase cluster I am seeing alot of this:
java.lang.RuntimeException: java.util.concurrent.TimeoutException
at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:75)
at com.couchbase.client.java.CouchbaseBucket.upsert(CouchbaseBucket.java:359)
at com.couchbase.client.java.CouchbaseBucket.upsert(CouchbaseBucket.java:354)
Below are the settings for my couchbase environment:
CouchbaseEnvironment: {sslEnabled=false, sslKeystoreFile='null', sslKeystorePassword='null', queryEnabled=false, queryPort=8093, bootstrapHttpEnabled=true, bootstrapCarrierEnabled=true, bootstrapHttpDirectPort=8091, bootstrapHttpSslPort=18091, bootstrapCarrierDirectPort=11210, bootstrapCarrierSslPort=11207, ioPoolSize=24, computationPoolSize=24, responseBufferSize=16384, requestBufferSize=16384, kvServiceEndpoints=1, viewServiceEndpoints=1, queryServiceEndpoints=1, searchServiceEndpoints=1, ioPool=NioEventLoopGroup, coreScheduler=CoreScheduler, eventBus=DefaultEventBus, packageNameAndVersion=couchbase-java-client/2.2.8 (git: 2.2.8, core: 1.2.9), dcpEnabled=false, retryStrategy=BestEffort, maxRequestLifetime=75000, retryDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=100, upper=100000}, reconnectDelay=ExponentialDelay{growBy 1.0 MILLISECONDS, powers of 2; lower=32, upper=4096}, observeIntervalDelay=ExponentialDelay{growBy 1.0 MICROSECONDS, powers of 2; lower=10, upper=100000}, keepAliveInterval=30000, autoreleaseAfter=2000, bufferPoolingEnabled=true, tcpNodelayEnabled=true, mutationTokensEnabled=false, socketConnectTimeout=1000, dcpConnectionBufferSize=20971520, dcpConnectionBufferAckThreshold=0.2, dcpConnectionName=dcp/core-io, callbacksOnIoPool=false, queryTimeout=75000, viewTimeout=75000, kvTimeout=2500, connectTimeout=5000, disconnectTimeout=25000, dnsSrvEnabled=false}
Im not really too sure what to look at here. As far as I can tell there should be a decent enough connection between the server where the app is running and the couchbase cluster. Any help or direction on this would be helpful. Here is a snippet from where the error is being thrown.
LockableItem<InnerVertex> lv = this.getInnerVertex(id);
lv.lock();
try {
String content;
try {
content = mapper.writeValueAsString(lv.item);
} catch (JsonProcessingException e) {
LOG.warning(e.getMessage());
return;
}
RawJsonDocument d = RawJsonDocument.create(VertexId.toKey(id), content);
bucket.upsert(d);
} finally {
lv.unlock();
}
I was searching for the answer. I got there are many solutions all are talking about exception. I also checked the jar code, it is called that it is timeout exception.
Root Cause Analysis
The error occured from the following section of couchbase: https://github.com/couchbase/couchbase-java-client/blob/master/src/main/java/com/couchbase/client/java/util/Blocking.java#L71
public static <T> T blockForSingle(final Observable<? extends T> observable, final long timeout,
final TimeUnit tu) {
final CountDownLatch latch = new CountDownLatch(1);
TrackingSubscriber<T> subscriber = new TrackingSubscriber<T>(latch);
observable.subscribe(subscriber);
try {
if (!latch.await(timeout, tu)) { // From here, this error occurs.
throw new RuntimeException(new TimeoutException());
}
}
If the timeout kicks in, a TimeoutException nested in a
RuntimeException is thrown to be fully compatible with the
Observable.timeout(long, TimeUnit) behavior.
Resource Link:
http://docs.couchbase.com/sdk-api/couchbase-java-client-2.2.0/com/couchbase/client/java/util/Blocking.html
Your configuration analysis and solution:
Your couchbase environment connectionTimeout is 5000ms or 5sec, which is the default value of connection timeout.
You need to increase this value to 10000ms or greater. Your problem will be solved.
//this tunes the SDK (to customize connection timeout)
CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder()
.connectTimeout(10000) //10000ms = 10s, default is 5s
.build();
A full Solution
Simonbasle has given a full solution in this tutorial:
From the short log, it looks like the SDK is able to connect to the
node, but takes a little much time to open the bucket. How good is the
network link between the two machines? Is this a VM/cloud machine?
What you can try to do is increase the connect timeout:
public class NoSQLTest {
public static void main(String[] args) {
try {
//this tunes the SDK (to customize connection timeout)
CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder()
.connectTimeout(10000) //10000ms = 10s, default is 5s
.build();
System.out.println("Create connection");
//use the env during cluster creation to apply
Cluster cluster = CouchbaseCluster.create(env, "10.115.224.94");
System.out.println("Try to openBucket");
Bucket bucket = cluster.openBucket("beer-sample"); //you can also force a greater timeout here (cluster.openBucket("beer-sample", 10, TimeUnit.SECONDS))
System.out.println("disconnect");
cluster.disconnect();
}
catch (Exception e) {
e.printStackTrace();
}
}
}
As a side note, you should always reuse the CouchbaseEnvironment,
CouchbaseCluster and Bucket instances once created (usually by making
them public static somewhere, or a Spring singleton, etc...). These
are thread safe and should be shared (and they are expensive to create
anyway).
Resource Link:
Couchbase connection timeout with Java SDK
Thanks for the question, and for #SkyWalker's Answer.
They helped when I encountered this annoying timeout.
For Spring Data Couchbase 2, adding the following to application.properties solved it
spring.couchbase.env.timeouts.connect=20000
I'm trying to connect to a remote hbase-0.94.8 installed on a ubuntu vm. I'm having a TableNotFoundException and this is my Java code:
Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "192.168.56.101");
HTableInterface usersTable = new HTable(config, "users");
Here is the full exception trace:
14/06/24 15:59:48 WARN client.HConnectionManager$HConnectionImplementation: Encountered problems when prefetch META table:
org.apache.hadoop.hbase.TableNotFoundException: Cannot find row in .META. for table: users, row=users,,99999999999999
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:158)
at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:52)
at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130)
at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:127)
at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:360)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:127)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:103)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:876)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:930)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
at com.heavenize.samples.hbase.UsersTool.main(UsersTool.java:37)
Exception in thread "main" org.apache.hadoop.hbase.TableNotFoundException: users
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:952)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
at com.heavenize.samples.hbase.UsersTool.main(UsersTool.java:37)
You can check for table before using it. To do it you may use HBaseAdmin.
Create HBaseAdmin instance and thet use isTableAvailable(String tableName) method.
Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "192.168.56.101");
HBaseAdmin admin = new HBaseAdmin(config);
if(admin.isTableAvailable(tableName))
{
HTableInterface usersTable = new HTable(config, "users");
}
I hope it will help you.