I'm trying to connect to a remote kafka zookeeper (zookeeperhost:1234) via SimpleConsumer, and pulling metadata for the topics that I've specified. However, I get this error:
RunnerMain failed :java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
I ran telnet zookeeperhost 1234 and got the following:
Trying <ip addr of host>...
Connected to zookeeperhost.somedomain.com.
Escape character is '^]'.
So it looks like I am able to reach the service, there is no security/permission issue.
Here is my code:
SimpleConsumer consumer = new SimpleConsumer ("zookeeperhost", 1234, 100000, 64 * 1024, "leader_lookup");
List<String> topics = Collections.singletonList("some_event");
TopicMetadataRequest req = new TopicMetadataRequest(topics);
TopicMetadataResponse resp = consumer.send(req); //this is the line that is causing the error
List<TopicMetadata> metaData = resp.topicsMetadata();
You need to connect SimpleConsumer to a kafka leader broker (port 9092 by default) not a zookeeper. Check this link for the details how to use SimpleConsumer: https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
Related
I am currently trying to listen for all UDP Traffic being broadcasted on the network using a specific port. For now lets say the port is 12345. When the data arrives the plan is to redirect the data out port 13579.
Currently I am trying to learn/use the library of PCAP4J. Using the tutorial found on their site (https://www.pcap4j.org/) I have been able to connect and look at data.
Sample Code Thus Far:
InetAddress ip = InetAddress.getByName("127.0.0.1");
PcapNetworkInterface nif = Pcaps.getDevByAddress(ip);
System.out.println("GOT NIF");
PromiscuousMode mode = PromiscuousMode.PROMISCUOUS;
int timeout = 10;
PcapHandle handle = nif.openLive(snapLen, mode, timeout);
System.out.println("GOT HANDEL");
Packet packet = handle.getNextPacketEx();
handle.close();
Questions:
Am I correct in assuming that I am looking at all traffic being sent to 127.0.0.1?
Is there an attribute of the packet that would allow me to say "This packet is on port 12345", or is there a way to configure PCAP4J to only listen to the port of 12345?
I am using MQ Java PCF API to retrieve information from an MQ installation. There are several constructors for the class PCFMessageAgent. There is one that accepts the host, port and channel name. I have shown an extract from the documentation.
public PCFMessageAgent(String host,
int port,
String channel)
throws MQException
Initialises a new PCFMessageAgent with a client connection to a queue manager.
The code I have is
PCFMessageAgent agent = new PCFMessageAgent(host, port, "SYSTEM.DEF.SVRCONN");
PCFMessage pcfCmd = new PCFMessage(MQConstants.MQCMD_INQUIRE_CHANNEL);
pcfCmd.addParameter(MQConstants.MQCACH_CHANNEL_NAME, channelName);
PCFMessage[] pcfResponse = agent.send(pcfCmd);
but I get a MQRC_NOT_AUTHORIZED error
Exception in thread "main" com.ibm.mq.MQException: MQJE001: Completion Code '2', Reason '2035'.
at com.ibm.mq.MQManagedConnectionJ11.<init>(MQManagedConnectionJ11.java:249)
at com.ibm.mq.MQClientManagedConnectionFactoryJ11._createManagedConnection(MQClientManagedConnectionFactoryJ11.java:450)
at com.ibm.mq.MQClientManagedConnectionFactoryJ11.createManagedConnection(MQClientManagedConnectionFactoryJ11.java:487)
at com.ibm.mq.StoredManagedConnection.<init>(StoredManagedConnection.java:97)
at com.ibm.mq.MQSimpleConnectionManager.allocateConnection(MQSimpleConnectionManager.java:194)
at com.ibm.mq.MQQueueManagerFactory.obtainBaseMQQueueManager(MQQueueManagerFactory.java:868)
Why do I get this error? How do I fix this.
I found this in the error logs. Any help will be much appreciated.
07/08/17 15:42:51 - Process(3294.15) User(user1) Program(amqrmppa)
Host(ubuntuvm-2) Installation(Installation1)
VRMF(8.0.0.7) QMgr(QM_FR2_2)
AMQ9777: Channel was blocked
EXPLANATION:
The inbound channel 'SYSTEM.DEF.SVRCONN' was blocked from address
'192.168.56.101' because the active values of the channel matched a record
configured with USERSRC(NOACCESS). The active values of the channel were
'CLNTUSER(user1)'.
ACTION:
Contact the systems administrator, who should examine the channel
authentication records to ensure that the correct settings have been
configured. The ALTER QMGR CHLAUTH switch is used to control whether channel
authentication records are used. The command DISPLAY CHLAUTH can be used to
query the channel authentication records.
----- cmqxrmsa.c : 1462 -------------------------------------------------------
07/08/17 15:42:51 - Process(3294.15) User(user1) Program(amqrmppa)
Host(ubuntuvm-2) Installation(Installation1)
VRMF(8.0.0.7) QMgr(QM_FR2_2)
AMQ9999: Channel 'SYSTEM.DEF.SVRCONN' to host '192.168.56.101' ended
abnormally.
EXPLANATION:
The channel program running under process ID 3294 for channel
'SYSTEM.DEF.SVRCONN' ended abnormally. The host name is '192.168.56.101'; in
some cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
----- amqrmrsa.c : 930 --------------------------------------------------------
The issue is that you are attempting to connect to the channel SYSTEM.DEV.SVRCONN. By default there is a CHLAUTH rule that blocks access to channels named SYSTEM.*.
You can create a new SVRCONN channel that does not have a name that starts with SYSTEM and it would get past this check.
As per ServerSocket(int port, int backlog), backlog indicates maximum length of the queue.
I created a simple server socket, with backlog as 1 using following code ServerSocket serverSocket = new ServerSocket(8001, 1); and my expectation was that when I will try to connect to this server with more than 1 client then I will get connection refused exception.
But when I tried to connect with this server from more than 1 client then I was able to connect even with 3 client in parallel and all were able to communicate with the server.
Am I missing something?
The backlog parameter can be adjusted up or down by the platform. The actual minimum backlog was initially 5 in BSD 4.3 and it is now more like 50 or even 500 on some platforms. There is no API to determine the actual value used.
I have for legacy reasons a Java activeMQ implementation of the Broker/Publisher over vanilla tcp transport protocol. I wish to connect a Python client to it, however all the "stomp" based documentation doesn't seem to have it, not over the stomp protcol, and when I try the basic examples I get the error on the Java Broker side:
[ActiveMQ Transport: tcp:///127.0.0.1:62860#5001] WARN org.apache.activemq.broker.TransportConnection.Transport - Transport Connection to: tcp://127.0.0.1:62860 failed: java.io.IOException: Unknown data type: 80
The Broker code is very vanilla in Java:
String localVMurl = "vm://localhost";
String remoterURL = "tcp://localhost:5001";
BrokerService broker = new BrokerService();
broker.addConnector(localVMurl);
broker.addConnector(remoterURL);
broker.setAdvisorySupport(true);
broker.start();
ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(localVMurl+"?create=false");
Connection connection = connectionFactory.createConnection();
and the Python just fails. I can't seem to find anything online using just basic "tcp://localhost:" connections from Python. Am I doing something wrong here?
import stomp
class MyListener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)
def on_message(self, headers, message):
print('received a message "%s"' % message)
conn = stomp.Connection(host_and_ports = [('localhost', 5001)])
conn.start()
conn.connect('admin', 'password', wait=True)
and I get the error:
IndexError: list index out of range
Without seeing the broker configuration it is a bit tricky to answer but from the error I'd guess you are trying to connect a STOMP client to the OpenWire transport which won't work, you need to have a STOMP TransportConnector configured on the broker and point the STOMP client there.
See the ActiveMQ STOMP documentation.
To add STOMP support for an embedded broker you'd do something along the lines of:
brokerService.addConnector("stomp://0.0.0.0:61613");
I'm begininng to use Mqtt and I have a hard time with handling an unreliable network.
I'm using a Paho Java Client (in groovy) to publish messages to a distant Mosquitto Broker.
Is there a way, when the broker is unreachable, to have the Paho client persist the message and automatically re-connect to the broker and publish the locally stored messages ? Do I have to handle everything myself, using for example a local broker ?
Here is my client building code
String persistenceDir = config['persistence-dir'] ?: System.getProperty('java.io.tmpdir')
def persistence = new MqttDefaultFilePersistence(persistenceDir)
client = new MqttAsyncClient(uri, clientId, persistence)
client.setCallback(this)
options = new MqttConnectOptions()
if (config.password) {
options.setPassword(config.password as char[])
options.setUserName(config.user)
}
options.setCleanSession(false)
client.connect(options)
And my publish code
def message = new MqttMessage(Json.encode(outgoingMessage).getBytes())
try {
client?.connect(options)
def topic = client.getTopic('processMsg')
message.setQos(1)
def token = topic.publish(message)
if (client) {
client.disconnect()
}
Thanks
The Paho client will only persist in-flight messages when it is connected to the broker.
Typically, when connectivity issues start to arrive you'll see message timeouts popping up
Timed out waiting for a response from the server (32000)
At that point the message will still be persisted.
However, when the connection is lost, and you start seeing this
Client is not connected (32104)
You should assume that the message has not been persisted by Paho.
You can debug this in org.eclipse.paho.client.mqttv3.internal.ClientComms :
/**
* Sends a message to the broker if in connected state, but only waits for the message to be
* stored, before returning.
*/
public void sendNoWait(MqttWireMessage message, MqttToken token) throws MqttException {
final String methodName = "sendNoWait";
if (isConnected() ||
(!isConnected() && message instanceof MqttConnect) ||
(isDisconnecting() && message instanceof MqttDisconnect)) {
this.internalSend(message, token);
} else {
//#TRACE 208=failed: not connected
log.fine(className, methodName, "208");
throw ExceptionHelper.createMqttException(MqttException.REASON_CODE_CLIENT_NOT_CONNECTED);
}
}
The internalSend will persist the message, but only if it is connected to the broker.
Also take into account that there is a maximum number of inflight messages that Paho can process. If it exceeds that it will also decide to not persist the message.
You could just setup a local broker and bridge that with the remote broker. That way you can queue up all your messages locally and when the remote broker comes back online all can be delivered.
Yes... After you get an exception that the message can't be delivered, it has to be either persisted or the message needs to be regenerated.
If you plan to use a local broker you can look at Really Small Message Broker (https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=d5bedadd-e46f-4c97-af89-22d65ffee070)