I have managed to install and configure Ganglia on my cluster. I do not want to just see all performance data on ganglia web interface but instead I want to read cluster information from other application (application may be Java or Python based). I am not able to find if it is possible or not.
Is there any API to read Ganglia data?
To test Ganglia I used telnet master 8649 and Ganglia showed me nice XML text on my console. But how do I do the same thing using Java or Python? I can definitely connect to 8649 using sockets but after that do I need to send something to Ganglia daemons?
I can help you to get an insight on this. But before that I must tell you, I am not a Java programmer, rather I am a C/C++ programmer. So, it means I can let you know, how things work in ganglia and you can find equivalent methods in Java/ Python to rewrite the codes you want.
Please be informed that there is no API in ganglia to achieve what you want to.
First consider below set up of ganglia to understand properly:
GS1 and GS2 are collecting system metrics and pushing them to GM.
So, according to your question, if you want to collect all such metrics by your own Java/ Python based application, then you may have to install the application on the Master server (i.e Replace GS with your own application).
GS1 and GS2 sends all collected metrics either by UDP unicast channel or UDP multicast channel. It is recommended that UDP unicast should be enabled in every gmond.conf for easier scalability.
I wouldn't discuss much on GS1 and GS2 as your question is more about replacing GM with your own tool.
GM uses two important libraries heavily to establish a UDP connection and translate data into its own readable format. They are APR (Apache Portable Runtime) to establish UDP connection and perform related activities and XDR (External Data Representation) to send data across networks and perform RPC.
You need to find APR and XDR equivalent libraries in Java and Python first. XDR is already available in Java and APR could be replaced by your own basic implementation to perform inter-network operations (i.e., create UDP socket, etc).
Open gmond.c source file of ganglia and go to line 1436. You will find a C function:
static void process_udp_recv_channel(const apr_pollfd_t *desc, apr_time_t now).
This function basically performs "UDP connection establishment" and "data translation into readable format" activities.
The call flow of the above function is shown below:
Now, let's expand the function at line 1436 to understand more.
The first argument in this function carries network parameters such as IP, Port, etc. The structure is expanded below. You can find similar object in Java also.
struct apr_pollfd_t {
apr_pool_t *p; /**< associated pool */
apr_datatype_e desc_type; /**< descriptor type */
apr_int16_t reqevents; /**< requested events */
apr_int16_t rtnevents; /**< returned events */
apr_descriptor desc; /**< #see apr_descriptor */
void *client_data; /**< allows app to associate context */
};
The second parameter has nothing to do, if SFLOW is disabled.
So, Start with creating a APR pool, UDP connection, etc.
socket = desc->desc.s;
channel = desc->client_data;
apr_pool_create(&p, global_context);
status = apr_socket_addr_get(&remotesa, APR_LOCAL, socket);
status = apr_sockaddr_info_get(&remotesa, NULL, remotesa->family, remotesa->port, 0, p);
/* Grab the data */
status = apr_socket_recvfrom(remotesa, socket, 0, buf, &len);
if(status != APR_SUCCESS)
{
apr_pool_destroy(p);
return;
}
apr_sockaddr_ip_buffer_get(remoteip, 256, remotesa);
/* Check the ACL */
if(Ganglia_acl_action( channel->acl, remotesa) != GANGLIA_ACCESS_ALLOW)
{
apr_pool_destroy(p);
return;
}
All declarations of variable can be found in the beginning of the function expanded (line 1439 to 1456).
Then, create XDR stream:
xdrmem_create(&x, buf, max_udp_message_len, XDR_DECODE);
Flush the data of the struct which saves metadata and metrics value:
memset( &fmsg, 0, sizeof(Ganglia_metadata_msg));
memset( &vmsg, 0, sizeof(Ganglia_value_msg));
fmsg (Ganglia_metadata_msg) and vmsg (Ganglia_value_msg) struct definitions can be found in gm_protocol.h header file. Re-write them in Java.
Then, figure out if the message received is "metadata" or "metrics values".
xdr_Ganglia_msg_formats(&x, &id); // this function is located in the source file gm_protocol_xdr.c and this file is generated by rpcgen.
Note: rpcgen is a rpc compiler and its explanation can be found in this question.
Note: Here is the link for gm_protocol_xdr.c.
Here, id is an enum and its declaration is shown below:
enum Ganglia_msg_formats {
gmetadata_full = 128,
gmetric_ushort = 128 + 1,
gmetric_short = 128 + 2,
gmetric_int = 128 + 3,
gmetric_uint = 128 + 4,
gmetric_string = 128 + 5,
gmetric_float = 128 + 6,
gmetric_double = 128 + 7,
gmetadata_request = 128 + 8,
};
typedef enum Ganglia_msg_formats Ganglia_msg_formats;
Based on the value of id, you can determine what kind of value the packets have.
For this purpose, this function calls an another function (which is in fact generated by rpcgen) to determine the kind of value the packet has and if found, it translates it to human readable format too.
The function is:
xdr_Ganglia_value_msg(&x, &vmsg);
You can find the full expansion of this function in gm_protocol_xdr.c from line 275.
After that you can do whatever you would like with these packets.
In the end, you must free all XDR variables and APR pools allocated.
I hope this gives you a fair idea to start with your own application.
Related
I would like to monitor a TCP/IP endpoint using TCP half-open or embryonic connection as defined in RFC793. The purpose of such monitoring is to detect whether the TCP endpoint is available or not. Hal-open monitoring is important to reduce the footprint of such monitoring for the TCP endpoint.
The TCP cinematic is SYN->SYN-ACK<-RST.
Is it possible to implement such monitoring in Java? Using Netty library?
It seems to be achievable but one needs to use another low-level implementation of the TCP for java.
This library might help: https://github.com/mlaccetti/rocksaw/blob/master/src/main/java/com/savarese/rocksaw/net/RawSocket.java
An example of low-level sending SYN command:
https://github.com/dangan249/RawSocket/blob/master/ccs/neu/edu/andang/RawSocketClient.java#L181-L182
private final byte SYN_FLAG = (byte) 2;
private final byte ACK_FLAG = (byte) 16;
...
// send the SYN packet
sendMessage(null, this.getCurrentSeqNum(), this.getCurrentACKNum(), SYN_FLAG);
which boils down to just writing a proper packet into the socket:
TCPHeader header = new TCPHeader( this.sourcePort, this.destPort, sequenceNum ,
ackNum, flags , AD_WINDOW_SIZE ) ;
TCPPacket packet = new TCPPacket( header );
...
this.rSock.write( this.destAddress, packet.toByteArray() ) ;
By doing that one can implement a needed sequence of operations including the wanted SYN->SYNC-ACK<-RST
#Andremoniy answer was refined to build a simple C binary doing half-open/embryonic check. The C binary binary has required RAW SOCKET Linux capability. It is executed from Java code using Runtime.exec().
The advantage comparing to using rocksaw library is to avoid running the whole Java process with additional capabilities required for doing such check.
I'm trying to create communication between simple Java App (using java.net.http.WebSocket class) and remote google-chrome run using google-chrome --remote-debugging-port=9222 --user-data-dir=.
Sending and receiving small messages works as expected, but there is an issue in case of bigger messages, 16kb.
Here is part of java source:
var uri = new URI("ws://127.0.0.1:9222/devtools/page/C0D7B4DBC53FB39F7A4BE51DA79E96BB");
/// create websocket client
WebSocket ws = HttpClient
.newHttpClient()
.newWebSocketBuilder()
.connectTimeout(Duration.ofSeconds(30))
.buildAsync(uri, simpleListener)
.join();
// session Id attached to chrome tab
String sessionId = "...";
// send message
String message = "{\"id\":1,\"method\":\"Runtime.evaluate\",\"params\":{\"expression\":\"document.body.style.backgroundColor = 'blue';\",\"returnByValue\":true,\"awaitPromise\":true,\"userGesture\":true},\"sessionId\":\"" + sessionId + "\"}";
// this works
ws.send(message, true);
// generate big string contains over 18k chars for testing purpose
String bigMessage = "{\"id\":2,\"method\":\"Runtime.evaluate\",\"params\":{\"expression\":\"[" + ("1,".repeat(9000)) + "1]\",\"returnByValue\":true,\"awaitPromise\":true,\"userGesture\":true},\"sessionId\":\"" + sessionId + "\"}";
// this doesn't work
ws.send(bigMessage, true);
Here is stack:
java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:345)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:376)
at java.net.http/jdk.internal.net.http.SocketTube.readAvailable(SocketTube.java:1153)
at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:821)
at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:175)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
...
I've tried basically the same by using puppeteer (nodejs library) and it works as expected.
I can't find any resource online about this issue.
Is there anything I'm missing in my example?
Here is url to simple example:
https://github.com/zeljic/websocket-devtools-protocol
Based on what I've seen so far, my best guess would be that Chrome Dev Tools do not process fragmented Text messages on that exposed webSocketDebuggerUrl endpoint. Whether Chrome Dev Tools can be configured to do so or not, is another question. I must note, however, that RFC 6455 (The WebSocket Protocol) mandates it:
Clients and servers MUST support receiving both fragmented and unfragmented messages.
There's one workaround I can see here. Keep in mind that this is unsupported and may change in the future unexpectedly. When running your client, specify the following system property on the command line -Djdk.httpclient.websocket.intermediateBufferSize=1048576 (or pick any other suitable size). As long as you keep sending your messages with true passed as boolean last argument to the send* methods, java.net.http.WebSocket will send messages unfragment, in a single WebSocket frame.
Well I had a similar issue when sending a big string by using web-sockets in java with a tomcat server.
There can be payload limit to send or receive in websocket server .
checkout org.apache.tomcat.websocket.textBufferSize in tomcat's doc. By default it is 8192 bytes try increasing the size.
Hi I'm tring to use jSS7 to build MAP interface to send message like this one :
https://code.google.com/p/jss7/source/browse/map/load/src/main/java/org/mobicents/protocols/ss7/map/load/Server.java
but I don't found any document or any demo to know how to change from USSD to SMS
To tell the truth, I'm a bit unclear of what you would like to do.
USSDs and SMSes are really two different things, although there are similarities.
USSDs are intended for bidirectional communication between the user and network applications.
The VLR will invoke (on behalf of the user) processUnstructuredSS-Request
while the gsmSCF would invoke either unstructuredSS-Request or unstructuredSS-Notify:
processUnstructuredSS-Request: Allows the user to send a request, getting back (optionally) a response)
unstructuredSS-Request: Allows the application to present a prompt to the user, getting back a (optionally) response.
unstructuredSS-Notify: Allows the application to present a message to the user, without response.
The key parameters in the different invoke and return result components are the
USSD-DataCodingScheme ::= OCTET STRING (SIZE (1))
-- The structure of the USSD-DataCodingScheme is defined by
-- the Cell Broadcast Data Coding Scheme as described in
-- TS 3GPP TS 23.038 [25]
USSD-String ::= OCTET STRING (SIZE (1..maxUSSD-StringLength))
-- The structure of the contents of the USSD-String is dependent
-- on the USSD-DataCodingScheme as described in TS 3GPP TS 23.038 [25].
More details in 3GPP TS 23.038
Now, considering the SMS. In general two MAP operations are involved in the handling of SMS:
- MO-ForwardSM (sent to the VMSC)
- MT-ForwardSM (sent to the GMSC)
In both cases the key parameters are:
- sm-RP-DA (the destination)
- sm-RP-OA (the origin)
- sm-RP-UI (the info)
The sm-RP-UI of type SignalInfo carries an SM-TL PDU (Short Message Transport Layer)
These are defined at 3GPP TS 23.040
So, where's the connection?
The connection is that TP-Data-Encoding-Schema one of the components of a SM-TL PDU is the same as the USSD-DataCodingScheme in the USSD.
And that defines how both the TP-User-Data and the USSD-String are to be encoded.
Now you should really descend into all the gory details on the implementation and use of the USSD service and the SMS service to see how those are used in different scenarios.
Unfortunately there is no clear manul of how to implement different MAP messages because of there big count. Use MAP protocol specification (3GPP TS 29.002) to learn more.
You can check also opensource mobicent SMSC GW source code:
https://code.google.com/p/smscgateway/
I want to build a client-server-application for some practice. I started off with a simple chat which was not so hard to do. I'm also able to identify different commands by just simply split a String, e.g. "command:msg". But I think that may be a little inconvenient. So I'm wondering if there are better ways to realize that. And I stumbled over that side:
http://www.javaworld.com/jw-01-1997/jw-01-chat.html?page=6
At the very end it says:
An alternative, however, and much more elegant solution, is to abstract the protocol behind a set of stream classes. The specifics of header construction and insertion can be handled automatically by the stream classes, and the client is then left with much the same interface as before: Clients write messages to a stream, but instead of flushing the stream, they call a method that attaches appropriate headers and sends the encapsulated message.
I don't really know what is meant by that. Could somebody explain it, or even better, give me a code example. Perhaps there may are other ways to do?
Let's say you want to send messages encapsulated as the link you sent:
| ID | len | message contents |.
What they mean with "to abstract the protocol behind a set of stream classes" is to create classes that extend stream classes which will put the correct ID and length on the encapsulated message for you.
For example, for an extended PrintWriter where you send two kinds of message:
ID 1 - normal message.
ID 2 - error message.
class MyProtocolPrintWriter extends PrintWriter {
public static final int NORMAL_MESSAGE = 1;
public static final int ERROR_MESSAGE = 2;
//put the constructor
public void writeMessage(String message) {
println(
String.format(
"%02x%02d%s", NORMAL_MESSAGE, message.length(), message));
}
public void writeErrorMessage(String message) {
println(
String.format(
"%02x%02d%s", ERROR_MESSAGE, message.length(), message));
}
}
Here's what one fairly successful chat network used for a protocol.
Internet Relay Chat
And here's a list of the commands that were implemented using the IRC protocol.
List of Internet Relay Chat commands
You would implement these commands as a set of stream classes. The user issues a command, and your stream class handles the specifics of the header construction and insertion into the stream.
Has anyone out there done the work of sitting on top of a packet capture interface (like jpcap) with an implementation of UDPSocket (for UDP datagrams) and InputStream (for TCP streams)?
I suppose it wouldn't be too hard to do given the callback API in jpcap, but has anyone out there already done it? Are there any issues with doing this (do I have to figure out how to reassemble a TCP stream myself, for example?)
I have not done this particular thing, but I do do a lot of work with parsing captured packets in C/C++. I don't know if there exist Java libraries for any of this.
Essentially, you need to work your way up the protocol stack, starting with IP. The pcap data starts with the link-level header, but I don't think there's much in it that you're concerned about, other than ignoring non-IP packets.
The trickiest thing with IP is reassembling fragmented datagrams. This is done using the More Fragments bit in the Flags field and the Fragment Offset field, combined with the Identification field to distinguish fragments from different datagrams Then you use the Protocol field to identify TCP and UDP packets, and the Header Length field to find the start of the corresponding header.
The next step, for both TCP and UDP, is demultiplexing, separating out the various connections in the captured packet stream. Both protocols identify connections (well, UDP doesn't have connections per se, but I don't have a better word handy) by the 4-tuple of the source and destination IP address and the source and destination port, so a connection would be a sequence of packets that matches on all 4 of these values.
Once that's done, for UDP, you're just about finished, unless you want to check the checksum. The Length field in the UDP header tells you how long the packet is; subtract 8 bytes for the header and there's your data.
TCP is somewhat more complicated, as you do indeed have to reassemble the stream, This is done using the sequence number in the header, combined with the length. The sum of these two tells you the next sequence number in the stream. Remember that you're keeping track of the traffic in two directions.
(This is a lot easier than writing an actual TCP implementation, as then you have to implement the Nagle algorithm and other minutiae.)
There's a lot of information on the net about the header formats; google "IP header" for starters. A network analyzer like Wireshark is indispensable for this work, as it will show you how your captured data is supposed to look. Indeed, as Wireshark is open source, you can probably find out a lot by looking at how it does things
Tcp reassembly can be done with JNetPcap. Here is a complete example:
final String SOME_PORT = 8888;
StringBuilder errbuf = new StringBuilder();
Pcap pcap = Pcap.openOffline("/dir/someFile.pcap", errbuf); //Can be replace with .openLive(...)
if (pcap == null) {
System.err.printf("Error: "+errbuf.toString());
return;
}
//Handler that receive Tcp Event one by one
AnalyzerListener<TcpStreamEvent> handler = new AnalyzerListener<TcpStreamEvent>() {
#Override
public void processAnalyzerEvent(TcpStreamEvent evt) {
JPacket packet = evt.getPacket();
Tcp tcp = new Tcp();
if (packet.hasHeader(tcp)) {
//Limiting the analysis to a specific protocol
if (tcp.destination() == SOME_PORT || tcp.source() == SOME_PORT) {
String data = new String(tcp.getPayload());
System.out.println("Capture data:{"+data+"}");
}
}
}
};
TcpAnalyzer tcpAnalyzer = JRegistry.getAnalyzer(TcpAnalyzer.class);
tcpAnalyzer.addTcpStreamListener(handler, null);
//Starting the capture
pcap.loop(Pcap.LOOP_INFINATE, JRegistry.getAnalyzer(JController.class), null);