Develop Reference

raw socket communication between two applications one in c and one in java

I have a application (acting as the server) written in c that is listening on a raw socket, with this socket descriptor: socket(AF_UNIX, SOCK_RAW, 0);. The 0 indicates IPPROTO_IP
I want to write a code in java (acting as the client) to send an ip packet to this raw socket that is listening. Now i read around that java needs a 3rd party library to manage raw sockets. People recommend junixsocket and juds so i'll use one of them i guess.
Junixsocket does something like this to define a raw socket:
File socketFile = new File("/path/to/your/socket");
AFUNIXSocket sock = AFUNIXSocket.newInstance();
sock.connect(new AFUNIXSocketAddress(socketFile));
So the question is:
Is it possible to make these 2 applications communicate with eachother through this raw socket? In java you need to establish the socketfilename and path whereas in c thats not compulsory. My c code does not specify a socketfilename or path so i dont know how to make them communicate on the same socket. How do i make sure they're both sending/receiving on the same raw socket? All this communication is happening only in local!
Thank you

You are using the unix domain sockets (AF_UNIX). They are effectively designed for processes running on the same kernel, and use filesystem pathnames for addressing (see man 4 unix for details). So you have to share a pathname between both processes.
Edit as requested:
To set up the unix domain socket in C app, you need an address of type sockaddr_un, and fill up its sun_path member with a path to your socket:
sockaddr_un sockaddr;
memset(&sockaddr, 0, sizeof(sockaddr));
sockaddr.sun_family = AF_UNIX;
strncpy(sockaddr.sun_path, "/path/to/your/socket", UNIX_MAX_PATH);
and either bind() or connect() your socket to this sockaddr.
However, it has nothing to do with SOCK_RAW. Correct me if I am wrong, but you want a raw socket over the Internet, that is socket(AF_INET, SOCK_RAW, 0).

Socket communication between Java and C# application

I have 02 applications, one is written in java and the another one is written in C#. I need to exchange data (Strings) between them thus I use a socket connection.
Currently the C# application acts as server and java application acts as client. The C# application will continuously listen on port "7777" and the Java application will make a socket to "localhost:7777" as start up and use that socket to send data to C# server. I don't close the connection between them since I need to exchange data frequently.
Everything is great until my C# server application sometimes needs to send some Strings to the Java client application. I think about using another port to make my java application becomes a server and the C# application will also become a client. Yet I don't feel that it is a good solution.
P/S: Yes, socket is Bi-direction, in my current application I can send string data from my C# application back to Java application, but only when the Java application first sent a string to the C# application and now it is waiting for the respond data from the C# application.
Thus I want to ask if there is another better solution ? Is it possible if we only use 1 port in total ? Thank you very much.

Your socket works both ways (bi-directional), so you won't need to create another. You can just get the output stream and input stream of the socket, on both sides of the connection, and use those for sending/receiving.
To send a string, the easiest way would probably be to use a DataOutputStream as you can write UTF-8 strings with a simple function.
Java side:
String blah = "hey";
DataOutputStream dataOs = new DataOutputStream(socket.getOutputStream());
dataOs.writeUTF(blah);
The C# side is slightly trickier as you need to account for the fact that the first two bytes sent from the java function writeUTF() will actually be the length (in bytes) of the string that follows. You can grab those 2 bytes first and then throw them into an int, shifting the bits as you go along. Then you can use that int as the size of the buffer when you request the string from the socket. Should look something like this.
C# side:
int length = 0;
byte[] sizeBuffer = new byte[2];
socket.Receive(sizeBuffer);
for (int i = 0; i < sizeBuffer.Length; i += 1)
{
length = length << 8;
length += sizeBuffer[i];
}
byte[] stringBuffer = new byte[length];
socket.Receive(stringBuffer);
string myString = Encoding.UTF8.GetString(stringBuffer, 0, stringBuffer.Length);

How to read ganglia information from other application?

I have managed to install and configure Ganglia on my cluster. I do not want to just see all performance data on ganglia web interface but instead I want to read cluster information from other application (application may be Java or Python based). I am not able to find if it is possible or not.
Is there any API to read Ganglia data?
To test Ganglia I used telnet master 8649 and Ganglia showed me nice XML text on my console. But how do I do the same thing using Java or Python? I can definitely connect to 8649 using sockets but after that do I need to send something to Ganglia daemons?

I can help you to get an insight on this. But before that I must tell you, I am not a Java programmer, rather I am a C/C++ programmer. So, it means I can let you know, how things work in ganglia and you can find equivalent methods in Java/ Python to rewrite the codes you want.
Please be informed that there is no API in ganglia to achieve what you want to.
First consider below set up of ganglia to understand properly:
GS1 and GS2 are collecting system metrics and pushing them to GM.
So, according to your question, if you want to collect all such metrics by your own Java/ Python based application, then you may have to install the application on the Master server (i.e Replace GS with your own application).
GS1 and GS2 sends all collected metrics either by UDP unicast channel or UDP multicast channel. It is recommended that UDP unicast should be enabled in every gmond.conf for easier scalability.
I wouldn't discuss much on GS1 and GS2 as your question is more about replacing GM with your own tool.
GM uses two important libraries heavily to establish a UDP connection and translate data into its own readable format. They are APR (Apache Portable Runtime) to establish UDP connection and perform related activities and XDR (External Data Representation) to send data across networks and perform RPC.
You need to find APR and XDR equivalent libraries in Java and Python first. XDR is already available in Java and APR could be replaced by your own basic implementation to perform inter-network operations (i.e., create UDP socket, etc).
Open gmond.c source file of ganglia and go to line 1436. You will find a C function:
static void process_udp_recv_channel(const apr_pollfd_t *desc, apr_time_t now).
This function basically performs "UDP connection establishment" and "data translation into readable format" activities.
The call flow of the above function is shown below:
Now, let's expand the function at line 1436 to understand more.
The first argument in this function carries network parameters such as IP, Port, etc. The structure is expanded below. You can find similar object in Java also.
struct apr_pollfd_t {
apr_pool_t *p; /**< associated pool */
apr_datatype_e desc_type; /**< descriptor type */
apr_int16_t reqevents; /**< requested events */
apr_int16_t rtnevents; /**< returned events */
apr_descriptor desc; /**< #see apr_descriptor */
void *client_data; /**< allows app to associate context */
};
The second parameter has nothing to do, if SFLOW is disabled.
So, Start with creating a APR pool, UDP connection, etc.
socket = desc->desc.s;
channel = desc->client_data;
apr_pool_create(&p, global_context);
status = apr_socket_addr_get(&remotesa, APR_LOCAL, socket);
status = apr_sockaddr_info_get(&remotesa, NULL, remotesa->family, remotesa->port, 0, p);
/* Grab the data */
status = apr_socket_recvfrom(remotesa, socket, 0, buf, &len);
if(status != APR_SUCCESS)
{
apr_pool_destroy(p);
return;
}
apr_sockaddr_ip_buffer_get(remoteip, 256, remotesa);
/* Check the ACL */
if(Ganglia_acl_action( channel->acl, remotesa) != GANGLIA_ACCESS_ALLOW)
{
apr_pool_destroy(p);
return;
}
All declarations of variable can be found in the beginning of the function expanded (line 1439 to 1456).
Then, create XDR stream:
xdrmem_create(&x, buf, max_udp_message_len, XDR_DECODE);
Flush the data of the struct which saves metadata and metrics value:
memset( &fmsg, 0, sizeof(Ganglia_metadata_msg));
memset( &vmsg, 0, sizeof(Ganglia_value_msg));
fmsg (Ganglia_metadata_msg) and vmsg (Ganglia_value_msg) struct definitions can be found in gm_protocol.h header file. Re-write them in Java.
Then, figure out if the message received is "metadata" or "metrics values".
xdr_Ganglia_msg_formats(&x, &id); // this function is located in the source file gm_protocol_xdr.c and this file is generated by rpcgen.
Note: rpcgen is a rpc compiler and its explanation can be found in this question.
Note: Here is the link for gm_protocol_xdr.c.
Here, id is an enum and its declaration is shown below:
enum Ganglia_msg_formats {
gmetadata_full = 128,
gmetric_ushort = 128 + 1,
gmetric_short = 128 + 2,
gmetric_int = 128 + 3,
gmetric_uint = 128 + 4,
gmetric_string = 128 + 5,
gmetric_float = 128 + 6,
gmetric_double = 128 + 7,
gmetadata_request = 128 + 8,
};
typedef enum Ganglia_msg_formats Ganglia_msg_formats;
Based on the value of id, you can determine what kind of value the packets have.
For this purpose, this function calls an another function (which is in fact generated by rpcgen) to determine the kind of value the packet has and if found, it translates it to human readable format too.
The function is:
xdr_Ganglia_value_msg(&x, &vmsg);
You can find the full expansion of this function in gm_protocol_xdr.c from line 275.
After that you can do whatever you would like with these packets.
In the end, you must free all XDR variables and APR pools allocated.
I hope this gives you a fair idea to start with your own application.

Telnet session vs TCP session

What's the difference between a TELNET and TCP session?
I have a simple server:
public static void main(String[] args) throws IOException {
ServerSocket ss = new ServerSocket(5555);
Socket socket = ss.accept();
System.out.println("Waiting for a client...");
InputStream sin = socket.getInputStream();
OutputStream sout = socket.getOutputStream();
DataOutputStream out = new DataOutputStream(sout);
out.writeUTF("\u001B[2J");
out.writeUTF("Hello client\r\n");
while (true) {
int b = sin.read();
sout.write(b);
}
}
I'm connecting via telnet and typing like this:
So I assume that packet sending by telnet client has no specific headers and pass to low level protocol (TCP in my case) completely without changing. Is it correct?

From http://en.wikipedia.org/wiki/Telnet#Telnet_data:
All data octets except 0377 are transmitted over the TCP transport as is. Therefore, a Telnet client application may also be used to establish an interactive raw TCP session, and it is commonly believed that such session which does not use the IAC (0377 character, or 255 in decimal) is functionally identical.[citation needed] This is not the case, however, because there are other network virtual terminal (NVT) rules, such as the requirement for a bare carriage return character (CR, ASCII 13) to be followed by a NULL (ASCII 0) character, that distinguish the telnet protocol from raw TCP sessions.[clarification needed] On the other hand, many systems now possess true raw TCP clients, such as netcat or socat on UNIX and PuTTY on Windows, which also can be used to manually "talk" to other services without specialized client software. Nevertheless, Telnet is still sometimes used in debugging network services such as SMTP, IRC, HTTP, FTP or POP3 servers, to issue commands to a server and examine the responses, but of all these protocols only FTP really uses Telnet data format.
Another difference of Telnet from a raw TCP session is that Telnet is not 8-bit clean by default. 8-bit mode may be negotiated, but high-bit-set octets may be garbled until this mode was requested, and it obviously will not be requested in non-Telnet connection. The 8-bit mode (so named binary option) is intended to transmit binary data, not characters though. The standard suggests the interpretation of codes 0000–0176 as ASCII, but does not offer any meaning for high-bit-set data octets. There was an attempt to introduce a switchable character encoding support like HTTP has,[3] but nothing is known about its actual software support.
I would type some more but really Wikipedia covered it nicely :)

Telnet is a name of services. and TCP is protocol to connect computer together with reliable, ordered, error-checked delivery of a stream.

Possible to sit on the network and receive a TCP stream/UDP datagrams?

Has anyone out there done the work of sitting on top of a packet capture interface (like jpcap) with an implementation of UDPSocket (for UDP datagrams) and InputStream (for TCP streams)?
I suppose it wouldn't be too hard to do given the callback API in jpcap, but has anyone out there already done it? Are there any issues with doing this (do I have to figure out how to reassemble a TCP stream myself, for example?)

I have not done this particular thing, but I do do a lot of work with parsing captured packets in C/C++. I don't know if there exist Java libraries for any of this.
Essentially, you need to work your way up the protocol stack, starting with IP. The pcap data starts with the link-level header, but I don't think there's much in it that you're concerned about, other than ignoring non-IP packets.
The trickiest thing with IP is reassembling fragmented datagrams. This is done using the More Fragments bit in the Flags field and the Fragment Offset field, combined with the Identification field to distinguish fragments from different datagrams Then you use the Protocol field to identify TCP and UDP packets, and the Header Length field to find the start of the corresponding header.
The next step, for both TCP and UDP, is demultiplexing, separating out the various connections in the captured packet stream. Both protocols identify connections (well, UDP doesn't have connections per se, but I don't have a better word handy) by the 4-tuple of the source and destination IP address and the source and destination port, so a connection would be a sequence of packets that matches on all 4 of these values.
Once that's done, for UDP, you're just about finished, unless you want to check the checksum. The Length field in the UDP header tells you how long the packet is; subtract 8 bytes for the header and there's your data.
TCP is somewhat more complicated, as you do indeed have to reassemble the stream, This is done using the sequence number in the header, combined with the length. The sum of these two tells you the next sequence number in the stream. Remember that you're keeping track of the traffic in two directions.
(This is a lot easier than writing an actual TCP implementation, as then you have to implement the Nagle algorithm and other minutiae.)
There's a lot of information on the net about the header formats; google "IP header" for starters. A network analyzer like Wireshark is indispensable for this work, as it will show you how your captured data is supposed to look. Indeed, as Wireshark is open source, you can probably find out a lot by looking at how it does things

Tcp reassembly can be done with JNetPcap. Here is a complete example:
final String SOME_PORT = 8888;
StringBuilder errbuf = new StringBuilder();
Pcap pcap = Pcap.openOffline("/dir/someFile.pcap", errbuf); //Can be replace with .openLive(...)
if (pcap == null) {
System.err.printf("Error: "+errbuf.toString());
return;
}
//Handler that receive Tcp Event one by one
AnalyzerListener<TcpStreamEvent> handler = new AnalyzerListener<TcpStreamEvent>() {
#Override
public void processAnalyzerEvent(TcpStreamEvent evt) {
JPacket packet = evt.getPacket();
Tcp tcp = new Tcp();
if (packet.hasHeader(tcp)) {
//Limiting the analysis to a specific protocol
if (tcp.destination() == SOME_PORT || tcp.source() == SOME_PORT) {
String data = new String(tcp.getPayload());
System.out.println("Capture data:{"+data+"}");
}
}
}
};
TcpAnalyzer tcpAnalyzer = JRegistry.getAnalyzer(TcpAnalyzer.class);
tcpAnalyzer.addTcpStreamListener(handler, null);
//Starting the capture
pcap.loop(Pcap.LOOP_INFINATE, JRegistry.getAnalyzer(JController.class), null);