Develop Reference

How do I send categorized and grouped logs into Stackdriver using the Java Client Library?

I'd like to aggregate multiple log.info log.warning and log.error calls, and possibly stack traces, into a single Stackdriver log line generated by the server interacting with my application code. The goal is to summarize a request handled by my Scala server, and then group as many logging statements as occurred during its execution, with any errors.
This is default behavior on GAE logging, but because I'm new to reading Java API's, I'm having trouble figuring out how to:
1/ create a custom MonitoredResource (?) representing, e.g., "API server", then specifying category within it (e.g. "production"). Specifically, do I have to create these via the REST API, even though I'm only doing it once for my deployment? Can I use something like Troposphere to define these in code and commit them in a repo?
2/ how the nouns MonitoredResource, MonitoredResourceDescriptor, LogEntry, LogEntryOperation and logName fit together, and where the categories "API Server" and "production" get defined, as well as logging statement groups like GET /foobar -> 200 response + 1834 bytes can be added (are those logNames?).
No need to write code for me, of course, but pointers and a high level overview to save me trial and error would be appreciated greatly.

You can group together multiple log entries for the same operation by using the LogEntryOperation field in the LogEntry (https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogEntryOperation).
In the Logs Viewer, you can group the log entries by filtering on the operation.id field using the advanced filters.
In the Java client library, you can set the Operation Id using https://googlecloudplatform.github.io/google-cloud-java/0.33.0/apidocs/com/google/cloud/logging/LogEntry.Builder.html#setOperation-com.google.cloud.logging.Operation-
1) The Monitored Resources you can use is a curated set defined by Google. You cannot define your own type. The supported resources are listed in https://cloud.google.com/logging/docs/api/v2/resource-list.
2) Basic concepts are described in https://cloud.google.com/logging/docs/basic-concepts.

1) The MonitoredResource monitors what you configure in MonitoredResourceDescriptor
I assume you can create it anyway you want (REST API or with the client libraries).
2) I am unsure where you want to describe "API Server" or "Production", the MonitoredResourceDescriptor is how to setup what to monitor. The LogEntry is the actual logs and the LogName is just a label you give this particular log. What you described should be something that a logEntry returns (the 200 code + other stuff).
I might be a bit confused at what you are asking. The best thing to do is to create a sample MonitoredResource and see how it works.

Entity & Entity Properties. Database design for effective searching

Last two days i've been searching suitable solution for the problem described below.
In my standalone notification-service module I have an abstract Message entity. Message has 'to', 'from', 'sentAt', 'receivedAt' and other attributes. The responsibility of the notification-service is to:
send new messages using different registered message providers (SMS, EMAIL, Skype , etc).
receive new messages from registered message providers
update status for already sent messages.
Notification-service module is developed as standalone module that is available by SOAP protocol. A lot of clients can use this module to send or searching through already received messages.
Clients want to attach some properties (~ smth like tags) while sending messages for further searching messages by these properties. These properties make a sense only in client's environment.
For example, Client A might want to send message and save following custom properties :
1. Internal system id of user whom system sends message
2. Distinguish flag (whether id related to users / admins or clients)
3. Notification flag (notification/alert/ ...)
Client B might want to send message and save another set of custom properties :
1. Internal system operator id (who sends sms)
2. Template id that was used to send message
Custom properties can be used by the clients to search already sent messages.
For example:
Client A could find SMS messages sent to administrator users in period between [Date 1; Date 2] that have 'alert' status.
Client B could find all notification sent by specified template.
Of course, data should be fetched page by page.
At first I created the following database model:
Database scheme
To find all messages with specified properties I tried to use query:
SELECT * FROM (SELECT message_id FROM custom_message_properties
WHERE CONCAT(CONCAT(key, ':'), value) IN ('property1:value1', 'property2:value2')
GROUP BY message_id having(count(*)) = 2)
as cmp JOIN message m ON cmp.message_id = m.id ORDER BY ID LIMIT 100 OFFSET 0
Query worked fine (although it seems me not very good) in database with small data. I decided to check results for ~ real awaited data .
So i generated 10 000 000 messages that have 40 000 000 custom properties and checked result. Execution time was ~ 2 minutes. The most time consumed operation was following sub-select:
SELECT message_id FROM custom_message_properties
WHERE CONCAT(CONCAT(key, ':'), value) IN ('property1:value1', 'property2:value2')
I understand that string comparison is very slow cause database index feature is not used. I decided to change database structure to merge 'key' and 'value' columns into single one. So i updated by database scheme :
Updated database scheme
I checked result again. Now execution time was ~20 seconds. It's much better but still is not suitable for production use.
So now I have no idea how to improve performance without significant changes in application architecture design.
The only one thought i have is to create separate table for each client with required client properties.
client(i)_custom_properties {
mid bigint, // foreign key references message (id)
p1 type1,
p2 type2,
......
pn type(n)
}
I have spent a lot of time while trying to find any useful information. I have also analyzed 'stackoverflow' database cause it seemed me that it should be quite the same. But in 'stackoverflow' there are ~ 50 000 different tags. Not so much that my database could have.
Any help is appreciated. Thanks, in advance!
Project environment that i use :
Postgres database (9.6)
Java 1.8
Spring modules (spring-boot, spring-data-jpa + hibernate, spring-ws, etc).

I have not found any suitable solution except creating additional table with client's properties for each client.
I know, that solution is not so flexible,
but now search query time is less than 1 second.
In future, I will try to solve the same problem using noSQL data storage.

How to read ganglia information from other application?

I have managed to install and configure Ganglia on my cluster. I do not want to just see all performance data on ganglia web interface but instead I want to read cluster information from other application (application may be Java or Python based). I am not able to find if it is possible or not.
Is there any API to read Ganglia data?
To test Ganglia I used telnet master 8649 and Ganglia showed me nice XML text on my console. But how do I do the same thing using Java or Python? I can definitely connect to 8649 using sockets but after that do I need to send something to Ganglia daemons?

I can help you to get an insight on this. But before that I must tell you, I am not a Java programmer, rather I am a C/C++ programmer. So, it means I can let you know, how things work in ganglia and you can find equivalent methods in Java/ Python to rewrite the codes you want.
Please be informed that there is no API in ganglia to achieve what you want to.
First consider below set up of ganglia to understand properly:
GS1 and GS2 are collecting system metrics and pushing them to GM.
So, according to your question, if you want to collect all such metrics by your own Java/ Python based application, then you may have to install the application on the Master server (i.e Replace GS with your own application).
GS1 and GS2 sends all collected metrics either by UDP unicast channel or UDP multicast channel. It is recommended that UDP unicast should be enabled in every gmond.conf for easier scalability.
I wouldn't discuss much on GS1 and GS2 as your question is more about replacing GM with your own tool.
GM uses two important libraries heavily to establish a UDP connection and translate data into its own readable format. They are APR (Apache Portable Runtime) to establish UDP connection and perform related activities and XDR (External Data Representation) to send data across networks and perform RPC.
You need to find APR and XDR equivalent libraries in Java and Python first. XDR is already available in Java and APR could be replaced by your own basic implementation to perform inter-network operations (i.e., create UDP socket, etc).
Open gmond.c source file of ganglia and go to line 1436. You will find a C function:
static void process_udp_recv_channel(const apr_pollfd_t *desc, apr_time_t now).
This function basically performs "UDP connection establishment" and "data translation into readable format" activities.
The call flow of the above function is shown below:
Now, let's expand the function at line 1436 to understand more.
The first argument in this function carries network parameters such as IP, Port, etc. The structure is expanded below. You can find similar object in Java also.
struct apr_pollfd_t {
apr_pool_t *p; /**< associated pool */
apr_datatype_e desc_type; /**< descriptor type */
apr_int16_t reqevents; /**< requested events */
apr_int16_t rtnevents; /**< returned events */
apr_descriptor desc; /**< #see apr_descriptor */
void *client_data; /**< allows app to associate context */
};
The second parameter has nothing to do, if SFLOW is disabled.
So, Start with creating a APR pool, UDP connection, etc.
socket = desc->desc.s;
channel = desc->client_data;
apr_pool_create(&p, global_context);
status = apr_socket_addr_get(&remotesa, APR_LOCAL, socket);
status = apr_sockaddr_info_get(&remotesa, NULL, remotesa->family, remotesa->port, 0, p);
/* Grab the data */
status = apr_socket_recvfrom(remotesa, socket, 0, buf, &len);
if(status != APR_SUCCESS)
{
apr_pool_destroy(p);
return;
}
apr_sockaddr_ip_buffer_get(remoteip, 256, remotesa);
/* Check the ACL */
if(Ganglia_acl_action( channel->acl, remotesa) != GANGLIA_ACCESS_ALLOW)
{
apr_pool_destroy(p);
return;
}
All declarations of variable can be found in the beginning of the function expanded (line 1439 to 1456).
Then, create XDR stream:
xdrmem_create(&x, buf, max_udp_message_len, XDR_DECODE);
Flush the data of the struct which saves metadata and metrics value:
memset( &fmsg, 0, sizeof(Ganglia_metadata_msg));
memset( &vmsg, 0, sizeof(Ganglia_value_msg));
fmsg (Ganglia_metadata_msg) and vmsg (Ganglia_value_msg) struct definitions can be found in gm_protocol.h header file. Re-write them in Java.
Then, figure out if the message received is "metadata" or "metrics values".
xdr_Ganglia_msg_formats(&x, &id); // this function is located in the source file gm_protocol_xdr.c and this file is generated by rpcgen.
Note: rpcgen is a rpc compiler and its explanation can be found in this question.
Note: Here is the link for gm_protocol_xdr.c.
Here, id is an enum and its declaration is shown below:
enum Ganglia_msg_formats {
gmetadata_full = 128,
gmetric_ushort = 128 + 1,
gmetric_short = 128 + 2,
gmetric_int = 128 + 3,
gmetric_uint = 128 + 4,
gmetric_string = 128 + 5,
gmetric_float = 128 + 6,
gmetric_double = 128 + 7,
gmetadata_request = 128 + 8,
};
typedef enum Ganglia_msg_formats Ganglia_msg_formats;
Based on the value of id, you can determine what kind of value the packets have.
For this purpose, this function calls an another function (which is in fact generated by rpcgen) to determine the kind of value the packet has and if found, it translates it to human readable format too.
The function is:
xdr_Ganglia_value_msg(&x, &vmsg);
You can find the full expansion of this function in gm_protocol_xdr.c from line 275.
After that you can do whatever you would like with these packets.
In the end, you must free all XDR variables and APR pools allocated.
I hope this gives you a fair idea to start with your own application.

Query a remote server using QuickFix/J in Java to get position of an instrument

I'm building a client for trading with a remote server using FIX protocol and QuickFix/J API.
I can send order, receive price updates, cancel orders etc...
I'm asked now to "query API for current position of an instrument".
So let's say I can submit an order for buying an instrument, and it doesn't get executed, I would like to receive from the server some information like "you are LONG on intrument X with quantity Y etc".
Is it possible using QuickFix/J API?
I have written a method like this
static void positionReport() throws SessionNotFound{
quickfix.fix50.PositionReport order = new quickfix.fix50.PositionReport();
SessionID sessionId = (SessionID) initiator.getSessions().get(0);
order.set(new Account("1005390"));
order.set(new SecurityID("4663789"));
order.set(new SecurityExchange("XETR"));
order.set(new Symbol("SAP"));
Session.sendToTarget(order, sessionId);
}
which sends FIX messages like this
8=FIX.4.29=9835=AP34=4949=HIQ6_ORDER52=20140324-
15:54:10.14256=HIQFIX1=100539048=466378955=SAP207=XETR10=199
and receives messages like this:
8=FIX.4.29=9935=334=6949=HIQFIX52=20140324-15:54:10.89156=HIQ6_ORDER45=4958=Invalid
MsgType372=AP373=1110=242
As you can see I get "Invalid message" error

Check your counterparty's documentation.
FIX is a fairly "dumb" protocol. It just provides a communication infrastructure. The default message definitions are best thought of as a list of suggested messages that you can use. Even if one message type is supported by two counterparties, it's possible that each of the two counterparties could use it in totally different ways.
Most connection providers only use a subset of these messages. You should check their documentation to see if they support the PositionRequest message, and to see how they want you to set the fields in it.

No you cannot do that using Quickfix, unless and until the counterparty is modelled to give you FIX acknowledgements to your specific liking. That is why you can add your customized FIX fields to the FIX XML config file.
373 tag says 11 -> 11 = Invalid MsgType
58 confirms it for you again.
Check your FIX XML config and check if your message is complete and if your counterparty allows the messages of type AP.

How could I identify multiple commands for network application (Java)

I want to build a client-server-application for some practice. I started off with a simple chat which was not so hard to do. I'm also able to identify different commands by just simply split a String, e.g. "command:msg". But I think that may be a little inconvenient. So I'm wondering if there are better ways to realize that. And I stumbled over that side:
http://www.javaworld.com/jw-01-1997/jw-01-chat.html?page=6
At the very end it says:
An alternative, however, and much more elegant solution, is to abstract the protocol behind a set of stream classes. The specifics of header construction and insertion can be handled automatically by the stream classes, and the client is then left with much the same interface as before: Clients write messages to a stream, but instead of flushing the stream, they call a method that attaches appropriate headers and sends the encapsulated message.
I don't really know what is meant by that. Could somebody explain it, or even better, give me a code example. Perhaps there may are other ways to do?

Let's say you want to send messages encapsulated as the link you sent:
| ID | len | message contents |.
What they mean with "to abstract the protocol behind a set of stream classes" is to create classes that extend stream classes which will put the correct ID and length on the encapsulated message for you.
For example, for an extended PrintWriter where you send two kinds of message:
ID 1 - normal message.
ID 2 - error message.
class MyProtocolPrintWriter extends PrintWriter {
public static final int NORMAL_MESSAGE = 1;
public static final int ERROR_MESSAGE = 2;
//put the constructor
public void writeMessage(String message) {
println(
String.format(
"%02x%02d%s", NORMAL_MESSAGE, message.length(), message));
}
public void writeErrorMessage(String message) {
println(
String.format(
"%02x%02d%s", ERROR_MESSAGE, message.length(), message));
}
}

Here's what one fairly successful chat network used for a protocol.
Internet Relay Chat
And here's a list of the commands that were implemented using the IRC protocol.
List of Internet Relay Chat commands
You would implement these commands as a set of stream classes. The user issues a command, and your stream class handles the specifics of the header construction and insertion into the stream.