Develop Reference

KStream-KStream leftjoin doesn't produce any value when the second matching value is not present

We have theses POJOs:
#Data
#NoArgsConstructure
#AllArgsConstructure
class MyPost {
private String content;
private SeenInfo seenInfo;
}
#Data
#NoArgsConstructure
#AllArgsConstructure
class SeenInfo {
private Integer seenCount;
//other fields...
}
and this left-join process in our application:
#Bean
public Function<KStream<String, MyPost>, Function<KStream<String, SeenInfo>, KStream<String, MyPost>>> joinProcess(Map<String, String> schemaConfig) {
return postStream ->
seenInfoStream -> {
SpecificAvroSerde<MyPost> postSerde = new SpecificAvroSerde<>();
SpecificAvroSerde<SeenInfo> seenInfoSerde = new SpecificAvroSerde<>();
postSerde.configure(schemaConfig, true);
seenInfoSerde.configure(schemaConfig, true);
return postStream.leftJoin(seenInfoStream,
(p, s) -> {
p.setSeenInfo(s);
return p;
},
JoinWindows.of(Duration.ofMinutes(5)),
StreamJoined.with(Serdes.String(),
postSerde,
seenInfoSerde));
};
}
Problem One:
When MyPost and SeenInfo matching values are present within 5 minutes, the join process produces two messages:
Message1: MyPost={ "content": "some text", "seenInfo": null}
Message2: MyPost={ "content": "some text", "seenInfo": { "seenCount": 1, ...}}
Problem Two:
If MyPost is present and SeenInfo is not, the join process will not return any data.
We Expect: Message: MyPost={ "content": "some text", "seenInfo": null}
What should we do to solve this problem?

Seems you are using an older version of Kafka Streams, that faces the first issue about "spurious left join results". It's fixed in Kafka 3.1 release (cf https://issues.apache.org/jira/browse/KAFKA-10847).
Compare https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Deprecate+24-hour+Default+Grace+Period+for+Windowed+Operations+in+Streams
For the second issue, yes, you should get an output -- if you use version 3.1, the only explanation for a missing output would be that your input streams "stopped/pause" and no new data arrives -- if there is no new data that join window won't close the the left-join result is "stuck" until time advances (based on record timestamps and thus required new input data).

How to exclude some fields when listing Grails domain?

I'm wondering how can I list Grails domain and exclude some fields at same time. I'm guessing solution must be simple but I just can not see it.
I prepared some example with domain User:
class User implements Serializable {
String username
String email
Date lastUpdated
String password
Integer status
static constraints = { }
static mapping = { }
}
At this point I want to list all users which have status below 2.
render User.findAllByStatusLessThen(2) as JSON
I want to render JSON response to clientside without some fields. For example I just want to render users with fields username and lastUpdated so rendered JSON would look like this:
[{"username": "user1", "lastUpdated":"2016-09-21 06:49:46"}, {"username": "user2", "lastUpdated":"2016-09-22 11:24:42"}]
What's the easiest way to achieve that?

Yeah.It's simple.Try below solutions
Solution 1
List userList = User.where{ status < 2 }.property("username").property("lastUpdated").list()
render userList as JSON
output
[{"user1", "2016-09-21 06:49:46"}, {"user2", "2016-09-22 11:24:42"}]
Solution 2 - using this you will get output in the Key-Value pair
List userList = User.findAllByStatusLessThen(2)?.collect{
[username : it.username, lastUpdated: it.lastUpdated]}
render userList as JSON
output
[{"username": "user1", "lastUpdated":"2016-09-21 06:49:46"}, {"username": "user2", "lastUpdated":"2016-09-22 11:24:42"}]

You are looking for Grails projections.
def result = Person.createCriteria().list {
lt("status", 2)
projections {
property('username')
property('lastUpdated')
}
} as JSON

Well if you want the result to be in key-value pair you can take advantage of HQL query
def query = """select new map(u.username as username, u.lastUpdated as lastUpdated) from User u where status < 2"""
def result = User.executeQuery(query)
println (result as JSON)
This will give you the output as below
[{"username": "user1", "lastUpdated":"2016-09-21 06:49:46"}, {"username": "user2", "lastUpdated":"2016-09-22 11:24:42"}]

Get all videos of a YouTube Playlist in Java

Context I'm working with Android Studio (Java). I want to obtain all the videos of a given playlist (or 50, I will get all the other after).
Problem I see people using url like
https://www.googleapis.com/youtube/v3/playlistItems?part=snippet&maxResults=50&playlistId=PLiEwZfNgb4fVrRzTonlVEMj6DB2Nmzg2M&key=AIzaSyC2_YRcTE9916fsmA0_KRnef43GbLzz8m0
but I don't know how to implement this in Java. I follow some tuto and I got a way to get information totally different like :
YouTube.Search.List query;
query = youtube.search().list("id,snippet");
query.setKey(MY_API_KEY);
query.setMaxResults((long)20);
query.setType("video");
query.setFields("items(id/videoId,snippet/title,snippet/description,snippet/thumbnails/default/url)");
And I really don't understand how do I get something else than a search.
Most of documentation is in english only...
EDIT
Ok, so I continued to try, I think I got a near solution, but I got an error.
private YouTube youtube;
private YouTube.PlaylistItems.List playlistItemRequest;
private String PLAYLIST_ID = "PLiEwZfNgb4fVrRzTonlVEMj6DB2Nmzg2M";
public static final String KEY = "AIzaSyC2_YRcTE9916fsmA0_KRnef43GbLzz8m0";
// Constructor
public YoutubeConnector(Context context)
{
youtube = new YouTube.Builder(new NetHttpTransport(),
new JacksonFactory(), new HttpRequestInitializer()
{
#Override
public void initialize(HttpRequest hr) throws IOException {}
}).setApplicationName(context.getString(R.string.app_name)).build();
}
public List<VideoItem> result()
{
List<PlaylistItem> playlistItemList = new ArrayList<PlaylistItem>();
try
{
/* HERE MUST BE MY PROBLEM ! */
playlistItemRequest = youtube.playlistItems().list("snippet");
playlistItemRequest.setPlaylistId(PLAYLIST_ID);
playlistItemRequest.setFields("items(id/videoId,snippet/title,snippet/description,snippet/thumbnails/default/url),nextPageToken,pageInfo");
playlistItemRequest.setKey(KEY);
String nextToken = "";
do {
playlistItemRequest.setPageToken(nextToken);
PlaylistItemListResponse playlistItemResult = playlistItemRequest.execute();
playlistItemList.addAll(playlistItemResult.getItems());
nextToken = playlistItemResult.getNextPageToken();
} while (nextToken != null);
}catch(IOException e)
{
Log.d("YC", "Could not initialize: "+e);
}
//[...]
}
Here is the error I got :
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"location" : "fields",
"locationType" : "parameter",
"message" : "Invalid field selection videoId",
"reason" : "invalidParameter"
} ],
"message" : "Invalid field selection videoId"
}
EDIT 2 Thanks to : Martijn Woudstra.
Correct line was :
playlistItemRequest = youtube.playlistItems().list("snippet,contentDetails");
//[...]
playlistItemRequest.setFields("items(snippet/title,snippet/description,snippet/thumbnails/default/url,contentDetails/videoId),nextPageToken,pageInfo");
//[...]
videoItem.setId(item.getContentDetails().getVideoId());

I know that is an old question but It is important to identify which resources are we using to understand how to get the proper information. There are many resources in the YouTube API v3, but we usually use the search, video, playlist and playlistItems.
According to the documentation the following JSON structure shows the format of a playlistItems resource:
{
"kind": "youtube#playlistItem",
"etag": etag,
"id": string,
"snippet": {
"publishedAt": datetime,
"channelId": string,
"title": string,
"description": string,
"thumbnails": {
(key): {
"url": string,
"width": unsigned integer,
"height": unsigned integer
}
},
"channelTitle": string,
"playlistId": string,
"position": unsigned integer,
"resourceId": {
"kind": string,
"videoId": string,
}
},
"contentDetails": {
"videoId": string,
"startAt": string,
"endAt": string,
"note": string,
"videoPublishedAt": datetime
},
"status": {
"privacyStatus": string
}
}
From this structure, we may suppose that there are three ways to get the videoId. But first it is important to know how we going to define the PARTS and the FIELDS of the resource.
To define the PARTS we use this code:
YouTube.PlaylistItems.List list = youtube.playlistItems().list("snippet");
In the previous line, "snippet" identifies a property that contains numerous fields (or child properties), including the title, description, position, and resourceId, so when we set "snippet" the API's response will contain all of those child properties.
Now, we also can limit the previous properties if we define the FIELDS. For example, in this code:
list.setFields("items(id/videoId,snippet/title,snippet/description," +
"snippet/thumbnails/default/url)");
If we call list.execute(), it will show an error because we didn't define id in the PARTS properties. Also, according to the JSON structure, id is a String and does not contains videoId as a child property. Ah!, but we can extract videoId from the resourceId? -Well, the answers is YES/NO. -Why so? Come on Teo, the JSON structure shows it clearly. -Yes, I can see that, but the documentation says:
If the snippet.resourceId.kind property's value is youtube#video, then this property will be present and its value will contain the ID that YouTube uses to uniquely identify the video in the playlist.
This means that sometimes may not be available. -Then, how we can get the videoId? -Well, we can add id or contentDetails to the PARTS resources. If we add id then defines fields like this:
YouTube.PlaylistItems.List list = youtube.playlistItems().list("id,snippet");
list.setFields("items(id,snippet/title,snippet/description," +
"snippet/thumbnails/default/url)");
If we add contentDetails then defines fields like this:
YouTube.PlaylistItems.List list = youtube.playlistItems()
.list("snippet,contentDetails");
list.setFields("items(contentDetails/videoId,snippet/title,snippet/description," +
"snippet/thumbnails/default/url)");
I hope this helps you guys.

id/videoId doesnt exist.
There is an id and a snippet/resourceId/videoId.
So my guess is your setfields aren't right.

How to I decode the headers of a message using Gmail-API using Java?

I am working on a fragment in android studio whose purpose is to display To/From, Subject, and the body of the message. So far, I am able to retrieve, decode, and display the body. I tried using a similar method for the headers but for some reason it isn't decoding properly, or my method calls aren't getting the correct information. Here is the code I am working with:
String user = "me";
String query = "in:inbox is:unread";
textView.setText("Inbox");
ListMessagesResponse messageResponse =
mService.users().messages().list(user).setQ(query).setMaxResults(Long.valueOf(1)).execute();
List<Message> messages = messageResponse.getMessages();
for(Message message : messages){
Message message2 = mService.users().messages().get(user, message.getId()).execute();
//Get Headers
byte[] headerBytes = Base64.decodeBase64(message2.getPayload().getParts().get(0).getHeaders().get(0).getName().toString().trim()); // get headers
String header = new String(headerBytes, "UTF-8");
//Get Body
byte[] bodyBytes = Base64.decodeBase64(message2.getPayload().getParts().get(0).getBody().getData().trim().toString()); // get body
String body = new String(bodyBytes, "UTF-8");
messageList.add(header);
messageList.add(body);
}
return messageList;
The section under // get body works. But the section under //Get Headers returns data with weird symbols which include black diamonds with white question marks inside and letters in random order. I have tried many different combinations and orders for the method calls in the Base64.decodeBase64 statement for headerBytes but wasn't able to succeed. Is there something I am missing?
Edit: I looked at the gmail-api documentation on the google developers site and I still am confused on how the header information is stored and how to retrieve specific things such as To, From, and Subject. That might be my problem since I may not be targeting the correct data.

If I list messages and get the first one, we can see what the message looks like:
Request
format = metadata
metadataHeaders = From,To,Subject
fields = payload/headers
GET https://www.googleapis.com/gmail/v1/users/me/messages/15339f3d12042fec?format=metadata&metadataHeaders=To&metadataHeaders=From&metadataHeaders=Subject&fields=payload%2Fheaders&access_token={ACCESS_TOKEN}
Response
{
"payload": {
"headers": [
{
"name": "To",
"value": "Emil <emtholin#gmail.com>"
},
{
"name": "From",
"value": "\"BernieSanders.com\" <info#berniesanders.com>"
},
{
"name": "Subject",
"value": "5,000,000"
}
]
}
}
As you can see, the values you are looking for are in the headers. You just have to sort them out in Java and you are done. The headers are not encoded like the body, so there is no need to do any decoding.

Plain string template query for elasticsearch through java API?

I have a template foo.mustache saved in {{ES_HOME}}/config/scripts.
POST to http://localhost:9200/forward/_search/template with the following message body returns a valid response:
{
"template": {
"file": "foo"
},
"params": {
"q": "a",
"hasfilters": false
}
}
I want to translate this to using the java API now that I've validated all the different components work. The documentation here describes how to do it in java:
SearchResponse sr = client.prepareSearch("forward")
.setTemplateName("foo")
.setTemplateType(ScriptService.ScriptType.FILE)
.setTemplateParams(template_params)
.get();
However, I would instead like to just send a plain string query (i.e. the contents of the message body from above) rather than build up the response using the java. Is there a way to do this? I know with normal queries, I can construct it like so:
SearchRequestBuilder response = client.prepareSearch("forward")
.setQuery("""JSON_QUERY_HERE""")
I believe the setQuery() method wraps the contents into a query object, which is not what I want for my template query. If this is not possible, I will just have to go with the documented way and convert my json params to Map<String, Object>

I ended up just translating my template_params to a Map<String, Object> as the documentation requires. I utilized groovy's JsonSlurper to convert the text to an object with a pretty simple method.
import groovy.json.JsonSlurper
public static Map<String,Object> convertJsonToTemplateParam(String s) {
Object result = new JsonSlurper().parseText(s);
//Manipulate your result if you need to do any additional work here.
//I.e. Programmatically determine value of hasfilters if filters != null
return (Map<String,Object>) result;
}
And you could pass in the following as a string to this method:
{
"q": "a",
"hasfilters": true
"filters":[
{
"filter_name" : "foo.untouched",
"filters" : [ "FOO", "BAR"]
},
{
"filter_name" : "hello.untouched",
"list" : [ "WORLD"]
}
]
}