I am trying to create a dataflow pipeline template, which required me to read data from bigquery. So what i need is to make my query dynamic using like Instant.now() but it seems the query is locked when creating the template
Some Code HERE
Some Code HERE
Some Code HERE
pipeline.apply("ReadFromBigQuery",
BigQueryIO.read(new DataTransformer(MyCustomObject.getQuery()))
.fromQuery(spec.getQuery())
.usingStandardSql()
.withQueryLocation("US")
.withoutValidation()
).apply("do Something 1",
Combine.globally(new CombineIterableAccumulatorFn<MyCustomObject2>())
).apply("do Something 2",
ParDo.of(new SendToKenshoo(param, param2)
);
My query is like this
SELECT * FROM `my-project-id.my-dataset.my-view` where PARTITIONTIME between TIMESTAMP('#currentDate') and TIMESTAMP('#tomorrowDate')
need to replace that #currentDate and #tomorrowDate using Instant.now() or any time function
please give me some example
note : i need to change the date on the code instead on query level like this
SELECT * FROM `my-project-id.my-dataset.my-view` where PARTITIONTIME between DATE_ADD(CURRENT_DATE(), INTERVAL -1 DAY) and CURRENT_DATE()
I'm not sure how you're sending those parameters to the query (via value provider, etc). However, I wouldn't recommend using templates for that because you need dynamic inputs. If you want to do that, I would use Flex Templates: https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates
Related
I am using the following code segment to do the insertion using JOOQ's UpdatableRecord.
public void acknowledgeDisclaimer(AcknowledgeDisclaimerReq acknowledgeDisclaimerReq) {
DisclaimerRecord disclaimerRecord = dslContext.newRecord(Disclaimer.DISCLAIMER);
disclaimerRecord.setDisclaimerForId(acknowledgeDisclaimerReq.getDealListingId());
disclaimerRecord.setDisclaimerForType("DEAL");
disclaimerRecord.setAcceptedAt(LocalDateTime.now());
disclaimerRecord.setAcceptedByOwnerId(acknowledgeDisclaimerReq.getLoggedInOwnerId());
int count = disclaimerRecord.store();
log.info("Inserted entry for disclaimer for deal: {}, owner: {}, id {}, insertCount: {}", disclaimerRecord.getDisclaimerForId(), disclaimerRecord.getAcceptedByOwnerId(), disclaimerRecord.getId(), count);
}
When setting the AcceptedAt data, I want to use the database's current timestamp instead of passing the JVM timestamp. Is there any way to do that in JOOQ?
UpdatableRecord.store() can only set Field<T> => T key/values, not Field<T> => Field<T>, so you cannot set an expression in your record. You can obviously run an explicit INSERT / UPDATE / MERGE statement instead.
Using triggers
The best way to ensure such a timestamp is set to the database timestamp whenever you run some specific DML on the table is to use a database trigger (you could make the trigger watch for changes in the ACCEPTED_BY_OWNER_ID value)
If you can't do this on the server side (which is the most reliable, because it will behave correctly for all database clients, not just the JDBC/jOOQ based ones), you might have a few client side options in jOOQ:
Using jOOQ 3.17 client side computed columns
jOOQ 3.17 has added support for stored (or virtual) client side computed columns, a special case of which are audit columns (which is almost what you're doing).
Using this, you can specify, for example:
<forcedType>
<generator><![CDATA[
ctx -> org.jooq.impl.DSL.currentTimestamp()
]]></generator>
<includeExpression>(?i:ACCEPTED_AT)</includeExpression>
</forcedType>
The above acts like a trigger that sets the ACCEPTED_AT date to the current timestamp every time you write to the table. In your case, it'll be more like:
<forcedType>
<generator><![CDATA[
ctx -> org.jooq.impl.DSL
.when(ACCEPTED_BY_OWNER_ID.isNotNull(), org.jooq.impl.DSL.currentTimestamp())
.else_(ctx.table().ACCEPTED_AT)
]]></generator>
<includeExpression>(?i:ACCEPTED_AT)</includeExpression>
</forcedType>
See a current limitation of the above here:
https://github.com/jOOQ/jOOQ/issues/13809
See the relevant manual sections here:
Client side computed columns
Audit columns
Should be something like disclaimerRecord.setAcceptedAt(DSL.now());
I m trying to sort the AEM query builder search results based on particular value of particular property. as we have in any database like MySQL we can sort based on column's value as well (for exp. ORDER BY FIELD('columnName','anyColumnName'). can we have something like this in AEM.
Suppose we have 5 Assets under path /content/dam/Assets.
Asset Name------------dc:title
1.jpg------------------Apple
2.jpg------------------Cat
3.jpg------------------Cat
4.jpg------------------Ball
5.jpg------------------Drag
I need assets on top of the results where dc:title = cat and also need other results also in sorting asc. expected result as given below
2.jpg------------------Cat
3.jpg------------------Cat
1.jpg------------------Apple
4.jpg------------------Ball
5.jpg------------------Drag
Note:- Using version AEM 6.2
You can use the orderby predicate with a value of #jcr:content/metadata/dc:title to sort by dc:title with the QueryBuilder. /libs/cq/search/content/querydebug.html is an interface to test queries on your instance. ACS Commons has a good breakdown of all out of the box predicates
If you want to pull Cats to the top of the results with a single query, you could write a custom predicate. The sample code from ACS Commons shows an example. Adobe has documentation as well.
I am facing a problem in Google bigquery. I have some complex computation need to do and save the result in Bigquery. So we are doing that complex computation in Java and saving result in google bigquery with the help of Google cloud dataflow.
But this complex calculation is taking around 28 min to complete in java. Customer requirement is to do within 20 sec.
So we switch to Google bigquery UDF option. One option is Bigquery legacy UDF. Bigquery legacy UDF have limitation that it is processing row one by one so we phased out this option. As we need multiple rows to process the results.
Second option is Scalar UDF. Big query scalar UDF are only can be called from WEB UI or command line and can not be trigger from java client.
If any one have any idea the please provide the direction on the problem how to proceed.
You can use scalar UDFs with standard SQL from any client API, as long as the CREATE TEMPORARY FUNCTION statements are passed in the query attribute of the request. For example,
QueryRequest queryRequest =
QueryRequest
.newBuilder(
"CREATE TEMP FUNCTION GetWord() AS ('fire');\n"
+ "SELECT COUNT(DISTINCT corpus) as works_with_fire\n"
+ "FROM `bigquery-public-data.samples.shakespeare`\n"
+ "WHERE word = GetWord();")
// Use standard SQL syntax for queries.
// See: https://cloud.google.com/bigquery/sql-reference/
.setUseLegacySql(false)
.build();
QueryResponse response = bigquery.query(queryRequest);
Big query scalar UDF are only can be called from WEB UI or command
line and can not be trigger from java client.
This is not accurate. Standard SQL supports scalar UDFs through CREATE TEMPORARY FUNCTION statement which can be used from any application and any client - it is simply part of the SQL query:
https://cloud.google.com/bigquery/docs/reference/standard-sql/user-defined-functions
To learn how to enable Standard SQL, see this documentation: https://cloud.google.com/bigquery/docs/reference/standard-sql/enabling-standard-sql
Particularly, simplest thing would be to add #standardSql tag at the beginning of SQL query.
Spring Data Cassandra 1.5.0 comes with a streaming API in CassandraTemplate. I'm using spring-data-cassandra 1.5.1. I have a code like:
String tableName = cassandraTemplate.getTableName(MyEntity.class).toCql();
Select select = QueryBuilder.select()
.all()
.from(tableName);
// In real world, WHERE statement is much more complex
select.where(eq(ENTITY_FIELD_NAME, expectedField))
List<MyEntity> result = cassandraTemplate.select(select, MyEntity.class);
and want to replace this code with iterable or Java 8 Stream in order to avoid fetching a big list of results to memory at once.
What I'm looking for is a method signature like CassandraOperations.stream(Select query, Class<T> entityClass), but it is not available.
The only available method in CassandraOperations accepts query string: stream(String query, Class<T> entityClass). I tried to pass here a string generated by Select like
cassandraTemplate.stream(select.getQueryString(), MyEntity.class)
But that fails with InvalidQueryException: Invalid amount of bind variables, because getQueryString() returns query with question mark placeholders instead of variables.
I see 3 options to get what I want, but every option looks bad:
Use Spring Query creation mechanism with Stream/Iterator expected return type (good only for simple queries) http://docs.spring.io/spring-data/cassandra/docs/current/reference/html/#repositories.query-methods.query-creation
Use raw CQL query and not to use QueryBuilder
Call select.getQueryString() and then substitute parameters again via BoundStatement
Is there any better way to stream selection results?
Thanks.
So, as of now the answer on my question is to wait until stable version of spring-data-cassandra 2.0.0 comes out:
https://github.com/spring-projects/spring-data-cassandra/blob/2.0.x/spring-data-cassandra/src/main/java/org/springframework/data/cassandra/core/CassandraTemplate.java#L208
Recently, one of our clients reported not being able to create a table based on a query against a view. That said, they were able to save the result of a query against a table into another table. This issue spawned a more implementation focused question using the Java client libraries. Specifically, is there any way to save the result set of a query against a view to a table using the Java client library? I will be digging and post anything that I find. That said, any early guidance would be appreciated!
To be specific and add more context, I note that the the following process failed when the query was run against a union view.
java -jar BigQueryToCloudExporter.jar ./GAFastAccessKey.p12 '' "
Select date(date_add('2014-08-09',floor(datediff(date(sec_to_timestamp(visitstarttime)),'2014-08-03')/7)*7,"DAY")) WeekEndDate
, hits.eventinfo.eventaction GA_RentalNo
, count(distinct visitID) PDP_PPC
FROM (TABLE_DATE_RANGE([Union_View.GA],
TIMESTAMP('2014-08-30'),
TIMESTAMP('2014-09-13')))
where hits.eventinfo.eventcategory='property attributes'
and brandId=121
--hits.eventinfo.eventcategory='property inquiry'
and trafficsource.medium like '%cpc%'
--and trafficsource.campaign not like '%ppb%'
and trafficsource.campaign like '%mpm%'
group each by WeekEndDate, GA_XXXXXX
order by WeekEndDate, GA_XXXXXX limit 100" StagingQueryTable QueryTable AVRO gs://XXXXXX/QueryTable*.avro
On the other hand, the following process succeeded when the query was made against a BigQuery table (keeping everything else same).
java -jar BigQueryToCloudExporter.jar ./GAFastAccessKey.p12 '' "
Select date(date_add('2014-08-09',floor(datediff(date(sec_to_timestamp(visitstarttime)),'2014-08-03')/7)*7,"DAY")) WeekEndDate
, hits.eventinfo.eventaction GA_XXXXXX
, count(distinct visitID) PDP_PPC
FROM (TABLE_DATE_RANGE([XXXXXX.ga_sessions_],
TIMESTAMP('2014-08-30'),
TIMESTAMP('2014-09-13')))
where hits.eventinfo.eventcategory='property attributes'
and brandId=121
--hits.eventinfo.eventcategory='property inquiry'
and trafficsource.medium like '%cpc%'
--and trafficsource.campaign not like '%ppb%'
and trafficsource.campaign like '%mpm%'
group each by WeekEndDate, GA_RentalNo
order by WeekEndDate, GA_XXXXXX limit 100" StagingQueryTable QueryTable AVRO gs://XXXXXX/QueryTable*.avro