I am trying to trigger a Google Dataflow job from Airflow and need help in sending a string from Airflow as parameter which could be read in Dataflow and used as an independent string.
Here´s the code for my DataflowTemplateOperator which sends the parameter named secretCode:
DataflowTemplateOperator(
task_id=TASK_ID,
job_name=JOB_NAME,
template=TEMPLATE_PATH,
parameters={
"secretCode": "123456"
},
dag=dag
)
I want to read the secretCode from PipelineOptions as String as send to the following ParDo but I don´t know how to do that. The code does not have anything to do with the input and output of the ParDo class. I just want to write the code to BigQuery.
val dataToTableRow: PCollection<TableRow> = myCustomDataStructure.apply(
"transform my data to table row",
ParDo.of(DataToTableRow())
)
I want to write the secret code returned from the PipelineOptions to BigQuery as I show in the code below but I don´t know how to get there:
class DataToTableRow : DoFn<myCustomDataStructure, TableRow>() {
#ProcessElement
fun processElement(#Element myData: myCustomDataStructure, outputReceiver: OutputReceiver<TableRow>) {
outputReceiver.output(getTableRow(myData))
}
private fun getTableRow(myData: myCustomDataStructure): TableRow {
return TableRow().set("ID", myData.id)
.set("SecretCode", secretCode)
}
}
I would appreciate some help regarding how to solve this problem. Thanks in advance.
You need to create your own interface that extends PipelineOptions and set your parameter here.
public interface SecretOptions extends PipelineOptions {
String getSecretCode();
void setSecretCode(String secretCode);
}
And then, register your interface on your Pipeline like this :
PipelineOptionsFactory.register(SecretOptions.class);
SecretOptions options = PipelineOptionsFactory.fromArgs(args)
.withValidation()
.as(SecretOptions.class);
Then you'll be able to access to your parameter anywhere on your pipeline using options.getSecretCode();
More informations on the documentation
Related
I have Sensors from them i want to Display the Data of the Sensor(Accelorometer, Gyroscope, Magnetometer) and log that recorded data.
From the Documentation I see following PUT and GET Requests that I need to perform. I also have a Example but that is written in java but I'm not familiar with Java.
PUT /Mem/DataLogger/Config that lists paths that you want to log.
In my case they would be /Meas/IMU9/104 and /Meas/Temp.
PUT /Mem/DataLogger/State to 3 (=LOGGING)
PUT /Mem/DataLogger/State to 2 (=READY)
GET "suunto:/MDS/Logbook/{serial}/Entries" on mobile to have a list of entries on the sensors datamemory. The one with biggest LogID is the one you just recorded
GET "suunto:/MDS/Logbook/{serial}/byId/{LogId}/Data" on mobile to get the recording as JSON.
For all that i need the http Plugin. I started with creating Final String's for the Entries.
My current Code looks like:
import 'package:flutter/material.dart';
import 'package:http/http.dart';
import 'package:mdsflutter/Mds.dart';
class DataLoggerActivity extends StatefulWidget {
final String URI_MDS_LOGBOOK_ENTRIES = "suunto://MDS/Logbook/{0}/Entries";
final String URI_MDS_LOGBOOK_DATA = "suunto://MDS/Logbook/{0}/ById/{1}/Data";
final String URI_LOGBOOK_ENTRIES = "suunto://{0}/Mem/Logbook/Entries";
final String URI_DATALOGGER_STATE = "suunto://{0}/Mem/DataLogger/State";
final String URI_DATALOGGER_CONFIG = "suunto://{0}/Mem/DataLogger/Config";
#override
_DataLoggerActivityState createState() => _DataLoggerActivityState();
}
class _DataLoggerActivityState extends State<DataLoggerActivity> {
#override
Widget build(BuildContext context) {
return Container();
}
}
First, the GET & PUT are not HTTP calls (so no HTTP plugin), but calls in MDS library. Please check the latest mdsflutter plugin and the example within.
Full disclosure: I work for the Movesense team
I have a data source service, which takes an observer as a parameter.
void subscribe(Consumer onEventConsumer);
I want to use flux as a response stream for RSocket.
How can I do this?
As I see it now, it should be something like
Flux<T> controllerMethod(RequestMessage mgs) {
var flux = Flux.empty();
dataSource.subscribe(event -> flux.push(event));
return flux;
}
But I have big doubts that it's a proper solution, and I'm new in the reactive approach, I don't know what methods I should use here?
As Simon already pointed out, this is what you use Flux.create for.
Take a look at the Getting Started Guide on projectreactor.io.
In short, you register a custom listener inside the lambda of the create method:
Flux<String> bridge = Flux.create(sink -> {
myEventProcessor.register(
new MyEventListener<String>() {
public void onDataChunk(List<String> chunk) {
for(String s : chunk) {
sink.next(s);
}
}
public void processComplete() {
sink.complete();
}
});
});
What you want to do is to pass the incoming elements on to a FluxSink, which will then publish those elements on the Flux.
this is a typical use case of Flux.create. you register an obsereer from inside the create lambda, which will pass the data it receives down to the provided FluxSink
I'm trying to migrate from using plain Retrofit to using the RxJava extension for retrofit in order to make chain of API calls on the background thread.
For example, I have an object called ModelGroup which has a list of ModelPerson objects. My goal is to do the following.
Send ModelGroup to the server and receive a response, which is an integer, representing the newly inserted ID, let's call it newGroupId.
For each ModelPerson in ModelGroup, set Person.groupId to newGroupId.
Send each person to the server.
If all ModelPerson objects from the ModelGroup were successfully updated with newGroupId then respond with onSuccess, otherwise onError.
My current solution can be seen below.
private void makeGroupInsert(ModelGroup modelGroup) {
int newGroupId = myApi.insertNewGroup(modelGroup.getName(), modelGroup.getRating())
.execute()
.body();
for (ModelPerson person : modelGroup.getPersons()) {
person.setGroupId(newGroupId);
String response = myApi.insertNewPerson(
person.getGroup_id(),
person.getFirst_Name(),
person.getLast_Name())
.execute()
.body();
if (!response.equals("success")) {
// One failed to update, send error to main thread.
}
}
// All succeeded, send success back to main thread.
}
Question
How can I achieve the same (or better) functionality using a RxJava + Retrofit solution?
EDIT 1
MyApi is defined below.
public interface MyApi {
#POST("insert_new_group")
Call<Integer> insertNewGroup(#Query("group_name") String groupName,
#Query("group_rating") int rating);
#POST("insert_new_person")
Call<String> insertNewPerson(#Query("group_id") int groupId,
#Query("first_name") String firstName,
#Query("last_name") String lastName);
}
First of all, you need to change Retrofit beans to use Observables. For example, it can look like the following line:
#POST("insert_new_group")
Observable<Integer> insertNewGroup(...
Then you can chain requests:
void updateData() {
myApi.insertNewGroup(modelGroup.getName(), modelGroup.getRating()) //Creating new group and getting its ID
.switchMap(this::setGroupIdAll) //Calling observable that will loop thru all persons and set their groupIDs
.subscribe(
(n) -> {/*you will get String after every 'insertNewPerson' run*/},
(e) -> {/*error handling*/}
);
}
Observable<String> setGroupIdAll(Integer id) {
return Observable.fromIterable(personsIterable) //personsIterable contains all your ModelPerson objects
.flatMap(this::updatePerson); //Call Observabl;e that will send updated person to the server
}
Observable<String> updatePerson(ModelPerson person) {
return myApi.insertNewPerson(
person.getGroup_id(),
person.getFirst_Name(),
person.getLast_Name());
}
I have tried so many way but i can't succeed. I haven't found any source code examples for Android(about rekognition)
there's a source code in JAVA in the Developer Guide but i cannot implement that even though I tried TT
I try to detect faces by sending an image file from an external storage(from the emulator)
I don't know what i did wrong(I'm not good at coding)
Here is my code
AmazonRekognitionClient amazonRekognitionClient;
Image getAmazonRekognitionImage;
DetectFacesRequest detectFaceRequest;
DetectFacesResult detectFaceResult;
File file = new File(Environment.getExternalStorageDirectory(),"sungyeol.jpg.jpg");
public void test_00(View view) {
ByteBuffer imageBytes;
try{
InputStream inputStream = new FileInputStream(file.getAbsolutePath().toString());
imageBytes = ByteBuffer.wrap(IOUtils.toByteArray(inputStream));
Log.e("InputStream: ",""+inputStream);
Log.e("imageBytes: ","");
getAmazonRekognitionImage.withBytes(imageBytes);
// Initialize the Amazon Cognito credentials provider
CognitoCachingCredentialsProvider credentialsProvider = new CognitoCachingCredentialsProvider(
getApplicationContext(),
"us-east-2:.......", // Identity Pool ID
Regions.US_EAST_2 // Region
);
//I want "ALL" attributes
amazonRekognitionClient = new AmazonRekognitionClient(credentialsProvider);
detectFaceRequest = new DetectFacesRequest()
.withAttributes(Attribute.ALL.toString())
.withImage(getAmazonRekognitionImage);
detectFaceResult = amazonRekognitionClient.detectFaces(detectFaceRequest);
detectFaceResult.getFaceDetails();
}
catch(Exception ex){
Log.e("Error on something:","Message:"+ex.getMessage());
}
and here is my errors
02-04 09:30:07.268 29405-29405/? E/InputStream:: java.io.FileInputStream#a9b23e7
02-04 09:30:07.271 29405-29405/? E/Error on something:: Message:Attempt to invoke virtual method 'com.amazonaws.services.rekognition.model.Image com.amazonaws.services.rekognition.model.Image.withBytes(java.nio.ByteBuffer)' on a null object reference
what is a null object reference?
i try to change the file path but he said no such file ... and when I change to this path, there's errors above.
by the way I've already asked a user for a permission to access a folder from Emulator in Android
please help me
PS. sorry for my bad English
Thank you in advance.
Now I am ok with the issues. I have been through many many things <3 <3 <3.
Thank you
I'm Thai and I had to try harder to find the solutions because there's lack of information in the particular language. Here are my solutions.
My solutions are:
0.There is an endpoint for setting for the Rekognition-->
http://docs.aws.amazon.com/general/latest/gr/rande.html#rekognition_region
1.On a "null object reference issue" I found that I have to create a new object first such as "Image image = new Image();" <-- The "new" command creates an object instance in that class
2.After the above error, there are more errors (Errors on NetworkOnMainThreadException), so I tried everything until I found this page -->
https://docs.aws.amazon.com/cognito/latest/developerguide/getting-credentials.html the page said that ...
Consequently, I looked up for more information about the AsyncTask and after that I created an AsyncTask class and then I move all my code about the initialize, the request, the response to the AsyncTask class. ตอนรันตอนท้ายๆน้ำตาจิไหล my code worked... TT and by the conclusion the sungyeol.jpg.jpg file worked
for example
private void testTask(){
.... all code in the main thread particularly on the requests and responses
from the services
//print the response or the result
//Log.e() makes the message in the android monitor red like an error
Log.e("Response:", [responseparameter.toString()]);
}
//create the inherited class from the AsyncTask Class
//(you can create within your activity class)
class AsyncTaskRunner extends AsyncTask<String,String,String>{
#Override
public String doInBackground(String ... input){
testTask(); // call the testTask() method that i have created
return null; // this override method must return String
}
}
//I've created a button for running the task
public void buttonTask(View view){
AsyncTaskRunner runner = new AsyncTaskRunner();
runner.execute();
}
for more information about the AsyncTask:
https://developer.android.com/training/basics/network-ops/connecting.html#AsyncTask
http://www.compiletimeerror.com/2013/01/why-and-how-to-use-asynctask.html#.WJdkqVOLTIU
I hope these help :)
basically I just want to know, if there are ways on how to export excel sheet file in Podio with the comments section included in a certain app using API.
There is no direct API call to export to excel along with comments. Export work on Items and Apps
There is a work around to do this.
1. Make a CommentAPI call as per code attached, you can get all the comments on Item, so that you can export them to your excel programmatically.
public class APICall implements Serializable {
public static void main(String as[]){
APICall apiObj = new APICall();
apiObj.apicall();
}
/**
*
*/
public void apicall()
{
try{
System.out.println("inside");
ResourceFactory resourceFactory = new ResourceFactory(new OAuthClientCredentials("<WS NAME>","<Your authkey>"),new OAuthUsernameCredentials("<username>", "<Password>"));
CommentAPI capi = new CommentAPI(resourceFactory);
Reference ref= new Reference(ReferenceType.ITEM,561530318);
List<Comment> cmts = capi.getComments(ref);
for(Comment e : cmts )
System.out.println(e.getValue());