I develop a high load enterprise application.
There are 2 services which should be scaled in specific way. They use Azure EventHubs for messaging. When load increase we need to create one more instance of a service and create one more topic (Event Hub) for communication with other services.
Is there a way to create event hub dynamically from java code? For example if I use Kafka I can just pass name of topic that doesn't exist and it will create it by itself. When I try to do it with Azure EventHubs I have such error:
The messaging entity 'sb://eventhubdev.servicebus.windows.net/newTopic' could not be found.
So... is it possible to create and delete it programmatically?
Google didn't help me with this question clearly enough.
There might be a solution to scale via java, but I would challenge that.
Scaling should be handled by your infrastructure (e.g. kubernetes) and not your code.
Furthermore, I don't know if the eventhub is dynamically enough to be scaled in the first place.
Providing the eventhub could be done via terraform.
See Link for further details:
https://www.terraform.io/docs/providers/azurerm/r/eventhub.html
After long investigation we decided to create new topic by direct API calls as described in this doc: https://learn.microsoft.com/en-us/rest/api/eventhub/eventhubs/createorupdate
Related
I am currently attempting to create an automated data export from an existing Google Datastore "Kind" to output to a json file.
I'm having trouble finding a suitable example that allows me to simply pull specific entities from the Datastore and push them out into an output file.
All the examples and documentation I've found assume I am creating an app-engine project to interface with the Datastore. The program I need to create would have to be local to sit on a server and query the Datastore to pull down the data.
Is my approach possible? Any advice on how to achieve this would be appreciated.
Yes, it is possible.
But you'll have to use one of the generic Cloud Datastore Client Libraries, not the GAE-specific one(s). You'll still need a GAE app, but you don't have to run your code in it, see Dependency on App Engine application.
You could also use one of the REST/RPC/GQL APIs, see APIs & Reference.
Some of the referenced docs contain examples as well.
I would like to design the best architecture for my following project: I have an application running on any device (desktop, mobile...) where users can publish or receive notifications with other users they share data.
Basically, a user can share with other users what he is doing on the application, other users being notified in real-time of the changes and vice-versa. And users are only able to receive notifications they are allowed by other users.
For example, when a user moves a widget on the screen, the application must store the new widget position, and also notify in real-time other users of this new position to perform the change on their screen. For this need, I would see an event-driven architecture with a publish-subscribe pattern. However, I guess I would also need to handle sync request-response pattern when the application needs to retrieve the list of users to share a widget for example.
I had a quick look at Streaming Data book by Manning where a streaming data architecture is described, but I don't know if this kind of architecture would fit my needs. One difference for example in the implementation part is that the event source producer can also be an event consumer in my application (in the book, the event source producer is a separate public streaming API and the real application is the only consumer)
My idea if I follow a bit the book would be the following: WebSocket for data ingestion and data access, a broker-like Kafka as message repository and a separate analysis service consuming Kafka topics and persisting data in DB. One interrogation is if I could use only one WebSocket for both data ingestion and data access.
Which detailed architecture and tools would you use to fit these needs?
For the implementation, I would consider javascript for the client part, and Java for the server part.
This is a pretty common use case for Kafka (leveraging both the broadcast and storage elements). There are some examples here which should help, although the context is slightly different:
https://github.com/confluentinc/kafka-streams-examples/tree/4.0.0-post/src/main/java/io/confluent/examples/streams/microservices
https://www.confluent.io/blog/building-a-microservices-ecosystem-with-kafka-streams-and-ksql/
In this example the CQRS pattern is used, so changes you make to the screen position would create events sent to kafka, then you create a view service that other application instances can (long) poll to get changes.
You could also implement this with a websocket. There are a few implementations of this on github but I haven't tried any of them personally. The one complexity is that, if you want to scale out to many nodes, you need some way to map messages in Kafka to open websockets (whereas mapping requests to kafka partitions in the REST example is handled automatically). Getting started with a single server implementation wouldn't require this complexity though.
I have been messing around with Spring Integration, and I'd like to use its capabilities in a project. I have two different applications, one is a business-sided one, which when users do certain actions, should send messages to another application. The other application should receive these messages (using some kind of queue (rabbitmq or another) to handle big loads) and store them, so it can use it to create realtime statistics of a number of applications running.
These messages will just contain information of the actions of the users, for example "bought N of product X" or "Used searchbar for Y".
This scenario is of course pretty simple, but I don't want to use any kind of XML configuration in my Spring applications. The examples I have seen so far all rely on XML, but I want to use some kind of annotations instead.
Spring Integration java dsl is already released which would serve your purpose.
A gentle introduction can be found at
https://dzone.com/articles/spring-integration-java-dsl
You can also checkout Camel (has a fluent dsl for EIP) which also gels well with spring
I am planning to upgrade Solr from single instance option to cloud option. Currently I have 5 cores and each one is configured with data import handler. I have deployed web application along with solr.war inside tomcat folder which will trigger full imports & delta-imports periodically according to my project needs.
Now, I am planning to create 2 shards for this application keeping half of my 5 cores data into each shard.I am not to understand how DIH will work in SolrCloud?
Is it fine if I start full-indexing from both shards?
Or I need to do full indexing from only one shard?
Architecture will look like below
It all depends on how you create your solr cloud: using composite id or implicit routing. Using composite id routing will take care of spreading the documents across all available shards. You can initiate the import from any solr cloud node. In the end the cloud environment will contain the imported document indices spread across all shards.
If you use implicit routing you have control where to keep each document index.
You do not have to use the DIH. Alternatively you can write a small app that uses the solr client to populate the index, which gives you more control.
After lots of googling and reading I finally decided to implement DIH as follows. Please let me know your comments if you feel there will be issues with this architecture.
I am looking to implement a way to transfer data from one application to another programmatically in Google app engine.
I know there is a way to achieve this with database_admin console but that process is very time inefficient.
I am currently implementing this with the use of Google Cloud Storage(GCS) but that involves querying data, saving it to GCS and then reading from GCS from different app and restoring it.
Please let me know if anyone knows a simpler way of transferring data between two applications programmatically.
Thanks!
Haven't tried this myself but it sounds like it should work: Use the data_store admin to backup your objects to GCS from one app, then use your other app to restore that file from GCS. This should be a good method if you only require a one time sync.
If you need to constantly replicate data from one app to another, introducing REST endpoints at one or both sides could help:
https://code.google.com/p/appengine-rest-server/ (this is in Python, I know, but just define a version of your app for the REST endpoint)
You just need to make sure your model definitions match on both sides (pretty much update the app at both sides with the same deployment code) and only have the side that needs to sync data track time of last sync and use the REST endpoints to pull in new data. Cron Jobs can do this.
Alternatively, create a PostPut callback on all of your models to make a POST call every time a model is written to your datastore to the proper REST endpoint on the other app.
You can batch update with one method, or keep a constantly updated version with the other method (at the expense of more calls).
Are you trying to move data between two App Engine applications or trying to export all your data from App Engine so you can move to a different hosting system? Your question doesn't have enough information to understand what you're attempting to do. Based on the vague requirements I would say typically this would be handled with a web service that you write in one application that exposes the data and the other application calls that service to consume the data. I'm not sure why Cloud Endpoints was down voted because that provides a nice way to expose your data as a JSON based web service with a minimum of coding fuss.
I'd recommend adding some more details into your question like exactly what are you trying to accomplish and maybe a mock data sample.
You could create a backup of your data using bulkloader and then restore it on another application.
Details here:
https://developers.google.com/appengine/docs/python/tools/uploadingdata?csw=1#Python_Downloading_and_uploading_all_data
Refer to this post if you are using Java:
Downloading Google App Engine Database
I don't know if this could be suitable for your concrete scenario, but Google Cloud Endpoints are definitely a simple way of transferring data programmatically from Google App Engine.
This is kind of the Google implementation of REST web services, so they allow you to share resources using URLs. This is still an experimental technology, but as long as I've worked with them, they work perfectly. Moreover they are very well integrated with GAE and the Google Plugin for Eclipse.
You can automatically generate an endpoint from a JDO persistent class and I think you can automatically generate the client libraries as well (although I didn't try that).