Let’s talk about streaming data… Streaming data is the foundation for many modern apps. From companies like Safe Watch Global’s Wihkum application focused on school safety and location data, to device telemetry information used for maintenance, or water level monitoring in rural Australia. Streaming data forms the basis of applications that help us make decisions in Real Time.
In this blog post, I will cover how to stream data in real-time to OCI Streams and then land the data into an Autonomous Database within OCI. To clarify, I am referring to Streaming Data, not Streaming Media. :-)
Setting up a stream
OCI Streams is a fully managed streaming data service within OCI. Streams allow multiple producers of data to send data into the stream, to be consumed by a reader application. If you were programming streaming data back in the 90’s and early 2000s like I was, you would probably remember dealing with sockets to achieve the same result. Yes, sockets and web sockets are alive and well, but thankfully, if all you need to do is send data messages to a server, fully managed services like Streams take a lot of the work off your plate, which means that you don’t need to manage infrastructure.
As a side note, for those already streaming data using Apache Kafka, OCI Streams is designed to be fully compatible with Kafka. This allows you to reuse some of your existing experiences in a Cloud Native environment.
You can easily set up your first stream in OCI. Select Analytics and AI from the hamburger menu, then select Streaming under the messaging heading.
Creating a new stream is as simple as providing a name for the new stream and selecting your OCI compartment.
Stream pools create a logical grouping of streams. You might want to have a single stream pool for a standard set of applications, such as a single pool for all standard applications. The stream pool will share some common configuration settings across all streams, such as encryption.
You have the option to tune your stream by changing the default retention time and the number of partitions. Each partition gives you 1MB/s of write capacity and 2MB/s of read capacity for the stream. I will leave these as the default for the moment.
Once your OCI Stream is set up, the next step is to configure a data producer to stream data to your stream.
Creating a stream client
Setting up a new stream client is refreshingly simple. It took me a few hours from start to finish to create my first sample application. Most of that time was spent on background tasks. Apparently, I had never set up the OCI CLI on my dev box because I was using a virtual machine last time. And, as usual, when I started Visual Studio, it had to update. It’s a bit like when I turn on the PlayStation… it always seems to need to update!
Setting up a new client application involves creating your application, installing the OCI Libraries via NuGet, and writing some refreshingly simple code to send data. Once I got past the VS Updates, most of the coding time was spent refining my code into a library, as I wanted to eventually make it cross-cloud compatible and make it easy for readers to retrieve the code from GitHub and incorporate it into their projects.
You can access the full code on GitHub in the usual spot.
https://github.com/BasementProgrammer/StreamingDataClient
Setting up your client application involves the following:
Install the OCI DotNet SDK packages:
For this application, I needed to install:
OCI.DotNetSDK.Common
OCI.DotNetADK.Streaming
To run the application locally, I also needed to install the OCI Command Line. This allowed me to set up a profile to connect to my OCI account. On my development machine, I have two profiles: one for connecting to my personal account and one for connecting to my work account.
Once you have set up your profile, verified connectivity to your account, and imported the libraries into your account, writing the code to send streaming data is relatively straightforward.
Create a streaming client
_config = new ConfigFileAuthenticationDetailsProvider("WORK");
_client = new StreamClient(_config);
This stream client will perform all the plumbing work to connect you to the OCI Streaming service.
Create a list of messages to send
This functionality allows you to send a set of messages in a batch. Depending on the logic of your application, you can balance the number of messages transmitted at a time.
List<PutMessagesDetailsEntry> messages = new List<PutMessagesDetailsEntry>();
foreach (string message in data)
{
PutMessagesDetailsEntry detailsEntry = new PutMessagesDetailsEntry
{
Key = Encoding.UTF8.GetBytes($"data"),
Value = Encoding.UTF8.GetBytes(message)
};
messages.Add(detailsEntry);
}
Finally, package the messages up into a request and send them to the stream via the client.
PutMessagesDetails messagesDetails = new PutMessagesDetails
{
Messages = messages
};
PutMessagesRequest putRequest = new PutMessagesRequest
{
StreamId = _streamConfig.StreamId,
PutMessagesDetails = messagesDetails
};
var response = _client.PutMessages(putRequest);
PutMessagesResponse putResponse = response.Result;
The call to PutMessages is an async call, so you will have to retrieve the result. Once you have sent the message, you can cycle through the results to check the values for the response codes. Each message will have a response.
foreach (PutMessagesResultEntry entry in putResponse.PutMessagesResult.Entries)
{
if (entry.Error != null)
{
Console.WriteLine($"Error({entry.Error}): {entry.ErrorMessage}");
}
else
{
Console.WriteLine($"Published message to partition {entry.Partition}, offset {entry.Offset}");
}
}
One last thing, the stream client is disposable, so please dispose of it when you are done.
The Web Client
As I mentioned, I have created a web client that you can use to connect to your own OCI Stream and send data.
This web interface will allow you to simulate multiple clients sending data to a single stream. The application is a work in progress, but will receive updates as I have time. So don’t expect the interface to remain as it stands in this image.
If you are going to use the sample code I have provided, you will at this point need to update the application configuration file appsettings.json with the values from your environment. here is the sample from the web client:
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"AllowedHosts": "*",
"ProfileName": "{{Your OCI CLI Profile Name}}",
"EndpointConfiguration": "{{Your endpoint name}}",
"StreamId": "{{OCID for your Stream}}"
}
Message Endpoint can be found my looking at the details for your stream and copying this value:
Summary
As you can see, implementing a streaming data client for OCI Streams, using .NET is pretty straightforward. The code can create the foundation for complex applications that deal with a wide range of use cases, such as monitoring.
I’ll be following up on this post with the implementation of the backend for the streaming service in OCI Functions, so be sure to check back!
