Welcome to Ensign U!

Welcome to Ensign U!

Rotational Labs | Friday, Oct 20, 2023 |  Python EventingML

Like any new tool, Ensign takes a little time to learn. But take it from us (a bunch of seasoned data science experts) – it will open up a new world of data science applications for you. Check out this beginner-friendly introduction!

Ensign is a flexible database and data streaming engine built for data scientists by data scientists. We make it easy for data teams to build and deploy real-time data products and services that make an impact, no matter your use case.

To help you on your learning journey, we’ve established Ensign U — a curriculum designed to get your first project up and running.

First, the Pre-Reqs

If you’re new to real-time data streaming, pubsub and event-driven architecture, here are a few recommended resources:

Make sure you have Python installed. We recommend using Python >= 3.9.

Step 1. Start with a Use Case

Start with a use case in mind. This can be any type of streaming data project that you’re interested in and your primary goal is to bring meaning to the data.

Looking for more inspiration?

Step 2. Pick Your Streaming Dataset(s)

Where is your data coming from? A data source can be a database, a public data API, a running web application, a bunch of IoT sensors, a file system, system logs, or other upstream sources.

Check out our Data Playground for publicly available APIs that stream data.

Step 3. Sketch Out Your Data Flow Architecture

Most data scientists want to start by training the machine learning model, since that’s how we’re trained. But this can lead you to a trained model that can’t fit into an application without a lot of painful MLOps.

In our experience, data science projects are much easier to deploy if you map out your data flows first, and then start modeling.

Watch these short videos:

Step 4. Set Up Your Project Infrastructure

Here’s where you need to create an account and log into Ensign. These videos will help.

  • Create a project: A project is a collection of topics and will serve as your data repository for streaming events.
  • Name your topics: A topic is a log of events that contain information you’re interested in.
  • Generate API keys: Use your API keys to connect your data source to your topic.
  • Connect using your API keys: Establish a client connection to the Ensign server by passing in the ClientSecret and ClientID from your API keys, either via environment variables or a gitignored file inside the repo.

Step 5. Install PyEnsign

Install the pyensign sdk. We recommend using Python >= 3.9.

You can pip install pyensign with:

pip install pyensign

Or, if you have an older version of pyensign installed (especially before v0.11b0), make sure you are on the latest release by upgrading your pyensign version using:

pip install --upgrade pyensign

Check out the docs if needed!

Step 6. Run the Minimal PyEnsign Example

Check that your API keys and your pyensign installation are working. You can do this by running the minimal python example.

Step 7. Run the Earthquake Data Ingestion Example Code

Run the earthquake example to ingest data into your topic using your own API keys.

Step 8. Query Your Ingested Data

Once you get data into your topic, you can view the data by logging into your Ensign account and navigating to Projects > Project Name > Topic Name. You will see the details page for your topic, and you can run the sample Topic Query provided to view a few of the events you have successfully ingested.

You can also query your data using EnSQL — see an example of how to do that with PyEnsign and your earthquake data here.


Now you have data flowing into your topic. The world is your oyster!

Use this same process each time you have a new data science project, and your chances of getting it deployed will be that much greater!

Here are some common data science projects that can add value to an organization:

  • Working with sensitive data? Apply transformations that mask or anonymize data from your data source.
  • Create real-time machine learning models that learn on the go.
  • Publish streams to web apps, mobile apps, dashboards, an
  • Publish data from different topics into a new a new topic
  • Collect data from edge devices and sensors.

Photo by Pablo Dodda on Flickr Commons.

Photo by Pablo Dodda

About This Post

Your end-to-end curriculum for getting started with Ensign, a database-meets-event-streaming platform for data science teams.

Written by:

Share this post:

Recommended  Rotations

View all

What is Incremental Machine Learning?

Most of us got into data science because it’s exciting (if chaotic) and there’s a constant stream of new ideas, which is thrilling (if intimidating). But if learning is all about keeping up, why can’t our models do it?

Aug 21, 2023

Streaming NLP Analytics Made Easy With HuggingFace LLMs and Ensign

Thinking about using a large language model (LLM) at your organization? Check out this tutorial to see how to bootstrap an MVP using an open source pre-trained model from HuggingFace and a free Ensign account.

Aug 1, 2023

MLOps 101: A Fresh Approach to Managing Models with Event Streams

If you can’t deploy your models, you might feel frustrated, but you aren’t alone — only 1 in 10 data science projects ever gets deployed. The fix? We need to shift our mental model towards “thinking in events.”

Jul 11, 2023
Enter Your Email To Subscribe