Mastering the Art of Data Freshness

Mastering the Art of Data Freshness

Edwin Schmierer | Friday, Mar 24, 2023 |  Eventing Flexstreams

Two decades ago Clive Humby proclaimed “data is the new oil”, marking an economic shift towards data products. But, if the intervening years have taught us anything, it’s to think of data more like fresh fruit.

A few weeks ago, Jordan Tigani (a founding engineer for BigQuery) introduced a new polemic — “Big Data is Dead”. He argued that most companies actually don’t have enough data to justify their spending on the data management tools that he admittedly once espoused, and asked the reader to consider whether they are in the “one percent” of companies who actually need and use big data. While his article makes some very important points (especially that data can be a liability as it grows and ages), we can’t help but wonder if it’s a bit of a circular argument. Just as money (when well-managed) tends to make more money, so does data tend to make more data.

Back in 2016, in “The Age of the Data Product”, our CEO Dr. Benjamin Bengfort wrote about the data product revolution. The key takeaway is that what differentiates the “big data one percent” from the rest is not just that they have more data, but that they’ve figured out how to build self-sustaining data engines that make more data.

A data product is an economic engine. It derives value from data and then produces more data, more value, in return…This is precisely what has led to the surplus of information and the resulting information revolution. More importantly, it is the generative effect that allows us to achieve better living through data, because more data products mean more data, which means even more data products, and so forth.

So here’s a fresh take — yes, big data matters, but “small and wide” data can prove just as valuable. It all comes down to mastering the art of data freshness.

A Trip To the Supermarket

Let’s start with a simple scenario. Imagine you visit the supermarket. You buy $100 worth of food. A week later, you’ve eaten less than $20 worth of the food you bought. What can you do? Most likely, you’ll buy less next time. Or, maybe you buy another fridge or possibly eat more, faster. The point is, you do something.

You’ll Get Nothing and Like It?

If you’re an executive, data is like the food that fuels your business. Arguably, it’s your most important asset. Maybe it’s key to your future AI plans (hello, ChatGPT!).

According to McKinsey, businesses use less than 20% of data they generate. Data silos persist despite the rise of data warehouses and data lakes. Why? It’s just too dang hard to move and synchronize data. And everyday it sits, it loses value. We heard the same after talking to over one hundred organizations, from startups to Fortune 500s, operating and building applications, data products, and services. And the problem is only getting worse.

“Data Value Gap”

We get it. Putting data in motion is hard. But still we have to ask: What executive would ever accept a less than 20% utilization on such an important asset?

A Better Option: FlexStreams

We built Ensign, our cloud-native, cloud agnostic data management platform, to make it easy to get data in motion quickly and securely so businesses can quickly innovate their Artificial Intelligence (AI) and automation strategies. With Ensign, you can set up any number of flexible and secure event streams - what we call FlexStreams - to get data to the right process at the right time and place with no additional investment in infrastructure or unexpected costs. It’s a “low ops”/ “no ops” way to get your data in motion quickly and securely to power your AI strategy. Simply put, generate more value at 1/10 the cost and power your AI strategy.

How can FlexStreams help your organization?

  • Cybersecurity: Detect and prevent cyber threats by analyzing network traffic and identifying anomalies to protect assets.
  • ESG Reporting: Build reporting capabilities across disparate data sources, from HR and ATS systems to energy usage and governance policies, to track ESG goals.
  • Knowledge Management: Produce and surface documents, policies, and insights on demand in real-time with accuracy to make better, faster decisions.
  • Sales Enablement: Generate predictive insights across CRMs, inventory systems, and other customer data sources to increase average revenue per use or annual contract value.
  • Supply Chain Optimization: Optimize supply chain processes by predicting demand, reducing inventory waste, and optimizing delivery routes to save costs.

You Can Do Something!

To all the frustrated execs out there, it’s time to do something! We don’t expect you to believe our word so we invite you to try yourself (or more realistically your technical team). No credit card or long-term contract required. Doesn’t matter if you’re in the cloud, on prem, or hybrid. Doesn’t matter what type of data you have. Doesn’t matter if you have legacy systems. We’ll meet you where you are. What does matter is you have to care about getting the most value from your data. If you do, we guarantee we’ll move the needle in the right direction for you.

What have you got to lose? Think about that the next time you take a trip to the supermarket.

  • Does it make sense to just stop gathering data? No!
  • Does it make sense to buy more cloud storage so your data can just sit around? Also probably no.
  • Or do you instead focus on keeping data fresh and putting it to use? That’s Ensign!

Want early access to a platform and community for developers building event-driven apps? Check out our free beta of Ensign.

Photo by Oleksandr K (oleksandr_k) via Flickr Commons

About This Post

In this post we consider what it takes to maximize the value of your data in a business context.

Written by:

Share this post:

Recent Rotations butterfly

View all

To LLM or Not to LLM (Part 2): Starting Simple

Sick of hearing about hyped up AI solutions that sound like hot air? 🧐 Let’s use boring old ML to detect hype in AI marketing text and see why starting with a simple ML approach is still your best bet 90% of the time.

Building an AI Text Detector - Lessons Learned

The LLMs boom has made differentiating text written by a person vs. generated by AI a highly desired technology. In this post, I’ll attempt to build an AI text detector from scratch!

May 15, 2024

To LLM or Not to LLM: Tips for Responsible Innovation

We’re seeing a proliferation of Large Language Models (LLMs) as companies seek to replicate OpenAI’s success. In this post, two AI engineers respond to LLM FAQs and offer tips for responsible innovation.

Enter Your Email To Subscribe