Category: Data & Cloud Migration

Databricks Genie One: Inside the Agentic Coworker Turning Business Data into Action

Posted on by inferenz.manage

Summary 

Databricks Genie One is the agentic coworker on Databricks’ Data Intelligence Platform, letting any business user, not just analysts, ask questions of governed data, save repeatable skills, and automate recurring work through scheduled tasks. It runs on Genie Agents and Metric Views for descriptive analytics, then extends into predictive use cases, like hospital readmission risk, through custom agents built on the Mosaic AI Agent Framework. This guide covers setup, accuracy and cost tuning, and field-tested best practices for rolling Genie One out across marketing, finance, sales, HR, and clinical operations teams. 

Databricks Genie One: End-to-End Architecture

A care coordination lead at a mid-size hospital system used to lose two days to a single readmission report: pulling numbers from three dashboards, emailing an analyst, and hoping the definitions matched. That wait is going away. 

Databricks Genie One, the agentic coworker built into the Data Intelligence Platform, lets any business user, not just analysts, ask questions of governed data, teach it repeatable skills, and hand it recurring work through scheduled tasks. Gartner expects 40% of enterprise applications to embed task-specific AI agents by the end of 2026, up from under 5% in 2025, and Genie One is Databricks’ clearest answer to that shift yet. 

This guide breaks down what Genie One does, the Genie Agents and Metric Views it runs on, how teams extend it into predictive work, and where the real accuracy and cost trade-offs live. 

care coordination lead at a mid-size hospital system used to lose two days to a single readmission report.

What Is Databricks Genie One?

Databricks Genie One is the general-purpose chat surface of the Data Intelligence Platform, built for people who have never written a line of SQL.

Where a Genie Agent is a governed, domain-scoped data building block, Genie One is the coworker any business user talks to. It draws its verified context from Genie Ontology and its trusted data and metrics from one or more Genie Agents, then layers on the capabilities a conversational agent needs to actually get work done. It answers questions, drafts documents and artifacts, takes action through MCP tools, etc. The two capabilities most relevant to day-to-day adoption are covered in depth below: skills and scheduled tasks.

Databricks announced it on June 16, 2026 at the Data + AI Summit, positioning it as a real step up from the original Genie, which only answered questions about data already sitting inside Databricks. 

Genie One reaches further. It works across structured and unstructured data, inside and outside the platform, and it ships on web, iOS, and Android. Marketing, finance, sales, HR, and clinical operations teams all get the same coworker, grounded in the same Unity Catalog permissions and lineage that govern everything else on the platform. 

At Inferenz, a data and AI solutions-led services company and Databricks partner, our team has believed that the differentiators sit underneath: Genie Agents, Metric Views, and, once the question turns predictive, custom agents, built on the Mosaic AI Agent Framework. 

Skills and scheduled tasks: how Genie One learns to work like you do 

Two features separate Genie One from a chatbot that forgets everything after each session. 

A skill is a task you teach the coworker once and reuse by name. Ask it to build your weekly metrics report, and it saves that approach as reusable, inspectable text under the open Agent Skills standard, the same convention Genie Code runs on. User skills currently sit in Public Preview, saved privately to your workspace, and Genie One applies one automatically unless you @-mention it directly.

Skills and scheduled tasks: how Genie One learns to work like you do

A scheduled task runs that same logic on a cadence you set, in plain English (something like “send me a daily briefing of new customer reviews”), then posts results into a chat thread plus an email. Schedules currently cap at daily frequency by default and need the Databricks SQL access entitlement to create or run. 

Dimension Skills Scheduled Tasks 
Trigger Manual, or auto-detected by Genie One Runs automatically on a set cadence 
Best for Repeatable, on-demand tasks Recurring reports and monitoring 
Output Chat response, document, or action Chat message plus an email notification 
How you set it up Ask Genie One to save an approach, or build one directly Natural-language request, or a manual form: Title, Instructions, Connections, Schedule, Timezone 
Governance note Ordinary files in your workspace folder, not hidden settings Capped at daily frequency by default; needs the SQL access entitlement 

Caregence-pairs-Genie-style-conversational-agents-for-hospital,-home,-care-and-hospice-operatorsGenie Agents: What They Are, and How to Set One Up

None of this works without trustworthy data underneath it, and that’s the job of a Genie Agent (what Databricks called a Genie Space through mid-2026). It’s a curated, conversational layer built over roughly 30 tables or views at most in current releases, configured with three things: instructions that teach Genie your vocabulary, SQL examples that anchor its query generation, and trusted assets, certified metrics Genie reuses instead of regenerating from scratch. 

Setting up Genie Agent step-by-step workflow

A user’s question becomes SQL, runs on a SQL Warehouse, and comes back with the generated query attached for verification. Nothing here is a black box. As of mid-2026, Agent Mode adds iterative reasoning for open-ended “why” and “what-if” questions, running several queries and returning a cited report instead of a single number. 

Setting one up is mostly configuration, not code: 

  1. Prerequisites: a Pro or Serverless SQL Warehouse, Unity Catalog SELECT privileges, and well-documented tables (column comments, keys, certified tags).
  2. Create the agent: pick your Unity Catalog sources, name it, attach a warehouse.
  3. Add data assets: start with 5 to 15 tables or Metric Views in one business domain, not the whole warehouse.
  4. Add context: instructions, SQL examples, trusted assets. This is the single highest-leverage step for accuracy.
  5. Add sample questions, and enable Agent Mode if users will ask open-ended, multi-step questions.
  6. Test, benchmark, and publish, then connect it to Genie One through Unity Catalog groups. 

At Inferenz, this is exactly the discipline we bring to Unity Catalog rollouts across healthcare Lakehouse environments: narrow scope first, governance built in from step one, not bolted on after. 

Metric views: why Genie One never gives two different answers 

Ask two people the same business question in different words, and a language model can generate two different SQL statements, and two different numbers. That’s the failure Metric Views were built to close. 

A Metric View is a Unity Catalog object that defines dimensions, measures, joins, and, in current releases, parameters that let one definition answer differently depending on what’s calling it: a dashboard, a Genie Agent conversation, or a Genie One skill. Write a readmission_rate_30d calculation once, certify it once, and every surface on the platform reuses that same logic instead of re-deriving it from scratch. 2026 updates added native median and percentile expressions (useful for skewed metrics like length of stay), cluster-by configuration for large fact tables, and wildcard expressions that cut boilerplate when composing layered views.

Genie Agent or Custom Predictive Agent: Which one do you actually need? 

A Genie Agent answers questions from data that already exists. “What’s our 30-day readmission rate this quarter?” sits squarely in its lane, even with Agent Mode’s deeper reasoning. The moment a question turns forward-looking, “which of my current inpatients is likely to be readmitted, and what should we do about it?”, you need a custom predictive agent built on the Mosaic AI Agent Framework instead. 

This is a genuinely different tool. You own the model choice, the tool calls, the orchestration, and the evaluation, all inside the same Unity Catalog governance boundary. Genie One can call either one the same way: as an MCP-connected tool, or wrapped inside a skill so a business user never sees the machinery underneath. 

Dimension Genie Agent Custom Predictive Agent 
Best for Descriptive, exploratory Q&A over governed tables and Metric Views Predictive, multi-step, tool-calling workflows 
Who owns the logic Databricks-managed reasoning and SQL generation You define the reasoning, tools, and orchestration 
Runs on A SQL Warehouse Model Serving endpoints, MLflow models, UC functions 
Typical output An answer, generated SQL, or a cited Agent Mode report A ranked list, a risk score, or a recommendation 
Reached from Genie One via Direct chat, skills, scheduled tasks An MCP tool connection, or a skill that wraps it 

Turning a readmission risk score into action in healthcare 

Hospital readmissions are one of the most closely watched numbers in healthcare, for good reason. According to CMS, historically about one in five Medicare patients discharged from a hospital are readmitted within 30 days, and the Hospital Readmissions Reduction Program financially penalizes hospitals that exceed their peer benchmark. The gap that matters here isn’t the prediction. It’s what happens between a risk score sitting in a dashboard and a care team acting on it before the patient walks out the door. 

A Genie-One-orchestrated version looks like this:  

  1. A care-coordination lead could define a Genie One skill called weekly_readmission_digest that pulls the latest 30-day readmission metrics from a Genie Agent
  2. It cross-references a custom predictive agent’s high-risk worklist, and formats both into a one-page summary.
  3. A scheduled task then runs that skill every Monday morning and delivers the digest to the unit’s chat thread and inbox, turning a report someone used to assemble by hand into something that simply shows up, grounded in the same governed data and metric definitions used everywhere else on the platform. 

Unifying fragmented clinical data and then acting on it is exactly the kind of work Inferenz does for hospital and home-based care operators moving from reactive reporting to real-time, value-based care, using predictive models built for clinical operations. 

None of this replaces clinical judgment. Any production system touching protected health information still needs to clear your organization’s HIPAA and model-governance review before it influences care.

Tuning accuracy and cost: The discipline behind reliable answers 

Genie One is only as good as the Genie Agents and Metric Views feeding it, so accuracy is a stack-wide habit, not a setting you flip once and forget. 

Start by writing down 20 to 50 representative questions with known-correct answers before touching a single instruction. Prioritize SQL examples and trusted assets over prose instructions, since concrete patterns anchor SQL generation far more reliably than descriptive text. Keep instructions short and free of contradictions, and re-run the benchmark after every schema change or instruction edit. Field reports cite 10 to 40 percent accuracy gains from this loop alone, applied consistently, against an agent nobody ever benchmarks. 

Result What It Means What To Do 
Pass Correct answer, correct grain Keep as a regression test 
Partial Right direction, wrong filter or period Add a targeted SQL example 
Fail, schema Can’t find or join the right tables Add column comments, keys, or a Metric View 
Fail, ambiguity Maps to more than one plausible metric Add a trusted asset to disambiguate 

Cost follows a similar rhythm. Serverless SQL Warehouses suit the bursty, ad-hoc pattern of conversational analytics better than always-on clusters, and Genie’s query-level attribution makes it possible to track cost per agent and manage the operating model, not just per warehouse, which is what makes chargeback to a specific business unit realistic. Agent Mode and scheduled tasks both add real compute. Budget for them separately from ad-hoc chat, and audit schedules nobody actually reads. 

Contact our Data and AI Experts

The bottom line 

Genie One gives every business team one coworker to talk to, instead of five dashboards and an analyst’s calendar. But the chat interface is the easy part. What makes it trustworthy is everything underneath: Genie Agents that turn governed Unity Catalog data into plain language, Metric Views that keep every surface computing the same number, and custom predictive agents that pick up exactly where descriptive analytics runs out of road. 

Treat the whole stack the way you would any production system: benchmark it, tune it on a short loop, and keep watching it after launch. The organizations already ahead here aren’t the ones with the flashiest chat interface. They are the ones who did the unglamorous data foundation work first. 

Frequently Asked Questions 

Parquet v2 in Azure Databricks: What Changed and Why It Matters

Posted on by spectrics

Summary  

Parquet file format v2 is now generally available for Delta Lake and Apache Iceberg tables in Azure Databricks Runtime 18.1 and above. It swaps in smarter encodings, richer page-level metadata, and INT64 timestamps to shrink file sizes and speed up querieswith zero changes to your existing SQL. Turn it on with a single table property and use REORG TABLE when you want your historical data rewritten too.

Introduction

Global data volumes are on pace to cross roughly 230–240 zettabytes by 2026, according to Statista estimates, and every terabyte of that sits somewhere, on someone’s storage bill. Most of it, if you’re running an Azure Databricks lakehouse, sits in Parquet file format files. So when the format underneath your Delta Lake or Apache Iceberg tables gets a meaningful upgrade, it’s worth fifteen minutes of your attention.

That upgrade is Parquet v2, and unlike a major platform migration, it doesn’t ask you to touch a single line of application code or rewrite a query. It changes how data is physically packed inside the files themselves which, in practice, means smaller files, better compression, and faster reads for the same data you already have.

A quick note on terms, if you’re newer to the stack: Apache Parquet is the columnar file format that stores data by column rather than by row, letting a query engine like Spark read only the columns a query actually needs. Delta Lake is the transactional layer Databricks builds on top of Parquet it adds ACID guarantees, schema enforcement, and time travel. Apache Iceberg is a similar open table format, increasingly used alongside or instead of Delta Lake in mixed-engine environments. Parquet v2 sits one level below both of them, at the file format itself, which is exactly why it works across both table types without any application-level rework.

In this Parquet v2 Azure Databricks guide, we’ll cover what Parquet v2 actually changes compared to Parquet v1, how to turn it on, what to check before you do, and where it fits in a broader Databricks storage optimization strategy.

What Is Parquet v2?

Parquet file formathas been the default columnar storage format for big data platforms for well over a decade, and for good reason, it stores data efficiently and let’s query engines skip columns a query doesn’t touch. In wide tables, that column-pruning alone can cut I/O dramatically; reading two columns out of a hundred means the engine never has to touch the other ninety-eight.

But the original Parquet file formatspec (v1) was designed for a different era of data volumes, and three limitations became increasingly visible as workloads scaled:

  • Integer and string compression left performance on the table.
  • Query engines had limited page-level metadata to use for skipping unnecessary data.
  • Timestamps relied on the older INT96 format, which compressed and filtered poorly.

Parquet v2 addresses all three with more efficient encodings, richer page metadata, and a modern INT64 timestamp format. According to Microsoft’s Azure Databricks documentation and Databricks’ own platform release notes, Parquet v2 is generally available for Delta Lake and Apache Iceberg tables starting in Databricks Runtime 18.1, with support for converting existing data added in Runtime 18.2.

Parquet v1 vs. Parquet v2, at a Glance

CapabilityParquet v1Parquet v2
Timestamp storageINT96INT64 (better compression, more accurate stats)
Integer / string encodingStandard RLE / dictionaryAdds DELTA_BINARY_PACKED and DELTA_LENGTH_BYTE_ARRAY for tighter packing
Page metadataBasic headersRicher v2 data page headers with per-page stats
Predicate pushdownLimited page-level skippingFiner-grained data skipping at the page level
Enable viaDefaultdelta.parquet.format.version / iceberg.parquet.format.version = 2.12.0
Reader compatibilityUniversalBroad, but verify external / third-party readers

What’s New in Parquet v2?

1. Better Compression for Integers and Strings

The headline change is how numeric and string values get stored. Parquet v2 introduces more efficient encoding techniques, including delta-based packing for integers and byte arrays, per the Apache Parquet specification, which typically produce:

  • Smaller Parquet files
  • Better compression ratios
  • Faster decoding during query execution

Independent benchmarking from the DuckDB engineering team gives a useful sense of scale here: enabling Parquet v2’s newer encodings produced files roughly 30% smaller with 15% faster writes under Snappy compression, and about 11% smaller with 24% faster writes under zstd, across their test datasets. On highly sequential data, think auto-incrementing IDs or evenly spaced timestamps, the gains were far more dramatic, with some columns shrinking by over 90%.

One honest caveat worth flagging: delta encoding isn’t a universal win. On columns with moderate entropydata that repeats in patterns but isn’t cleanly sequential, delta encoding can occasionally produce larger files than v1, because it turns repeating values into effectively random deltas that compress worse. It’s a good reason to test against a representative sample of your own tables rather than assuming uniform gains.

2. Improved Data Page Headers

Parquet files are divided into pages, and in Parquet v2, each page carries richer metadata, allowing Databricks to determine whether a page contains relevant data before it’s ever read. That directly improves:

  • Predicate pushdown
  • Data skipping
  • Query performance on filtered datasets

If your query filters sales data for a single month, Databricks can skip the pages that don’t contain data from that period, cutting the volume of data scanned, and the compute cost that comes with it.

3. INT64 Timestamps Instead of INT96

Older Parquet files stored timestamps in the INT96 format. Parquet v2 replaces it with the more efficient INT64 timestamp format, which brings:

  • Better compression
  • More accurate statistics
  • Faster filtering on timestamp columns
  • Better compatibility with modern analytics engines

Since most enterprise data warehouse tables lean heavily on timestamp columns event logs, transaction records, IoT telemetry this single change tends to have an outsized, noticeable impact on real-world query performance.

How to Enable Parquet v2

If you’re using Databricks Unity Catalog managed tables, Azure Databricks may automatically upgrade compatible tables to Parquet v2 for you. To enable it manually, set the table property directly.

Existing Delta table

ALTER TABLE table_name
SET TBLPROPERTIES (
'delta.parquet.format.version' = '2.12.0'
);

Existing Iceberg table

ALTER TABLE table_name
SET TBLPROPERTIES (
'iceberg.parquet.format.version' = '2.12.0'
);

New Delta table

CREATE TABLE table_name (...)
TBLPROPERTIES (
'delta.parquet.format.version' = '2.12.0'
);

New Iceberg table

CREATE TABLE table_name (...)
USING iceberg
TBLPROPERTIES (
'iceberg.parquet.format.version' = '2.12.0'
);

Existing Data Isn’t Automatically Converted

This is the detail most teams miss on their first pass.

Changing the table property only affects new data written after the change. Your existing Parquet files stay in their original format, which means a single table can temporarily hold a mix of Parquet v1 and Parquet v2 files side by side.

If you want your historical data converted too, Databricks Runtime 18.2 and above provides the REORG TABLE command:

REORG TABLE table_name
APPLY (
SET PARQUET (FORMAT_VERSION = '2.12.0')
);

This rewrites every existing file using Parquet v2, so the entire table benefits from the format change.

Can You Roll Back?

Yes. If you hit a compatibility issue downstream, you can convert the table back to Parquet v1 just as easily:


REORG TABLE table_name
APPLY (
SET PARQUET (FORMAT_VERSION = '1.0.0')
);

This rewrites the data files again and restores the table to the older format a genuinely low-risk way to test Parquet v2 in a non-production environment before committing.

Things to Check Before Enabling Parquet v2

Parquet v2 delivers real gains, but it’s worth verifying compatibility if your data is read outside Databricks. Before flipping the switch, check for:

  • External Apache Iceberg readers that may not yet support Parquet v2. DuckDB’s own engineering team, for instance, has noted that several mainstream query engines still default to writing (and in some cases reading) older Parquet encodings for exactly this reason.
  • Delta Sharing or other external sharing methods confirm recipient tools can actually read Parquet v2 files before you share.
  • Materialized views and streaming tables, which aren’t upgraded automatically and need to be enabled manually.

If your tables are read exclusively from within Databricks, compatibility generally isn’t a concern at all.

Should You Use Parquet v2?

For most Azure Databricks workloads, yes. Parquet v2 offers real advantages without requiring any application-side changes:

  • Reduced storage usage
  • Better compression
  • Faster query execution
  • Improved predicate pushdown
  • Better timestamp handling

If your data is read exclusively within Azure Databricks, enabling Parquet v2 is a low-effort, high-leverage way to improve performance. If external tools, BI platforms, or third-party engines also touch your data, verify compatibility first.

A practical rollout sequence looks like this:

  1. Let Databricks Unity Catalog managed tables upgrade automatically where applicable.
  2. Enable Parquet v2 on Databricks-only tables first.
  3. Run REORG TABLE if you want existing data converted.
  4. Test external readers before enabling Parquet v2 on shared datasets.

Final Thoughts

Parquet v2 is the kind of improvement that works quietly in the background. You keep writing the same SQL, running the same pipelines, using the same BI tools but your data becomes more storage-efficient, and your queries, more often than not, come back a little faster.

For enterprises running Azure Databricks at scale, this is a low-effort upgrade with a real payoff, provided you verify compatibility for anyone reading your data outside the platform. As a Databricks consulting partner, Inferenz has helped enterprise data teams evaluate exactly this kind of platform-level change as part of broader Delta Lake and lakehouse cost-optimization engagements. Through its data engineering and integration services, organizations have optimized storage architectures, improved data performance, and reduced cloud infrastructure costs, where a one-line table property, applied correctly across the right tables, adds up to a measurable line item on the cloud bill.

Frequently Asked Questions

Databricks Data + AI Summit 2026: The Lakehouse Just Became Something Bigger

Posted on by inferenz.manage

Summary 

Databricks Data + AI Summit 2026 was not a feature release, but a declaration. The Lakehouse is no longer just where enterprises store and query data. It is where agents do the job for your business. Here is what changed, what it means, and why it matters now. 

Introduction 

Every year, the tech industry produces a hundred summits that announce things. 

Databricks Data and AI Summit 2026 (DAIS) was different.  

What Databricks put on the table in San Francisco this June was an architectural argument about where enterprise AI is actually headed, and it landed with the kind of coherence that makes you reconsider how you have been thinking about your data stack. 

The theme, if you had to name it, was this: the Lakehouse is now the control plane for the agentic enterprise. Not just a place to store data. The place where agents govern, reason, act, and get held accountable for what they do. 

For Inferenz, a Databricks partner building agentic AI solutions in healthcare and enterprise, several of these announcements land directly in the infrastructure we build on and deploy for clients.  

Here is our read on what mattered most and what you should actually do about it. 

The context problem is finally being taken seriously 

If you have ever deployed an AI model and watched it produce a confidently wrong answer, you already know the core problem DAIS 2026 addressed. 

It is not model quality. It is context. 

As Ali Ghodsi, founder and CEO of Databricks put it during the keynote: “Most enterprise AI today is just guessing with false confidence. If you’re a CFO and AI can’t tell you why margins changed, that’s not an AI problem. That’s a context problem.” 

Genie Ontology is Databricks’s answer to that. It is a live context layer that continuously reads your data, documents, queries, and applications to build a machine-readable map of what your business actually means by its own terms.  

  • What does “active user” mean in your system?  
  • What is your definition of “churn”?  
  • When did your ARR calculation change, and why? 

This is not a static data dictionary someone fills in once and forgets. Genie Ontology updates continuously, weighs sources by authority (similar to how PageRank works), and feeds that knowledge directly into Unity Catalog’s semantic layer.  

The downstream effect: every agent, every dashboard, and every AI-generated report pulls from one shared, authoritative understanding of your business rather than each making its own guesses. 

The company with the best context layer will have a larger AI advantage than the company with the most data. That sentence from the Bain team covering the summit deserves to sit with you for a moment. 

Genie One: an AI coworker that actually knows your business 

Genie One is now generally available, and it is a significant step past what most enterprise AI assistants can actually do. 

  • It connects to over 50 applications, including Gmail, Slack, Teams, Jira, and Confluence.  
  • It can answer questions grounded in your actual governed lakehouse data, draft documents, schedule tasks, monitor changes, and explain why something happened.  
  • On a benchmark of 28 real-world enterprise data questions, Genie answered 84.5% correctly on the first attempt. The best general-purpose coding agent on the same test scored 52.4%. 

The difference is the ontology layer underneath. Genie is not searching documents. It is reasoning against a live, governed representation of your business. That is what separates a useful answer from a plausible one. 

For enterprise teams evaluating where to start with agentic AI, Genie One is the fastest path to ROI for non-technical business users. No seat-based pricing, either. Each user gets 150 DBUs of free LLM usage per month, with pay-as-you-go beyond that. 

LTAP: Forty years of infrastructure debt, addressed 

Here is a problem most enterprises have accepted as permanent: your transactional systems and your analytical systems have always been two separate things. Separate databases, separate formats, ETL pipelines running between them, two slightly different copies of the same data that never quite agreed. 

LTAP (Lake Transactional/Analytical Processing) changes that.  

The mechanics: Lakebase, Databricks’s serverless PostgreSQL database (now at 12 million launches per day), stores transactional data directly in Unity Catalog using Delta and Iceberg formats.  

No ETL. No sync. Hidden copies disappear. Every analytical engine reads the same governed file.

For AI agents, this is foundational. An agent that needs to read a customer’s live order history and then run six months of purchasing analysis currently must query two systems and reconcile two copies of data. With LTAP, there is one copy, one governance layer, and one point of truth. 

New Lakebase capabilities at the summit:  

  • cross-cloud disaster recovery 
  • git-style database branching (spin up a full-fidelity clone of production in sub-seconds for safe testing), and  
  • Lakebase Search, which brings hybrid vector and full-text retrieval natively into PostGRES. 

Lakehouse//RT, powered by a new engine called Reyden, rounds this out with sub-100ms query latency at 12,000 queries per second directly on Delta and Iceberg tables. PointClickCare’s benchmarks showed it running more than a third faster than their prior warehouse, on their own healthcare dataset, without a separate serving system. 

Agent Bricks: The platform that does the other 99% 

Building an AI agent is not hard anymore. The hard part is everything else. 

Memory across sessions. Security when agents execute code. Cost management when agents run at scale. Evaluation. Monitoring. Governance of what they can access. That is the 99% of engineering work that does not show up in demos but determines whether your deployment works in production. 

Agent Bricks is now a full-stack platform for exactly that. Over 100,000 agents have been built on it. AstraZeneca, 7-Eleven, Fox, and Block all run production agents on Agent Bricks. The 2026 expansion added: 

  • Managed agent memory powered by Lakebase, persistent across sessions 
  • MCP-connected retrieval from Unity Catalog and external tools like GitHub, Jira, and Google Drive 
  • Secure sandboxed compute environments for code execution 
  • Multi-model support: OpenAI, Anthropic, Gemini, Qwen, Grok, all governed under Unity Catalog 
  • Omnigent, a meta-orchestration layer for managing agents across different frameworks, models, and tools when your stack is not monolithic 

For teams building with Claude Code SDK, LangGraph, CrewAI, or OpenAI Agent SDKs, Omnigent is the layer that lets these coexist under one governance model instead of sprawling across disconnected stacks. 

Databricks also moved five products into the free tier: Genie Code, Serverless GPUs, Lakebase, Agent Bricks, and Lakeflow Designer. You can now prototype an entire agentic application from data pipeline to agent logic to served endpoint without spending anything. 

Unity AI Gateway: Governance that happens at runtime 

This is the announcement that regulated industries have been waiting for. 

Traditional AI governance asked: who can access which data, which model is approved? That works for humans. It breaks down for agents that act autonomously, spawn subagents, call external tools, and generate outputs at volume. 

Unity AI Gateway governs what agents actually do at the moment they do it.  

  • Hard spend caps.  
  • Real-time PII detection.  
  • Prompt injection prevention.  
  • Full trace capture of every tool call, MCP interaction, and subagent action.  
  • Security policies written in SQL that respond to agent behavior in context, not just static rules applied at the edge. 

At Inferenz, our work in healthcare AI has always required governance to be a first-class concern. What Unity AI Gateway represents is exactly the infrastructure required to move agentic AI from pilot deployments into production clinical environments.  

We cover this in more depth through our Generative and Agentic AI services and the governance architecture that underpins our Caregence platform. 

Is-Your-AI-Infrastructure-Ready-for-Agent-Scale-Governance

OpenSharing: Open standards win again! 

Databricks launched Delta Sharing in 2021 to solve cross-organizational data sharing without copying files. It became the most widely adopted open data-sharing protocol in the industry. 

OpenSharing extends that logic to the full AI stack. Data, models, agent skills, and Genie Agents can now be shared across organizations and clouds via a single Linux Foundation-hosted open protocol. 

The practical enterprise use case is Genie Agent Sharing: share a governed AI interface with a partner or customer, giving them curated access to your data and reasoning capabilities without exposing your underlying logic, proprietary calculations, or source tables. You control what they can ask, how much data they can export, and how many requests they can make. 

SecureConnect removes the networking headache: cross-cloud storage connections without per-recipient firewall configuration. 

What this means for Inferenz clients 

Inferenz is a Databricks partner. We build on this platform. Several of the Databricks summit announcements directly expand what we can deliver: 

  • Genie Ontology strengthens the semantic layer that our healthcare clients need for AI to reason correctly about clinical terms, payer rules, and care metrics without every agent reinventing the definition. 
  • Lakebase and LTAP close the gap between transactional care data and the analytical models that power Caregence predictive risk intelligence. Patient records that update in real time can now feed directly into risk models without ETL delays. 
  • Agent Bricks governance and Unity AI Gateway provide the runtime controls our healthcare deployments require. HIPAA-compliant agentic AI is not just a compliance checkbox. It is an architecture. These capabilities make that architecture standard rather than custom-built for every engagement. 

For enterprise clients working on data and cloud modernization or evaluating where agentic AI fits in their stack, the LTAP architecture eliminates an entire tier of infrastructure that was previously unavoidable. One governed copy of data, one permission model, one source of truth for both operational and analytical AI.

Five things worth acting on now 

Most enterprises left the summit with a list of things to watch. These five are worth starting this quarter. 

Define your semantic layer before your agents do it for you.  

Genie Ontology is only as good as what Unity Catalog already knows. If your organization has never agreed on what “revenue” or “active user” officially means, that conversation is now blocking your AI roadmap. 

Consolidate your database tier.  

Running a separate operational database alongside Databricks? Lakebase and LTAP give you a clear path to one governed system. The git-style branching alone makes the evaluation worth an afternoon. 

Audit your agent governance.  

Most AI pilots have no runtime enforcement. If your governance stops at the data catalog, it is not governance. Unity AI Gateway fixes that, but only if you implement it. 

Prototype on Agent Bricks before building custom.  

Lakebase, Agent Bricks, and Serverless GPUs are all free tier now. There is no budget justification for building a custom agentic stack before you have tested what is already there. 

Treat context as a strategic asset.  

The next AI advantage will not come from model selection. It will come from the organization whose agents have the clearest, most authoritative understanding of what the business means. That is a semantic architecture decision, not a procurement one.

What-Would-Your-Data-Stack-Look-Like-If-It-Was-Built-for-Agents

Final thought 

The debate in enterprise AI used to be about which model to choose. DAIS 2026 made clear that this was always the wrong question. 

The model is not the constraint. The architecture around it is. 

Context, governance, live data, and runtime control are the infrastructure that determines whether your AI delivers or stalls. Databricks built a year’s worth of announcements around exactly those four things. 

For enterprises that have been waiting for the infrastructure to catch up to the ambition, it just did.

Frequently Asked Questions 

Maximizing Speed, Revenue & Insights with the Right Data Warehouse Design 

Posted on by inferenz.manage

Summary

Data warehouse design decides how fast your teams get answers, how much they trust the numbers, and how easily you can scale analytics and AI. This guide breaks down architecture approaches, schema options, and implementation patterns, with clear “use when” guidance for each. 

Introduction: Understanding Data Warehouse Designs 

In today’s data-driven world, organizations rely on data warehouses to consolidate, organize, and analyze massive volumes of information. But building a data warehouse is not just about storing data – it’s about designing it in a way that maximizes speed, accuracy, and business value. 

A data warehouse design determines how data is structured, stored, and accessed. It affects everything from query performance to reporting accuracy, machine learning capabilities, and regulatory compliance.  

Choosing the right design is crucial because a poorly designed warehouse can slow analytics, increase costs, and lead to incorrect business decisions. 

Why Data Warehouse Design Matters 

  • Performance: Ensures queries run quickly, enabling real-time dashboards and faster decision-making. 
  • Scalability: Supports data growth without costly re-engineering. 
  • Data Quality & Governance: Reduces redundancy, ensures consistency, and provides audit traceability. 
  • Business Alignment: Reflects how the business measures success, making analytics intuitive for end-users. 

The following designs apply to organizations that provide data in batches. Details on warehouse design for organizations that provide real-time data, will be covered separately. 

Simple Data Warehouse Architecture Diagram (3-Layer View) 

Simple Data Warehouse Architecture Diagram (3-Layer View)

Source systems 
ERP, CRM, product apps, files, APIs, event streams 

Ingestion and integration 
ETL or ELT, CDC, data quality checks, standardization 

Warehouse and modeling layers 
Architecture approach (Kimball, Inmon, Data Vault, Anchor) 
Schema design (star, snowflake, galaxy, 3NF) 
Implementation patterns (wide tables, aggregates, hybrid) 

Consumption 
BI tools, dashboards, ad-hoc queries, ML workflows

Data Warehouse Architecture / Design Approaches 

Data warehouse architecture defines the overall strategy and methodology for building a data warehouse, guiding how data is collected, integrated, stored, and accessed for analysis. Unlike individual schema designs that focus on table structures, these approaches provide a high-level blueprint for enterprise data management and analytics. 

Kimball Dimensional Modeling 

Kimball focuses on building dimensional models around business processes, often as data marts that roll up into a broader analytical layer. It is popular because it is easy to understand and fast for BI. 

Kimball Dimensional Modeling

Use when 

  • Business users need intuitive reporting quickly 
  • Requirements are stable and well understood 
  • You want incremental delivery with visible wins 

Best fit 

  • BI dashboards, finance and revenue reporting, sales and marketing analytics 

Typical impact 

  • Faster time to value, strong user adoption, simpler reporting model 

Example scenario 
Marketing needs campaign performance dashboards quickly. Kimball supports focused data marts, conformed dimensions, and fast reporting delivery. 

Inmon top-down approach (enterprise-first EDW) 

Inmon starts with a centralized enterprise data warehouse, usually in normalized 3NF structures. Data marts are derived later for performance and ease of reporting. It takes longer to build but supports consistent enterprise definitions. 

Inmon top-down approach (enterprise-first EDW)

Use when 

  • A single version of truth is required across functions and regions 
  • Governance and standardization are priorities 
  • Integration across many systems is complex 

Best fit 

  • Large enterprises with strict KPI consistency and governance needs 

Typical impact 

  • Higher trust in metrics, stronger control, better enterprise alignment 

Example scenario 
A global company needs standardized KPIs across regions. Inmon supports centralized definitions and reduces conflicting reports. 

Data Vault modeling (scalable and auditable) 

Data Vault organizes data into Hubs (business keys), Links (relationships), and Satellites (descriptive history). It separates raw ingestion from business logic, which helps with change, traceability, and long-term integration. 

Data Vault modeling (scalable and auditable)

Use when 

  • Source systems change often 
  • Historical tracking and auditability matter 
  • You expect new domains and sources over time 

Best fit 

  • Telecom, finance, insurance, regulated industries, complex enterprise integration 

Typical impact 

  • Faster onboarding of sources, fewer breakages from schema drift, stronger lineage 

Example scenario 
A telecom adds new products and pricing models often. Data Vault reduces the blast radius of change and keeps history intact. 

Anchor modeling (high adaptability in a normalized style) 

Anchor modeling uses Anchors (core entities), Attributes, and Ties (relationships). It is designed for frequent change. You can add new attributes without redesigning large parts of the model. 

Anchor modeling (high adaptability in a normalized style)

Use when 

  • Business attributes and rules change frequently 
  • You need flexibility without major table redesign 
  • You want a long-lived model that evolves with the business 

Best fit 

  • Fast-changing SaaS environments and evolving product analytics needs 

Typical impact 

  • Less rework, easier schema evolution, better maintainability 

Example scenario 
A SaaS business keeps adding customer attributes. Anchor modeling supports this without downtime-heavy redesigns. 

CTA

Schema designs: logical and physical models 

Schemas define how tables are structured. They affect join patterns, usability, and performance. 

Star schema 

The star schema is a central fact table that connects to denormalized dimensions. It is widely used because it is fast and easy to query. 

Star schema

Use when 

  • You want fast BI and simple reporting 
  • Many users run ad-hoc analysis 
  • Business teams need clear dimensions and metrics 

Best fit 

Dashboards, KPI reporting, analytics that depend on speed 

Snowflake schema 

Dimensions are normalized into sub-tables, often to manage hierarchies and reduce redundancy. It can save storage but adds joins. 

Snowflake schema

Use when 

  • Dimension hierarchies are complex 
  • Storage efficiency matters 
  • Slightly slower queries are acceptable 

Best fit 

Large product catalogs, structured hierarchies, domains with frequent hierarchy updates 

Galaxy schema (fact constellation) 

Multiple fact tables share dimension tables. It supports cross-process analytics across domains like orders, shipments, returns, and inventory. 

Galaxy schema (fact constellation)

Use when 

  • You need analysis across multiple business processes 
  • Shared dimensions create enterprise views of the customer or product 

Best fit 

E-commerce, supply chain, end-to-end customer journey analytics 

Normalized 3NF enterprise warehouse 

Highly normalized tables reduce redundancy and enforce integrity. It is strong for integration and governance, but reporting queries can be slower without downstream marts. 

Normalized 3NF enterprise warehouse

Use when 

  • The warehouse is a system of record 
  • Audit and regulatory demands are high 
  • Integration consistency matters more than reporting speed 

Best fit 

Enterprise integration layer, regulated domains, “one source of truth” requirements 

Physical implementation patterns 

These patterns influence performance and cost once architecture and schemas are chosen. 

Wide tables 

Wide tables store facts and useful attributes together in a denormalized structure. They reduce joins and speed up analytics and ML feature use.

Wide tables 

Use when 

  • ML feature pipelines suffer from join complexity 
  • Query speed is more important than storage 
  • Data models are stable enough for denormalization 

Best fit 

  • AI feature stores, customer 360 analytics, experimentation analytics 

Hybrid designs 

Hybrid designs mix approaches and optimize each layer for its job. A common pattern is: raw integration layer (often Data Vault), then dimensional marts for BI, then wide tables for ML and performance-heavy use cases.

Hybrid designs

Use when 

  • You support BI, advanced analytics, and ML together 
  • Workloads differ by team and tool 
  • You want both governance and speed 

Best fit 

  • Modern enterprise data platforms where one model cannot satisfy every use case 

Practical selection guide 

  • Fast reporting and quick wins: Kimball + star schema 
  • Enterprise consistency and governance: Inmon or 3NF EDW feeding marts 
  • Frequent source change and deep audit needs: Data Vault 
  • Rapidly evolving attributes and long-term flexibility: Anchor modeling 
  • Cross-domain process analytics: Galaxy schema 
  • Performance-heavy analytics and ML features: Wide tables 
  • Mixed workloads across BI and AI: Hybrid layered approach 

Conclusion 

The best data warehouse design is the one that fits your business reality, not the one that looks best on a whiteboard. Every architecture choice shapes what happens downstream: dashboard speed, reporting trust, integration effort, governance strength, and how ready your teams are for advanced analytics and AI. 

For most U.S. enterprises, the smartest path is to separate concerns. Use a strong data warehouse architecture for integration and traceability, choose the right data warehouse schema design for reporting, and apply performance patterns like wide table design only where they make sense. In many environments, that naturally leads to a hybrid data warehouse architecture, where Data Vault modeling supports scalable ingestion, Kimball dimensional modeling powers BI adoption, and curated layers enable ML without breaking reporting. 

Whether you choose the Inmon approach, a pure dimensional strategy, or a layered model, the goal stays the same: reduce friction between data teams and decision-makers. When the design is right, analytics becomes faster, costs become predictable, and the warehouse becomes a stable foundation for growth, modernization, and AI-driven outcomes. 

CTA 2

Frequently Asked Questions

    FinOps in Real-World Practice: Transforming Cloud Spend into Strategic Value

    Posted on by inferenz.manage

    Summary

    As cloud adoption grows in fintech, cloud cost management becomes harder because usage and pricing shift every hour. FinOps helps teams link spend to real outcomes like cost per transaction, fraud checks, and feature delivery. Learn how fintech teams apply FinOps in daily operations, using tagging, visibility, forecasting, and automation to turn cloud spend into strategic value.Cloud spend to strategic value with FinOps

    Introduction

    Cloud makes fintech faster. Teams can ship features quickly, scale during peak transaction windows, and run analytics without buying hardware. 

    The catch is simple: consumption pricing turns every new workload into a variable cost line. And in fintech, workloads spike for reasons that feel “business as usual” such as payout cycles, fraud bursts, seasonal lending, or a partner API change.

    FinOps exists to keep that variability from becoming chaos. The FinOps Foundation defines FinOps as an operational framework and cultural practice that maximizes business value from cloud and technology through timely, data-driven decisions and shared financial accountability across engineering, finance, and business teams. 

    This guide shows what FinOps looks like when you apply it day to day in fintech environments, where speed, governance, and predictability matter at the same time.

    Why fintech teams feel cloud cost pressure sooner

    Fintech cloud usage tends to concentrate in a few expensive areas:

    • Always-on customer experiences: low-latency apps, APIs, identity, and observability.
    • Risk and fraud analytics: streaming, feature stores, model training, and bursty compute.
    • Data platforms: warehouses and lakehouses that grow quietly with retention, audit, and regulatory needs.
    • Security controls: logging, monitoring, scanning, and encryption overhead that is necessary, but rarely “free.”

    And cloud spend keeps climbing across industries. Gartner forecasts public cloud end-user spending at $723.4B in 2025. 

    So, the question for fintech leaders is rarely “should we spend less?” It’s “how do we spend with intent, and prove it with numbers?”

    That’s where FinOps becomes a business discipline, not a billing exercise.

    Three phases of FinOps

    FinOps in daily operations: the practices that change outcomes

    1) Unify teams around shared financial accountability

    FinOps works when engineering and finance stop treating cloud cost as someone else’s job. The practical shift looks like this:

    • Finance gets clear ownership views: by product, environment, and business line.
    • Engineering gets fast feedback loops: cost impact is visible before and after a release.
    • Product and leadership get unit economics: cost per transaction, cost per active customer, cost per underwriting decision, cost per fraud check.

    Example
    Before launching a new real-time payments feature, the platform team reviews expected throughput, storage growth, and observability overhead with finance. They agree on a target unit cost (say, cost per 1,000 transactions) and track it weekly. If unit cost rises, teams investigate whether it came from higher log volume, unbounded retries, or an over-sized compute tier.

    What Inferenz typically adds here is the operating model: who owns which cost domains, what gets reviewed weekly versus monthly, and how teams turn cost data into decisions without slowing delivery.

    2) Make cost visibility usable with tagging, allocation, and clean data

    Visibility is more than a dashboard. It’s consistent, trusted allocation that supports action.

    For fintech teams, a tagging and allocation baseline usually includes:

    • Product / business line
    • Environment (prod, staging, dev)
    • Cost center
    • Workload type (API, batch, streaming, ML training, BI)
    • Data classification (helps align cost with governance and audit needs)

    Tools such as AWS Cost Explorer and Azure Cost Management help, but they depend on clean tagging and consistent account structure.

    Quick win that matters:
    Create a “no tag, no launch” gate for production infrastructure as a guardrail that prevents unknown spend from becoming permanent.

    Data quality and governance blog

    3) Shift from month-end surprises to real-time decisions

    FinOps teams operate on short cycles because cloud changes daily. When cost signals arrive a month later, the money is already gone.

    In real practice, fintech teams do things like:

    • Auto-shutdown non-critical environments after hours
    • Rightsize compute based on actual utilization
    • Use commitment planning (Savings Plans, Reserved Instances) where usage is steady
    • Move storage to lower-cost tiers with policy-based lifecycle rules

    FinOps Foundation guidance frames this as a loop across visibility, optimization, and operations. 

    Example
    A fraud model retrains nightly. The pipeline grew over time and now runs on larger nodes than needed. FinOps flags the change in cost per training run, the data team confirms stable runtime targets, and the platform team applies right-sizing and schedule controls. The end result is predictable spend without weakening detection.

    4) Treat forecasting like a product KPI, not a finance exercise

    Forecasting is where fintech teams often struggle because demand is real-time and spiky. Still, you can forecast well if you forecast the right thing.

    Instead of asking, “What will AWS bill be next month?”, focus on:

    • forecasted unit volumes (transactions, API calls, onboarding checks)
    • expected model usage (training runs, inference calls)
    • the unit cost curve (cost per 1,000 events)

    Then tie cloud spend to those business drivers.

    Cloud spend management remains a widespread challenge, which makes forecasting discipline a differentiator.

    Where Inferenz fits: building data pipelines that merge billing exports, usage telemetry, and product metrics so forecasts reflect how the business actually runs, beyond what the invoice says.

    How fintech teams scale FinOps by maturity

    How fintech teams scale FinOps by maturity

    Common roadblocks and how to get past them

    Three obstacles to scaling FinOps

    • Resistance from teams
      Engineers may assume cost controls will slow delivery. Fix that by using automation, clear thresholds, and fast feedback, not manual approvals.
    • Complex pricing and confusing bills
      Cloud pricing is hard. The fix is to translate billing into “engineering terms” such as runtime, storage growth, egress, and query patterns.
    • Inconsistent governance
      If tagging rules vary by team, visibility collapses. Standardize the minimum required tags and enforce them with policy.

    Recommended practices for sustainable FinOps adoption in fintech

    Recommended practices for sustainable FinOps adoption in fintech

    1. Start with 1 or 2 high-impact domains
      Common picks: fraud analytics pipeline, core API platform, data warehouse.
    2. Define unit economics everyone understands
      Cost per transaction, cost per onboarded customer, cost per underwriting decision.
    3. Automate guardrails
      Idle cleanup, tag enforcement, budget alerts, and anomaly detection.
    4. Make the weekly FinOps review short and decisive
      Review top cost drivers, anomalies, and planned changes for next week.
    5. Tie spend to business outcomes
      Revenue growth, authorization rates, fraud loss reduction, time-to-ship, or customer experience KPIs.

    Final thoughts

    FinOps becomes valuable in fintech when it connects cloud spend to product reality: usage, risk controls, and customer outcomes. With the right allocation, unit economics, and automation, teams keep speed while making spend predictable and defensible.

    CTA Contact Us

    Frequently asked questions

    Data Quality & Governance: The Strategic Blueprint for Sustainable Organizational Success

    Posted on by inferenz.manage

    In an era defined by data, organizations are navigating a fundamental paradox: they are data-rich but insight-poor. The sheer volume of information, intended to be a strategic asset for every Fortune 100 contender and nimble startup alike, often becomes a source of complexity and confusion. 

    Without a structured approach, this asset quickly turns into a liability, leading to flawed strategies, missed opportunities, and eroded trust. The solution is not more data, but better, more reliable data, managed under a coherent strategic framework. This is the essence of data quality and governance: the strategic blueprint for transforming data chaos into a sustainable competitive advantage.

    The data imperative: Why trustworthy data is non-negotiable

    In today’s digital economy, every critical business function relies on data. From personalizing a customer journey to optimizing supply chains with big data analytics, the accuracy and reliability of the underlying information dictate the outcome. 

    Poor Data Quality directly translates to poor decision-making, misguided strategies, and inefficient operations. When leadership cannot trust the numbers presented in a Business intelligence dashboard, strategic planning becomes a game of guesswork, and the organization’s ability to respond to market shifts is severely compromised. 

    Trustworthy data is the foundational prerequisite for organizational agility and resilience.

    The Promise of AI: unlocking potential through data excellence

    AI initiatives promise to change industries. However, AI is not magic; it is a sophisticated consumer of data. 

    Machine learning algorithms are only as effective as the data they are trained on. Biased, incomplete, or inaccurate data leads to flawed models, unreliable predictions, and potentially disastrous business outcomes. A staggering number of AI projects fail to move from pilot to production, not because the algorithms are weak, but because the data foundation is unstable. 

    True AI Readiness begins with a deep commitment to data quality and governance, ensuring that your most advanced initiatives are built on a bedrock of trust.

    Setting the stage: Data Quality and Governance as your strategic foundation

    Viewing data quality and governance as mere compliance obligations or IT-centric tasks is a critical strategic error. Instead, they must be positioned as the central pillars of an organization’s data strategy: the keys to why Data Quality and governance drive digital success. A robust governance framework acts as the control system, defining the rules of engagement for all data assets, while a commitment to data quality ensures those assets are fit for purpose. 

    Together, they create an environment where data can be confidently accessed, shared, and leveraged to drive innovation and create tangible business value, forming the strategic blueprint for enduring success.

    The indispensable foundation: Unpacking Data Quality and Governance

    Before building a data-driven enterprise, leaders must understand the core components of its foundation. Data Quality and data governance are distinct but deeply interconnected disciplines. One cannot succeed without the other. Governance provides the structure, rules, and accountability, while quality represents the tangible, measurable state of the data itself.

    Defining Data Quality: dimensions of trust

    Data Quality is not a single attribute but a multi-dimensional concept, often defined by standards like ISO/IEC 25012. To be considered high-quality, data must meet several key criteria:

    • Accuracy: Does the data correctly reflect the real-world object or event it describes?
    • Completeness: Are all the necessary data points present?
    • Consistency: Is the data uniform across different systems and applications?
    • Timeliness: Is the data available when it is needed for analysis and decision-making?
    • Uniqueness: Are there duplicate records that could skew analysis and operations?
    • Validity: Does the data conform to the defined format, type, and range (e.g., a valid email address format)?

    Assessing and improving data across these dimensions is the first step toward building a trusted data ecosystem.

    Defining Data Governance: The strategic framework for control and value

    Data governance frameworks provide the structure for managing an organization’s data assets. This is not about restricting access but about enabling responsible use. A comprehensive framework establishes the necessary policies, standards, procedures, and controls. It clearly defines who can take what action, with which data, under what circumstances, and using which methods. These Data policies are the rulebook that guides every user in the organization, ensuring that data is handled securely, ethically, and in a way that maximizes its value while minimizing risk.

    The Intertwined Nature: How robust governance ensures data Integrity and quality

    Data governance is the engine that drives Data Quality. Without a governance framework, efforts to clean up data are temporary fixes at best. Governance establishes the roles and processes needed to maintain data excellence over time. It defines Data stewards who are accountable for specific data domains, implements procedures for data entry and validation, and provides a mechanism for resolving data issues. This structured approach is what ensures Data Integrity: the overall accuracy, consistency, and reliability of data throughout its lifecycle. Governance transforms data quality from a reactive, project-based activity into a proactive, embedded discipline.

    The Cost of Neglect: Addressing Data Trust Issues and Mitigating Reputational Damage

    Ignoring data quality and governance carries a steep price. Inaccurate customer data leads to poor service and lost sales. Flawed financial data can result in compliance failures and hefty fines. 

    According to Gartner, the average organization loses $12.9 million annually due to poor data quality. Operationally, bad data creates immense inefficiency as employees spend valuable time hunting for reliable information or correcting errors. Perhaps most damaging is the erosion of trust. When customers lose faith in your ability to manage their information, or when executives can no longer rely on reports to guide the business, the resulting reputational damage can be irreversible.

    Crafting Your Strategic Blueprint: Core Pillars of Effective Governance

    An effective data governance program is not a one-size-fits-all solution. It must be a carefully designed blueprint tailored to the organization’s specific needs, maturity, and strategic goals. However, several core pillars are universally essential for success.

            1. Roles and Responsibilities: Empowering Data Stewardship and Leadership

    Data governance is a team sport that requires clear accountability. A successful program establishes a hierarchy of roles, starting with executive sponsorship from a Chief Data Officer (CDO) or a similar leader who champions the vision. The most critical on-the-ground role is that of Data stewards. These individuals, typically business experts from various departments, are entrusted with overseeing specific organizational data assets. They are responsible for defining data standards, monitoring quality, and ensuring that Data policies are followed within their domain, acting as the crucial link between IT and the business.

            2. Master Data Management (MDM): Achieving a Single, Trusted View of Key Data

    Many organizations struggle with fragmented data, where information about a single customer, product, or supplier exists in multiple, often conflicting, versions across different systems. Master data management (MDM) is the discipline and technology used to resolve this chaos. MDM creates a single, authoritative “golden record” for critical data entities. By creating a central, trusted source of master data, organizations remove inconsistencies. They simplify processes. They make sure all analytics and decisions are based on a shared, accurate view of the business.

            3. Designing Your Target Operating Model for Data Governance: Structure and Workflow

    A Target Operating Model (TOM) for data governance outlines how people, processes, and technology will work together to execute the governance strategy. It defines the structure of the governance council or committee, the workflows for data issue resolution, and the processes for creating and enforcing policies. The TOM serves as the practical implementation plan, detailing how governance will be embedded into the daily operations of the business. It clarifies reporting lines, meeting cadences, and the escalation paths for data-related issues, turning abstract policy into concrete action.

            4. The Data Lifecycle: Ensuring Quality and Governance from Inception to Archival

    Data is not static; it has a lifecycle that begins with its creation and ends with its eventual archival or deletion. Applying data quality and governance principles consistently across this entire journey is essential for maintaining trust and value over time.

    Holistic Data Lifecycle Management: A Continuous Journey

    Effective data lifecycle management requires a holistic view. This includes managing data creation, storage, usage, sharing, and eventual retirement. Governance procedures must be applied at each stage. For example, data quality checks should be implemented at the point of data entry, access controls must govern its use, and retention policies should dictate how long it is stored. This continuous oversight ensures that Data Integrity is maintained from start to finish.

    Data Lineage: Tracing Data’s Journey and Transformations

    Data lineage provides a complete audit trail of data’s journey through an organization’s systems. It documents where data originated, what transformations it underwent, and how it is used in various reports and applications. This visibility is crucial for building trust. Data lineage is essential for fixing errors. It helps analyze the impact before system changes. It also meets rules for tracking data for regulatory compliance. When a user can see the source and history of a data point, they have more confidence in its accuracy.

    Quality and Governance in Modern Data Architectures

    The rise of big data technologies, Data lakes, and Cloud computing has introduced new challenges for governance. The sheer volume, velocity, and variety of data make manual oversight impossible. To adapt, modern governance frameworks must use metadata management tools to automatically list data assets in a data lake. Implement governance controls within cloud platforms. Design a “data middle platform” that enforces policies and quality checks on data as it moves between systems. This ensures a single, governed Data Lake environment rather than a data swamp.

    Managing Data Migration and Integration with Quality in Mind

    Data migration and system integration projects are high-risk moments for Data Quality. Moving data between systems without proper planning can introduce errors and corrupt information. A robust governance framework is essential to guide these projects. It requires data profiling before migration to find quality problems. It sets clear mapping rules for integration. It demands thorough checks and reconciliation after moving data. This ensures no data is lost or damaged during transfer.

    Driving Business Value: Turning Trustworthy Data into Strategic Advantage

    The ultimate goal of data quality and governance is not simply to have clean, well-managed data. It is to leverage that data as a strategic asset to drive tangible business outcomes, create competitive differentiation, and foster sustainable growth.

    Powering Better Decision-Making and Business Intelligence

    The most direct benefit of a strong data governance program is the improvement in strategic and operational decision-making. When executives and managers trust the data in their Business intelligence dashboards and reports, they can make faster, more confident choices. Governed data eliminates the ambiguity and debate over whose numbers are correct, allowing teams to focus on analyzing insights and taking action rather than questioning data validity.

    Fueling Advanced Analytics and AI Initiatives

    High-quality, well-documented, and easily accessible data is the essential fuel for advanced analytics and AI Initiatives. Predictive maintenance models, customer churn predictions, and other machine learning algorithms depend on a rich history of reliable data. A governance framework makes sure data is available. It ensures data lineage is clear. It also confirms data is suitable for advanced applications. This greatly raises the chance of success for an organization’s top projects.

    Enhancing Customer and User Experience with Reliable Data

    Reliable data is the foundation of a superior customer experience. A single, accurate view of the customer, enabled by MDM, allows for true personalization, targeted marketing, and seamless service interactions. When a user contacts support, they expect the agent to have their complete and correct history. Inaccurate or incomplete data leads to frustrating, disjointed experiences that damage customer loyalty and brand perception.

    Optimizing Business Processes and Operational Efficiency

    Clean, consistent, and timely data is a powerful catalyst for operational excellence. It streamlines business processes by removing the friction caused by data errors. For example, accurate product data reduces shipping errors in logistics, correct supplier data ensures timely payments in procurement, and valid employee data simplifies HR and payroll processes. These efficiencies compound across the organization, reducing operational costs and freeing up employee time for more value-added activities.

    Enabling Data Accessibility and Responsible Data Sharing

    A common misconception is that governance is about locking data down. In reality, good governance supports responsible data access. By establishing clear ownership, security classifications, and access policies, governance creates a framework for Data Accessibility where data can be shared confidently and securely across the organization. This “data democratization” empowers more users to access the data they need to perform their jobs effectively while ensuring that sensitive information is protected.

    Mitigating Risk & Ensuring Trust: The Compliance and Security Imperative

    In an increasingly regulated world, robust data governance is no longer optional; it is a fundamental component of risk management. It provides the necessary controls and oversight to protect the organization from regulatory penalties, security breaches, and the associated reputational fallout.

    Navigating the Complex Landscape of Regulatory Compliance

    Organizations today face a complex web of privacy laws and data protection regulations, such as the EU’s GDPR and the California Consumer Privacy Act (CCPA). Adhering to these rules requires a deep understanding of what data is collected, where it is stored, and how it is used. Data governance frameworks manage regulatory compliance. They document data processing activities, handle consent, and enforce policies. These ensure data is used according to legal rules.

    Proactive Risk Management: Data Audit and Data Observability for Continuous Oversight

    Instead of reacting to data breaches or quality failures, leading organizations are adopting proactive risk management strategies. This includes regular data audits to assess compliance with internal policies and external regulations. The emerging field of Data Observability goes a step further, using automated tools to continuously monitor the health of data pipelines and systems. This provides real-time alerts on data quality degradation, schema changes, or anomalous data patterns, allowing teams to identify and resolve issues before they impact the business.

    Establishing Clear Data Issue Escalation and Resolution Processes

    Even with the best controls, data issues will inevitably arise. A key function of data governance is to establish clear, efficient procedures for identifying, escalating, and resolving these issues. A defined data issue escalation path ensures that when a user spots a problem, they know exactly who to report it to. This process guarantees that the right Data stewards and technical teams are engaged quickly to perform root cause analysis and implement a lasting solution, preventing the same issue from recurring.

    The Human Element & Cultural Transformation: Building a Data-Driven Organization

    Ultimately, technology and policies are only part of the solution. Achieving a truly data-driven organization requires a cultural transformation. It means fostering a shared sense of responsibility for data quality across all departments and empowering every employee with the skills and knowledge to treat data as a critical enterprise asset. This cultural shift, supported by strong leadership and continuous training, is what turns a governance blueprint into a living, breathing reality.

    Conclusion

    Data quality and governance are not mere technical exercises or compliance hurdles; they are the strategic blueprint for sustainable success in the digital age. By implementing a robust framework built on clear roles, effective processes, and enabling technologies like Master data management, organizations can transform their data from a chaotic liability into their most powerful asset. This change helps make smarter decisions. It improves the customer experience and increases operational efficiency. It also creates a necessary base for successful AI initiatives.

    The journey begins by treating data as a core business function, not an IT afterthought. It requires building a culture of accountability where everyone understands their role in preserving Data Integrity and upholding quality. By committing to this blueprint, organizations can confidently navigate the complexities of the modern data landscape, mitigate risk, and unlock the full potential of their information assets. By investing in this plan, your organization can do more than manage data. It can actively use data to find new ways to innovate. It can reduce risks and gain a lasting competitive edge.

    The Far-Reaching Impact of Model Drift and its Data Drama

    Posted on by inferenz.manage

    Background Summary

    Model drift is more than a real data science headache, it’s a silent business killer. When the data your AI relies on changes, predictions falter, decisions suffer, and trust erodes. This guide explains what drift is, why it affects every industry, and how a mix of smart monitoring, robust data pipelines, and AI-powered cleaning tools can keep your models performing at their peak.-Imagine launching a new product, rolling out a service upgrade, or opening a flagship store after months of preparation, only to find customer complaints piling up because something invisible changed behind the scenes. In AI, that invisible culprit is often model drift.

    Your model worked perfectly in testing. Predictions were accurate, dashboards lit up with promising KPIs.  But months later, results dip, costs climb, and customer trust erodes. What changed? 

    The data feeding your model no longer reflects the real world it serves. 

    This article breaks down why that happens, why it matters to every industry, and how modern tools can stop drift before it damages outcomes.

    What is “Data Drama”?

    “Data drama” means wrestling with disorganized, inconsistent, or incomplete data when building AI solutions, leading to model drift. Model drift refers to the degradation of a model’s performance over time due to changes in data distribution or the environment it operates in.

    Think of it as junk in the trunk: if your AI is the car, bad data makes for a bumpy ride, no matter how powerful the engine is.

    Picture a hospital that wants to use AI to predict patient health risks:

    • Patient names are sometimes written “Jon Smith,” “John Smith,” or “J. Smith.”
    • Some records are missing phone numbers or have outdated addresses.
    • The hospital’s old records are stored in paper files or weird formats.

    Even if the AI is “smart,” it struggles to learn from such confusing information. There are three primary types of drifts that affect the scenarios:

    • Data drift (covariate shift): The input distribution P(x) changes. Example: new user behavior, seasonal trends, new data sources.
    • Concept drift: The relationship between features and target P(y∣x) changes. Example: fraud tactics evolve customer churn reasons shift.

    Label drift (prior probability shift): The distribution of P(y) changes. Common in imbalanced classification tasks.

    Why is this a problem?

    • Silent failures: Drift isn’t always obvious models can keep running, just poorly.
    • Bad decisions: In finance, healthcare, or logistics, this can mean misdiagnoses, delays, or big financial losses.
    • Customer frustration: Imagine getting your credit card blocked for every vacation you take.
    • Wasted resources: Fixing a broken model after damage is harder (and costlier) than preventing it.
    • Time wasted: Engineers spend up to 80% of their time cleaning data instead of building useful solutions.
    • Hidden mistakes: Flawed data can make the AI give wrong answers—like approving the wrong credit card application or missing a fraud alert.
    • Loss of trust: If the AI presents inaccurate results, users quickly lose faith in the technology.

    Why is it hard to catch?

    • Most production pipelines don’t monitor live feature distributions or prediction confidence.
    • Business KPIs may degrade before engineers notice any statistical performance drop.
    • Retraining isn’t always feasible daily, especially without label feedback loops.

    How can we solve the data drama?

    Today, AI itself helps clean and fix messy data, making life easier for both techies and non-techies. Here’s a step-by-step technical approach for managing drift in production systems: 

               1. Track key statistical metrics on input data:

        • Population stability index (PSI)
        • Kullback-leibler divergence (KL Divergence)
        • Kolmogorov-smirnov (KS) test
        • Wasserstein distance (for continuous features)

    Implementation example:

    Tools: Evidently AI, WhyLabs, Arize AI

              2. Monitoring model performance without labels

    If you can’t get real-time labels, use proxy indicators:

        • Confidence score distributions (are they shifting?)
        • Prediction entropy or uncertainty variance
        • Output class distribution shift

    Example using fiddler AI:


    # Detect divergence from training output distributions

              3. Retraining pipelines & model registry integration

    Build retraining workflows that:

        • Pull recent production data
        • Recompute features
        • Revalidate on held-out test sets
        • Re-register the model with metadata

    Example stack:

        • Feature store: Feast / Tecton
        • Training pipelines: MLflow / SageMaker Pipelines / Vertex AI
        • CI/CD: GitHub Actions + DVC

    Registry: MLflow or SageMaker Model Registry

    Tools & solutions 

    This is broken down by stages of the solution pipeline:

    1.Understanding what data is missing

    Before solving the problem, you need to identify what is missing or irrelevant in your dataset.

    ToolPurposeFeatures
    Great expectationsData profiling, testing, validationDetects missing values, schema mismatches, unexpected distributions
    Pandas profiling / YData profilingExploratory data analysisGenerates auto-EDA reports; useful to check data completeness
    Data contracts (openLineage, dataplex)Define expected data schema and sourcesEnsures the data you need is being collected consistently

     

     2. Data collection & logging infrastructure

    To fix missing data, you need to collect more meaningful, raw, or contextual signals—especially behavioral or operational data.

    ToolUse CaseIntegration
    Apache kafkaReal-time event loggingCaptures user behavior, app events, support logs
    Snowplow analyticsUser tracking infrastructureWeb/mobile event tracking pipeline for custom behaviors
    SegmentCustomer data platformCollects customer touchpoints and routes to data warehouses
    OpenTelemetryObservability for servicesTrack service logs, latency, API calls tied to user sessions
    Fluentd / LogstashLog collectorsIntegrate service and system logs into pipelines for ML use

     

    3. Feature engineering & enrichment

    Once the relevant data is collected, you’ll need to transform it into usable features—especially across systems.

    ToolUse CaseNotes
    FeastOpen-source feature storeManages real-time and offline features, auto-syncs with models
    TectonEnterprise-grade feature platformCentralized feature pipelines, freshness tracking, time-travel
    Databricks feature storeNative with Delta LakeIntegrates with MLflow, auto-tracks lineage
    DBT + SnowflakeFeature pipelines via SQLGreat for tabular/business data pipelines
    Google vertex AI feature storeFully managedIdeal for GCP users with built-in monitoring

     

    4. External & third-party data integration

    Some of the most relevant data may come from external APIs or third-party sources, especially in domains like finance, health, logistics, and retail.

    Data typeTools / APIs
    Weather, locationOpenWeatherMap, HERE Maps, NOAA APIs
    Financial scoresExperian, Equifax APIs
    News/sentimentGDELT, Google Trends, LexisNexis
    Support ticketsZendesk API, Intercom API
    Social/feedbackTrustpilot API, Twitter API, App Store reviews

     

    5. Data observability & monitoring

    Once new data is flowing, ensure its quality, freshness, and availability remain intact.

    ToolCapabilities
    Evidently AIData drift, feature distribution, missing value alerts
    WhyLabsReal-time observability for structured + unstructured data
    Monte CarloData lineage, freshness monitoring across pipelines
    Soda.ioData quality monitoring with alerts and testing
    DatafoldData diffing and schema change tracking

     

    6. Explainability & impact analysis

    You want to make sure your added features are actually helping the model and understand their impact.

    ToolUse Case
    SHAP / LIMEExplain model decisions feature-wise
    Fiddler AICombines drift detection + explainability
    Arize AIReal-time monitoring and root-cause drift analysis
    Captum (for PyTorch)Deep learning explainability library

     

    Why model drift is every business’s problem

    Model drift may sound like a technical glitch, but its consequences ripple across industries in ways that hurt revenue, efficiency, and trust.

    • Healthcare – A drifted model can misread patient risk levels, causing missed diagnoses, delayed interventions, or unnecessary tests. In critical care, this can directly affect patient outcomes.
    • Finance – Inconsistent data patterns can produce incorrect credit scoring or flag legitimate transactions as fraudulent, frustrating customers and damaging loyalty.
    • Retail & E-commerce – Changing buying behavior or seasonal demand shifts can lead to inaccurate demand forecasts, resulting in overstock that ties up cash or stockouts that push customers to competitors.
    • Manufacturing & supply chain – Predictive maintenance models can miss early signs of equipment wear, leading to unplanned downtime that halts production lines.

    The common thread?

    • Revenue impact – Poor predictions lead to lost sales opportunities and operational waste.
    • Compliance risk – In regulated sectors, drift can create breaches in reporting accuracy or fairness obligations.

    Brand reputation – Customers and partners lose trust if decisions feel inconsistent or incorrect.

    The cost of ignoring model drift

    The business case for tackling drift is backed by hard numbers:

    • Data quality issues cost organizations an average of $12.9 million annually.
    • For predictive systems, downtime can cost $125,000 per hour on an average depending on the industry.
    • Recovery from a drifted model, retraining, redeployment, and regaining lost customer trust, can take weeks to months, costing far more than prevention.

    Implementing automated drift detection can reduce model troubleshooting time drastically.  Early intervention can prevent revenue losses in industries where decisions are AI-driven.

    In other words, the cost of not acting is often several times higher than the cost of building proactive safeguards.

    From detection to prevention

    Drift management is about more than catching problems, it’s about designing systems that keep models healthy and relevant from the start.

    ApproachWhat It Looks LikeOutcome
    ReactiveModel performance dips → business KPIs drop → engineers scramble to investigate.Higher downtime, lost revenue, longer recovery cycles.
    ProactiveContinuous monitoring of data and predictions → alerts trigger retraining before business impact.Minimal disruption, sustained model accuracy, preserved customer trust.

    Why proactive wins:

    • Reduces firefighting and emergency fixes.
    • Ensures AI systems adapt alongside market or operational changes.
    • Turns drift management into a competitive advantage, keeping predictions accurate while competitors struggle with outdated models.

     

    Takeaway

    In fast-moving markets, your AI is only as good as the data it learns from. Drift happens quietly, but its effects ripple loudly across customer experiences, operational efficiency, and revenue. By combining continuous monitoring with adaptive retraining, businesses can turn model drift from a costly disruption into a controlled, measurable process.

    The real win is beyond the fact that it fixes broken predictions. Now you can build AI systems that grow alongside your business, staying relevant and reliable in any market condition.

    QA in the Modern Data Stack: Using Python, Zephyr Scale & Unity Catalog for End-to-End Quality Assurance

    Posted on by inferenz.manage

     

    Integrated QA framework using Python, Zephyr Scale & Unity Catalog

    Introduction

    Quality Assurance (QA) in the software world has moved beyond functional testing and interface validation. As modern enterprises shift toward data-centric architectures and cloud-native platforms, QA now involves ensuring data accuracy, integrity, governance, and system compliance end to end.

    In a recent enterprise project, I worked on migrating a legacy Customer Relationship Management (CRM) system to Microsoft Dynamics 365 (MS D365). It wasn’t merely a technology shift. It involved moving large data volumes, aligning new business rules, setting up strong governance layers, and ensuring uninterrupted business operations.

    In this article, I’ll share how QA was handled across this transformation using Zephyr Scale for test management, Python for automation, and Databricks Unity Catalog for governance and access control.

    QA challenges in migrating to Microsoft Dynamics 365

    Migrating from a legacy CRM to a modern cloud platform brings unique QA challenges. The main focus areas included:

    Focus AreaQA ObjectiveCommon Issues
    Data ValidationEnsure data integrity and accuracy post-migrationMissing, duplicate, or corrupted records
    Functional TestingValidate end-to-end workflows across Bronze → Silver → Gold layersBreaks in business logic or incomplete process flow
    Integration TestingVerify KPI accuracy in downstream systemsData mismatch or inconsistent calculations

    This was my first experience in a hybrid QA setup—where data engineering and cloud CRM validation worked together. Automation became essential from the start.

    Test management with Zephyr Scale in Jira

    We used Zephyr Scale within Jira to manage all QA activities. It ensured complete traceability from test case creation → execution → defect resolution.

    The test planning followed an iterative Agile structure:

    SprintPhaseDescription
    Sprint 1System Integration Testing (SIT)Validation of data flow, transformations, and business rules
    Sprint 2User Acceptance Testing (UAT)Final stage readiness checks before production deployment

    Sample migration test case

    Objective: Validate that data from the Bronze layer is accurately transferred to the Silver layer.

    Steps:

    1. Query record counts in the Bronze schema.  
    2. Query corresponding counts in the Silver schema.  
    3. Compare totals and sample values.  
    4. Confirm no data loss or duplication.

    Zephyr Scale offered complete visibility—allowing both QA and business teams to align quickly and demonstrate readiness during go-live reviews.

    Writing effective test scenarios and cases

    In a data migration project, QA must cover both systems—the old CRM and the new MS D365—along with the underlying Databricks Lakehouse layers.

    The following scenarios formed the backbone of our testing effort:

    • Data validation: Ensuring every record from the old subscription is fully and accurately migrated.
    • Schema validation: Confirming the data flow through Bronze → Silver layers, with cleansing and normalization (3NF) applied.
    • KPI validation: Verifying 16 business KPIs for accuracy, completeness, and correct duration (annual or quarterly).
    • Governance validation: Checking access permissions, lineage, and audit logs for compliance.

    This structured approach ensured coverage across the technical and business sides of the migration.

    QA automation with Python

    Manual validation quickly became impractical with large datasets and frequent syncs. Automation was the only sustainable approach.

    Automated checks included:

    • Record counts between schemas/tables/columns
    • Schema conformity checks in migrated tables
    • Data Validation from Bronze to Silver to Gold
    • Naming convention checks
    • Storage location validations
    • KPI Calculations

    This automation saved countless hours and ensured we caught discrepancies quickly.

    Sample script:

    These automated tests reduced QA time, enabled early detection of errors, and ensured reliable validation across migration batches.

    Unity Catalog: Governance in the data pipeline

    Data governance was as important as data accuracy in this project. Using Databricks Unity Catalog, we centralized security, access, and lineage validation for all datasets.

    As part of QA, we validated:

    Governance CheckQA Objective
    Access ControlEnsure only authorized users can view Personally Identifiable Information (PII).
    Schema LockingValidate that schema versions remain consistent across deployments.
    Audit LoggingConfirm all data access events are recorded and retrievable.

    Testing with Unity Catalog reinforced compliance while maintaining transparency across teams.

    End-to-end QA workflow in the migration

    Each tool contributed to the overall assurance model:

    StepTool UsedQA Outcome
    Test scenario creationZephyr Scale + JiraLinked to user stories for visibility
    Data validationPython automationVerified migration accuracy
    Governance checksUnity CatalogValidated access control and data lineage
    ReportingZephyr dashboardsWeekly QA progress reports

     

    Workflow overview

    StageProcessPrimary ToolQA Outcome
    1Data migration from legacy CRMMigration scriptsSource-to-target data movement
    2Data lake layeringDatabricks (Bronze → Silver → Gold)Data transformation and enrichment
    3Automated validationPythonRecord and schema verification
    4Governance enforcementUnity CatalogRole-based access, lineage, and audit logging
    5Test managementZephyr ScaleTest execution tracking and reporting
    6Issue managementJiraTicketing, sign-off, and visibility

    This structure built confidence through traceability and consistent automation cycles.

    Key takeaways from the CRM to D365 transition

    • Treat CRM migration as a business transformation, not just data movement.
    • Use Zephyr Scale for transparent test tracking.
    • Automate frequent checks using Python to maintain speed and precision.
    • Leverage Unity Catalog for governance assurance and compliance.

    Final thoughts

    Migrating to Microsoft Dynamics 365 while building a modern data stack highlighted how deeply QA intersects with data engineering and governance.

    By combining Zephyr Scale, Python automation, and Unity Catalog, we achieved a QA framework that was:

    • Structured for traceability,
    • Automated for efficiency, and
    • Governed for compliance.


    This foundation now serves as a blueprint for future enterprise migrations, ensuring data trust from ingestion to insight.

    How We Reduced DynamoDB Costs and Improved Latency Using ElastiCache in Our IoT Event Pipeline

    Posted on by inferenz.manage

    Background Summary

    For executives, architects, and healthcare leaders exploring AI-powered platforms, this article explains how Inferenz tackled real-time IoT event enrichment challenges using caching strategies. 

    By optimizing AWS infrastructure with ElastiCache and Lambda-based microservices, we not only achieved a 70% latency improvement and 60% cost reduction but also built a scalable foundation for agentic AI solutions in business operations. The result: faster insights, lower costs, and an enterprise-ready model that can power predictive analytics and context-aware services.

    Overview

    When working with real-time IoT data at scale, optimizing for performance, scalability, and cost-efficiency is mandatory. In this blog, we’ll walk through how our team tackled a performance bottleneck and rising AWS costs by introducing a caching layer within our event enrichment pipeline.

    This change led to:

    • 70% latency improvement
    • 60% reduction in DynamoDB costs
    • Seamless scalability across millions of daily IoT events

    Business impact for enterprises

    • Faster insights: Sub-second enrichment drives better clinical and operational decisions.
    • Lower TCO: Cutting database costs by 60% reduces IT spend and frees budgets for innovation.
    • Scalability with confidence: Handles millions of IoT events daily without trade-offs.

    Future-ready foundation: Supports predictive analytics, patient engagement tools, and compliance reporting.

    Scaling real-time metadata enrichment for IoT security events

    In the world of commercial IoT security, raw data isn’t enough. We were tasked with building a scalable backend for a smart camera platform deployed across warehouses, offices, and retail stores environments that demand both high uptime and actionable insights. These cameras stream continuous event data in real-time motion detection, tampering alerts, and system diagnostics into a Kafka-based ingestion pipeline.

    But each event, by default, carried only skeletal metadata: camera_id, timestamp, and org_id. This wasn’t sufficient for downstream systems like OpenSearch, where enriched data powers real-time alerts, SLA tracking, and search queries filtered by business context.

    To make the data operationally valuable, we needed to enrich every incoming event with contextual metadata, such as:

    • Organization name
    • Site location
    • Timezone
    • Service tier / SLA
    • Alert routing preferences

    This enrichment had to be low-latency, horizontally scalable, and fault-tolerant to handle thousands of concurrent event streams from geographically distributed locations. Building this layer was crucial not only for observability and alerting, but also for delivering SLA-driven, context-aware services to enterprise clients.

    The challenge: redundant lookups, latency bottlenecks, and soaring costs

    All organizational metadata such as location, SLA tier, and alert preferences was stored in Amazon DynamoDB. Our initial enrichment strategy involved embedding the lookup logic directly within Logstash, where each incoming event triggered a real-time DynamoDB query using the org_id.

    While this approach worked well at low volumes, it quickly unraveled at scale. As the number of events surged across thousands of cameras, we ran into three critical issues:

    • Redundant reads: The same org_id appeared across thousands of events, yet we fetched the same metadata repeatedly, creating unnecessary load.
    • Latency overhead: Each enrichment added ~100–110ms due to network and database round-trips, becoming a bottleneck in our streaming pipeline.
    • Escalating costs: With read volumes spiking during traffic bursts, our DynamoDB costs began to grow rapidly threatening long-term sustainability.

    This bottleneck made it clear: we needed a smarter, faster, and more cost-efficient way to enrich events without hammering the database.

    Our event pipeline architecture

    LayerTechnologyPurpose
    Event IngestionApache KafkaStream raw events from IoT cameras
    ProcessingLogstashEvent parsing and transformation
    Enrichment LogicRuby Plugin (Logstash)Embedded custom logic for enrichment
    Org Metadata StoreAmazon DynamoDBSource of truth for organization data
    Caching LayerAWS ElastiCache for RedisFast in-memory cache for org metadata
    Search IndexAmazon OpenSearch ServiceStores enriched events for analytics

    Our solution: using AWS ElastiCache for read-through caching

    To reduce DynamoDB dependency, we implemented read-through caching using AWS ElastiCache for Redis. This managed Redis offering provided us with a high-performance, secure, and resilient cache layer.

    New enrichment flow:

    1. Raw event is read by Logstash from Kafka
    2. Inside a custom Ruby filter:
      • Check ElastiCache for cached org metadata.
      • If cache hit → use cached data.
      • If cache miss → query DynamoDB, then write to ElastiCache with TTL.
    3. Enrich the event and push to OpenSearch.

    Logstash snippet using ElastiCache

    Note: ElastiCache is configured inside a private subnet with TLS enabled and IAM-restricted access.

    Results: performance and cost improvements

    After integrating ElastiCache into the enrichment layer, we saw immediate improvements in both speed and cost.

    MetricBefore (DynamoDB Only)After (ElastiCache + DynamoDB)
    Avg. DynamoDB Reads/Minute~100,000~20,000 (80% reduction)
    Avg. Enrichment Latency~110 ms~15 ms
    Cache Hit RatioN/A~93%
    OpenSearch Indexing Lag~5 seconds<1 second
    Monthly DynamoDB Cost$$$ (~60% savings)

     

    Enterprise-grade benefits of using ElastiCache

    • In-memory speed: Sub-millisecond access time
    • TTL-based invalidation: Ensures freshness without complexity
    • Secure access: Deployed inside VPC with TLS and IAM controls
    • High availability: Multi-AZ replication with automatic failover
    • Integrated monitoring: CloudWatch metrics and alarms for hit/miss, memory usage

    Scaling smarter: enrichment as a stateless microservice

    As our event volume and platform complexity grew, we realized our architecture needed to evolve. Embedding enrichment logic directly inside Logstash limited our ability to scale, debug, and extend functionality. The next logical step was to offload enrichment to a dedicated, stateless microservice, giving us clearer separation of concerns and unlocking platform-wide benefits.

    Evolved architecture:

    Whether deployed as an AWS Lambda function or a containerized service, this microservice became the single source of truth for enriching events in real time.

    Output flow description:

    • Cameras → Kafka
    • Kafka → Logstash
    • Logstash → AWS Lambda Enrichment
    • Lambda → Redis (ElastiCache)
      • If cache hit → Return metadata
      • If cache miss → Query DynamoDB → Update cache → Return metadata
    • Logstash → OpenSearch

    Why it worked: key benefits

    • Decoupled logic:
       By removing enrichment from Logstash, we gained flexibility in testing, deploying, and scaling independently.
    • Version-controlled rules:
       Enrichment logic could now be maintained and versioned via Git making schema updates traceable and deployable through CI/CD.
    • Reusable across teams:
       The microservice exposed a central API that could be leveraged not just by Logstash, but also by alerting engines, APIs, and other consumers.
    • Improved observability:
       With AWS X-Ray, CloudWatch dashboards, and retry logic in place, we had deep visibility into cache hits, fallback rates, and enrichment latency.

    Enterprise-grade security & monitoring

    To ensure the new design was production-ready for enterprise environments, we baked in security and monitoring best practices:

    • TLS-in-transit enforced for all connections to ElastiCache and DynamoDB
    • IAM roles for fine-grained access control across Lambda, Logstash, and caches
    • CloudWatch metrics and alarms for Redis hit ratio, memory usage, and fallback load
    • X-Ray tracing enabled for full latency transparency across the enrichment path

    This architecture proved to be robust, cost-effective, and scalable handling millions of events daily with low latency and high reliability.

    From optimization to transformation

    While caching solved immediate performance and cost challenges, its broader value lies in enabling enterprise-grade AI adoption. By combining IoT enrichment with caching, even healthcare organizations can unlock:

    • Predictive patient care (anticipating risks from real-time signals)
    • Automated compliance reporting for HIPAA and SLA adherence
    • Scalable patient-caregiver coordination through AI-driven scheduling and alerts

    This architecture is a blueprint for how agentic AI can operate at scale in healthcare ecosystems.

    Conclusion

    Introducing caching into the enrichment pipeline delivered more than performance gains. By adopting AWS ElastiCache with a microservice-based model, the system now enriches millions of IoT events with sub-second speed while keeping costs under control. For enterprises, this architecture translates into faster insights for caregivers, stronger SLA compliance, and predictable operating costs.

    The design also creates a future-ready foundation for agentic AI in enterprises. Enriched data can now flow directly into predictive analytics, business tools, and compliance systems. Instead of reacting late, organizations can respond to real-time signals with agility and confidence.

    At Inferenz, we view caching as a strategic enabler for enterprise-grade AI. It allows security platforms to be faster, more resilient, and prepared for the next wave of intelligent automation.

    Key takeaways

    • Cache repeated lookups like org metadata to reduce both latency and cloud database costs
    • Use ElastiCache as a production-grade, scalable caching layer
    • Decouple enrichment logic using microservices or Lambda for better maintainability and control
    • Monitor cache hit ratios and fallback patterns to tune performance in production

    As your system grows, always ask: “Is this database call necessary?”
    If the data is static or semi-static, caching might just be your smartest optimization.

    FAQs

    Data Observability in Snowflake: A Hands-On Technical Guide

    Posted on by inferenz.manage

    Background summary

    In the US data landscape, ensuring accurate, timely, and trustworthy analytics depends on robust data observability. Snowflake offers an all-in-one platform that simplifies monitoring data pipelines and quality without needing external systems. 

    This guide walks US data engineers through practical observability patterns in Snowflake: from freshness checks and schema change alerts to advanced AI-powered validations with Snowflake Cortex. Build confidence in your data delivery and accelerate decision-making with native Snowflake tools.

    Introduction to data observability

    Data observability is the proactive practice of continuously monitoring the health, quality, and reliability of your data pipelines and systems without manual checks. For US-based data teams, this means answering critical operational questions like:

    • Is the daily data load complete and on time?
    • Are schema changes breaking pipeline logic?
    • Are key metrics stable or exhibiting unusual drift?
    • Are these pipeline resources being queried as expected?

    Replacing outdated scripts with automated, real-time observability reduces risk and speeds issue resolution.

    Why Snowflake is the ideal platform for data observability in the US?

    Snowflake’s unified architecture brings data storage, processing, metadata, and compute resources into one scalable cloud platform, especially beneficial for US enterprises with complex compliance and scalability requirements. Key advantages include:

    • Direct access to system metadata and query history for real-time insights.
    • Built-in Snowflake Tasks for scheduling observability queries without external jobs.
    • Snowpark support to embed Python logic for custom anomaly detection and validation.
    • Snowflake Cortex, a game-changing AI observability tool with native Large Language Model (LLM) integration for intelligent data evaluation and alerting.
    • Seamless integration with popular US monitoring and communication tools such as Slack, PagerDuty, and Grafana.

    These features empower US data engineers to build scalable observability frameworks fully on Snowflake.

    Core observability patterns to implement in Snowflake

    1. Data freshness monitoring

    Verify that your critical tables update as expected daily with timestamp comparisons.
    By scheduling this as a Snowflake Task and logging results, you catch delays early and comply with SLAs vital for US business responsiveness.

    2. Trend monitoring with row counts

    Sudden spikes or drops in row counts can signal data quality issues. Collect daily counts and compare to a rolling 7-day average. Use Snowflake Time Travel to audit past states without complex bookkeeping.

    3. Schema change detection

    Changes in table schemas can break consuming applications.
    Snapshotted regularly, this helps detect unauthorized or accidental alterations.

    4. Value and distribution anomalies via Snowpark

    Leverage Python within Snowpark to check data distributions and business logic rules, such as:

    • Null value rate spikes
    • Unexpected new categorical values
    • Numeric outliers beyond thresholds

    For US compliance or finance sectors, these anomaly detections support regulation-ready controls.

    5. Advanced AI checks with Snowflake Cortex

    Snowflake Cortex enables embedding LLMs directly in SQL to evaluate complex data conditions naturally and intelligently. 

    This eliminates complex manual rules while providing human-like explanations for data integrity, rising in demand across US enterprises with AI-driven reporting .

    How it works?

    The basic idea is to leverage LLMs to evaluate data the way a human might—based on instructions, patterns, and past context. Here’s a deeper look at how this works in practice:

    1. Capture metric snapshots
       You gather the current and previous snapshots of key metrics (e.g., client_count, revenue, order_volume) into a structured format. These could come from daily runs, pipeline outputs, or audit tables.
    2. Convert to JSON format
       These metric snapshots are serialized into JSON format—Snowflake makes this easy using built-in functions like TO_JSON() or OBJECT_CONSTRUCT().
    3. Craft a prompt with business logic
       You design a prompt that defines the logic you’d normally write in Python or SQL. For example:

    4. Invoke the LLM using SQL
       With Cortex, you can call the LLM right inside your SQL using a statement like:\

    5. Interpret the output
      The response is a natural language or simple string output (e.g., ‘Failed’, ‘Passed’, or a full explanation), which can then be logged, flagged, or displayed in a dashboard.

    Building a comprehensive observability framework in Snowflake

    A robust framework typically includes:

    • Config tables defining what to monitor and rules to trigger alerts.
    • Scheduled SNOWFLAKE Tasks to execute data quality checks and log metrics.
    • Centralized metrics repository tracking historical results.
    • Alert notifications routed to US-favored channels (Slack, email, webhook).
    • Dashboards (via Snowsight, Snowpark-based apps, Grafana integrations) visualizing trends and failures in real-time.

    Snowflake’s 2025 innovations such as Snowflake Trail and AI Observability increase visibility into pipelines, enhancing time-to-detect and time-to-resolve issues for US data teams.

    Conclusion

    Data observability is crucial for US data engineering teams aiming for trustworthy analytics and regulatory compliance. Snowflake provides an unparalleled integrated platform that brings together data, metadata, compute, and AI capabilities to monitor, detect, and resolve data quality issues seamlessly. By implementing the observability strategies outlined here, including Snowflake Tasks, Snowpark, and Cortex, data teams can reduce manual overhead, accelerate root-cause analysis, and ensure data confidence. Snowflake’s continuous innovation in observability cements its position as the go-to cloud data platform for US enterprises seeking operational excellence and trust in their data pipelines.

    Frequently asked questions (FAQs)