Databricks Partner in South Africa: More Than Just a Badge (2026)
As a Databricks partner in South Africa, we go beyond certifications. We build working data pipelines that deliver real insights, not just slide decks.
Read7 min
Databricks partner
Databricks partner practice — lakehouse, pipelines, governance and agents.
We build on the Databricks lakehouse to unify data engineering, analytics and machine learning on one governed platform. Delta Lake pipelines, Unity Catalog governance, and — since Agent Bricks — production agents that run next to the data instead of copying it somewhere else.
Why Databricks, and who it is actually for
Databricks is the platform we reach for when the hard part is engineering rather than reporting. If your data arrives as streams and semi-structured files rather than tidy tables, if your team writes Python and Spark rather than only SQL, or if machine learning is a first-class workload rather than a side project, the lakehouse earns its keep. If you mostly need a governed warehouse and dashboards with minimal platform engineering, we will usually point you at Snowflake or BigQuery instead — and we would rather say that in the first meeting than the third invoice.
The architecture has not changed as much as the marketing suggests: Delta Lake tables in a medallion layout, bronze for raw and immutable, silver for cleaned and conformed, gold for business-ready and tested. What has changed is the governance layer around it. Unity Catalog has moved from a permissions system to the thing that gives both people and agents a shared definition of what your data means, which matters enormously once something non-human starts querying it at three in the morning.
For South African organisations the deployment question comes first, because Databricks runs on AWS, Azure and Google Cloud rather than on infrastructure of its own. That makes your workspace region a function of the cloud you already use, and it is worth confirming against Databricks' published region availability rather than assuming — we do that as part of the assessment, before anyone signs anything, because the POPIA answer depends on it.
We are also an independent member of Anthropic's Claude Partner Network, which is unusually relevant here: Databricks and Anthropic signed a five-year partnership to make Claude models available natively on the Data and AI Platform, reachable from a SQL query or a serving endpoint without replicating data out to a model vendor. That means the reasoning layer and the governed data sit inside one boundary, and we can be honest with you about which of the two is actually limiting your results.
What we build with Databricks
Lakehouse architecture & medallion design
Workspace, catalog and schema design with a bronze, silver and gold medallion layout on Delta Lake — plus the naming, ownership and testing conventions that keep it navigable after the first six months.
Ingestion & streaming pipelines
Batch and streaming ingestion with Auto Loader, Structured Streaming and declarative pipelines, including change-data-capture from SQL Server, Oracle and Postgres source systems.
Unity Catalog governance
Centralised access control, lineage, data classification, row and column-level security, and the Glossary and Domains work that gives every team — and every agent — one definition of a customer.
Machine learning & MLflow
Feature engineering, training, experiment tracking and model serving with MLflow, including the monitoring that tells you a model has drifted before a business user does.
Agent Bricks & AI applications
Production agents grounded in lakehouse data and vector search, connected to external systems over MCP, with the AI Gateway logging, guardrails and PII detection switched on from the first deployment rather than the first incident.
Cost engineering & platform operations
Cluster policies, serverless sizing, job orchestration and DBU cost attribution by team and workload, so platform spend is a managed number rather than a monthly surprise.
AI on Databricks
What Databricks is doing with AI
At the 2026 Data + AI Summit in June, Databricks pushed hard on one idea: agents belong inside the governance boundary that already covers your data. Here is the stack as it stands in September 2026 and what each piece is for.
Agent Bricks
Databricks' agent platformThe place you build and operate agents in production. It brings model choice, context from the lakehouse and governance into one platform, with built-in tools registered and governed in Unity Catalog rather than scattered across notebooks. The document search subagent is now roughly three times faster than the previous generation.
Managed memory on Lakebase
Shipped at Data + AI Summit 2026Agents can manage their own context and session history through managed memory, backed by Lakebase underneath. It is the difference between an agent that starts every conversation from nothing and one that behaves like a colleague who was in the last meeting.
Lakebase
Fully managed serverless PostgresServerless Postgres with compute and storage decoupled, sitting alongside the lakehouse rather than bolted on. The feature that matters operationally is instant copy-on-write branching: you can branch a production database to debug an AI agent without copying sensitive data into a test environment.
Unity AI Gateway
Runtime governance layerOne runtime governance layer across models, agents, tools and MCP servers. Instead of governing data in one system and AI in another, requests from an agent are subject to the same catalog that governs the tables underneath.
Contextual Service Policies
BetaGovernance that moves past who can access a tool to what the tool may do in a given interaction. An administrator can allow, deny, or require human approval for specific actions — writing to a sensitive folder, pushing code — which is the control most agent deployments discover they need only after something goes wrong.
Glossary & Domains in Unity Catalog
Announced at Data + AI Summit 2026A governed, shared source of business meaning for both people and agents. This is the Databricks answer to the semantic-layer problem: an agent that does not know your definition of an active customer will confidently invent one.
MCP support in Unity Catalog
AvailableAgents connect securely to external systems — Google Drive, Jira, Slack, GitHub — through the Model Context Protocol, with those connections registered and governed in the catalog rather than configured per notebook and forgotten.
Claude models, natively
Five-year Anthropic partnershipDatabricks and Anthropic signed a five-year agreement to offer Claude natively on the Data and AI Platform. Claude is reachable from a SQL query or a model-serving endpoint across AWS, Azure and Google Cloud deployments, so there is no manual data replication to a model vendor and access stays governed by Unity Catalog.
Model choice in Agent Bricks
Multi-vendorAgent Bricks connects Claude, GPT-5, Gemini and other leading models to lakehouse data, vector search and external systems over MCP — with all usage governed through the AI Gateway, including logging, safety guardrails and PII detection. Model selection becomes a per-agent cost and quality decision.
How South African teams use Databricks with us
01
Consolidating a fragmented data estate
Multiple source systems landed into one medallion lakehouse with tested transformations and full lineage, so finance, operations and the regulator stop receiving three different versions of the same number.
02
Streaming and event-driven analytics
Transaction and event streams ingested continuously into Delta tables with quality enforced at each layer, giving fraud, operations and risk teams minutes-old data instead of yesterday's extract.
03
Machine learning that reaches production
Feature pipelines, MLflow experiment tracking, model serving and drift monitoring — built so the model that scored well in a notebook still scores well in November, and someone is alerted when it does not.
04
Document intelligence at scale
Claude models called natively from the lakehouse to extract and classify fields from contracts, claims and supplier invoices, with personal information detected and masked by the AI Gateway before a model sees it.
05
Governed internal agents
Agent Bricks agents grounded in lakehouse data and connected to Jira, Slack or a document store over MCP, with contextual policies requiring human approval for any action that writes rather than reads.
06
Bringing platform cost back under control
Cluster policies, serverless right-sizing, job consolidation and DBU attribution per team — usually the fastest measurable win on an estate that grew organically and has never been reviewed.
From our Databricks practice
What we're building on Databricks.
What we're building, testing and shipping on Databricks — written up as we go.
Databricks questions we get asked
- Should we choose Databricks, Snowflake or BigQuery?
- It depends on your workloads and your team, and we will tell you honestly when Databricks is not the answer. Databricks suits heavy data engineering, streaming and machine learning, and teams comfortable in Python and Spark. Snowflake suits strong governance, data sharing and business-user AI with less platform engineering. BigQuery suits Google Cloud-centric teams and serverless analytics. We build on all three, which is exactly why the free AI assessment starts with your constraints rather than our preferences.
- Where does Databricks run, and can our data stay in South Africa?
- Databricks is deployed on top of AWS, Azure or Google Cloud rather than on its own infrastructure, so your workspace region follows the cloud you already use — and all three of those providers operate South African regions. Because service and region availability changes, we confirm the specific region against Databricks' published availability as part of the assessment rather than assuming it, and we document the result for your information officer before any data moves.
- What is Agent Bricks, and how is it different from just calling an API?
- Calling a model API from a notebook gives you an answer and no governance. Agent Bricks gives you an agent platform where tools are registered and governed in Unity Catalog, context comes from the lakehouse and vector search, external systems connect over MCP, and every call runs through an AI Gateway with logging, guardrails and PII detection. The difference shows up the day your compliance team asks what an agent accessed last Tuesday.
- Can we use Claude models on Databricks?
- Yes, natively. Databricks and Anthropic signed a five-year partnership to offer Claude models directly through the Data and AI Platform, available from a SQL query or a model-serving endpoint across AWS, Azure and Google Cloud deployments. Because the models are served inside the platform, you are not replicating data out to a separate model vendor and access remains governed by Unity Catalog. As an independent member of Anthropic's Claude Partner Network we use Claude where reasoning depth earns its cost, and cheaper models where it does not.
- Do we need Unity Catalog if we already have access controls?
- You need it more than you think, and the reason has changed. Unity Catalog began as centralised permissions and lineage, which is valuable but arguably optional on a small estate. It is now the layer that gives agents a governed definition of your business through Glossary and Domains, and that enforces Contextual Service Policies on what a tool may actually do. Once something non-human is querying your data, catalog governance stops being hygiene and starts being the control.
- How do we stop Databricks costs running away?
- Cost on Databricks is mostly a design problem rather than a pricing problem. We set cluster policies so nobody spins up a cluster twenty times the size of the job, right-size serverless workloads, consolidate overlapping jobs, and attribute DBU consumption by team and workload so the conversation about spend has names attached to it. On estates that grew organically this is usually the fastest measurable win in an engagement.
- Are you an official Databricks partner?
- We describe ourselves as a Databricks partner practice: that phrase describes our delivery capability on the platform, not a signed programme membership, and Automation Architects is independent of Databricks, Inc. and not affiliated with, sponsored by or endorsed by it. Our only formal programme membership is Anthropic's Claude Partner Network. We would rather be precise about that than let a badge do the talking.
Let's build it on Databricks
Tell us what you're trying to achieve and we'll map the right approach on Databricks — no obligation.
Book a free AI assessmentDatabricks and the Databricks logo are trademarks of Databricks, Inc.. Automation Architects is an independent consultancy and is not affiliated with, sponsored by, or endorsed by Databricks, Inc.. References to Databricks describe the technologies we implement and do not imply certification of any specific outcome.