Data teams across industries are rethinking how to build and run systems that go beyond simply storing information, transforming data into genuine intelligent insights. These systems also need to be interoperable. AI models, feature pipelines, business intelligence (BI) reports, and batch jobs often span multiple teams and engines. Achieving cross-boundary data sharing without duplication or refactoring has become a primary requirement.
Previously, enterprises relied on a two-tier architecture: a data warehouse optimized for business intelligence and reporting, and a data lake designed for large-scale AI and machine learning (ML). This separation incurred numerous costs, including complex data migration, specialized engineering, and duplicate data storage between systems, with little to no synchronization.
Cloudera’s open data lake warehouse unified architecture addresses this challenge by consolidating analytics (BI and ad-hoc queries) and AI (predictive and generative AI) workloads onto a single, controlled data infrastructure. Leveraging open table formats like Apache Iceberg, this unified data architecture helps enterprises “drive computing power into the data” (rather than the other way around) and lays the foundation for running AI workloads closer to the data. AI workloads on the intelligent lake can run directly on controlled, versioned, high-quality data.
As a leading data and AI platform company, Cloudera is commit to applying AI technology to enterprise data in complex environments. Leveraging its mature open-source infrastructure, Cloudera delivers a consistent cloud experience that integrates public cloud, data centers, and edge computing.
The Importance of Open Infrastructure for Running AI Workloads
Over the past decade, enterprises have gradually realized that performance and scalability alone are not enough; flexibility and interoperability are the keys to long-term success. This is especially true for AI workloads, which rely on the ability to access different data sources, frameworks, and tools without limit by proprietary formats or systems.
Against this backdrop, open table formats such as Apache Iceberg have reshaped the architecture of data platforms. Iceberg separates the logical definition of a table from its physical storage layout, allowing multiple engines and frameworks to read and write the same data with full transactional guarantees. This openness supports the continuous evolution of infrastructure and the adoption of new computing engines without rewriting existing processes.
Running a production-grade pipeline requires a unified platform that connects data, models, and governance mechanisms across all stages of the AI lifecycle. At its core are feature engineering pipelines, which continuously transform raw structured, semi-structured, and unstructured data into features usable for AI while maintaining data lineage and reproducibility for model training and evaluation.
Beyond traditional machine learning, generative single data and metadata plane. Furthermore, a scalable inference layer is crucial for the secure and efficient deployment and operation of these models.
As AI workloads become increasingly multimodal and agent-based, access to directories and metadata is becoming more critical. AI pipelines, retrieval systems, and autonomous agents all rely on metadata to discover datasets, reproduce training states, and maintain data lineage. Open directories provide these systems with a universal way to query, register, and track datasets, regardless of where or how they are processed.
A Cloudera’s open infrastructure helps enterprises support a wide range of analytics, predictive, and generative AI workloads.
Cloudera’s unified data and AI platform
Cloudera’s open data lake warehouse architecture, built on open infrastructures such as Apache Iceberg and REST Catalog, integrates data engineering, analytics, and AI into a single, controlled architecture. The platform is design so that workloads—whether analytics or AI—run where the data resides. By eliminating the cumbersome steps of data migration or replication, teams can build a lifecycle encompassing data ingestion, transformation, analysis, and model operations, with complete data lineage and governance capabilities.
Figure 1: Cloudera’s data and AI platform built on open infrastructure (Apache Iceberg)
Next, we will review how the various components of the Cloudera platform (Figure 1) support teams in building machine learning pipelines and generative AI applications, covering all stages of the data and AI lifecycle from ingestion to inference, while operating as a unified, interoperable platform. Each component of the platform is built on open standards, ensuring flexibility and interoperability across environments.
Storage: Apache Iceberg
Apache Iceberg, the foundation of Cloudera’s Lakehouse architecture, is an open, versioned, and transactional table format. Iceberg supports schema evolution, data versioning, and atomic operations, ensuring consistent operation of analytics and AI workloads on the same controlled data. Cloudera provides a controlled and versioned infrastructure that ensures different models, hints, or retrieval tasks are based on a consistent and traceable view of data.
Iceberg’s native features, such as pattern evolution, and AI datasets This approach aligns perfectly with the evolution of AI. In Cloudera’s Intelligent Lakeware, feature storage, training datasets, and retrieval corpora can all share the same Iceberg table, freezing a consistent view for training through snapshot technology while continuously receiving new data for inference. This design reduces the barriers between analytics tables and dedicated AI storage.
Data Ingestion: Cloudera Data in Motion
Cloudera DataFlow, built on Apache NiFi, lays the foundation for continuous data migration to an intelligent lake warehouse. It enables low-latency data ingestion from various enterprise data sources, including databases, APIs, IoT devices, and event logs, supporting both batch and streaming workloads. NiFi natively integrates with the latest innovations of Apache Iceberg, allowing data write directly to the open data lake warehouse architecture without intermediate storage. The tight coupling between NiFi and Iceberg simplifies data pipeline complexity and brings data ingestion closer to the Open Tables format itself.
In real-time application scenarios, NiFi, Apache Kafka, and Apache Flink together form an event-driven data ingestion architecture. NiFi handles data orchestration and routing, Kafka provides persistent streaming, and Flink performs real-time data augmentation before data is persist to Iceberg. This design ensures that data remains fresh and controllable for downstream consumers. This continuous flow… Multimodal data flow is the core driving force behind AI workloads in smart lake warehouses. By continuously providing real-time data in a consistent governance manner through Iceberg tables, enterprises can provide timely, domain-specific information to generative AI systems, thereby making RAG pipelines and agent workflows more accurate, reliable, and stable.
Table of Contents: Cloudera Iceberg REST Catalog
Cloudera Iceberg REST Catalog, based on the open REST specification, provides a centralized and interoperable metadata service, allowing third-party engines that support the open specification, such as Snowflake, Redshift, and Databricks, to access Iceberg tables with zero copies. This is crucial for enterprises, as they are no longer limit to a single computing engine provided by a single platform, thus gaining the flexibility to choose the most suitable computing resources based on business needs. Users can use their preferred tools, while Cloudera’s security and governance policies ensure consistency across all data types and environments.
Figure 2: Cloudera’s Iceberg REST Catalog achieves interoperability with third-party engines.
This catalog layer is crucial for feature engineering pipelines, agent workflows, and retrieval system dynamics, enabling them to dynamically find and access controlled datasets. AI agents can query Iceberg tables using the REST Catalog, much like querying a knowledge graph of enterprise data. They can discover available tables, interpret their schemas, and analyze table metadata (such as partitions, snapshots, and lineage) to determine the dataset to use.
Security and Governance: Cloudera SDX
Cloudera Shared Data Experience (SDX) is a unified security and governance framework that encompasses services from data ingestion to inference. SDX provides a unified layer for data lineage, auditing, access control, and policy enforcement, ensuring that workloads inherit the same security model regardless of where they run. It integrates with enterprise identity systems (LDAP, SSO, OAuth) and supports fine-grained, role- and attribute-based access control for both structured and unstructured data.
By combining SDX with an open data lake warehouse unified architecture, Cloudera ensures that data, models, and AI agents operate within the same controlled boundary, providing transparency, reproducibility, and trust for analytics and generative AI workloads.
Cloudera Data and AI Services
A unified service layer integrates the various functionalities the team needs for transformation, analysis, and…deployAI. All operations are based on the same controlled data.
Data Engineering: Cloudera Data Engineering is built on open-source Apache Spark and Apache Airflow, providing serverless services that enable the building, orchestration, and scaling of data pipelines directly on Iceberg tables, thus providing reliable, reproducible ETL and feature pipelines for analytics and AI work in hybrid environments.
AI Services: Cloudera’s AI service layer enables full lifecycle operation of AI, from model training and fine-tuning to secure deployment. All stages are natively run on the Iceberg platform and within the same controlled data architecture. This service integrates model development, registration, and inference into a unified workflow, achieving seamless integration of data engineering and AI operations.
Figure 3: Cloudera AI’s AI Workbench and Inference Services
Cloudera AI Workbench
Cloudera AI Workbench is a collaborative environment for data scientists, analysts, and engineers to develop, fine-tune, and test models. It integrates notebooks, low-code application builders (AMPs), and dedicated studios covering all stages of AI development. To accelerate AI development and deployment, Cloudera AI Workbench supports four AI studios, bridging the gap between business and technology teams and fostering collaboration on AI projects.
- Synthetic Data Studio generates synthetic datasets for testing and model training when real-world data is limited or constrained.
- Fine-Tuning Studio leverages enterprise-grade datasets to tune open-source underlying models to improve relevance and accuracy.
- RAG Studio builds RAG pipelines that connect large language models (such as OpenAI, Anthropic, and Amazon Bedrock) with relevant private data to generate context-aware and relevant outputs.
- Agent Studio supports the creation of multi-step agent workflows, leveraging models, MCPs, APIs, and internal data sources to automate domain-specific tasks.
These capabilities all run on an open data lake warehouse architecture based on the Iceberg infrastructure, enabling teams to access the data needed for specific tasks in a controlled, zero-replication manner.
Cloudera MCP Server
Cloudera is also expanding the openness of its AI platform with a range of emerging MCP services, including the open-source Cloudera AI Workbench MCP Server. Designed for AI system integration, this service enables agent and tool invocation capabilities within the AI Workbench. It provides a framework for large language models to securely interact with Cloudera AI Workbench features and components, bringing models, data, and applications into automated enterprise workflows. In this architecture, agents can reason, execute, and automate tasks within a trusted and regulated Cloudera environment, meeting the security, controllability, and auditability requirements of regulated industries.
Cloudera AI Inference Service
Cloudera AI Inference Service brings models to production environments through auto-scaling, high availability, and end-to-end observability. This service supports traditional machine learning models and large language models, providing predictions and responses with low latency. Models can be deployed as REST or gRPC endpoints with enterprise-grade security, ensuring reliable and consistent access for applications and agents.
Cloudera AI Registry is integrated into the inference layer, providing centralized model lifecycle management with an MLflow-compatible API for tracking, version control, artifact storage, and traceability. Users can choose from a variety of open and enterprise language models, such as Llama, Cohere, Gemma, and Mistral.
The inference layer also includes built-in monitoring and observability, allowing teams to track latency, throughput, and model bias while maintaining complete data lineage and compliance through SDX governance. This ensures that model predictions are interpretable and traceable, a key requirement for enterprise-grade AI.
The future is driven by AI, and AI is driven by data
The success of AI depends not only on the capabilities of models or intelligent agents but also on the data architecture. Intelligent Lakehouse provides this foundation, unifying analytics, operations, and AI workloads onto a single, controlled data plane. Built on open standards, it ensures seamless interoperability of data, metadata, and models across different tools, cloud platforms, and teams. IDC predicts that by 2028, 60% of Chinese enterprise data platforms will have built HTAP architectures to unify transaction processing and analytics workloads, thereby supporting AI agents and enabling real-time data access and continuous intelligence.
Cloudera AI Workbench, AI Inference Service, and the integrated AI Registry together form a data-to-AI lifecycle based on an open lakeware architecture. This technology stack is built directly on controlled Iceberg tables and open metadata access, ensuring that every model, cue, and agent operates on trusted, versioned data.
The future of enterprise AI will no longer be defined by proprietary technology stacks but by open infrastructure that unifies data, governance, and intelligence through shared standards and transparent interoperability.
To learn more about how to securely prepare, integrate, and analyze data at scale using Cloudera, check out our product demo or sign up for a free 5-day trial.
