Enterprise Data Platform
I designed and built a centralized data platform that integrates data from more than 20 operational, financial, marketing, and analytical source systems into Snowflake.
The platform provides a reliable foundation for reporting, operational alerts, self-service analytics, and AI-assisted data access. It replaces separate, source-specific integrations with a shared architecture for ingestion, transformation, governance, and monitoring.
Data ingestion
The platform uses two primary ingestion methods:
- Airbyte replicates operational database tables into Snowflake. It handles standard database synchronization while reducing the amount of custom ingestion code.
- Python ETL pipelines integrate systems that require specialized logic, including REST APIs, SQL queries, SFTP transfers, SharePoint files, and CSV uploads.
Across the 20-plus source systems, the Python pipelines handle:
- Authentication and credential management
- API pagination and rate limits
- Full and incremental loading
- Schema and data-type validation
- Record normalization and deduplication
- Source-specific business rules
- Retry and error-handling logic
- Bidirectional API synchronization
Some integrations only load data into Snowflake. Others also send processed data back to operational systems through APIs.
Snowflake data platform
Snowflake is the platform’s central storage and processing layer. Data from the source systems is organized into separate raw, transformed, and reporting-ready structures.
Transformation queries:
- Standardize names, dates, identifiers, and data types
- Remove duplicate or invalid records
- Join related data from multiple systems
- Apply shared financial and operational rules
- Produce stable tables and views for downstream consumers
This separation prevents dashboards from depending directly on changing source-system structures. It also gives reporting and AI tools consistent interfaces to data from all 20-plus integrations.
Semantic and presentation layers
Power BI semantic models sit between Snowflake and the final dashboards. They define relationships, calculated measures, time-based logic, and governed business definitions.
For example, a metric such as year-to-date revenue is defined once and reused across reports. This reduces conflicting calculations and produces consistent results across business functions.
The curated data supports:
- Executive and operational Power BI dashboards
- Self-service business analysis
- Conversational analytics
- AI skills grounded in governed data
- Snowflake alerts
- Microsoft Teams notifications
Pipeline execution
Custom pipelines run as scheduled container jobs in Azure Container Apps. Each pipeline is packaged in a Docker image and stored in Azure Container Registry.
A shared Python framework reads the job configuration, selects the correct pipeline, and executes it. This allows integrations for more than 20 systems to share common infrastructure while keeping their extraction and transformation logic isolated.
The execution framework provides:
- Independent pipeline execution
- Environment-based configuration
- Reusable database and API utilities
- Structured logging
- Consistent exception handling
- Scheduled and on-demand execution
- Centralized operational monitoring
Monitoring and alerts
Pipeline failures and selected business conditions generate automated notifications. Snowflake evaluates configured conditions, while webhooks deliver alerts to Microsoft Teams.
This makes failures visible without requiring manual inspection of every pipeline. It also allows the platform to monitor business events, not only technical job status.