Databricks Architect

Role: Senior Data\-bricks Architect Location: REMOTE Domain\- Retail / Grocery Work Type: Contract Experience: 10\+ years overall IT experience, with 5\+ years in Databricks / Cloud Data Engineering / Data Architecture Role Overview We are looking for an experienced Senior Databricks Architect to lead the architecture and technical execution of a large\-scale migration for a leading Retail/Grocery client. The architect will be responsible for designing the target Databricks Lakehouse architecture, defining migration patterns, guiding the data engineering team, and ensuring successful modernization of existing Talend\-based ETL/ELT workloads onto Databricks. The role requires strong hands\-on expertise in Databricks, Apache Spark, SQL, cloud data platforms, data architecture, ETL modernization, and migration strategy, along with a good understanding of Retail/Grocery business processes and data domains. Key Responsibilities Architecture \& Solution Design · Lead the end\-to\-end architecture for migration of Talend ETL/ELT workloads to Databricks. · Define the target\-state Databricks Lakehouse architecture, including data ingestion, transformation, storage, orchestration, governance, and consumption layers. · Design scalable, secure, highly available, and cost\-optimized data solutions. · Define architecture standards, design patterns, coding standards, and best practices for Databricks development. · Establish migration frameworks and reusable patterns for converting Talend jobs into Databricks/Spark\-based pipelines. · Evaluate existing Talend jobs and determine appropriate migration strategies, including re\-platforming, re\-engineering, consolidation, or retirement. · Provide technical leadership for complex data pipelines and integration scenarios. Talend to Databricks Migration · Analyze existing Talend workflows, jobs, mappings, dependencies, schedules, and data transformations. · Develop migration strategies for batch and incremental data processing workloads. · Translate Talend transformations and business rules into PySpark/SQL/Databricks implementations. · Identify opportunities to simplify and optimize legacy ETL processes during migration. · Define data reconciliation and validation strategies to ensure parity between Talend and Databricks outputs. · Establish migration sequencing based on business criticality, dependencies, complexity, and risk. · Support migration of high\-volume and business\-critical data pipelines. Databricks \& Data Engineering · Design and implement solutions using Databricks, Apache Spark, Delta Lake, PySpark, and SQL. · Design Medallion Architecture (Bronze, Silver, Gold) and appropriate data processing patterns. · Implement incremental processing, CDC, SCD Type 1/Type 2, data quality, error handling, and audit frameworks. · Optimize Spark workloads, Delta tables, SQL queries, partitioning, clustering, and job execution. · Leverage Delta Live Tables / Lakeflow capabilities, Workflows, Unity Catalog, and other Databricks platform services where appropriate. · Design data pipelines integrating structured and semi\-structured data from databases, files, APIs, and enterprise applications. · Establish monitoring, logging, alerting, operational support, and performance\-management patterns. Cloud \& Integration · Strong experience with at least one major cloud platform, preferably Azure or AWS. · Design integration between Databricks and cloud storage, databases, messaging platforms, APIs, and enterprise applications. · Experience with technologies such as ADLS/S3, Azure Data Factory, Kafka, REST APIs, relational databases, and cloud\-native services is highly desirable. · Define secure connectivity, networking, secrets management, and access\-control patterns. Retail / Grocery Domain · Work closely with business and technology stakeholders to understand Retail/Grocery data requirements. · Experience with retail data domains such as: o Product / Item o Store / Location o Customer / Loyalty o Sales / Transactions o Pricing o Promotions o Inventory o Supply Chain o Vendors / Suppliers o Orders o Forecasting o Merchandising · Understand retail\-specific challenges such as high\-volume transaction processing, store\-level data, product hierarchies, promotions, pricing, inventory movements, and customer analytics. · Translate business requirements into scalable data architecture and technical solutions. Technical Leadership · Act as the senior technical authority for the Databricks migration program. · Provide technical direction and mentoring to data engineers and developers. · Conduct architecture/design reviews and code reviews. · Troubleshoot complex production and performance issues. · Collaborate with offshore/onshore teams, enterprise architects, infrastructure teams, security teams, and business stakeholders. · Prepare architecture diagrams, technical design documents, migration plans, and implementation standards. · Communicate technical risks, dependencies, and recommendations effectively to client leadership. Required Technical Skills Databricks · Strong hands\-on experience with Databricks. · Apache Spark / PySpark. · Delta Lake. · Databricks Workflows/Jobs. · Unity Catalog. · Databricks SQL. · Performance tuning and optimization. · Medallion/Lakehouse architecture. · Experience with modern Databricks data engineering capabilities. Data Engineering · Strong SQL and data modeling skills. · ETL/ELT architecture. · Batch and incremental data processing. · CDC and SCD implementation. · Data quality and reconciliation frameworks. · Large\-volume data processing. Pay: $70\.00 \- $90\.00 per hour Work Location: Remote