Cloud Platform Engineer

Job Description

– Cloud Platform Engineer

Job Title

Cloud Platform Engineer

About the Role

We are seeking a skilled Cloud Platform Engineer (EL2) to join our team for a Cloud Data Modernization initiative.

The ideal candidate

will have strong hands-on experience with Databricks as the primary data platform supported by Snowflake and Azure/AWS cloud infrastructure. The role involves modernizing and migrating on-premises ETL workloads to cloud-based data platforms developing scalable data pipelines implementing CI/CD practices and ensuring strong data quality observability security and performance. The successful candidate should be comfortable working with Databricks Apache Spark PySpark Azure Data Factory ADLS Gen2 Snowflake Python SQL and GitHub Actions.

Key Responsibilities

Contribute to the migration and modernization of on-premises ETL workloads to cloud platforms. Develop and maintain scalable data engineering solutions using Databricks as the primary platform. Build data pipelines using Databricks Apache Spark PySpark and Azure Data Factory (ADF). Work with Snowflake as a secondary data platform for data engineering and analytics workloads. Develop and support integrations using ADF Self-hosted Integration Runtime (SHIR) Logic Apps ADLS Gen2 and Blob Storage. Review existing on-premises ETL processes and identify opportunities for cloud modernization and optimization. Implement Bronze/Silver/Gold (Medallion) architecture and Lakehouse patterns where applicable. Work with Delta Lake for scalable and reliable data processing. Develop data transformations and ETL/ELT processes using Python PySpark and SQL. Implement automated data validation reconciliation data quality monitoring and observability processes. Optimize Databricks and Snowflake workloads for performance scalability reliability and cost efficiency. Implement and maintain CI/CD pipelines using GitHub Actions. Follow Git branching pull-request code-review and engineering best practices. Collaborate with data engineers cloud engineers architects QA teams business analysts and other stakeholders. Support production data pipelines and troubleshoot data performance integration and deployment issues. Ensure data solutions follow applicable security governance compliance and data-quality standards. Leverage AI-assisted development tools to improve engineering productivity code quality testing and documentation. Required Technical SkillsDatabricks – Primary Skill Strong hands-on experience with Databricks. Experience developing data pipelines using PySpark and Apache Spark. Good understanding of Delta Lake and Lakehouse architecture. Experience with Databricks notebooks jobs/workflows clusters and optimization. Understanding of Medallion Architecture – Bronze Silver and Gold layers. Experience optimizing Spark jobs transformations partitions joins and workloads. Azure Cloud Hands-on experience with Azure data services such as Azure Data Factory (ADF) Self-hosted Integration Runtime (SHIR) Azure Data Lake Storage Gen2 (ADLS Gen2) Azure Blob Storage Azure Logic Apps Good understanding of cloud-based data integration and storage architectures. Snowflake – Secondary Skill Hands-on experience with Snowflake for data engineering and analytics workloads. Experience developing SQL-based transformations and ETL/ELT processes. Understanding of Snowflake data loading warehouse concepts and performance optimization. Experience supporting Snowflake migration or modernization initiatives is desirable. Programming & Data Engineering Strong experience with Python and/or PySpark. Strong SQL development skills. Experience writing complex queries joins stored procedures views CTEs and data transformations. Experience developing enterprise ETL/ELT pipelines. Understanding of structured and semi-structured data processing. DevOps & CI/CD Hands-on experience with Git and GitHub. Experience with GitHub branching strategies and Pull Requests. Experience developing and maintaining GitHub Actions CI/CD pipelines. Understanding of automated testing deployment and release processes. Experience participating in code reviews and following engineering best practices. Data Quality & Observability Experience implementing data validation and reconciliation frameworks. Knowledge of data quality dimensions such as accuracy completeness consistency and timeliness. Experience with pipeline monitoring and data observability. Ability to identify and troubleshoot data quality issues. Cloud Platform Experience Strong experience with Azure is preferred. Experience with AWS is an advantage. Understanding of cloud storage compute networking security and identity concepts. Ability to work across cloud environments and adapt to new cloud technologies. AI-Assisted Engineering Experience using AI-assisted development tools such as GitHub Copilot Databricks Assistant ChatGPT Claude Similar AI coding assistants Ability to use AI tools for code generation debugging refactoring test development documentation and engineering productivity while following enterprise security and governance standards.

Preferred Qualifications

Experience with Delta Lake and Lakehouse architectures. Experience implementing Bronze/Silver/Gold architecture. Experience with Airflow or other orchestration frameworks. Experience with data modeling and database design. Knowledge of data governance and data quality best practices. Experience with Databricks and Snowflake performance optimization. Experience with healthcare payer data such as Claims Membership Enrollment Provider Clinical Financial data Experience leveraging AI/ML for data engineering automation. Experience with other cloud platforms and data solutions. Azure or Databricks certification is a plus.

Key Responsibilities

by PriorityPrioritySkill / AreaPrimaryDatabricksPrimaryPySpark / Apache SparkPrimaryCloud Data EngineeringPrimaryAzure Data PlatformSecondarySnowflakeSecondarySQL / PythonSecondaryETL/ELT & Data MigrationRequiredGitHub / GitHub Actions / CI/CDRequiredData Quality & ValidationGood to HaveDelta Lake / LakehouseGood to HaveMedallion ArchitectureGood to HaveAirflowGood to HaveHealthcare Payer DataGood to HaveAI-Assisted Development

Key Competencies

Strong analytical and problem-solving skills. Ability to work independently and manage ambiguity. Strong understanding of cloud data engineering concepts. Ability to troubleshoot complex data and pipeline issues. Strong focus on data quality security scalability and performance. Excellent communication and collaboration skills. Ability to work effectively within Agile delivery methodologies. Ability to learn and adopt new technologies quickly. Ability to work under tight deadlines and proactively escalate critical issues.

Work Location

Remote