Position Overview & Specifications
Hi,
Position: Databricks LeadLocation: RemoteClient: Innover Digital
Job Summary As a Senior Databricks Lead, responsible for implementing Delta Lake architecture and optimizing data pipelines in Azure using Databricks, while ensuring compliance with governance standards and driving project delivery through collaboration with Project Managers and guidance to Data Engineers. Major Tasks, Responsibilities, and Key AccountabilitiesImplement Delta Lake architecture for efficient storage, versioning, and management of large-scale structured and semi-structured data in Azure environment using Azure Databricks.Integrate Databricks with various data sources and systems, including relational databases, data lakes, cloud storage, and streaming platforms, ensuring seamless data ingestion and synchronization.Optimize Spark jobs, SQL queries, and data pipelines for performance and efficiency, leveraging Databricks features like caching, partitioning, and cluster configurationImplement data governance policies, access controls, and encryption mechanisms to ensure compliance with regulatory requirements and protect sensitive dataSet up monitoring and alerting systems to track Databricks cluster performance, resource utilization, and job execution, and troubleshoot issues to ensure system availability and reliability.Document architectural designs, data flows, and configurations, and establish best practices and guidelines for Databricks development, deployment, and maintenance.Competence in unity catalog, DevOps and DataOps practices, including familiarity with tools such as GitHub Actions, Terraform, Azure DevOps, and data governance tools.Skilled in orchestrating and automating data pipelines to establish a DevOps model effectively.Be involved in all project phases and drive delivery working closely with Project ManagerSupport & Guide Data Engineers with technical challenges, oversee deploymentRequired Qualifications:Minimum of 10+ years of experience in architecture, design, implementation, and analytics solutions in the Data Engineering domain.Proven track record of designing and implementing at least 2 projects end-to-end in Databricks.Extensive experience (at least 3+ years) in Databricks, including expertise in various components such as Delta Lake, Unity Catalog, Databricks workflows orchestration, Security management, and Platform governance.Proficiency in Databricks Lakehouse concept and its implementation within enterprise environments.Strong understanding of Data warehousing and familiarity with various governance and security standards pertaining to Databricks.Knowledgeable in cluster optimization and its integration with diverse cloud services.Ability to develop complex data pipelines efficiently.Proficient in SQL and spark-sql with a strong emphasis on performance optimization skills to enhance efficiency and reduce costs.Experience in designing both batch and streaming data pipelines.Extensive knowledge of Spark and Hive data processing framework.Understanding of writing unit test cases and integration tests.Responsible for establishing best practices around Databricks CI/CD.Comprehensive understanding of Spark Architecture, encompassing Spark Core, Spark SQL, Data Frames, Spark Streaming, and RDD caching.
Thanks & Regards,Himanshu Kumar-----------------------------------------------Next Level Business Services, Inc.Talent Solutions | Digital Transformation | Data AnalyticsUS: +1 904-344-3863 |Fax: +1 (608) 646-8326E-mail: Himanshu.kumar1@recruiter.nlbtech.com
Position: Databricks LeadLocation: RemoteClient: Innover Digital
Job Summary As a Senior Databricks Lead, responsible for implementing Delta Lake architecture and optimizing data pipelines in Azure using Databricks, while ensuring compliance with governance standards and driving project delivery through collaboration with Project Managers and guidance to Data Engineers. Major Tasks, Responsibilities, and Key AccountabilitiesImplement Delta Lake architecture for efficient storage, versioning, and management of large-scale structured and semi-structured data in Azure environment using Azure Databricks.Integrate Databricks with various data sources and systems, including relational databases, data lakes, cloud storage, and streaming platforms, ensuring seamless data ingestion and synchronization.Optimize Spark jobs, SQL queries, and data pipelines for performance and efficiency, leveraging Databricks features like caching, partitioning, and cluster configurationImplement data governance policies, access controls, and encryption mechanisms to ensure compliance with regulatory requirements and protect sensitive dataSet up monitoring and alerting systems to track Databricks cluster performance, resource utilization, and job execution, and troubleshoot issues to ensure system availability and reliability.Document architectural designs, data flows, and configurations, and establish best practices and guidelines for Databricks development, deployment, and maintenance.Competence in unity catalog, DevOps and DataOps practices, including familiarity with tools such as GitHub Actions, Terraform, Azure DevOps, and data governance tools.Skilled in orchestrating and automating data pipelines to establish a DevOps model effectively.Be involved in all project phases and drive delivery working closely with Project ManagerSupport & Guide Data Engineers with technical challenges, oversee deploymentRequired Qualifications:Minimum of 10+ years of experience in architecture, design, implementation, and analytics solutions in the Data Engineering domain.Proven track record of designing and implementing at least 2 projects end-to-end in Databricks.Extensive experience (at least 3+ years) in Databricks, including expertise in various components such as Delta Lake, Unity Catalog, Databricks workflows orchestration, Security management, and Platform governance.Proficiency in Databricks Lakehouse concept and its implementation within enterprise environments.Strong understanding of Data warehousing and familiarity with various governance and security standards pertaining to Databricks.Knowledgeable in cluster optimization and its integration with diverse cloud services.Ability to develop complex data pipelines efficiently.Proficient in SQL and spark-sql with a strong emphasis on performance optimization skills to enhance efficiency and reduce costs.Experience in designing both batch and streaming data pipelines.Extensive knowledge of Spark and Hive data processing framework.Understanding of writing unit test cases and integration tests.Responsible for establishing best practices around Databricks CI/CD.Comprehensive understanding of Spark Architecture, encompassing Spark Core, Spark SQL, Data Frames, Spark Streaming, and RDD caching.
Thanks & Regards,Himanshu Kumar-----------------------------------------------Next Level Business Services, Inc.Talent Solutions | Digital Transformation | Data AnalyticsUS: +1 904-344-3863 |Fax: +1 (608) 646-8326E-mail: Himanshu.kumar1@recruiter.nlbtech.com