Position Overview & Specifications
About Our Client:Our healthcare client is committed to leveraging innovative data engineering solutions, particularly the Medallion architecture, to drive operational excellence and deliver transformative insights in patient care.
Must-Haves:- Minimum of 3 years' experience with Databricks, PySpark, SQL, Spark clusters, and Jupyter Notebooks.- Expertise in building data lakes using the Medallion architecture and working with delta tables in the delta file format.- Familiarity with CI/CD pipelines and Agile methodologies, ensuring efficient and collaborative development practices.- Strong understanding of ETL processes, data modeling, and data warehousing principles.- Experience with data visualization tools like Power BI is a plus.- Knowledge of cybersecurity data, particularly vulnerability scan data, is preferred.- Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.
Responsibilities:- Develop, test, and maintain large-scale data processing systems using Databricks, PySpark, SQL, Spark clusters, delta tables, and the innovative Medallion architecture.- Collaborate closely with cybersecurity analysts to gather data requirements and deliver effective solutions aligned with Medallion architecture principles.- Ensure data quality and implement robust data governance standards, leveraging the scalability and efficiency offered by the Medallion architecture.- Design and implement ETL processes, including data cleansing, transformation, and integration, optimizing performance within the delta file format framework.- Build and manage data lakes based on Medallion architecture principles, ensuring scalability, reliability, and adherence to best practices.- Monitor and optimize data pipelines, integrating CI/CD practices to streamline development and deployment processes.- Collaborate with cross-functional team members to implement data analytics projects, utilizing Jupyter Notebooks and other tools to harness the power of the Medallion architecture.- Embrace Agile methodologies throughout the development lifecycle to promote iterative and collaborative development practices, enhancing the effectiveness of Medallion-based solutions.
Must-Haves:- Minimum of 3 years' experience with Databricks, PySpark, SQL, Spark clusters, and Jupyter Notebooks.- Expertise in building data lakes using the Medallion architecture and working with delta tables in the delta file format.- Familiarity with CI/CD pipelines and Agile methodologies, ensuring efficient and collaborative development practices.- Strong understanding of ETL processes, data modeling, and data warehousing principles.- Experience with data visualization tools like Power BI is a plus.- Knowledge of cybersecurity data, particularly vulnerability scan data, is preferred.- Bachelor's or Master's degree in Computer Science, Information Systems, or a related field.
Responsibilities:- Develop, test, and maintain large-scale data processing systems using Databricks, PySpark, SQL, Spark clusters, delta tables, and the innovative Medallion architecture.- Collaborate closely with cybersecurity analysts to gather data requirements and deliver effective solutions aligned with Medallion architecture principles.- Ensure data quality and implement robust data governance standards, leveraging the scalability and efficiency offered by the Medallion architecture.- Design and implement ETL processes, including data cleansing, transformation, and integration, optimizing performance within the delta file format framework.- Build and manage data lakes based on Medallion architecture principles, ensuring scalability, reliability, and adherence to best practices.- Monitor and optimize data pipelines, integrating CI/CD practices to streamline development and deployment processes.- Collaborate with cross-functional team members to implement data analytics projects, utilizing Jupyter Notebooks and other tools to harness the power of the Medallion architecture.- Embrace Agile methodologies throughout the development lifecycle to promote iterative and collaborative development practices, enhancing the effectiveness of Medallion-based solutions.