Senior Scala Spark Data Architect

Role Overview: As a Senior Scala Data Engineer, you will be responsible for designing and implementing robust, scalable data engineering solutions using Apache Spark (Scala) and Spark structured streaming. Your expertise in software design and distributed systems will be crucial in developing end-to-end data pipelines on Kubernetes clusters, orchestrated via Airflow. Your strong foundation in software engineering and knowledge of CI/CD processes will enable you to deliver reliable, scalable, and robust data solutions. Key Responsibilities: - Design & implement robust, scalable, batch & real-time data engineering solutions using Apache Spark (Scala) & Spark structured streaming. - Architect well-structured Scala projects using reusable, modular, and testable codebases aligned with SOLID principles and clean architecture principles & practices. - Develop, Deploy & Manage Spark jobs on Kubernetes clusters, ensuring efficient resource utilization, fault tolerance, and scalability. - Orchestrate data workflows using Apache Airflow manage DAGs, task dependencies, retries, and SLA alerts. - Write and maintain comprehensive unit tests and integration tests for Pipelines / Utilities developed. - Work on performance tuning, partitioning strategies, and data quality validation. - Use and enforce version control best practices (branching, PRs, code review) and continuous integration (CI/CD) for automated testing and deployment. - Write clear, maintainable documentation (README, inline docs, docstrings). - Participate in design reviews and provide technical guidance to peers and junior engineers. Qualification Required: - Strong experience in Scala and Java languages. - Proficiency in Big Data Orchestration tools like Airflow, Spark on Kubernetes, Yarn, Oozie. - Experience with Big Data Processing technologies such as Hadoop, Kafka, Spark & Spark Structured Streaming. - Knowledge of SOLID & DRY principles with Good Software Architecture & Design implementation experience. - Advanced Scala experience including Functional Programming, using Case classes, Complex Data Structures & Algorithms. - Proficient in developing automated frameworks for unit & integration testing. - Experience with Docker, Helm, and related container technologies. - Proficient in deploying and managing Spark workloads on Kubernetes clusters. - Experience in evaluation and implementation of Data Validation & Data Quality. - DevOps experience in Jenkins, Maven, Github, Github actions, CI/CD. Role Overview: As a Senior Scala Data Engineer, you will be responsible for designing and implementing robust, scalable data engineering solutions using Apache Spark (Scala) and Spark structured streaming. Your expertise in software design and distributed systems will be crucial in developing end-to-end data pipelines on Kubernetes clusters, orchestrated via Airflow. Your strong foundation in software engineering and knowledge of CI/CD processes will enable you to deliver reliable, scalable, and robust data solutions. Key Responsibilities: - Design & implement robust, scalable, batch & real-time data engineering solutions using Apache Spark (Scala) & Spark structured streaming. - Architect well-structured Scala projects using reusable, modular, and testable codebases aligned with SOLID principles and clean architecture principles & practices. - Develop, Deploy & Manage Spark jobs on Kubernetes clusters, ensuring efficient resource utilization, fault tolerance, and scalability. - Orchestrate data workflows using Apache Airflow manage DAGs, task dependencies, retries, and SLA alerts. - Write and maintain comprehensive unit tests and integration tests for Pipelines / Utilities developed. - Work on performance tuning, partitioning strategies, and data quality validation. - Use and enforce version control best practices (branching, PRs, code review) and continuous integration (CI/CD) for automated testing and deployment. - Write clear, maintainable documentation (README, inline docs, docstrings). - Participate in design reviews and provide technical guidance to peers and junior engineers. Qualification Required: - Strong experience in Scala and Java languages. - Proficiency in Big Data Orchestration tools like Airflow, Spark on Kubernetes, Yarn, Oozie. - Experience with Big Data Processing technologies such as Hadoop, Kafka, Spark & Spark Structured Streaming. - Knowledge of SOLID & DRY principles with Good Software Architecture & Design implementation experience. - Advanced Scala experience including Functional Programming, using Case classes, Complex Data Structures & Algorithms. - Proficient in developing automated frameworks for unit & integration testing. - Experience with Docker, Helm, and related container technologies. - Proficient in deploying and managing Spark workloads on Kubernetes clusters. - Experience in evaluation and implementation of Data Validation & Data Quality. - DevOps experience in Jenkins, Maven, Github, Github actions, CI/CD.