Data Engineer

As a Data Engineer at our company, you will be responsible for designing, developing, and maintaining real-time and batch data pipelines in Databricks. Your key responsibilities will include: - Integrating Apache Kafka with Databricks using Structured Streaming. - Implementing robust data ingestion frameworks using Databricks Autoloader. - Building and maintaining Medallion Architecture pipelines across Bronze, Silver, and Gold layers. - Implementing checkpointing, output modes, and appropriate processing modes in structured streaming jobs. - Designing and implementing Change Data Capture (CDC) workflows and Slowly Changing Dimensions (SCD) Type 1 and Type 2 logic. - Developing reusable components for merge/upsert operations and window function-based transformations. - Handling large volumes of data efficiently through proper partitioning, caching, and cluster tuning techniques. - Collaborating with cross-functional teams to ensure data availability, reliability, and consistency. Qualifications required for this role include: - Strong hands-on experience with Apache Kafka, including integration, topic management, and schema registry (Avro/JSON). - Expertise in Databricks and Spark Structured Streaming. - Proficiency in writing efficient queries and transformations using SQL & Spark SQL. - Knowledge of data governance practices, including schema enforcement, data quality checks, and monitoring. Additionally, it would be beneficial if you have: - Strong coding skills in Python and PySpark. - Experience working in CI/CD environments for data pipelines. - Exposure to cloud platforms such as AWS, Azure, or GCP. - Understanding of Delta Lake, time travel, and data versioning. - Familiarity with orchestration tools like Airflow or Azure Data Factory. This position is based in Mumbai and requires 3-6 years of experience. If you are passionate about data engineering and have a proven track record of success in building and optimizing data pipelines, we encourage you to apply for this exciting opportunity. As a Data Engineer at our company, you will be responsible for designing, developing, and maintaining real-time and batch data pipelines in Databricks. Your key responsibilities will include: - Integrating Apache Kafka with Databricks using Structured Streaming. - Implementing robust data ingestion frameworks using Databricks Autoloader. - Building and maintaining Medallion Architecture pipelines across Bronze, Silver, and Gold layers. - Implementing checkpointing, output modes, and appropriate processing modes in structured streaming jobs. - Designing and implementing Change Data Capture (CDC) workflows and Slowly Changing Dimensions (SCD) Type 1 and Type 2 logic. - Developing reusable components for merge/upsert operations and window function-based transformations. - Handling large volumes of data efficiently through proper partitioning, caching, and cluster tuning techniques. - Collaborating with cross-functional teams to ensure data availability, reliability, and consistency. Qualifications required for this role include: - Strong hands-on experience with Apache Kafka, including integration, topic management, and schema registry (Avro/JSON). - Expertise in Databricks and Spark Structured Streaming. - Proficiency in writing efficient queries and transformations using SQL & Spark SQL. - Knowledge of data governance practices, including schema enforcement, data quality checks, and monitoring. Additionally, it would be beneficial if you have: - Strong coding skills in Python and PySpark. - Experience working in CI/CD environments for data pipelines. - Exposure to cloud platforms such as AWS, Azure, or GCP. - Understanding of Delta Lake, time travel, and data versioning. - Familiarity with orchestration tools like Airflow or Azure Data Factory. This position is based in Mumbai and requires 3-6 years of experience. If you are passionate about data engineering and have a proven track record of success in building and optimizing data pipelines, we encourage you to apply for this exciting opportunity.