Data Engineering - AWS

As a Data Engineer, you will be responsible for designing, developing, and optimizing large-scale data processing pipelines using PySpark. Your key responsibilities will include: - Utilizing technologies such as Hadoop, Hive, and HDFS for data ingestion, transformation, and management. - Ensuring high performance and reliability of ETL jobs in production environments. - Collaborating with Data Scientists, Analysts, and stakeholders to deliver robust data solutions. - Implementing data quality checks and maintaining data lineage for transparency and auditability. - Handling ingestion, transformation, and integration of structured and unstructured data sources. - Potentially leveraging Apache NiFi for automated, repeatable data flow management. - Writing clean, efficient, and maintainable code in Python and Java. - Contributing to architecture, performance tuning, and scalability strategies. Qualifications required for this role include: - 57 years of experience in data engineering. - Strong hands-on experience with PySpark and distributed data processing. - Solid understanding of ETL principles, data warehousing, and data modeling. - Experience with large-scale datasets and performance tuning. - Familiarity with SQL and NoSQL databases. - Proficiency in Python and intermediate knowledge of Java. - Experience with Git and CI/CD pipelines. In addition to the above, the company values hands-on experience with Apache NiFi, real-time streaming pipeline development experience, exposure to cloud platforms like AWS, Azure, or GCP, and familiarity with Docker or Kubernetes. As a Data Engineer, you will be responsible for designing, developing, and optimizing large-scale data processing pipelines using PySpark. Your key responsibilities will include: - Utilizing technologies such as Hadoop, Hive, and HDFS for data ingestion, transformation, and management. - Ensuring high performance and reliability of ETL jobs in production environments. - Collaborating with Data Scientists, Analysts, and stakeholders to deliver robust data solutions. - Implementing data quality checks and maintaining data lineage for transparency and auditability. - Handling ingestion, transformation, and integration of structured and unstructured data sources. - Potentially leveraging Apache NiFi for automated, repeatable data flow management. - Writing clean, efficient, and maintainable code in Python and Java. - Contributing to architecture, performance tuning, and scalability strategies. Qualifications required for this role include: - 57 years of experience in data engineering. - Strong hands-on experience with PySpark and distributed data processing. - Solid understanding of ETL principles, data warehousing, and data modeling. - Experience with large-scale datasets and performance tuning. - Familiarity with SQL and NoSQL databases. - Proficiency in Python and intermediate knowledge of Java. - Experience with Git and CI/CD pipelines. In addition to the above, the company values hands-on experience with Apache NiFi, real-time streaming pipeline development experience, exposure to cloud platforms like AWS, Azure, or GCP, and familiarity with Docker or Kubernetes.