Job Description

Responsibilities:

Provide L3 support for all Apache Spark and Airflow related issues for the Data Platform. Design, develop, and maintain complex workflow, data pipelines, processes, and system architecture using Java, Python, PySpark, Apache Spark, Airflow and AWS technologies for efficient data processing and orchestration. Work closely with AWS services like S3, Glue, EMR, Redshift, and other related technologies to design and optimize data infrastructure. Implement data integration and transformation processes to ensure optimal performance and reliability of data pipelines. Optimize and fine-tune existing data pipelines / Airflow to improve efficiency, scalability, and maintainability. Troubleshoot and resolve issues related to data pipelines, ensuring smooth operation and minimal downtime. Develop and maintain documentation for data pipelines, processes, and system architecture. Stay updated with the latest industry trends and best practices related to data engineering and AWS services.

Requirements:

Bachelor's degree in Computer Science, Engineering, or a related field. At least 10 years of experience in software development, ETL, and data engineering, specifically working with Apache Spark, Apache Airflow and AWS technologies i.e. strong knowledge of AWS services, particularly S3, Glue, EMR, Redshift, and AWS Lambda. Proficient in the design and support of complex workflow for various business divisions to load data marts tables from S3 staging layer to AWS Redshift on daily basis in a 24 x 7 BAU environment. Strong proficiency (at least 8 years) in Java, Python, PySpark and SQL for data processing and manipulation, with experience with optimizing and scaling data pipelines for performance and efficiency. Strong understanding of Snowflake Data Lake is preferred. Strong understanding of data modelling, ETL processes and tools, and data warehousing concepts and tools. Excellent problem-solving skills and ability to work in a fast-paced, collaborative environment. * Effective communication skills and the ability to articulate technical concepts to non-technical stakeholders.