Lead Data Engineer
Job Description
We are seeking a highly skilled and motivated Data Engineer with hands-on experience in PySpark, Python, and a strong grasp of AWS cloud services including Databricks, RDS MySQL, and Redshift. As a Data Engineer, you will work closely with data scientists, analysts, and other engineering teams to design and develop scalable data pipelines, ensure high data quality, and support analytical and machine learning initiatives.
Design, develop, and maintain efficient and scalable ETL pipelines using PySpark and Python.
Build and manage Data Lake and data warehouse architectures on AWS.
Work with Databricks to process large volumes of data and create analytics-ready datasets.
Develop and optimize SQL queries to extract and manipulate data from RDS MySQL and Amazon Redshift databases.
Collaborate with Data Scientists and Analysts to understand data requirements and translate them into data models or pipeline solutions.
Ensure data integrity, quality, and security across various platforms.
Implement automated monitoring, logging, and alerting for data workflows.
Stay current with industry best practices and emerging technologies in big data and cloud computing.
Required Skills and Qualifications:
8+ years of experience as a Data Engineer or in a similar role working with data infrastructure. Strong proficiency in Python and PySpark for data processing. Hands-on experience with AWS Cloud Services, especially AWS Databricks or Databricks on Azure/GCP equivalent. Experience in developing large-scale data pipelines and batch/streaming ETL processes. Strong command over SQL for writing complex queries and performance tuning. Familiarity with data quality, lineage, and governance tools. Experience with version control systems like Git and CI/CD pipelines is a plus.
Preferred Qualifications:
Experience with orchestration tools such as Airflow, Glue, or Step Functions. Working knowledge of Delta Lake, Parquet, and other big data file formats. Exposure to containerization tools like Docker or Kubernetes. * Understanding of data security and compliance in cloud environments.
MNCJobz.com will not be responsible for any payment made to a third-party. All Terms of Use are applicable.
Related Jobs
Job Detail
-
Job IdJD1676232
-
IndustryNot mentioned
-
Total Positions1
-
Job Type:Full Time
-
Salary:Not mentioned
-
Employment StatusPermanent
-
Job LocationSG, Singapore
-
EducationNot mentioned