Job Description

Job scopes

• Lead end-to-end data migration projects from on-premises environments to Databricks, ensuring minimal downtime.
• Work with architects to design solutions that meet both functional and non-functional requirements.
• Process near real-time data using Auto Loader and DLT pipelines.
• Identify and resolve data-related issues, providing support to ensure data availability and integrity.
• Optimize AWS and Databricks resource usage to control costs while meeting performance and scalability requirements.
• Stay up-to-date with AWS, Databricks services, and data engineering best practices to recommend and implement new technologies and techniques.
• Proactively implement engineering methodologies, standards, and leading practices.
• Evaluate potential technical solutions and make recommendations to resolve data issues, especially in performance assessment for complex data transformations and long-running data processes.

Job requirements

• Bachelor's or master's degree in Computer Science, Data Engineering, or a related field.
• Minimum 5 years of experience in data engineering, with expertise in AWS or Azure services, Databricks, and/or Informatica Intelligent Data Management Cloud (IDMC).
• Proficiency in Python, Java, or Scala for building data pipelines.
• Excellent knowledge of SQL and NoSQL databases.
• Must have a background in Python and be able to identify, communicate, and mitigate risks and issues.
• Hands-on experience in configuring Databricks clusters, writing PySpark code, and building CI/CD pipelines for deployment.
• Highly experienced in optimization techniques such as Z-ordering, Auto Compaction, and vacuuming.
• Familiarity with data modeling and schema design.
• Experience with big data technologies like Apache Spark and Hadoop on Databricks.
• Knowledge of AWS services focusing on data and architecture.
• Knowledge of containerization and orchestration tools like Docker and Kubernetes.
• Familiarity with data visualization tools like Tableau or Power BI.
• Understanding of DevOps principles for managing and deploying data pipelines.
• Experience with version control systems (e.g., Git) and CI/CD pipelines.
• Knowledge of data governance and data cataloging tools, especially Informatica IDMC (Intelligent Data Management Cloud).
• Databricks certifications, and Informatica certifications are a plus.
• Excellent problem-solving and analytical skills.
• Excellent communication and collaboration skills.

Cristina Malabuyoc Malijan EA License No. 02C3423 Personnel Registration No. R1111547Please note that your response to this advertisement and communications with us pursuant to this advertisement will constitute informed consent to the collection, use and/or disclosure of personal data by ManpowerGroup Singapore for the purpose of carrying out its business, in compliance with the relevant provisions of the Personal Data Protection Act 2012. To learn more about ManpowerGroup's Global Privacy Policy, please visit https://www.manpower.com.sg/privacy-policy

Manpower