Job Description

-------------------

Data Engineer should be able to understand the Business requirements, Functional and Technical requirements and should build effective data transformation jobs in Python, PySpark/SCALA, Python Framework. Should have strong hands-on working expertise in creating the optimized data pipelines in Pyspark/Python/Scala. Produce unit tests for Spark transformations and helper methods. Understand the complex transformation logic and translate and build data pipelines in Pyspark/Spark-SQL/Hive logic to ingest the data from source systems to Data Lake (Hive/Hbase/Parquet)/ Enterprise Data Domain tables. Work closely with Business Analysts team to review the test results and obtain sign off. Prepare necessary design/operations documentation for future usage. Perform peers Code quality review and be gatekeeper for quality checks. Hands-on coding, usually in a pair programming environment. Working in highly collaborative teams and building quality code. The candidate must exhibit a good understanding of data structures, data manipulation, distributed processing, application development, and automation.
Familiar with Oracle, Spark streaming, Kafka, ML.


Have knowledge on RDMS concepts, hands on experience on PLSQL etc. To develop an application by using Hadoop tech stack and delivered effectively, efficiently, on-time, in-specification and in a cost-effective manner. Ensure smooth production deployments as per plan and post-production deployment verification. This Hadoop Developer will play a hands-on role to develop quality applications within the desired timeframes and resolving team queries.
Technical Requirements:


Hadoop data engineer with total 4-6years of experience and should have strong experience in Hadoop, Spark, Pyspark, Scala , Hive, Spark-SQL, Python, Impala, CI/CD, Git, Jenkins, Agile Methodologies, DevOps, Cloudera Distribution. Strong Knowledge in data warehousing Methodology and Change Data Capture. Relevant 5+ years of Hadoop & Spark/Pyspark experience is mandatory. Good Knowledge and experience in any RDBMS database (MariaDB or SQL Server OR MySQL or Oracle ). Knowledge on stored procedures is an added advantage. Have exposure to TWS jobs for scheduling. Strong in enterprise data architectures and data models. Good experience in Core Banking, Finance domain. * Exposure in AML Domain preferred, not mandatory.