Job Description

The Role This position will be responsible for designing data lake/data warehouse using Microsoft Azure and AWS, including the necessary development work in the relevant Extract-Transform-Load (ETL) The Role This position will be responsible for designing data lake/data warehouse using Microsoft Azure and AWS, including the necessary development work in the relevant Extract-Transform-Load (ETL) integration tool. Job Responsibilities Design and develop large scale, high-volume, and high-performance data structures with large sets of data from different sources. Provide technical expertise in the Big Data space (SQL Server, Spark, Python, Redshift, Graph) and in Azure Synapse analytics. Build data pipelines that are scalable, repeatable, and secure, and can serve multiple purposes. Hands on experience in tools like Informatica, Azure Data factory (ADF) or any other integration tool Process semi and unstructured data into format suitable for analysis. Able to use API and perform API management using Azure APIM and logic app. Create a platform on top of stored data sources for users to perform any kind of ad-hoc queries and advanced analytics Create data catalogs and data flow diagrams, and recognize interdependencies between components Develop and define key business questions and build data sets that answer those questions. Gather and process raw data at scale (including writing scripts, write SQL queries, etc.) to build features that will be used in modeling Leads project team with full responsibility in all stages (i.e. design, analysis, coding, testing and integration) of project implementation, involving complex, secure and performant data solutions and models Builds and manages relationships throughout the organization Reviews and evaluates designs and project activities for compliance with architecture, security and quality guidelines and standards provides tangible feedback to improve product quality and mitigate failure risk. Drives innovation and integration of new technologies into projects and activities in the big data space. Collaborates and communicates with PMO regarding project progress and issue resolution. Subject Matter Expert (SME), representing the data engineering team in large and complex data analytical projects. Provides guidance and mentorship to fellow staff members. Job Requirements 5-7 years of hands-on experience developing high availability, scalable, medium to large scale ETL and Data Warehouse systems. In depth knowledge of Cloud Technologies: Micro Services/API, Azure Synapse, , ADLS, AWS, IAM, S3, Amazon, AWS APIs, AWS CLI, Amazon Kinesis, Cloud Formation To code efficiently in scripting languages such as Powershell, Python, SQL Develop data integration from multiple data sources, processing large amounts of structured and unstructured data Proficient in distributed computing principles, Azure Architecture and querying tools Knowledge of ETL techniques and frameworks Ability to analyze a problem, develop and engineer a full solution from concept, to deployment and to production. Team player with a long term vision supporting the overall team goal Passion for quality and attention to detail Ability to effectively work with distributed teams, interface with business stakeholders and manage your own time with an eye on timely delivery of the final solution. Knowledge of creating and /or consuming Web API from various platforms is an added advantage A proven track record in managing and controlling multiple projects and working within a team environment. Significant prior experience interfacing between business and technical teams to gather requirements, estimate projects and report progress is required. Must be motivated, independent and self-sufficient. Able to receive an assigned task and see it through to completion with minimal supervision. Candidate should have excellent interpersonal and customer facing skills.