Job Description

Role Overview:

ETL & Data Engineer to design, build, and maintain robust data pipelines and backend services that power AI-driven operations. The role involves working with high-volume IT and cloud data, optimizing ETL processes, and integrating with AIOps platforms and ML pipelines.

Key Responsibilities:

Build and maintain scalable ETL pipelines for batch and real-time data ingestion, transformation, and loading from diverse sources (IT infrastructure, cloud, monitoring systems, APIs). Implement data validation, cleansing, and normalization for consistent AI model input. Develop backend services and APIs to support data ingestion, metadata management, and configuration. Optimize ETL jobs for performance, fault tolerance, and low latency. Integrate with AIOps platforms and ML pipelines using REST APIs or event-driven architectures. Schedule and monitor ETL workflows using tools like Airflow, Prefect, or Dagster. Support CI/CD pipelines for deploying ETL services and full-stack applications.

Required Skills & Tools:

Programming & Scripting: Python, Go (Golang), Java, Ruby, JavaScript/TypeScript (Next.js) ETL & Data Engineering: Apache NiFi, Spark, Airflow, Flink, Kafka, Talend Orchestration: Airflow, Prefect, Dagster Data Storage & Lakes: PostgreSQL, MongoDB, Elasticsearch, Snowflake, BigQuery, S3, GCS, Azure Blob Streaming Platforms: Kafka, Kinesis, Pub/Sub

Good to Have:

Experience with AIOps & Observability tools like Splunk, Dynatrace, AppDynamics, New Relic, Elastic Stack Familiarity with ITSM systems (ServiceNow) and CMDB integrations * Understanding of metrics, logs, and traces for AI-driven operations