Design, build, and operate the services and libraries for data orchestration and workflow platform (Airflow, TaskService) used company-wide.
Develop, scale, and optimize distributed data processing jobs on Spark running across Amazon EMR, Kubernetes, and Databricks.
Collaborate with engineering teams to migrate and modernize big data workloads, such as moving jobs onto Databricks or Spark-on-Kubernetes.
Enhance platform reliability and observability through metrics, alerting, tracing (OpenTelemetry), and on-call tooling.
Build and maintain internal tools and lightweight UIs to help engineers monitor, debug, and operate data pipelines.
Own projects end-to-end, from design and code review through deployment, rollout, and production support.
Debug and resolve complex issues in large-scale, distributed production systems.
Contribute to the technical roadmap and define best practices for data pipeline development company-wide.
Requirements
Strong software engineering fundamentals with production experience in Python, Scala, or Java.
Experience building or operating distributed data processing systems (e.g., Spark, Databricks, Hadoop) and/or workflow orchestration tools (e.g., Airflow).
Comfortable working with cloud infrastructure (AWS, Kubernetes) and CI/CD pipelines.
Solid grounding in data structures, algorithms, and distributed systems design.
Strong debugging and troubleshooting skills for large-scale production systems.
Some experience with or interest in building internal tools, dashboards, or lightweight UIs.
Effective communication skills and ability to collaborate with engineering teams across the company.
Pragmatic, ownership-driven approach to building and operating software.
2+ years of software development/engineering experience.
Bachelor’s or Master’s degree in computer science, relevant engineering field, or equivalent experience.
Nice to haves
Experience with SQL and data warehousing technologies (e.g., Snowflake, Vertica).
Knowledge of Kubernetes operators and cluster administration.
Familiarity with observability tooling such as OpenTelemetry, Grafana, and Prometheus.
Experience carrying production on-call responsibilities for critical systems.
Exposure to machine learning pipelines or ad-tech / real-time bidding systems.
Perks and benefits
Global team with diverse backgrounds, experiences, and perspectives fostering an inclusive and supportive environment.
Commitment to equal opportunity employment and creating an inclusive hiring experience.
Support for accommodation requests or accessibility needs during the hiring process.