Senior Data Platform Engineer

Requirements

• Design, develop, operate, and maintain scalable, reliable, fault-tolerant, and high-performance big data platforms.
• Work with distributed systems in all shapes and flavors (databases, file systems, compute, etc.).
• Shape and maintain continuous improvements and deployment of new clusters.
• Implement observability solutions (logging, monitoring, alerting) to ensure system reliability and performance.
• Optimize performance of large-scale distributed data platforms with appropriate cluster configurations, and resource management.
• Practice sustainable incident response and blameless postmortems, with a focus on automation and root cause analysis.
• Improve security and compliance by implementing access controls, encryption, and best practices for data governance.
• Design and build automation tools to reduce toil and enhance the reliability of data platforms.
• Self-service infrastructure to empower data teams while maintaining platform stability.
• Familiar with infrastructure automation and configuration management tools (e.g. Ansible, Terraform, Puppet).
• Strong system administrator/platform engineer background with experience in large-scale distributed systems.
• Experienced with infrastructure and private cloud systems (on-premise).
• A team player with strong communication skills, able to work closely with diverse stakeholders (analysts, data scientists, data engineers, infrastructure, and security teams).
• Experience developing and maintaining distributed data and compute systems (Hadoop, Druid, Trino, etc).
• Experience with Kubernetes (k8s, Docker).
• Experience with Hadoop ecosystems (Hive, YARN, HDFS, Kerberos).

Nice to have

• Experience as a Data Platform Engineer or Site Reliability Engineer for Data Platforms
• Fluency in Python. Java, Golang, or Rust are also appreciated.
• Experience in observability & monitoring tools (Prometheus, Grafana, OpenTelemetry).
• Experience with on-call rotations or incident management in high-scale distributed systems.

What you'll be doing

• Most days are dedicated to development of new features or preparing Kubernetes clusters for the Big Data platform.
• Collaborate with data and ML scientists and engineers to develop and roll out tools that enhance platform performance.
• Design, build, and deliver scalable systems as a Platform Engineer.
• Build new clusters and services while ensuring end users are unblocked and systems maintain agreed SLO uptime.