Staff Software Engineer - OpenTelemetry Python instrumentation | UK | Remote

Requirements

  • 8+ years of experience in software development with strong proficiency in Python.
  • Experience debugging production software, language internals, libraries, and asynchronous code.
  • Interest in AI applications and agent systems with curiosity about instrumentation and troubleshooting.
  • Experience working in open-source communities and eagerness to engage with the OpenTelemetry community.
  • Familiarity with observability tooling such as Grafana, Prometheus, and OpenTelemetry.
  • Experience with cloud-based software or systems engineering and understanding of production environments and operational risks.

Nice to Haves

  • Experience working on open source observability projects, ideally OpenTelemetry or Prometheus.
  • Experience in Python instrumentation techniques, profiling, runtime diagnostics, or asynchronous context propagation.
  • Hands-on technical leadership experience, including setting technical direction and leading project teams.
  • Experience instrumenting AI frameworks or agent workflows, or familiarity with OpenTelemetry semantic conventions for generative AI.
  • Proven contributions to or maintenance of Open Source projects with successful pull requests and community collaboration.
  • Rust experience for exploring Python SDK performance improvements and Rust-backed implementations.

What You'll Be Doing

  • Develop improvements to the OpenTelemetry Python SDK and instrumentation libraries with a focus on AI applications and agent workflows.
  • Collaborate in open-source communities and work with maintainers on specifications, APIs, and implementations.
  • Build Python instrumentation that connects agent runs, model calls, and tool execution including asynchronous code.
  • Contribute to evolving OpenTelemetry conventions for generative AI with consideration for sensitive data and metric cardinality.
  • Improve Python SDK performance using profiling and benchmarks, balancing runtime overhead, compatibility, and maintainability.
  • Act as an OpenTelemetry expert, teaching best practices within Grafana Labs and the broader community.
  • Support the technical direction and vision of the team and contribute to business conversations on OpenTelemetry strategy.
  • Own projects from problem definition through design, implementation, testing, documentation, and release.
  • Collaborate openly with GTM, Product, and Support teams to resolve instrumentation issues and provide practical guidance.

Perks and Benefits

  • 100% remote global culture with a diverse, collaborative team across 40+ countries.
  • Competitive compensation range including Restricted Stock Units (RSUs) for ownership in company success.
  • Scalable and innovative work environment with transparency, autonomy, and trust.
  • Defined career growth pathways with dedicated IC career tracks and opportunities to progress without management roles.
  • Approachable leadership and transparent communication fostering a high-trust, low-ego culture.
  • In-person onboarding to connect with team members and learn about company culture and practices.
  • Generous global annual leave policy of 30 days, including 3 Grafana Shutdown Days for team-wide disconnection.
  • Commitment to diversity, equity, and inclusion as an equal opportunity employer.
AI Summary ✨

Similar jobs

  • FICC Options Execution - Senior Software Engineer (Crypto)

    DRWLondon, UKa day ago
    NewRegistered sponsor
  • Senior Backend Engineer - Personal and Business Pricing

    WiseGreater London, UKa day ago
    NewRegistered sponsor
    ≈ €105k – €133k / year base
  • Software Engineer, Guardian

    OpenAILondon, UK2 days ago
    NewRemoteRegistered sponsor
  • See all jobs in UK →