What You'll Be Doing

• Own prioritized competitive benchmarking initiatives across search, vector and hybrid retrieval, and analytical and observability workloads.
• Build and maintain automated benchmark harnesses, datasets, query sets, and load profiles that reflect real-world usage, including concurrent ingestion and querying.
• Design fair comparisons with documented system versions, configurations, tuning decisions, resource budgets, and functional differences.
• Evaluate latency distributions, throughput, resource consumption, and cost efficiency; measure retrieval performance at comparable recall or relevance targets.
• Adapt and extend workload-appropriate tools such as Elasticsearch Rally, ClickBench, and vector search benchmark frameworks.
• Investigate performance differences using profiling, system metrics, and controlled experiments; collaborate with engineering teams to identify causes and validate improvements.
• Evaluate relevant self-managed and managed cloud deployments, including serverless offerings, accounting for scaling behavior and service constraints.
• Work with engineering and technical content partners to publish reproducible benchmark artifacts and clear technical reports explaining methodology, findings, limitations, and tradeoffs.
• Partner with engineering and product management to prioritize benchmark coverage and translate findings into actionable product improvements.

Requirements

• Solid software engineering skills with experience building maintainable automation and analysis tools in Python or a relevant programming language.
• Hands-on experience evaluating and diagnosing performance in distributed systems, databases, or search infrastructure.
• Depth in at least one relevant area (search, vector retrieval, analytical databases, or observability systems) and ability to learn unfamiliar systems.
• Experience designing representative datasets and workloads including query patterns, concurrency, and ingestion behavior.
• Sound experimental judgment including repeated measurements, variability analysis, warm-up and cache effects, and distinguishing meaningful changes from measurement noise.
• Experience building reproducible test environments and documenting configurations, procedures, and results.
• Clear technical writing and communication skills with ability to explain complex findings and limitations.
• Strong collaboration skills and ability to translate performance evidence into applicable engineering recommendations.

Nice to Haves

• Experience with Elasticsearch, Lucene, Rally, ClickBench, or vector search benchmark frameworks.
• Experience measuring retrieval quality alongside latency, throughput, and resource usage.
• Familiarity with cloud performance and cost analysis including autoscaling and serverless systems.
• Public technical writing, open-source contributions, or published benchmark studies.

Perks and Benefits

• Competitive pay based on your work, not your previous salary.
• Health coverage for you and your family in many locations.
• Flexible locations and schedules to craft your own calendar in many roles.
• Generous vacation days each year.
• Matching donations up to $2000 (or local currency equivalent) for financial donations and service.
• Up to 40 hours each year to volunteer for projects you love.
• Minimum of 16 weeks parental leave supporting parenthood.