What You Will Be Doing

• Own prioritized competitive benchmarking initiatives across search, vector and hybrid retrieval, analytical and observability workloads.
• Build and maintain automated benchmark harnesses, datasets, query sets, and load profiles that reflect real-world usage, including concurrent ingestion and querying.
• Design fair comparisons with documented system versions, configurations, tuning decisions, resource budgets, and functional differences.
• Evaluate latency distributions, throughput, resource consumption, and cost efficiency; measure performance at comparable recall or relevance targets for retrieval workloads.
• Adapt and extend workload-appropriate tools such as Elasticsearch Rally, ClickBench, and vector search benchmark frameworks.
• Investigate performance differences using profiling, system metrics, and controlled experiments; collaborate with engineering teams to identify causes and validate improvements.
• Evaluate self-managed and managed cloud deployments, including serverless offerings, considering scaling behavior and service constraints.
• Work with engineering and technical content partners to publish reproducible benchmark artifacts and clear technical reports explaining methodology, findings, limitations, and tradeoffs.
• Partner with engineering and product management to prioritize benchmark coverage and translate findings into actionable product improvements.

Requirements

• Solid software engineering skills with experience building maintainable automation and analysis tools in Python or other relevant programming languages.
• Hands-on experience evaluating and diagnosing performance in distributed systems, databases, or search infrastructure.
• Depth in at least one relevant area—search, vector retrieval, analytical databases, or observability systems—and the ability to learn unfamiliar systems.
• Experience designing representative datasets and workloads, including query patterns, concurrency, and ingestion behavior.
• Sound experimental judgment involving repeated measurements, variability analysis, warm-up and cache effects, and differentiating meaningful changes from measurement noise.
• Experience building reproducible test environments and documenting configurations, procedures, and results.
• Clear technical writing and communication skills to explain complex findings and their limitations.
• Strong collaboration skills and ability to translate performance evidence into engineering recommendations.

Nice to Haves

• Experience with Elasticsearch, Lucene, Rally, ClickBench, or vector search benchmark frameworks.
• Experience measuring retrieval quality alongside latency, throughput, and resource usage.
• Familiarity with cloud performance and cost analysis, including autoscaling and serverless systems.
• Public technical writing, open-source contributions, or published benchmark studies.

Perks and Benefits

• Competitive pay based on your work rather than previous salary.
• Health coverage for you and your family in many locations.
• Flexible work locations and schedules for many roles.
• Generous vacation days every year.
• Financial matching up to $2000 (or local currency equivalent) for donations and service.
• Up to 40 hours per year for volunteer projects.
• Minimum of 16 weeks parental leave to support parenthood.
• Inclusive culture celebrating diverse perspectives, backgrounds, and experiences.
• Commitment to accessibility and accommodations during application and recruitment processes.