What you'll be doing

Design scalable frontier storage and queueing systems with high-throughput writes, efficient lookups, deduplication, and data-focused scheduling.
Develop a shared crawl execution layer for reliable and polite large-scale fetching of dynamic websites while managing resource use.
Build extensible pipelines for various content types including open-web, licensed, partner content, text, PDFs, code repositories, and multimodal assets.
Define stable interfaces for source connectors to enable easy addition of new sources without platform rebuild.
Collaborate with parsing and curation teams to enhance retention of valuable code, math, and structured content and improve data yield.
Implement provenance and policy enforcement in partnership with legal, privacy, security, and data governance teams.
Own production readiness aspects including capacity planning, observability, fault tolerance, recovery, and cost efficiency.
Instrument the pipeline end-to-end and establish service objectives for throughput, freshness, reliability, and training cutoffs compliance.
Work with research and product teams to evaluate crawls and external datasets, delivering reproducible data releases linked to training experiments.
Coordinate dependencies across teams balancing immediate data delivery with long-term platform development.

Requirements

Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, or related field, or equivalent experience.
15+ years of experience in software engineering with expertise in designing and operating large-scale distributed systems.
Extensive knowledge in web crawling or related infrastructure including distributed scheduling, stateful services, storage, and high-throughput data processing.
Strong hands-on skills in engineering, system design, architecture, implementation, performance analysis, and production debugging.
Experience with batch or streaming pipelines, consistent processing, retries, deduplication, backpressure, and recovery, with proven improvements in throughput and reliability.
Proven ability delivering data platforms for research, ML, search, or technical users with clear interface translation and measurable outcomes.
Strong technical communication and leadership abilities influencing engineering, product, research, and governance teams.

Experience architecting a web-wide crawl platform and scaling crawl frontier storage or consolidating acquisition systems.
Direct experience providing foundation-model pretraining data focusing on targeted discovery, quality-aware crawl ranking, and technical content acquisition.
Expertise implementing publisher consent, training-use opt-outs, dataset licensing controls, or provenance infrastructure at scale.
Background with browser-based or agentic data collection, programmable extraction, and sandboxed environments for tool-use or post-training evaluation.
Experience onboarding licensed or multimodal datasets, reusable partner integrations, or delivering streaming feeds with low latency for retrieval and grounding.