Design and enhance scalable, fault-tolerant data architectures supporting real-time and batch processing using Apache Kafka, Apache Flink, and lakehouse technologies.
Improve data semantics, classification, and discovery to enhance data accessibility and usability across eBay.
Develop and maintain automated data pipelines for efficient and reliable data processing from multiple sources into lakehouse architecture.
Optimize data consumption through materialized views and data subsumption techniques to reduce redundancy and improve performance.
Collaborate with data scientists, business analysts, and engineering teams to convert requirements into effective data solutions.
Explore and integrate emerging data engineering technologies to improve scalability, efficiency, and reliability of data infrastructure.
Requirements
Experience with Apache Kafka, Apache Flink, and other streaming technologies for large-scale data processing.
Strong understanding of lakehouse architectures and hands-on experience in large-scale data solution implementation.
Knowledge of data semantics, data discovery, and governance for effective data management.
Experience designing, developing, and automating data pipelines, including materialized views.
Familiarity with data subsumption techniques to reduce data redundancy and improve system performance.
Proficiency in Java, Scala, Python, or similar programming languages.
5+ years of relevant software engineering or data engineering experience.