Fiber AI (YC S23)
|Software Engineer
Remote, United States, US
Summary
Led the development of high-performance search infrastructure and data solutions, optimizing system efficiency and cost for an AI-focused startup.
Highlights
Engineered a scalable and performant search engine to query over 850M+ documents using Elasticsearch, reducing query computation time from 60-120 seconds to just 3-5 seconds.
Implemented advanced storage optimization and data sharding techniques, cutting annual AWS bills for elastic clusters from $100K to $15K annually, achieving an 85% cost reduction.
Developed a highly resilient LinkedIn job-posts scraper, capable of extracting 13M+ jobs in 2 weeks while navigating complex anti-scraping policies.
Designed and implemented solutions to geocode 30M+ locations, integrating rate limiters and concurrency mechanisms for stable, high-volume cron job execution.
Utilized JavaScript's Streams and advanced data structures/algorithms to optimize CPU-intensive computations, reducing cron job completion time from 3 weeks to 1 week.