Senior Web Scraping Engineer
hace 2 días
Newcastle upon Tyne
Senior Web Scraping Engineer 📍 Remote | 🌍 International Team | 🕒 Full-time At WebAutomation, we help businesses unlock the power of public web data at scale. Our products and custom scraping solutions empower organizations to access accurate, structured, and reliable datasets. We are an international, remote-first team passionate about cutting-edge web scraping technologies and large-scale data automation. Role Overview We are looking for a Senior Web Scraping Engineer with strong expertise in Python and Node.js to lead complex data extraction projects. You will design scalable scrapers, overcome advanced anti-bot systems, and build robust pipelines that power our products. This role is ideal for someone who thrives on solving tough technical challenges, can effectively manage multiple projects in parallel, and quickly adapts to changing priorities in a fast-paced environment. Key Responsibilities • Build and maintain advanced scrapers using Python, Node.js, Scrapy, Puppeteer, Playwright, Selenium, • Optimize pipelines for scalability, efficiency, and minimal resource usage, • Debug, maintain, and enhance existing scraping systems under changing site structures, • Handle multiple scraping projects simultaneously while meeting deadlines and quality standards, • Adapt quickly to daily priority changes and shifting project requirements, • Collaborate with product and data teams to ensure accuracy and reliability, • Mentor junior engineers and contribute to best practices in large-scale scraping, • Proficiency in reverse engineering web applications using browser DevTools, network analysis, and debugging proxies (Charles, Burp Suite, mitmproxy), • Expertise in analyzing and extracting data from GraphQL endpoints, including introspection queries and schema discovery, • Experience with WebSocket reverse engineering for real-time data extraction, • Proficiency in analyzing API authentication flows (OAuth, JWT tokens, API key rotation, session management), • Knowledge of browser automation detection evasion (navigator properties, WebDriver detection, Chrome DevTools Protocol), • Experience bypassing CloudFlare, PerimeterX, DataDome and other anti-bot solutions, • Skills in analyzing and replicating complex request signatures and encryption mechanisms Must-Have Experience • 5+ years in web scraping, backend engineering, or data automation, • Expert in Python (asyncio, threading) and Node.js (async/await), • Hands-on with Scrapy, Puppeteer, Playwright, Selenium, • Strong debugging and problem-solving skills for complex scrapers, • Strong multi-tasking skills with the ability to prioritize work in a dynamic environment, • Database experience (SQL/NoSQL, Redis) and caching systems, • Fluent English (written & spoken) Nice to Have • Experience with Flask, Django, FastAPI, • Docker, Kubernetes, and CI/CD pipelines, • R&D or security research background, • Open-source contributions in scraping/data engineering What We Offer • Competitive salary based on expertise, • 100% remote & flexible work environment, • Opportunity to work on cutting-edge web & app scraping projects, • A collaborative international team that values innovation and autonomy, • Professional growth through challenging projects & mentorship 🚀 Ready to push the boundaries of web data extraction? Apply now and join us at WebAutomation to shape the future of large-scale web scraping.