List of libraries, tools and APIs for web scraping and data processing.

Async Python 3.6+ web scraping micro-framework based on asyncio

Web Scan Lazy Tools - Python Package

SpideyX a multipurpose Web Penetration Testing tool with asynchronous concurrent performance with multiple mode and configurations.

Crawlzone is a fast asynchronous internet crawling framework for PHP.

Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)

🌌 High productivity semi-automatic crawler generator 🛠️🧰

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖

Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.

A Python framework to build polite, but tenacious crawlers / scrapers with a MariaDB backend

Easily crawl news portals or blog sites using Storm Crawler.

Web crawling & scraping framework for Node.js on top of headless Chrome browser

🔍 A powerful web-crawling framework, based on aiohttp.

BFS and DFS BASED Project.A neural network-powered web crawler that intelligently extracts, classifies, and processes web content using deep learning

An intelligent proxy server. Provide durable, real-time, high-quality proxies as a middleman or datasource server.

Crawler written in TypeScript using ES6 generators.

基于python协程池、用法灵活的高性能爬虫框架

A crawler program to extract all of the data and the price for symbols in the global stock exchange.

🚀 THIS WEB SCRAPING TEMPLATE PROVIDES YOU WITH A GREAT STARTING POINT WHEN CREATING WEB SCRAPING BOTS. 🤖