An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.
29
Public repositories
74,276
Total stars
808
Followers
The Scrapy project has a significant public presence on GitHub, showcasing a wide range of repositories primarily written in Python, HTML, C++, DIGITAL Command Language, and Shell. Notable repositories include Scrapy, a high-level web crawling and scraping framework, and Scrapyd, a service daemon for running Scrapy spiders.
Top languages
Public repositories
scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
scrapyd
A service daemon to run Scrapy spiders
scrapely
A pure-python HTML screen-scraping library
dirbot
Scrapy project to scrape public web directories (educational) [DEPRECATED]
quotesbot
This is a sample Scrapy project for educational purposes
parsel
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors
scrapyd-client
Command line client for Scrapyd server
w3lib
Python library of web-related functions
cssselect
CSS Selectors for Python
queuelib
Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python
loginform
Fill HTML login forms automatically
slybot
No description provided for this repository.
protego
A pure-Python robots.txt parser with support for modern conventions.
itemadapter
Common interface for data container classes
scrapy.org
The scrapy.org website (old code)
itemloaders
Library to populate items using XPath and CSS with a convenient API
booksbot
A crawler for http://books.toscrape.com
scrapy-bench
A CLI for benchmarking Scrapy.
scrapy-lint
A linter for Scrapy projects.
scurl
Performance-focused replacement for Python urllib
pypydispatcher
A fork of http://pydispatcher.sourceforge.net/ with PyPy support
xtractmime
https://mimesniff.spec.whatwg.org/ implementation for Python
base-chromium
base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/
scrapy-itemloader
[Archived] Library to populate Scrapy items using XPath and CSS with a convenient API
form2request
Python library to build HTTP requests out of HTML forms
url-chromium
url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url
gsoc2014-integration-tests
GSoC2014 - Scrapy Integration tests project
scrapy-bench-speedcenter
Codespeed for scrapy-bench
sphinx-scrapy
Sphinx extension for documentation in the Scrapy ecosystem
Frequently asked questions
What does scrapy build on GitHub?
Scrapy builds a variety of tools and libraries on GitHub, including Scrapy for web crawling, Scrapyd for managing Scrapy spiders, and Parsel for data extraction. These projects are designed for web scraping and data collection.
Which programming languages does scrapy use?
The primary programming languages used by Scrapy on GitHub are Python and HTML, with additional contributions in C++, DIGITAL Command Language, and Shell. This diverse language use supports various functionalities in their projects.
Are scrapy's repositories public?
Yes, all of Scrapy's repositories on GitHub are public. This transparency allows users and developers to collaborate, contribute, and review the code, making it accessible for educational and practical use.
Is this exposure intended?
Monitor Scrapy project with RepoGuard and get alerted the moment a new public repository appears.
Monitor this account