A robots.txt parser alternative to Python's robotparser module
Project Links
Meta
Author: Philip Semanchuk
Classifiers
Development Status
- 5 - Production/Stable
Intended Audience
- Developers
License
- OSI Approved :: BSD License
Operating System
- MacOS :: MacOS X
- POSIX
- Unix
Environment
- Win32 (MS Windows)
Programming Language
- Python
- Python :: 2
- Python :: 3
Topic
- Utilities
Robotexclusionrulesparser is an alternative to the Python standard library module robotparser. It fetches and parses robots.txt files and can answer questions as to whether or not a given user agent is permitted to visit a certain URL.
This module has some features that the standard library module robotparser does not, including the ability to decode non-ASCII robots.txt files, respect for Expires headers and understanding of Crawl-delay and Sitemap directives and wildcard syntax in path names.
Complete documentation (including a comparison with the standard library module robotparser) is available in ReadMe.html.
Robotexclusionrulesparser is released under a BSD license.
Files in release
No dependencies