NodeJS robots.txt parser with support for wildcard (*) matching.
-
Updated
Mar 29, 2022 - JavaScript
#
robots-exclusion-standard
Here are 9 public repositories matching this topic...
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
search-engine
whitelist
user-agent
seo
crawling
twitterbot
robots-txt
googlebot
crawlers
web-crawling
bingbot
robots-exclusion-standard
blocking-bots
web-robots
search-engine-optimization
baiduspider
-
Updated
Nov 4, 2021
Go robots.txt parser
-
Updated
Nov 27, 2017 - Go
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
-
Updated
May 6, 2022 - JavaScript
Alternative robots parser module for Python
-
Updated
Apr 25, 2022 - Python
Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping
web-scraping
robots-txt
web-crawling
hacktoberfest
robots-exclusion-standard
robots-exclusion-protocol
robots-txt-parser
-
Updated
May 2, 2022 - PHP
Fully native robots.txt parsing component without any dependencies.
-
Updated
Dec 12, 2020 - JavaScript
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
robots-txt
robots
meta-tags
robots-parser
robots-exclusion-standard
robotstxt
robots-tag
robots-exclusion-protocol
x-robots-tag
-
Updated
Dec 3, 2021 - C#
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
-
Updated
Aug 20, 2020 - C++
Improve this page
Add a description, image, and links to the robots-exclusion-standard topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the robots-exclusion-standard topic, visit your repo's landing page and select "manage topics."