canCrawl() returns incorrect result when matching middle of path #8
Description
robots.txt:
User-agent: *
Disallow: /rss
Allow: /
canCrawl() thinks this means /home/rssa cannot be crawled but that is incorrect.
{{ message }}
robots.txt:
User-agent: *
Disallow: /rss
Allow: /
canCrawl() thinks this means /home/rssa cannot be crawled but that is incorrect.