robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.

Version: 0.4.1
Depends: R (≥ 3.0.0)
Imports: stringr (≥ 1.0.0), httr (≥ 1.0.0), magrittr
Suggests: knitr, rmarkdown, dplyr, testthat, covr
Published: 2017-09-01
Author: Peter Meissner [aut, cre], Oliver Keys [ctb], Rich Fitz John [ctb]
Maintainer: Peter Meissner <retep.meissner at gmail.com>
BugReports: https://github.com/ropenscilabs/robotstxt/issues
License: MIT + file LICENSE
URL: https://github.com/ropenscilabs/robotstxt
NeedsCompilation: no
Materials: README NEWS
CRAN checks: robotstxt results

Downloads:

Reference manual: robotstxt.pdf
Vignettes: using_robotstxt
Package source: robotstxt_0.4.1.tar.gz
Windows binaries: r-devel: robotstxt_0.4.1.zip, r-release: robotstxt_0.4.1.zip, r-oldrel: robotstxt_0.4.1.zip
OS X El Capitan binaries: r-release: robotstxt_0.4.1.tgz
OS X Mavericks binaries: r-oldrel: robotstxt_0.4.1.tgz
Old sources: robotstxt archive

Reverse dependencies:

Reverse suggests: spiderbar

Linking:

Please use the canonical form https://CRAN.R-project.org/package=robotstxt to link to this page.