spiderbar: Parse and Test Robots Exclusion Protocol Files and Rules

The 'Robots Exclusion Protocol' <http://www.robotstxt.org/orig.html> documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' <https://github.com/seomoz/rep-cpp> C++ library for processing these 'robots.txt' files.

Version: 0.2.0
Depends: R (≥ 3.2.0)
Imports: Rcpp
LinkingTo: Rcpp
Suggests: testthat, covr, robotstxt
Published: 2017-09-25
Author: Bob Rudis (bob@rud.is) [aut, cre], SEOmoz, Inc [aut]
Maintainer: Bob Rudis <bob at rud.is>
BugReports: https://github.com/hrbrmstr/spiderbar/issues
License: MIT + file LICENSE
URL: https://github.com/hrbrmstr/spiderbar
NeedsCompilation: yes
SystemRequirements: C++11
Materials: README NEWS
CRAN checks: spiderbar results

Downloads:

Reference manual: spiderbar.pdf
Package source: spiderbar_0.2.0.tar.gz
Windows binaries: r-devel: spiderbar_0.2.0.zip, r-release: spiderbar_0.2.0.zip, r-oldrel: spiderbar_0.2.0.zip
OS X El Capitan binaries: r-release: spiderbar_0.2.0.tgz
OS X Mavericks binaries: r-oldrel: spiderbar_0.2.0.tgz

Linking:

Please use the canonical form https://CRAN.R-project.org/package=spiderbar to link to this page.