cneskey / sanders

WIP!!! — Web-scraping and Web-crawling Content Parsing, Validation and Sanitization Helpers

sanders

Web-scraping and Web-crawling Content Parsing, Validation and Sanitization Helpers

Description

When researchers crawl/scrape the web for content they are talking to strange computers running software created by a diverse array of humans. Content -- even content that is supposed to adhere to internet stadards -- can have very rough edges that need to be smoothed out to be useful and potentially less harmful to the systems running the scraping and analysis code. Methods are provided that sand off the rough edges of many different types of scraped content and metadata.

The following functions are implemented:

• normalize_media_types: Sand the rough edges off of bare, scraped media (MIME) types into a smooth data frame

Installation

devtools::install_github("hrbrmstr/sanders")

Usage

library(sanders)

# current verison
packageVersion("sanders")

## [1] '0.1.0'

content_type <- c("text/html; charset=utf-8", "text/css",
                  "text/javascript; charset=UTF-8", "text/javascript",
                  "application/x-javascript", "text/plain; charset=utf-8")

x <- normalize_media_types(content_type)

x

## # A tibble: 6 x 3
##          type      subtype     params
##         <chr>        <chr>     <list>
## 1        text         html <list [1]>
## 2        text          css <list [0]>
## 3        text   javascript <list [1]>
## 4        text   javascript <list [0]>
## 5 application x-javascript <list [0]>
## 6        text        plain <list [1]>

x$params

## [[1]]
## [[1]][[1]]
## # A tibble: 1 x 2
##       key value
##     <chr> <chr>
## 1 charset utf-8
## 
## 
## [[2]]
## list()
## 
## [[3]]
## [[3]][[1]]
## # A tibble: 1 x 2
##       key value
##     <chr> <chr>
## 1 charset utf-8
## 
## 
## [[4]]
## list()
## 
## [[5]]
## list()
## 
## [[6]]
## [[6]][[1]]
## # A tibble: 1 x 2
##       key value
##     <chr> <chr>
## 1 charset utf-8

Test Results

library(sanders)
library(testthat)

date()

## [1] "Thu Sep 21 18:50:27 2017"

test_dir("tests/")

## testthat results ========================================================================================================
## OK: 0 SKIPPED: 0 FAILED: 0
## 
## DONE ===================================================================================================================