generated from carpentries/workbench-template-rmd
-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.Rmd
180 lines (142 loc) · 7.79 KB
/
index.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
---
title: "Web scraping with R and rvest"
site: sandpaper::sandpaper_site
---
The lessons below were designed for those interested
in using R to obtain data from websites.
This introduction assumes no programming experience and should be used
as a supplement to one of the other Carpentries R lessons, either before
taking an R lesson for a quick introduction to R, or after an introduction
to R, to examine another use case. The lessons can be taught in about 2 hours.
The first lesson introduces web scraping, then follows an introduction to
HTML, scraping data using rvest, plotting the scraped data and a conclusion.
This lesson assumes no prior knowledge of R or RStudio and no programming
experience.
## Episodes
1. [Before we start](00-before-we-start.html)
2. [Introduction: What is web scraping?](01-intro.html)
3. [What is HTML?](02-html.html)
4. [Web scraping with rvest](03-scraping.html)
5. [Plotting scraped data](04-plotting.html)
6. [Conclusion](05-conclusion.html)
## Preparations
Carpentry's teaching is hands-on, and to follow this lesson
learners must have R and RStudio installed on their computers. They also need
to be able to install a number of R packages, create directories, and download
files.
To avoid troubleshooting during the lesson, learners should follow the
instruction below to download and install everything beforehand.
If they are using their own computers this should be no problem,
but if the computer is managed by their organization's IT department
they might need help from an IT administrator.
### Install R and RStudio
R and RStudio are two separate pieces of software:
- **R** is a programming language that is especially powerful for data
exploration, visualization, and statistical analysis
- **RStudio** is an integrated development environment (IDE) that makes using
R easier. In this course we use RStudio to interact with R.
If you don't already have R and RStudio installed, follow the instructions for
your operating system below. You have to install R before you install RStudio.
#### Windows
- Download R from the
[CRAN website](https://cran.r-project.org/bin/windows/base/release.htm).
- Run the `.exe` file that was just downloaded
- Go to the [RStudio download page](https://www.rstudio.com/products/rstudio/download/#download)
- Under *All Installers*, download the RStudio Installer for Windows.
- Double click the file to install it
- Once it's installed, open RStudio to make sure it works and you don't get any
error messages.
##### MacOS
- Download R from
the [CRAN website](https://cran.r-project.org/bin/macosx/).
- Select the `.pkg` file for the latest R version
- Double click on the downloaded file to install R
- It is also a good idea to install [XQuartz](https://www.xquartz.org/) (needed
by some packages)
- Go to the [RStudio download page](https://www.rstudio.com/products/rstudio/download/#download)
- Under *All Installers*, download the RStudio Installer for MacOS.
- Double click the file to install RStudio
- Once it's installed, open RStudio to make sure it works and you don't get any
error messages.
##### Linux
- Follow the instructions for your distribution
from [CRAN](https://cloud.r-project.org/bin/linux), they provide information
to get the most recent version of R for common distributions. For most
distributions, you could use your package manager (e.g., for Debian/Ubuntu run
`sudo apt-get install r-base`, and for Fedora `sudo yum install R`), but we
don't recommend this approach as the versions provided by this are
usually out of date. In any case, make sure you have at least R 3.3.1.
- Go to the
[RStudio download page](https://www.rstudio.com/products/rstudio/download/#download)
- Under *All Installers*, select the version that matches your distribution and
install it with your preferred method (e.g., with Debian/Ubuntu `sudo dpkg -i rstudio-YYYY.MM.X-ZZZ-amd64.deb` at the terminal).
- Once it's installed, open RStudio to make sure it works and you don't get any
error messages.
### Update R and RStudio
If you already have R and RStudio installed, first check if your R version is
up to date:
- When you open RStudio your R version will be printed in the console on
the bottom left. Alternatively, you can type `sessionInfo()` into the console.
If your R version is 4.0.0 or later, you don't need to update R for this
lesson. If your version of R is older than that, download and install the
latest version of R from the R project website
[for Windows](https://cran.r-project.org/bin/windows/base/),
[for MacOS](https://cran.r-project.org/bin/macosx/),
or [for Linux](https://cran.r-project.org/bin/linux/)
- It is not necessary to remove old versions of R from your system,
but if you wish to do so you can check
[How do I uninstall R?](https://cran.r-project.org/bin/windows/base/rw-FAQ.html#How-do-I-UNinstall-R_003f)
- Note: The changes introduced by new R versions are usually backwards-compatible.
That is, your old code should still work after updating your R version.
However, if breaking changes happen, it is useful to know that you can have
multiple versions of R installed in parallel and that you can switch between
them in RStudio by going to `Tools > Global Options > General > Basic`.
- After installing a new version of R, you will have to reinstall all your packages
with the new version. For Windows, there is a package called `installr` that can
help you with upgrading your R version and migrate your package library.
To update RStudio to the latest version, open RStudio and click on
`Help > Check for Updates`. If a new version is available follow the
instruction on screen. By default, RStudio will also automatically notify you
of new versions every once in a while.
### Install required R packages
During the course we will need a number of R packages. Packages contain useful
R code written by other people. We will use the packages
`rvest` and `tidyverse`.
To try to install these packages, open RStudio and copy and paste the following
command into the console window (look for a blinking cursor on the bottom left),
then press the <kbd>Enter</kbd> (Windows and Linux) or <kbd>Return</kbd> (MacOS)
to execute the command.
```{r, eval=FALSE}
install.packages(c("tidyverse", "rvest", "ggplot2", "httr", "stringr"))
```
Alternatively, you can install the packages using RStudio's graphical user
interface by going to `Tools > Install Packages` and typing the names of the
packages separated by a comma.
R tries to download and install the packages on your machine.
When the installation has finished, you can try to load the
packages by pasting the following code into the console:
```{r, eval=FALSE}
library(tidyverse)
library(rvest)
library(ggplot2)
library(httr)
library(stringr)
```
If you do not see an error like `there is no package called ‘...'` you are good
to go!
### Updating R packages
Generally, it is recommended to keep your R version and all packages
up to date, because new versions bring improvements and important bugfixes.
To update the packages that you have installed, click `Update` in the
`Packages` tab in the bottom right panel of RStudio, or go to
`Tools > Check for Package Updates...`.
Sometimes, package updates introduce changes that break your old code,
which can be very frustrating. To avoid this problem, you can use a package
called `renv`. It locks the package versions you have used for a given project
and makes it straightforward to reinstall those exact package version in a
new environment, for example after updating your R version or on another
computer. However, the details are outside of the scope of this lesson.
## Acknowledgments
This lesson draws material from the [R Ecology lesson](https://datacarpentry.org/R-ecology-lesson/),
[R Social Science lesson](https://datacarpentry.org/r-socialsci) and
the [Introduction to Web Scraping lesson](https://librarycarpentry.org/lc-webscraping).