-
Notifications
You must be signed in to change notification settings - Fork 10
/
Copy path03-testing.Rmd
161 lines (101 loc) · 6.71 KB
/
03-testing.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
# Application Testing: `shinytest`
- You've developed a nice app
- You've put it in production
- You want to be confident that it will keep running in the future
**Things that can change/break a Shiny application**
- Modifying code
- Upgrading the `shiny` package
- Upgrading other packages
- Upgrading R
- External data source changes or fails
## Testing Options
- Manual testing
- time intensive
- inconsistent
- Automated testing (hard)
- web browser
- simulated user interactions
- tests for graphical elements
Enter: Snapshot-based testing [Webinar Slide Deck](https://github.com/rstudio/webinars/blob/master/48-shinytest/shinytest.pdf)
- Can be easier to create tests
- Can test an entire application
- More sensitive to spurious changes
**Automated Testing for Shiny Apps**
- [Blog](https://resources.rstudio.com/rstudio-blog/shinytest-automated-testing-for-shiny-apps)
`shinytest` is a package (available on CRAN) to perform automated testing for Shiny apps.
Basic `shinytest` proceedure:
Support for `shinytest` is available in RStudio v1.2 preview.
**Installation**
`install.packages("shinytest")`
**Note:** When running `shinytest` for the first time, you may be prompted by the RStudio IDE or package warning messages to install some dependencies. `shinytest` requires a headless web browser (PhantomJS) to record and run tests.
- To install it, run shinytest::installDependencies()
- If it is installed, make sure the phantomjs executable can be found via the PATH variable.
**Record Tests**
- Run `recordTest()` to launch the app in a test recorder.
- Create the tests by interacting with the application - this will allow the recorder to snapshot the application state at various points.
- Quit the test recorder. This action will trigger the following events:
- The test script will be saved as a .R file in a subdirectory of the application named `tests/`.
- If you are running in the RStudio IDE, it will automatically open this file in the editor.
- The test script will be run, and the snapshots will be saved in a subdirectory of the `tests/` directory.
To record tests from `R`, run the following:
```
library(shinytest)
recordTest("path/to/the/app") #Replace with the correct path
```
To record tests from RStudio v1.2, when an application file (app.R, server.R, ui.R or global.R) is open in the editor, a button labeled _Run App_ will appear at the top of the editor pane. Click on the small black triangle next to this button to reveal the menu of extended options.
This launches the Shiny application to be tested in a separate R process. We'll refer to this as the **target app**. At the same time, the current R process lauches a special Shiny application which displays the target app in an iframe along with some controls. We'll refer to this as the **recorder app**. You should see something like this:
The panel on the right displays some controls for the test recorder, as well as a list of **recorded events**. As you interact with the target app, you will see those interactions appear in the recorded events list.
For testing a Shiny application, interacting with the inputs is only one part of the equation. It's also necessary to check that the application produces the correct outputs. This is accomplished by taking **snapshots** of the application's state.
To take a snapshot of the application's state, click the _Take snapshot_ button on the recorder app. This will record all input values, output values, and exported values.
**Running Tests**
When you quit the test recorder, it will automatically run the test script. There are three separate components involved in running tests:
1. First is the **test driver**. This is the R process that coordiates the testing and controls the web browser. When working on creating tests interactively, this is the R process that you use.
2. Next is the **Shiny process**, also known as the **server**. This is the R process that runs the target Shiny application.
3. Finally, there is the **web browser**, also known as the **client**, which connects to the server. This is a headless web browser - one which renders the web page internally, but doesn't display the content to the screen (PhantomJS).
So, when you exit the test recorder, it will by default automatically run the test script and print something like this:
```
Saved test code to /path/to/app/tests/mytest.R
Running mytest.R
====== Comparing mytest ...
No existing snapshots at mytest-expected/. This is a first run of tests.
Updating baseline snapshot at tests/mytest-expected
Renaming tests/mytest-current
=> tests/mytest-expected.
```
This is the result of running `testApp()`, which can also be manually run by providing the desired application and test like this:
`testApp("exampleApp", "mytest")`
The built-in integreation with RStudio v1.2 provides `Run Tests` as a drop down menu option in your Shiny app source file (see it located under the _Record Test_ option in the screenshot above).
**Subsequent Test Runs**
After the initial test run, you can continue to run the tests to check for changes in application behavior.
If there are any differences between current and expected results, the test output will look something like this:
```
Running mytest.R
====== Comparing mytest ...
Differences detected between mytest-current/ and mytest-expected/:
Name Status
001.json != Files differ
001.png != Files differ
Would you like to view the differences between expected and current results [y/n]?
```
To view failed tests in the RStudio IDE, go to the _Build tab_ and make sure the _issues toggle_ is selected:
For each test with different results, you can see the differences between the expected and current results.
**Testing Code**
The `shinytest` package was created for testing Shiny applications on the interaction-level. To test Shiny code and functions, we suggest using the `testthat` package.
[Info](http://testthat.r-lib.org/)
[GitHub](https://github.com/r-lib/testthat)
Learn about testing and how to setup test workflow and structure: [R packages by Hadley Wickham](http://r-pkgs.had.co.nz/tests.html)
## Activity: `shinytest`
**First: Open app.R and use 'Record Test'**
**Discussion:**
_Understanding shinytest_
- What does the recording file create?
- What challenges might our app pose to testing?
- Can you run the tests and get a success?
- How might you automate tests?
**Deliverable: Try `shinytest`**
- A set of tests that can catch unintentional changes to our app.