d-alex-hughes / 241_revisions

Experiments and Causality

Include a link: http://beta.mybinder.org/v2/gh/binder-examples/r/master?filepath=index.ipynb

[](https://mybinder.org/v2/gh/d-alex-hughes/241_revisions/master?filepath=week_02%2Fnotebook_2.5.ipynb)]

Introduction

This course introduces students to experimentation in the social sciences. This topic has increased considerably in importance since 1995, as researchers have learned to think creatively about how to generate data in more scientific ways, and developments in information technology has facilitated the development of better data gathering. Key to this area of inquiry is the insight that correlation does not necessarily imply causality. In this course, we learn how to use experiments to establish causal effects, and how to be appropriately skeptical of findings from observational data.

Our goals for each student in the course:

• Become skeptical about claims of causality. When faced with a piece of research on observational data, you should be able to tell stories that illustrate possible flaws in the conclusions.
• Understand why experimentation solves the basic causal-inference problem. You should be able to describe several examples of successful experiments and what makes you feel confident about their results.
• Appreciate the difference between laboratory experiments and field experiments.
• Appreciate how information systems and websites can be designed to make experimentation easy in the modern online era
• Understand how to quantify uncertainty, using confidence intervals and statistical power calculations
• Understand why control groups and placebos are both important.
• Design, implement, and analyze your own field experiment.
• Appreciate a few examples of what can go wrong in experiments. Examples include administrative glitches that undo random assignment, inability to fully control the treatment (and failure to take this inability into account), and spillovers between subjects.

Description

This course begins with a discussion of the issues with causal inference based on observational data. We recognize that many of the decisions that we care about, whether they be business related or theoretically motivated, are essentially causal in nature.

The center of the course builds out an understanding of the mechanics of estimating a causal quantity. We present two major inferential paradigms, one new and one you are likely familiar with. We first present randomization inference as a unifying, intuitive inferential paradigm. We then demonstrate how this paradigm sits in complement to the classical frequentist inferential paradigm. These concepts in hand, we turn focus to the design of experiments and place particular focus both answering the question that we set out to answer, and achieving maximally powered experiments through design.

The tail of the course pursues two parallel tracks. In the first, students form a research question that requires a causal answer and design and implement the experiment that best answers this question. At the same time, new content presented in the course focuses on the practical stumbling blocks in running an experiment and the tests to detect these stumbling blocks.

Computing is primarily conducted in R, though several students have worked in python for the entirety of the course.

Specifics

Office Hours

Each instructors’ office hours are open to all students in all sections. We’re happy to talk about any aspect of the course, or the data science program in our office hours.

Day	Time	Instructor	Link
Monday	12:30-1:30p	Alex	https://zoom.us/j/100396929
Tuesday	6:30-8:30p	Daniel	https://berkeley.zoom.us/j/2154325716
Wednesday	5:30-6:30p	Alex	https://zoom.us/j/100396929
Thursday	5:30-6:30	Micah	https://zoom.us/j/2949146457

Schedule

Week	Topics	Async Reading	Sync Reading	Assignment Due
1	Experimentation k	FE 1, NYT	Feynman, News 1, 2, Predict or Cause	–
2	Apples to Apples	FE 2; Lewis & Reiley (p. 1-2.5, §1; §2A-B)	MTGI 1,5,8,9; Lakatos (O) Rubin, sections 1 & 2	Essay 1
3	Quantifying Uncertainty	FE 3.0, 3.1, 3.4	Blackwell, Lewis and Rao 1, 3.1, 3.2	PS 1, Revised Essay 1
4	Blocking and Clustering	FE 3.6.1, 3.6.2, 4.4, 4.5	(O): Cluster Estimator, BlockTools	Essay 2
5	Covariates and Regression	MM 1, FE 4.1-3, MM 2, MHE p. 16-24	Opower (O): FE Appendix B (p. 453)	PS 2, Revised Essay 2
6	Regression; Multi-factor Experiments	MM 6.1, MM 95-97, FE 9.3.3, 9.4	Montgomery Sections 1, 3.0, 3.1, 3.2, 3.5, 4.2, Skim 5	Vote on Projects
7	HTE	FE 9, Multiple Comparisons, and Demo	RI HTE, Goodson (O): JLR 1, 2, 3.1, 4.3	–
8	Incomplete Control of Delivery	FE 5	G&G 2005; TD, Ch 7; TD, Ch 9	PS 3
9	Spillover	FE 8 and lyft and (O) uber	Miguel and Kremer; Blake and Cohey 2, 3	Progress Report
10	Problems, Diagnostics and the Long View	FE 11.3	DiNardo and Pischke, Simonsohn	–
11	Causality from Observation?	MM 3.1, 4.1, 5.1	Incinerators, Washington, Dee (O): Lalive, Rubin, Section 3	PS 4
12	Attrition, Mediation, Generalizabilty	FE 7, 10, Bates 2017	Alcott and Rogers	Peer Eval 1
13	Creative Experiments	FE 12, (O): Ny Mag, Science, FE 13	Broockman Irregularities, Hughes et al. (O): Uber Platform
14	Final Thoughts		Freedman	PS 5
15	–	(O): Retracted LaCour, (tl;dr), Podcat (audio))		Final Paper, Peer Eval 2

Books

We use two books in this course, and read a third book in the second week. We recommend that you buy a paper copy of the two textbooks (we’ve chosen textbooks that have a fair price), and would understand if you digitally read the third book.

• Field Experiments: Design and Analysis is the core textbook for the cousre. It is available at Amazon for $40.
• Mastering Metrics is the secondary textbook for the course. It is available at Amazon for $20.
• More than Good Intentions is the third book for the cousre. It is available at Amazon for $10, new, or $3 used. But, you could also read this digitally.

Articles

• We have made all the articles we read in the couse available in the repository. However, it is a great practice to get used to establishing a VPN to gain access to all the journal articles that are available through the library subscription service. Instructions for connecting are here. Journal access is one of the greatest benefits to belonging to a University, we suggest you use it.
• David has made a great resource that has suggestions for further reading. You can access this here.

Grading and Scoring

Assignments

• Problem Sets (50%, 10% each) A series of problem sets, with some questions drawn from FE, many requiring programming or analysis in R.
  • We encourage you to work together on problem sets, because great learning can come out of helping each other get unstuck. We ask that each person independently prepare his or her own problem-set writeup, to demonstrate that you have thought through the ideas and calculations and can explain them on your own. This includes making sure you run any code yourself and can explain how it works. Collaboration is encouraged, but mere copying will be treated as academic dishonesty.
  • At this point, the course has lived for a number of semesters, and we have shared solution sets each semester. We note in particular that struggling with the problems is a key part of the learning in this course. Copying from past solutions constitutes academic dishonesty and will be punished as such; you should know that we have included language in the solutions that will make it clear when something has been merely copied rather than understood.
• Essays (20%, 10% each)
  • In the first essay we will ask you to examine an existing causal claim that is based on observational data.
  • In the second essay we will ask you to propose an experiment that you and two or three classmates can conduct.
• Class Experiment (30%) In teams of 3-5 studetns, carry out a pilot experiment that measures a causal effect of interest.