Tying in Evidence
Evidence is critical to putting the “science” in political science. there are many kinds of evidence and many kinds of standards and measures to consider when evaluating evidence. But whatever the evidence you use, it is important that it be a direct test of the implications of your theory.
How do I use evidence effectively?
The effectiveness of your evidence is measured by the degree to which it tests your theory or argument. The size of a dataset, the richness of a case-study, or the novelty of an experimental design do not matter if these tests are not directly tied to the argument being made. Identifying how your evidence relates to your argument requires three steps.
The first is to identify the individual claims or causal steps being made by your theory. You can think of this as “unpacking” the steps in your argument. What does your theory say about how the world works?
Then, you will need to draw out the implications of those claims for what we should expect to observe in the real world. If your theory is true, what would we expect to see in X case, or X data? What would we observe that would prove your claims wrong? Making sure that your argument is falsifiable is necessary in order to complete this second step.
Finally, you will explain how your evidence tests your claims by capturing the phenomenon you are interested in and how it reflects (or fails to reflect) the expectations of your theory. Your evidence should capture a real world example of the implications you previously described.
The following video will show you how to unpack the causal steps of your argument, identify the implications of your theory, and link your evidence to those implications. You can find the slides used in the video below.
What kind of data can I use?
Unpacking the claims made in your argument and testing the implications of those claims against the real world will clarify how your evidence supports your argument. Those three steps can be applied regardless of the type of evidence you collect.
The specific kinds of evidence you use to substantiate your argument will vary based on the type of argument you make, the best way to test it, the data available to you, and the conventions of the subfield you are working in. Nevertheless, you should always relate your evidence to your argument by explaining how it fits with the implications of each of your claims.
Your evidence is typically drawn from analysis of some data, such as an observational dataset, a series of experiments, a set of historical records, interviews, or a case study of a particular country. You can analyze that data in a range of ways to test your argument, such as through statistical regression, close reading, or process tracing. Whatever evidence you use, what is most important is that you explain how it puts the implications of your argument to the test.
There is no definitive list of the types of evidence that are viable in political science, but some are more common than others. Below, you can find examples of some common types of evidence and how political scientists deploy them to test their arguments.
Case studies involve looking closely at a particular place or historical moment to test predictions about what should happen under a certain theory against facts about what did happen. While a single case study can provide evidence for an argument, political scientists more often use multiple case studies to evaluate theories with greater nuance. A common method of analysis is the “method of difference,” in which a researcher compares two cases that have common background characteristics differ in some outcome of interest. The researcher then shows how their theory points to some point of difference that explains why these two cases diverged. While case studies may be drawn from secondary sources alone, they often draw on other sources of data, such as archival documents, original interviews, or observational datasets.
Surveys generate data by asking a representative sample of people a series of questions and quantifying their answers, yielding a dataset of observations about their opinions, beliefs, and behaviors. When coupled with data about respondents’ identities, they provide powerful tools for testing theories about how people think about politics. Like interviewers, survey researchers must carefully design their surveys to effectively operationalize the independent and dependent variables implied by the theories they’d like to test and make sure respondents’ answers are clear and informative. Surveys can be administered multiple times to the same sample over a given period to gather data about how peoples’ opinions change in response to changes in their political environment.
Observational datasets are collections of observations across many different units like countries or legislators and/or track observations within a single unit over time. The “N” simply refers to the number of observations in the sample: a dataset can include a few hundred observations or a few hundred thousand, often even more. Typically, political scientists use regression analysis to test their predictions by detecting whether the correlation between variables implied by a theory appear in the data. A wide variety of different types of observations enable researchers to “control” for other factors that might influence the relationship of interest such that they can distinguish their explanation from other potential explanatory forces. For example, a political scientist might assemble a dataset of congressional floor votes to test whether close elections lead representatives to cross the aisle more often. They might also collect and control for measures of legislator ideology to rule out the possibility that competitive elections yield more moderate candidates who prefer centrist policy.
Interviews involve sitting down with political actors, be they elites or ordinary people, and asking them questions to draw out data about their judgments, thought processes, and inner lives not easily captured in surveys or observational datasets. What separates interviewing in social science from interviews in journalism is that researchers typically ask multiple interviewees the same set of questions rather than going with the flow of each conversation. This enables researchers to identify important points of variation between different types of interviewees and attribute variation in answers to those differences. Thus, the design of interview questions and the selection of interviewees are important elements of ensuring interviews genuinely test theories as opposed to just offering a description of the world. What would we expect a respondents to say if your theory was true? What could they say that would falsify your theory?
An experiment is a study aims to test a claim by isolating the effect of some variable (a “treatment”) on some outcome by creating a stylized environment that holds all other factors constant. In an experiment, a researcher will randomly assign members of a sample to treatment and control groups, administer a treatment to the treatment group, and then measure the differences between the two groups in some outcome of interest. So long as the experiment is properly designed, any significant difference between treatment and control groups can be attributed to the treatment. For that reason, much rides on designing the experiment right: a researcher typically must show that the treatment has been truly randomized and that both it and the measured outcome reasonably operationalize the independent and dependent variables specified by the theory. For many research questions, experiments are not a viable option – we can’t randomize a pool of countries or experimentally apply a treatment like war. In these cases, political scientists will attempt to approximate experimental methods through statistical methods or exploiting semi-random events.
Like case studies, archival research involves gathering original data from historical documents to test theories that explain why a political event or outcome occurred. Archival research itself doesn’t test theories, but it yields novel data sources that can both enrich our understanding of cases and test claims outright through other forms of analysis. For example, primary sources like official memos or letters can yield insights about the factors that influenced actors’ decisions at a specific historical juncture. Alternately, archival material can be compiled into observational datasets that can be analyzed using quantitative methods. However one chooses to go about analyzing archival records, the same social scientific principles apply: records do not just speak for themselves, but must be assessed in reference to the expectations of one’s theory.