Methodology of the Pluralists’ Guide

A funny thing happened on the way to launching this website: some of the people we consulted raised concerns about our methodology. These extremely helpful people forced us to clarify our aims and refine our approach, with results that may be of some use if shared publicly. So what follows is a short explanation of how we think we’re doing what we think we’re doing.

Perhaps the first thing to say is that we actually have gone to the trouble of thinking about what counts as an appropriate methodology. We are not formally trained in survey methods, and we did not earn our academic stripes by means of formal study in the social sciences. But a) with the humility that distinguishes the professional philosopher from people in other fields, we went into autodidact mode, read a bunch of stuff on survey methods, and decided afterwards that we knew more than enough to go forward. This led to b) our colleagues in sociology and statistics enjoying some hearty laughter at our expense, and then generously agreeing to advise us on how to gather reliable data and how to frame our inquiry. (We hereby thank these kind souls for their uncompensated labor, and gratefully promise to pester them as often as possible going forward.) If you don’t yet feel that you are in reasonably good hands here, we’ll add that c) we all believe in truth and knowledge, and at least one of us believes in epistemology, and has written rather a great deal about it. (Another one of us was tempted to say d) I’m not a statistician, but I play one on the web. You can thank the rest of us for putting an end to that idea.)

As a result of these guided reflections on method, we chose to conduct opinion polls of recognized experts, using one of the standard web-based survey tools. The questions we used can be found here The aim of the anonymous polls – we know who our respondents are, of course, but we don’t know how any individual answered any particular question – was to gather information on two questions: Which departments are likely to provide substantive support for research in specific areas? And: Which departments are likely to provide a welcoming climate for students from underrepresented and traditionally excluded populations? We posed the substantive question to experts in American philosophy, Continental philosophy, Feminist philosophy, and the critical philosophy of race and ethnicity. And we posed the climate question to experienced observers of the paths that people from underrepresented gender, racial, ethnic, and sexual identity populations take through the profession.

In both the climate and the substantive surveys, our aim was to give our respondents an alternative to thinking broadly about some abstract feature called ‘department quality.’ Accordingly, we invited them to evaluate specific statements about the likelihood that individual graduate programs would responsibly oversee graduate study in particular areas. In the substantive surveys, for example, these statements took the following form: “Students will be encouraged to pursue work in [the relevant field],” “Students will have access to a community of scholars who share their interest in [the relevant field],” “Students in this department will have no trouble finding mentors to guide work in [the relevant field],” and “In this department, a student will be able to write a state of the art dissertation in [the relevant field].”

The respondents could agree or disagree with these statements in varying levels of intensity expressed along a five-point Likert scale. (They also had the option of answering “Don’t know enough” so as not to force them into evaluating departments about which they felt insufficiently well-informed.) This enabled us to generate numerical values between 1 and 5 for each department on each statement, and also to generate averages between 1 and 5 for each department across all the questions.

Although these averages created what can only be called a rank ordering, we did not regard this ranking as a list of the top-however-many schools. We did not want to declare someone number 1; rather we sought to identify promising and plausible places for students with interests in specific areas to study. For that reason, what we might have rendered as top-10 lists or top-25 lists have instead become alphabetical lists of schools that we strongly recommend in each area (all of which had overall averages between 4 and 5), and of schools that we recommend, full stop (with overall averages between 3 .5 and 4).

We offer these lists, and we solicited the information used to generate them, in the same spirit in which people in the academy typically make recommendations about graduate schools to their students. The idea in those (we hope) familiar cases is that the advisor understands both the intellectual substance and the professional landscape of the relevant areas of study, and on the strength of this understanding makes recommendations to his or her advisees about where these subfields are being successfully taught and productively explored. Our basic idea was to expand this familiar practice, and get recognized experts and leaders in each of these fields to declare themselves, somewhat systematically, on the same sorts of issues – to indicate, in a way calculated to deepen the impressionistic judgments of overall quality that we all make, which programs they thought presented good options for graduate study in the relevant fields.

The helpful folks whom we consulted describe this approach as a poll of expert opinion, one of the standard methods used by social scientists.  Having said this, we should point out that, as philosophers, we are sometimes inclined to think that social scientists have something approaching consensus about the methods they should use, and about the paradigms for understanding their methods and projects. This is of course not so, as our friends in the field were quick and eager to remind us. In work related to surveys and public opinion research as in any academic field, there is disagreement as to the best methods for asking questions and evaluating answers. We believe it is important to keep this fact in mind: to remember that there is not one best way to gather the kind of data we seek. It is, however, equally important to remember that there are better and worse ways. We believe strongly that we have chosen some of the better ways, and that we have as a result made a responsible attempt at providing an accurate picture of certain areas of philosophical study for prospective students in those areas.