Week 9
Qualitative Literacy

Soci—316

Sakeef M. Karim
Amherst College

SOCIAL RESEARCH

Unit II Begins
March 24^th

A Small Update

Full Page

Qualitative Literacy

Criteria for Good Qualitative Research

Suppose you were given two books, each based entirely on one year of ethnographic observation, and were told that one of them is a sound piece of empirical social science and the other, though interesting and beautifully written, is empirically unsound. What criteria would you use to tell the difference? One can ask the same question of a different kind of qualitative research. Suppose the two books were instead based on in-depth interviews with the same set of respondents, and you were informed that one is empirically sound and the other is not. What criteria would you use?

(Small and Calarco 2022:ix, EMPHASIS ADDED)

Criteria for Good Qualitative Research

We have posed our criteria question to many people who would especially want to have an answer … Indeed, just about everyone who answered our question expressed some uncertainty … That uncertainty partly reflects what one of us has called an absence of “qualitative literacy,” the ability to read, interpret, assess, and evaluate qualitative evidence competently.

(Small and Calarco 2022:x, EMPHASIS ADDED)

Criteria for Good Qualitative Research

One Precondition: Exposure

[T]here is one precondition all good field projects possess. It is a high degree of what we call “exposure.” The core advantage of both ethnographic observation and in-depth interviews is the direct contact each method has with the social world or its people. The greater the contact, the better the data. In interview research, exposure derives from the number of hours spent talking to respondents, and in-depth interviewers generally agree that more hours of interviewing lead to better data. In participant observation, exposure derives directly from the number of hours exposed to the field, and ethnographers generally agree that more time in the field produces better data.

(Small and Calarco 2022:18–19, EMPHASIS ADDED)

Criteria for Good Qualitative Research

One Precondition: Exposure

Survey-influenced interviewers believe that a large sample is a precondition for quality. But that belief is simply one interpretation of the idea of exposure, and not a necessary one. An interview study of 120 people interviewed for 1 hour will have far less exposure than one with 40 people interviewed four times each for 3 hours per interview. The first will have 120 hours of exposure; the second, 480. Interviewers from different perspectives will differ on which of the two approaches is preferable … [b]ut they will certainly agree that, ceteris paribus, 480 hours of data are far more than 120, and that more data tend to yield greater qualitative insight about those who were interviewed.

(Small and Calarco 2022:19, EMPHASIS ADDED)

Criteria for Good Qualitative Research

One Precondition: Exposure

[R]esearchers from an experimental or comparative perspective may believe that studying multiple sites is a precondition for quality in an ethnography. But a study of four sites, each observed 20 hours a week for ten weeks, will have far less exposure than one of a single site observed 20 hours a week for eighteen months. The first will have 800 hours of exposure; the second, 1,560. Researchers will differ on which of the two approaches they prefer. But they will almost certainly agree that 1,560 hours of data are more than 800, and that more hours in a given field site will yield greater insight into what happens there.

(Small and Calarco 2022:19–20, EMPHASIS ADDED)

Criteria for Good Qualitative Research

Five Key Indicators

In Qualitative Literacy, Small and Calarco (2022) discuss five key indicators that can be used to distinguish well-executed from poorly executed approaches
to qualitative data collection:

• Cognitive Empathy

• Heterogeneity

• Palpability

• Follow-Up

• Self-Awareness

Cognitive Empathy

A Working Definition

[E]ffective field-based research almost always shows clear signs of cognitive empathy. Among the most important objectives of qualitative empirical research, cognitive empathy is the degree to which the researcher understands how those interviewed or observed view the world and themselves—from their perspective.

(Small and Calarco 2022:23, EMPHASIS ADDED)

A Working Definition

And An Example

A researcher can report someone’s belief that abortion constitutes killing an already-started life without understanding how that person arrived at their particular belief, why other beliefs about abortion do not compel the person as this one does, what significance that belief carries, and more. As the example suggests, cognitive empathy, the ability to understand what another understands, is a matter of degree. The better a researcher empathizes, the more accurately the scholar can describe a person’s views; reveal the meaning of those views to the person who holds them; account for their origins; explain how coherent, reflective, or consistent they are; anticipate the person’s views about other matters; and convince the person that they have been understood.

(Small and Calarco 2022:23–24, EMPHASIS ADDED)

Modes of Cognitive Empathy

The particular kind of “understanding” that Dilthey, Weber, and others refer to amounts to what we have called cognitive empathy … [A] qualitative researcher may, at minimum, attain this kind of understanding about three things: perception, meaning, and motivation. The first, the foundation of any study in which cognitive empathy matters, is merely how others perceive themselves and the world around them. Meaning is the significance people assign to what they see, think, say, or do … Motivations are thornier … [M]otivated actions are those an individual takes with a particular goal or outcome in mind.

(Small and Calarco 2022:23–24, EMPHASIS ADDED)

Examples and Group Conversations

Toggling Cognitive Empathy

In groups of 2-3, use pp. 29-46 in Small and Calarco (2022) to adjust the baseline sentences presented in the next two slides—with an eye to maximizing cognitive empathy.

Toggling Cognitive Empathy

In In-Depth Interviews

Adjust this excerpt to maximize cognitive empathy.

Seventeen-year-old Maria lived in a predominantly black, high-poverty neighborhood in Philadelphia. She explained, "I don't like my neighborhood, and I can't wait to get out. My school counselor says that my grades are good enough to get me into Penn State, but I have to do well on the SAT to get a scholarship. I can't wait."

Toggling Cognitive Empathy

In Participant Observation

Adjust this excerpt to maximize cognitive empathy.

She stepped off the bus. Heading north, she tiptoed around barbeque leftovers on the sidewalk—foil, charcoal, a few paper plates. As she crossed the street, she saw two men in their early twenties spraying graffiti on the brick wall of a corner townhouse.

Heterogeneity

A Working Definition

Exposure … increases the researcher’s overall sensitivity to subtleties in speech and action that would have been missed at the start of the study. In the finished narrative, one can detect this sensitivity when depictions exhibit high heterogeneity, which we define as the degree to which the perceptions, experiences, motivations, and other aspects of the population or context studied are represented as diverse … Effective qualitative works tend to depict people and places that are far more diverse than a newcomer to those studied might have expected—even if the study involves people from a single field site or demographic group.

(Small and Calarco 2022:47, EMPHASIS ADDED)

A Working Definition

An Elaboration

The more exposure a researcher has to the world of those studied, the more difference the researcher detects. The more hours over multiple sessions the interviewer spends with the respondent, the more the former can detect the contradictions, mood changes, shifts in opinions, and variations in perceptions in the latter. The more time an ethnographer spends at a field site, whether a large corporation or low-income neighborhood or high-performing school, the more differences the researcher detects in its members, its characteristics, and the fluctuations in both over time.

(Small and Calarco 2022:48–49, EMPHASIS ADDED)

A Working Definition

An Elaboration

It is important to note that depicting heterogeneity … is not synonymous with what social scientists sometimes mean by “examining variation.” Heterogeneity is an indicator of good work because … such work depicts the degree of diversity in people or contexts that insiders—members of a particular group or community—know there to be.

(Small and Calarco 2022:49, EMPHASIS ADDED)

A Working Definition

An Elaboration

Some social scientists use the term “variation” to describe a feature of research design … believing that studies must be set up to include not one but multiple kinds of people, groups, study sites, or contexts … We do not believe that this kind of variation (or the inclusion of explicit comparison groups) is necessary for a study to be empirically sound … [M]any outstanding ethnographies are based on research in a single neighborhood or organization. Moreover, a study of a single site or group can still exhibit high heterogeneity. We use “heterogeneity” to describe not a feature of the research design but the character of the written report … [W]hen qualitative research, whether interview based or observational, is convincing, it almost always depicts heterogeneity among people or their contexts in one way or another.

(Small and Calarco 2022:49, EMPHASIS ADDED)

Examples and Group Conversations II

Depicting Heterogeneity

In groups of 2-3, use pp. 52-79 in Small and Calarco (2022) to adjust some of the sentences presented in the next two slides—with an eye to
depicting heterogeneity.

Depicting Heterogeneity

In In-Depth Interviews

• The Individual
• The Group

Adjust this excerpt to depict heterogeneity of understanding, experience, and motivations.

Sofia, who moved to New York without papers when she was nineteen, was embedded in a support network that provided information about jobs, help during emergencies, and emotional aid. She explained, in Spanish, "When I need something, I know I can turn to other people from my hometown. I got my cleaning job from my cousin, who told her boss about me."

Adjust this excerpt to depict heterogeneity as an exception to a group pattern
or characteristic of the group.

Mexican immigrants in New York are embedded in a network of support that provides information about jobs, help during emergencies, and emotional support. Sofia explained, in Spanish, "When I need something, I know I can turn to to other people from my hometown." Patricio said something similar: "I was lucky to have friends that helped me when I lost my apartment." Miguel said, "I made it because I have a network that supports me."

Depicting Heterogeneity

In Participant Observation

• The Individual
• The Group
• The Place

Adjust this excerpt to depict heterogeneity in a given experience or the heterogeneity of experiences.

Please scroll down inside the box to read the full excerpt

Sofia was embedded in a support network that provided information about jobs, help during emergencies, and emotional aid. At one of the community center's monthly meetings, Sofia was a few minutes late. In the meeting room, about a dozen women were seated around a table talking in pairs or groups of threes. She walked around the table, greeting everyone. She thanked one of them, her cousin Josefa, for the tip about the job. She thanked Yanina for the movie tickets, but laughed that her daughter slept her way through the film. The entire process took several minutes, after which she sat down, starting the meeting a few minutes late.

Adjust this excerpt to depict heterogeneity in a given experience or the heterogeneity of experiences.

The women in the community center had formed a network of support that provided information about jobs, help during emergencies, and emotional support. Sofia had found her job through her cousin, who also participated in the meetings. Yanina had shared with others movie tickets she had won in a raffle. Claudia had baby-sat for Sofia, when the latter's daughter was too sick for day care.

Adjust this excerpt to depict heterogeneity in a given experience or the heterogeneity of experiences.

Seventeen-year-old Maria lived in a predominantly black, high-poverty neighborhood in Philadelphia. She explained, "I don't like my neighborhood, and I can't wait to get out. My school counselor says that my grades are good enough to get me into Penn State, but I have to do well on the SAT to get a scholarship. I can't wait"

A Final Exercise

Create a Stylized Example

Practice Qualitative Literacy

Create Stylized Passage Related to Your Project

Please enter some stylized text first.

Modify Your Text

Harmonized with previous tab.

(Carr et al. 2020)

More Qualitative Literacy
March 26^th

A Reminder

The Next Two Weeks

Full Page

Palpability

A Working Definition

Our third indicator, palpability, can be detected in as little as a single sentence, because it is a feature of every piece of evidence reported. Palpability, as we use the term, is the extent to which the reported findings are presented concretely rather than abstractly … [E]ven concise discussions that aim to neither demonstrate empathy nor reveal heterogeneity can differ dramatically in quality, by virtue of how concrete the evidence reported is.

(Small and Calarco 2022:80, EMPHASIS ADDED)

A Working Definition

An Elaboration

To understand concreteness, consider its opposite, abstraction. For a researcher to identify any pattern in the data, they must perform an abstraction: they must decide that two direct observations—two statements uttered by interviewees, two actions noticed in the field, and so forth—are versions of a general phenomenon, even an only slightly more general phenomenon “For example, a researcher interviewing mothers must decide that when one interviewee says,”My husband and I argued” and another says “Jeff was being annoying,” they are both expressing the general phenomenon, spousal conflict, rather than, say, one expressing that the couple argued and the other that the spouse was joking. Over the course of a study, small abstractions of this kind become larger and more encompassing abstractions that result in a scientific contribution” (Small and Calarco 2022:81). … Abstraction of one or another form is the foundation of social science, the feature that converts a single observation into a principle, theory, or hypothesis. But abstractions are not, in fact, data, and a study of any kind that reports only abstractions …. is presenting an interpretation of the data, not the data themselves.

(Small and Calarco 2022:81, EMPHASIS ADDED)

A Working Definition

An Elaboration

[O]nly quotations and field reports have a chance of constituting palpable evidence. But quotations and field notes are not necessarily enough; closeness matters, too. By closeness we refer to proximity to the phenomenon of interest … Interviewers can elicit quotations that are themselves abstractions or generalizations of the phenomenon; ethnographers can fail to either observe or record the phenomenon of interest closely. In either case, a clue that the researcher failed to get close is their reliance on generalizations in the quotations or field notes … Ultimately, only reports of distinct persons, statements, perceptions, meanings, motivations, events, actions, responses, and places have a chance of constituting palpable evidence.

(Small and Calarco 2022:81–82, EMPHASIS ADDED)

Palpable Statements

An Example

Baseline "Mothers reported being more concerned about their children's safety than their spouses are."

Is this solution more palpable?

×

Examples and Group Conversations III

Getting More Palpable

Sort yourself into groups of 2-3. Some groups will review Palpability in In-Depth Interviewing (Small and Calarco 2022:84–89), while others will review Palpability in Participant Observation (Small and Calarco 2022:91–96). In both cases,
discuss why the (relevant) passage below needs to be tweaked.

Interviewer

How much do you worry about the safety of your child?

Gina

A lot. I'm her mother. It's my job to. Especially because she's not old enough yet to know what's safe and what isn't. So, I have to keep an eye on her all the time.

Interviewer

And what about your partner? How important is safety to him?

Gina

[Laughs] I mean, of course he thinks her safety is important. But we have some ... [pauses] different ideas about what's actually safe.

Interviewer

Who do you think is more concerned about your child's safety—you or your partner?

Gina

Me, definitely. I tend to worry a lot more than he does.

The couples I observed disagreed at least occasionally about children's safety, and when they did, mothers displayed more concern than fathers did. For example, Isabel and Akio often argued about issues such as car seats, with Isabel taking the more careful stance.

↓ Scroll to Read More

Follow-Up & Self-Awareness

A Working Definition

Follow-Up

We define follow-up as the extent to which the researcher collected data to answer questions that arose during the data-collection process itself … [F]ollowing up is not just a feature of interviews; it plays a role in ethnographies as well, and it in fact can describe an orientation to the entire data-collection process, representing a foundation of true scientific discovery.

(Small and Calarco 2022:99, EMPHASIS ADDED)

A Working Definition

Self-Awareness

[F]ield-workers do not merely collect but in fact produce the data they later analyze; they co-create, in an important sense, the transcripts and field notes … But field-workers matter to the data collected in an even more fundamental sense: who they are—their gender, race, height and weight, level of extraversion, grace or awkwardness, hairstyle, clothing, vocabulary and intonation, class background, religion, fundamental beliefs, sympathy or disdain for others, and ignorance, and every other aspect of their identity—affects how easily they reach others, what others do and say in front of them, and how they interpret what they have heard or observed … We define self-awareness, narrowly, as the extent to which the researcher understands the impact of who they are on those interviewed and observed—and thus on the collected data.

(Small and Calarco 2022:119, EMPHASIS ADDED)

A Final Group Exercise

Follow-Up and Self-Awareness

Form groups of 2-3. Some groups will review the chapter on Follow-Up and discuss its implementation in interviews/ethnographic work. Others will review the chapter on Self-Awareness and discuss how it is achieved in interview- or ethnographically-oriented research. In both cases, Qualitative Literacy (Small and Calarco 2022) is your north star.

One Last Exercise?

Modify Your Stylized Example

Practice Qualitative Literacy

Create Stylized Passage Related to Your Project

Please enter some stylized text first.

Modify Your Text

Harmonized with baseline.

Inherited from previous step.

Inherited from previous step.

Inherited from previous step.

Inherited from previous step.

```{=html}

Practice Qualitative Literacy

Create Stylized Passage Related to Your Project

Please enter some stylized text first.

Modify Your Text

Harmonized with baseline.

Inherited from previous step.

Inherited from previous step.

Inherited from previous step.

Inherited from previous step.

Enjoy the Weekend

References

Carr, Deborah S., Elizabeth Heger Boyle, Benjamin Cornwell, Shelley J. Correll, Robert Crosnoe, et al. 2020. The Art and Science of Social Research. Second Edition. New York: W. W. Norton & Company, Inc.

Small, Mario Luis, and Jessica McCrory Calarco. 2022. Qualitative Literacy: A Guide to Evaluating Ethnographic and Interview Research. University of California Press.