Sarnecka Lab Blog: Writing Workshop: Review the Literature

[Note: Most of the material from these writing workshop blog posts, plus a lot more never-blogged material, is now available in my book, The Writing Workshop: Write More, Write Better, Be Happier in Academia.]

When you start working on a new research topic, you must familiarize yourself with what’s already been done. You do this so you can design your own work to be a useful contribution. Because this is such an important early step, many PhD programs require their students to write a literature review, or to take an exam on their knowledge of the literature. Reviewing the scientific literature on any topic is a lot of work, but the steps are simple: (1) Make your reading list; (2) Read the literature; (3) Write the review.

Step 1. Make Your Reading List

Before you start, decide how you will keep track of the books and articles you’ve read, along with your notes about them. I like the Zotero app, which is free and not only keeps track of your library and notes, but also automatically generates in-text citations and reference lists in the format of your choice. There are other apps like Zotero, or you could just keep a set of paper or electronic files. The important thing is that you can find the readings and notes again later if you want to refer back to them.

The easiest way to build a reading list is to start with a review article or chapter, but an empirical article with a good introduction also works. The readings cited in that article are candidates for your reading list. You don’t have to read all of them, but skim the titles and add them to your reading list if they seem useful or interesting. Then, when you read those articles, do the same thing with their reference lists, and so on. You can also search in the other direction, using Google Scholar to find publications that have cited your key article in the years since it came out. This is especially useful if your key article is an old one.

To take a silly, fictional example, let’s imagine that you’re interested in doing some kind of work with rodents and their psychology or behavior. You might start out by reading a well-cited article reviewing the history of behavioral research across a range of rodent species, and adding many of the papers cited in that article to your reading list. You soon discover that the literature on rats is many times larger than the literature on any other rodent species, so you decide to find some interesting phenomenon in rat behavior and investigate it in another rodent.

Reading about the rats, you come across some studies suggesting that rats’ emotions are systematically related to their body temperature. You think this is really interesting and has big implications for how animal behavior could be affected by changes in global climate. After more reading, you start to think that hamsters would be a good species to study, and your reading list expands to include several sub-lists: (1) Studies of how temperature affects animal behavior; (2) studies of rodent psychology; (3) studies of hamsters. One day in a meeting, your advisor mentions that the lab has a bag of tiny rat-sized mood rings left over from an earlier study, and that you’re welcome to use them if you want. You do a little checking and find out that the rings also fit hamsters, so you tentatively plan to do a study with hamsters and mood rings, measuring something about responses to the environment. Now you have a fourth sub-list: Studies using mood rings to measure something in small animals, preferably rodents.

How do you know when to stop adding readings to the your list? I suggest that you use your term plan to determine how long your list will be. For example, if you have allotted one semester (16 weeks) to review the literature, and you estimate that given your schedule you can review 15 readings per week (3 readings/day, 5 days/week) then your list should be no more than 16x15=240 items long.

This may seem like an odd way to think about a reading list. Of course the goal is not to read a certain number of items, but to gain a good understanding of literature in a given area. But how can you possibly know when you have gained a good understanding? You can’t. All you can do is compose what seems like a good list, start reading and taking notes, and trust that if you’ve missed anything important, you’ll discover it later.

For clarity, this post treats the steps of making a list, reading the literature and writing the review as if they happened in order, one after the other. But in fact, I think it makes sense to build your list as you go, rather than trying to generate the whole list before you start reading. You can use your central article or chapter to identify perhaps 10 or 20 readings-- enough to keep you reading and taking notes for a few days or a week. As you read, you will continue to discover new sources and add them to your list.

If you are a student, it may be useful to check in with your advisor about your reading list as it develops. If you have missed something important, your advisor may be able to tell you so. If you seem to be veering into irrelevant territory, your advisor may be able to redirect you. It’s also perfectly fine to consult with different people about different sublists. For example, if your advisor has expertise in mood rings but not hamsters, you can ask another faculty member--a hamster expert--to take a look at your hamster list.

Step 2. Read the Literature

Although I read all the time for pleasure, I somehow got all the way through my undergraduate years without having to read much for school. (As an undergraduate I studied Russian and Japanese, partly because language classes didn’t require a lot of reading; when I started graduate school, I had never read a scientific paper.) So my idea of reading was to sit down in a comfy chair with a cup of tea, put my feet up, open the work to Page 1, and read at a leisurely pace until I reached the end.

In a PhD program, this way of reading didn’t work at all. I would clear two or three hours in my schedule and try to tackle a stack of articles and book chapters. At the end of that time, I would have read only one article (maybe not even a whole one) and I would not have understood most of that. Week after week, I went to my classes feeling guilty. I was worried that other people would be able to tell that I was unprepared; I was angry at myself for not being able to keep up with the reading; I was angry at the professors for assigning such a ridiculous amount of reading in the first place.

One day I confessed my troubles to a professor in a different department, whom I trusted. (Thank you, Armand Lauffer of the University of Michigan.) He said something like, “You need to learn how to spend about an hour on a book. Study the table of contents, flip through and look at the figures, read maybe the first chapter, maybe the last chapter. In about an hour, you can get a pretty good idea of what the book says, and that’s all you need.”

At the time, I thought what he was describing sounded like cheating. Looking back, I don’t know what I was thinking: It was clearly impossible to read every word of every article in the time I had, but I still thought that not reading every word was wrong, and I felt terribly worried and guilty about it. It took me years to realize that Armand Lauffer was right: Reading the academic literature is nothing like reading for pleasure. If you try to sit down and read all the relevant work in your area the same way you would read Tolkien’s Lord of the Rings, you’ll die of old age--if not boredom-- before you get halfway through.

You need a systematic way of picking the most important points out of an article or chapter without wasting time. Of course, if you genuinely find the reading pleasurable and you want to read every word, and you have the time to do that, don’t let me stop you. But I’m assuming that you just want to get a basic understanding of what’s been done in your field up to now. To avoid confusion let’s give this activity a different verb: Let’s call it surveying.

How to Survey an Empirical Journal Article

If you are new to an area, expect to spend 30-60 minutes surveying an empirical article. (Don’t worry; after a few years, you will be able to do it in under 10 minutes.) There are times when surveying isn’t appropriate; for example, when I review a manuscript or a grant proposal, I do feel obligated to read every word. But for the purposes of a literature review, surveying is just fine. You can always come back later and read the details if necessary.

1. Read the title.

Make note of any words you don’t understand, or any questions you have.

2. Read the abstract.
The abstract is a summary of the whole paper, and it’s worth taking the time to read carefully, several times over if necessary. Again, make note of any questions you have or words you don’t understand.

3. Scroll through the paper and look at the figures.
Figures that illustrate methods or models should give you a good idea of what the researchers did. Figures that illustrate results should show you what the researchers found, and good ones will show you what the researchers’ hypotheses were. Again, make note of any questions you have.

4. Skim through the rest of the paper for the answers to your questions.
The rest of your reading of this article will be guided by the questions you wrote down in Steps 1-3. With experience, you will be able to find the answers to most questions quickly. Here are some examples of common question types, and where to find the answers.

What does [word] mean? Or, what does [abbreviation] stand for? All technical terms and abbreviations should be defined the first time they are used. But sometimes authors break this rule in the title and abstract, where word counts are limited. So look in the introduction for the definition of your mystery word or abbreviation. If you are reading the paper on a screen, you can save time by using the ‘find’ function to search. If the word or abbreviation is not defined in the paper, shame on those authors. You can decide whether to look it up online (but be careful, because scientific terms can be used very differently across different subfields) or just let it go. Personally, I usually let it go. An author who doesn’t bother to define terms isn’t trying very hard to communicate with me, and is not entitled to more of my valuable time and attention.

What question did the authors set out to answer? You can usually find this information in the last paragraph of the introduction. If the information is not there, the authors haven’t organized their introduction properly. Again, you can choose whether to search further or let it go.

What did they measure, and how did they measure it? This information is in the method section, along with information about who the participants were (for experiments with human subjects) and (hopefully) all the other details you would need in order to replicate the study.

What did the authors find? (Not what they think it means.) How did they analyze their data? This information is in the results section.

What do the authors think their findings mean? This information is in the discussion.

5. Write a few sentences to a paragraph of notes.
The final step in surveying an article is to write a paragraph or so for your own notes, summarizing the authors’ central claim and the main evidence for it. Also note any details that are particularly relevant to your work. For example, if the study used a design or an analysis that you might use, make a note of it.

How to Survey a Nonfiction Book

If you are new to an area, expect to spend an hour or two surveying a book. The more relevant it is to your interests, the more closely you will want to read it. If you don't know how relevant it is, assume it's minimally relevant. You can always go back and re-read anything you want in detail. (This list is adapted from Cutler, 2002.)

1. Examine the outside of the book (front and back).
Study the title, the ‘blurbs’ or comments on the covers, and the messages on the end flaps.

2. Note the author’s name and any biographical information about him/her.
Is this person a researcher? A journalist? What else have they published? Are they selling something?

3. Note the copyright date.
You have to know when a book was written in order to put it in context. An author writing twenty years ago was responding to different debates in the field than an author writing today. Also take note of the book’s publication history: The number of editions, revisions and reprints. This may give you a clue to how widely a book has been read, although academic books tend to sell fewer copies and be reprinted less often than other kinds of books.

4. Read the preface, introduction or forward.
This is where the author indicates what they have tried to do in the book.

5. Study the table of contents closely.
This is the outline of the whole book.

6. Flip through the book and look at any figures, photographs, etc.
These pack a lot of information into a small space, and give your eyes a break from reading.

7. If there is an overall summary or general discussion, read it closely.

8. Flip through the end matter-- indexes, bibliography, glossary, appendices, etc.
In particular, look for complete references to cited research. Authors should back up their factual claims with peer-reviewed studies so that you can go and look at the work for yourself. Beware of authors who make factual claims without telling you where they come from.

9. Write a paragraph or two of notes.
The last step in surveying a book is the same as the last step in surveying an article--write a paragraph or two of notes restating the author’s central claims and the main evidence for them.

How I Learned to Survey Literature

When I was a graduate student, despite Prof. Lauffer’s sage advice, I initially didn’t believe that surveying articles and books was good enough. I was convinced that I needed to read every word of every study or I would be a fraud. But I realized that surveying before reading could help me take in the information from an article or book better, so I came up with a three-step plan.

First, I would survey the article or book as described above and write a summary paragraph. Second, I would actually read the whole thing. Third, I would go back to the paragraph I had written and update it based on my new, more complete understanding of the reading. I would correct any errors and add any important details that I had missed.

I followed all three steps for perhaps ten or twenty articles, and noticed something odd: I almost never made any changes to the paragraph I wrote after surveying. I was probably spending 15 or 20 minutes on the first step, and then two or three hours on the second one. (And no time at all on the third one because I hardly ever made any changes to my initial notes.)

Finally it dawned on me that the second and third steps were a waste of my time. The understanding that I had after just surveying the book or article was fine-- the value added by actually reading every word was negligible. That’s when I started surveying the literature instead of reading it.

Step 3. Write Your Literature Review

The term ‘literature review’ or ‘lit review’ can refer to two very different things.

The lit review in an empirical journal article is part of the introduction section. It is a few paragraphs to a few pages long. It contains just the background information that a reader of the article needs to know in order to understand why the authors did what they did and what their findings mean. In particular, the lit review traces the authors’ line of reasoning from some big, broadly interesting question (e.g., How do changes in the environment affect animal psychology?) to the narrow, specific, operational question of the study (e.g., Does shaving hamsters make them angry?)
A review article is a much longer and more comprehensive document: Perhaps ten to fifty pages long, depending on the discipline. It presents an overview of research in a particular area, or on a particular topic. It contains the background information that a researcher needs to know in order to start doing new, original research in the area. Of course no article contains everything there is to know about a topic, but a good review article provides an introduction to the area. Let’s assume that this is the kind of literature review you need to write.

Assemble your reading notes into a rough draft.

If you followed the steps above, you’ll have a few sentences or a paragraph of notes about each article or book that you surveyed. Arrange these in some kind of logical grouping and slap some subheadings on the groups (e.g., Temperature and Rodent Behavior; Animal Emotions; Studies of Hamsters ; Studies Using Mood Rings) and Boom! There’s your rough draft. It will be a bunch of brief and unconnected paragraphs like the following.

Lawsky & Delaney (1964) used mood rings to measure rats’ propensity to cry at sentimental movies. Found no such propensity; concluded that rats are cold-hearted creatures devoid of empathy. Included measures of test-retest reliability for rat mood rings.

Pearl et al. (1968) created sentimental movies starring rat actors; showed that rat audiences were more sympathetic to rat protagonists than to human protagonists, as measured by mood rings and the number of tissues used by weeping rats. Lawsky et al wrote commentary claiming that number-of-tissues measure was confounded, because rats were likely hoarding tissues as nesting material.

Lawsky, Delaney & Herman (1975) measured television-viewing preferences in small mammals using daytime TV. Found that chinchillas, skunks and a subset of gerbils enjoyed talk shows; squirrels and chipmunks preferred game shows (especially Wheel of Fortune and Jeopardy) but became hostile and agitated when their favorite contestants lost. Follow-up study showed that skunks became depressed after watching daytime dramas, although chinchillas did not. Studies did not include hamsters, but game-show results with squirrels and chipmunks did establish that rodent anger can be measured using mood rings.

Grossman, Liljeholm & Chernyak (1991) had guinea pigs wear mood rings for several weeks in an office environment; found that they were happiest on Fridays. Possible confound: Guinea pigs were given extra food on Fridays so that lab assistants could come in late on Saturday mornings. Authors conducted follow-up experiment with extra food given on Tuesdays; found that guinea pigs switched preference to Tuesdays. Overall conclusion: Guinea pigs are happy when they get extra food.

B. Add topic sentences, transitional phrases, and introductory or summarizing statements.

It’s not enough to just paste all your reading notes into a document: You have to say something about them. In addition to the notes about each reading, you must add transitional words, sentences and paragraphs to organize the information into some kind of story. It doesn’t have to be a good story. In fact, it will almost certainly be a boring story. But this step--turning your list of notes into a story--is where a lot of thinking and learning happens, so you can’t skip it.

Your review article doesn’t have to include all the notes you took about every reading. When you figure out where each reading fits into your story, you might decide to include just a sentence or two about some readings, and go into more detail about others. For example, you might revise the reading notes in the rough draft above to produce the following paragraph. (Topic sentences, transitional phrases and introductory/summarizing statements are highlighted.)

Mood rings have been used to reliably measure animals’ emotions for more than 50 years. One of the first studies to use them was by Lawsky & Delaney (1964), who measured rats’ propensity to cry at sentimental movies. They found no such propensity and concluded that rats are cold-hearted creatures devoid of empathy. Their conclusions were later credibly challenged by Pearl et al. (1968), who found that rats did show empathy when the movies featured rat protagonists. However, the Lawsky & Delaney study was important because it included the first measures of test-retest reliability for rat mood rings.

Mood rings have been used successfully with a wide range of small mammals. Lawsky, Delaney & Herman (1975) measured daytime TV preferences in chinchillas, gerbils, squirrels, chipmunks, and skunks. The method was later extended to guinea pigs by Grossman, Liljeholm & Chernyak (1991). Although Lawsky et al. did not include hamsters as subjects, they were able to measure anger in squirrels and chipmunks who watched game shows, thus establishing that rodents’ anger can be measured using mood rings.

Note how the highlighted portions tie the information together and tell the reader why it’s important to the question under discussion in the literature review. For the purposes of your planned ‘angry hamster’ research, it’s not so important what the early studies with mood rings showed. You care about them because they established that mood rings could be used to measure small animals’ emotions-- particularly anger, which is the emotion you plan to measure. And although none of these studies used hamsters, you can argue that mood rings have been used successfully in quite a wide range of other species, so it’s reasonable to try using them with hamsters.