Wednesday, 1 March 2017

A systematic literature review approach

Lavellée, Robillard and Mirsalari wrote a paper in 2014 which aimed to explore, then assist computing students in preparing a systematic academic literature review.

The authors found that there were two sets of expertise required: that of gathering the information to ensure completeness, and creating a literature review that could be repeated by a different person later to yield largely the same information. They wanted to develop a process which became complete and replicable - providing the methodology was consistent (Lavallée et al., 2014).

They developed a process which they have named iterative systematic review (iSR), and have chunked down the approach to the literature review into eight tasks (Lavallée et al., 2014, p. 175-6):
  1. "Review planning: Plan the review effort and training activities." This should include drawing up a plan, a tutorial in how to do this, communicating what sets a quality paper apart, how to read statistics and results, and an introduction to research biases.
  2. "Question formulation: Define the research questions." The authors suggest a very generic question to begin. For example, “What has already been written on subject X?”, then narrowing down to the research question. Read the article here and here.
  3. "Search strategy: Define the review scope and search strings." This section includes how to prepare search key words and Boolean strings for databases, and how to adjust these as material is, or is not, found.
  4. "Selection process: Define inclusion and exclusion criteria." In this task, researchers use article titles, abstracts, conclusions and keywords to determine utility.
  5. "Strength of the evidence: Define what makes a high quality paper." This section utilised a checklist - sadly not provided - which students could use to work out the significance of a paper. However, students could gain a reasonable idea from citations and journal rankings (albeit rough).
  6. "Analysis: Extract the evidence from the selected papers." Students need to be able to extract relevant information here, which is challenging. Careful reading and following of citations will help us here: as will carefully noting context. The authors suggest tables to consolidate evidence, but also remind us that the use of quantitative evidence to evaluate qualitative data is not necessarily useful.
  7. "Synthesis: Structure the evidence in order to draw conclusions." Lavallée et al comment on the poor quality of student synthesis in this section. This is something that needs to be paid careful attention to. By first doing a précis, students start to see what is important in the article, which will aid their later write up (see more here). 
  8. "Process monitoring: Ensure the process is repeatable and complete." Students will learn over time what is 'enough', and what needs repair, when they find that they are unable to transition to the next stage.

While the authors did manage to gain some level of repeatability, they have not yet managed to confirm completeness (Lavallée et al., 2014).

You can download the article by Lavallée et al here.


  • Lavallée, Mathieu, Robillard, Pierre-N. & Mirsalari, Reza (2014). Performing systematic literature reviews with novices: An iterative approach. Education, IEEE Transactions on, August 2014, Volume 57, issue 3 (pp. 175-181).


  1. Replies
    1. This comment has been removed by the author.

    2. Thanks David! Glad you liked it - the author's of the article did a great job, didn't they! :-D


Thanks for your feedback. The elves will post it shortly.