Knowledge Organisers for learning: Examples, non-examples and concept maps in university mathematics

1.   Introduction

Concept mapping is becoming an increasingly popular educational tool in various educational settings ^[1,2]. Based on the principles of Novak's ^[3] theory of meaningful learning and assimilation, a concept map is a graphical representation of knowledge, depicting linear hierarchical relationships and cross-relations between related concepts. It is assumed that concept mapping promotes effective meaning-making by prompting learners to logically synthesise and organise information in a hierarchy of concepts underpinned by the identification of a unifying logical structure ^{[1,2,3,4,5,6]}. At the higher levels of the conceptual hierarchy, the most general concepts are used, from which arrows emanate, identifying increasingly specific concepts. Cross-relations connect concepts located in different branches of a concept map ^[3]. Identifying and representing the relationships between concepts prompts a learner to engage in the analysis of knowledge, the application of inductive and deductive thinking, and the evaluation of understanding. A recent meta-analysis, published in November 2022, concluded that the concept mapping method is more effective for the improvement of students' analyticity (as a cognitive disposition) and for all motivational critical thinking dispositions (open-mindedness, truth-seeking, inquisitiveness) than traditional teaching methods ^[2].

Pertaining to concrete learning outcomes, another meta-analysis of 142 studies concluded that learning with concept maps produces a moderate, statistically significant effect (g = .58, p < .001) compared to other common forms of learning ^[1]. Furthermore, moderator analysis showed that learners gained greater benefits when creating concept maps (g = .72, p < .001) when compared to simply studying concept maps (g = .43, p < .001). Of the 142 studies included in the meta-analysis, 118 were concerned with STEM subjects. A large majority were grounded in learning natural sciences: biology, physics, and chemistry. At the tertiary level, three studies investigated concept mapping as a learning tool in statistics courses ^[7,8,9]. However, none of the studies included in the meta-analysis considered concept mapping in the tertiary mathematics context.

A small number of qualitative case studies are reported in the mathematics education literature, illustrating possible implementations, mostly at a school level, and alluding to the potential viability of concept mapping in mathematics classrooms ^{[10,11,12,13,14,15]}. There are some isolated examples of the use of concept mapping in teacher training in the UK, USA and Australia ^{[15,16,17,18]}, albeit such use may not be fully reported in research journals. And some work has been done to describe the training needed to enable students to construct concept maps in a study of Grade-8 Chinese classrooms ^[19] and ascertain the nature of the conceptual understanding held by this group of learners ^[20,21].

Thus, it appears that there is a dearth of research that explores the specifics of concept mapping in the context of mathematics education at the tertiary level, with one exception. In our recent paper, we investigated the use of concept mapping as a weekly task completed by learners in a large service mathematics course at the University of (N = 219) ^[22]. We focused on examining the relationships between the quality of student concept mapping and two major outcome variables: overall course achievement and one of the most fundamental psychological constructs – self-efficacy (which was measured using a validated instrument, MASE ^[23]). Using hierarchical multiple regression, we showed that concept mapping performance explains a statistically significant amount of variance in both the final exam scores and the Emotional Regulation factor of assessment self-efficacy after accounting for other conventional coursework assessments. Second, the association with the emotional regulation efficacy measure suggests that concept mapping as a learning activity that involves more positivity about the ability to succeed in facing challenges than a typical assignment. This indicates the potential for more perseverance and effortful learning while actively engaged in meaning-making and the type of relational reasoning prompted by concept mapping ^[22].

In this article, using methods analysis, we report an explorative case study to complement our previous study with three aims. The first is to ascertain student perceptions on embedding concept mapping as a regular assessment/learning tool as part of a weekly Knowledge Organiser. Secondly, we examine the relationship between the implementation indicators, such as time-on-task, their marks, and student perceptions of the utility of Knowledge Organisers. And lastly, we use the data to inform a set of guidelines for design, development, assessment and implementation that could be used by other practitioners. We next present a brief overview of the research literature from educational psychology and mathematics education as a theoretical justification for using the Knowledge Organisers' constituent components: examples of a given concept, non-examples and a concept map.

2.   Theoretical and empirical foundation

2.1.   What is learning?

When discussing teaching and learning, it is beneficial to consider the latest research on human cognition. In educational psychology, a commonly utilised model to describe the way in which human cognitive structure and functions are organised is presented as "Human Cognitive Architecture" ^[24]. The central idea is the interaction between three components: sensory memory, working memory and long-term memory, which is depicted diagrammatically in Figure 1 (reprinted from ^[25]). Working memory represents a central structure processing information that is coming from (1) external sources (such as visual and audio information), through a temporary storage facility of sensory memory and (2) the information stored in the long-term memory.

One of the main functions of working memory is processing and encoding the information for long-term memory storage. Long-term memory is conceptualised as "the store holding all knowledge acquired during the process of learning" ^[24]. When receiving new, unfamiliar information, working memory is known to have limitations in both the amount of information it can accommodate and the period during which the information is stored ^[24,26]. However, experimental studies showed that the limitations of working memory were no longer a barrier when it came to receiving information that had been stored in long-term memory. Therefore, according to the cognitive load theory, the ultimate goal of learning is acquiring and preserving information in long-term memory ^[24,26].

2.2.   How is knowledge stored?

A cognitive structure unit in which knowledge is organised and stored in long-term memory is called a schema ^[27,28]. It allows for numerous relevant elements of knowledge to be organised into a single body of information (illustrated in Figure 1). Hence, in the process of learning, we incorporate more complicated elements with the primary elements of relevant information that pre-exist in our long-term memory.

Another major role of schemas is to diminish the burden of working memory. As mentioned previously, there are limitations on how many elements can be taken in concurrently. However, Sweller et al. ^[26] argue that there are no limitations to "the size, complexity and sophistication of elements" in a schema. In fact, schemas that have been developed over a long duration can potentially accumulate information with a large capacity ^[27]. Therefore, research asserts that the formation of schemas in long-term memory and their utility in the working memory encapsulates the main mechanism of learning.

2.3.   Examples, non-examples and concept maps

We next provide a brief overview of the literature concerning the use of examples and non-examples in mathematics and more general research pertaining to concept mapping.

2.3.1.   Examples in mathematics education

Examples have been widely accepted as one of the main educational tools in mathematics, based on early research establishing their effectiveness ^[29]. They are commonly used in mathematics education, and they are considered to play an important role in fostering an understanding of mathematical concepts and proofs ^{[30,31,32,33]}. An example is defined by researchers in various ways. It is the "illustrations of concepts and principles" ^[34], "any mathematical object from which it is expected to generalise" ^[30] as well as "a mathematical object satisfying the definition of some concept" ^[35]. In tertiary-level mathematics, examples are often provided by a teacher for the students as part of instructional explanations ^[36]. However, constructing examples is also a beneficial learning activity. Research has alluded to various benefits as an outcome of having students construct their own examples. This has been well documented at a wide range of educational levels in the mathematics education literature ^[32,34,36].

Key notions considered by researchers are a concept image and an example space. A concept image is a term first used by Tall and Vinner ^[37] to denote the whole cognitive structure that is associated with a particular concept. This conceptualisation is somewhat similar to what educational psychologists describe as schema ^[27,28], with a marked resemblance in their describing characteristics. However, the term 'schema' is reserved for use in a general, non-discipline-specific sense.

An example space is a certain subset within the concept image in which the individual's prior knowledge of examples and the methods of creating those examples is combined ^[31]. Goldenberg and Mason ^[38] view example spaces as inescapable components of the learners' experience. That is, learning more about a concept includes gaining access to further examples, "as well as enriching the interconnections and extending the triggers and resonances affording access to those spaces" (p. 190). Teaching effectively includes making use of tasks and interactions through which learners gain access to examples, to construction methods, and of course to mathematically relevant features of different examples.

Marton's ^[39] theory of variation is often used as a foundation for explaining how learners can learn concepts from examples. People have a natural disposition for detecting variation in objects within close temporal and physical proximity. As such, it is believed that varying not too many factors at once (nor too few) prompts learners' natural sense-making processes to be activated, which are conducive to discerning generality. The generalising is enabled by noticing dimensions of (possible) variation and ranges of permissible change, thereby observing generality through the particular ^[38].

2.3.2.   Non-examples and how they are used

Some considerations have been given to the use of non-examples as pedagogical tools in the literature. Zaslavsky and Shir ^[40] defined non-examples as "a statement that is not equivalent to its commonly accepted definition" and emphasised their importance in mathematics learning. They posited that the benefits are derived because to be able to distinguish statements that are not equivalent to the actual definition, students must engage in substantial logical considerations. In early research, Henderson ^[41] simply referred to a non-example as "[a]n object which is not a member of the referent set" and focused on pedagogical observations that non-examples are useful to demonstrate a particular condition often missed by students needed to satisfy the definition of a concept.

Unlike examples, which are frequently used as a tool to support students' learning, non-examples do not commonly appear in mathematics education ^[40]. However, as the definitions of non-examples imply, the benefit of incorporating non-examples into mathematics education is that they "serve to clarify boundaries" ^[42], making it a crucial part of forming an understanding of a concept ^[43]. Especially, constructing non-examples of a concept along with its examples is highly effective when each non-example has only a single feature that is illustratively not satisfied by definition ^[29].

More recently, the use and importance of non-examples have been demonstrated through the research undertaken by Fukawa-Connelly and Newton ^[30]. They have manipulated the model of example space, which was developed by Mason and Watson ^[31], and broadened its usage to the whole set of examples that students may encounter, including non-examples of the concept of interest, thereby revealing the potential of this method for effective learning.

During the design stage of our project, we took the research on examples and non-examples into account by prompting students to construct examples/non-examples of a selected concept as a preliminary step before asking students to construct a concept map. It is important to note the principal difference between constructing an example space and a concept map. Constructing an example space of a concept primarily exploits features of the concept by listing and contrasting variations of examples (and non-examples). Whereas constructing a concept map of the selected concept may involve an explication of its (albeit limited) example space, the main focus is on the identification of a group of related (different) concepts and connections between them. As such, the main difference can be conceived as the difference between a local conceptual understanding (example space) versus a more global conceptual understanding (concept map) based on the explicated relations between numerous concepts. We elaborate on what is meant by concept mapping next.

2.3.3.   Concept maps: Overview

Concept maps were first used as a learning tool at Cornell University as part of a research project led by Joseph Novak in 1972 ^[6,44]. The main purpose of a concept map is to produce a visual representation of organised information about a chosen concept ^[45]. The concepts are often presented in enclosed shapes (rectangles or ovals), and directed lines are used to connect any two related concepts. The most general concept that embraces all other concepts is presented at the top (or sometimes centre), with more specific concepts cascading out of the main one, thus illuminating a hierarchy of the presented concepts ^[6]. "Linking words or linking phrases" ^[6] are short descriptions on the directed lines that specify how the connected concepts are related. Some examples of such words/phrases include "is an example of", "generalises to" and "contains".

Novak ^[6], the pioneer of concept mapping, described its key features as follows:

● Hierarchy: The concepts are presented in a manner that reveals the hierarchical structure from general concepts to specific concepts in descending order. It should be noted that the hierarchical relationship between concepts may vary depending on the context. Deciding on a "focus question" ^[6], which is a specific question that the concept mapper seeks to answer, can guide in deciding on the hierarchical relationship between concepts.

● Cross-links: The connections used to show the relationship between concepts that have been derived under different strands developed from the main concept are called cross-links. The process of identifying cross-links is highly dependent on the existing knowledge of the concept mapper.

● Examples: Incorporating specific examples into a concept map enhances clarity in understanding the concept that may be abstract when presented on its own.

2.3.4.   Concept mapping as an effective learning strategy

In general, concept mapping is assumed to make learning efficient, with various reasons provided in the literature. Schroeder et al. ^[1] posited that the reasons why constructing and studying concept maps might be beneficial can be broadly categorised into three types: concept maps promote meaningful learning, concept maps reduce the extraneous load or both.

Concept mapping promotes meaningful learning because it requires learners to engage deeply with the material by focusing on the organisational structure of a set of related concepts and producing elaborative connections among them ^[45,46]. Meaningful learning is thought to occur when new knowledge is created and assimilated into the existing schemas, in accordance with Human Cognitive Architecture and cognitive load theory, which were discussed earlier ^[26]. The term 'knowledge elaboration' is often used in reference to meaningful learning, emphasising the importance of using prior knowledge to expand and refine new insights utilising processes such as organising, restructuring, interconnecting, integrating new elements of information, and identifying relations between them ^[47]. Research has shown that knowledge elaboration is the key mechanism behind the success of well-known learning strategies such as self-explanations ^[48] and elaborative interrogation ^[49]. That is, the knowledge elaboration strategies and the processes involved in a successful concept mapping activity are manifestly similar. According to Karpicke and Blunt ^[50], "concept mapping bears the defining characteristics of an elaborative study method: It requires students to enrich the material they are studying and encode meaningful relationships among concepts within an organised knowledge structure" (p. 772).

However, the findings are not clear cut to conclude that concept mapping is the most effective learning strategy. For example, in an attempt to compare the effects of retrieval practice and concept mapping, O'Day and Karpicke ^[51] conducted two randomised controlled experiments. Undergraduate students were randomly assigned to various study groups and were given a biology text to study. Surprisingly, the results of both studies demonstrated that the combination of concept mapping and retrieval practice was no more beneficial than retrieval practice alone, calling the proclaimed learning benefits of concept mapping into question. This is in line with the older result by Karpicke and Blunt ^[50], which showed that practising retrieval produces greater gains in meaningful learning than elaborative studying with concept mapping. However, these studies were not undertaken in mathematics learning contexts; thus, the generalisability of their conclusions is limited.

2.3.5.   Concept mapping in mathematics

In mathematics education settings and more broadly, it is well understood that learning is not just an accumulation of new snippets of information in long-term memory, as in a computer ^[52]. Rather, learning is influenced by two major factors: (1) what is presented and in what format and (2) the extent of the cognitive processing that the learner is actively engaged in during learning. Thus, learning is viewed as a generative activity. This conception of learning is a theoretical advance that emerged from the unification of ideas that came out from the cognitive revolution with constructivist ideas about the importance of meaning-making while learning. This well-evidenced theory predicts effective learning to consist of three stages: (1) a learner actively selects the relevant aspects of incoming information by paying attention (via sensory memory), (2) which is followed by organising this information into a coherent cognitive structure in working memory, and (3) finally, integrating this cognitive structure with relevant prior knowledge activated from long-term memory ^[52].

In parallel, mathematics education research extended these ideas to distinguish key characteristics of different types of mathematical understanding and knowledge. These ideas can be traced back nearly 50 years ago to Skemp's ^[53] distinction between 'relational understanding' and 'instrumental understanding' of mathematics. The latter describes a limiting yet commonly occurring understanding based on knowing a set of rules without understanding the reasons, whereas "learning relational mathematics consists of building up a conceptual structure (schema)" ^[53]. Subsequent research divided mathematics knowledge into two types: procedural knowledge and conceptual knowledge ^[54], with the latter defined as "knowledge that is rich in relationships. It can be thought of as a connected web of knowledge, a network in which the linking relationships are as prominent as the discrete pieces of information" (pp. 3-4). This dichotomy was subsequently questioned and reconceptualised by Star ^[55] to include the dimension accounting for quality.

The current mathematics cognition perspective asserts that conceptual understanding is achieved when a sufficiently well-organised schema has been encoded into long-term memory ^[56]. From this perspective, concept mapping is conducive to developing conceptual understanding and, thus, effective learning because it fosters meaningful learning by requiring learners to engage deeply with the material and focus on the logical structure underlying and unifying a set of related concepts. Arguably, this is a very different process from retrieval practice.

Generally, research on assessment draws a clear distinction between summative assessments, used to mark academic progress after a set unit of material (i.e., assessment of learning) and formative assessments, used to track student progress during the learning process for the provision of feedback (i.e., assessment for learning) ^[57]. However, this distinction has been blurred in recent tertiary mathematics studies. For example, contradicting the assumption that formative and summative assessment approaches are incompatible, Buchholtz et al. ^[58] showed how these assessment types could be combined in university mathematics teacher education. Indeed, the central consideration for mathematics education at a tertiary level has been about what type of reasoning is elicited and assessed by various tasks. In other words, in parallel with the distinction between procedural and conceptual knowledge, an important question is how to design a valid and reliable method to assess different types of understanding (such as instrumental versus relational in the sense of Skemp ^[53].

The research debates in mathematics education concerning the assessment of procedural knowledge have been more or less settled. In contrast, much controversy is centred around the supposedly designed tasks to assess conceptual knowledge, which do not always align with theoretical claims about mathematical understanding ^[59]. However, one of the biggest issues discussed in undergraduate mathematics is that most questions on exams and coursework are 'imitative'; that is, questions that can be solved by performing prescribed algorithms and recalling analogous (if not identical) solutions ^[60,61,62].

Against this theoretical and empirical backdrop, our latest finding mentioned in the introduction is not unexpected ^[22]. We found that concept mapping, used as an assessment, can detect learners' mathematical abilities that are not discerned by conventional assessments and, thus, have the potential to assess conceptual understanding. Moreover, there is evidence in support of the use of concept mapping as an effective learning tool linking performance to improved affective outcomes (such as self-efficacy). Given this promising result, further investigations on incorporating concept mapping into practice are warranted. As such, in this study, we investigate the feasibility of incorporating concept mapping in university mathematics as both an assessment tool and a learning strategy to foster relational reasoning.

2.4.   Research questions

Given the ample evidence and theoretical foundations pertaining to the benefits of using examples, non-examples and concept mapping for learning, we designed, developed, implemented, and evaluated an innovation in a natural classroom setting of a university mathematics course. First, a lecturer provided an explanation of what a concept map is and demonstrated a few examples during the first class of a university semester. Students were notified that a new type of assessment, called a Knowledge Organiser, would be used weekly during the semester. Each week a key concept was selected, and students were required to fill in a Knowledge Organiser, which consisted of three prompts: (1) state its definition, (2) provide examples and non-examples, and (3) create a concept map of the concept that includes other related concepts and connections.

The present exploratory case study aimed to ascertain the Knowledge Organisers' implementation feasibility and evaluate it from a learner's perspective. To that end, a survey was conducted at the end of the semester to collect data for answering the following research questions:

RQ1. What are students' general perceptions of the Knowledge Organisers?

RQ2. Is there a relationship between students' perceptions of the concept mapping tasks and their performance on the tasks?

RQ3. How much time do students spend on the tasks and is there a relation between the time students spent on the concept mapping tasks and their performance?

RQ4. What pedagogical strategies can be formulated to guide future implementations of concept mapping at the undergraduate level?

3.   Methods

3.1.   Research site

The study was conducted at a large research-intensive university (Auckland, New Zealand) in an undergraduate mathematics course covering Calculus Ⅱ, Linear Algebra Ⅱ, and Introduction to Ordinary Differential Equations, serving the needs of students majoring in a variety of disciplines such as computer science, finance, economics and other sciences. Unfortunately, for a large proportion of non-mathematics majors taking this course, a lack of interest in mathematics contributes to low intrinsic motivation coupled with suboptimal engagement with the course. An additional challenge is the size of the course: the enrolment numbers range from 350 to 550 students per semester. The course is delivered over 12 teaching weeks with the following weekly structure: three 1-hour lectures and one 1-hour tutorial (25 to 30 students per room working on problems). The lectures are purposefully designed not to be transmission-style lecturer's monologues and include active learning activities such as Think-Pair-Share, quizzing etc.

This study was undertaken in Semester 1, 2021 (March—June), when, internationally, many places were influenced by the COVID-19 pandemic. However, due to the elimination strategy with closed borders, most educational institutions have been running as normal since late 2020 in New Zealand, with a few exceptions. As a result, most courses were delivered face-to-face except for the first two weeks of the semester.

3.2.   Participants and setting

In the trial semester, 355 students were enrolled in the course, with 35 students studying overseas, thus completing the course online. All students were invited to participate in the study, and 323 provided their consent to the use of their data for research purposes. An important aspect of the course was weekly tutorials (practical sessions) where students worked on assigned mathematical problems in small groups. Attending a weekly tutorial for ten weeks was compulsory for all students enrolled, with participation marks awarded. Additionally, students were expected to submit written solutions to short assignments, referred to as "marked problems", which were also assigned weekly.

3.3.   Innovation: Knowledge Organisers

In semester one, 2021, Knowledge Organisers were assigned as part of the question set in each tutorial (1 question out of 4) and twice as the 'marked problem' for the week, for which students' work was marked. The template for a Knowledge Organiser consisted of sections to state the definition of an assigned concept, provide examples/non-examples and elaborate in the form of a concept map. The template and guidelines for completing Knowledge Organisers given to students can be found in the Appendix (Figures 4 and 5). To encourage students to produce their own examples and non-examples, we asked them not to copy them from the coursebook. Additionally, when demonstrating a model Knowledge Organiser (an example is given in Figure 5), we emphasised that many variations of a 'correct' answer are possible as a disclaimer.

An example of a concept map as the elaboration part of a Knowledge Organiser on Vector Space is given in Figure 2. The overarching idea of this concept map is a consolidated macro view of the concept of Vector Space as a mathematical structure used to generalise physical spaces such as $\mathbb{R}$, ${\mathbb{R}}^{2}$, ${\mathbb{R}}^{3}$ and, more generally, ${\mathbb{R}}^{n}$. The map also demonstrates that this structure serves as a unifying concept that defines fundamental subspaces related to matrices: the Nullspace, the Column space, and an Eigenspace. Moreover, the concept is utilised in solving differential equations since the solution set of a linear homogeneous ordinary differential equation is a vector space, which enables the formulation of efficient techniques for solving them. Thus, by presenting the relations between a group of concepts from different branches of Mathematics, such as Linear Algebra, Calculus, and Ordinary Differential Equations, the concept map promotes relational reasoning at a higher level – it pinpoints the overarching structure that exists at a higher level of abstraction. Arguably, the pedagogical value of identifying and illuminating this higher-level structure that unifies disparate-to-students mathematical concepts could be impactful.

As mentioned, students were expected to complete a Knowledge Organiser in weekly tutorials starting in Week 2. (Due to Covid-19, students had an option not to attend an in-person tutorial and could submit their solutions online to get a full mark. Only a small proportion of students preferred to do that.)

Moreover, two Knowledge Organiser tasks were set as 'marked problems': the selected concepts were Series in Week 3 and Vector Space in Week 7. Students were given at least one week to complete each Knowledge Organiser. Thus, two of the ten Knowledge Organisers were collected for marking.

An introductory session on Knowledge Organisers was included as part of the semester's first class, and an example was uploaded on Canvas (Learning Management System) for students to access anytime. However, the quality of student submissions of the first set of Knowledge Organisers revealed that some students did not clearly understand the concept mapping task. Therefore, the first author, who was an instructor on the course, provided another explanation session during a class in Week 4. The session provided a more detailed overview of the main features and characteristics of an effective concept map and included examples of high-quality concept maps, which were contrasted with low-quality concept maps.

Students' work on concept mapping tasks was assessed using a specially designed rubric developed and validated in a previous study ^[63]. As part of the study, several rubrics were developed based on previous research, and their predictive evaluation capacity was compared. This was done by comparing the scores assigned according to four different rubrics with their corresponding predictive power of the students' performance in various forms of assessment using linear regression analysis. The results showed that the "Ratio method" was an optimal rubric for assessing students' performance. The Ratio method assigns a numerical score equal to the inverse ratio of the number of concepts to the number of relationships connecting them. For example, the score for the concept map in Figure 2 is obtained by the inverse ratio of the number of concepts = 12 and the number of relationships between them = 15. Hence the score is equal to 15/12 = 1.25. According to this method, for a fixed number of concepts, higher scores are indicative of high-level elaboration manifesting through more cross-links identified between the group of related concepts. At the low level, fewer relations would be used in a map, mostly of generality/specificity type. In practice, the Ratio method would necessitate the use of a fixed lower bound for the number of concepts used in a map, which would depend on the minimal number of related concepts specified by an instructor in relation to the topic content.

3.4.   Data collection

3.4.1.   Questionnaire

In addition to coursework marks, the data for this research was collected through an online survey software, Qualtrics, which recorded participants' responses. An ethics approval was granted by the University of Auckland Human Participants Ethics Committee on 25 February, 2021 for three years (reference Number UAHPEC21976).

The questionnaire was given out at the end of the semester and included nine items/questions. The first seven items asked students to rate the extent to which they agreed with the statement about the Knowledge Organisers (e.g., "I found the knowledge-organiser activity valuable", "The knowledge-organisers made me confused", "I would like to see similar activities in other mathematics courses", "I feel that it was not a good use of my time"). The full list of Items 1-7 is provided in the Appendix. A Likert 5-point scale was used for Items 1 to 7, and descriptive statistics were obtained. The statements alternated between positive (Items 1, 3, 5, 6, 7) and negative (Items 2, 4). Hence, before finding the average of each student's response on the questionnaire, the scores on the negative statements were inversed (by subtracting the raw score from 6).

Additionally, the next item (Item 8) asked students: "On average, how much time did it take you to complete a single knowledge-organiser?", with 1- less than 10 minutes; 2 - 10-30 minutes; 3 - 30-50 minutes and 4 - more than 50 minutes given as possible responses.

The last question on the survey was: "What was the most beneficial aspect of the knowledge-organiser for you, if any?", an open-ended question that allowed the students to freely express their opinions about the Knowledge Organisers. Student responses to this question were analysed qualitatively.

3.4.2.   Concept map scores

Concept map scores were obtained using the Ratio method described in the 'Innovation: Knowledge Organisers' (section 3.3). They were used in the quantitative analysis in order to test for associations between student performance on the Knowledge Organisers and their perceptions and the time spent on-task.

3.5.   Data analysis

The study employed a mixed-methods approach to data analysis, combining the complementary power of quantitative and qualitative analyses ^[64].

3.5.1.   Qualitative analysis

The responses to the open-ended question, "What was the most beneficial aspect of the knowledge-organiser for you, if any?" were analysed using an inductive thematic analysis ^[65]. The second author, who has experience in postgraduate level research, had collected all the responses, read and re-read them to familiarise themselves with the dataset. In this process, keywords or key ideas were sought in each response. Then, the responses were first sorted into two major categories depending on whether the statement about the Knowledge Organisers was positive or negative. Next, within the two major categories, the subcategories were determined based on the particular aspect the students found favouring or disfavouring the Knowledge Organisers. In ensuring an objective process, the categorisation of the second author was compared with that of the first author, who is an experienced researcher. The two authors reviewed the decisions, and the responses' subcategories were confirmed after some adjustments.

3.5.2.   Quantitative analysis

Quantitative analyses were conducted to investigate: (1) the descriptive statistics, (2) the relationship between student perceptions and the concept map scores, and (3) the relationship between the time spent to complete a Knowledge Organiser and the concept map scores. Spearman's correlation and point-biserial correlation tests were carried out to address (2) and (3), respectively.

4.   Results

To answer RQ1, the responses to the questionnaire on Knowledge Organisers were analysed.

4.1.   Item 1 – Item 7

The descriptive statistics of the responses to the first seven items of the questionnaire (listed in Figure 6) are presented in Table 1.

A total of 251 students provided responses to the first seven items. As reported in Table 1, all means are within the range of 2.80 to 3.25, with the standard deviations between 1.00 and 1.07. The median and the mode of Item 1 to Item 7 are all 3.

4.2.   Item 8 – self-reported time-on-task

The frequency statistics of the responses to Item 8 ("On average, how much time did it take you to complete a single knowledge-organiser?") are presented in Table 2.

The mean of Item 8 responses was 2.49, with a standard deviation of 0.85. Of the students who participated in this study, 43.25% responded that the Knowledge Organisers took between 10-30 minutes to complete, and 33.33% responded that the task took between 30-50 minutes. The fact that over 75% of students spent between 10-50 minutes provided reassurance that the majority of students engaged with the task meaningfully, with only a small proportion spending unreasonably little time on the task.

4.3.   Question 9: Open-ended question

Unlike other questions in the survey, Question 9 ("What was the most beneficial aspect of the Knowledge Organiser for you, if any?") was an open-ended question soliciting student feedback on the new task. Student responses were analysed qualitatively using an inductive thematic analysis ^[65], as described in Data Analysis (section 3.5).

The frequency statistics of the responses are shown in Table 3.

Unlike the previous items in the questionnaire, this question had the lowest response rate, with only 150 students providing a comment. However, of those who had responded, a high proportion of the students (over 75%) stated a beneficial aspect of the Knowledge Organisers.

In determining the categories and the subcategories of the responses, following the methodology employed, common keywords or key ideas were used as a guideline in comparing and finding similarities between student responses. For the responses that discussed multiple aspects, it was decided that they would be allocated to the subcategory of the aspect for which they emphasised the most. Hence, each student response was mapped onto no more than one subcategory.

Figure 3 presents a thematic map of this qualitative analysis, which was developed according to the methodology described in Data Analysis (section 3.5). For illustration purposes, the size of each oval representing a subcategory is proportional to the number of responses belonging to that subcategory. The number of responses represented by each oval ranges from 1 to 25.

The two categories (positive and negative perceptions) and the subcategories of the responses with representative quotes are presented in Table 4 and Table 5, respectively.

Notably, out of 145 responses, only six referred to the use of examples/non-examples as part of their Knowledge Organiser tasks, with the majority of positive comments explicitly pointing out aspects of the concept mapping activity. This is in line with the expectations requiring disproportionally more effort in constructing a concept map than providing an example/non-example. These results will be analysed further in the Discussion section.

4.4.   Statistical tests

In order to answer RQ2, we conducted a point-biserial correlation test between the concept map score (obtained through the Ratio method) and the student perceptions classified as either positive or negative according to their responses to Question 9 (N = 143). Unless otherwise stated, data are mean ± standard deviation. Analyses were done to ensure that there were no outliers, the data were normally distributed, and the homogeneity of variances existed. A statistically significant correlation was obtained between the concept map scores and the student perceptions, rpb(141) = .242, p = .004, with positive perceptions resulting in higher concept map scores compared to negative perceptions, M = .77 (SD = .52) vs. M = .46 (SD = .51). However, the perceptions accounted for only 5.86% of the variability in the concept map scores.

To answer RQ3, we used Spearman's rank-order correlation test to inspect the relationship between the concept map score obtained using the Ratio method and the response to Item 8 on the questionnaire, which asked about the time spent to complete a single Knowledge Organiser (N = 249). The monotonicity of the relationship was validated prior to the analysis. As expected, the results of this analysis revealed a positive correlation between concept map scores and the time students spent on completing the Knowledge Organisers, rs(247) = .147, p = .021.

5.   Discussion

To address RQ1, we first note that the descriptive statistics for Items 1–8 are neutral overall (Table 1). Hence, not much can be inferred from these values. However, it is important to point out a misalignment in the results of the quantitative analysis (Items 1–7) and the qualitative analysis (Question 9). This could be because 251 students have responded to Item 1 – Item 7, whereas only 150 students have responded to Question 9. Hence, it is possible that the students who did not find the Knowledge Organisers beneficial would have simply chosen not to answer Question 9. Perhaps, this is a very plausible explanation for why the results from Items 1–7 tend to be neutral, whereas the positive opinions are much more prevalent in the data from Question 9 (114 positives vs 31 negatives).

In answering RQ1, an important result to note in the qualitative analysis is the five major subcategories identified in the analysis capturing themes in positive opinions expressed by students regarding their perceptions of the Knowledge Organisers. The subcategories are Connection/Relation, Revision, Helpful for Understanding, Summary/Conciseness and Improvement in Learning. Analysed through the lens of Human Cognitive Architecture and mathematics cognition theory, it is clear that these categories are aligned with various stages in developing and forming schemas. In order to develop a schema, the information must be attended to first (revised and understood), and then the connections between various aspects of information must be sought (related and connected). Importantly, the conciseness of the Knowledge Organisers' format reduces the burden of organising complex information and thereby could support working memory function by reducing its limitations. This, in turn, would make it more explicit (or 'pre-packaged') to be stored in the long-term memory as a branch of a complex network connecting various units of cognitive structure. Students' representative quotes about the Knowledge Organisers such as "It can help me sort out some knowledge points that cannot be clarified in the course book and find out the relationship between them", "Zooming out, to see all connections and aspects of a given topic", "Forcing myself to identify the connections between certain concepts", and "Understanding what we learned and how it related to other concepts. Because the diagram is useful to visualise understanding", demonstrate that the students were able to perceive and pinpoint the key mechanism afforded by the Knowledge Organisers that could make their learning more effective. However, this can not be considered as a groundbreaking, surprising result.

Regarding RQ2 and RQ3, we observed that students' performance on concept mapping is positively correlated with both their perceptions of Knowledge Organisers and the time spent completing the tasks. However, correlation does not imply causation. One possibility is that students with positive dispositions towards the Knowledge Organisers had the extra drive and motivation to spend more time on the Knowledge Organisers, thereby achieving a higher mark. On the other hand, students who generally get good grades tend to be more diligent in completing all course assessments with enhanced effort. Hence, another possibility is that in the process of engaging with the Knowledge Organisers' tasks on a weekly basis, the students could have developed their appreciation of this activity, identifying the benefits and, thus, developing positive perceptions of the tasks as a result. Either way, it seems that a deliberate attempt to increase the time students spend on the task is worthwhile. We discuss the rationale for this and offer some concrete options below.

The correlational evidence suggests that the students with negative perceptions had not engaged with the task in a meaningful way. This can be explained by the well-known Expectancy-Value Theory of achievement motivation developed by Wigfield and Eccles ^[66]. The theory posits that if a learner thinks a task is valuable for whatever reason, then the learner is likely to be motivated to succeed, thus putting more effort into learning. Since Knowledge Organisers are quite different from tasks that typically appear in mathematics education, it is probable that some students did not discern any value in completing these tasks, resulting in low engagement.

Regarding RQ4, one of the major goals of this study was to ascertain the feasibility of incorporating Knowledge Organisers into a mathematics classroom by conducting an implementation and evaluation study to complement the findings reported in our previous paper ^[22]. Based on the results of this study, we obtained the following insights. The design of the assessment task was partially successful. Specifically, the template and the instructions provided to learners seemed to serve the designated purpose. The rubric for marking student concept mapping based on the Ratio method is objective and easy to use ^[63]. Our concern about students 'gaming' the system by including many 'wrong' concepts and connections has not materialised. Albeit, this possibility can not be ruled out from occurring in general. Thus, our recommendation is to include a qualitative aspect of assessing a concept map (partial marks for effort) to counterbalance sole reliance on a purely quantitative measure.

However, we identified a severe limitation in the design of the overall innovation due to underestimating how novel and confusing concept mapping was for students at the start of the semester. As reported previously, after marking concept maps submitted in Week 3, extra instructional resources were provided to learners urgently: a more detailed explanation of how to construct a high-quality concept map was made during class in Week 4, together with examples of concept maps of high- and low-quality. This measure seemed sufficient, as evidenced by marked improvements in student outputs. Therefore, based on the triangulation of the results from qualitative and quantitative analyses, we conclude that the feasibility of future implementation would depend on several factors.

First, providing a quality explanation of the concept mapping activity at the start of the semester and emphasising its benefits is crucial. We found that provision and discussion of examples/non-examples of the actual concept maps of mathematical concepts were needed for learners to make progress. It would have been better to be done before they attempted the tasks. Second, based on the interpretation of the data through the lens of Expectancy-Value Theory ^[66], our recommendation is to embed the Knowledge Organiser into high-stake assessments such as tests and the final exam and inform students about it. As suggested by the data analysis, this simple measure may improve learning outcomes through increased motivation and, thus, engagement as a result of students' improved perceptions about the value of these tasks. As the theory posits, the more learners value the task, the more they are motivated to engage and complete it. Arguably, if the students are aware that these novel Knowledge Organisers are there to help them master a skill that will be assessed on the exam, they would be more inclined to value the tasks and hence, invest more time and effort. This, in turn, may result in more efficient and effective learning, given our current understanding of Human Cognitive Architecture and preliminary results from previous studies.

6.   Final remarks

The need to seek research-grounded solutions to improve practice has been emphasised in the mathematics education literature. A major concern has been flagged that classroom-based interventions in mathematics education are rarely undertaken and evaluated ^[33]. Contributing to the discipline in this vein, this exploratory case study reports on the design, development, implementation and evaluation of innovative tasks in a large classroom setting involving over 300 students. By utilising a mixed-methods approach and triangulation of the findings from qualitative and quantitative analyses, we were able to discern critical aspects pertaining to the feasibility of implementation and evaluate learners' perceptions. A major flaw in the initial design of the innovation was revealed: underestimation of novelty for learners necessitating extra instructional guidance and resources at the start of the trial. Moreover, the analytical insights derived from this investigation identified another implementational limitation affecting less-motivated students who did not engage with the tasks meaningfully. Based on the results of the data analysis and their theoretical interpretations, we were able to formulate implications for practice.

The design principle of our innovation is generalisable and transferable to other educational settings as a blueprint for an assessment structure and related instruction that could be utilised in mathematics education broadly. Stylianides and Stylianides ^[67] propose three dimensions of evaluation of classroom interventions: (1) how amenable it is to scaling up, (2) how practicable it is for curricular integration, and (3) how capable it is of producing long-lasting effects. Evaluated this way, our innovation can arguably be deemed effective for the first two criteria: the number of students utilising Knowledge Organisers is unlimited; it is practicable for incorporation into existing curricular structures at any level. However, ongoing research is needed to determine long-lasting effects in the context of tertiary education.

It is important to note that the idea of using a knowledge organiser has been introduced previously. Many studies have explored the use of various knowledge organisers, such as an advance organiser, a post-organiser, and knowledge of the behavioural objective in research dating back to the 1970s ^[68,69]. However, not much has been done concerning the assessment of a global conceptual understanding using concept mapping. Future studies could investigate whether or not it is principally different from a local conceptual understanding in the sense of an example space (based on the variation theory ^[30,38]); and if so, its effectiveness needs to be compared with other learning activities in mathematics education in properly controlled settings. On a different note, an important line of inquiry may be investigating collaborative concept mapping activity as part of small-group problem-solving sessions to analyse how the global conceptual understanding of a group of related concepts could be networked and negotiated among learners. Moreover, further theoretical and empirical research can attempt to answer one of the key questions: whether a learner's ability to produce a high-quality Knowledge Organizer accurately reflects their local and global conceptual knowledge and understanding of mathematics.

Appendix