Mathematical modeling and mining real-world Big education datasets with application to curriculum mapping

1.   Introduction

Curriculum mapping is an important tool of modern educational design. A clearly mapped curriculum is able to show the links and relationships amongst the different courses in the curriculum to students and other stakeholders. An important part for the educational curriculum design of a course is to determine the prerequisites for the subjects contained in the course ^[1]. The prerequisite knowledge can be defined as the skills and information which are necessary for a student to succeed in a given instructional subject within a curriculum ^[2]. The subject prerequisites serve two purposes: 1) It ensures that a student has the necessary background knowledge before undertaking the subject to be studied; and 2) It forms a scaffolding for learning amongst the subjects so that students are led to take subjects with increasing levels of complexity (e.g., beginning from introductory subjects, to subjects with developing materials, and ending with subjects requiring more complex materials). This issue is particularly important for the rising popularity of online-based education ^[3] and the emergence of Massive Online Open Courses (MOOCs) ^[4]. One challenge to be addressed is that subjects with similar titles from different universities may put barriers for setting up a reasonable, time-saving learning path because the student may be unable to distinguish them before an intensive research on all subjects related to the degree from all universities. Therefore, it would be necessary to supply optimized relations amongst unlearned subjects based on the subjects previously learned.

Several authors have proposed mathematical models for linking the dependencies of courses within a curriculum to identify useful relationships and connections. The authors in ^[5] used a complex systems approach to propose a mathematical model termed as the Curriculum Prerequisite Network. In their work, the authors used a mathematical graph theory model (directed acyclic graph) where the nodes were used for representing the courses and the links between nodes represented the course prerequisites. Another approach was taken by authors in ^[6] which used KDD (Knowledge Discovery in Databases) methods to mine useful information from a medical and healthcare curriculum. Their methodology used the CRISP-DM (Cross-Industry Standard Process for Data Mining) reference model ^[7] and investigated the following queries: 1) Educational disciplines which do not conform to the curriculum data; 2) Identification of overlapping areas across the curriculum; 3) Identification of discipline clusters within the curriculum; and 4) Identification of courses which belong to important parts of the curriculum.

Some recent approaches and investigations for mining prerequisite information from education curriculum datasets have been proposed by authors in ^[8,9]. The authors in ^[8] proposed an approach termed as concept graph learning (CGL). In this work, the authors identified an issue where the course descriptions from different universities may contain descriptions and terms which vary amongst the universities. The authors proposed using an intermediate mapping termed as "universal concepts". The key idea is that courses from different universities would use the same terms in the universal concepts mapping and this allows courses from different universities to be mapped to the common space. The model can then be used to predict implicit prerequisites amongst the various courses. Their work was validated from educational and curriculum datasets retrieved from four universities in the United States. The authors in ^[9] presented a survey of analytics and techniques for Big education datasets. The work also reported on graph learning for real-world educational datasets from four universities in Australia. The works in ^[8,9] have demonstrated the validity of mining curriculum data from education datasets. However, the full potential and various challenges of the curriculum mapping has not been investigated as only the data from four universities have been considered in the earlier works.

Considering the above challenges, in this paper, we discuss an approach for modeling and mining real-world Big education datasets with application to curriculum mapping. We use concept graph-based learning techniques, discuss data representations and techniques which are more suited for large datasets, and validate the work on nineteen real-world education datasets crawled online from university websites in Australia. The practical challenges to collect and extract the relevant data from the university websites are also discussed in the paper. The structure of this paper is as follows. Section 2 introduces the description of education graph learning and the data pre-processing. The proposed graph-based learning approach is discussed in Section 3. Section 4 gives discussions for the experimental results and validates the effectiveness of the proposed approach. Section 5 gives some concluding remarks.

2.   Description of education graph learning and data pre-processing

2.1.   Description of education graph learning

The overview of the education graph learning process is shown in Figure 1. The top part of the diagram shows examples of subjects in different universities. The second part of the diagram shows the Principal Concepts Pool (PCP), where the concepts of each subject are extracted by natural language processing (NLP) in the form of keywords. Given the ground truth data of prerequisite relations amongst subjects and the groups of concepts (keywords) of each subject, the relations of keywords between different group can be established, where every group of keywords represent one subject. After training based on all ground truth data, the relations of concepts will be achieved. Such learned relations amongst concepts are capable of inferring relations amongst subjects. For example, if many of the concepts of subject 1 are prerequisites of concepts of subject 2, it can be concluded that subject 1 should be a prerequisite of subject 2.

2.2.   Data pre-processing and representation

The experimental data was provided by real-world datasets crawled online from nineteen Australian university websites. The experimental data was obtained by regular web scraping techniques using Python on the respective university subject data websites which are available on the Internet. There were several challenges that had to be addressed for the data pre-processing. A practical challenge that was faced during the data collection process was to dynamically extract the generated subject data from the university websites. This process known as web-scraping was accomplished using Selenium ^[10]. A second challenge which was encountered was to clean the data to make it suitable for the experimental analytics. The raw data was cleaned by four methods including: 1) Conversion of data to lowercase—all characters were converted to lowercase; 2) Tokenization—sentences were broken down into tokens where each token represents a word, symbol or number; 3) Word stemming—removing the redundancy over words by conversion to their root words (e.g., conversion of "calculating", "calculated" to the root word "calculate"; and 4) Removal of stop words (e.g., "and", "the", "that"). The data representation used the Bag of Words (BoW) ^[11,12] approach. The BoW is a popular representation technique used for NLP to extract key attributes from large-scale text-based documents and objects where histograms of words are used as features to perform the classification.

The details of the experimental data are shown in Table 1. The first column shows the names of the universities in the experiment. The second column shows the number of subjects related to Computer Science (CS) and Information Technology (IT) in each university. The third column shows the total number of prerequisite relations relevant to CS and IT subjects in each university. The fourth column shows the number of key words retrieved from all CS and IT subjects in each university. The PCP contained a total of 30,840 concept keywords.

3.   Methods

3.1.   Notation and methodology

Table 2 shows a summary of the notations and terminology following the symbols used by the authors in ^[8]. Given a training set of courses with a bag-of-concepts representation per course as a row in matrix X, and a list of known prerequisite links per course as a row in matrix Y, the objective is to optimize the matrix A whose elements specify both the direction (sign) and the strength (magnitudes) of each link between concepts.Table 2. Summary of notations and terminology.

The relation between two subjects (subject i and subject j) can be expressed as:

The objective is to find a matrix $A$ by minimizing the cost function:

where, $Tr$ represents the trace of a matrix and $H$ is the conjugate transpose. We discuss two approaches to perform the optimization: 1) Rank approach; and 2) Learning a sparse concept graph.

3.2.   Rank approach

It should be noticed that $c$ or the length of $x$ is large, and increases the computation cost of Eq (1). Therefore, it becomes necessary to solve the following problem.

The inner product in the angle brackets of Eq (3) is 0. The gradient of the Eq (3) over $A$ is

An observation is that ${X}^{T}$ is a $c\times n$matrix, and the matrix $M$ is a $n\times n$ matrix. Therefore, we only have to calculate the $M$ matrix, then the matrix $A$ can be projected back by Eq (5). The Rank approach algorithm based on Projected Gradient Decent (PGD) ^[8] is shown in Figure 2.

3.3.   Learning a sparse concept graph

A finding from the data collected from the 19 universities is that $X$ is a sparse matrix. The matrix $A$ is also a sparse matrix as many concepts have no relations between each other. Therefore, an optimization could be performed by calculating the cost function based on sparse matrices. The cost function is changed as shown in Eq (6).

The Learning a sparse concept graph algorithm based on Projected Gradient Decent (PGD) is shown in Eq (7) and Figure 3. For sparse techniques involving tree-structured data, the technique in ^[13] can be considered.

4.   Experimental results and discussion

4.1.   Within-university prediction results and discussion

The crawled and pre-processed data from Section 2.2 were split into training sets and test sets. For an initial investigation, the models were used for within-university prediction. In this experiment, the models were trained from subject data from one university, and tested by the data from the same university. In this case, one third of the data set was used to train the model, another one third was used for validation, and the final one third was used to test the model. Validation was performed using three-fold cross validation. Two metrics MAP (mean average precision) and AUC (area under the curve) were used to evaluate the performance of the algorithms. MAP ^[14] is the average precision which is a popular metric used to measure the performance of models for the applications or tasks such as document/information retrieval, object detection. AUC ^[15] is the measure of the ability of a classifier to distinguish between classes and is used as a summary of the receiver operating characteristic (ROC) curve, which is a graph showing the performance of a classification model at all classification thresholds. Table 3 shows the within-university performance for Algorithms 1 and 2. The following observations can be made from the comparisons in Table 3: 1) Algorithm 1 gave better performance than Algorithm 2 for the majority of the universities tested; and 2) In general, the AUC scores were higher than the MAP scores for the within-university performance. Based on the observations in 1) and 2), we will focus on using Algorithm 1 with the AUC metric for further investigations.

4.2.   Cross-university prediction results and discussion

For a next investigation, the models were used for cross-university prediction. In this experiment, the models were trained from subject data from one university, and tested by the data from a different university. The keywords from the various universities were combined to create a dictionary which was then used for coding concepts for all universities. Table 4 shows the cross-university performance using Algorithm 1 and the AUC metric. The following observations can be made from the comparisons in Table 4: 1) For most universities, the inter-university performance (shown by the diagonals of the matrix) gave better prediction performance that the cross-university performance. However, in some cases, the cross-university performance gave better performance. This shows that having universal concepts and keywords amongst different universities could allow for significant cross-university prediction (i.e., transfer learning amongst universities, e.g., UoA and UCan which gave a performance of 0.907); and 2) There were some cross-university AUC results which gave zero values. This shows the converse situation where the different universities do not have any keywords or have very few keywords in common. In this case, the subject data learnt from one university is not able to help in the prediction performance for the other university.

For a final investigation, we performed a comparison of the number of keywords versus the sum of the predictive performance for all universities. Figure 4 shows a summary of the results for the various universities. Although, there were universities with many keywords which gave high predictive performance, there were also universities with fewer keywords which gave comparable performance (e.g., USydney with 2518 keywords gave a performance of 4.694, however UTasmania with 1075 keywords gave a comparable performance of 4.590). This shows that the keywords which are used are more important than the total number of keywords. It is important to note that the experimental data was automatically provided by real-world datasets crawled online from Australian university websites. No further pre-processing or keyword selection was performed other than that discussed in Section 2.2.

5.   Conclusions

This paper has proposed an approach for modeling and mining curriculum Big data from real-world education datasets crawled online from university websites. The practical scenario is to enable students to have a study plan to complete a course by accumulating credits on top of subjects he or she has already completed from different universities. Other than the practical use for students, the work also demonstrates the value of transfer learning amongst different universities for Big education data where the subject data from universities can be used for predicting pre-requisite links amongst subjects from other universities. The work has been validated on nineteen (50% of the total number of public universities in Australia) real-world education datasets from university websites in Australia and showed good performance. Our future work aims to extend the investigation to all 38 public universities in Australia, and also investigate the transfer learning for universities in different countries.

Conflict of interest

The authors declare there is no conflict of interest.