KnE Social Sciences | The Economies of Balkan and Eastern Europe Countries in the Changed World (EBEEC) | pages: 343-356

, , , and

1. Introduction

Many fields in daily life are effected by technological developments. These new technologies bring innovative methods in education as well as in working life. Education goes on in human's life after graduate from secondary or higher education. The people from business life, to solve their problems easier and promote themselves, have to continue educational work in certain periods. So, organizations have to provide some education opportunity for their workers. Absolutely, technological developments provide new environments about interactive methods in education life for the students and other users.

Participation of the people from training or business life, encountered some difficulties. In this point, distance education is the new opportunity to take education for people who have financial problems, lack of time or some disabilities. In this situations, distance education is a good option. Information Technologies can be maintained with learning management system (LMS). Each LMS has some benefits for main target. Therefore, selecting of LMS is serious work for institution or individual organizations.

This study is focused on the specific decision making process which LMS should be chosen depend on their main interests when building distance education system in some institutions or trade organizations. Firstly, some information about different LMS will be examined. The efficient factors of selecting LMS will be revealed. And finally, according to priorities of these factors, different LMS will be compared by the Analytical Hierarchy Process (AHP) which is one of the most popular decision making methods.

2. Methodology

2.1. Learning Management System (LMS)

The education system is perhaps at the top of the systems affected by technological changes, and is undergoing fundamental changes. Learners are increasing and becoming more varied, demanding more flexible and easy to reach learning contents, not want to be dependent on only one learning method want to learn a lot of thing at the time and place where they can reach. Internet is offering many tools and application packages to educators that can be used at all stages of teaching, and such tools increase the effectiveness and efficiency of teaching [3].

Distance education is very likely the fastest growing area of education. This brings new benefits but also difficulties for students. From the results of comparison it can be said that distance education is comparable to internal in the quality of knowledge of successful students. Distance and combined education managed by information and communication technology is a modern form of education. There is not many analysis for distance education, knowledge or evaluation [8].

Distance education firstly starts with delivering written and printed text with postal services. Then continue with the use of radio and television. With this second way thousands of people educate at once. Most of the open universities in the world are started with second generation [2]. The third generation of distance education utilizes information and communication technologies (ICT) to provide interaction in addition to content delivery. The third generation of distance education allows personalization of content depending upon learners' learning preferences. In fourth generation is based on online delivery via the Internet. Fifth one is intelligent flexible learning model based on the interactive nature of the internet [2].

Kesim(2012) was found that historical development process of distance education respectively as follow: correspondence study era, broadcasting era, open universities era, teleconferencing era and internet era. As Kesim(2012) declared that the first era specified is the correspondence study era. The most important features of this era are text based communication and instruction that was focused on postal correspondence. When the second era is being examined, broadcasting of television and radio make a big contribution to learning and teaching processes. The open universities era is accepted as the third era and is specified as the historical development process of distance learning. This period reflects the organizational development process of distance education through new approaches. Open universities are the new organizational structures that were characterized as a new perspective. The next era is called the teleconferencing era. Especially real time interaction of students via video and audio conferences reflect a significant transformation process for distance education. Computer networks and satellites are the main variables that make the distance learning process effective in the teleconferencing era. In conclusion, the fifth era of distance learning is called the internet era. The most prominent feature of this period is considered to be internet technologies.

The most important advantage for learner in distance education is getting knowledge without the constraint of time and space, especially in an asynchronous distance e-learning system. The other important factor is that e-learning may save internal training cost for some enterprises organizations in a long-term strategy. Also, the e-learning can be used as an alternative self-training for assisting or improving the traditional classroom teaching [5].

E-learning and learning management systems give alternative learning opportunities for students. Students include workers especially, and those remote students who cannot avail themselves for face-to-face teaching and learning due to a myriad of factors (Dzandu and Tang, 2015). E-learning courses can be used in several areas such as primary and higher education, training for employees in organizations [13].

Main criteria's of e-learning are defined as quality of web learning platform, e-learning materials, synchronous learning, learning records and self-learning by Chao and Chen(2009). And also e-learning materials are the most important one. Contents of material are expected to be user friendly. The web-based elearning system (WELS) has emerged as a new means of skill training and knowledge acquisition, encouraging both academia and industry to invest resources in the adoption of this system. The capability and flexibility of the web-based e-learning system (WELS) have been demonstrated in both training and education, resulting in its adoption by the academia as well as the industry Shee and Wang, 2008). The e-learning course is used as a synonym for information systems. Performance of information systems can be measured by effectiveness (Dzandu and Tang, 2015).

At the beginning many people think that distance learning is very hard system for them [8]. But it was also found that the most students in distance education are more successful than students who take face to face education. They are also successful at the exams. For success of distance education, learning contents should be prepare by the help of new technologies with considering students opinion [3,8,11].

LMS have demonstrated to be a useful tool in learning processes, especially for on-line students, and they are consolidated in most universities worldwide. New advanced features can now be introduced in LMSs aimed at enhancing the experience of students and lecturers with these platforms and even to improve academic results [12].

LMS are web-based systems that allow instructors and/or students to share materials, submit and return assignments, and communicate online [11]. Quality and scales of learning management system can be measured by costs for software, update and supports. The solution is open source systems. Open source (OS), is the source code of a software that is readily available to the public for extension and modification depending on the user's needs [4]. Even though commercial LMS available in the market for a price, open source LMS are available with no cost [4]. Nowadays there are many open source LMS available in the sector, each having its strengths and weaknesses. For this reason, it is important for a prospective user to be well informed in order to make the best decision [4].

Learning management is not only for academical organizations but also organizations whom apply training courses [7]. E-learning is an important role in teaching and learning in both different levels of schools and commercial or industrial organizations in literature [5].

Tools and materials in LMS are supported learning courses. Tools can be assessments, exams, grades, communications tools (messages, chat rooms, forums and etc.), registration, scheduling, and course and participants statistics. And materials can be interactive applications, course contents, documents and papers, audio and video files. Sustainability and reusability of learning objects are important [20]. Also users and system logs are kept in LMS. Users in LMS are system administrators, course coordinators, instructors, students and guest. There is two groups of LMS; Free LMS are Moodle, Claroline, Atutor etc., and commercial LMS's are Blackboard, SharepointLMS, and WebCT.

Factors in Learning Management Systems are listed [5] as personal characteristics and system instruction, Participant motivation and system interaction, range of instruction materials and accuracy, webpage design and display of instruction materials, e-learning environment, webpage connection, course quality and work influence, learning records, instruction materials. Possible factors that effecting the selection of LMS can be license costs, flexibility of learning contents, security and market share.

2.2. AHP

Analytic Hierarchy Process (AHP) which is a multi-criteria decision making (MCDM) method for addressing complicated problems reduces the waste of time and improves the accuracy of decision making [20]. MCDM Methods are listed as ELECTRE, TOPSIS, AHP, Multiplicative Exponential Weighting

(MEW), Simple Additive Weighting (SAW), AHP, DEMATEL, Fuzzy Integral. [5,20]. "Which MCDM method is perfect?" question is hard to answer because searching for the perfect and best MCDM method is critical and valuable [20].

AHP method is simple, flexible and usable with both quantitative and qualitative criteria. AHP does not use difficult mathematical expression. It is transparent for decision maker experts for rankings. In addition, it allows sensitivity analysis with relative priorities by changing ranking values. AHP method is used for complex MCDM problems with both single and multi-dimensional MCDM problems with relative values. [20]. Some of Multi Criteria Decision methods can be used in comparison of e-learning in literature. [5]. These are Fuzzy integral, DEMATEL, and AHP.

AHP is a decision making tools and is developed by Saaty. Especially, it is used to analyze and structure for complex decision problems [9]. AHP is defined as a decision and forecasting method giving the percentage distribution of decision points in terms of the factors affecting the decision (Yaraloğlu, 1994). Singaravel and Selvaraj (2015) state that AHP is a method for each data for weight criteria of each objective are interested in data from set of pair wise comparison (Triantaphyllou, 1995). Due to AHP methods based on user perception, weight of variables very important to final accurate result. Saaty (2008) state that AHP constructed by 4 following steps:

  • Define the problem and determine knowledge that sought.

  • According to main object sort variables from lowest number to highest number.

  • Construct a set of pairwaise comparison matrixes. Every matrix is constructed by the each variables.

  • Use the priorities obtained from the comparisons to weigh the priorities in the level immediately below (Saaty, 2008). Weighting the factors is very important step and it can be changed for decision makers.

When make a comparisons, it is very important that scale number of variables for each decision points. The fundamental scale of absolute numbers are in Table 1.

Table 1

The fundamental scale of absolute numbers. Resource: Saaty, (2008).

Relative Importance <statement> <title>Definition </title> </statement> Explanation
1 Equal Importance Both activities are equals for objective
3 Moderate Importance Slightly favour one activity over another
5 Strong Importance Strongly favour one activity over another
7 Very Strong Importance Very strongly over another
9 Extreme Importance One activity over another is of the highest possible order of affirmation
2,4,6,8 Intermediate values

To use AHP Process, following steps have to be done.

Step 1: Describe all aspect of decision making problems. In this step, determine decision points and decision variables that effected decision points. Making this process properly, effect the whole problem solve process.

Step 2: Generated the factors comparison matrix. This matrix is square matrix has n x n dimensions. Diagonal of the matrix takes the value 1. The matrix is showed below:


A is Factors Comparison Matrix a 11 = a 22 = ⋯ = a n n = 1 Diagonal matrix n×n dimensions

when generate A Matrix, use the fundamental scale of absolute numbers (Yaraloğlu, 2010). If user assume that first factor is more important than third factors, A matrix's first row third columns value is 3. If opposite event occur, first column third row value is 1/3. If user that first factor equal important with the third factor, so value of A matrix first row and third column is 1. The comparison is done just values above the diagonal of A matrix. The values below diagonal of A matrix is calculate like this:


Step 3: In this step, determine the percentage importance of factors distribution. To make this, every element of matrix is divided sum of it's columns (Alexander, 2012). Thus, create the B column vector. This vector and formulation are showed below.


This calculation is done as long as number of factors and create B column vectors. Sum of each B vectors column element is 1. These columns are synthesized and create C Matrix [18]. Then, it can be determined percentage level of relative importance of the factors with C Matrix. Sum C Matrix's each row elements and arithmetic mean is taken with formula and create W column vector. Values of W column vector shows superiority degrees of factors. C Matrix, W column vector and formula of W column vector are shown below.


Step 4: In this process, find consistency Index (CI) and Consistency Ratio (CR). Firstly, Multiply A Matrix and W column and create D Matrix. Then create E values and λ. E, λ, CI and CR formulas are shown in Table 2.


If CR values is lower than 0.1, it shows that decision maker made comparison consistently. Otherwise this event shows that there is a problem about calculation.

Step 5: In this process the steps before this apply for all factors. Rows and columns of comparison matrix is created by all decision points. So, these matrix have m x m dimensions and are repeated until the number of factors. Then S vectors are created by result of all factor matrix. This is shown below:


Step 6: S i vectors are merged and create K matrix has mxn dimensions. Then K matrix multiply with the W vector and create L vector has m element. Formulas are shown below.


2.3. Comparison of LMS

Steps of LMS selection process of effective system are [5];

  • determination of the affecting factors,

  • questionnaire collection and statistical analysis,

  • weighting these factors,

  • evaluation of the entire performance according to these weighted factors.

In this section, compare four LMS (Sakai, Moodle, BlackBoard, SharePoint LMS) with AHP Process. Determined license cost, flexibility, security and market share as factors. Every decision point has value of factor. These values are created by the user based on experiment and knowledge. Table 3 is shown these values.

Step1: Make decision matrix for each factor. While determine the compare values of factors, used following method. These method is created by the authors.

Step2: In this section, make decision matrix for all factors and using all AHP steps. Weight of factors are given by the user. For different objects, examine different weight of factor. Following area shows these steps.

Factor is License Cost:


License Cost Sakai Moodle BlackBoard SharePoint LMS
Sakai 1 3 7 5
Moodle 0.333 1 5 3
BlackBoard 0.14285 0.2 1 0.333
SharePoint LMS 0.2 0.333 3 1
Sum of Columns 1.675 4.533 16 9.333

Factor is Flexibility:


Flexibility Sakai Moodle BlackBoard SharePoint LMS
Sakai 1 0.333 5 5
Moodle 3 1 7 7
BlackBoard 0.2 0.14285 1 1
SharePoint LMS 0.2 0.14285 1 1
Sum of Columns 4.4 1.62 14 14

Factor is Security:


Security Sakai Moodle BlackBoard SharePoint LMS
Sakai 1 1 0.2 0.2
Moodle 1 1 0.2 0.2
BlackBoard 5 5 1 1
SharePoint LMS 5 5 1 1
Sum of Columns 12 12 2.4 2.4

Factor is Market Share:


Market Share Sakai Moodle BlackBoard SharePoint LMS
Sakai 1 0.333 0.14285 1
Moodle 3 1 0.2 3
BlackBoard 7 5 1 7
SharePoint LMS 1 0.333 0.14285 1
Sum of Columns 12 6.66 1.48 12

2.4. Results

In this section, use to the different weight values for each object are determined. For example, License Cost is more important than the other factor. Thus, defining 0.40 as weight of first factor (License Cost), and the other values are 0.20. These steps are repeated for four factors. Some results are shown below.

As shown in Table 5, when decision makers have different priority, best choice of learning management system shows variety. If License cost is the priority for decision makers, Sakai is the best choice LMS with 0.31 point. If flexibility is the priority for decision makers, Moodle is the best choice LMS with 0.34 points. And if security and market share of LMS have priority for decision makers, Blackboard is the best choice. As a result of this AHP Process, license costs and flexibility factors have priority open source learning management system is the best choice. If security and market share of LMS is the priority, one of the commercial software will be the best choice.

A mobile application is developed by authors for making comparison easily between different LMS (Figure 1-2). According to giving priority to factor weight, the application allow to find best choice and worst choice from all results. It uses AHP method. Some screenshots about the application shown in tables below.

Figure 1

Main Screen, factor and decision points of application.

Figure 2

Selecting weight of factors, decision matrix and final result.


Easy to use by the user of this application, the weight values of factors are given by the system. Used four factors to compare learning management systems. The number of factor may be increased, by the future works.

3. Conclusion

Table 2

RI Values. Resource: Yaralioğlu, (2010).

1 0 7 1.32
2 0 8 1.41
3 0.58 9 1.45
4 0.90 10 1.49
5 1.12 11 1.51
6 1.24 12 1.48
Table 3

The factor values for each decision point.

Decision Points License Cost Flexibility Security Market Share
SAKAI 9 7 5 3
MOODLE 7 9 5 5
SharePOINT LMS 5 3 9 3
Table 4

Compare method to find decision matrix values

First Decision Point Second Decision Point Difference Value Final Value
1 1 0 1
1 3 -2 1/3
1 5 -4 1/5
1 7 -6 1/7
1 9 -8 1/9
3 1 2 3
3 3 0 1
3 5 -2 1/3
3 7 -4 1/5
3 9 -6 1/7
5 1 +4 5
5 3 +2 3
5 5 0 1
5 7 -2 1/3
5 9 -4 1/5
7 1 +6 7
7 3 +4 5
7 5 +2 3
7 7 0 1
7 9 -2 1/3
9 1 +8 9
9 3 +6 7
9 5 +4 5
9 7 +2 3
9 9 0 1
Table 5

Best and worst choices for different factors.

Distance Education System is better alternative for all institution emphasis on education. So, giving a new chance to educate for students or workers that has some problems such as financial, disabilities, lack of time or geographical gaps. Internet based LMS is required when distance education is needed. But there are many LMS which are open source and commercial. Selection of correct LMS may be the big problem for decision makers. Because each LMS has some different features. And every decision maker's expectation from LMS, may be different. In this study, some factors license cost, flexibility, security and market share are used for choosing process. Two open source software (Sakai and Moodle) and two commercial software (BlackBoard and SharePoint LMS) are selected as decision points. The values of decision points related to each factor were determined by the some specific information. Then compared all decision points with AHP. Reveal that which LMS is best, when altered the weight of factor values for main object. According to some basic comparative weight factor values, open sources LMS (Sakai and Moodle) is more advantage than the other pair (BlackBoard and SharePoint LMS) for License cost and flexibility. On the other hand, commercial softwares (BlackBoard and SharePoint LMS) is more advantage than the other pair (Sakai and Moodle) for security and market share.

To compare LMS easily, the mobile application is developed for the all users who don't know anything about AHP. In future study, the number of factors and decision points may be increase and with this way, evaluate compare process more widely.



Alexander M., Decision Making Using the Analytic Hierarchy Process (AHP) and SAS/IML, The Proceedings of the South East SAS Users Group2012Durham, NC


Aoki K., Generations of Distance Education: Technologies, Pedagogies, and Organizations, International Conference on New Horizons in Education, INTE2012. Procedia - Social and Behavioral Sciences2012 Page: 1183-1187.


Cavus N., Distance Learning and Learning Management Systems. Procedia - Social and Behavioral Sciences, Year: 2015, Volume: No. 191, Page: 872-877.


Cavus N., Zabadi T., A Comparison of Open Source Learning Management Systems , Procedia - Social and Behavioral Sciences, A Comparison of Open Source Learning Management Systems , Procedia - Social and Behavioral Sciences, Year: 2014, Volume: No. 143, Page: 521-526.


Chao R.-J., Chen Y.-H., Evaluation of the criteria and effectiveness of distance e-learning with consistent fuzzy preference relations, Expert Systems with Applications, Year: 2009, Volume: 36, Issue: 7, Page: 10657-10662. DOI: 10.1016/j.eswa.2009.02.047


Dzandu M. D., Tang Y., Beneath a learning management system - Understanding the human information interaction in information systems, Volume: No: 3, 6th International Conference on Applied Human Factors and Ergonomics (AHFE 2015) and the Affiliated Conferences, AHFE, 2015., Procedia Manufacturing2015 Page: pp


El-Mekawy M., Rusu L., Perjons E., An evaluation framework for comparing business-IT alignment models: A tool for supporting collaborative learning in organizations, Computers in Human Behavior, Year: 2015, Volume: 51, Page: 1229-1247. DOI: 10.1016/j.chb.2014.12.016


Fojtik R., Comparison of Full-Time and Distance Learning. 4th World Conference on Educational Technology Researches, WCETR-2014. Procedia - Social and Behavioral Sciences, Comparison of Full-Time and Distance Learning. 4th World Conference on Educational Technology Researches, WCETR-2014. Procedia - Social and Behavioral Sciences2015 Page: 402-407.


Hanine M., Boutkhoum O., Tikniouine A., Agouti T., Application of an integrated multi-criteria decision making AHP-TOPSIS methodology for ETL software selection, SpringerPlus, Year: 2016, Volume: 5, Issue: 1, article no. 263, DOI: 10.1186/s40064-016-1888-z


Kesim E., The Economic Contributions of Developing E-Learning Technologies from the Perspective of Educational Organizations, No.4, Year: 2012, Page: 2070


Lonn S., Teasley S. D., Saving time or innovating practice: Investigating perceptions and uses of Learning Management Systems, Computers and Education, Year: 2009, Volume: 53, Issue: 3, Page: 686-694. DOI: 10.1016/j.compedu.2009.04.008


Muñoz A., Lasheras J., Capel A., Cantabella M., Caballero A., OntoSakai: On the optimization of a Learning Management System using semantics and user profiling, Expert Systems with Applications, Year: 2015, Volume: 42, Issue: 15-16, Page: 5995-6007. DOI: 10.1016/j.eswa.2015.04.019


Pereira F. A. D. M., Ramos A. S. M., Gouvêa M. A., Da Costa M. F., Satisfaction and continuous use intention of e-learning service in Brazilian public organizations, Computers in Human Behavior, Year: 2015, Volume: 46, Page: 139-148. DOI: 10.1016/j.chb.2015.01.016


Saaty T., Decision Making with the Analytic Hierarchy Process, Volume: 1, 2008


Shee D. Y., Wang Y.-S., Multi-criteria evaluation of the web-based e-learning system: A methodology based on learner satisfaction and its applications, Computers and Education, Year: 2008, Volume: 50, Issue: 3, Page: 894-905. DOI: 10.1016/j.compedu.2006.09.005


Singaravel B., Selvaraj T., Optimization of machining parameters in turning operation using combined topsis and ahp method, Tehnicki Vjesnik, Year: 2015, Volume: 22, Issue: 6, Page: 1475-1480. DOI: 10.17559/TV-20140530140610


Triantaphyllou E., Mann S. H., Using the Analytic Hierarchy Process for Decision Making In Engineering Applications: Some Challenges, Volume: 2, International Journal of Industrial Engineering: Applications and Practice1995 Page: 35-44.


Wu Q., Ren H., Gao W., Ren J., Multi-criteria assessment of building combined heat and power systems located in different climate zones: Japan-China comparison, Energy, Year: 2016, Volume: 103, Page: 502-512. DOI: 10.1016/


Yaralioglu K., Yaralıoğlu K., Karar Verme Yöntemleri, Detay Yayincilik, Ankara. Yaralioglu, Ege Academic Review, Year: 2010, Volume: 3, Issue: 1, Page: 119-137.


T. , Yigit T. et. al., Multi Criteria Decision Making System for Learning Object Repository, Procedia - Social and Behavioral Sciences, No.141, Year: 2014, Page: 813-816.



  • Downloads 25
  • Views 176



ISSN: 2518-668X