INTRODUCTION
In life, humans are often confronted by complex decision-making problems, such as buying stocks, choosing a university, selecting a healthcare supplier or purchasing a house. In the latter example, the decision-maker has to decide between a number of houses and does so dependent upon multiple criteria like price, location and aesthetics. Multi-criteria Decision-Making (MCDM) is a field of study, which attempts to solve these problems. It can be understood as a tool to structure the decision-making process in an attempt to find the most optimal solution. MCDM has already found application in numerous fields (Chakraborty et al., 2023; Emovon & Oghenenyerovwho, 2020; Tzeng & Huang, 2011; Zwiegelaar & Rahbarimanesh, 2022). Almost every decision-making problem employing MCDM consists of the following steps: Defining the decision-making problem, listing alternatives, determining criteria, weighting the criteria, comparing the alternatives with regards to the defined criteria, evaluating the performance of the alternatives and selecting the preferred alternative. While the fundamental steps remain consistent across the various Multi-Criteria Decision-Making (MCDM) methods, each method approaches these steps in their own unique way. Another key distinction within MCDM methods is the difference between multi-objective decision-making and multi-attribute decision-making. While
© 2024. Published by AHFE Open Access. All rights reserved. 175
MODM methods do not operate on a predetermined set of alternatives but are restricted by a set of optimal objective constraints (e.g. the temperature for the country to travel to should measure between 10
- C–20
- C in the mean), the number of alternatives in MADM is predetermined and limited (e.g. a list of countries to travel to). Due to the scope of this paper, it will be mainly concerned with MADM methods. Similarly, methods that employ fuzzy logic, probabilistic methods, or group decision-making will either not be addressed or will receive limited coverage. To provide the reader with a better overview of MADM methods, a concise summary will still follow. Probabilistic methods are used to handle uncertainty in the decision-making process, which is useful when dealing with incomplete information. Similarly, fuzzy logic is utilized when decision-makers struggle to give precise numerical preferences. To paint a clearer picture, the first addresses uncertainty in external events, while the latter does so for indecisiveness within the decision- maker. While all MADM methods can be used in a group decision-making process, it still requires extra steps to funnel the different interests and preferences of stakeholders into one.
According to (Bozorg-Haddad et al., 2021) the MCDM decision-making process encompasses four distinct analytical processes: descriptive analysis, predictive analysis (which is restrained to decision under uncertainty), normative analysis and prescriptive analysis. Descriptive models are focused on describing and predicting the behaviour of an individual. In MCDM, these models account for the unique learning history of the decision-maker, capturing their preferences, weighting the criteria or making comparative judgements. Normative models, on the other hand, are concerned with the perfectly rational human and attempt to improve the decision-making outcome through rationality. In addition, Prescriptive models provide a framework guiding the decision-maker to the optimal choice of action. Lastly, if there is uncertainty involved in the decision-making process, predictive analysis provides the necessary probabilities of events happening. For example, a predictive model states how likely, it would be to assume that a caterer provides quality food for an upcoming festival. Since, most decision- makers are experts in their respective fields, they have gathered experience on the optimal course of action and MADM methods can either incorporate it to a smaller or lager degree. A subjective MADM approach relies on the intuition and instinct of the decision-maker to find the preferred alternative, while an objective approach focuses on providing a framework that helps in identifying the best solution independent of the decision-makers experience (Bozorg-Haddad et al., 2021). Regardless of the choice of either an objective or subjective approach, due to their involvement the decision-makers and their unique learning history, they will always influence the decision-making outcome.
Since, MCDM methods originated from the field of operational research the psychological standpoint has been of less relevance, although undeniably important to any decision-making process. There are many questions concerning the use of MADM methods, like: can MADM methods assess and reflect the preferences of the decision-maker, and if so, do the MADM methods outperform the decisions of the decision-makers? Hence, this paper
Through the Psychological Lens: Unveiling Biases in Multi-Criteria Decision-Making 177
serves to provide an overview of MADM methods in light of psychological methodology, its challenges and ideas for further research. The structure of the paper is as follows: First, a short description of some of the most important methods, a general outlook on the problems in MCDM and at last, the MADM Methods are analysed based upon their employed methodology.
MADM METHODS Even though the procedure remains roughly the same, each method possesses its own unique assumptions, ideas and approaches to solving decision- making problems.
Analytical Hierachy Process (AHP) starts by defining the decision-making problem and expressing it through a set of criteria and alternatives in a hierarchical nature. At the top of the hierarchy stands the goal of the decision-maker; below are the criteria important to the goal; and at the bottom are the feasible alternatives. The idea of AHP is that the criteria can be best understood as single attribute decision-making problems because every alternative is evaluated with respect to one criterion at a time. Thus, providing a better overview of the decision-making problem at hand. Now, to find out which criterion is the most important in the decision-making process each criterion is compared head-to-head and rated on a 9-point scale. This procedure is called pairwise comparison and allows the decision- maker to determine the weights and the impact each criterion should have on the overall decision. The comparison of the alternatives in respect to the given criterion follows the same procedure (Saaty, 1990). If the pairwise comparison process takes a long time, AHP calls for revisions in order to see if judges have changed their minds drastically (Saaty, 1986).
ELECTRE I uses a two-step process to find a set of ideal solutions. First, in the step of aggregation, solutions which cannot be fully dominated by other alternatives are selected, while alternatives which can be dominated are removed. Meaning, that the alternatives have to perform at least as well or better in regard to some of the criteria when compared to each other. In the second step (Exploitation), the remaining alternatives are again compared and either are labeled: indifferent to each other, one is preferable to another, or no decisive statement about their relationship can be made and is thus incomparable. Through Exploitation a set of optimal solutions emerges from which the decision-maker can select the preferred alternative (Figueira et al., 2013). Again, these two steps make use of pairwise comparison to establish outranking relations. However, in ELECTRE there is no 9-point scale employed to state the dominance of one alternative over another. The method of pairwise comparison simply refers to the procedure of comparing two elements at a time.
PROMETHEE uses functions to assess the preferences of the decision- maker. These functions capture, in a pairwise fashion, the preference of an alternative with regard to each criterion in a dimensionless form [0;1]. The preference function possesses parameters that have to be calibrated by the decision-maker to make use of their knowledge. The extracted preferences are then multiplied by the weights of the criteria and summed up to assess the
overall performance of every alternative. However, the weighting procedure of the criteria is not clearly defined but rather just restricted by the condition that all weights have to sum up to 1 (Bozorg-Haddad et al., 2021). Thus, several methods can be employed to measure attribute importance. The point allocation method allows the decision-maker to distribute 100-points amongst all the criteria from which the weights can be derived directly. From the example of the introduction, Aesthetics could receive 50 points, price 30 and location 20. While the direct rating procedure lets the decision-maker rate the criteria importance on a scale from 0-100, and then needs to be standardized. There are more methods to assess these preferences, but these are some of the most common ones (Bozorg-Haddad et al., 2021).
The best worst method (BWM) follows the same procedure as AHP. However, instead of comparing every element with each other during pairwise comparison, the required number of comparisons is greatly reduced in BWM. Through the use of pairwise comparisons inconsistencies can be introduced into the decision-making process (Kuo & Chen, 2023). For example, an inconsistency would be: A is preferred B ∧is preferred C; However A is not preferred C
To address this issue, BWM establishes the best and the worst criterion and compares every criterion to these two. Thus, reducing the number of comparisons needed as well as the number of logical inconsistencies. Not only are the criteria compared in such a way, but also the attribute values of the alternatives. This is the distinguishing feature of BWM (Mi et al., 2019).
METHOD CRITIQUES The Validity of Preferences
To evaluate the performance of alternatives or the weights of criteria in MADM, the preferences of the decision-maker are approximated. As mentioned before, a variety of methods can be applied to measure the underlying preferences of the decision-maker. Methods like pairwise comparison, the point allocation or direct rating method make use of the decision-maker’s knowledge and are subsequently called subjective weighting methods (Odu, 2019). The theoretical foundation of these methods rests upon psychological latent variable models (LVM). These models assume that manifest behavior is an indicator of some latent variable, which is inaccessible to our empirical investigation (Borgstede & Eggert, 2023). The behavior as a manifest or observable variable is said to be related to its latent or not observable variable in some way. For example, if a participant ticks the box ‘strongly agree’ in a survey assumed to measure assertiveness, the tick as the manifest behavior is the indicator for some value of the measured attribute assertiveness. In the case of MADM, the assessing of preferences is also a latent variable model. MADM often uses some elicitation method to weight the criteria or it compares the alternatives concerning their performance using a 9-point scale. The data produced through these methods is said to be an indicator for the latent variable preferences.
Through the Psychological Lens: Unveiling Biases in Multi-Criteria Decision-Making 179
However, the concept of preferences remains vague and is often not clearly defined. In the context of a weighting procedure, the term preference can mean vastly different things. It can refer to the estimated importance of criteria because they are of interest to the stakeholders or because they are important to the decision-maker himself. In the first case, entirely different processes would be involved in estimating the importance to the stakeholders when compared to assessing one’s own preferences. The former involves some estimation of variables external to oneself, whereas the latter does not. To expand on this further, preference can be understood as the perceived value of an alternative inherent to the decision-maker when making a comparison or it could also be just the choice of an alternative over another. Arguably, depending on the concept, a different measurement procedure would be best suited to assess the preferences of the decision-maker. This problem is often ascribed to the appropriation of everyday language for scientific use (Borgstede & Leising, 2019). In everyday language, a single word can refer to different constructs. For instance, someone proficient in speaking many languages might be deemed intelligent, but intelligence can also be equated with a high IQ. Both examples might speak for prowess in cognitive ability, their meaning however is dependent on context. Similarly, the measurement of intelligence depends on the measurement instrument employed. The construct of intelligence is ultimately made up of different facets, and depending on the instrument used, the facets which are measured may vary. Hence, both context and the definition of our theoretical construct are important to the measuring procedure.
Looking at the validity of the concept of ‘attribute importance’ (which corresponds to the criteria weights), it becomes increasingly apparent that a clear definition of the construct ‘preferences’ is lacking. Van Ittersum et al. (2007) conducted research on the nomological and convergent validity of different methods to measure attribute importance as employed in MADM. Validity is a key concept used in psychology to measure how far the employed instrument actually measures what it should measure. As latent variables are inaccessible to our empirical investigation, it is necessary to prove that we are truly measuring the defined construct. To assess the convergent validity, tests which should measure theoretically similar constructs are correlated. While the nomological validity measures to what degree the causal relations between theoretically similar constructs are one and the same. Even though the measurement of attribute importance should produce similar results irrespective of the instrument used, results vary between instruments. Suggesting that the different weighting techniques measure different constructs and produce different weights (Van Ittersum et al., 2007). Van Ittersum et al. (2007) propose that the different methods employed measure different aspects of attribute importance. But this perspective cannot explain the lack of validity in all cases.
Consistency A formal challenge is the consistency of judgments. Even when a decision- maker expresses a preference for A over B and B over C, it does not
necessarily mean that he will consequently prefer A over C.
If A ≻B ∧B ≻C then A ≻C However, sometimes under empirical investigation:
A ≻B ∧B ≻C; A ≺C
There are a few possible explanations for why this could be the case. First, logical inconsistencies can be introduced through pairwise comparison because the process of discriminating between two stimuli may not be deterministic and the perception of stimuli varies (Thurstone, 1994). Thurstone (1994) proposed a potential solution to this problem with his law of comparative judgment. Decision-makers are presented with alternatives or criteria and are asked to judge them as better, worse, or equivalent. To account for variance in perception and ensure consistent judgments, it is necessary to evaluate the same element multiple times. After making judgments, the mean frequency with which they preferred an alternative over another is assessed. Using the mean frequencies, a scale is constructed depicting the distance between alternatives or criteria. Thus, creating a ranking, which also shows the degree of preference through the distance between datapoints. Subsequently, there should be no logical inconsistencies, and a definitive ranking of alternatives is given to any MADM problem. As some of the criteria involved in MADM are not easily quantifiable, Thurstone (Thurstone, 1927) also provides an example of how a scale can be constructed in the case of social values. In his application of the law of comparative judgement, Thurstone evaluates the seriousness of crimes. The fundamental premise is that, if a crime A is compared to crime B and rated worse by 90% of the judges, while crime B compared to crime C is considered worse 50% of the time, then the distance between A and B should be greater than that between B and C. As aforementioned, the distance between the compared alternatives is expressed through the frequency of preference. The measurement of perceived differences between two stimuli can be done on a group or individual level (Thurstone, 1994).
Another possible explanation for the occurrence of logical inconsistencies is that the stimulus C in the example may possess properties that have not been accounted for, thereby making it preferable over stimulus A. The decision-maker might not fully understand all the factors influencing their decision. Thus, it might be beneficial to observe the decision-makers behavior in a choice experiment, where stimulus properties are systematically varied, and the choice by the decision-maker is measured.
Findings suggest, that in AHP, logical consistency are commonly introduced through its synthesis procedure (inclusion of irrelevant alternatives), its normalization procedure, criteria weights (e.g. all alternatives are equally preferred in regard to one criterion), misuse of the method, the uncertainty of the decision-making process or structural dependency between the criteria and alternatives (Borgstede & Eggert, 2023). It also should be noted that while there have been various attempts to measure the degree of inconsistency, there have also been attempts to limit the
Through the Psychological Lens: Unveiling Biases in Multi-Criteria Decision-Making 181
inconsistencies by making fewer comparisons, as in the Best-Worst-Method or revising the judgements as in AHP (Mi et al., 2019; Rezaei, 2015; Saaty, 1986, 1990). Although, the BWM introduces greater consistency into the decision-making process in relation to the law of comparative judgment, it may decrease the accuracy of the results as it will inevitably provide less information about the preferences of the decision-maker. Additionally, a consistent pairwise comparison matrix does not automatically entail the validity of expert judgments (Kuo & Chen, 2023).
Expertise and Experts Often, MADM methods try to consider the experience of an expert in hopes of achieving better results. However, there is no agreement on the definition of an expert (Day, 2002; Weinstein, 1993). It is a difficult endeavor to define what an expert really is and every field of study has its own perspective (Hill & Ready-Campbell, 2011; van Dijk et al., 2020). This makes it hard for MADM methods to identify an expert decision-maker and to make use of his experience. There are various approaches which can be used to identify experts (Caley et al., 2014; Shanteau et al., 2002). However, these approaches rarely find application in MADM methods to identify if the decision-maker is really an expert (Franek & Kashi, 2014; Keshavarz Ghorabaee et al., 2017). Even if the decision-maker is an expert, the question still remains: whether the assessment made by decision-makers outperforms that of a non- expert. Additionally, one should also consider the bias that experts might also have, such as outcome bias which experienced pilots can also be prone to (Martins et al., 2023). According to (Hogarth et al., 2015) this is highly dependent on the context. The inherent reliability of the decision-making environment makes his judgment more or less accurate. If future events can be predicted through the use of past data, this makes an ‘expert’ judgement more reliable. If this is not the case, non-experts may perform just as well as experts (Green & Armstrong, 2007).
Furthermore, the expert does not necessarily use more information than the layman to inform his decision making process, but he does weigh criteria differently when compared to the layman (Shanteau, 1992). Thus, it is not necessary to include many decision-making criteria to make use of the decision-makers experience.
More importantly, there is a debate concerning if MADM methods are even able to make use of the decision-makers experience. Harries and Harvey (Harries & Harvey, 2000) argue that participants’ judgments seems more reflective of how the criteria ought to have been used rather than how they actually have been used. Hence, it becomes questionable if MADM methods can effectively utilize the acquired experience of the decision- maker. Nevertheless, (Riquelme, 2001) demonstrated that humans should be able to articulate their judgment policies. In an experiment, subjects were asked to give a holistic judgement on their intention to buy a mobile phone plan and subsequently had to weight the importance of the criteria employed to describe the mobile phone plan. Weights derived from holistic judgments were compared with the weights stated by the subjects after the
trial. The level of correlation between derived and self-explicated weights suggests that humans can accurately describe their own judgment policies. However, certain methods do not even lay claim to incorporate the experience of the decision-maker and just try to account for the wishes and interests of the stakeholders. In this case, criteria do not necessarily need to reflect the most optimal properties according to the decision-maker, but rather represent properties the stakeholders deem to be important. These methods then systematically eliminate alternatives until one is found which best fits the optimal outcome for the stakeholders’ interest and wishes (Bozorg-Haddad et al., 2021).
CONCLUSION
In conclusion, Multi-Attribute Decision-Making (MADM) methods offer powerful tools for tackling complex decision-making problems, but their effectiveness and reliability are subject to various challenges, especially when evaluated from a psychological standpoint. The paper highlighted the key steps involved in MADM processes, such as defining the problem, listing alternatives and criteria, weighting criteria, comparing alternatives, evaluating performance, and selecting the preferred option. Several MADM methods were examined, including Analytical Hierarchy Process (AHP), Best Worst Method (BWM), ELECTRE, and PROMETHEE. The analysis focused on identifying general problems inherent in MADM approaches. These include the validity of preferences, consistency in judgements and the role of expertise and experts. Each of these issues presents significant challenges that must be addressed to ensure the reliability and effectiveness of MADM processes.
Looking ahead, future research should focus on addressing these challenges and exploring new avenues for improvement. This includes refining measurement techniques for assessing preferences, enhancing our knowledge of preferences or attribute importance, and better understanding the role of expertise in decision-making. Especially, further investigation into pairwise comparison methods can contribute to advancing the field and improving its applicability in real-world decision-making scenarios.
Overall, while MADM methods hold great potential for aiding decision- makers in navigating complex choices, ongoing research and refinement are necessary to fully realize their benefits and overcome their inherent challenges. By addressing these issues, MADM can continue to evolve as a valuable tool for decision-making in various domains.
REFERENCES
Borgstede, M., & Eggert, F. (2023). Squaring the circle: From latent variables to theory-based measurement. Theory & Psychology, 33(1), 118–137. https://doi.or g/10.1177/09593543221127985 Borgstede, M., & Leising, D. (2019). Hypothetical Constructs. https://doi.org/10.
1007/978-3-319-28099-8_679-1
Through the Psychological Lens: Unveiling Biases in Multi-Criteria Decision-Making 183
Bozorg-Haddad, O., Loaiciga, H., & Zolghadr-Asli, B. (2021). A Handbook on Multi-Attribute Decision-Making Methods. https://doi.org/10.1002/ 9781119563501 Caley, M. J., O’Leary, R. A., Fisher, R., Low-Choy, S., Johnson, S., & Mengersen, K.
(2014). What is an expert? A systems perspective on expertise. Ecology and Evolution, 4(3), 231–242. https://doi.org/10.1002/ece3.926 Chakraborty, S., Raut, R. D., Rofin, T. M., & Chakraborty, S. (2023). A comprehensive and systematic review of multi-criteria decision-making methods and applications in healthcare. Healthcare Analytics, 4, 100232. https://doi.org/ 10.1016/j.health.2023.100232 Day, J. (2002). What is an expert? Radiography, 8(2), 63–70. https://doi.org/10.
1053/radi.2002.0369 Emovon, I., & Oghenenyerovwho, O. S. (2020). Application of MCDM method in material selection for optimal design: A review. Results in Materials, 7, 100115. https://doi.org/10.1016/j.rinma.2020.100115 Figueira, J., Greco, S., Roy, B., & Słowi´nski, R. (2013). An overview of ELECTRE
methods and their recent extensions. Journal of Multi-Criteria Decision Analysis, 20. https://doi.org/10.1002/mcda.1482 Franek, J., & Kashi, K. (2014). A Review and Critique of MADM Methods and Applications in Business and Management. International Journal of the Analytic Hierarchy Process, 6. https://doi.org/10.13033/ijahp.v6i2.254 Green, K., & Armstrong, J. (2007). The Ombudsman: Value of Expertise for
Forecasting Decisions in Conflicts. Interfaces, 37, 287–299. https://doi.org/10. 1287/inte.1060.0262 Harries, C., & Harvey, N. (2000). Taking advice, using information and knowing what you are doing. Acta Psychologica, 104(3), 399–416. https://doi.org/10.1016/s0001-6918(00)00038-x Hill, S., & Ready-Campbell, N. (2011). Expert Stock Picker: The Wisdom of (Experts in) Crowds. International Journal of Electronic Commerce, 15(3), 73–102. https: //doi.org/10.2753/JEC1086-4415150304 Hogarth, R., Lejarraga, T., & Soyer, E. (2015). The Two Settings of Kind and
Wicked Learning Environments. Current Directions in Psychological Science, 24, 379–385. https://doi.org/10.1177/0963721415591878 Keshavarz Ghorabaee, M., Amiri, M., Zavadskas, E. K., & Antucheviciene, J. (2017).
Supplier evaluation and selection in fuzzy environments: A review of MADM approaches. Economic Research-Ekonomska Istraživanja, 30(1), 1073–1118. https://doi.org/10.1080/1331677X.2017.1314828 Kuo, T., & Chen, M.-H. (2023). On using pairwise comparison in the analytic hierarchy process: Validity is goal while consistency is means. Inf. Sci., 648(C). https://doi.org/10.1016/j.ins.2023.119630 Martins, A. P. G., Köbrich, M. V., Carstengerdes, N., & Biella, M. (2023). All’s well
that ends well? Outcome bias in pilots during instrument flight rules. Applied Cognitive Psychology, 37(2), 433–442. https://doi.org/10.1002/acp.4046 Mi, X., Tang, M., Liao, H., Shen, W., & Lev, B. (2019). The state-of-the-art survey on integrations and applications of the best worst method in decision making: Why, what, what for and what’s next? Omega, 87, 205–225. https://doi.org/10.1016/j. omega.2019.01.009 Odu, G. (2019). Weighting methods for multi-criteria decision making technique.
Journal of Applied Sciences and Environmental Management, 23, 1449. https: //doi.org/10.4314/jasem.v23i8.7
Rezaei, J. (2015). Best-worst multi-criteria decision-making method. Omega, 53, 49–57. https://doi.org/10.1016/j.omega.2014.11.009 Riquelme, H. (2001). Do consumers know what they want? Journal of Consumer
Marketing, 18(5), 437–448. https://doi.org/10.1108/07363760110398772 Saaty, T. L. (1986). Axiomatic Foundation of the Analytic Hierarchy Process.
Management Science, 32(7), 841–855. Saaty, T. L. (1990). How to make a decision: The analytic hierarchy process. European Journal of Operational Research, 48(1), 9–26. https://doi.org/10.1016/0377-2217(90)90057-I Shanteau, J. (1992). How much information does an expert use? Is it relevant? Acta Psychologica, 81(1), 75–86. https://doi.org/10.1016/0001-6918(92)90012-3 Shanteau, J., Weiss, D. J., Thomas, R. P., & Pounds, J. C. (2002). Performance-based assessment of expertise: How to decide if someone is an expert or not. European Journal of Operational Research, 136(2), 253–263. https://doi.org/10.1016/S0377-2217(01)00113-8 Thurstone, L. L. (1927). The method of paired comparisons for social values. The
Journal of Abnormal and Social Psychology, 21(4), 384–400. https://doi.org/10. 1037/h0065439 Thurstone, L. L. (1994). A law of comparative judgment. Psychological Review, 101(2), 266–270. https://doi.org/10.1037/0033-295X.101.2.266 Tzeng, G.-H., & Huang, J.-J. (2011). Multiple Attribute Decision Making: Methods
and Applications. Chapman and Hall/CRC. https://doi.org/10.1201/b11032 van Dijk, E. E., van Tartwijk, J., van der Schaaf, M. F., & Kluijtmans, M. (2020).
What makes an expert university teacher? A systematic review and synthesis of frameworks for teacher expertise in higher education. Educational Research Review, 31, 100365. https://doi.org/10.1016/j.edurev.2020.100365 Van Ittersum, K., Pennings, J. M. E., Wansink, B., & van Trijp, H. C. M. (2007).
The validity of attribute-importance measurement: A review. Journal of Business Research, 60(11), 1177–1190. https://doi.org/10.1016/j.jbusres.2007.04.001 Weinstein, B. D. (1993). What is an expert? Theoretical Medicine, 14(1), 57–73.
https://doi.org/10.1007/BF00993988 Zwiegelaar, J., & Rahbarimanesh, A. (2022). Application of Multi Criteria Decision Making (MCDM) to Analyse the Impact of External Environment on Innovation Ecosystem. International Conference on Information Systems 2022 Special Interest Group on Big Data Proceedings. https://aisel.aisnet.org/sigbd2022/3