Analytic Hierarchy Process (AHP) | Quantifying Evaluation Criteria Through Pairwise Comparisons

The Analytic Hierarchy Process (AHP) is a mathematical decision-making method that decomposes complex decisions involving multiple criteria into a hierarchical structure. Through pairwise comparisons, it determines the relative importance of evaluation criteria and produces an overall assessment of competing alternatives.

Developed by Thomas L. Saaty in the 1970s, AHP has been applied to a wide range of problems, including business strategy, R&D project selection, capital investment, policy evaluation, and location selection.

When making decisions, we usually consider several factors simultaneously. For example, choosing a rental apartment may involve rent, distance from the nearest station, floor space, neighborhood conditions, and available facilities. These factors do not necessarily have equal importance. Moreover, different apartments have different advantages and disadvantages. One may be conveniently located but expensive, while another may be spacious but poorly connected to public transportation.

AHP addresses such complex judgments by breaking them down into relatively simple questions: “Which of these two factors is more important, and by how much?” By repeating these comparisons, the method calculates the relative weights of evaluation criteria and combines them to assess alternatives.

A distinctive feature of AHP is that it does not attempt to eliminate subjective judgment. Instead, it converts subjective judgments into explicit numerical values and provides a way to examine their internal consistency.

However, a mathematically calculated evaluation should not automatically be regarded as an objectively correct decision.

Basic Principles of AHP

AHP typically organizes a decision problem into three hierarchical levels: the overall goal, evaluation criteria, and alternatives.

For example, in selecting a rental apartment, the overall goal might be “Choose the apartment that best meets my needs.”

Below this goal, evaluation criteria such as commuting convenience, rent, and living environment are defined. The alternatives—Apartments A, B, and C—are placed at the lowest level.

For more complex problems, criteria may be divided into additional subcriteria. For instance, “Living Environment” might be divided into floor space, natural lighting, noise levels, and available facilities.

This hierarchical decomposition serves a purpose beyond simplifying calculations. It helps decision-makers clarify what they value, identify potentially overlooked factors, and understand how different considerations contribute to the final decision.

AHP then compares elements at the same hierarchical level in pairs, with reference to their parent element.

For example:

  • How much more important is commuting convenience than rent?
  • How much more important is rent than the living environment?
  • How much more important is commuting convenience than the living environment?

The results are processed mathematically to derive the relative importance of each criterion.

Deriving Weights Through Pairwise Comparisons

AHP commonly uses Saaty’s Fundamental Scale, which assigns numerical values from 1 to 9 to judgments of relative importance.

ValueMeaning
1Equal importance
3Moderate importance
5Strong importance
7Very strong importance
9Extreme importance
2, 4, 6, 8Intermediate judgments

For example, if criterion A is judged to be strongly more important than criterion B, the comparison is assigned a value of 5. The reverse comparison receives a value of 1/5.

Suppose we have three evaluation criteria for choosing an apartment: commuting convenience, rent, and living environment.

The pairwise comparison matrix might be:

CommutingRentLiving Environment
Commuting126
Rent1/213
Living Environment1/61/31

In this example, commuting convenience is considered twice as important as rent and six times as important as the living environment. Rent is considered three times as important as the living environment.

All diagonal elements of the matrix are 1, and elements on opposite sides of the diagonal are reciprocals. Such a matrix is called a reciprocal matrix.

In the conventional eigenvector method of AHP, the principal eigenvector of this matrix is calculated and normalized so that its elements sum to 1.

The calculation is expressed as:

Aw = λmax w

Here, A represents the pairwise comparison matrix, w is the vector of criterion weights, and λmax is the maximum eigenvalue.

For the matrix above, the resulting weights are:

  • Commuting convenience: 0.60
  • Rent: 0.30
  • Living environment: 0.10

Thus, commuting convenience accounts for 60% of the relative weighting, rent for 30%, and the living environment for 10%.

Another approach is to calculate weights using the geometric mean of each row. The geometric mean method is computationally straightforward and has been widely studied in pairwise comparison-based decision analysis.

However, when pairwise judgments are inconsistent, the eigenvector and geometric mean methods may produce somewhat different weights.

Evaluating the Consistency of Judgments

An important feature of AHP is its ability to assess the consistency of pairwise comparisons.

Suppose a decision-maker judges that criterion A is twice as important as criterion B, and B is three times as important as criterion C.

Under perfectly consistent multiplicative judgments, A should be six times as important as C.

However, when directly comparing A with C, the same person might assign a value of 4 rather than 6.

Such inconsistencies naturally arise when people make multiple comparisons.

AHP measures consistency using the Consistency Index (CI), calculated from the maximum eigenvalue of the comparison matrix.

CI = (λmax − n) / (n − 1)

Here, n is the number of elements being compared.

For a perfectly consistent matrix, λmax equals n, resulting in CI = 0.

AHP also uses the Random Index (RI), which represents the average consistency index of randomly generated reciprocal comparison matrices of the same size.

The Consistency Ratio (CR) is calculated as:

CR = CI / RI

A CR of 0.10 or less is commonly used as a practical guideline for acceptable consistency. However, this threshold does not guarantee that the underlying judgments are correct. For matrices with two or fewer elements, the usual CR-based consistency assessment is not meaningful.

When CR is relatively high, users are generally encouraged to reconsider their comparisons.

Nevertheless, inconsistency is not always undesirable. It may indicate that the relationships among criteria are not fully understood or that the decision-maker is uncertain about their preferences.

The purpose should not be merely to improve the numerical consistency of the matrix, but to understand why inconsistencies have occurred.

Most importantly, perfectly consistent judgments can still be based on inappropriate assumptions. Even when CR equals zero, the resulting decision may be misleading if the original evaluation framework is flawed.

Example: Choosing a Rental Apartment

Let us return to the apartment selection example.

Suppose Apartments A, B, and C are compared under the three criteria discussed above.

Pairwise comparisons of the apartments under each criterion produce the following local priorities.

CriterionWeightApartment AApartment BApartment C
Commuting convenience0.600.600.300.10
Rent0.300.200.500.30
Living environment0.100.400.400.20

For each criterion, the local priorities of Apartments A, B, and C sum to 1.

In the conventional additive aggregation approach, the overall score is calculated by multiplying the local priority under each criterion by the corresponding criterion weight and summing the products.

Overall Score = Σ (Criterion Weight × Local Priority)

For Apartment A:

0.60 × 0.60 + 0.30 × 0.20 + 0.10 × 0.40 = 0.46

The overall results are:

ApartmentOverall ScoreRank
Apartment A0.461
Apartment B0.372
Apartment C0.173

Apartment A receives the highest overall score.

Because commuting convenience has the greatest weight, Apartment A can rank first despite receiving a relatively low score for rent.

However, this raises an important question.

What if the rent for Apartment A substantially exceeds the decision-maker’s maximum budget?

Under a simple weighted-sum model, the disadvantage of excessive rent may be compensated for by superior commuting convenience.

Consequently, Apartment A might still receive the highest overall score, even though it is not a feasible choice.

This illustrates a fundamental limitation of additive evaluation models.

Limitations of AHP: Is a Weighted Sum Really Appropriate?

Conventional AHP commonly combines criterion weights and alternative priorities through an additive aggregation model.

This approach makes decision-making relatively straightforward, but it has important limitations, particularly when criteria interact or when certain conditions cannot be compensated for by other advantages.

Consider two factors involved in apartment selection: distance from the nearest station and commuting frequency.

For someone who commutes to an office every day, proximity to a station may be extremely important. For someone who works entirely from home, it may be much less important.

In other words, the importance of station proximity depends on another condition: commuting frequency.

Similarly, satisfaction with rent does not necessarily change linearly.

A small increase in rent may have little effect within an affordable range. However, once a strict budget limit is exceeded, an apartment may become unacceptable regardless of its other advantages.

At least three distinct problems need to be considered.

Nonlinear Utility

The relationship between an evaluation criterion and the satisfaction it produces is not necessarily linear.

For example, the difference between living five and ten minutes from a station may be substantial, while the difference between 25 and 30 minutes may be less important to some people.

A nonlinear utility function can represent such preferences more appropriately than a simple linear scale.

Interactions Between Criteria

The importance or effect of one criterion may depend on another.

For example, proximity to a station becomes more valuable as commuting frequency increases.

The combined effect cannot always be represented adequately by treating both criteria as independent contributors to an additive score.

Non-Compensatory Conditions

Some requirements cannot be offset by advantages elsewhere.

A strict budget limit, a mandatory safety requirement, or a legal restriction may disqualify an alternative regardless of its overall weighted score.

Such requirements must be represented explicitly rather than treated as ordinary criteria that can be compensated for.

AHP is a valuable method for quantifying relative importance, but conventional additive aggregation cannot fully address all these situations.

This does not mean AHP is entirely linear. The process of deriving weights from pairwise comparisons involves eigenvector calculations and is not itself a simple linear combination. The principal limitations concern the final aggregation of priorities and the assumptions underlying the evaluation model.

Extensions and Alternatives to AHP

Various extensions and related multi-criteria decision-making methods have been developed to address limitations associated with conventional AHP.

ANP: Modeling Dependencies Among Criteria

The Analytic Network Process (ANP) was developed by Thomas Saaty as an extension of AHP.

While AHP generally assumes a hierarchical structure, ANP allows dependencies and feedback relationships among elements within a network.

For example, when evaluating R&D projects, technical feasibility, market potential, development costs, and competitive advantage may influence one another.

Low technical feasibility may increase development costs, while higher development costs may reduce commercial competitiveness.

ANP provides a framework for determining priorities while considering such dependencies.

However, ANP does not automatically provide a general solution for arbitrary nonlinear utility functions or complex preference interactions.

Dependencies among evaluation elements and nonlinear relationships among preferences are related but distinct modeling problems.

Nonlinear Utility Functions

Multi-Attribute Utility Theory (MAUT) and related approaches can represent nonlinear relationships between criterion values and utility.

For example, an increase in monthly rent from $1,000 to $1,100 may have a different perceived impact from an increase from $2,000 to $2,100.

A nonlinear utility function can reflect these differences.

However, even when each criterion is transformed through a nonlinear utility function, combining the resulting utilities through a simple weighted sum does not automatically capture interactions between criteria.

Choquet Integrals: Modeling Interactions

The Choquet integral is a non-additive aggregation method that can account for synergy, redundancy, and other interactions among criteria.

For example, technical expertise and manufacturing capabilities may create a greater competitive advantage when combined than when evaluated independently.

A Choquet integral can represent such relationships through a non-additive measure of the importance of criterion combinations.

However, this flexibility introduces additional parameters. As the number of criteria increases, specifying and interpreting these interactions can become difficult.

Mandatory Conditions and Sensitivity Analysis

In practical decision-making, relatively simple extensions may be sufficient.

For apartment selection, users might first eliminate alternatives that exceed a strict rental budget and then apply AHP to the remaining candidates.

They can also vary criterion weights and examine how the rankings change.

If a small change in the importance of commuting convenience causes the preferred apartment to change, the decision is sensitive to that particular judgment.

Such analysis helps clarify how strongly the final result depends on the assumptions used in the evaluation.

The Problem of Rank Reversal

AHP has also been associated with a long-standing debate concerning rank reversal.

Rank reversal occurs when adding or removing an alternative changes the relative ranking of alternatives that were already being evaluated.

For example, suppose Apartment A ranks above Apartment B. If a third apartment, C, is introduced and is very similar to A, the relative ranking of A and B may change under certain aggregation procedures.

One reason is that local priorities are often normalized so that the scores of all alternatives under a criterion sum to 1.

Adding or removing alternatives can therefore change the relative values assigned to existing candidates.

Rank reversal is not necessarily irrational in every situation. A new alternative may change the decision context or the meaning of relative comparison.

However, in other situations, decision-makers may reasonably expect the ranking between existing alternatives to remain unchanged when their characteristics have not changed.

Different AHP synthesis approaches, including Distributive Mode and Ideal Mode, have been developed and discussed in relation to this issue.

The important question is not simply whether rank reversal occurs, but what the rankings are intended to represent and whether the aggregation method is appropriate for the decision.

Evaluating R&D and New Business Projects

AHP can be applied to the selection and prioritization of research and development projects and new business opportunities.

Possible evaluation criteria include technical feasibility, market potential, novelty, compatibility with existing technological capabilities, development time, and required investment.

By comparing these criteria pairwise, decision-makers can make explicit which considerations are most important to their R&D strategy.

However, dependencies and interactions among criteria require careful attention.

For example, highly novel research projects may involve greater technical uncertainty. If novelty and technical feasibility are evaluated independently through a simple weighted sum, particularly ambitious projects may be disadvantaged.

Similarly, projects intended to generate short-term commercial returns and projects designed to build long-term technological capabilities may require different evaluation frameworks.

This raises a more fundamental question: should all R&D projects be ranked using a single set of criteria and weights?

In some cases, separating projects into categories or applying different strategic objectives may be more appropriate than forcing all projects into one ranking.

The value of AHP therefore extends beyond selecting the project with the highest numerical score.

It can also provide a structured process through which stakeholders discuss priorities, clarify strategic assumptions, and examine differences in judgment.

Applications to Personnel Evaluation and Admissions

AHP can also be applied to decisions involving multiple evaluation criteria, such as personnel assessment and university admissions.

For example, criteria might include professional expertise, achievements, collaboration skills, problem-solving ability, and future potential.

Pairwise comparisons could be used to establish the relative importance of these criteria.

However, applications involving people require particular caution.

If evaluation criteria are poorly defined, numerical comparisons may simply reproduce existing subjective biases. If certain criteria are assigned inappropriate importance, even a highly consistent AHP model may produce unfair outcomes.

Mathematical consistency does not establish ethical legitimacy or fairness.

In personnel evaluation and admissions, the validity of criteria, reliability of measurements, fairness across individuals and groups, and transparency of decision-making are essential considerations.

AHP may help make the evaluation process more explicit, but it cannot automatically guarantee that the resulting judgments are fair or justified.

AHP in the Age of AI

Large Language Models (LLMs) create new possibilities for applying AHP.

Traditionally, users needed to define evaluation criteria, identify alternatives, and complete numerous pairwise comparisons themselves.

These preparatory activities can require considerable knowledge and effort.

With LLM assistance, users can describe a decision problem in natural language and receive suggestions for relevant evaluation criteria and alternatives.

For example, a user might ask:

“How should we prioritize our company’s potential R&D projects?”

An LLM could suggest criteria such as technical feasibility, market potential, novelty, compatibility with existing technologies, and potential synergies.

It could also help identify overlapping criteria or important factors that may have been overlooked.

However, allowing an LLM to automatically determine all pairwise comparison values introduces a significant concern.

The importance of different criteria depends on the decision-maker’s objectives, organizational strategy, and values.

Weights generated from general knowledge should not automatically be treated as representing the user’s preferences.

A more appropriate division of responsibilities is to let the LLM assist with identifying criteria and alternatives, use a mathematical engine to calculate weights and consistency measures, and leave the final value judgments to the user.

LLMs may also support critical examination of evaluation results.

For example, if a particular R&D project receives the highest overall score, an LLM might raise questions such as:

  • Are market potential and expected profitability being counted twice?
  • Has the relationship between technical feasibility and development time been considered?
  • Have mandatory requirements or critical constraints been overlooked?
  • Would the ranking change significantly under different assumptions?

Users can then revise the evaluation model and recalculate the results using a dedicated mathematical engine.

This approach combines the flexibility of AI-assisted reasoning with the transparency and reproducibility of structured mathematical analysis.

AHP in ThinkNavi

ThinkNavi approaches AHP not merely as a method for ranking alternatives, but as a tool for examining evaluation criteria, relative importance, judgment consistency, and the validity of decision-making assumptions.

ThinkNavi Free is planned to provide tools that allow users to enter criteria and alternatives manually, calculate weights through pairwise comparisons, evaluate alternatives, and examine consistency.

The planned integration with decision matrices will also support mandatory conditions, nonlinear utility functions, adjustments to criterion weights, and sensitivity analysis.

These capabilities will enable users to explore how changes in the evaluation model influence their conclusions.

In the LLM-assisted version, additional functions may include suggestions for evaluation criteria and alternatives, information retrieval, identification of problematic assumptions, and critical examination of evaluation results.

The broader ConceptMiner product family may also integrate AHP and related decision-making methods to evaluate hypotheses, R&D themes, and business opportunities generated through concept exploration.

AHP makes it possible to break complex judgments into relatively simple pairwise comparisons and derive quantitative results through a systematic mathematical process.

However, those results depend on the selected criteria, assigned weights, evaluated alternatives, and aggregation procedures.

The goal should not be to automatically accept the alternative with the highest calculated score. It should be to understand how decisions are made, examine the assumptions behind them, and revise the evaluation framework when necessary.

AHP is one of the most established methods for connecting subjective human judgment with mathematical analysis. Combined with computers and AI, it offers further opportunities to make decision-making more transparent, reflective, and systematic.


Related Thinking Methods:

References and Further Reading

  • Thomas L. Saaty, The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation, McGraw-Hill, 1980.
  • Thomas L. Saaty, “Axiomatic Foundation of the Analytic Hierarchy Process,” Management Science, 32(7), pp. 841–855, 1986.
  • Thomas L. Saaty, “How to Make a Decision: The Analytic Hierarchy Process,” Interfaces, 24(6), pp. 19–43, 1994.
  • Thomas L. Saaty, “Decision Making — The Analytic Hierarchy and Network Processes (AHP/ANP),” Journal of Systems Science and Systems Engineering, 13(1), pp. 1–35, 2004.
  • Ido Millet and Thomas L. Saaty, “On the Relativity of Relative Measures — Accommodating Both Rank Preservation and Rank Reversals in the AHP,” European Journal of Operational Research, 121(1), pp. 205–212, 2000.
  • INFORMS — How to Make a Decision: The Analytic Hierarchy Process

.