When we judge whether two objects are similar, we compare their characteristics. For products, we might examine price, performance, dimensions, and applications. For companies, we might compare revenue, profitability, R&D expenditure, and technological capabilities.
It seems reasonable to assume that adding more characteristics will provide a more detailed understanding of objects and improve the accuracy of classification and decision-making.
However, increasing the number of features does not necessarily improve analytical accuracy.
In multidimensional data, increasing the number of variables can cause the size of the data space, distances between points, and the distribution of information to behave in ways that differ considerably from our everyday intuition.
For example, under certain conditions, distances between data points become relatively similar as dimensionality increases. This phenomenon is known as Distance Concentration.
When the relative difference between the nearest and farthest points becomes small, classification and similarity judgments based on distance may become less informative.
Another surprising phenomenon occurs in high-dimensional spheres: most of the volume is concentrated in a thin region near the surface. This illustrates how high-dimensional geometry differs fundamentally from the geometry of two or three dimensions.
These phenomena are related to a broader family of problems commonly discussed under the term Curse of Dimensionality.
However, the Curse of Dimensionality, concentration of volume near the surface of a sphere, and Distance Concentration are not identical phenomena. Nor does high dimensionality necessarily make distance-based analysis or classification meaningless.
The important distinction is that increasing the number of features is not the same as improving our ability to identify meaningful differences among objects.
This issue is relevant not only to Machine Learning but also to Statistics, Multivariate Analysis, Optimization, Control Theory, scientific modeling, and philosophical questions concerning human cognition and Concept Formation.
What Is Dimensionality?
In data analysis, dimensionality usually refers to the number of variables or coordinates used to represent an object.
For example, representing a product by its price and weight produces two-dimensional data.
Adding durability produces a three-dimensional representation.
Including power consumption, physical dimensions, operating speed, temperature characteristics, material composition, and other properties produces a higher-dimensional representation.
In general, an object (x) represented by (d) variables can be expressed as a vector:
$$
x=(x_1,x_2,\ldots,x_d)
$$
This is a (d)-dimensional vector.
Two-dimensional and three-dimensional data can be represented visually, but directly visualizing data with tens, hundreds, or thousands of dimensions is difficult for humans.
Nevertheless, high-dimensional data are common.
Images contain large numbers of pixel values. Spectroscopic measurements provide observations across many wavelengths. Text embeddings may represent sentences and documents as vectors containing hundreds or thousands of numerical components.
However, the number of recorded variables is not necessarily equal to the number of independent degrees of freedom present in the data.
This distinction becomes important when considering Intrinsic Dimension and Effective Dimension.
What Is the Curse of Dimensionality?
The Curse of Dimensionality broadly refers to problems in which increasing the number of dimensions causes computational requirements, data requirements, or estimation difficulties to grow rapidly.
The term originated in the work of mathematician Richard Bellman on Dynamic Programming.
In his 1957 book Dynamic Programming and subsequent research, Bellman examined the computational difficulties associated with optimization problems involving many state variables.
Consider a variable that can take ten possible values.
With one variable, there are ten possible states.
With two variables, there are 100 combinations.
With three variables, there are 1,000 combinations.
More generally, if each of (d) variables is divided into (m) possible values, the total number of combinations is:
$$
N=m^d
$$
| Number of Dimensions | Combinations with 10 Values per Dimension |
|---|---|
| 1 | 10 |
| 2 | 100 |
| 3 | 1,000 |
| 5 | 100,000 |
| 6 | 1,000,000 |
| 10 | 10,000,000,000 |
With ten dimensions, the number of combinations reaches ten billion.
This example assumes that all combinations are explicitly enumerated. Not every high-dimensional problem requires such computation.
Nevertheless, it illustrates how rapidly state spaces and search spaces can expand as the number of dimensions increases.
The difficulty is relevant not only to Dynamic Programming and Optimization but also to statistical estimation, Design of Experiments, numerical computation, and data analysis.
The Curse of Dimensionality Includes Several Different Problems
The Curse of Dimensionality does not refer to a single mathematical phenomenon.
It encompasses several distinct difficulties.
| Problem | What Happens |
|---|---|
| Combinatorial explosion | The number of possible states or combinations increases rapidly |
| Data sparsity | Observations become sparse relative to the size of the data space |
| Estimation difficulty | Insufficient information is available to estimate numerous parameters reliably |
| Distance Concentration | Relative differences among distances may become smaller |
| Accumulation of noise | Irrelevant variables may dominate distances and judgments |
| Computational complexity | Searching, estimating, and optimizing may require substantial resources |
These problems are related, but they do not occur under identical conditions or have identical solutions.
For example, a state space may become computationally difficult to search even when Distance Concentration is not present.
Conversely, a dataset may exhibit poor distance discrimination even when sufficient computational resources are available.
It is therefore misleading to treat high-dimensional data as inherently unusable.
The first step is to identify which problem is actually occurring.
Why Data Become Sparse in High-Dimensional Spaces
One common difficulty associated with increasing dimensionality is Data Sparsity.
Consider two variables, each ranging from zero to one. Their values can be represented within a square.
Dividing each axis into ten intervals produces 100 cells.
Applying the same division to a ten-dimensional space produces ten billion cells.
Even with one million observations, it would be impossible to populate every cell with sufficient data.
This illustrates the difficulty of obtaining dense coverage when data are distributed broadly across a high-dimensional space.
However, real datasets do not necessarily fill their entire ambient space.
When variables are strongly correlated, the actual degrees of freedom may be much smaller than the number of recorded variables.
A dataset with 1,000 variables does not necessarily require observations covering every possible combination in a 1,000-dimensional space.
This distinction is one reason why identifying underlying structure is so important in high-dimensional analysis.
Why Volume Concentrates Near the Surface of a High-Dimensional Ball
To understand the unusual geometry of high-dimensional spaces, consider the volume of a ball.
In two dimensions, the corresponding measure is the area of a disk. In three dimensions, it is the volume of an ordinary sphere.
The volume of a (d)-dimensional ball with radius (R) is:
$$
V_d(R)=C_dR^d
$$
where:
$$
C_d=\frac{\pi^{d/2}}{\Gamma(d/2+1)}
$$
The important property is that the volume is proportional to the radius raised to the power (d).
Now consider a unit ball with radius 1 and an inner ball with radius 0.9.
The proportion of the total volume contained within the inner ball is:
$$
\frac{V_d(0.9)}{V_d(1)}=0.9^d
$$
Therefore, the proportion of volume contained within the outer shell, between radii 0.9 and 1, is:
$$
1-0.9^d
$$
| Number of Dimensions | Volume Within the Outer 10% of the Radius |
|---|---|
| 2 | 19.0% |
| 3 | 27.1% |
| 10 | 65.1% |
| 30 | 95.8% |
| 100 | Approximately 99.997% |
In 100 dimensions, almost the entire volume lies between radii 0.9 and 1.
This result is difficult to anticipate from our everyday experience of two-dimensional and three-dimensional geometry.
However, it does not mean that the surface itself has nonzero volume.
Rather, an outer shell of finite thickness occupies almost all the volume of the ball.
It is also important to recognize the assumptions. This result describes the geometry of a high-dimensional ball and the distribution of points sampled uniformly from its volume.
It does not imply that every real high-dimensional dataset is distributed in this way.
The phenomenon is commonly described as Concentration in a Thin Shell.
Concentration Near a Sphere and Concentration on a Sphere
The expression “concentration on a sphere” can refer to different phenomena that should be distinguished.
The first is the concentration of volume near the boundary of a high-dimensional ball, described above.
The second concerns the concentration of probability distributions and geometric quantities on a high-dimensional sphere.
For example, when a point is sampled uniformly from the surface of a high-dimensional unit sphere, its coordinate in any fixed direction is likely to be close to zero.
Geometrically, this means that much of the sphere’s surface measure lies near the equator perpendicular to that direction.
More generally, the tendency of random quantities in high-dimensional spaces to remain close to characteristic values is studied under Concentration of Measure.
Concentration of Measure is a broad subject in high-dimensional geometry and probability theory.
Concentration in a thin shell, concentration around an equator, and Distance Concentration are related phenomena, but they should not be treated as interchangeable formulations of the same theorem.
The following sections focus on Distance Concentration because of its importance in data analysis.
Random Directions Become Almost Orthogonal in High Dimensions
In two dimensions, two vectors are perpendicular only when they form a right angle.
In high-dimensional spaces, however, two randomly selected directions are likely to be almost perpendicular.
Consider two independent unit vectors (u) and (v), sampled uniformly from the surface of a (d)-dimensional unit sphere.
Both vectors have length 1.
The expected value of their inner product is:
$$
E[u\cdot v]=0
$$
Its variance is:
$$
\operatorname{Var}(u\cdot v)=\frac{1}{d}
$$
As dimensionality increases, the variance decreases.
Consequently, the inner product tends to be concentrated near zero.
Because the inner product of two unit vectors equals the cosine of the angle between them, an inner product near zero corresponds to an angle close to 90 degrees.
The squared Euclidean distance between the vectors is:
$$
|u-v|^2=2-2(u\cdot v)
$$
Therefore, as the inner product concentrates near zero:
$$
|u-v|\approx\sqrt{2}
$$
This is a probabilistic result for independently sampled, uniformly distributed unit vectors.
It does not imply that meaningful relationships in real datasets or learned vector representations necessarily disappear in high dimensions.
What Is Distance Concentration?
Distance Concentration refers to the tendency of distances between data points to become relatively similar in high-dimensional spaces under certain probability distributions and distance measures.
In a low-dimensional space, we can often distinguish intuitively between points that are very close and points that are far apart.
In high-dimensional data, however, the relative differences between distances may become smaller.
This does not mean that the absolute distances necessarily decrease.
In fact, typical Euclidean distances may increase as dimensionality grows.
The important issue is the reduction in variation relative to the magnitude of the distances.
When this occurs, differences between nearby and distant points may become less informative for certain analytical tasks.
This matters for methods based on Nearest Neighbor Search, classification, and clustering.
In their 1999 paper “When Is ‘Nearest Neighbor’ Meaningful?”, Kevin Beyer and colleagues demonstrated that, under particular conditions, the relative difference between nearest-neighbor and farthest-neighbor distances can shrink as dimensionality increases.
However, the paper does not establish that nearest-neighbor analysis becomes meaningless for every high-dimensional dataset. It also discusses conditions and examples in which the phenomenon does not occur.
Understanding Distance Concentration Mathematically
Consider a simple probabilistic model.
Suppose two data points (X) and (Y) are sampled independently from a (d)-dimensional standard normal distribution:
$$
X,Y\sim N(0,I_d)
$$
Each coordinate has mean zero and variance one, and the coordinates are independent.
Their difference follows:
$$
X-Y\sim N(0,2I_d)
$$
The squared Euclidean distance is:
$$
D^2=|X-Y|^2
$$
Its expected value is:
$$
E[D^2]=2d
$$
Its variance is:
$$
\mathrm{Var}(D^2)=8d
$$
The standard deviation of the squared distance is therefore:
$$
\sigma(D^2)=\sqrt{8d}
$$
The coefficient of variation, defined as the ratio of the standard deviation to the mean, is:
$$
CV(D^2)
=\frac{\sqrt{8d}}{2d}
=\sqrt{\frac{2}{d}}
$$
As (d) increases, this coefficient approaches zero.
| Dimension (d) | Coefficient of Variation of Squared Distance |
|---|---|
| 10 | Approximately 44.7% |
| 100 | Approximately 14.1% |
| 1,000 | Approximately 4.5% |
The expected squared distance increases linearly with dimensionality, while its relative variation decreases.
Consequently, the typical Euclidean distance becomes concentrated around a value of approximately:
$$
D\approx\sqrt{2d}
$$
However, this example depends on the specific assumption of independent normally distributed coordinates.
The coefficient of variation calculated here refers to squared distance, not distance itself.
The same numerical relationships do not necessarily hold for real datasets. Distance Concentration depends on the underlying probability distribution, correlations among variables, the distance measure, and the number and arrangement of data points.
Why Nearest and Farthest Points May Become Difficult to Distinguish
To understand how Distance Concentration affects nearest-neighbor analysis, consider a query point (q) and several candidate points.
Let (D_{\min}) be the distance to the nearest candidate and (D_{\max}) the distance to the farthest candidate.
One possible measure of their relative contrast is:
$$
C=\frac{D_{\max}-D_{\min}}{D_{\min}}
$$
where (D_{\min}>0).
The smaller this value becomes, the smaller the difference between nearest and farthest distances relative to the nearest-neighbor distance.
For example, if the nearest distance is 10 and the farthest is 20, then:
$$
C=1
$$
If the nearest distance is 100 and the farthest is 101, then:
$$
C=0.01
$$
In the second case, the relative difference is much smaller.
Nevertheless, a small relative contrast does not automatically make nearest-neighbor identification useless.
Even small distance differences may provide useful information if they are stable and reflect meaningful structure.
Conversely, larger differences may be unreliable if they are dominated by noise.
The important contribution of Beyer and colleagues was to examine under which conditions distance-based discrimination becomes less meaningful, rather than to reject nearest-neighbor analysis solely because dimensionality is high.
Does Distance Concentration Always Occur in High Dimensions?
High-dimensional data are sometimes described as if all points inevitably become equally distant.
This is incorrect.
Distance Concentration depends on the data distribution and the properties of the distance measure.
For example, a dataset may have 1,000 recorded dimensions, but only a few variables may exhibit substantial variation while the others remain nearly constant.
In that situation, meaningful distance differences may remain.
Similarly, if the data contain clearly separated groups, distances within groups and between groups may remain distinguishable.
A small number of highly informative characteristics may also preserve meaningful relationships despite a large number of recorded dimensions.
The conditions under which Distance Concentration does not arise have been examined in theoretical research, including “When Is ‘Nearest Neighbour’ Meaningful: A Converse Theorem and Implications” (2009).
The effectiveness of nearest-neighbor analysis should therefore not be judged by dimensionality alone.
It is necessary to examine distance distributions, neighborhood stability, the structure of the data, and the objectives of the analysis.
Relevant Features and the Accumulation of Noise
An important problem in high-dimensional data is the mixture of relevant and irrelevant features.
Suppose we use 100 measurements to evaluate product quality.
Only five of these variables may be strongly associated with product defects, while the remaining 95 may contain little information relevant to the objective.
If every variable is treated equally in a distance calculation, differences among the irrelevant features may accumulate and obscure the contribution of the five important variables.
In this case, adding variables increases the amount of recorded information without necessarily increasing the information useful for identifying defects.
Selecting the relevant variables may produce a more effective classification.
However, deciding which variables are relevant presents another problem.
When the objective is to predict a known category of defects, features can be evaluated according to their relationship with that outcome.
When the objective is to discover previously unknown defects or patterns, removing features solely because they are unrelated to known outcomes may eliminate information needed for discovery.
Feature Selection therefore involves more than simply reducing dimensionality.
It requires considering what the features are intended to discover, explain, or predict.
Euclidean Distance and Cosine Similarity
In multidimensional analysis, Cosine Similarity is widely used alongside Euclidean Distance.
Euclidean Distance measures the magnitude of the difference between two vectors:
$$
d(x,y)=\sqrt{\sum_{k=1}^{d}(x_k-y_k)^2}
$$
Cosine Similarity measures the relationship between their directions:
$$
\cos(x,y)
=\frac{x\cdot y}{|x||y|}
$$
Cosine Similarity is commonly used with text embeddings.
However, switching from Euclidean Distance to Cosine Similarity does not automatically eliminate Distance Concentration.
When both vectors are normalized to unit length:
$$
|x-y|^2=2-2\cos(x,y)
$$
Therefore, Euclidean Distance and Cosine Similarity produce the same nearest-neighbor ranking for unit-normalized vectors.
The tendency of random high-dimensional directions to become almost orthogonal directly affects Cosine Similarity as well.
The important issue is not simply the name of the distance measure.
It is whether that measure adequately represents the characteristics and relationships relevant to the analytical objective.
Distance Metrics Change Neighborhood Relationships
Several distance measures are available for multidimensional analysis.
Manhattan Distance sums the absolute differences between coordinates:
$$
d_1(x,y)=\sum_{k=1}^{d}|x_k-y_k|
$$
Euclidean Distance is:
$$
d_2(x,y)=\left(\sum_{k=1}^{d}|x_k-y_k|^2\right)^{1/2}
$$
More generally, Minkowski Distance is:
$$
d_p(x,y)=\left(\sum_{k=1}^{d}|x_k-y_k|^p\right)^{1/p}
$$
with (p\geq1) for the usual metric interpretation.
Changing the distance measure changes how differences among features contribute to the result.
In their 2001 paper “On the Surprising Behavior of Distance Metrics in High Dimensional Spaces,” Charu Aggarwal, Alexander Hinneburg, and Daniel Keim examined how different distance measures behave in high-dimensional spaces.
Their research illustrates the importance of selecting appropriate distance measures when studying high-dimensional similarity.
However, no single distance metric is universally superior for every high-dimensional dataset.
Some research also investigates expressions involving (p<1), but these may fail to satisfy the usual metric axioms and should not be confused with conventional distances.
Hubness: Why Some Points Become Neighbors of Many Others
Another interesting phenomenon in high-dimensional data is Hubness.
Hubness occurs when a relatively small number of points appear repeatedly in the nearest-neighbor lists of many other points.
For example, suppose we identify the ten nearest neighbors of every point in a dataset.
Some variation in how often each point appears as a neighbor is expected.
However, certain high-dimensional datasets exhibit pronounced inequality: a few points are selected as neighbors unusually often.
These points are called Hubs.
Miloš Radovanović, Alexandros Nanopoulos, and Mirjana Ivanović investigated this phenomenon in their 2010 paper “Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data.”
Hubness can influence nearest-neighbor classification, clustering, and information retrieval.
For example, a particular document may appear among the nearest neighbors of many unrelated search queries.
This may indicate that the document is genuinely relevant to a broad range of subjects.
Alternatively, it may reflect geometric characteristics of the embedding space that cause the document to be selected excessively often.
Hubness is not identical to Distance Concentration.
It concerns the distribution of how frequently points appear in nearest-neighbor lists.
Furthermore, strong Hubness does not necessarily occur in every high-dimensional dataset.
Intrinsic Dimension and Effective Dimension
A crucial distinction in high-dimensional analysis is the difference between the number of observed variables and the number of independent degrees of freedom present in the data.
Consider a dataset with 1,000 measured variables.
If all variables vary independently, the underlying structure may be extremely complex.
However, numerous variables may be controlled by only a few common factors.
For example, 100 measurements might all respond to changes in a single underlying physical state.
In that case, they do not necessarily represent 100 independent pieces of information.
This motivates the concepts of Intrinsic Dimension and Effective Dimension.
Intrinsic Dimension generally concerns the number of degrees of freedom required to describe the structure on which the data lie.
Effective Dimension is used in several related but distinct ways, including measures of how variation is distributed across directions or components.
The two terms should not be treated as mathematically interchangeable.
For example, data represented in a 1,000-dimensional space may actually lie near a manifold with approximately ten intrinsic dimensions.
Levina and Bickel investigated methods for estimating Intrinsic Dimension using distances among neighboring observations.
However, Intrinsic Dimension cannot always be estimated reliably as a single fixed number.
The results may depend on the distribution of observations, noise, sample size, and local structure.
The Manifold Hypothesis and Dimensionality Reduction
The Manifold Hypothesis proposes that data represented in high-dimensional spaces may actually be concentrated near structures having substantially lower intrinsic dimensionality.
For example, a human face image consists of numerous pixels.
However, much of the variation among images of the same face may be influenced by a smaller number of factors, such as facial orientation, illumination, and expression.
If such underlying structure exists, it may be possible to represent the data using fewer dimensions.
Principal Component Analysis (PCA) reduces dimensionality by identifying directions of linear variation.
Nonlinear dimensionality reduction methods, including UMAP, seek representations based on features such as local neighborhood relationships.
UMAP incorporates ideas associated with manifold learning and topological structure.
However, dimensionality reduction does not preserve every aspect of the original data.
Furthermore, UMAP relies on neighborhood relationships in the original high-dimensional space.
If the original distances or neighborhoods are poorly suited to the analytical objective, those problems may influence the reduced representation.
Dimensionality reduction should therefore not be assumed to eliminate the Curse of Dimensionality automatically.
Dimensionality Reduction constructs a representation of selected structures in high-dimensional data using fewer coordinates. Its validity must be evaluated in relation to the intended analytical purpose.
What Does Dimensionality Reduction Preserve?
Different dimensionality reduction methods emphasize different properties.
Some seek to retain as much variance as possible.
Others emphasize approximate distance preservation, local neighborhoods, group separation, or information useful for prediction.
| Objective | Property Emphasized |
|---|---|
| Variance preservation | Representing data variation through a smaller number of components |
| Approximate distance preservation | Maintaining distances among objects |
| Local neighborhood preservation | Preserving relationships among nearby objects |
| Visualization | Producing interpretable two-dimensional or three-dimensional representations |
| Predictive performance | Retaining information relevant to target variables |
| Noise reduction | Reducing the influence of unnecessary variation |
These objectives do not necessarily coincide.
For example, a direction containing substantial variance is not automatically important for classification.
Similarly, a method preserving local neighborhoods may not accurately preserve distances between widely separated groups.
The quality of dimensionality reduction cannot therefore be evaluated using one universal criterion.
If the reduced data will subsequently be used for clustering or another analytical method, it is important to examine whether the relationships required for that task have been preserved.
Is High Dimensionality Always a Disadvantage? The Blessing of Dimensionality
High-dimensional geometry does not always create difficulties.
Under certain conditions, high dimensionality can support data separation, probabilistic analysis, and mathematical approximation.
These beneficial aspects are sometimes discussed under the term Blessing of Dimensionality.
For example, the tendency of random directions to become almost orthogonal can be useful in constructing representations that are relatively distinct from one another.
Concentration of Measure can also make certain probabilistic quantities more predictable and mathematically tractable.
Another important result is the Johnson–Lindenstrauss Lemma.
It shows that a finite collection of points can be mapped into a lower-dimensional Euclidean space while approximately preserving pairwise distances within a specified tolerance.
Under the relevant conditions, the required target dimension is related to quantities such as the logarithm of the number of points, rather than directly to the original ambient dimension.
This demonstrates that preserving useful geometric relationships does not always require retaining every coordinate of an extremely high-dimensional representation.
However, the Blessing of Dimensionality does not imply that increasing dimensionality automatically improves classification or prediction.
It is essential to distinguish the conditions under which high dimensionality is beneficial from those under which it creates difficulties.
The Curse of Dimensionality and the Ugly Duckling Theorem
The Ugly Duckling Theorem, developed by Satosi Watanabe, is a mathematical result concerning similarity and classification.
For a finite set of distinguishable objects, the theorem shows that if all possible predicates are counted equally, every pair of distinct objects shares the same number of predicates.
Consequently, simply counting all possible properties without distinguishing their importance cannot determine which objects are more similar.
This result has an interesting conceptual resemblance to Distance Concentration in high-dimensional spaces.
For example, when numerous features are treated equally in a distance calculation, differences among objects may become less distinctive under certain conditions.
However, the two results are mathematically different.
| Aspect | Ugly Duckling Theorem | Distance Concentration |
|---|---|---|
| Mathematical foundation | Logic and combinatorics | High-dimensional geometry and probability theory |
| What is compared | Predicates shared by objects | Distances between objects |
| Main assumptions | All possible predicates are counted equally | Particular probability distributions, distance measures, and other conditions |
| Result | Every pair of distinct objects shares the same number of predicates | Relative differences among distances may become smaller |
| Importance of feature selection | Necessary to distinguish similarity meaningfully | May be essential for obtaining informative distance relationships |
The Ugly Duckling Theorem is an exact combinatorial result under its assumptions.
Distance Concentration is a phenomenon that depends on the statistical and geometric properties of the data.
Neither theorem can be derived directly from the other merely by increasing the number of dimensions.
The common methodological insight is that collecting many characteristics and treating them indiscriminately does not necessarily produce meaningful similarity judgments.
However, this is a conceptual connection between the two results, not a claim that they describe the same mathematical mechanism.
Implications for Epistemology: What Makes Things Similar?
When humans recognize similarities, they do not compare every possible characteristic equally.
Consider the Moon and an apple.
They differ greatly in appearance, size, material composition, and location.
However, physics allows us to understand aspects of their motion through common laws of gravity.
The significant correspondence concerns physical relationships, not superficial appearance.
If the objective were to classify food, there would normally be no reason to treat the Moon and an apple as members of the same category.
Thus, what counts as similarity depends on which characteristics or relationships are considered relevant.
The Ugly Duckling Theorem provides a formal perspective on this problem.
Distance Concentration adds another consideration: even when numerous characteristics are represented numerically, a distance measure does not automatically produce distinctions that are informative for the intended purpose.
However, this does not establish that similarity is entirely subjective.
Evolutionary relationships, physical properties of materials, and conditions associated with equipment failures are examples of structures that can be investigated empirically.
The important task is to determine which characteristics are meaningful for the problem and evaluate their relevance through evidence.
The Number of Features Is Not the Same as Depth of Understanding
Multivariate Analysis and AI make it possible to represent objects using large numbers of variables.
However, describing objects in great detail is not equivalent to understanding their most important structures.
Consider a manufacturing process monitored by 1,000 sensors.
These measurements may provide extensive information about the process.
However, if product defects are primarily influenced by a few specific conditions, treating all sensor values equally may not be the most effective approach.
Conversely, if the analysis considers only variables already known to cause defects, it may overlook previously unrecognized signals of abnormal behavior.
Feature reduction and the discovery of unknown patterns therefore involve different objectives.
For predicting known outcomes, selecting features according to their relationships with the target variable may be effective.
For exploratory analysis, feature representations and methods that do not depend exclusively on known target variables may be necessary.
This distinction is important when comparing predictive modeling with exploratory data analysis.
The Curse of Dimensionality in Multivariate Analysis
Multivariate Analysis examines relationships among objects using multiple variables.
Examples include PCA, Factor Analysis, Clustering, Regression Analysis, and Discriminant Analysis.
In these methods, the relationship between the number of variables and the number of observations can be particularly important.
For example, regression models involving many explanatory variables may become difficult to estimate reliably when the number of observations is limited.
Strong correlations among explanatory variables may also create problems such as Multicollinearity.
In Clustering, irrelevant variables may dominate the distance calculation and obscure meaningful differences among objects.
However, these problems do not all have the same mathematical cause.
Parameter estimation difficulties, strong correlations, and Distance Concentration are distinct phenomena.
Understanding the Curse of Dimensionality therefore requires identifying the limitations relevant to each analytical method rather than attributing every difficulty to the number of variables alone.
Distance Concentration in Clustering
Clustering methods organize data into groups using similarities, distances, or related structural criteria.
For example, k-means minimizes an objective function based on distances between data points and cluster centers.
Hierarchical clustering uses distances between objects or clusters.
Growing Neural Gas (GNG) uses distances between input vectors and reference vectors associated with network nodes during learning.
In such methods, the interpretation of distance is important.
If distances become strongly concentrated, neighborhood relationships may become more sensitive to small changes in noise or analytical conditions.
However, meaningful cluster structure may still exist even when some degree of Distance Concentration is present.
The effectiveness of clustering should therefore be evaluated using more than distance distributions alone.
Cluster stability, relationships with external variables, and interpretability may also be important.
Furthermore, visually distinct clusters are not necessarily meaningful for the analytical objective.
Even if within-cluster distances are small and between-cluster distances are large, the resulting classification may fail to explain important operational or technological differences.
Main Approaches to High-Dimensional Problems
Various methods can help address the difficulties associated with high-dimensional data.
However, no single method resolves every aspect of the Curse of Dimensionality.
| Approach | Main Purpose | Important Limitation |
|---|---|---|
| Feature Selection | Retain important variables while removing unnecessary ones | May eliminate features relevant to unknown patterns |
| PCA | Summarize linear variation using fewer components | High variance does not necessarily imply importance for the objective |
| Nonlinear Dimensionality Reduction, including UMAP | Construct lower-dimensional representations using local structure | Original distances and relationships are not perfectly preserved |
| Feature Weighting | Emphasize characteristics relevant to the objective | Requires a basis for assigning weights |
| Changing the Distance Metric | Define similarity more appropriately for the problem | No metric is universally superior |
| Regularization | Reduce excessive model complexity and unstable estimation | Effectiveness depends on model assumptions |
| Collecting More Data | Improve the information available for estimation | May not provide sufficient coverage of the entire space |
| Exploiting Local Structure | Analyze relationships within relevant neighborhoods | Local structures may themselves be difficult to estimate reliably |
The appropriate response depends on the problem.
If computational cost is the main difficulty, more efficient algorithms may be needed.
If Distance Concentration is the problem, feature representation or the distance metric may need reconsideration.
If the objective is discovering unfamiliar patterns, feature selection based entirely on known outcomes may be inappropriate.
Does Reducing Dimensionality Mean Discarding Information?
A natural concern about Dimensionality Reduction is that removing features may also remove important information.
This concern is justified.
Dimensionality Reduction can involve information loss.
However, when many variables are strongly correlated, they may contain overlapping information.
Removing irrelevant or noisy variation may help reveal the underlying structure.
Conversely, features with low variance or low frequency are not necessarily unimportant.
For example, when detecting rare abnormalities, a feature that almost never changes under normal conditions may become highly informative when a failure occurs.
A selection procedure based only on total variance could underestimate its significance.
Information reduction should therefore not be treated as an objective in itself.
The aim is to reduce unnecessary complexity while preserving information relevant to the intended analysis.
Implications for Science and Engineering
The Curse of Dimensionality is important in many scientific and engineering fields beyond Machine Learning.
Physics investigates systems involving large numbers of degrees of freedom.
Control Engineering may require predicting and controlling systems with many state variables.
Statistics addresses the estimation of models involving numerous variables from limited observations.
Design of Experiments must deal with rapidly growing combinations of factors and levels.
| Field | Representative High-Dimensional Challenge |
|---|---|
| Control Engineering | Expansion of state spaces and computational difficulty of optimal control |
| Statistics | Estimation of high-dimensional parameters and overfitting |
| Design of Experiments | Rapid growth of combinations of factors and levels |
| Materials Science | Relationships among composition, structure, and numerous physical properties |
| Physics | Representation and analysis of systems with many degrees of freedom |
| Marketing | Complex customer classifications involving many attributes |
| Decision Analysis | Increasing numbers of evaluation criteria and judgment requirements |
However, high dimensionality can also reveal regularities.
In Statistical Physics, for example, systems with very many degrees of freedom can exhibit stable macroscopic behavior that is amenable to statistical analysis.
Thus, high dimensionality should not be understood simply as the presence of too much information to process.
Some phenomena become more difficult with increasing dimensionality, while others become more regular or analytically tractable.
Implications for Decision Analysis
Businesses often evaluate alternatives using many criteria.
For example, candidates for new business investment may be assessed according to market size, growth rate, profitability, technological capabilities, competitiveness, R&D costs, and commercialization time.
Adding criteria can support more comprehensive evaluations.
However, it does not necessarily improve decision quality.
If multiple criteria measure nearly the same underlying characteristic, that aspect may receive excessive influence.
Similarly, numerous weakly relevant criteria may dilute the contribution of more important ones.
Increasing the number of criteria also increases the effort required to collect information and assign weights.
However, duplicated evaluation criteria and weighting problems in Decision Analysis are not mathematically identical to Distance Concentration.
Their conceptual relationship lies in the observation that adding more characteristics and treating them equally does not necessarily improve discrimination among alternatives.
Implications for Concept Formation and Cross-Domain Exploration
Similarity also plays an important role in Concept Formation and Cross-Domain Exploration.
For example, we may describe a technology using many detailed features and search for other technologies with similar characteristics.
However, if all details of the original technology are emphasized equally, the results may be dominated by closely related technologies within the same discipline.
By abstracting technological functions and relational structures, it may become possible to investigate solutions from unfamiliar domains.
Consider replacing the category “self-healing materials” with the functional description:
“Repeatedly delivering repair agents to damaged locations.”
This formulation may reveal supply networks and transport mechanisms from other fields that are not normally classified as self-healing materials.
This is not simply a matter of reducing the number of variables.
It involves changing which characteristics and relationships are regarded as significant.
The Curse of Dimensionality and the Ugly Duckling Theorem provide useful perspectives on why such feature selection matters.
However, Cross-Domain Exploration does not directly solve these mathematical problems.
Nor does an abstract similarity necessarily imply that a technology can be transferred successfully.
The characteristics used for exploration and those used to evaluate discovered candidates may need to serve different purposes.
The Curse of Dimensionality and Concentration Phenomena in ThinkNavi
ThinkNavi regards the Curse of Dimensionality and high-dimensional concentration phenomena as foundational concepts for understanding the geometry of multivariate data and examining Feature Selection, Similarity, Classification, and Concept Formation.
These are technical issues in Statistics and Machine Learning, but they also relate to the more general question of how objects should be represented through selected characteristics.
For example, products may be classified using dimensions, prices, and performance indicators.
The resulting classification depends on which characteristics are included.
Adding more features may produce a richer representation, but adding irrelevant characteristics can make meaningful differences more difficult to identify.
ThinkNavi Free could provide interactive calculations and visualizations illustrating these phenomena.
For example, users could change the number of dimensions and calculate the proportion of a high-dimensional ball’s volume contained within its outer shell.
They could also generate random multidimensional data and examine how mean distances, variation, nearest-neighbor distances, and farthest-neighbor distances change with dimensionality.
Another possible application would allow users to modify feature selections and weights and observe the resulting changes in distances and clustering.
However, simulations should explicitly identify the assumed probability distributions and dependencies among features.
A phenomenon observed in random simulated data cannot automatically be generalized to all real high-dimensional datasets.
An LLM-assisted version could help explain and compare Feature Selection, distance measures, and Dimensionality Reduction methods in relation to the user’s analytical objectives.
However, recommending a particular method without examining the actual data structure would be inappropriate.
Implications for ConceptMiner R&D
ConceptMiner R&D represents information from technical documents using embedding vectors and analyzes conceptual structures.
High-dimensional similarity and distance are therefore important considerations in its processing pipeline.
For example, directly clustering the original high-dimensional embedding vectors may produce different results from clustering vectors after dimensionality reduction with UMAP.
If Distance Concentration or accumulated noise affects the original space, direct distance calculations may fail to capture useful neighborhood relationships.
On the other hand, dimensionality reduction may lose information or alter relationships that existed in the original space.
Neither approach is therefore universally superior on theoretical grounds.
When GNG is trained using vectors produced by UMAP, it learns the distance structure of the UMAP-transformed space, not the geometry of the original embedding space itself.
This may be useful for the analytical objective.
However, it is important to evaluate how well relevant semantic and technological relationships have been preserved.
Possible evaluation methods include nearest-neighbor overlap, stability of clustering under different conditions, and interpretation of the results using technological knowledge.
An additional issue is that semantic similarity between embeddings does not necessarily correspond to similarity in technological function or structure.
For example, self-healing materials and biological substance transport systems may use substantially different terminology and concern different physical objects.
However, they may share an abstract supply-network structure.
Discovering such relationships may require more than general semantic similarity.
Extraction of technological elements, functional abstraction, and analysis of relational correspondences may be necessary.
ConceptMiner R&D can therefore treat embedding-based information organization and exploration based on extracted technological elements as complementary but distinct activities.
Calculating distances appropriately and determining what should count as meaningful technological similarity are different problems.
What Should Be Examined in Practical Analysis?
When analyzing high-dimensional data, it is important to examine more than the number of dimensions.
| Evaluation Item | Main Question |
|---|---|
| Ambient dimension | How many variables or features are recorded? |
| Meaning of features | What does each variable represent in relation to the objective? |
| Relationships among features | How much redundancy or correlation exists? |
| Intrinsic structure | Can the data be described using fewer degrees of freedom? |
| Distance distribution | How large and variable are the distances? |
| Neighborhood stability | Do neighborhood relationships change when features or metrics change? |
| Cluster stability | Does the identified structure persist under different analytical conditions? |
| External validity | Do the relationships correspond to meaningful technological or operational characteristics? |
| Effects of dimensionality reduction | Which properties of the original space have been preserved or changed? |
For example, embedding vectors can be examined using both Euclidean Distance and Cosine Similarity.
Neighborhood rankings before and after dimensionality reduction can also be compared.
However, such measurements cannot by themselves establish whether the resulting relationships are technologically meaningful.
Where appropriate, document contents, domain expertise, and independent evaluation data should be used to assess the results.
Conclusion: More Dimensions Do Not Necessarily Mean More Meaning
The Curse of Dimensionality describes a broad family of difficulties arising as the number of features or state variables increases, including computational complexity, growing data requirements, and statistical estimation problems.
High-dimensional spaces also exhibit geometric properties that differ from everyday intuition.
The volume of a high-dimensional ball is concentrated near its surface, and independently sampled random directions tend to be almost orthogonal.
Under particular probability distributions and distance measures, Distance Concentration can cause relative differences among distances to become small.
However, these phenomena do not arise identically in every high-dimensional dataset.
The data distribution, correlations among features, Intrinsic Dimension, distance measure, and analytical objective all influence whether meaningful structures can be identified.
Addressing the Curse of Dimensionality therefore requires more than simply reducing the number of variables.
It requires examining what information should be preserved and which relationships are important.
The comparison with the Ugly Duckling Theorem extends this discussion to the broader problems of cognition and Concept Formation.
The Ugly Duckling Theorem demonstrates that counting every possible feature equally does not provide a meaningful distinction among object pairs.
Distance Concentration demonstrates that, under certain conditions, distances calculated from numerous features may become less informative for distinguishing objects.
Although these are mathematically different phenomena, they share an important methodological implication: simply increasing the number of characteristics does not guarantee better classification or deeper understanding.
This insight is relevant not only to Multivariate Analysis and Machine Learning but also to R&D, Decision Analysis, technology exploration, and Concept Formation.
Describing an object in greater detail is not the same as understanding its essential characteristics and relationships.
In an era when AI makes it possible to gather enormous amounts of information and analyze increasingly large sets of features, the ability to determine which characteristics matter—and why they matter—becomes even more important.
Primary English Terms: Curse of Dimensionality; Concentration Phenomena in High-Dimensional Spaces
Related Terms: High-Dimensional Geometry, Concentration of Measure, Thin-Shell Concentration, Distance Concentration, Almost Orthogonality, Nearest Neighbor Search, Hubness, Intrinsic Dimension, Effective Dimension, Manifold Hypothesis, Dimensionality Reduction, Feature Selection, Feature Weighting, Metric Learning, Blessing of Dimensionality
Related Thinking Methods and Theories:
- What Is Machine Learning? | From a Technology for Specialists to an Accessible Tool for Analysis and Discovery
- The Ugly Duckling Theorem — Where Does Similarity Come From?
- Concept Formation and Categorization | Change the Categories, Change What You See
- Epistemology | What Can We Know, and How?
- Multivariate Analysis — Discovering Relationships and Structures Among Multiple Variables
- Principal Component Analysis,
- Clustering,
- UMAP,
- Growing Neural Gas,
- Machine Learning,
- No Free Lunch Theorem,
- Structure-Mapping Theory | Understanding Relational Correspondences and Generating New Inferences Across Domains
- Cross-Domain Exploration | Discovering Unfamiliar Technologies, Knowledge, and Solutions Beyond Disciplinary Boundaries
- Concept Exploration | Discovering Emerging Concepts and Unknown Possibilities
References and Further Reading
- Richard Bellman, Dynamic Programming, Princeton University Press, 1957. A foundational work on Dynamic Programming and the computational challenges underlying the Curse of Dimensionality.
- Richard Bellman, Adaptive Control Processes: A Guided Tour, Princeton University Press, 1961. An examination of optimization and control problems involving multidimensional state spaces.
- Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft, “When Is ‘Nearest Neighbor’ Meaningful?,” Database Theory—ICDT ’99, Lecture Notes in Computer Science, Vol. 1540, pp. 217–235, 1999. A foundational investigation of Distance Concentration and the conditions affecting nearest-neighbor analysis in high-dimensional data.
- Charu C. Aggarwal, Alexander Hinneburg, and Daniel A. Keim, “On the Surprising Behavior of Distance Metrics in High Dimensional Spaces,” Database Theory—ICDT 2001, Lecture Notes in Computer Science, Vol. 1973, pp. 420–434, 2001. A study of the behavior of different distance measures in high-dimensional spaces.
- “When Is ‘Nearest Neighbour’ Meaningful: A Converse Theorem and Implications,” Journal of Complexity, Vol. 25, No. 4, pp. 385–397, 2009. A theoretical examination of conditions under which Distance Concentration does not occur.
- Miloš Radovanović, Alexandros Nanopoulos, and Mirjana Ivanović, “Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data,” Journal of Machine Learning Research, Vol. 11, pp. 2487–2531, 2010. A study of Hubness and its implications for nearest-neighbor relationships and classification.
- Elizaveta Levina and Peter J. Bickel, “Maximum Likelihood Estimation of Intrinsic Dimension,” Advances in Neural Information Processing Systems 17, 2004. Research on estimating Intrinsic Dimension using local neighborhood distances.
- Leland McInnes, John Healy, and James Melville, “UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction,” arXiv:1802.03426, 2018. A foundational account of nonlinear dimensionality reduction based on manifold learning and neighborhood structure.
- Alexander N. Gorban and colleagues, “Blessing of Dimensionality: Mathematical Foundations of the Statistical Physics of Data,” 2018. An examination of high-dimensional geometric and probabilistic properties that can be beneficial for data analysis.
- University of California, Berkeley, “High-Dimensional Space.” Educational material on high-dimensional volumes, concentration phenomena, and geometric properties.
- Satosi Watanabe, Knowing and Guessing: A Quantitative Study of Inference and Information, Wiley, 1969. A foundational discussion of similarity, classification, inference, and the Ugly Duckling Theorem.