Categorical Magnitude and Entropy

Stephanie Chen & Juan Pablo Vigneaux · 2023 · arxiv arXiv:2303.00879

Prereqs: 🍞 Leinster 2021 (magnitude, Hill numbers). 🍞 Baez, Fritz, Leinster 2011 (entropy characterization) helps.

Shannon entropy and magnitude are the same invariant in disguise. Under a uniform distribution, log(magnitude) = entropy. The paper unifies them through a single categorical construction — the Euler characteristic of an enriched category.

Entropy from a uniform distribution

Shannon entropy of a uniform distribution over n outcomes is log(n). Magnitude of n equally-spaced points is n. So log(magnitude) = log(n) = entropy. This is the simplest case of the unification.

Scheme

; Uniform case: log(magnitude) = entropy

(define (shannon-entropy p)
  (- (apply + (map (lambda (pi)
    (if (> pi 0) (* pi (log pi)) 0)) p))))

(define (log-magnitude-uniform n)
  ; For n equally-spaced points, magnitude = n
  (log n))

(for-each (lambda (n)
  (let* ((p (make-list n (/ 1.0 n)))
         (H (shannon-entropy p))
         (logM (log-magnitude-uniform n)))
    (display "n=") (display n)
    (display " H=") (display H)
    (display " log|A|=") (display logM)
    (display (if (< (abs (- H logM)) 0.0001) " ✓" " ✗"))
    (newline)))
  '(2 3 5 10 20))
; They agree everywhere. Not a coincidence.

Beyond uniform: the weighted case

For non-uniform distributions, the connection goes through weighted magnitude. Given a metric space with a probability distribution (weights), the log of the weighted magnitude recovers the entropy of that distribution. The uniform case is the special case where all weights are equal.

Scheme

; Weighted magnitude: assign weights to points
; For discrete points with metric d(i,j) = infinity if i≠j:
; magnitude with weights w = sum(w_i) when all pairs are "far"
; In this trivial metric, log(sum(w)) relates to entropy

(define (shannon-entropy p)
  (- (apply + (map (lambda (pi)
    (if (> pi 0) (* pi (log pi)) 0)) p))))

; Non-uniform distribution
(define p '(0.5 0.3 0.2))

; Exponential of entropy = effective number of outcomes
(define exp-H (exp (shannon-entropy p)))

(display "p = ") (display p) (newline)
(display "H(p) = ") (display (shannon-entropy p)) (newline)
(display "exp(H) = ") (display exp-H) (newline)
(display "# outcomes = ") (display (length p)) (newline)
; exp(H) < 3: non-uniform distribution has fewer
; "effective" outcomes than actual outcomes.
; This is the diversity-magnitude bridge.

The Euler characteristic connection

Both entropy and magnitude arise as the Euler characteristic of an enriched category. A finite metric space is a category enriched over [0,∞). Its Euler characteristic is the magnitude. A finite probability space is a category enriched over [0,1]. Its Euler characteristic is exp(entropy). Same construction, different enrichment.

Scheme

; Euler characteristic = magnitude for enriched categories
; Finite metric space enriched over [0, infinity)
; Finite probability space enriched over [0, 1]

; For a 3-point metric space with distances:
(define (similarity d) (exp (- d)))

; Points at distances 0, 1, 2 from each other
(define Z '((1.0    0.368  0.135)
            (0.368  1.0    0.368)
            (0.135  0.368  1.0)))
; Z_ij = e^(-d_ij), diagonal = 1

; Magnitude = sum of weights w where Zw = 1
; For well-separated points, magnitude ~ number of points
; For clustered points, magnitude ~ 1

(display "Similarity matrix Z:") (newline)
(for-each (lambda (row)
  (display "  ") (display row) (newline)) Z)
(display "Well-separated: magnitude ~ 3") (newline)
(display "If all distances -> 0: magnitude -> 1") (newline)
(display "Same structure gives entropy via different enrichment.")

Why this matters

The unification means entropy and magnitude aren't separate concepts that happen to look similar. They're the same functor evaluated on different enriched categories. 🍞 Baez, Fritz, and Leinster characterized entropy as the unique information loss measure. 🍞 Leinster characterized magnitude as the unique notion of size for metric spaces. Chen and Vigneaux show these uniqueness results are two faces of one theorem.

Scheme

; The bridge: same construction, two enrichments
; Enrichment [0, inf) -> magnitude (geometric size)
; Enrichment [0, 1]   -> exp(entropy) (information content)

(define (exp-entropy p)
  (exp (- (apply + (map (lambda (pi)
    (if (> pi 0) (* pi (log pi)) 0)) p)))))

; Uniform: both agree
(define uniform-4 '(0.25 0.25 0.25 0.25))
(display "uniform(4): exp(H) = ")
(display (exp-entropy uniform-4))
(display ", magnitude of 4 distinct points = 4")
(newline)

; Non-uniform: exp(H) gives effective size
(define skewed '(0.9 0.05 0.03 0.02))
(display "skewed: exp(H) = ")
(display (exp-entropy skewed))
(newline)
(display "Fewer effective outcomes — same as fewer effective points.")

Notation reference

Paper	Scheme	Meaning
\|A\|	; magnitude (Euler characteristic)	Size of enriched category A
H(p)	(shannon-entropy p)	Shannon entropy
log\|A\| = H	(log magnitude) = H	The unification (uniform case)
Z_ij = e^(-d_ij)	(similarity d)	Similarity matrix from distances
V-Cat	; category enriched over V	Enriched category (V = metric or probability)

Neighbors

Other paper pages

🍞 Leinster 2021 — magnitude and diversity (the metric side)
🍞 Baez, Fritz, Leinster 2011 — entropy characterization (the information side)
🍞 Sato 2023 — divergences on monads (related information measures)

Foundations (Wikipedia)

Translation notes

The examples demonstrate the uniform-case identity (log magnitude = entropy) and the intuition behind weighted magnitude. Chen and Vigneaux's actual construction works through the Euler characteristic of categories enriched over a quantale, unifying the metric-space and probability-space cases via a change-of-base functor. For example: the "same construction, two enrichments" example on this page computes exp(H) and compares it to a count. In the paper, the comparison is between two Euler characteristics: one for a [0,∞)-enriched category (metric space) and one for a [0,1]-enriched category (probability space), connected by a lax monoidal functor between the enrichment bases. The numerical agreement is the same; the functorial explanation is not.

Uniform case: Exact. Non-uniform and magnitude examples: Simplified.

Ready for the real thing? arxiv

Read the paper. Start at §2 for the enriched category setup, §4 for the magnitude-entropy theorem.

Framework connection: The magnitude-entropy unification gives the Natural Framework's information budget a single invariant — compression ratio is the same functor applied to different enriched categories. (The Natural Framework)

← Ho, Wu 2026 · 20 of 21 by june.kim