Entropy and Diversity

Tom Leinster · 2021 · Cambridge University Press

Prereqs: basic probability (what a distribution is). 🍞 Baez, Fritz, Leinster 2011 provides the entropy characterization.

Hill diversity numbers form a one-parameter family: at q=0 you count species, at q=1 you get the exponential of Shannon entropy, at q=2 you get Simpson's reciprocal. Magnitude extends this to metric spaces — a single number measuring the "effective size" of a space.

Hill diversity numbers

Given a probability distribution p over n species, the Hill number of order q is:

D_q = (Σ pᵢ^q)^1/(1−q)

This is the "effective number of species." A community with 10 equally-common species has D_q = 10 for all q. A community dominated by one species has D_q close to 1.

Scheme

; Hill diversity number of order q
; D_q(p) = (sum(p_i^q))^(1/(1-q))

(define (hill-number p q)
  (if (= q 1)
    ; Limit as q -> 1: exp of Shannon entropy
    (exp (- (apply + (map (lambda (pi)
      (if (> pi 0) (* pi (log pi)) 0)) p))))
    (expt (apply + (map (lambda (pi) (expt pi q)) p))
          (/ 1 (- 1 q)))))

; Uniform distribution: 5 equally common species
(define uniform '(0.2 0.2 0.2 0.2 0.2))
(display "uniform(5): D0=")
(display (hill-number uniform 0))
(display " D1=")
(display (hill-number uniform 1))
(display " D2=")
(display (hill-number uniform 2))
(newline)

; Dominated distribution
(define dominated '(0.9 0.05 0.03 0.01 0.01))
(display "dominated:  D0=")
(display (hill-number dominated 0))
(display " D1=")
(display (hill-number dominated 1))
(display " D2=")
(display (hill-number dominated 2))

q=0: species richness

At q=0, all nonzero probabilities contribute equally. D₀ just counts the number of species present. Rare and common species count the same.

Scheme

; D_0 = number of species with nonzero probability

(define (d0 p)
  (length (filter (lambda (pi) (> pi 0)) p)))

(display "5 species, one rare: ")
(display (d0 '(0.5 0.3 0.1 0.09 0.01)))
(newline)
(display "5 species, one absent: ")
(display (d0 '(0.5 0.3 0.1 0.1 0.0)))
(newline)
; q=0 is a blunt instrument — it ignores how common each species is

q=1: exponential of Shannon entropy

At q=1 (the limit), D₁ = exp(H), where H is Shannon entropy. This is the "effective number of equally-common species" — entropy converted from bits to a count.

Scheme

; D_1 = exp(H) where H = -sum(p_i log p_i)

(define (shannon-entropy p)
  (- (apply + (map (lambda (pi)
    (if (> pi 0) (* pi (log pi)) 0)) p))))

(define (d1 p) (exp (shannon-entropy p)))

(define p '(0.25 0.25 0.25 0.25))
(display "4 uniform: H=")
(display (shannon-entropy p))
(display " D1=")
(display (d1 p))
(newline)

(define q '(0.7 0.1 0.1 0.1))
(display "4 skewed:  H=")
(display (shannon-entropy q))
(display " D1=")
(display (d1 q))
; D1 < 4 because one species dominates

q=2: Simpson's reciprocal

At q=2, D₂ = 1/Σpᵢ². This is the reciprocal of the probability that two randomly chosen individuals belong to the same species. Dominant species are heavily weighted.

Scheme

; D_2 = 1 / sum(p_i^2) — Simpson's reciprocal

(define (d2 p)
  (/ 1 (apply + (map (lambda (pi) (* pi pi)) p))))

(define uniform '(0.2 0.2 0.2 0.2 0.2))
(define skewed '(0.8 0.05 0.05 0.05 0.05))

(display "uniform(5): D2 = ") (display (d2 uniform)) (newline)
(display "skewed:     D2 = ") (display (d2 skewed)) (newline)
; Skewed community has low effective diversity
; because most random pairs hit the dominant species

Magnitude — effective size of a metric space

Magnitude generalizes diversity to metric spaces. Given a finite metric space with distance matrix d, the magnitude is the sum of the weight vector w satisfying Zw = 1, where Z_ij = e^(-d_ij). It measures the "effective number of points" — points close together contribute less than points far apart.

Scheme

; Magnitude of a finite metric space
; Z_ij = exp(-d_ij), solve Zw = 1, magnitude = sum(w)

; Simple example: 3 points on a line at positions 0, 1, t
(define (magnitude-3 t)
  ; Z matrix for points at 0, 1, t
  ; Z = ((1, e^-1, e^-t), (e^-1, 1, e^-(t-1)), (e^-t, e^-(t-1), 1))
  ; For t=1: two points coincide, magnitude should be 2
  ; For t>>1: three well-separated points, magnitude ~ 3
  (let* ((z01 (exp -1)) (z02 (exp (- t))) (z12 (exp (- (abs (- t 1)))))
         ; Solve 3x3 system (simplified for demonstration)
         (det (+ 1 (* 2 z01 z02 z12) (- (* z01 z01)) (- (* z02 z02)) (- (* z12 z12)))))
    (/ (+ 1 1 1
          (- z01) (- z01) (- z02) (- z02) (- z12) (- z12)
          (* 2 z01 z02) (* 2 z01 z12) (* 2 z02 z12)
          (- (* 2 z01 z02 z12)))
       det)))

; More intuitive: just show the concept
(display "3 points at 0, 1, 2:") (newline)
(display "  well-separated -> magnitude ~ 3") (newline)
(display "3 points at 0, 0.01, 0.02:") (newline)
(display "  clustered -> magnitude ~ 1") (newline)
(display "Magnitude counts effective distinct points.")

; Magnitude of a finite metric space
; Z_ij = exp(-d_ij), solve Zw = 1, magnitude = sum(w)

; Simple example: 3 points on a line at positions 0, 1, t
(define (magnitude-3 t)
  ; Z matrix for points at 0, 1, t
  ; Z = ((1, e^-1, e^-t), (e^-1, 1, e^-(t-1)), (e^-t, e^-(t-1), 1))
  ; For t=1: two points coincide, magnitude should be 2
  ; For t>>1: three well-separated points, magnitude ~ 3
  (let* ((z01 (exp -1)) (z02 (exp (- t))) (z12 (exp (- (abs (- t 1)))))
         ; Solve 3x3 system (simplified for demonstration)
         (det (+ 1 (* 2 z01 z02 z12) (- (* z01 z01)) (- (* z02 z02)) (- (* z12 z12)))))
    (/ (+ 1 1 1
          (- z01) (- z01) (- z02) (- z02) (- z12) (- z12)
          (* 2 z01 z02) (* 2 z01 z12) (* 2 z02 z12)
          (- (* 2 z01 z02 z12)))
       det)))

; More intuitive: just show the concept
(display "3 points at 0, 1, 2:") (newline)
(display "  well-separated -> magnitude ~ 3") (newline)
(display "3 points at 0, 0.01, 0.02:") (newline)
(display "  clustered -> magnitude ~ 1") (newline)
(display "Magnitude counts effective distinct points.")

Notation reference

Paper	Scheme	Meaning
ᵍD(p)	(hill-number p q)	Hill diversity of order q
H(p)	(shannon-entropy p)	Shannon entropy
\|A\|	; magnitude	Magnitude (effective size)
Z_ij = e^(-d_ij)	(exp (- d))	Similarity matrix entry
w	; weight vector, Zw = 1	Magnitude weights

Neighbors

Other paper pages

🍞 Baez, Fritz, Leinster 2011 — characterizes Shannon entropy via functoriality
🍞 Chen, Vigneaux 2023 — unifies magnitude and entropy categorically
🍞 Sato 2023 — divergences on monads (related information measures)

Related foundations

∞ Lebl Ch.8 Metric Spaces — the metric space structure that magnitude measures

Foundations (Wikipedia)

Translation notes

The Hill number computations are exact for finite distributions. The magnitude example is conceptual — solving the linear system Zw = 1 for arbitrary metric spaces requires matrix inversion, which the examples only sketch. For example: the magnitude discussion describes 3 points on a line. In the book, magnitude is defined for arbitrary enriched categories via the Euler characteristic of a category — a construction that subsumes metric spaces, posets, and graphs under one definition. The Hill numbers are the same; the categorical generalization of magnitude is not.

Hill number examples: Exact. Magnitude example: Analogy.

Ready for the real thing? The book is published by Cambridge University Press (2021). Start at Chapter 2 for Hill numbers, Chapter 6 for magnitude.

Framework connection: Hill diversity and magnitude measure the Natural Framework's information budget — exp(entropy) is the effective number of distinguishable states at each pipeline stage. (The Natural Framework)

← Kidney, Wu 2021 · 13 of 21 by june.kim Smithe 2021 · 15 of 21 →