# Entropy (information theory)
## Microsim
### Live player
<div class="microsim-player">
<iframe src="https://editor.p5js.org/sciencenibber/full/alvczWLrB" width="100%" height="620" frameborder="0" sandbox="allow-scripts allow-same-origin"></iframe>
</div>
<div class="microsim-fallback">
<img src="Microsims/thumbs/Entropy_(information_theory).png" alt="Entropy_(information_theory) microsim poster" style="width:100%;border:1px solid #4445;border-radius:6px;">
<p><em>Live microsim (desktop) · <a href="https://editor.p5js.org/sciencenibber/sketches/alvczWLrB">open sketch in the p5.js editor</a></em></p>
</div>
**Editor URL:** https://editor.p5js.org/sciencenibber/sketches/alvczWLrB
**Description (100 words):**
This Pattern E sketch makes the Shannon-entropy formula H(X) = - Sum_i p_i * log2(p_i) literally visible by decomposing it into three stacked rows of bars: probability p_i (top), surprisal I_i = -log2(p_i) (middle, in bits), and per-symbol contribution p_i * I_i (bottom). The contributions sum to H(X), shown live on a stacked vertical gauge with a red H_max = log2(N) reference line. Drag any bar slider and the others auto-renormalize so Sum p stays 1; choose alphabet size N in [2, 8] and load preset distributions (uniform, geometric, English-letter, DNA, deterministic). A binary-entropy-curve inset traces H(p) with a marker that tracks p_0 when N = 2.
```js
// =============================================================================
// Entropy_(information_theory).js — Wikitube microsim
// Article: Entropy_(information_theory)
// URL : en.wikitube.io/wiki/Entropy_(information_theory)
// Pattern: Information §10 Pattern E — Entropy meter
//
// WHAT THIS SHOWS
// ---------------
// Most "entropy meter" sketches present H(X) = - Sum p_i * log2(p_i) as a
// single readout. This one decomposes the formula visually so the reader
// SEES why H is what it is. For each symbol i three stacked bars are drawn:
//
// Row 1 (top) : p_i — probability of symbol i
// Row 2 (middle) : I_i = -log2(p_i) — surprisal (bits per occurrence)
// Row 3 (bottom) : p_i * I_i — contribution to H
//
// The Shannon entropy is then literally the sum of the bottom row:
//
// H(X) = Sum_i p_i * (-log2 p_i) (bits)
//
// A stacked accumulator strip on the right visualises that sum in real time.
// A small binary-entropy curve in the lower-right panel traces
// H(p) = -p log2 p - (1-p) log2(1-p), with a marker at the current value
// whenever the alphabet is binary.
//
// PARAMETERS
// ----------
// N — alphabet size, integer in [2, 8] (slider, top-right)
// p_i — per-symbol probability, real in [0, 1] (bar sliders)
// I_i — surprisal -log2(p_i) of symbol i (derived)
// H — Shannon entropy in bits (readout, BL)
// H_max — log2(N), the uniform-distribution maximum (readout, BL)
// eta — efficiency H / H_max in [0, 1] (readout, BL)
//
// PITFALLS HONORED (per pitfalls.md, 2026-04-30)
// ----------------------------------------------
// * disableFriendlyErrors keeps the FES off in production.
// * ASCII-only inside string / template / text() arguments. Anything fancy
// (Sigma, mu, log subscripts) lives in COMMENTS only.
// * Slider thumbs DO NOT sit on top of HUD readouts — controls are in their
// own bottom strip and readouts have their own band above them.
// * log base 2 done via Math.log2 (NOT Math.log, which is natural log).
// * Edge case: p_i = 0 contributes 0 to H (limit p log p -> 0 as p -> 0).
// * Renormalisation on slider drag keeps Sum p_i exactly 1; otherwise H is
// meaningless.
//
// =============================================================================
const ARTICLE = "Entropy_(information_theory)";
p5.disableFriendlyErrors = true;
// -- Information-room palette (P5_JS_EDITOR §10 "Shared constants") ----------
const BG = 246;
const INK = [40, 48, 60];
const BAR = [70, 130, 200]; // probability bar (row 1)
const BAR_HI = [220, 110, 60]; // bar currently being edited
const SUR = [120, 80, 180]; // surprisal bar (row 2, purple)
const CONTRIB = [220, 110, 60]; // contribution bar (row 3, orange)
const EDGE = [120, 130, 150]; // axis / gridline
const LOWP = [200, 205, 215]; // background tick line color
const ACCENT = [80, 180, 120]; // efficiency / sum gauge accent (green)
// -- Layout -----------------------------------------------------------------
// Canvas 720 x 520. Three horizontal bands of bars, then a control strip.
// Right-edge column reserved for the H accumulator and binary-entropy panel.
const W = 720, H = 520;
const M = { left: 80, right: 200, top: 70, bottom: 130 };
// Three bar rows occupy the main panel between M.top and (H - M.bottom).
// Each row gets ~1/3 of that vertical band, with a small gap between rows.
function rowBounds(row /* 0..2 */) {
const yTop = M.top + 4;
const yBot = H - M.bottom - 4;
const total = yBot - yTop;
const rowH = total / 3 - 8;
return {
y0: yTop + row * (total / 3),
y1: yTop + row * (total / 3) + rowH,
};
}
// -- DOM controls (created in setup()) --------------------------------------
let nSlider; // N — alphabet size
let presetSel; // distribution preset dropdown
let barSliders = []; // per-symbol probability sliders, length == N
let lastEditedIdx = -1; // index of the slider the user is dragging right now
// -- State ------------------------------------------------------------------
let probs = [0.5, 0.5]; // start with a fair coin (N=2)
let symbols = ["A", "B"]; // labels under the bars
// =============================================================================
// setup() — build canvas, DOM controls, initial layout
// =============================================================================
function setup() {
createCanvas(W, H);
pixelDensity(2); // crisp text on retina
textFont("system-ui");
// Alphabet-size slider in the top-right strip.
nSlider = createSlider(2, 8, 2, 1);
nSlider.position(W - 200, 12);
nSlider.style("width", "120px");
nSlider.input(onAlphabetSizeChanged);
// Preset dropdown.
presetSel = createSelect();
presetSel.position(W - 200, 38);
presetSel.style("width", "180px");
presetSel.option("uniform");
presetSel.option("biased (geometric)");
presetSel.option("English letters (top N)");
presetSel.option("DNA bases (typical)");
presetSel.option("deterministic");
presetSel.changed(onPresetChanged);
rebuildBarSliders(2); // start with N=2, fair coin
}
// =============================================================================
// draw() — render frame
// =============================================================================
function draw() {
background(BG);
// Read every parameter once at the top of draw() into named locals — the
// P5_JS_EDITOR §10 control-layout convention. The math step then references
// them by information-theory name.
const N = probs.length;
const surprisal = probs.map(p => (p > 0 ? -Math.log2(p) : 0)); // I_i, bits
const contrib = probs.map((p, i) => p * surprisal[i]); // p_i * I_i
const H_bits = contrib.reduce((a, b) => a + b, 0); // Shannon H
const H_max = Math.log2(N); // uniform max
const eta = H_max > 0 ? H_bits / H_max : 0;
drawDecompositionPanel(N, surprisal, contrib);
drawHGauge(H_bits, H_max, contrib);
drawBinaryCurvePanel(probs);
drawHud(N, H_bits, H_max, eta);
}
// =============================================================================
// Math primitives
// =============================================================================
// renormalize(p, fixedIdx, fixedVal)
// The slider for `fixedIdx` was just set to `fixedVal`. Distribute the
// remaining mass (1 - fixedVal) over the OTHER bars in proportion to
// their current heights, so Sum p stays exactly 1. If the others are
// all zero, fall back to a uniform spread of the remaining mass.
function renormalize(p, fixedIdx, fixedVal) {
const v = constrain(fixedVal, 0, 1);
const remaining = 1 - v;
let othersSum = 0;
for (let i = 0; i < p.length; i++) if (i !== fixedIdx) othersSum += p[i];
const next = p.slice();
next[fixedIdx] = v;
if (othersSum > 1e-9) {
const k = remaining / othersSum;
for (let i = 0; i < p.length; i++) if (i !== fixedIdx) next[i] = p[i] * k;
} else {
const each = remaining / (p.length - 1);
for (let i = 0; i < p.length; i++) if (i !== fixedIdx) next[i] = each;
}
return next;
}
// =============================================================================
// DOM-control change handlers
// =============================================================================
function onAlphabetSizeChanged() {
const N = nSlider.value();
rebuildBarSliders(N);
probs = new Array(N).fill(1 / N); // reset to uniform
syncSlidersToProbs();
}
function onPresetChanged() {
const N = probs.length;
probs = presetDistribution(presetSel.value(), N);
syncSlidersToProbs();
}
function presetDistribution(name, N) {
let raw;
if (name === "uniform") {
raw = new Array(N).fill(1 / N);
} else if (name === "biased (geometric)") {
raw = [];
for (let i = 0; i < N; i++) raw.push(Math.pow(0.5, i)); // (1/2)^i
} else if (name === "English letters (top N)") {
// Lewand 2000: E T A O I N S H R L D C U M
const eng = [0.1270, 0.0905, 0.0817, 0.0751, 0.0697, 0.0675,
0.0633, 0.0609];
raw = eng.slice(0, N);
} else if (name === "DNA bases (typical)") {
// Approximate human-genome composition: A 0.295, T 0.295, G 0.205, C 0.205.
const dna = [0.295, 0.295, 0.205, 0.205, 0.0001, 0.0001, 0.0001, 0.0001];
raw = dna.slice(0, N);
} else if (name === "deterministic") {
raw = new Array(N).fill(0);
raw[0] = 1; // all mass on first symbol
} else {
raw = new Array(N).fill(1 / N);
}
const s = raw.reduce((a, b) => a + b, 0);
return s > 0 ? raw.map(v => v / s) : new Array(N).fill(1 / N);
}
function rebuildBarSliders(N) {
for (const s of barSliders) s.remove();
barSliders = [];
symbols = ["A", "B", "C", "D", "E", "F", "G", "H"].slice(0, N);
const x0 = M.left, w = W - M.left - M.right;
for (let i = 0; i < N; i++) {
const s = createSlider(0, 1, 1 / N, 0.001);
const cx = x0 + (i + 0.5) * (w / N);
s.position(cx - 35, H - M.bottom + 70);
s.style("width", "70px");
// input() not changed() — chart updates while the user is dragging.
s.input(((idx) => () => onBarSliderInput(idx))(i));
barSliders.push(s);
}
}
function onBarSliderInput(idx) {
lastEditedIdx = idx;
const v = barSliders[idx].value();
probs = renormalize(probs, idx, v);
// Push the (possibly altered) other-bar values back into the DOM sliders so
// visible thumb positions agree with the model. Skip the one being dragged.
for (let i = 0; i < probs.length; i++) {
if (i === idx) continue;
barSliders[i].value(probs[i]);
}
}
function syncSlidersToProbs() {
for (let i = 0; i < probs.length; i++) barSliders[i].value(probs[i]);
}
// =============================================================================
// Rendering — left/main panel: three-row decomposition
// =============================================================================
function drawDecompositionPanel(N, surprisal, contrib) {
const x0 = M.left, w = W - M.left - M.right;
const cell = w / N;
const barW = cell * 0.55;
// Row 0: probability p_i in [0, 1].
drawRow(0, "p_i", N, cell, barW, x0, probs,
(v) => v, BAR, "p_i", 1.0, 0.25);
// Row 1: surprisal I_i = -log2 p_i. Cap visual scale at 4 bits so a
// probability of 1/16 fills the row; rarer events extend off the bar
// but the numeric label still renders the actual surprisal.
drawRow(1, "I_i = -log2 p_i", N, cell, barW, x0, surprisal,
(v) => Math.min(v / 4, 1), SUR, "bits", 4.0, 1.0);
// Row 2: contribution p_i * I_i. Cap at 1 bit per symbol since a single
// bin can never contribute more than 1/e * log2(e) ~ 0.531 bits when the
// entropy is maximized over its own probability, but the SUM Sum_i p_i I_i
// can still reach log2(N). Headroom of 1 bit keeps bars readable.
drawRow(2, "p_i * I_i (contribution to H)", N, cell, barW, x0, contrib,
(v) => Math.min(v / 1, 1), CONTRIB, "bits", 1.0, 0.25);
}
// drawRow(row, title, N, cell, barW, x0, values, normFn, color, unitLabel,
// scaleMax, gridStep) — render one of the three decomposition rows.
//
// * row : 0 (top, probability), 1 (middle, surprisal), 2 (bottom, contribution)
// * title : header text drawn left of the row
// * values : length-N array of the row's quantities
// * normFn(v) : map a value into [0, 1] for visual height
// * scaleMax : the quantity at the row's full height (for gridline labels)
// * gridStep : tick spacing in the row's units
function drawRow(row, title, N, cell, barW, x0, values, normFn, color, unitLabel,
scaleMax, gridStep) {
const { y0, y1 } = rowBounds(row);
const rowH = y1 - y0;
// Row title and unit label, drawn in the left margin.
noStroke(); fill(INK); textSize(11); textAlign(RIGHT, CENTER);
text(title, x0 - 10, y0 + rowH * 0.35);
fill(...EDGE); textSize(10);
text("(" + unitLabel + ")", x0 - 10, y0 + rowH * 0.65);
// Row gridlines + tick labels.
stroke(LOWP); strokeWeight(1);
for (let v = 0; v <= scaleMax; v += gridStep) {
const yy = y1 - (v / scaleMax) * rowH;
line(x0, yy, x0 + N * cell, yy);
}
// Row baseline + left axis.
stroke(EDGE); strokeWeight(1.2);
line(x0, y1, x0 + N * cell, y1);
line(x0, y0, x0, y1);
// Tick label at the row's max.
noStroke(); fill(EDGE); textSize(9); textAlign(RIGHT, CENTER);
text(nf(scaleMax, 1, 1), x0 - 4, y0);
text("0", x0 - 4, y1);
// Bars.
for (let i = 0; i < N; i++) {
const cx = x0 + (i + 0.5) * cell;
const t = normFn(values[i]); // height in [0, 1]
const top = y1 - t * rowH;
const isHi = (i === lastEditedIdx);
fill(...(isHi ? BAR_HI : color));
noStroke();
rect(cx - barW / 2, top, barW, y1 - top, 3);
// Numeric value above each bar.
fill(INK); textSize(10); textAlign(CENTER, BOTTOM);
const display = isFinite(values[i]) ? nf(values[i], 1, 2) : "inf";
text(display, cx, top - 2);
// Symbol labels under row 0 only.
if (row === 0) {
fill(INK); textSize(13); textAlign(CENTER, TOP);
text(symbols[i], cx, y1 + 6);
}
}
textAlign(LEFT, BASELINE);
}
// =============================================================================
// Rendering — H accumulator gauge (right column, top half)
// =============================================================================
// Stacked vertical bar to the right of the decomposition panel. Each contrib
// segment stacks on top of the previous one; the total height is H. A tick
// at log2(N) marks the maximum entropy for the current alphabet size.
function drawHGauge(H_bits, H_max, contrib) {
const x = W - M.right + 32;
const w = 28;
const yTop = M.top + 4;
const yBot = H - M.bottom - 70;
const gh = yBot - yTop;
// Cap visual scale at log2(8) = 3 bits (the max possible H_max in this sketch).
const SCALE_MAX = Math.log2(8);
// Background track + ticks at every integer bit.
noStroke(); fill(LOWP);
rect(x, yTop, w, gh, 3);
stroke(EDGE); strokeWeight(1);
for (let b = 0; b <= 3; b++) {
const yy = yBot - (b / SCALE_MAX) * gh;
line(x - 4, yy, x + w + 4, yy);
noStroke(); fill(INK); textSize(10); textAlign(LEFT, CENTER);
text(b + " bit" + (b === 1 ? "" : "s"), x + w + 8, yy);
stroke(EDGE);
}
// H_max tick (dashed-ish) — drawn as a bold red line.
const yMax = yBot - (H_max / SCALE_MAX) * gh;
stroke(220, 60, 60); strokeWeight(1.6);
line(x - 6, yMax, x + w + 6, yMax);
noStroke(); fill(220, 60, 60); textSize(10); textAlign(LEFT, CENTER);
text("H_max", x + w + 8, yMax - 12);
// Stack the contributions p_i * I_i from the bottom up.
let yCur = yBot;
for (let i = 0; i < contrib.length; i++) {
const segH = (contrib[i] / SCALE_MAX) * gh;
if (segH < 0.5) continue;
// Alternate hue slightly per symbol for stack visibility.
const alpha = 200 - i * 18;
fill(CONTRIB[0], CONTRIB[1], CONTRIB[2], alpha);
noStroke();
rect(x, yCur - segH, w, segH);
yCur -= segH;
}
// Outline.
noFill(); stroke(EDGE); strokeWeight(1);
rect(x, yTop, w, gh, 3);
// Numeric H readout at the gauge head.
noStroke(); fill(INK); textSize(11); textAlign(LEFT, BOTTOM);
text("H = Sum p_i * I_i", x - 8, yTop - 16);
text(" = " + H_bits.toFixed(3) + " bits", x - 8, yTop - 4);
textAlign(LEFT, BASELINE);
}
// =============================================================================
// Rendering — binary entropy curve panel (right column, bottom half)
// =============================================================================
// Tiny inset showing H(p) = -p log2 p - (1-p) log2(1-p) for p in [0, 1].
// When N == 2, a marker is drawn at the current p_0; otherwise the panel
// shows only the curve as a reference for what the binary case looks like.
function drawBinaryCurvePanel(p) {
const x = W - M.right + 16;
const y = H - M.bottom - 50;
const w = M.right - 32;
const h = 50;
// Panel background.
noStroke(); fill(255);
rect(x, y, w, h, 3);
stroke(EDGE); strokeWeight(1); noFill();
rect(x, y, w, h, 3);
// Title.
noStroke(); fill(INK); textSize(10); textAlign(LEFT, TOP);
text("binary H(p)", x + 4, y + 2);
// Curve.
noFill(); stroke(...SUR); strokeWeight(1.4);
beginShape();
const STEPS = 64;
for (let s = 0; s <= STEPS; s++) {
const pp = s / STEPS;
const Hp = (pp <= 0 || pp >= 1) ? 0 :
-pp * Math.log2(pp) - (1 - pp) * Math.log2(1 - pp);
const xx = x + 4 + (w - 8) * pp;
const yy = y + h - 4 - Hp * (h - 14); // H_max = 1 fills the panel
vertex(xx, yy);
}
endShape();
// Marker only when N=2; the curve is for binary entropy specifically.
if (p.length === 2) {
const pp = p[0];
const Hp = (pp <= 0 || pp >= 1) ? 0 :
-pp * Math.log2(pp) - (1 - pp) * Math.log2(1 - pp);
const mx = x + 4 + (w - 8) * pp;
const my = y + h - 4 - Hp * (h - 14);
noStroke(); fill(...BAR_HI);
circle(mx, my, 6);
fill(INK); textSize(9); textAlign(LEFT, BOTTOM);
text("p=" + pp.toFixed(2) + ", H=" + Hp.toFixed(2), x + 4, y + h - 2);
} else {
fill(...EDGE); textSize(9); textAlign(LEFT, BOTTOM);
text("(set N=2 to see marker)", x + 4, y + h - 2);
}
textAlign(LEFT, BASELINE);
}
// =============================================================================
// HUD — Wikitube watermark (Betterfire Standard rule 2)
// =============================================================================
function drawHud(N, H_bits, H_max, eta) {
// ---- Top-left: title + Wikitube URL ----
noStroke(); fill(20);
textAlign(LEFT, TOP); textSize(20);
text("Entropy (information theory)", 16, 14);
fill(110); textSize(12);
text("Wikitube microsim - en.wikitube.io/wiki/" + ARTICLE, 16, 40);
// ---- Top-right: control hints ----
fill(110); textSize(11); textAlign(RIGHT, TOP);
text("slider: N (alphabet size)", W - 16, 14);
text("dropdown: preset distribution", W - 16, 30);
text("bar sliders: per-symbol p_i (auto-renormalized)", W - 16, 46);
// ---- Bottom-left: live readouts in canonical info-theory symbols ----
fill(INK); textAlign(LEFT, BOTTOM); textSize(12);
text("N = " + N, 16, H - 78);
text("H(X) = " + H_bits.toFixed(4) + " bits", 16, H - 62);
text("H_max = log2(" + N + ") = " + H_max.toFixed(4) + " bits", 16, H - 46);
text("efficiency = H / H_max = " + eta.toFixed(4), 16, H - 30);
// ---- Bottom-right: canonical equation footer ----
fill(80); textAlign(RIGHT, BOTTOM); textSize(11);
text("H(X) = - Sum_i p_i * log2(p_i) = E[ I(X) ] (bits)", W - 16, H - 78);
text("uniform p_i = 1/N => H = log2(N) = H_max", W - 16, H - 62);
text("deterministic p = (1, 0, ..., 0) => H = 0", W - 16, H - 46);
textAlign(LEFT, BASELINE);
}
```
## Links (Wikipedia order)
<!-- injected from _registry/childlinks/Entropy_(information_theory).json (2026-07-30T02:09:12Z) -->
`842_(compression_algorithm)` · `A-law_algorithm` · `A_Mathematical_Theory_of_Communication` · `Adaptive_Huffman_coding` · `Adaptive_coding` · `Adaptive_differential_pulse-code_modulation` · `Algebraic_code-excited_linear_prediction` · `Approximate_entropy` · `Arithmetic_coding` · `Asymmetric_numeral_systems` · `Asymptotic_equipartition_property` · `Audio_codec` · `Average_bitrate` · `Axiom` · `Bayesian_inference` · `Bernoulli_process` · `Binary_entropy_function` · `Binary_logarithm` · `Bit` · `Bit_rate` · `Boltzmann's_entropy_formula` · `Boltzmann_constant` · `Brotli` · `Burrows–Wheeler_transform` · `Byte-pair_encoding` · `Bzip2` · `Canonical_Huffman_code` · `Chain_code` · `Channel_capacity` · `Characterization_(mathematics)` · `Checksum` · `Chroma_subsampling` · [[Claude_Shannon]] · `Code-excited_linear_prediction` · `Coding_tree_unit` · `Color_space` · `Combinatorics` · [[Communication_channel]] · [[Companding]] · `Compressed_data_structure` · `Compressed_suffix_array` · `Compression_artifact` · `Computer_program` · `Concave_function` · [[Conditional_entropy]] · `Conditional_mutual_information` · `Conditional_probability` · `Constant_bitrate` · `Context_mixing` · `Context_tree_weighting` · `Continuous_function` · `Convex_conjugate` · [[Convolution]] · `Counting_measure` · `Cross-entropy` · `Cryptanalysis` · `Data_communication` · [[Data_compression]] · `Data_compression_symmetry` · `Daubechies_wavelet` · `David_A._Huffman` · `David_Ellerman` · `David_J._C._MacKay` · `Deblocking_filter` · `Decision_tree_learning` · `Deflate` · `Delta_encoding` · [[Delta_modulation]] · `Density_matrix` · `Dictionary_coder` · [[Differential_entropy]] · [[Differential_equation]] · [[Differential_pulse-code_modulation]] · `Directed_information` · [[Discrete_cosine_transform]] · `Discrete_sine_transform` · [[Discrete_wavelet_transform]] · `Display_resolution` · `Diversity_index` · `Dominance_(ecology)` · `Duality_(mathematics)` · `Dynamic_Markov_compression` · [[Dynamic_range]] · [[Dynamical_system]] · `E_(mathematical_constant)` · `Edwin_Thompson_Jaynes` · `Elias_gamma_coding` · `Embedded_zerotrees_of_wavelet_transforms` · `Encyclopedia_of_Mathematics` · [[Entropy]] · `Entropy_(disambiguation)` · `Entropy_(statistical_thermodynamics)` · `Entropy_coding` · `Entropy_estimation` · `Entropy_in_thermodynamics_and_information_theory` · `Entropy_power_inequality` · `Entropy_rate` · `Erica_Klarreich` · `Eta` · `Event_(probability_theory)` · [[Expected_value]] · `Exponential-Golomb_coding` · `FM-index` · [[Fast_Fourier_transform]] · `Fibonacci_coding` · `Film_frame` · `Fisher_information` · `Fourier_transform` · `Fractal_compression` · `Frame_rate` · `Generalized_relative_entropy` · `Geometry_of_Quantum_States` · `Golomb_coding` · `Grammar-based_code` · `Graph_entropy` · `H-theorem` · `Hamming_distance` · `Hartley_(unit)` · `History_of_entropy` · `History_of_information_theory` · `Huffman_coding` · `Hutter_Prize` · `ISBN` · [[Image_compression]] · `Image_resolution` · `Incremental_encoding` · `Information_content` · `Information_dimension` · `Information_fluctuation_complexity` · `Information_geometry` · [[Information_theory]] · `Interlaced_video` · [[John_von_Neumann]] · [[Joint_entropy]] · `Joy_A._Thomas` · `János_Aczél_(mathematician)` · `Kolmogorov_complexity` · `Kullback–Leibler_divergence` · `LHA_(file_format)` · `LZ4_(compression_algorithm)` · `LZ77_and_LZ78` · `LZFSE` · `LZMA` · `LZRW` · `LZWL` · `LZX` · `Landauer's_principle` · `Lapped_transform` · `Latency_(audio)` · `Lebesgue_measure` · `Lempel–Ziv–Oberhumer` · `Lempel–Ziv–Stac` · `Lempel–Ziv–Storer–Szymanski` · `Lempel–Ziv–Welch` · `Levenshtein_coding` · `Levenshtein_distance` · `Limit_of_a_function` · `Limiting_density_of_discrete_points` · `Line_spectral_pairs` · `Linear_predictive_coding` · `Liouville_function` · `LogSumExp` · `Log_area_ratio` · `Logarithm` · `Logistic_regression` · `Lossless_compression` · `Lossy_compression` · `Ludwig_Boltzmann` · [[Machine_learning]] · `Macroblock` · `Mark_Adler` · `Markov_model` · `Maximum_entropy_thermodynamics` · `Maxwell's_demon` · `Microstate_(statistical_mechanics)` · `Modified_Huffman_coding` · `Modified_discrete_cosine_transform` · `Motion_compensation` · `Motion_estimation` · `Move-to-front_transform` · `Mu-law_algorithm` · `Mutual_information` · `NLab` · [[Nat_(unit)]] · `Natural_logarithm` · `Noisy-channel_coding_theorem` · [[Nyquist–Shannon_sampling_theorem]] · `One-time_pad` · `PAQ` · `Partition_of_a_set` · `Peak_signal-to-noise_ratio` · `Permutation` · `Perplexity` · `Phil_Katz` · `Pigeonhole_principle` · `Pixel` · `PlanetMath` · `Prediction_by_partial_matching` · `Prefix_code` · `Principle_of_maximum_entropy` · `Prior_probability` · [[Probability_density_function]] · `Probability_distribution` · `Probability_space` · `Proportionality_(mathematics)` · `Psychoacoustics` · `Pyramid_(image_processing)` · `Quanta_Magazine` · `Quantities_of_information` · `Quantization_(image_processing)` · [[Quantization_(signal_processing)]] · `Random_variable` · `Randomness` · `Range_coding` · `Rate–distortion_theory` · `Re-Pair` · `Redundancy_(information_theory)` · `Rolf_Landauer` · `Rosetta_Code` · `Run-length_encoding` · `Rényi_entropy` · `Sample_entropy` · [[Sampling_(signal_processing)]] · [[Science_(journal)]] · [[Second_law_of_thermodynamics]] · `Sequitur_algorithm` · `Set_partitioning_in_hierarchical_trees` · `Shannon's_source_coding_theorem` · `Shannon_(unit)` · `Shannon_coding` · `Shannon–Fano_coding` · `Shannon–Fano–Elias_coding` · `Shannon–Hartley_theorem` · [[Signal_processing]] · `Silence_compression` · `Smallest_grammar_problem` · `Snappy_(compression)` · `Sound_quality` · `Species_richness` · [[Speech_coding]] · `Standard_test_image` · `Stationary_process` · `Statistical_dispersion` · `Statistical_mechanics` · `Stochastic_process` · `Sub-band_coding` · `Telecommunications_network` · `Terence_Tao` · `Ternary_numeral_system` · `Texture_compression` · [[Thermodynamic_system]] · `Thomas_M._Cover` · `Timeline_of_information_theory` · `Trace_(linear_algebra)` · `Transform_coding` · `Tunstall_coding` · `Typical_set` · `Unary_coding` · [[Uncertainty_principle]] · `Units_of_information` · `Universal_code_(data_compression)` · `Variable_bitrate` · `Video` · `Video_codec` · `Video_compression_picture_types` · `Video_quality` · `Von_Neumann_entropy` · `Warped_linear_predictive_coding` · `Warren_Weaver` · `Wavelet_transform` · [[Wayback_Machine]] · `YouTube` · `Zstd`
## From the Real GENERATIVE library
*Entropy (information theory) — placed from the Real G.E.N.E.R.A.T.I.V.E. course library (Information room). Source: Wikimedia Commons (via Wikipedia article media). [Details & license](https://commons.wikimedia.org/wiki/File:Binaryerasurechannel.png).*
> In information theory, the entropy of a random variable quantifies the average level of uncertainty or information associated with the variable's potential states or possible outcomes. This measures the expected amount of information needed to describe the state of the variable, considering the distribution of probabilities across all potential states. ([Wikipedia](https://en.wikipedia.org/wiki/Entropy_%28information_theory%29))
<!-- REAL-GENERATIVE-MEDIA:END -->
> **Room:** [[Information]] · **Status:** ✅ shipped
## Overview
**Entropy (information theory)** is [[Claude_Shannon|Claude Shannon]]'s 1948 quantitative measure of the *uncertainty* — equivalently, the *expected information content* — of a discrete random variable. For a source X taking values in {x_1, ..., x_n} with probability mass function p(x_i), Shannon defined the entropy as
H(X) = - Sum_i p(x_i) log_2 p(x_i) (bits)
which can be read as the expected **surprisal** I(x) = -log_2 p(x): rare outcomes carry more information per occurrence, common ones less. Entropy is bounded by 0 (one outcome is certain) and log_2 n (uniform), and the binary entropy function H(p) = -p log_2 p - (1-p) log_2 (1-p) traces the canonical concave curve peaking at p = 0.5 with H = 1 bit. From this single quantity follow the three central theorems of the field — Shannon's source coding theorem (entropy is the lossless-compression floor in expected bits per symbol), the noisy-channel coding theorem (channel capacity is sup_p I(X;Y), and reliable communication is possible up to that rate), and the asymptotic equipartition property (long sequences concentrate on a typical set of size ~2^(nH)). Joint, conditional, and relative entropies, mutual information, and modern cross-entropy losses are all algebraic combinations of this formula.
## See also
- Room hub: [[Information]]
- p5.js Editor conventions: P5 JS EDITOR
- Wiki root: MAIN
---
*Scaffolded by `generative-microsim` from row 0 of the Information sheet on 2026-04-30T12:37:19Z.*
Letters: entropy · mined_information · probability · distribution · flow · kanji_radicals · measurement · discretization
<!-- REAL-GENERATIVE-MEDIA:START -->
<!-- CRAFT-LINK:START g12 -->
*Built to the [[WT!P5_js_Microsim_Master_Class|p5.js Master Class]].*
<!-- CRAFT-LINK:END -->
## Wikipedia : Wikitube
**Strict pair:** [Wikipedia](https://en.wikipedia.org/wiki/Entropy_%28information_theory%29) : [Wikitube](https://en.wikitube.io/wiki/Entropy_%28information_theory%29)
## Previous hub tags
Tree parents: [[Information_theory]] · [[Systems_theory]].
Legacy hubs: `GENERATIVE`.
---
*Sources: 1 legacy note. Minted wave 1, 2026-07-30 (v1.6 order).*