Information_theory - Wikiparency

# Information theory Information theory quantifies the storage, transmission, and compression of information under uncertainty. Shannon entropy, mutual information, channel capacity, and coding theorems provide the mathematical foundation for communication, cryptography, statistical inference, and the informational view of physical and biological systems.  ## Microsims (promoted from legacy — three.js first) ### Microsim ### Live player <div class="microsim-player"> <iframe src="https://editor.p5js.org/sciencenibber/full/byCRgwg34" width="100%" height="620" frameborder="0" sandbox="allow-scripts allow-same-origin"></iframe> </div> <div class="microsim-fallback"> <img src="Microsims/thumbs/Information_theory.png" alt="Information_theory microsim poster" style="width:100%;border:1px solid #4445;border-radius:6px;"> <p><em>Live microsim (desktop) · <a href="https://editor.p5js.org/sciencenibber/sketches/byCRgwg34">open sketch in the p5.js editor</a></em></p> </div> **Editor URL:** https://editor.p5js.org/sciencenibber/sketches/byCRgwg34 **Description (100 words):** This microsim demonstrates the binary [[Entropy|entropy]] function, the simplest case of Shannon entropy. A single slider sets the bias p of a coin (the probability of outcome 1). As you drag p from 0 to 1, the sketch plots H(p) = -p*log2(p) - (1-p)*log2(1-p), the number of bits of uncertainty in one flip. The orange dot tracks your current point on the curve, and the bar below shows how the two probabilities split. Entropy peaks at exactly 1 bit when p = 0.5 — the fair coin, the least predictable outcome — and falls to 0 bits at either certainty. ```js // ===================================================================== // Article : Information theory // Slug : Information_theory // Wikitube : en.wikitube.io/wiki/Information_theory // Room : Information // // Idea : Information theory measures the "surprise" carried by a // random outcome. This sketch demonstrates the binary // entropy function — the Shannon entropy of a single coin // flip with bias p. The reader drags the bias p from 0 to 1 // and watches the entropy H(p) rise to a maximum of exactly // 1 bit at p = 0.5 (a fair coin) and fall to 0 bits at the // two certain outcomes (p = 0 or p = 1). The curve plotted // is the canonical "entropy bump"; the moving dot is the // reader's current p; the bar at the bottom shows how the // two probabilities split. Entropy is maximised exactly // when the outcome is least predictable. // // Equation : H(p) = -p*log2(p) - (1-p)*log2(1-p) (bits) // general: H(X) = -Sum p(x) * log2 p(x) // ===================================================================== // Rule §3 — single source of truth. The HUD URL line, the editor save // name, and any cross-references all derive from this constant. const ARTICLE = "Information_theory"; // Rule §4 — disable the Friendly Error System for ship. p5.disableFriendlyErrors = true; // ---------- runtime state ---------- let pSlider; // bias p of the Bernoulli source (P of outcome "1") // layout constants computed in setup() let PLOT_X, PLOT_Y, PLOT_W, PLOT_H; // entropy-curve plot box function setup() { // Rule §5 — canvas inside setup, standard size, 2x density. createCanvas(720, 520); pixelDensity(2); // Plot box for the H(p) curve. Computed from width/height so the // sketch survives a resize. PLOT_X = width * 0.14; PLOT_Y = height * 0.20; PLOT_W = width * 0.72; PLOT_H = height * 0.46; // Rule §6 — controls in setup, positioned explicitly, labelled in draw. // Range 0..1 in fine steps; default 0.5 = the fair coin, the maximum. pSlider = createSlider(0, 1, 0.5, 0.01); pSlider.position(180, height - 28); pSlider.style("width", "300px"); } function draw() { background(248); // ---------- read controls ---------- const p = pSlider.value(); // ---------- math ---------- const H = binaryEntropy(p); // current entropy in bits // ---------- reference geometry (rule §8 layer 1) ---------- // Plot frame + gridlines, neutral grey, drawn first. stroke(180); strokeWeight(1); noFill(); rect(PLOT_X, PLOT_Y, PLOT_W, PLOT_H); // Horizontal gridline at H = 1 bit (the maximum) and label. stroke(215); line(PLOT_X, PLOT_Y, PLOT_X + PLOT_W, PLOT_Y); // H = 1 line(PLOT_X, PLOT_Y + PLOT_H / 2, PLOT_X + PLOT_W, PLOT_Y + PLOT_H / 2); // H = 0.5 // Vertical gridline at p = 0.5 (the maximiser). line(PLOT_X + PLOT_W / 2, PLOT_Y, PLOT_X + PLOT_W / 2, PLOT_Y + PLOT_H); noStroke(); fill(140); textFont("system-ui"); textSize(10); textAlign(RIGHT, CENTER); text("1 bit", PLOT_X - 6, PLOT_Y); text("0.5", PLOT_X - 6, PLOT_Y + PLOT_H / 2); text("0", PLOT_X - 6, PLOT_Y + PLOT_H); textAlign(CENTER, TOP); text("p = 0", PLOT_X, PLOT_Y + PLOT_H + 4); text("p = 0.5", PLOT_X + PLOT_W / 2, PLOT_Y + PLOT_H + 4); text("p = 1", PLOT_X + PLOT_W, PLOT_Y + PLOT_H + 4); // ---------- active geometry (rule §8 layer 2) ---------- // The entropy curve H(p) across the full range, in blue. stroke(40, 90, 200); strokeWeight(2); noFill(); beginShape(); for (let i = 0; i <= 240; i++) { const px = i / 240; // p value 0..1 const py = binaryEntropy(px); // H in bits 0..1 vertex(PLOT_X + px * PLOT_W, PLOT_Y + (1 - py) * PLOT_H); } endShape(); // The current point on the curve (orange dot + drop lines). const dotX = PLOT_X + p * PLOT_W; const dotY = PLOT_Y + (1 - H) * PLOT_H; stroke(220, 130, 40); strokeWeight(1); line(dotX, dotY, dotX, PLOT_Y + PLOT_H); // drop to the p axis line(PLOT_X, dotY, dotX, dotY); // across to the H axis noStroke(); fill(220, 130, 40); circle(dotX, dotY, 9); // Probability split bar under the plot, showing p vs 1-p. const barY = PLOT_Y + PLOT_H + 28; const barH = 16; noStroke(); fill(40, 90, 200); rect(PLOT_X, barY, p * PLOT_W, barH); // P(outcome=1) = p fill(220, 60, 60); rect(PLOT_X + p * PLOT_W, barY, (1 - p) * PLOT_W, barH); // P(outcome=0) = 1-p // ---------- HUD watermark (rule §2) ---------- noStroke(); textFont("system-ui"); // §2a — top-left title block. fill(20); textSize(20); textAlign(LEFT, TOP); text("Information theory", 16, 14); textSize(12); fill(110); text("Wikitube microsim - en.wikitube.io/wiki/" + ARTICLE, 16, 40); // §2b — top-right control hints. textAlign(RIGHT, TOP); textSize(11); fill(110); text("slider: p (bias of the coin, P of outcome 1)", width - 16, 14); text("H peaks at 1 bit when p = 0.5 (least predictable)", width - 16, 30); // §2c — bottom-left live readouts (canonical parameter symbols). textAlign(LEFT, BOTTOM); textSize(13); fill(40, 90, 200); text("p = " + nf(p, 1, 2), 16, height - 64); fill(220, 60, 60); text("1 - p = " + nf(1 - p, 1, 2), 16, height - 46); fill(20); text("H(p) = " + nf(H, 1, 3) + " bits", 16, height - 4); // Slider label (rule §7) — to the LEFT of the slider, right-aligned. textAlign(RIGHT, CENTER); textSize(12); fill(60); text("p (coin bias)", 172, height - 28 + 8); // §2d — bottom-right equation footer (ASCII only — see pitfalls.md). textAlign(RIGHT, BOTTOM); textSize(11); fill(80); text("H(p) = -p*log2(p) - (1-p)*log2(1-p)", width - 16, height - 8); } // ---------- helpers (rule §10) ---------- // Binary (Shannon) entropy in bits. The 0*log(0) = 0 convention is // handled explicitly so the endpoints return exactly 0 instead of NaN. function binaryEntropy(p) { if (p <= 0 || p >= 1) return 0; return -p * log2(p) - (1 - p) * log2(1 - p); } // Base-2 logarithm (p5 only ships natural log). function log2(x) { return Math.log(x) / Math.log(2); } ``` ### MicroSim spec - **Recommended sim type:** stochastic process / random walk - **Microsimmability score:** 78/100 - **Layout:** drawing region (canvas) on top; control region (sliders/buttons) below. ### Parameters (tunable controls) - `Source bias` - `Noise` - `Code rate` ### What animates Symbols from a biased source pass through a noisy channel and you watch [[Entropy|entropy]] and error change. ### Learning objective Relate source entropy, noise, and coding to reliable communication.   ## Microsims — p5.js ### Information theory (p5.js) · `H = −Σ p log p` <div class="microsim-player"> <iframe src="https://editor.p5js.org/sciencenibber/full/MFFPM7AEH" width="100%" height="480" frameborder="0" loading="lazy" sandbox="allow-scripts allow-same-origin" title="Information theory — p5.js microsim"></iframe> </div> *Shannon's mathematics of how much a signal can carry and how far it can be compressed.* **Open in the editor:** [▶ fork this sketch](https://editor.p5js.org/sciencenibber/sketches/MFFPM7AEH) · movement *IX · Foundations & the rest of the toolbox* · library `p5js` ### Related microsims Live sims on neighbouring articles — 6 of them inside this article's own Wikipedia link tree: - [[Analog_signal]] *(in tree)* - [[Autocorrelation]] *(in tree)* - [[Bifurcation_theory]] *(in tree)* - [[Calculus]] *(in tree)* - [[Chaos_theory]] *(in tree)* - [[Communication_channel]] *(in tree)* *Sim hosted off-article; the article owns the reference, not the runtime (WIKI_RULES §10.4). Placed by `g08_place_microsims.py`.*  > **Also on this page:** 1 further p5.js sketch already published for this article live further down. Per WIKI_RULES §5 a collision promotes rather than forks — they are one shelf, not rivals; this block is the §10.4 *current best* reference.   ## Images <figure class="wt-gifplate"> <img src="https://commons.wikimedia.org/wiki/Special:FilePath/Preferential_attachment.gif" alt="Network Growth" loading="lazy" decoding="async"> <figcaption><strong>Network Growth</strong> — Watch network structure emerge from preferential connection rules.<br> <span class="wt-credit">Wikimedia Commons · <strong>licence pending verification</strong> (run <code>g17_gif_verify.py</code> on a networked lane) · <a href="https://commons.wikimedia.org/wiki/File:Preferential_attachment.gif">Details</a></span></figcaption> </figure> *The hub concept of [[PORTAL_Information_theory]]. Still companion to the 2 live microsims above — §15 keeps the player first, the plate sits in the image slot.*  ## Reveal %%REVEAL:p5%% %%REVEAL:d3%% --- *Concept aligned with [Wikipedia](https://en.wikipedia.org/wiki/Information_theory); adapted text, where present, is licensed [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/).* ## Overview [[Claude_Shannon|Claude Shannon]] built the theory on [[Mathematics|mathematics]] and [[Statistics|statistics]], in the same mid-century ferment that produced [[Norbert_Wiener|Norbert Wiener]]'s [[Cybernetics|cybernetics]] and [[W._Ross_Ashby|Ashby]]'s variety calculus — entropy as requisite variety, the channel as a regulator. Its machinery now underwrites [[Computer_science|computer science]] and the [[Theory_of_computation|theory of computation]], carries [[Physics|physics]] into thermodynamic and quantum readings of the bit, and meets [[Machine_learning|machine learning]] and [[Artificial_intelligence|artificial intelligence]] wherever a [[Neural_network_(machine_learning)|neural network]] compresses experience. Structure and strategy stay adjacent: codes travel on graphs from [[Graph_theory|graph theory]], signaling games borrow [[Game_theory|game theory]] and [[Decision_theory|decision theory]], control loops price feedback in bits through [[Control_theory|control theory]], and the [[Information_system|information system]] and [[Systems_biology|systems biology]] read organizations and organisms as channels within [[Systems_science|systems science]].  *Built to the [[WT!P5_js_Microsim_Master_Class|p5.js Master Class]].*  ## Wikipedia : Wikitube **Strict pair:** [Wikipedia](https://en.wikipedia.org/wiki/Information_theory) : [Wikitube](https://en.wikitube.io/wiki/Information_theory) ## Previous hub tags Hubs: `GENERATIVE`, `Systems`. Portals: [[PORTAL_Systems]], [[PORTAL_Information_theory]].