# Differential entropy
## Microsim
### Live player
<div class="microsim-player">
<iframe src="https://editor.p5js.org/sciencenibber/full/89fUuxHoL" width="100%" height="620" frameborder="0" sandbox="allow-scripts allow-same-origin"></iframe>
</div>
<div class="microsim-fallback">
<img src="Microsims/thumbs/Differential_entropy.png" alt="Differential_entropy microsim poster" style="width:100%;border:1px solid #4445;border-radius:6px;">
<p><em>Live microsim (desktop) · <a href="https://editor.p5js.org/sciencenibber/sketches/89fUuxHoL">open sketch in the p5.js editor</a></em></p>
</div>
**Editor URL:** https://editor.p5js.org/sciencenibber/sketches/89fUuxHoL
**Description (100 words):**
This sketch lets you watch the **differential entropy** h(X) = − ∫ f(x) log₂ f(x) dx swing in real time as you reshape one of four named continuous densities — Gaussian, Uniform, Exponential, or Laplace. The two sliders rescale (sigma) and shift (mu) the chosen family; the main panel plots f(x), the side panel sweeps h(X) versus sigma so you see the underlying log-of-spread relationship, and the bottom-left readout colours h(X) **green when positive, orange when negative**. Crank sigma below about 0.4 and the Gaussian peaks above f(x) = 1 — the dashed orange "above this line, h goes negative" guide shows exactly why that happens.
```js
// =====================================================================
// Differential_entropy.js -- Wikitube microsim, Information room
// ---------------------------------------------------------------------
// ARTICLE Differential_entropy
// ROOM Information
// PATTERN E -- Entropy and information measures (Information sec.10)
// AUTHORED 2026-04-30 (generative pipeline, scheduled run)
//
// PURPOSE
// Visualise how the differential entropy
//
// h(X) = - integral f(x) log f(x) dx (in bits, log = log2)
//
// varies with the shape of a continuous probability density f(x).
// The reader picks one of four canonical density families
// (Gaussian, Uniform, Exponential, Laplace), drags two sliders that
// scale and shift the density, and watches h(X) update in real time.
// The teaching beat the sketch is built around: differential
// entropy is NOT discrete entropy. It can be negative when the
// density spikes above 1, and rescaling X by a constant a shifts
// h by exactly log|a| -- the curve of h vs sigma in the side
// panel is literally a log curve, no matter which family is loaded.
//
// CONTROLS (DOM, top-left of canvas)
// Family select Gaussian / Uniform / Exponential / Laplace
// sigma slider spread parameter (sigma for Gauss/Lapl, half-width
// for Uniform, 1/lambda for Exponential).
// mu slider location parameter (mean for Gauss/Lapl, centre
// for Uniform, offset for Exponential).
// reset button snap sigma=1, mu=0, family=Gaussian.
//
// READOUTS (HUD bottom-left)
// h(X) in bits current differential entropy, rounded to 3 dp.
// units note "negative h is OK -- see article".
// sigma, mu echoed live so the reader can ground their
// intuition against numeric values.
//
// EQUATION (HUD bottom-right, ASCII)
// h(X) = - integral f(x) log2 f(x) dx
//
// PARAMETER TABLE
// FAMILIES index into closed-form differential entropies:
// Gaussian: 0.5 * log2(2*pi*e * sigma^2)
// Uniform: log2(2*sigma) (width = 2*sigma)
// Exponential:log2(e * sigma) (mean = sigma)
// Laplace: log2(2*e * sigma)
// SIGMA_RANGE 0.05 .. 4.0 (slider step 0.01)
// MU_RANGE -3.0 .. 3.0 (slider step 0.05)
// X_RANGE -6 .. 6 (canvas plot domain)
// N_PLOT_SAMPLES 720 (curve resolution; 60-fps friendly)
//
// FILES
// Local archive : Articles/Information/Microsims/Differential_entropy.js
// Editor URL : captured live from window.location.href after save.
// =====================================================================
const ARTICLE = "Differential_entropy";
p5.disableFriendlyErrors = true;
// ---------- palette (Information room standard, sec.10) -------------
const BG = 246;
const INK = [40, 48, 60];
const BAR = [70, 130, 200];
const EDGE = [120, 130, 150];
const TOKEN = [220, 110, 60];
const ACCEPT= [80, 180, 120];
const LOWP = [200, 205, 215];
// ---------- physical / informational constants ----------------------
const BASE = 2; // Shannon entropy is in bits
const LN2 = Math.log(2);
const TWO_PI_E = 2 * Math.PI * Math.E;
// ---------- plot domain (data coordinates, not pixels) ---------------
const X_MIN = -6, X_MAX = 6;
const N_PLOT_SAMPLES = 720;
// Side-panel "h vs sigma" sweep:
const SIGMA_SWEEP_MIN = 0.05;
const SIGMA_SWEEP_MAX = 4.0;
const N_SWEEP = 220;
// ---------- DOM controls (created in setup) -------------------------
let familySel, sigmaSlider, muSlider, resetBtn;
let family = "Gaussian";
let sigma = 1.0;
let mu = 0.0;
function setup() {
createCanvas(windowWidth, windowHeight);
textFont("Helvetica");
textSize(13);
noStroke();
// family select -- the four canonical "named" densities
familySel = createSelect();
familySel.position(20, 60);
familySel.option("Gaussian");
familySel.option("Uniform");
familySel.option("Exponential");
familySel.option("Laplace");
familySel.changed(() => { family = familySel.value(); });
// sigma slider -- 0.05 .. 4.0 (spread)
sigmaSlider = createSlider(0.05, 4.0, 1.0, 0.01);
sigmaSlider.position(20, 100);
sigmaSlider.style("width", "200px");
// mu slider -- -3 .. 3 (location)
muSlider = createSlider(-3.0, 3.0, 0.0, 0.05);
muSlider.position(20, 140);
muSlider.style("width", "200px");
// reset button -- snap to canonical Gaussian(0,1)
resetBtn = createButton("reset");
resetBtn.position(20, 180);
resetBtn.mousePressed(() => {
family = "Gaussian"; familySel.selected("Gaussian");
sigmaSlider.value(1.0); muSlider.value(0.0);
});
}
// ---------- pdf evaluator: f(x | family, sigma, mu) -----------------
// Each branch is the textbook density. Domain checks are inline so
// that, e.g., the Exponential is exactly zero left of mu rather than
// silently producing NaN under the log.
function pdf(x) {
switch (family) {
case "Gaussian": {
// standard Gaussian density
const z = (x - mu) / sigma;
return Math.exp(-0.5 * z * z) / (sigma * Math.sqrt(2 * Math.PI));
}
case "Uniform": {
// uniform on [mu - sigma, mu + sigma]; height = 1 / (2*sigma)
return (x >= mu - sigma && x <= mu + sigma) ? 1 / (2 * sigma) : 0;
}
case "Exponential": {
// shifted exponential; mean = sigma; support x >= mu
if (x < mu) return 0;
return Math.exp(-(x - mu) / sigma) / sigma;
}
case "Laplace": {
// Laplace (double-exponential); scale = sigma
return Math.exp(-Math.abs(x - mu) / sigma) / (2 * sigma);
}
}
return 0;
}
// ---------- closed-form differential entropy h(X) in bits -----------
// These are the textbook results -- see MacKay ITILA chap.8 and any
// Cover & Thomas table. Using the closed form keeps the readout
// numerically stable; the integral form is only used for the curve
// shading (qualitative) below.
function differentialEntropyBits() {
switch (family) {
case "Gaussian":
return 0.5 * Math.log2(TWO_PI_E * sigma * sigma);
case "Uniform":
// h = log2(2 * sigma)
return Math.log2(2 * sigma);
case "Exponential":
// h = log2(e * sigma)
return Math.log2(Math.E * sigma);
case "Laplace":
// h = log2(2 * e * sigma)
return Math.log2(2 * Math.E * sigma);
}
return 0;
}
// ---------- main draw loop ------------------------------------------
function draw() {
background(BG);
// pull slider values once at top of draw -- the math step
// references them by information-theory name.
sigma = sigmaSlider.value();
mu = muSlider.value();
// layout: main plot occupies the centre, side plot occupies right.
const plotX = 250, plotY = 70;
const plotW = width - 520, plotH = height - 200;
drawDensityPlot(plotX, plotY, plotW, plotH);
drawSweepPanel(width - 250, plotY, 220, plotH);
drawControlsLabel();
drawHud();
drawReadouts();
}
// ---------- main density plot ---------------------------------------
function drawDensityPlot(x0, y0, w, h) {
// panel background and frame
noStroke(); fill(255); rect(x0, y0, w, h, 4);
stroke(...EDGE, 80); strokeWeight(1); noFill(); rect(x0, y0, w, h, 4);
// axis: x in [X_MIN, X_MAX], y in [0, yMax]
// yMax adapts to the current max density so very narrow Gaussians
// remain on screen (visualising the "negative h" case).
let yMax = 0.001;
for (let i = 0; i < N_PLOT_SAMPLES; i++) {
const x = X_MIN + (X_MAX - X_MIN) * i / (N_PLOT_SAMPLES - 1);
yMax = Math.max(yMax, pdf(x));
}
// give a 10% headroom so the peak does not kiss the panel top
yMax *= 1.10;
// grid + zero-axis line
stroke(...LOWP); strokeWeight(1);
for (let g = 0; g <= 4; g++) {
const yy = y0 + h * g / 4;
line(x0, yy, x0 + w, yy);
}
for (let g = 0; g <= 6; g++) {
const xx = x0 + w * g / 6;
line(xx, y0, xx, y0 + h);
}
// the height-1 line: anything above this is where f(x) > 1, and the
// local contribution to h is NEGATIVE. Painting it makes the
// "differential entropy can be negative" beat visible.
if (yMax > 1) {
const yOne = y0 + h * (1 - 1 / yMax);
stroke(...TOKEN); strokeWeight(1);
drawingContext.setLineDash([6, 4]);
line(x0, yOne, x0 + w, yOne);
drawingContext.setLineDash([]);
noStroke(); fill(...TOKEN); textAlign(LEFT, BOTTOM); textSize(11);
text("f(x) = 1 (above => negative entropy contribution)", x0 + 8, yOne - 2);
}
// x-axis ticks every integer
noStroke(); fill(...INK); textAlign(CENTER, TOP); textSize(11);
for (let xi = Math.ceil(X_MIN); xi <= Math.floor(X_MAX); xi++) {
const px = x0 + w * (xi - X_MIN) / (X_MAX - X_MIN);
text(xi.toString(), px, y0 + h + 4);
}
// the density curve itself, drawn as an open polyline plus a
// light fill underneath (alpha so the grid still shows through).
noFill(); stroke(...BAR); strokeWeight(2);
beginShape();
for (let i = 0; i < N_PLOT_SAMPLES; i++) {
const x = X_MIN + (X_MAX - X_MIN) * i / (N_PLOT_SAMPLES - 1);
const y = pdf(x);
const px = x0 + w * (x - X_MIN) / (X_MAX - X_MIN);
const py = y0 + h * (1 - y / yMax);
vertex(px, py);
}
endShape();
// shaded fill under the curve
noStroke(); fill(...BAR, 40);
beginShape();
vertex(x0, y0 + h);
for (let i = 0; i < N_PLOT_SAMPLES; i++) {
const x = X_MIN + (X_MAX - X_MIN) * i / (N_PLOT_SAMPLES - 1);
const y = pdf(x);
const px = x0 + w * (x - X_MIN) / (X_MAX - X_MIN);
const py = y0 + h * (1 - y / yMax);
vertex(px, py);
}
vertex(x0 + w, y0 + h);
endShape(CLOSE);
// mu marker -- vertical line at the mean / centre
const muPx = x0 + w * (mu - X_MIN) / (X_MAX - X_MIN);
stroke(...TOKEN); strokeWeight(1.5); line(muPx, y0, muPx, y0 + h);
// panel title
noStroke(); fill(...INK); textAlign(LEFT, BOTTOM); textSize(12);
text("density f(x) -- family = " + family, x0 + 6, y0 - 4);
}
// ---------- side panel: h(X) as sigma sweeps ------------------------
// This is the "single-formula readout" the standards (sec.10 Pattern
// E reskin) ask for: a small entropy-vs-parameter curve that shows
// the current point as a highlighted dot.
function drawSweepPanel(x0, y0, w, h) {
noStroke(); fill(255); rect(x0, y0, w, h, 4);
stroke(...EDGE, 80); strokeWeight(1); noFill(); rect(x0, y0, w, h, 4);
// sweep h vs sigma over [SIGMA_SWEEP_MIN, SIGMA_SWEEP_MAX]
const orig = sigma;
let yMin = Infinity, yMax = -Infinity;
const samples = [];
for (let i = 0; i < N_SWEEP; i++) {
const s = SIGMA_SWEEP_MIN
+ (SIGMA_SWEEP_MAX - SIGMA_SWEEP_MIN) * i / (N_SWEEP - 1);
sigma = s;
const hv = differentialEntropyBits();
samples.push([s, hv]);
yMin = Math.min(yMin, hv);
yMax = Math.max(yMax, hv);
}
sigma = orig;
// pad y range so the line never touches the frame
const span = Math.max(0.5, yMax - yMin);
yMin -= span * 0.05; yMax += span * 0.05;
// axes: zero line in the y range gets a stronger stroke so the
// viewer can see when h crosses through zero (the "negative
// entropy" boundary).
if (0 > yMin && 0 < yMax) {
const yZero = y0 + h * (1 - (0 - yMin) / (yMax - yMin));
stroke(...EDGE); strokeWeight(1);
line(x0, yZero, x0 + w, yZero);
}
// the sweep curve
noFill(); stroke(...BAR); strokeWeight(1.5);
beginShape();
for (const [s, hv] of samples) {
const px = x0 + w * (s - SIGMA_SWEEP_MIN)
/ (SIGMA_SWEEP_MAX - SIGMA_SWEEP_MIN);
const py = y0 + h * (1 - (hv - yMin) / (yMax - yMin));
vertex(px, py);
}
endShape();
// the current operating point -- a filled circle
const hNow = differentialEntropyBits();
const px = x0 + w * (sigma - SIGMA_SWEEP_MIN)
/ (SIGMA_SWEEP_MAX - SIGMA_SWEEP_MIN);
const py = y0 + h * (1 - (hNow - yMin) / (yMax - yMin));
noStroke(); fill(...TOKEN); circle(px, py, 9);
// panel title and axis labels
noStroke(); fill(...INK); textAlign(LEFT, BOTTOM); textSize(12);
text("h(X) [bits] vs sigma", x0 + 6, y0 - 4);
textAlign(LEFT, TOP); textSize(10);
text("sigma -->", x0 + 6, y0 + h + 4);
}
// ---------- HUD: title block, equation, control hints ---------------
function drawHud() {
// top-left title block (article + wikitube URL)
noStroke(); fill(0, 200); rect(8, 8, 380, 26);
fill(255); textSize(13); textAlign(LEFT, TOP);
text(ARTICLE + " :: en.wikitube.io/wiki/" + ARTICLE, 16, 14);
// top-right control hint strip
fill(0, 160); rect(width - 280, 8, 272, 26);
fill(255); textAlign(LEFT, TOP); textSize(12);
text("drag sliders | switch family | reset", width - 270, 14);
// bottom-right equation strip
fill(0, 160); rect(width - 380, height - 32, 372, 26);
fill(255); textAlign(LEFT, TOP); textSize(12);
text("h(X) = - integral f(x) log2 f(x) dx", width - 370, height - 26);
}
// ---------- readouts (bottom-left) ----------------------------------
function drawReadouts() {
const hNow = differentialEntropyBits();
noStroke();
fill(0, 160); rect(8, height - 100, 360, 92);
fill(255); textAlign(LEFT, TOP); textSize(12);
text("family = " + family, 16, height - 94);
text("sigma = " + sigma.toFixed(3), 16, height - 78);
text("mu = " + mu.toFixed(3), 16, height - 62);
// entropy readout coloured by sign so the negative case is loud
fill(hNow < 0 ? color(...TOKEN) : color(...ACCEPT));
text("h(X) = " + hNow.toFixed(3) + " bits"
+ (hNow < 0 ? " (negative -- ok!)" : ""),
16, height - 46);
fill(...LOWP); textSize(10);
text("differential entropy is unitless once "
+ "you've fixed the units of X", 16, height - 22);
textSize(13);
}
// labels for the DOM controls that live above the canvas
function drawControlsLabel() {
noStroke(); fill(...INK); textAlign(LEFT, TOP); textSize(12);
text("density family", 20, 44);
text("sigma (spread)", 20, 84);
text("mu (location)",20, 124);
}
function windowResized() {
resizeCanvas(windowWidth, windowHeight);
}
```
## Links (Wikipedia order)
<!-- injected from _registry/childlinks/Differential_entropy.json (2026-07-30T02:09:12Z) -->
`Absolute_continuity` · `Almost_everywhere` · `Asymptotic_equipartition_property` · `Beta_distribution` · `Beta_function` · `Bit` · [[Calculus_of_variations]] · `Cauchy_distribution` · `Change_of_variables` · `Channel_capacity` · `Chi-squared_distribution` · `Chi_distribution` · [[Conditional_entropy]] · `Conditional_mutual_information` · `Covariance` · `Digamma_function` · `Directed_information` · `Edwin_Thompson_Jaynes` · `Elementary_Principles_in_Statistical_Mechanics` · `Encyclopedia_of_Mathematics` · [[Entropy_(information_theory)]] · `Entropy_estimation` · `Entropy_rate` · `Erlang_distribution` · `Estimator` · `Exponential_distribution` · `Gamma_distribution` · `Gamma_function` · `If_and_only_if` · [[Information_theory]] · `Invariant_measure` · `Jacobian_matrix_and_determinant` · [[Joint_entropy]] · [[Josiah_Willard_Gibbs]] · `Kullback–Leibler_divergence` · `Laplace_distribution` · `Lebesgue_measure` · `Limiting_density_of_discrete_points` · `Log-normal_distribution` · `Logarithm` · `Logistic_distribution` · `Matrix_(mathematics)` · `Maxwell–Boltzmann_distribution` · `Multivariate_normal_distribution` · `Mutual_information` · [[Nat_(unit)]] · `Noisy-channel_coding_theorem` · `Normal_distribution` · `Pareto_distribution` · `Physical_Review_E` · `PlanetMath` · [[Probability_density_function]] · `Probability_measure` · `Quantile_function` · [[Quantization_(signal_processing)]] · `Rate–distortion_theory` · `Rayleigh_distribution` · `Shannon's_source_coding_theorem` · `Shannon–Hartley_theorem` · `Stack_Exchange` · `Student's_t-distribution` · `Support_(mathematics)` · `Triangular_distribution` · `Weibull_distribution`
## From the Real GENERATIVE library
*Differential entropy — placed from the Real G.E.N.E.R.A.T.I.V.E. course library (Information room). Source: Wikimedia Commons (via Wikipedia article media). [Details & license](https://commons.wikimedia.org/wiki/File:Binaryerasurechannel.png).*
> Differential entropy (also referred to as continuous entropy) is a concept in information theory that began as an attempt by Claude Shannon to extend the idea of (Shannon) entropy (a measure of average surprisal) of a random variable, to continuous probability distributions. Unfortunately, Shannon did not derive this formula, and rather just assumed it was t ([Wikipedia](https://en.wikipedia.org/wiki/Differential_entropy))
<!-- REAL-GENERATIVE-MEDIA:END -->
<!-- LOCAL-MEDIA-PASS:START -->
## From the vault media library
!Differential entropy thumb.png
*Differential Entropy — from the vault's own media holdings, placed 2026-07-09. MTN / Wikitube.io original · CC BY-SA 4.0.*
<!-- LOCAL-MEDIA-PASS:END -->
> **Room:** [[Information]] · **Status:** ✅ shipped
## Overview
**Differential entropy** is the continuous-variable analogue of Shannon's discrete [[Entropy|entropy]]. For a real-valued random variable X with probability [[Density|density]] function f(x), its differential entropy is
h(X) = − ∫ f(x) log f(x) dx,
measured in bits when log is base 2 and in nats when log is the natural logarithm. Introduced by [[Claude_Shannon|Claude Shannon]] in his 1948 paper *A Mathematical Theory of Communication*, it preserves many of the structural identities of the discrete case — chain rule, conditioning never increases it, mutual information stays non-negative — but it loses two properties readers usually take for granted. First, h(X) is **not invariant** under change of variables: rescaling X by a constant a shifts h by log|a|, so a differential entropy of −3.2 bits is meaningful only relative to the units of X. Second, h(X) **can be negative**, infinite, or undefined; a sharply peaked density (a tall narrow Gaussian, a delta-like spike) produces a negative value, signalling that the density itself exceeds 1 over some region. Among all continuous distributions on the real line with a fixed mean and variance, the Gaussian achieves the maximum differential entropy — the continuous analogue of the discrete uniform's maximality. This makes the Gaussian the natural noise floor for channel-capacity arguments and the entropy-maximising prior in maximum-entropy inference.
## See also
- Room hub: [[Information]]
- p5.js Editor conventions: P5 JS EDITOR
- Wiki root: MAIN
---
*Scaffolded by `generative-microsim` from row 0 of the Information sheet on 2026-04-30T13:39:25Z.*
Letters: entropy · distribution · mined_density · exponential · mined_information · discretization · flow · mined_switch
<!-- REAL-GENERATIVE-MEDIA:START -->
<!-- CRAFT-LINK:START g12 -->
*Built to the [[WT!P5_js_Microsim_Master_Class|p5.js Master Class]].*
<!-- CRAFT-LINK:END -->
## Wikipedia : Wikitube
**Strict pair:** [Wikipedia](https://en.wikipedia.org/wiki/Differential_entropy) : [Wikitube](https://en.wikitube.io/wiki/Differential_entropy)
## Previous hub tags
Tree parent: [[Information_theory]].
Legacy hubs: `GENERATIVE`.
---
*Sources: 1 legacy note. Minted wave 1, 2026-07-30 (v1.6 order).*