# Joint entropy
## Microsim
### Live player
<div class="microsim-player">
<iframe src="https://editor.p5js.org/sciencenibber/full/iOkGPXBpB" width="100%" height="620" frameborder="0" sandbox="allow-scripts allow-same-origin"></iframe>
</div>
<div class="microsim-fallback">
<img src="Microsims/thumbs/Joint_entropy.png" alt="Joint_entropy microsim poster" style="width:100%;border:1px solid #4445;border-radius:6px;">
<p><em>Live microsim (desktop) · <a href="https://editor.p5js.org/sciencenibber/sketches/iOkGPXBpB">open sketch in the p5.js editor</a></em></p>
</div>
**Editor URL:** https://editor.p5js.org/sciencenibber/sketches/iOkGPXBpB
**Description (100 words):**
A 4 x 4 joint distribution p(X, Y) parametrised as a bivariate-Gaussian-style kernel on the integer lattice, controlled by four sliders -- centres cx and cy, spread sigma, correlation rho. The blue heatmap paints each cell by joint mass; faint orange rings overlay the product distribution p(X) * p(Y), so the rings sit exactly on the heatmap colours when rho = 0 and visibly drift away from them when rho moves. Marginal bar charts above and right show p(X) and p(Y). The dark readout panel reports H(X), H(Y), H(X) + H(Y), max{H(X), H(Y)}, the headline H(X, Y), and I(X ; Y), with an INDEPENDENT flag that flips green when the gap collapses below 0.01 bits.
```js
// =====================================================================
// Joint_entropy.js -- Wikitube microsim, Information room
// ---------------------------------------------------------------------
// ARTICLE Joint_entropy
// ROOM Information
// PATTERN E -- Entropy and information measures (sec.10 P5_JS_EDITOR)
// AUTHORED 2026-04-30 (generative pipeline, scheduled run)
//
// PURPOSE
// Visualise the joint entropy
//
// H(X, Y) = - sum_{x, y} p(x, y) log2 p(x, y)
//
// for a 4 x 4 joint distribution p(X, Y) the reader edits live, and
// make the two governing inequalities visible at all times:
//
// max{H(X), H(Y)} <= H(X, Y) <= H(X) + H(Y)
//
// The right-hand bound is mutual-information slack
// I(X ; Y) = H(X) + H(Y) - H(X, Y)
// which collapses to zero exactly when X and Y are independent.
// The left-hand bound is reached when one variable is a deterministic
// function of the other.
//
// PARAMETRISATION (the 'shape' of the joint)
// The reader does NOT poke 16 cells directly -- 16 sliders is a usability
// disaster. Instead the joint is parametrised as an UNNORMALISED
// Gaussian-style kernel on the 4 x 4 grid,
//
// p(i, j) ~ exp( -[ (i-cx)^2 - 2 rho (i-cx)(j-cy) + (j-cy)^2 ]
// / (2 sigma^2 (1 - rho^2)) )
//
// then renormalised to sum to one. Four sliders fully control it:
// cx in [0, N-1] centre of mass on the X axis
// cy in [0, N-1] centre of mass on the Y axis
// sigma in [0.4, 4] spread; large sigma -> uniform joint
// rho in [-0.95, 0.95] correlation; rho = 0 -> independent
//
// This single closed form already covers every regime the article
// talks about:
// sigma -> infty -> uniform joint (max H(X,Y))
// sigma -> 0, |rho| -> 1 -> degenerate diagonal (min H(X,Y))
// rho = 0 -> exact independence (I(X;Y) = 0)
// rho near +/- 1 -> max coupling (I(X;Y) max)
//
// CONTROLS (DOM, top-left of canvas, three rows)
// cx slider X centre
// cy slider Y centre
// sigma slider spread
// rho slider correlation
// reset button snap defaults (cx = cy = 1.5, sigma = 1.5, rho = 0)
//
// READOUTS (HUD bottom-left, dark panel)
// H(X) marginal entropy of X (bits)
// H(Y) marginal entropy of Y (bits)
// H(X) + H(Y) sum of marginals (bits) <- upper bound
// H(X, Y) joint entropy (the headline) (bits)
// I(X ; Y) gap to the upper bound (bits)
// max{H(X), H(Y)} lower bound (bits)
// independence flag when I(X;Y) < 0.01 bits, prints "INDEPENDENT"
//
// EQUATION (HUD bottom-right, ASCII only)
// H(X,Y) = -sum p(x,y) log2 p(x,y)
// max(H(X),H(Y)) <= H(X,Y) <= H(X) + H(Y)
//
// VISUAL ANATOMY
// centre : 4 x 4 joint heatmap p(X, Y) with each cell painted by
// probability mass and labelled with its value.
// above : marginal p(X) horizontal bars + numeric labels.
// right : marginal p(Y) vertical bars + numeric labels.
// overlay : faint outline of the PRODUCT distribution p(X) * p(Y) drawn
// on top of the joint, so the reader can see exactly how far
// the joint deviates from independence. When rho = 0 the
// outline matches the heatmap cell-for-cell.
//
// PARAMETER TABLE
// N 4 alphabet size for both X and Y.
// X_LABELS A B C D canonical alphabet for printing.
// Y_LABELS a b c d lowercase to keep marginals visually distinct.
// EPS 1e-12 floor for log arguments (log2(0) sentinel).
//
// FILES
// Local archive : Articles/Information/Microsims/Joint_entropy.js
// Editor URL : captured live from window.location.href after save.
// =====================================================================
const ARTICLE = "Joint_entropy";
p5.disableFriendlyErrors = true;
// ---------- palette (Information room standard, sec.10) -------------
const BG = 246;
const INK = [40, 48, 60];
const BAR = [70, 130, 200];
const EDGE = [120, 130, 150];
const TOKEN = [220, 110, 60];
const ACCEPT= [80, 180, 120];
const LOWP = [200, 205, 215];
// ---------- alphabet ------------------------------------------------
const N = 4;
const X_LABELS = ["A", "B", "C", "D"];
const Y_LABELS = ["a", "b", "c", "d"];
const EPS = 1e-12;
// ---------- DOM controls (created in setup) -------------------------
let sCx, sCy, sSigma, sRho, btnReset;
// ---------- derived state (recomputed every draw) -------------------
let joint = []; // 4x4 matrix p(x, y)
let pX = []; // 4-vector marginal of X
let pY = []; // 4-vector marginal of Y
let prod = []; // 4x4 outer product p(x) * p(y) -- "what indep looks like"
// ---------- helpers -------------------------------------------------
function log2safe(v) {
// log2 with a floor so an exact-zero probability does not propagate NaN.
// p log p with p = 0 is taken to be 0 by Shannon convention.
return v <= EPS ? 0 : Math.log2(v);
}
function entropyVec(v) {
// Shannon entropy of a 1-D probability vector, in bits.
let H = 0;
for (const p of v) if (p > EPS) H -= p * Math.log2(p);
return H;
}
function entropyMat(m) {
// Shannon entropy of a 2-D probability matrix, in bits.
let H = 0;
for (const row of m) for (const p of row) if (p > EPS) H -= p * Math.log2(p);
return H;
}
function rebuildJoint(cx, cy, sigma, rho) {
// Build an unnormalised 4x4 Gaussian-style kernel, then renormalise.
// The form is the classic bivariate-Gaussian density evaluated on the
// integer lattice 0..N-1 x 0..N-1. rho is clamped strictly inside +/- 1
// to keep the (1 - rho^2) denominator strictly positive.
const r = Math.max(-0.99, Math.min(0.99, rho));
const denom = 2 * sigma * sigma * (1 - r * r);
const W = [];
let total = 0;
for (let i = 0; i < N; i++) {
const row = [];
for (let j = 0; j < N; j++) {
const dx = i - cx;
const dy = j - cy;
const q = (dx * dx - 2 * r * dx * dy + dy * dy) / denom;
const w = Math.exp(-q);
row.push(w);
total += w;
}
W.push(row);
}
// Normalise; total > 0 always because exp is positive.
for (let i = 0; i < N; i++) for (let j = 0; j < N; j++) W[i][j] /= total;
return W;
}
function marginalRows(m) {
// Marginalise out the column index -> p(X = i) = sum_j m[i][j].
const out = new Array(N).fill(0);
for (let i = 0; i < N; i++) for (let j = 0; j < N; j++) out[i] += m[i][j];
return out;
}
function marginalCols(m) {
// Marginalise out the row index -> p(Y = j) = sum_i m[i][j].
const out = new Array(N).fill(0);
for (let j = 0; j < N; j++) for (let i = 0; i < N; i++) out[j] += m[i][j];
return out;
}
function outerProduct(a, b) {
// Outer product a x b -- used to draw the "independent" reference.
const M = [];
for (let i = 0; i < N; i++) {
const row = [];
for (let j = 0; j < N; j++) row.push(a[i] * b[j]);
M.push(row);
}
return M;
}
// ---------- p5 lifecycle --------------------------------------------
function setup() {
// 720 x 520 fixed canvas matches the global Betterfire standard from
// sec.5; the Information starter uses windowWidth/windowHeight but the
// entropy-meter family stays well inside 720 x 520 and benefits from
// predictable text layout for the dark readout panel.
createCanvas(720, 520);
pixelDensity(2);
textFont("system-ui");
// Three rows of sliders fit comfortably under the title block
// (rows at y = 56, 84, 112). Each slider is 180 px wide; its label
// is drawn in draw() to the LEFT of the slider so DOM-rendered names
// do not float over the canvas.
sCx = createSlider(0, N - 1, 1.5, 0.01); sCx.position(108, 52).style("width", "180px");
sCy = createSlider(0, N - 1, 1.5, 0.01); sCy.position(108, 80).style("width", "180px");
sSigma = createSlider(0.4, 4.0, 1.5, 0.01); sSigma.position(108, 108).style("width", "180px");
sRho = createSlider(-0.95, 0.95, 0.0, 0.01); sRho.position(108, 136).style("width", "180px");
btnReset = createButton("reset");
btnReset.position(108, 164);
btnReset.mousePressed(() => {
sCx.value(1.5); sCy.value(1.5); sSigma.value(1.5); sRho.value(0.0);
});
}
function draw() {
background(BG);
// --------- pull every slider value once at the top --------------
// Reading slider.value() returns a string sometimes; force-cast to
// float to guarantee arithmetic works. This is a sec.6 best practice.
const cx = parseFloat(sCx.value());
const cy = parseFloat(sCy.value());
const sigma = parseFloat(sSigma.value());
const rho = parseFloat(sRho.value());
// --------- rebuild distributions and entropies ------------------
joint = rebuildJoint(cx, cy, sigma, rho);
pX = marginalRows(joint);
pY = marginalCols(joint);
prod = outerProduct(pX, pY);
const Hx = entropyVec(pX);
const Hy = entropyVec(pY);
const Hxy = entropyMat(joint);
const Ixy = Hx + Hy - Hxy;
const minH = Math.max(Hx, Hy);
const sumH = Hx + Hy;
// --------- canvas regions (lay out in pixel coords) -------------
// Heatmap centre: 280 x 280 grid sitting at (320, 60).
const G_X = 320, G_Y = 60, G_W = 280, G_H = 280;
const CELL = G_W / N;
// --------- joint heatmap ---------------------------------------
drawHeatmap(G_X, G_Y, G_W, G_H, joint);
// --------- product-distribution overlay (faint) ----------------
drawProductOverlay(G_X, G_Y, G_W, G_H, prod);
// --------- marginal bars: top (X) and right (Y) ----------------
drawMarginalTop (G_X, G_Y - 50, G_W, 40, pX, X_LABELS);
drawMarginalRight(G_X + G_W + 10, G_Y, 40, G_H, pY, Y_LABELS);
// --------- readouts (dark panel, bottom-left) ------------------
drawReadouts(20, height - 200, 280, 180, Hx, Hy, Hxy, sumH, minH, Ixy);
// --------- HUD: title + controls hint + equation ---------------
drawSliderLabels();
drawHud();
drawEquationFooter();
}
// ---------- visual primitives ---------------------------------------
function drawHeatmap(x, y, w, h, m) {
// The heatmap paints each cell with opacity proportional to its
// probability mass. We do NOT use HSB / colour ramps to keep the
// sketch feeling like the rest of the Information palette; instead a
// single accent (BAR) varies in alpha from 20 (near-zero) to 240
// (concentration). The MAX cell defines 'full ink' so the contrast
// stays meaningful even for very flat distributions.
let mx = 0;
for (const row of m) for (const v of row) if (v > mx) mx = v;
const cs = w / N;
noStroke();
for (let i = 0; i < N; i++) {
for (let j = 0; j < N; j++) {
const a = mx > 0 ? 20 + 220 * (m[i][j] / mx) : 20;
fill(BAR[0], BAR[1], BAR[2], a);
rect(x + i * cs, y + j * cs, cs - 1, cs - 1, 3);
// Cell numeric (3 decimals). ASCII only, sec.9.
noStroke();
fill(...INK);
textAlign(CENTER, CENTER);
textSize(11);
text(m[i][j].toFixed(3), x + i * cs + cs / 2, y + j * cs + cs / 2);
}
}
// Frame the whole heatmap so it reads as one unit.
noFill();
stroke(...EDGE);
strokeWeight(1);
rect(x, y, w, h);
}
function drawProductOverlay(x, y, w, h, m) {
// The 'product' distribution p(X) * p(Y) is what the joint WOULD look
// like if X and Y were independent. Draw it as a circle inside each
// cell whose radius scales with that cell's product probability. When
// rho = 0 the circles tile the heatmap exactly because joint == product;
// when rho != 0 the rings drift away from the heatmap colours, which
// is exactly the visual cue for I(X;Y) > 0.
let mx = 0;
for (const row of m) for (const v of row) if (v > mx) mx = v;
const cs = w / N;
noFill();
stroke(220, 110, 60, 200);
strokeWeight(1.2);
for (let i = 0; i < N; i++) {
for (let j = 0; j < N; j++) {
const r = mx > 0 ? (cs * 0.45) * Math.sqrt(m[i][j] / mx) : 0;
if (r > 0.5) circle(x + i * cs + cs / 2, y + j * cs + cs / 2, 2 * r);
}
}
}
function drawMarginalTop(x, y, w, h, p, labels) {
// Marginal p(X) drawn as columns above the heatmap. Each bar's height
// scales linearly with mass; the label sits below at the heatmap's
// top edge so readers can read the label and the bar simultaneously.
const cs = w / N;
noStroke();
fill(...LOWP);
rect(x, y, w, h, 2);
fill(...BAR, 200);
for (let i = 0; i < N; i++) {
const bh = p[i] * h;
rect(x + i * cs + 4, y + h - bh, cs - 8, bh);
}
// Numeric and letter label at top of each column.
fill(...INK);
textAlign(CENTER, BOTTOM);
textSize(10);
for (let i = 0; i < N; i++) text(p[i].toFixed(2), x + i * cs + cs / 2, y - 2);
textAlign(CENTER, TOP);
textSize(11);
for (let i = 0; i < N; i++) text(X_LABELS[i], x + i * cs + cs / 2, y + h + 2);
}
function drawMarginalRight(x, y, w, h, p, labels) {
// Marginal p(Y) drawn as rows to the right of the heatmap. Bar widths
// scale linearly; the label sits to the right of each row.
const cs = h / N;
noStroke();
fill(...LOWP);
rect(x, y, w, h, 2);
fill(...BAR, 200);
for (let j = 0; j < N; j++) {
const bw = p[j] * w;
rect(x, y + j * cs + 4, bw, cs - 8);
}
fill(...INK);
textAlign(LEFT, CENTER);
textSize(10);
for (let j = 0; j < N; j++) text(p[j].toFixed(2), x + w + 4, y + j * cs + cs / 2);
}
function drawReadouts(x, y, w, h, Hx, Hy, Hxy, sumH, minH, Ixy) {
// Dark, slightly translucent readout panel. The headline H(X,Y) is
// larger and brighter; the bounds and the gap are quieter; the
// independence flag flips ACCEPT-coloured when Ixy is below threshold.
noStroke();
fill(0, 200);
rect(x, y, w, h, 6);
fill(255);
textAlign(LEFT, TOP);
textSize(13);
text("H(X) = " + Hx.toFixed(3) + " bits", x + 14, y + 10);
text("H(Y) = " + Hy.toFixed(3) + " bits", x + 14, y + 30);
text("H(X) + H(Y) = " + sumH.toFixed(3) + " bits", x + 14, y + 50);
text("max{H(X), H(Y)} = " + minH.toFixed(3) + " bits", x + 14, y + 70);
// Headline H(X, Y).
fill(...TOKEN);
textSize(15);
text("H(X, Y) = " + Hxy.toFixed(3) + " bits", x + 14, y + 95);
// Mutual-information gap. We allow tiny negative values from FP
// round-off and clamp them to 0 for display.
const Ixy_disp = Math.max(0, Ixy);
fill(255);
textSize(13);
text("I(X ; Y) = " + Ixy_disp.toFixed(3) + " bits", x + 14, y + 120);
// Independence flag: I(X;Y) below 0.01 bits is, for our purposes,
// independent. This is well above FP noise floor and well below the
// smallest informative signal a 4x4 joint can carry.
if (Ixy_disp < 0.01) {
fill(...ACCEPT);
text("INDEPENDENT (rho = 0)", x + 14, y + 145);
} else {
fill(...EDGE);
text("dependent (gap = " + Ixy_disp.toFixed(3) + " bits)", x + 14, y + 145);
}
}
function drawSliderLabels() {
// Slider labels go to the LEFT of the slider, right-aligned. We keep
// every symbol the literature uses (cx, cy, sigma, rho) and add a
// one-line clarification only where the symbol is ambiguous.
noStroke();
fill(...INK);
textAlign(RIGHT, CENTER);
textSize(12);
text("cx", 102, 62);
text("cy", 102, 90);
text("sigma", 102, 118);
text("rho", 102, 146);
text("", 102, 174); // reset btn has no label
// Print the live value to the right of each slider strip.
textAlign(LEFT, CENTER);
text(parseFloat(sCx.value()).toFixed(2), 296, 62);
text(parseFloat(sCy.value()).toFixed(2), 296, 90);
text(parseFloat(sSigma.value()).toFixed(2), 296, 118);
text(parseFloat(sRho.value()).toFixed(2), 296, 146);
}
function drawHud() {
// sec.2: Wikitube watermark. Title (line 1) is the human-readable
// article title; URL (line 2) is built from the ARTICLE constant so it
// stays in lockstep with the saved sketch name.
noStroke();
fill(20);
textSize(20);
textAlign(LEFT, TOP);
text("Joint entropy", 16, 12);
fill(110);
textSize(12);
text("Wikitube microsim - en.wikitube.io/wiki/" + ARTICLE, 16, 36);
// Top-right control hint, two lines of sec.2b form.
textAlign(RIGHT, TOP);
textSize(11);
fill(110);
text("sliders: cx, cy, sigma, rho (Gaussian-style joint p(X,Y))", width - 12, 12);
text("orange rings = product p(X) p(Y); equality at rho = 0", width - 12, 28);
}
function drawEquationFooter() {
// sec.2d / sec.9: ASCII equation footer in the bottom-right. Two
// lines is fine here because the pair of inequalities IS the lesson
// and a single-line collapse would hide the upper bound.
noStroke();
fill(80);
textSize(11);
textAlign(RIGHT, BOTTOM);
text("H(X,Y) = -sum p(x,y) log2 p(x,y)", width - 12, height - 26);
text("max(H(X),H(Y)) <= H(X,Y) <= H(X) + H(Y)", width - 12, height - 12);
}
```
## Links (Wikipedia order)
<!-- injected from _registry/childlinks/Joint_entropy.json (2026-07-30T02:09:12Z) -->
`Asymptotic_equipartition_property` · `Bit` · `Channel_capacity` · [[Conditional_entropy]] · `Conditional_mutual_information` · [[Differential_entropy]] · `Directed_information` · [[Entropy_(information_theory)]] · `Entropy_rate` · [[Information_theory]] · `Integral` · `Limiting_density_of_discrete_points` · `Mutual_information` · `Noisy-channel_coding_theorem` · `Quantities_of_information` · `Random_variable` · `Rate–distortion_theory` · `Shannon's_source_coding_theorem` · `Shannon–Hartley_theorem` · `Subadditivity` · `Theresa_M._Korn` · `Venn_diagram`
## From the Real GENERATIVE library
*Joint entropy — placed from the Real G.E.N.E.R.A.T.I.V.E. course library (Information room). Source: Wikimedia Commons (via Wikipedia article media). [Details & license](https://commons.wikimedia.org/wiki/File:Binaryerasurechannel.png).*
> In information theory, joint entropy is a measure of the uncertainty associated with a set of variables.[2] ([Wikipedia](https://en.wikipedia.org/wiki/Joint_entropy))
<!-- REAL-GENERATIVE-MEDIA:END -->
<!-- LOCAL-MEDIA-PASS:START -->
## From the vault media library
!Joint entropy thumb.png
*Joint Entropy — from the vault's own media holdings, placed 2026-07-09. MTN / Wikitube.io original · CC BY-SA 4.0.*
<!-- LOCAL-MEDIA-PASS:END -->
> **Room:** [[Information]] · **Status:** ✅ shipped
## Overview
**Joint entropy**, written H(X, Y), measures the total uncertainty in a *pair* of discrete random variables observed together — the average number of bits needed to encode one outcome of the joint random variable (X, Y) drawn from the joint distribution p(x, y). Its definition is the natural Shannon generalisation, H(X, Y) = -Sum_{x, y} p(x, y) log_2 p(x, y), and it satisfies one tight inequality and one chain-rule equality that organise most of the rest of [[Information_theory|information theory]]: max{H(X), H(Y)} <= H(X, Y) <= H(X) + H(Y), with the upper bound met precisely when X and Y are statistically independent and the lower bound met when one variable is a deterministic function of the other. The gap H(X) + H(Y) - H(X, Y) is the *mutual information* I(X ; Y), the bits per symbol that observing one variable tells you about the other; the two chain-rule decompositions H(X, Y) = H(X) + H(Y | X) = H(Y) + H(X | Y) connect joint entropy to the conditional entropies that quantify channel noise. Joint entropy is the rate floor for any code that compresses (X, Y) pairs without distortion, and it is the foundation of multivariate information theory: every later quantity — [[Conditional_entropy|conditional entropy]], mutual information, transfer [[Entropy|entropy]], multivariate entropy rates — is a difference of joint entropies of subsets of variables.
## See also
- Room hub: [[Information]]
- p5.js Editor conventions: P5 JS EDITOR
- Wiki root: MAIN
---
*Scaffolded by `generative-microsim` from row 0 of the Information sheet on 2026-04-30T17:41:12Z.*
Letters: entropy · mined_information · chart_glyph_dictionary · distribution · probability · lattice · measurement · discretization
<!-- REAL-GENERATIVE-MEDIA:START -->
<!-- CRAFT-LINK:START g12 -->
*Built to the [[WT!P5_js_Microsim_Master_Class|p5.js Master Class]].*
<!-- CRAFT-LINK:END -->
## Wikipedia : Wikitube
**Strict pair:** [Wikipedia](https://en.wikipedia.org/wiki/Joint_entropy) : [Wikitube](https://en.wikitube.io/wiki/Joint_entropy)
## Previous hub tags
Tree parent: [[Information_theory]].
Legacy hubs: `GENERATIVE`.
---
*Sources: 1 legacy note. Minted wave 1, 2026-07-30 (v1.6 order).*