Unit 4: Multivariate Correlation, Anscombe's Quartet & PCA 40 mins study timeADVANCED

Multivariate Relationships, Anscombe's Quartet & PCA Intuition

Explore linear vs monotonic relationships, why Anscombe's quartet proves summary statistics are insufficient without visualisation, and how PCA projects high-dimensional feature spaces onto orthogonal principal axes.

Verified: Faculty Peer Review Board

Learning Objectives

•Contrast Pearson linear correlation with Spearman rank-order correlation.
•Demonstrate the pedagogical significance of Anscombe's Quartet and Simpson's Paradox.
•Compute and interpret covariance matrices and eigenvectors in PCA.
•Construct a Scree Plot to determine dimensional retention thresholds.

Essential Prerequisites

•Descriptive statistics, variance/covariance, and basic matrix algebra

Layer 1: Intuition & Why It Matters

The Core Mental Model

“Imagine projecting the 3D shadow of a flying bird onto a wall. If the light comes from the front, you only see a thin sliver. If the light comes from above, the shadow captures the full wingspan. PCA automatically rotates the camera to the angle where the shadow casts the widest possible wingspan (maximum variance).”

Why This Exists

Real datasets frequently contain 50 to 500 features with heavy multicollinearity. PCA compresses high-dimensional noise into a compact set of orthogonal signals, preventing overfitting and enabling 2D visual discovery.

Beginner Foundation

Correlation does not equal causation. Furthermore, high correlation can be entirely manufactured by a single outlier, or obscured by non-linear relationships. Spearman ranking helps uncover curves that Pearson misses.

Micro Concepts Decomposition

MICRO CONCEPT 1Canonical Object

Pearson vs. Spearman Correlation

Pearson r measures linear correlation between continuous variables and is sensitive to outliers. Spearman rho evaluates monotonic relationships by computing Pearson on ranked values.

Key Takeaway: Pearson: linear relationship; Spearman: monotonic relationship (resistant to non-linear scaling and outliers).

MICRO CONCEPT 2Canonical Object

Anscombe's Quartet & Why Plots are Mandatory

Anscombe's Quartet consists of four datasets with nearly identical summary statistics (mean, variance, correlation r = 0.816, regression line y = 3.0 + 0.5x), yet drastically different graphs. Summary statistics alone can conceal non-linear patterns and leverage points.

Key Takeaway: Never trust summary statistics without plotting the data. Different distributions can produce identical mean and correlation.

MICRO CONCEPT 3Canonical Object

Principal Component Analysis (PCA) Intuition

PCA is an unsupervised linear dimensionality reduction technique. It rotates the coordinate axes to align with directions of maximum variance. Principal Component 1 (PC1) captures the largest variance; PC2 is orthogonal to PC1 and captures the next largest variance.

Key Takeaway: PCA finds orthogonal directions of maximum variance; PC1 explains the most variance, PC2 is perpendicular to PC1.

MICRO CONCEPT 4Canonical Object

Scree Plot & Cumulative Explained Variance

Eigenvalues represent the variance captured by each principal component. A Scree Plot graphs eigenvalues against component number. The 'elbow' identifies the optimal number of dimensions to retain.

Key Takeaway: Use the Scree Plot elbow and cumulative explained variance ratio (e.g. >= 85%) to choose component count.

Layer 3 & 4: Formal Specification & Mechanism

Hardware State Machine Architecture

Given centered data matrix X (n x p), the covariance matrix is Sigma = (1 / (n - 1)) * X^T * X. Singular Value Decomposition (SVD) of Sigma yields eigenvectors V and eigenvalues Lambda: Sigma * v_i = lambda_i * v_i. The principal component scores are Z = X * V.

Step-by-Step PCA Protocol: 1. Standardize features to mean = 0, variance = 1 (StandardScaler). 2. Compute the p x p Covariance/Correlation matrix. 3. Compute eigenvalues and eigenvectors via Eigendecomposition. 4. Sort eigenvalues in descending order. 5. Select top k components where cumulative variance >= 80-90%. 6. Project original data onto top k eigenvectors.

Layer 7: Interactive Laboratory

Interactive Simulator

EDA • VISUALIZATIONMultivariate Relationships & Collinearity Heatmap Visualizer

Launch Fullscreen Lab

EDA • BIVARIATE ANALYSISPearson r & Linear Fit

Correlation Matrix & Scatter Plot Regression Laboratory

Pearson r:0.993

Target Correlation (r):0.8

Inject High-Leverage Outlier Point (95, 10)

Anscombe's / Simpson's Rule:

Pearson correlation r = Cov(X, Y) / (σ_X × σ_Y) measures only linear association. Notice how toggling a single extreme leverage outlier dramatically flips or degrades the regression slope! Never report correlation without visualizing the scatter plot.

Layer 5: Step-by-Step Worked Numerical Example

End-to-End Execution Trace

Anscombe's Quartet Statistics: For all 4 distinct datasets: Mean of X: 9.0 Sample variance of X: 11.0 Mean of Y: 7.50 Sample variance of Y: 4.125 Correlation between X and Y: 0.816 Linear regression line: y = 3.00 + 0.500x Dataset 1: Simple linear relationship with scatter Dataset 2: Smooth quadratic curve (non-linear!) Dataset 3: Strict linear line with 1 extreme outlier Dataset 4: All X values identical at 8 except 1 extreme leverage point!

Layer 6: Active Runtime CodeLab

Step-by-Step Code Execution (PYTHON)

AURXON Sandbox

AURXON Digital Engine

Font

main.pyGlacier Light

Ln 1 • Python 3.12

import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
 
# Synthetic multi-feature dataset
np.random.seed(42)
X = np.random.multivariate_normal(
    mean=[0, 0, 0],
    cov=[[3.0, 2.5, 1.2],
         [2.5, 3.0, 1.0],
         [1.2, 1.0, 2.0]],
    size=100
)
 
# Step 1: Standardize
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
 
# Step 2: Fit PCA
pca = PCA(n_components=3)
X_pca = pca.fit_transform(X_scaled)
 
print("Explained Variance Ratio per Component:")
for i, ratio in enumerate(pca.explained_variance_ratio_):
    print(f"  PC{i+1}: {ratio*100:.2f}%")
 
cumulative_var = np.cumsum(pca.explained_variance_ratio_)
print(f"Cumulative Variance (PC1 + PC2): {cumulative_var[1]*100:.2f}%")

774 chars • 30 lines • Ln 1UTF-8 • 4 Spaces

Interactive Terminal Shell

Sandbox Terminal Ready

Click Run Code or press Ctrl+Enter to compile and execute.

⚡ AURXON Bitstream Runtime v4.8IDRAS Academic Virtual Node

Common Student Pitfalls & Mistakes

Where Students Lose Marks

❌ Mistake: Applying PCA without first standardizing features that have different units (e.g. age in years vs income in dollars).

✓ Correct Understanding: PCA is sensitive to variable variance. Always scale variables with StandardScaler so high-magnitude columns don't dominate principal components artificially.

Layer 8: Practice & Knowledge Verification

Active Assessment Quiz

Interactive Assessment EngineQuestion 1 of 1

Multivariate Relationships, Anscombe's Quartet & PCA Intuition — Practice Questions

ADVANCED LevelScore: 0/0

Why must datasets be standardized using StandardScaler prior to running Principal Component Analysis (PCA)?

Academic Evaluation Preparation

Viva Examination & University Scoring Strategy

Standard Viva Examination Questions

Q1: Why must features be standardized before PCA?

Answer: PCA maximizes variance. If one feature is measured in thousands (e.g. salary) and another in single digits (e.g. years of experience), the unscaled feature will dominate PC1 regardless of its true informational value.

How to Write High-Scoring University Exam Answers

Differentiate Pearson and Spearman correlation coefficients. Explain Anscombe's Quartet and interpret its pedagogical lesson. Derive the geometric and algebraic formulation of PCA, explaining eigenvectors, eigenvalues, and how Scree plots determine component retention.