Loss Functions, Convex Optimization & Gradient Descent Dynamics

Consider linear hypothesis h_θ(x) = θ^T x. Mean Squared Error (MSE) Cost Function: J(θ) = (1 / 2m) * Σ_{i=1}^m (h_θ(x^{(i)}) - y^{(i)})^2 Gradient Vector: ∇J(θ) = [ ∂J/∂θ_0, ∂J/∂θ_1, ..., ∂J/∂θ_n ]^T where ∂J/∂θ_j = (1 / m) * Σ_{i=1}^m (h_θ(x^{(i)}) - y^{(i)}) * x_j^{(i)} Parameter Update Rule: θ := θ - α * ∇J(θ), where α > 0 is the learning rate. Variants: 1. Batch Gradient Descent: Computes gradient over entire training set m. 2. Stochastic Gradient Descent (SGD): Updates parameters for each training sample (noisy, faster escapes from saddle points). 3. Mini-Batch: Computes gradient over batch size B in [32, 256].