Convergence Analysis in Deep Learning (Part 1)
Unify definitions and assumptions regarding convexity, strong convexity, smoothness, and stochastic gradients, and derive inequalities …
Convergence Analysis
A series on convergence analysis for optimization, distributed learning, and federated learning.
Convergence Analysis in Deep Learning (Part 2)
Derive first-order method convergence bounds under smooth/non-smooth and convex/non-convex conditions, clarifying step sizes, output points, …
Convergence Analysis in Deep Learning (Part 3)
Deriving strong convex error bounds and non-convex stationary point bounds for mini-batch SGD under assumptions of conditionally unbiased …
Convergence Analysis in Distributed Machine Learning (Part 4)
Deriving strong convexity convergence bounds for synchronous, asynchronous, and K-Async SGD, clarifying assumptions of independent sampling, …
Convergence Analysis in Federated Learning (Part 5)
Deriving non-convex stationary point bounds and strongly convex error bounds under full participation from FedAvg's local updates, …