Fractional Calculus in Backpropagation Learning: Theory, Algorithms, and Convergence Analysis

Source abstract

We develop a mathematical framework for integrating fractional calculus into the backpropagation learning algorithm. We present two complementary approaches: (i) fractional-order gradient descent (FOGD), which replaces the integer-order derivative in the optimizer with a Caputo or Grünwald–Letnikov fractional derivative, inducing power-law memory in weight updates; and (ii) fractional-order neural networks (FONN), which employ fractional derivatives within the network's activation and chain rule. We prove the equivalence of the Grünwald–Letnikov and Riemann–Liouville definitions under explicit regularity and finite-part conditions, establish the Mittag-Leffler solution to fractional gradient flow, and derive its power-law asymptotics via complete monotonicity and Hankel contour integration (rather than by a heuristic Tauberian argument). We state a corrected fractional Taylor theorem valid only for functions that are not classically $C^1$ at the expansion point, and we explicitly note the well-known counterexample for standard smooth functions. We prove convergence of stochastic fractional gradient descent on non-convex objectives at rate $O(T^{-1/2}) + O(T^{-\alpha})$ under an explicitly stated bounded-trajectory assumption, and we identify the memory-bound lemma as an assumption rather than a proven result. Numerical verification confirms the theoretical predictions where the assumptions hold. The framework generalizes SGD, momentum, and Nesterov acceleration as the special case $\alpha = 1$, connects to anomalous diffusion on loss landscapes, and yields an emergent power-law learning-rate schedule.