TY - JOUR

T1 - Stochastic relaxed inertial forward-backward-forward splitting for monotone inclusions in Hilbert spaces

AU - Cui, Shisheng

AU - Shanbhag, Uday

AU - Staudigl, Mathias

AU - Vuong, Phan

N1 - Publisher Copyright:
© 2022, The Author(s).

PY - 2022/11

Y1 - 2022/11

N2 - We consider monotone inclusions defined on a Hilbert space where the operator is given by the sum of a maximal monotone operator T and a single-valued monotone, Lipschitz continuous, and expectation-valued operator V. We draw motivation from the seminal work by Attouch and Cabot (Attouch in AMO 80:547–598, 2019, Attouch in MP 184: 243–287) on relaxed inertial methods for monotone inclusions and present a stochastic extension of the relaxed inertial forward–backward-forward method. Facilitated by an online variance reduction strategy via a mini-batch approach, we show that our method produces a sequence that weakly converges to the solution set. Moreover, it is possible to estimate the rate at which the discrete velocity of the stochastic process vanishes. Under strong monotonicity, we demonstrate strong convergence, and give a detailed assessment of the iteration and oracle complexity of the scheme. When the mini-batch is raised at a geometric (polynomial) rate, the rate statement can be strengthened to a linear (suitable polynomial) rate while the oracle complexity of computing an ϵ-solution improves to O(1 / ϵ). Importantly, the latter claim allows for possibly biased oracles, a key theoretical advancement allowing for far broader applicability. By defining a restricted gap function based on the Fitzpatrick function, we prove that the expected gap of an averaged sequence diminishes at a sublinear rate of O(1 / k) while the oracle complexity of computing a suitably defined ϵ-solution is O(1 / ϵ1+a) where a> 1. Numerical results on two-stage games and an overlapping group Lasso problem illustrate the advantages of our method compared to competitors.

AB - We consider monotone inclusions defined on a Hilbert space where the operator is given by the sum of a maximal monotone operator T and a single-valued monotone, Lipschitz continuous, and expectation-valued operator V. We draw motivation from the seminal work by Attouch and Cabot (Attouch in AMO 80:547–598, 2019, Attouch in MP 184: 243–287) on relaxed inertial methods for monotone inclusions and present a stochastic extension of the relaxed inertial forward–backward-forward method. Facilitated by an online variance reduction strategy via a mini-batch approach, we show that our method produces a sequence that weakly converges to the solution set. Moreover, it is possible to estimate the rate at which the discrete velocity of the stochastic process vanishes. Under strong monotonicity, we demonstrate strong convergence, and give a detailed assessment of the iteration and oracle complexity of the scheme. When the mini-batch is raised at a geometric (polynomial) rate, the rate statement can be strengthened to a linear (suitable polynomial) rate while the oracle complexity of computing an ϵ-solution improves to O(1 / ϵ). Importantly, the latter claim allows for possibly biased oracles, a key theoretical advancement allowing for far broader applicability. By defining a restricted gap function based on the Fitzpatrick function, we prove that the expected gap of an averaged sequence diminishes at a sublinear rate of O(1 / k) while the oracle complexity of computing a suitably defined ϵ-solution is O(1 / ϵ1+a) where a> 1. Numerical results on two-stage games and an overlapping group Lasso problem illustrate the advantages of our method compared to competitors.

KW - Complexity

KW - Dynamic sampling

KW - Monotone operator splitting

KW - Stochastic approximation

KW - Variance reduction

UR - http://www.scopus.com/inward/record.url?scp=85139379457&partnerID=8YFLogxK

U2 - 10.1007/s10589-022-00399-3

DO - 10.1007/s10589-022-00399-3

M3 - Article

AN - SCOPUS:85139379457

SN - 0926-6003

VL - 83

SP - 465

EP - 524

JO - Computational Optimization and Applications

JF - Computational Optimization and Applications

IS - 2

ER -