Optimization and Control (math.OC)

Abstract: The push-sum algorithm is probably the most important distributed averaging approach over directed graphs, which has been applied to various problems including distributed optimization. This paper establishes the explicit absolute probability sequence for the push-sum algorithm, and based on which, constructs quadratic Lyapunov functions for push-sum based distributed optimization algorithms. As illustrative examples, the proposed novel analysis tool can improve the convergence rates of the subgradient-push and stochastic gradient-push, two important algorithms for distributed convex optimization over unbalanced directed graphs. Specifically, the paper proves that the subgradient-push algorithm converges at a rate of $O(1/\sqrt{t})$ for general convex functions and stochastic gradient-push algorithm converges at a rate of $O(1/t)$ for strongly convex functions, over time-varying unbalanced directed graphs. Both rates are respectively the same as the state-of-the-art rates of their single-agent counterparts and thus optimal, which closes the theoretical gap between the centralized and push-sum based (sub)gradient methods. The paper further proposes a heterogeneous push-sum based subgradient algorithm in which each agent can arbitrarily switch between subgradient-push and push-subgradient. The heterogeneous algorithm thus subsumes both subgradient-push and push-subgradient as special cases, and still converges to an optimal point at an optimal rate. The proposed tool can also be extended to analyze distributed weighted averaging.

Abstract: In his recent research M. K. Tam (2018) considered a framework for the analysis of iterative algorithms which can be described in terms of a structured set-valued operator. At each point in the ambient space, the value of the operator can be expressed as a finite union of values of single-valued paracontracting operators. He showed that the associated fixed point iteration is locally convergent around strong fixed points. This result generalizes a theorem due to Bauschke and Noll (2014). In the present paper we generalize the result of Tam and show the global convergence of his algorithm for an arbitrary starting point. An analogous result is also proved for the Krasnosel'ski-Mann iterations.

Abstract: We study the training dynamics of shallow neural networks, in a two-timescale regime in which the stepsizes for the inner layer are much smaller than those for the outer layer. In this regime, we prove convergence of the gradient flow to a global optimum of the non-convex optimization problem in a simple univariate setting. The number of neurons need not be asymptotically large for our result to hold, distinguishing our result from popular recent approaches such as the neural tangent kernel or mean-field regimes. Experimental illustration is provided, showing that the stochastic gradient descent behaves according to our description of the gradient flow and thus converges to a global optimum in the two-timescale regime, but can fail outside of this regime.

Abstract: We study generalized games with full row rank equality constraints and we provide a strikingly simple proof of strong monotonicity of the associated KKT operator. This allows us to show linear convergence to a variational equilibrium of the resulting primal-dual pseudo-gradient dynamics. Then, we propose a fully-distributed algorithm with linear convergence guarantee for aggregative games under partial-decision information. Based on these results, we establish stability properties for online GNE seeking in games with time-varying cost functions and constraints. Finally, we illustrate our findings numerically on an economic dispatch problem for peer-to-peer energy markets.

Abstract: Given two nonempty and disjoint intersections of closed and convex subsets, we look for a best approximation pair relative to them, i.e., a pair of points, one in each intersection, attaining the minimum distance between the disjoint intersections. We propose an iterative process based on projections onto the subsets which generate the intersections. The process is inspired by the Halpern-Lions-Wittmann-Bauschke algorithm and the classical alternating process of Cheney and Goldstein, and its advantage is that there is no need to project onto the intersections themselves, a task which can be rather demanding. We prove that under certain conditions the two interlaced subsequences converge to a best approximation pair. These conditions hold, in particular, when the space is Euclidean and the subsets which generate the intersections are compact and strictly convex. Our result extends the one of Aharoni, Censor and Jiang ["Finding a best approximation pair of points for two polyhedra", Computational Optimization and Applications 71 (2018), 509-523] which considered the case of finite-dimensional polyhedra.

Abstract: Existing planning approaches for onshore wind farm siting and network integration often do not meet minimum cost solutions or social and environmental considerations. In this paper, we develop an approach for the multi-objective optimization of turbine locations and their network connection using a Quota Steiner tree problem. Applying a novel transformation on a known directed cut formulation, reduction techniques, and heuristics, we design an exact solver that makes large problem instances solvable and outperforms generic MIP solvers. Although our case studies in selected regions of Germany show large trade-offs between the objective criteria of cost and landscape impact, small burdens on one criterion can significantly improve the other criteria. In addition, we demonstrate that contrary to many approaches for exclusive turbine siting, network integration must be simultaneously optimized in order to avoid excessive costs or landscape impacts in the course of a wind farm project. Our novel problem formulation and the developed solver can assist planners in decision making and help optimize wind farms in large regions in the future.