I will discuss how these very simple, yet ubiquitous, algorithms have a complex behavior in high-dimensional non-convex settings. \\ \\ (The thesis defense will be streamed on \\ https://eu.bbcollab.com/guest/12a6a5bb79834cd69bc2760e7c968180)