Various comments

[mateuszbaran]

A few comments about line searches, potentially relevant to the work here.

1. Hager-Zhang line search is in my experience a very good method for gradient-based line searches but it seems to be absent from NonlinearSolve.jl , other than through LineSearchesJL. Is it planned to be re-implemented here?
2. Nonlinear gradient-based line searches actually need to compute inner products between gradient at a point and vector-transported descent direction. See here for example: https://arxiv.org/pdf/2207.01855.pdf . It means that for constrained optimization we need to do the transport (projection would often be fine but not necessarily always the best choice).
3. The inner product actually doesn't require the full gradient. Optim.jl currently doesn't exploit it but it's something that crossed my mind when analyzing the code. We should sometimes be able to compute such products faster than the full gradient + dot. For example forward mode AD would handle this perfectly fine. Maybe that's something you could consider.
4. Default max stepsize: I've encountered some problems in which the stepsize found by line search greatly exceeded (by orders of magnitude) the injectivity radius of exponential map. Manopt.jl patches it by using a manifold-dependent max stepsize (for example $\pi$ for a sphere). LineSearches.jl always defaults to Inf.

[Vaibhavdixit02]

Hager-Zhang line search is in my experience a very good method for gradient-based line searches but it seems to be absent from NonlinearSolve.jl , other than through LineSearchesJL. Is it planned to be re-implemented here?

I think it would be implemented. We will come up with a more concrete roadmap for this repo very soon since we seem to have a promising GSoC candidate to work on this!