Add the ability to store the trace for nonlinear solve algorithms

[avik-pal]

Fixes #58 (kind of). This is meant to add tracing capability to native NonlinearSolve solvers

Checklist

• Any code changes were done in a way that does not break public API
• All documentation related to code changes were updated
• The new code follows the contributor guidelines, in particular the ScioML Style Guide and COLPRAC.
• Upstream Patches
  • DiffEqBase needs 3 new kwargs
  • SciMLBase needs an additional storage for trace
  • Bump compact entries
• Extend to all solvers
  • NR
  • TR
  • GN
  • LM
  • Broyden
  • Klement
  • PT
• Display which algorithm is printing the trace (needed for polyalgorithms)
• Add option to log every k iterations

Additional context

Add any other context about the problem here.

Example Usage

julia> solve(prob; show_trace = Val(true), trace_level = TraceAll(150))

Algorithm: GeneralKlement(linsolve = LUFactorization())

----     -------------        -----------          -------             
Iter     f(u) inf-norm        Step 2-norm          cond(J)             
----     -------------        -----------          -------             
0        3.50000000e+07       0.00000000e+00       1.00000000e+00      
Final    3.50000000e+07      
----------------------      

Algorithm: GeneralBroyden()

----     -------------        -----------          -------             
Iter     f(u) inf-norm        Step 2-norm          cond(J)             
----     -------------        -----------          -------             
0        3.50000000e+07       0.00000000e+00       1.00000000e+00      
Final    3.50000000e+07      
----------------------      

Algorithm: NewtonRaphson(ad = AutoForwardDiff())

----     -------------        -----------          -------             
Iter     f(u) inf-norm        Step 2-norm          cond(J)             
----     -------------        -----------          -------             
0        3.50000000e+07       0.00000000e+00       1.31206964e+13      
Final    3.50000000e+07      
----------------------      

Algorithm: NewtonRaphson(ad = AutoForwardDiff(), linesearch = BackTracking())

----     -------------        -----------          -------             
Iter     f(u) inf-norm        Step 2-norm          cond(J)             
----     -------------        -----------          -------             
0        3.50000000e+07       0.00000000e+00       3.12767074e+05      
150      1.96894976e+07       1.62771357e+01       2.54208181e+09      
300      1.95822332e+07       8.59874893e+00       3.54255023e+09      
450      1.95254850e+07       5.64893544e+00       5.30212288e+09      
600      1.94937971e+07       4.75390720e+00       7.69360853e+09      
750      1.94830888e+07       2.36799303e+00       1.23415632e+10      
900      1.94776278e+07       1.58184438e+00       1.84208351e+10      
Final    1.94758224e+07      
----------------------      

Algorithm: TrustRegion(ad = AutoForwardDiff(), radius_update_scheme = Simple)

----     -------------        -----------          -------             
Iter     f(u) inf-norm        Step 2-norm          cond(J)             
----     -------------        -----------          -------             
0        3.50000000e+07       0.00000000e+00       8.15768984e+11      
150      1.76071235e+07       1.55362216e+03       2.50639043e+12      
300      1.76071235e+07       1.55362216e+03       2.50639043e+12      
450      1.76071235e+07       1.55362216e+03       2.50639043e+12      
600      1.76071235e+07       1.55362216e+03       2.50639043e+12      
750      1.76071235e+07       1.55362216e+03       2.50639043e+12      
900      1.76071235e+07       1.55362216e+03       2.50639043e+12      
Final    1.76071235e+07      
----------------------      

Algorithm: TrustRegion(ad = AutoForwardDiff(), radius_update_scheme = Bastin)

----     -------------        -----------          -------             
Iter     f(u) inf-norm        Step 2-norm          cond(J)             
----     -------------        -----------          -------             
0        3.50000000e+07       0.00000000e+00       8.16936903e+11      
150      8.33325158e-01       2.82025866e-03       1.58940933e+11      
300      8.33325110e-01       2.82025866e-03       1.61054537e+11      
450      8.33325063e-01       2.82025866e-03       1.63225116e+11      
600      8.33325017e-01       2.82025866e-03       1.65455004e+11      
750      8.33324971e-01       2.82025866e-03       1.67746665e+11      
900      8.33324926e-01       2.82025866e-03       1.70102704e+11      
Final    8.33324896e-01      
----------------------      
u: 3-element Vector{Float64}:
   29.8340911480401
 2349.223979895471
 1271.6287520542758

[codecov]

Codecov Report

Attention: 30 lines in your changes are missing coverage. Please review.

Comparison is base (46912f2) 94.04% compared to head (1b9c188) 93.07%.

Files	Patch %	Lines
src/trace.jl	79.06%	18 Missing ⚠️
src/trustRegion.jl	56.25%	7 Missing ⚠️
src/NonlinearSolve.jl	90.47%	2 Missing ⚠️
src/utils.jl	66.66%	2 Missing ⚠️
src/gaussnewton.jl	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #292      +/-   ##
==========================================
- Coverage   94.04%   93.07%   -0.97%     
==========================================
  Files          20       21       +1     
  Lines        1896     2065     +169     
==========================================
+ Hits         1783     1922     +139     
- Misses        113      143      +30     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ChrisRackauckas]

We definitely need to update https://docs.sciml.ai/NonlinearSolve/stable/basics/solve/#solver_options as well.