Optimization with Float32

[ArtemStrashko]

Hello,
Can I use ConjugateGradient with Float32 function value and gradient? I have been running calculations with Float64 and everything has been fine, but with Float32 I get the following error:

MethodError: no method matching OptimKit.LineSearchPoint(::Float32, ::Float32, ::Float64, ::ITensor, ::Float32, ::ITensor, ::ITensor)
Closest candidates are:
  OptimKit.LineSearchPoint(::T, ::T, ::T, ::X, ::T, ::G, ::G) where {T<:Real, X, G} at /mnt/home/astrashko/.julia/packages/OptimKit/xpmbV/src/linesearches.jl:6

[Jutho]

Could you provide the full stack trace. As you can see from the error message, it seems like a `Float64' value slipped into the calculation. I would need some more details to figure out whether this is an error on my behalf (because I somewhere hard-coded a Float64 value where I shouldn't have) or whether this happens in the computation of the function value or gradient ?

[ArtemStrashko]

Here it is:

MethodError: no method matching OptimKit.LineSearchPoint(::Float32, ::Float32, ::Float64, ::ITensor, ::Float32, ::ITensor, ::ITensor)
Closest candidates are:
  OptimKit.LineSearchPoint(::T, ::T, ::T, ::X, ::T, ::G, ::G) where {T<:Real, X, G} at /mnt/home/astrashko/.julia/packages/OptimKit/xpmbV/src/linesearches.jl:6

Stacktrace:
  [1] (::HagerZhangLineSearch{Rational{Int64}})(fg::Function, x₀::ITensor, η₀::ITensor, fg₀::Tuple{Float32, ITensor}; retract::Function, inner::typeof(OptimKit._inner), initialguess::Float64, acceptfirst::Bool)
    @ OptimKit ~/.julia/packages/OptimKit/xpmbV/src/linesearches.jl:257
  [2] optimize(fg::var"#28#29"{Vector{ITensor}, Vector{Int64}, Matrix{ITensor}, Matrix{ITensor}, Matrix{ITensor}}, x::ITensor, alg::GradientDescent{Float64, HagerZhangLineSearch{Rational{Int64}}}; precondition::typeof(OptimKit._precondition), finalize!::typeof(OptimKit._finalize!), retract::Function, inner::typeof(OptimKit._inner), transport!::Function, scale!::typeof(OptimKit._scale!), add!::Function, isometrictransport::Bool)
    @ OptimKit ~/.julia/packages/OptimKit/xpmbV/src/gd.jl:43
  [3] optimize(fg::Function, x::ITensor, alg::GradientDescent{Float64, HagerZhangLineSearch{Rational{Int64}}})
    @ OptimKit ~/.julia/packages/OptimKit/xpmbV/src/gd.jl:18
  [4] optimizeMPS(mps::Vector{ITensor}, mpo::Vector{ITensor}, images::Matrix{ITensor}, labels::Vector{Int64}, parameters::Dict{String, Any})
    @ Main ~/Desktop/FI_projects/DeepMPS/code/Optimization.jl:102
  [5] optimizeMPS(mps::Vector{ITensor}, mpo::Vector{ITensor}, images::Matrix{ITensor}, labels::Vector{Int64})
    @ Main ~/Desktop/FI_projects/DeepMPS/code/Optimization.jl:74
  [6] optimizeFullTN(add_noise::Bool, parameters::Dict{String, Any})
    @ Main ~/Desktop/FI_projects/DeepMPS/code/Optimization.jl:51
  [7] optimizeFullTN(add_noise::Bool) (repeats 2 times)
    @ Main ~/Desktop/FI_projects/DeepMPS/code/Optimization.jl:5
  [8] top-level scope
    @ ./timing.jl:210 [inlined]
  [9] top-level scope
    @ ./In[7]:0
 [10] eval
    @ ./boot.jl:360 [inlined]
 [11] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:1116

[Jutho]

It does not seem easy to find out where the Float64 value came from without further details on your code. Could you at least check that val, grad = fg(your_initial_x) does not produce a Float64 value for either val or norm(grad).

[Jutho]

If you have code that you can share to reproduce the bug, I can take a more detailed look.

[ArtemStrashko]

Hi,
Sorry, I finally found that it was my fault: somehow broadcasting division by an integer across Float32 tensor elements, changed tensor elements type to Float64.
Thank you,
Artem.