Attempt to access 0-element Vector{Float64} at index [1]

[Affie]

Just capturing so long, will investigate further when I have a chance:

• All masters and julia 1.6.0-beta1
• Tried to solve loaded fg

    nested task error: BoundsError: attempt to access 0-element Vector{Float64} at index [1]
    Stacktrace:
      [1] getindex
        @ ./array.jl:801 [inlined]
      [2] calcPPE(var::DFGVariable{Point2}, varType::Point2; method::Type{MeanMaxPPE}, solveKey::Symbol)
        @ IncrementalInference ~/.julia/dev/IncrementalInference/src/FGOSUtils.jl:204
      [3] calcPPE(var::DFGVariable{Point2}; method::Type{MeanMaxPPE}, solveKey::Symbol)
        @ IncrementalInference ~/.julia/dev/IncrementalInference/src/FGOSUtils.jl:215
      [4] setPPE!(subfg::LightDFG{SolverParams, DFGVariable, DFGFactor}, label::Symbol, solveKey::Symbol, method::Type{MeanMaxPPE}, newPPEVal::Nothing)
        @ IncrementalInference ~/.julia/dev/IncrementalInference/src/FGOSUtils.jl:552
      [5] setPPE!
        @ ~/.julia/dev/IncrementalInference/src/FGOSUtils.jl:550 [inlined]
      [6] updateFromSubgraph_StateMachine(csmc::CliqStateMachineContainer{BayesTreeNodeData, LightDFG{SolverParams, DFGVariable, DFGFactor}, LightDFG{SolverParams, DFGVariable, DFGFactor}, MetaBayesTree})
        @ IncrementalInference ~/.julia/dev/IncrementalInference/src/CliqueStateMachine.jl:751
      [7] (::StateMachine{CliqStateMachineContainer})(userdata::CliqStateMachineContainer{BayesTreeNodeData, LightDFG{SolverParams, DFGVariable, DFGFactor}, LightDFG{SolverParams, DFGVariable, DFGFactor}, MetaBayesTree}, timeout::Nothing; pollinterval::Float64, breakafter::Function, verbose::Bool, verbosefid::Base.TTY, verboseXtra::IncrementalInference.CliqStatus, iterlimit::Int64, injectDelayBefore::Nothing, recordhistory::Bool, housekeeping_cb::IncrementalInference.var"#377#379"{IncrementalInference.TreeClique})
        @ FunctionalStateMachine ~/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82
      [8] initStartCliqStateMachine!(dfg::LightDFG{SolverParams, DFGVariable, DFGFactor}, tree::MetaBayesTree, cliq::IncrementalInference.TreeClique, timeout::Nothing; oldcliqdata::BayesTreeNodeData, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, show::Bool, incremental::Bool, limititers::Int64, upsolve::Bool, downsolve::Bool, recordhistory::Bool, delay::Bool, logger::Base.CoreLogging.SimpleLogger, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol)
        @ IncrementalInference ~/.julia/dev/IncrementalInference/src/CliqueStateMachine.jl:63
      [9] tryCliqStateMachineSolve!(dfg::LightDFG{SolverParams, DFGVariable, DFGFactor}, treel::MetaBayesTree, cliqKey::Int64, timeout::Nothing; oldtree::MetaBayesTree, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, limititers::Int64, downsolve::Bool, incremental::Bool, delaycliqs::Vector{Symbol}, recordcliqs::Vector{Symbol}, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol)
        @ IncrementalInference ~/.julia/dev/IncrementalInference/src/SolverAPI.jl:110
     [10] (::IncrementalInference.var"#433#436"{MetaBayesTree, Bool, Bool, Base.TTY, Bool, Bool, Vector{Symbol}, Vector{Symbol}, Symbol, LightDFG{SolverParams, DFGVariable, DFGFactor}, MetaBayesTree, Nothing, ProgressMeter.ProgressUnknown, Int64, Int64})()
        @ IncrementalInference ./task.jl:406

[Affie]

I think it's related to this issue: #993
Caused by this line:

IncrementalInference.jl/src/GraphProductOperations.jl

Line 322 in 6ddd346

setkde && 0 < size(getPoints(newPts),2) ? setValKDE!(dfg, sym, newPts, false, infdim) : nothing

Perhaps it should rather be size(getPoints(newPts), 1)

julia> size([],2)
1

Also maybe for multiproc:

IncrementalInference.jl/src/CliqStateMachineUtils.jl

Line 510 in 6ddd346

setValKDE!(subfg, fr, result[1], false, result[2])

IncrementalInference.jl/src/CliqStateMachineUtils.jl

Line 504 in 6ddd346

setValKDE!(subfg, fr, downresult[fr][1], false, downresult[fr][2])

[Affie]

I haven't changed it because it might be a symptom of a larger problem:
Perhaps related to ongoing work in #1010? It's with the default of useLikelihoodMessages = true and I think there are no messages sent up for the specific variable.

[dehann]

oops, we should also check order of operations here:

setkde && 0 < size(getPoints(newPts),2) ?

is && inside one if, or for a secondary outer if? I'm not sure which we meant here...

Then size(getPoints(newPts),2) is cheap way to see how may samples there are (i.e. number of columns). So the ,2 is "right", but the size([],2)==1 could be a real problem, however, this should be a Matrix not Vector, right?

Suggestion

Looks like idea of this line is to set values if something exists or just do nothing if empty. So lets just use length?

( setkde && 0 < length(getPoints(newPts)) ) ? setValKDE!(dfg, sym, newPts, false, infdim) : nothing

[dehann]

Tried to solve loaded fg

Would it be possible to share that file perhaps please?

[Affie]

It was the saved fg in this issue: JuliaRobotics/RoME.jl#380
Unfortunately, I don't have an MWE of it.

[dehann]

Oops okay thanks, this might be a bit slow until we see it again.

[dehann]

Issues Leo is seeing (from solving his 00139_fg.tar.gz example) might be related to this:

BoundsError: attempt to access 0-element Array{Float64,1} at index [1]
Stacktrace:
getindexSolve Progress: approx max 3216, at iter 854 	 Time: 0:03:38 at ./array.jl:809 [inlined]
 [2] calcPPE[ Info: CSM-5 Clique 8 finished
([ Info: CSM-5 Clique 11 finished
DFGVariableSolve Progress: approx max 3216, at iter 867 	 Time: 0:03:38{Point2}, ::Point2[ Info: CSM-5 Clique 10 finished
[ Info: CSM-5 Clique 14 finished
[ Info: CSM-5 Clique 20 finished
; method::Type{DistributedFactorGraphs.MeanMaxPPE}, solveKey::Symbol) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/FGOSUtils.jl:204
 [3] calcPPE(::DFGVariable{Point2}; method::Type{DistributedFactorGraphs.MeanMaxPPE}, solveKey::Symbol) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/FGOSUtils.jl:215
 [4] setPPE!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::Symbol, ::Symbol, ::Type{DistributedFactorGraphs.MeanMaxPPE}, ::Nothing) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/FGOSUtils.jl:552
 [5] setPPE! at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/FGOSUtils.jl:550 [inlined]
 [6] updateFromSubgraph_StateMachine(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/CliqueStateMachine.jl:755
 [7] (::StateMachine{CliqStateMachineContainer})(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Nothing; pollinterval::Float64, breakafter::Function, verbose::Bool, verbosefid::Base.TTY, verboseXtra::IncrementalInference.CliqStatus, iterlimit::Int64, injectDelayBefore::Nothing, recordhistory::Bool, housekeeping_cb::IncrementalInference.var"#382#384"{IncrementalInference.TreeClique}) at /home/labuser/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82
 [8] initStartCliqStateMachine!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::IncrementalInference.TreeClique, ::Nothing; oldcliqdata::BayesTreeNodeData, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, show::Bool, incremental::Bool, limititers::Int64, upsolve::Bool, downsolve::Bool, recordhistory::Bool, delay::Bool, logger::Base.CoreLogging.SimpleLogger, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/CliqueStateMachine.jl:63
 [9] tryCliqStateMachineSolve!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Int64, ::Nothing; oldtree::MetaBayesTree, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, limititers::Int64, downsolve::Bool, incremental::Bool, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/SolverAPI.jl:110
 [10] (::IncrementalInference.var"#439#442"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})() at ./task.jl:356┌ Warning: printCliqHistorySummary -- No CSM history found.

ERROR: TaskFailedException:
BoundsError: attempt to access 0-element Array{Float64,1} at index [1]
Stacktrace:
 [1] getindex at ./array.jl:809 [inlined]
 [2] calcPPE(::DFGVariable{Point2}, ::Point2; method::Type{DistributedFactorGraphs.MeanMaxPPE}, solveKey::Symbol) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/FGOSUtils.jl:204
 [3] calcPPE(::DFGVariable{Point2}; method::Type{DistributedFactorGraphs.MeanMaxPPE}, solveKey::Symbol) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/FGOSUtils.jl:215
 [4] setPPE!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::Symbol, ::Symbol, ::Type{DistributedFactorGraphs.MeanMaxPPE}, ::Nothing) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/FGOSUtils.jl:552
 [5] setPPE! at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/FGOSUtils.jl:550 [inlined]
 [6] updateFromSubgraph_StateMachine(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/CliqueStateMachine.jl:755
 [7] (::StateMachine{CliqStateMachineContainer})(::CliqStateMachineContainer{BayesTreeNodeData,LightDFG{SolverParams,DFGVariable,DFGFactor},LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree}, ::Nothing; pollinterval::Float64, breakafter::Function, verbose::Bool, verbosefid::Base.TTY, verboseXtra::IncrementalInference.CliqStatus, iterlimit::Int64, injectDelayBefore::Nothing, recordhistory::Bool, housekeeping_cb::IncrementalInference.var"#382#384"{IncrementalInference.TreeClique}) at /home/labuser/.julia/packages/FunctionalStateMachine/2JZFG/src/StateMachine.jl:82
 [8] initStartCliqStateMachine!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::IncrementalInference.TreeClique, ::Nothing; oldcliqdata::BayesTreeNodeData, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, show::Bool, incremental::Bool, limititers::Int64, upsolve::Bool, downsolve::Bool, recordhistory::Bool, delay::Bool, logger::Base.CoreLogging.SimpleLogger, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/CliqueStateMachine.jl:63
 [9] tryCliqStateMachineSolve!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Int64, ::Nothing; oldtree::MetaBayesTree, verbose::Bool, verbosefid::Base.TTY, drawtree::Bool, limititers::Int64, downsolve::Bool, incremental::Bool, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, solve_progressbar::ProgressMeter.ProgressUnknown, algorithm::Symbol) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/SolverAPI.jl:110
 [10] (::IncrementalInference.var"#439#442"{MetaBayesTree,Bool,Bool,Base.TTY,Bool,Bool,Array{Symbol,1},Array{Symbol,1},Symbol,LightDFG{SolverParams,DFGVariable,DFGFactor},MetaBayesTree,Nothing,ProgressMeter.ProgressUnknown,Int64,Int64})() at ./task.jl:356
Stacktrace:
 [1] sync_end(::Channel{Any}) at ./task.jl:314
 [2] macro expansion at ./task.jl:333 [inlined]
 [3] taskSolveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree, ::Nothing; oldtree::MetaBayesTree, drawtree::Bool, verbose::Bool, verbosefid::Base.TTY, limititers::Int64, limititercliqs::Array{Pair{Symbol,Int64},1}, downsolve::Bool, incremental::Bool, multithread::Bool, skipcliqids::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, delaycliqs::Array{Symbol,1}, smtasks::Array{Task,1}, algorithm::Symbol) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/SolverAPI.jl:49
 [4] solveTree!(::LightDFG{SolverParams,DFGVariable,DFGFactor}, ::MetaBayesTree; timeout::Nothing, storeOld::Bool, verbose::Bool, verbosefid::Base.TTY, delaycliqs::Array{Symbol,1}, recordcliqs::Array{Symbol,1}, limititercliqs::Array{Pair{Symbol,Int64},1}, injectDelayBefore::Nothing, skipcliqids::Array{Symbol,1}, eliminationOrder::Nothing, variableOrder::Nothing, eliminationConstraints::Array{Symbol,1}, variableConstraints::Nothing, smtasks::Array{Task,1}, dotreedraw::Array{Int64,1}, runtaskmonitor::Bool, algorithm::Symbol, multithread::Bool) at /home/labuser/.julia/packages/IncrementalInference/8DImq/src/SolverAPI.jl:371
 [5] top-level scope at REPL[16]:1

[dehann]

Reliably on clique 4, and with .useMsgLikelihoods=true (still the 00139_fg.tar.gz case)

julia> tree[4]
TreeClique (id=4)
  frontals:    [:l2, :x69]
  separator:   [:l8, :l6, :l9, :x64, :l15, :l12, :l14, :l11, :l7, :l5, :x114, :l10, :l1, :l3, :l4]
  status:      NULL
  allmarginld: false
  potentials:  [:x69l6f1, :x69l8f1, :x69l9f1]
 messages

Using eliminationOrder:

00139fg_eliminationorder.zip

[dehann]

The graph does solve fine with getSolverParams(fg).useMsgLikelihoods=false.

This problem links directly back to #1010 which is the current highest priority issue. My recommendation is to set getSolverParams(fg).useMsgLikelihoods=false before solving and return to using =true once 1010 is completed.

• there is a bug here to be fixed before 1010, but don't want Leo to be held up any longer. We will tackle this bug in due course and might take a week or two to get round to it.

for reference, the 00139_fg.tar.gz graph solves here locally in around 3 mins 30 sec

cc @lemauee, I think setting =false should fix the issue you are getting in JuliaRobotics/RoME.jl#418 with slightly degraded performance after a single solve. Since your use case is to solve the graph many times, the useMsgLikelihoods=false option will be diminished by many solves. I will continue testing here and report back if I learn anything new.

For completeness, we expect 1010 to improve numerical quality over useMsgLikelihoods=false usage, and therefore 1010 is the main thing for us to focus on at this time.

[dehann]

Pretty sure I fixed this problem and closed the issue. This looks like a duplicate issue.

[dehann]

It was a bad line of code in one of the function argument defaults, hence the stack trace was hard to read. Basically the code was trying to create some default list, but the source vector had length 0. This produce some error which looked exactly the same as the title of this issue. Also this issue has not shown up since i added a fix.