msnoise cc stack -m test fails

[asyates]

Testing the new stack command, i am getting errors due to the data not being found for the moving stack. Can see this also when running msnoise utils test:

Lets STACK !
2024-01-24 12:13:07.485426 msnoise [pid 154472][INFO]: There are STACKS jobs for some days to recompute for YA.UV05.00:YA.UV10.00
2024-01-24 12:13:07.486294 msnoise [pid 154472][INFO]: Processing YA.UV05.00:YA.UV10.00-ZZ-1
2024-01-24 12:13:07.622975 msnoise [pid 154472][ERROR]: not enough data for mov_stack=5
2024-01-24 12:13:07.623753 msnoise [pid 154472][INFO]: Processing YA.UV05.00:YA.UV10.00-ZZ-2
2024-01-24 12:13:07.662694 msnoise [pid 154472][ERROR]: not enough data for mov_stack=5
2024-01-24 12:13:07.677489 msnoise [pid 154472][INFO]: There are STACKS jobs for some days to recompute for YA.UV05.00:YA.UV06.00
2024-01-24 12:13:07.678284 msnoise [pid 154472][INFO]: Processing YA.UV05.00:YA.UV06.00-ZZ-1
2024-01-24 12:13:07.716789 msnoise [pid 154472][ERROR]: not enough data for mov_stack=5
2024-01-24 12:13:07.717501 msnoise [pid 154472][INFO]: Processing YA.UV05.00:YA.UV06.00-ZZ-2
2024-01-24 12:13:07.755518 msnoise [pid 154472][ERROR]: not enough data for mov_stack=5
2024-01-24 12:13:07.768700 msnoise [pid 154472][INFO]: There are STACKS jobs for some days to recompute for YA.UV06.00:YA.UV10.00
2024-01-24 12:13:07.769459 msnoise [pid 154472][INFO]: Processing YA.UV06.00:YA.UV10.00-ZZ-1
2024-01-24 12:13:07.807445 msnoise [pid 154472][ERROR]: not enough data for mov_stack=5
2024-01-24 12:13:07.808167 msnoise [pid 154472][INFO]: Processing YA.UV06.00:YA.UV10.00-ZZ-2
2024-01-24 12:13:07.846215 msnoise [pid 154472][ERROR]: not enough data for mov_stack=5

I haven't looked into the specifics of the .nc files at this stage, but can see that in STACKS2 directory being created (running a different example) that i have only one 001 day file per station-pair and channel. If this is similar to hdf5 files, i guess this is expected.

msnoise cc stack_old runs fine, but i guess then mwcs and stretching (new and old) both based on STACKS2 directory (at least seems that way based on --help documentation and compute_dtt_old not outputting anything).

[asyates]

Okay so turns out it's just a change in behaviour when number of 1-days CCFs is less than mov_stack size.

i.e. for the tests, which use only 1-day of data, having mov_stack = 5 would not return an error in old codes as it would output the 1-day CCF in the place of the 5-day stack based on setting low and high indexing.

old code:

                                for mov_stack in mov_stacks:
                                   if i < mov_stack:
                                       low = 0
                                       high = mov_stack
                                   else:
                                       low = i - mov_stack + 1
                                       high = i + 1
                                   newdata = False
                                   for uday in datelist[low:high]:
                                       if uday in updated_days:
                                           newdata = True
                                           break
                                   if newdata:
                                       corr = stack_total[low:high]

whereas in the new code, it returns an error if mov_stack is larger than number of ccfs, which is why test failing i.e.

new code:

                for mov_stack in mov_stacks:
                    #print(mov_stack, mov_stacks, len(dr.times))
                    if mov_stack > len(dr.times):
                        logger.error("not enough data for mov_stack=%i" % mov_stack)
                        continue

So not an issue really, but still tests should be updated so will leave open (and have updated title). Perhaps to include 2 days of data and set mov_stack = 2 for example... or even just mov_stack = 1?

[ThomasLecocq]

I'm thinking of adding a larger test dataset, but that should be automagically downloaded with https://github.com/fatiando/pooch , making the default code base super light (and not 30 MB as now). Include more than 5 days of data, and also horizontal components, what do you think?

[ThomasLecocq]

pooch is a fact now :)