Handle delete markers for objects in ES

[nl0]

• catalog
  • bucket overview
    • filter-out deleted objects from bucket summary (images and sample)
    • fix object count query
  • search results
    • show 'Delete Marker' labels
• search lambda
  • expose delete_marker by default in search action
  • expose delete_marker in images and sample actions

caveat: images and sample files result sets are filtered from delete markers after querying / limiting, so when delete markers are encountered, the sets will be smaller than the respective limits (NUM_PREVIEW_IMAGES = 100 and NUM_PREVIEW_FILES = 20)

[nl0]

some python tests are broken for some reason (it has something to do with dependencies)

[sir-sigurd]

some python tests are broken for some reason (it has something to do with dependencies)

PyPI is having problems, but even when deps are installed tests fail here.

[fiskus]

Can you provide example of how can I test it manually? Or it's too complicated?

I see "Error resolving revision" on /b/aneesh-ai2-temp/packages/akarve/pkg5 and don't know if it how it supposed to be.

[akarve]

fixed unit tests and did delete_marker filtering in ES instead (or is there some reason why the client needs to do it?). @nl0 please take a look

[nl0]

@fiskus

Can you provide example of how can I test it manually? Or it's too complicated?

hook up to the searchminimal stack and observe:

• no deleted files in overview image grid
• no deleted files in summary (sample)

you can also try uploading and removing files and see that uploading increments total file count and removing (creating a delete marker) does not.

the last thing is the display of delete marker versions in search results: try searching the filename that was deleted and you should see 'Delete Marker' in the version stack.

I see "Error resolving revision" on /b/aneesh-ai2-temp/packages/akarve/pkg5 and don't know if it how it supposed to be.

delete markers for packages arent handled in this PR

[codecov-io]

Codecov Report

Merging #1997 (c1f5052) into master (bbd44d6) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #1997      +/-   ##
==========================================
- Coverage   46.74%   46.71%   -0.04%     
==========================================
  Files         332      332              
  Lines       16439    16450      +11     
  Branches     2191     2195       +4     
==========================================
  Hits         7684     7684              
- Misses       7856     7867      +11     
  Partials      899      899              

Flag	Coverage Δ
api-python	89.35% <ø> (ø)
catalog	7.39% <0.00%> (-0.01%)	⬇️
lambda	92.40% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...alog/app/components/SearchResults/SearchResults.js	0.00% <0.00%> (ø)
catalog/app/containers/Bucket/Overview.js	0.00% <0.00%> (ø)
catalog/app/containers/Bucket/requests.js	0.00% <0.00%> (ø)
catalog/app/utils/search.js	0.00% <ø> (ø)
lambdas/search/index.py	88.50% <ø> (ø)
lambdas/search/tests/test_search.py	100.00% <ø> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bbd44d6...c1f5052. Read the comment docs.

[fiskus]

I tried added and deleted files in b/searchminimal on master. Don't see deleted files already.
On this branch I can see deleted files in overview for /b/aneesh-ai2-temp

So, I failed to reproduce

[nl0]

I tried added and deleted files in b/searchminimal on master. Don't see deleted files already.
On this branch I can see deleted files in overview for /b/aneesh-ai2-temp

So, I failed to reproduce

you should use the searchminimal stack (that is, api gateway and registry from that stack, bucket doesn't really matter). also, it's probably broken right now if Aneesh's lambda changes are deployed there, so i think we should deploy both latest catalog and lambda changes and test after that.

[fiskus]

I understand, I used searchminimal stack and searchminimal bucket