Update the item only if it exists in the cache

[pingsutw]

TL;DR

I found that Propeller keeps sending the request to the agent even workflow is done.

The work queue only has unique items, so we add itemID to it. The workers won't process the item again after the task is done.

Before:

	enqueueBatches	workqueue	sync (10 workers)
t1	obj(1)	obj(1)
t2
t3	obj(2)	obj(1), obj(2)
t4
t5	obj(1)	obj(1), obj(2), obj(1)
t6			obj(1), obj(2), obj(1)

After:

	enqueueBatches	workqueue	sync (10 workers)
t1	1	1
t2
t3	2	1, 2
t4
t5	1	1, 2
t6			1, 2

100 workflows, each has 100 tasks.
Before:

After:

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

^^^

Tracking Issue

NA

Follow-up issue

NA

[codecov]

Codecov Report

Attention: 10 lines in your changes are missing coverage. Please review.

Comparison is base (b35cc95) 58.98% compared to head (aedaf76) 59.05%.
Report is 31 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4117      +/-   ##
==========================================
+ Coverage   58.98%   59.05%   +0.07%     
==========================================
  Files         618      621       +3     
  Lines       52708    53091     +383     
==========================================
+ Hits        31088    31353     +265     
- Misses      19140    19239      +99     
- Partials     2480     2499      +19     

Flag	Coverage Δ
unittests	59.05% <72.97%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
flytestdlib/cache/auto_refresh.go	78.28% <72.97%> (+2.62%)	⬆️

... and 37 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pingsutw]

Updates:

• Still add the item to the workqueue
• Worker only updates the item if it exists in the cache
• To prevent items from being added back to the cache after deletion, update the item with a lock.

If the Item is added back to the cache after deletion, worker will run forever.

1. Add item to the workqueue (there is no status in the item since we didn't call plugin.status in the sync function)
2. Update LRU cache
3. Repeat 1

[EngHabu]

We shouldn't stop the timer here, should we? just log and continue? the stop should happen after sync is called

[pingsutw]

updated it

[EngHabu]

We should add a check for if the len(newBatch) == 0, we don't need to sync at that point...

[pingsutw]

updated it

[EngHabu]

I wanted to avoid locking the entire dataset for these operations... any reason to?

[EngHabu]

Why only update if it's already there, what if it has been evicted since?

[pingsutw]

If we don't check if it exist in the cache, we might have this scenario.

worker1 removes item from the caches -> worker2 adds the item to cache again. (workerqueue can have duplicate item)

If we don't use lock, we might have this scenario.

1. w.toDelete.Remove (worker 1)
2. w.lruMap.Contains(id) (woker 2)
3. w.lruMap.Remove(key) (woker 1)
4. w.lruMap.Add(id, item) (woker 2) # we add item back to the cache again.