Eviction issues

[simaocat]

Thank you very much for your reply about performance issue. However, in the current testing, I have encountered some issues regarding eviction, and I would like to discuss them.

1. The value database.hard.max_keys serves as the maximum number of buckets for the hash table (the maximum number is statically allocated in advance, and there is currently no resize hash functionality). When all buckets are allocated, it becomes impossible to find an available bucket, resulting in insertion failure, which effectively limits the maximum number of keys. However, the configuration memory.limits.hard.max_memory_usage does not seem to truly limit the maximum available memory. When the memory usage exceeds memory.limits.soft.max_memory_usage, eviction begins to work. If the insertion rate is faster than the eviction rate at this time, the memory usage will still gradually increase and exceed the limit set by memory.limits.hard.max_memory_usage, leading to actual memory usage surpassing this configured value. From the actual operation of my code, the memory continues to increase and exceeds the value of memory.limits.hard.max_memory_usage. However, in practical production environments, memory.limits.hard.max_memory_usage is more useful than database.hard.max_keys because the size of the values inserted by users varies greatly. Users often do not know what value to configure for database.hard.max_keys, but they know how much maximum available memory memory.limits.hard.max_memory_usage should be set to, for example, 80% of the system memory. Therefore, it is hoped that memory.limits.hard.max_memory_usage can effectively limit the maximum available memory for actual data size. When the actual memory usage exceeds this value, insertion should report an error, and wait for eviction to continuously reduce memory usage below memory.limits.hard.max_memory_usage before allowing further data insertion.

2. The value of the metric cachegrand_db_keys_count displayed in Prometheus requests shows a significant discrepancy compared to the number of keys observed using the keys * command in the Redis client. What could be causing this issue?

3. In the main branch, when running the stress testing program, the cachegrand log starts to report errors after a while when evictions occur, and this error happens every time. It is hoped that this error will not be carried over into version 0.5.
   [2025-01-14T11:04:42Z][ERROR ][worker][id: 12][cpu: 12][transaction_rwspinlock] Possible transactional spinlock stuck detected for thread 540153 in src/data_structures/hashtable/mcmp/hashtable_op_get.c:200 [2025-01-14T11:04:43Z][ERROR ][worker][id: 11][cpu: 11][transaction_rwspinlock] Possible transactional spinlock stuck detected for thread 540152 in src/data_structures/hashtable/mcmp/hashtable_op_delete.c:278

[danielealbano]

1. Already opened a bug for it max_memory_usage must take into account the keys as well, not just for the value #439 but will check it more in detail once I will stabilise the main branch.

2. Very interesting, the value is read by summing the statistics from each worker thread so if it "shows a significant discrepancy" there might be a bug in how the statistics are tracked, I will check it out

3. As mentioned in another reply, the main branch is currently under heavy changes, more specifically how the hashtable is managed which also requires to update a number of commands (e.g. the keys expiration is currently broken in main).

[simaocat]

May I ask, according to what was mentioned earlier, will the stable version 0.5 be released as scheduled by the end of this month or the beginning of next month?

[danielealbano]

That's the goal although it might already slide by one week, however might slide further as I work on cachegrand in during my (very little) free time and it's not my primary activity :)

[simaocat]

After testing, a potential bug was discovered that is present in both version 0.4 and the main branch, and your confirmation is needed:

cachegrand/src/storage/db/storage_db.c

Lines 1982 to 1995 in eb2f796

	if (storage_db_op_delete_by_index_all_databases(
	db,
	&transaction_delete,
	true,
	key_eviction_list_entry->value,
	&entry_index)) {
	keys_evicted_count++;
	counters->keys_count--;
	counters->data_size -= (int64_t)entry_index->value.size;
	} else {
	// Should never fail, but just in case increment the amount of keys to evict by one and continue
	evict_first_n_keys++;
	}

In the storage_db_keys_eviction_run_worker function in the file storage_db.c, after the storage_db_op_delete_by_index_all_databases function is called successfully, the data_size of the counter is decremented by the size of the value. However, at this point, the size of the value has already become 0. Therefore, the actual size is not subtracted, which leads to the outer storage_db_keys_eviction_calculate_close_to_hard_limit_percentage function using the old data_size for judgment. As a result, evictions will still be executed repeatedly, wasting a lot CPU time.

Please review it if it's really a bug. Thanks!

[danielealbano]

Yes, it seems that by mistake the keys count and data size is decreased twice

[simaocat]

Hi, how is your v0.5 coming along? Cachegrand is a great project that has been very helpful to me, so I'd like to make a small donation. And I'm really looking forward to its release. Would you be willing to share the current v0.5 source code tar file with me in advance? This way, I can conduct early testing, identify any issues, and apply it to my actual project. If that's possible, my email address is: [email protected]. Looking forward to your reply.