Should introduce deleting state for ledger deletion

[hangc0276]

Is your enhancement request related to a problem? Please describe.
In current ledger deletion, we divided it into two separate steps.

1. Remove all the waiting to delete ledgers from ledger list and update the newest ledger list into meta store.
2. In the meta store update callback operation, delete the waiting to delete ledgers from storage system, such as BookKeeper or Tiered storage.

Due to the separate step, we can't ensure the ledger deletion transaction. If the first step succeed and the second step failed, it will lead to ledgers can't be deleted from storage system forever. The second step maybe fail by broker restart or storage system deletion failed.

pulsar/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java

Lines 2532 to 2561 in bd68b6f

	store.asyncUpdateLedgerIds(name, getManagedLedgerInfo(), ledgersStat, new MetaStoreCallback<Void>() {
	@Override
	public void operationComplete(Void result, Stat stat) {
	log.info("[{}] End TrimConsumedLedgers. ledgers={} totalSize={}", name, ledgers.size(),
	TOTAL_SIZE_UPDATER.get(ManagedLedgerImpl.this));
	ledgersStat = stat;
	metadataMutex.unlock();
	trimmerMutex.unlock();
	for (LedgerInfo ls : ledgersToDelete) {
	log.info("[{}] Removing ledger {} - size: {}", name, ls.getLedgerId(), ls.getSize());
	asyncDeleteLedger(ls.getLedgerId(), ls);
	}
	for (LedgerInfo ls : offloadedLedgersToDelete) {
	log.info("[{}] Deleting offloaded ledger {} from bookkeeper - size: {}", name, ls.getLedgerId(),
	ls.getSize());
	asyncDeleteLedgerFromBookKeeper(ls.getLedgerId());
	}
	promise.complete(null);
	}
	@Override
	public void operationFailed(MetaStoreException e) {
	log.warn("[{}] Failed to update the list of ledgers after trimming", name, e);
	metadataMutex.unlock();
	trimmerMutex.unlock();
	promise.completeExceptionally(e);
	}
	});

In our customer's environment, we have found many orphan ledgers cause by the above reason.

Describe the solution you'd like
We'd better introduce a deleting state for the waiting to delete ledgers. The desired ledger deletion should follow the three steps.

1. Remove all the waiting to delete ledgers from ledger list and update the newest ledger list into meta store.
2. Write the waiting to delete ledgers into meta store marked as deleting state.
3. Use another thread to periodically check the deleting state ledgers, and do ledger delete operation on storage system. Each topic's owner control their own topics' ledger deletion.
4. After the ledger deletion complete, remove the ledger from meta store.

[wuzhanpeng]

I can help with this issue~

[hangc0276]

@wuzhanpeng Thanks for your help, I have assigned this issue to you. If you have any questions or need any help, please ping me at any time. You'd better write a proposal first.

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[tisonkun]

Superseded by #16569