Content may never get advertised on failure to persist

[masih]

If the error happens because the advertisement failed to persist then that content will never get advertised.
Looking at the code, such errors can happen before call to NotifyPut e.g. in metadata encoding on markets side or inside the provider before persistence.

Note, this failure scenario is different from a case where the index provider fails to announce the new advertisement. Since if an indexing advertisement is added to the chain of advertisements as long as one announcement is successful the un-announced advertisements are indexed.

Further, the announce-all functionality will not be helpful here since it is intended to be a one-off operation run after migration and not something a SP is expected to execute regularly.

Two things to consider:

• expose error types on index-provider so that on the markets side we can differentiate critical persistence errors from announcement errors. If latter, the code execution can safely log and ignore the error.
• document what should happen if advertisement creation fails but deal succeeds. Right now this will result in content never getting advertised.

Originally posted by @masih in #673 (comment)