Clarify the lifecycle of indexes with the lifecycle of a deal

[jacobheun]

The purpose of this issue is to clarify and discuss the various potential state changes for indexing over the lifecycle of a deal that has been marked for indexing. This is not intended to cover the existing lifecycle of indexing, but the desired lifecycle. Once solidified, we can create accompanying issues to resolve discrepancies in the current implementation to match the desired state.

Legend

• 🚧 - Needs discussion & decision
• 🍏 - Alignment reached

The Lifecycle of Indexing

🚧 Indexing a new Deal

Note: Not covering how deals are identified for indexing in this section as there is a separate effort to solidify requirements around that. See #689 and filecoin-project/notary-governance#666 for more details.

For discussion purposes, let’s assume that once the above issues are complete, there will be some way to identify if a specific deal should be indexed or not, and that there will be a mechanism to account for this for existing deals.

When a new deal has been successfully published, if an unsealed copy exists and the deal is marked for indexing, it should be immediately registered with the index provider/marked for indexing.

🚧 Deletion of an unsealed copy

When an unsealed copy is deleted today, indexes are not removed. There is currently support in the Network Indexers to include metadata on whether or not the data is unsealed, but it’s not being leveraged correctly today (all announced indexes are being marked as unsealed).

We need a mechanism to detect the removal of unsealed copies (as they can be rm’d manually). The section on Repairing Indexes below, speaks to how this might be accomplished. Upon detection of deletion we can perform one of the following actions (need to decide between the options):

Option 1 - Remove the indexes(recommended): When an unsealed copy is removed, as the unsealing process is a non trivial operation, we should assume the copy will not become available in a short time frame. As such, the local indexes should be removed and we should announce the deletions to the network indexers. This frees up space both locally and on indexers. If unsealed copies were expected to be created/deleted often, then this option might be less reasonable, but this is not the case today.

Option 2 - Update Index Metadata: When an unsealed copy is removed we can update the metadata of the indexes for that deal to specify that no unsealed copy exists. This would still allow discovery of the SP who has the content, but retrieval would not function without an unseal. The advantage of this option is that a client could pay the Unseal price to get the data, knowing who has it. However, it’s worth noting that retrieval flows requiring unsealing are not particularly clear and would need further work to likely become viable.

If this option is chosen, we may want to change the indexing logic of sector expiration and will definitely need to change how removed sectors are handled.

🚧 A sector is unsealed

When we detect a sector has been unsealed, and that sector is eligible for indexing, it should be registered with the index provider for reindexing (assuming unsealed deletion - option1 is selected).

🚧 Expiration of a sector

As long as the unsealed copy of the sector exists, the indexes should also exist. No changes should occur until the unsealed copy is removed.

🚧 Removal of a sector

Same as sector expiration, this should be a no-op, as index changes would be triggered from changes to the unsealed copy only.

If unsealed deletion option 2 is selected, removal of a sector when there is no unsealed copy, will require deletion of indexes and announcement to the network indexers.

🚧 Repairing Indexes

One of the issues facing retrieval reliability is index metadata getting out of sync with unsealed copies (or the lack thereof). There are several reasons this may be occurring, but it often requires manual intervention by SP’s to repair and the visibility into when this needs to happen is not clear. A proposal that has been discussed recently is to have an automatic repair job for indexing, to automatically ensure unsealed copies, eligible for indexing receive an integrity check and are repaired if there is an issue. This would NOT include automatic unsealing of data as this is a resource intensive process.

An extension of this proposal, given that unsealed copies may be deleted or created manually by SP’s, is to have new index creation, and repair all belong to this “repair” service. This service could be a background process that is continually repairing/registering/removing indexes with limited resource consumption. This could remove some operational overhead for common errors reported with retrievals. Specifics of how this could/should work can be flushed out in a followup issue.

Related Issues & Discussions

• IndexerAnnounceAllDeals logic and intent is not clear #882
• Scope default indexing to only FIL+ deals with FastRetrieval enabled notary-governance#666
• Capability to skip deals for indexing: need config to mark content as not retrievable in market #689
• Remove deals from the index if they fail to seal on time or are slashed #940
• legacy format deals without unsealed copy fails during recovery process #690
• Status of store deal gets stuck in Announcing for a long time #920

[LaurenSpiegel]

For option 1, why is the local index removed? Is there a local index for the sealed copy that is separate from the unsealed copy and this operation would just remove the local index for the unsealed copy?

[davidd8]

One general observation is that the data lifecycle is bifurcating into sealed and unsealed data, which adds complexity. The more we can maintain the same state across sealed and unsealed data, the simpler it will be to reason about these systems. In that vein, I lean towards option (1) with the assumption that SPs only remove unsealed copies when also removing sealed copies.

This may imply expiration and removal of sectors should also require the removal of unsealed copies, and the time before the unsealed copy gets deleted is insignificant in the grand scheme of things. Why would an SP hold on to unsealed copies of data that they are no longer storing on-chain?

I'm not clear how the repair functionality works - is it to re-index unsealed copies, or find unsealed copies that are indexed? If this is possible, another idea is to trigger a repair from a failed retrieval attempt. But again, I'm not sure how the SP would know whether or not they have the unsealed copy of the requested CID, if they can't find it in their local index.

[dirkmc]

For option 1, why is the local index removed?

When a client queries the SP to ask for the data, the SP

1. Gets the index for the data
2. Gets a reader over the data
3. Serves the data using the index + reader

If the unsealed copy has been removed, it's not possible to serve the deal's data. So we should return a "not found" response to queries at step 1.

[dirkmc]

This may imply expiration and removal of sectors should also require the removal of unsealed copies

I think the answer depends on our expectations of unsealing. The system was designed with the expectation that unsealing would soon become cheap and fast, however that hasn't happened in practice. Today it takes multiple hours to unseal data. If we assume that's not going to change soon, then in practical terms retrievability is unrelated to sealed data. So for the purposes of retrieval, what matters is the unsealed copy, and keeping the index in sync with the unsealed copy.

Why would an SP hold on to unsealed copies of data that they are no longer storing on-chain?

They may have separate contractual obligations for retrieval vs storage, and they may earn money from retrieval (eg serving popular data). I agree that this is more complicated conceptually so we need to decide if it's worth the additional complexity to allow those use cases.

I'm not clear how the repair functionality works - is it to re-index unsealed copies, or find unsealed copies that are indexed?

It is to re-index unsealed copies. There may not be an index for a unsealed copy of data if

• the SP never created an index (when the dagstore was introduced, SPs had to run a manual process to add indexes)
• something went wrong with indexing
• the unsealed copy was deleted, and later restored from a sealed copy

Note also that indexes may become corrupted (we've seen this happen in practice with dagstore indexes) so we want a way to detect and repair them.

[jacobheun]

So for the purposes of retrieval, what matters is the unsealed copy, and keeping the index in sync with the unsealed copy.

Exactly this. Option 2 is the thing that muddies the water here, and why I advocate for just sticking with option 1 until we have a very clear story with good UX around handling retrieval for sealed data. Options 1 makes the logic pretty straightforward:

"For all unsealed data keep the local indexes up to date, and for all unsealed copies marked for index announcement, keep network indexers up to date.".

We don't need to care about sealed data at all for indexing purposes. It doesn't matter if the sealed sector is expired/removed/etc, we only ever care about the unsealed data, and once it's gone we cleanup the PieceDirectory(soon replacing the dagstore).

Is there a local index for the sealed copy that is separate from the unsealed copy and this operation would just remove the local index for the unsealed copy?

To help clarify this:

• No indexes ever exist for sealed data.
  • If we've generated indexes from the unsealed data we can associate the sealed sector for any indexed cid by the associated Piece cid, but as long as there is no unsealed copy retrieval will fail until it's unsealed.

Note also that indexes may become corrupted (we've seen this happen in practice with dagstore indexes) so we want a way to detect and repair them.

+1. We need to assume corruption is going to happen, we're dealing with a lot of data. Automating a "scan and repair" approach could help us mitigate failures and reduce some operational overhead for SPs.

[LaurenSpiegel]

If the sealed data is not tracked in the Network Indexer and not in the local index:

1. How does the client get the CID if they know which SP holds it (what does the flow look like)?
2. How do they get the CID if they do not know which SP holds it (what does the flow look like)?

[jacobheun]

1. How does the client get the CID if they know which SP holds it (what does the flow look like)?
2. How do they get the CID if they do not know which SP holds it (what does the flow look like)?

I'm going to assume that by CID you mean the actual content of the cid for these (correct me if that's wrong).

For both of these, I'll start by addressing the sealed data problem. If there is sealed data that a client wants back there are 2 options for getting it.

• Contacting the SP and telling them to unseal it. - This could be a reasonable option for scenarios like disaster recovery, or archival, where the intent is that hopefully you won't actually need to access the data, but you can contact the SP to regain access to it. This could keep storage costs low, while users might pay a premium for the unsealing and/or retrieval. AFAIK, this is likely the go to option SP's are doing, but I could be wrong.
• Retrieval triggers an unseal - This logic exists today where retrieval attempts will automatically trigger an unseal, but the UX for either party is not all that clear. As unsealing is resource intensive, SP's often set very high unseal prices to prevent it from being triggered. We could and should eventually improve this flow, but my sense is it's a low priority as the expectation is that data that should be readily accessible be unsealed. The idea is that as a client I can pay the SP to unseal the data, and come back later to retrieve the data once unsealing has been happened. If we want to dig into this option, we should create another issue/discussion as there's a lot to unpack here.

1. How does the client get the CID if they know which SP holds it (what does the flow look like)?

• The unseal is triggered by one of the above mentioned mechanisms. (taking ~3h+)
• The unsealed copy is indexed (At this point the client could retrieve directly from the SP)
  • This could be regenerating indexes (option 1) or "repairing" them (option 2)
• Indexing updates are announced to the network indexers
  • At this point any client could query the indexers for the data

2. How do they get the CID if they do not know which SP holds it (what does the flow look like)?

The client should ideally know the PieceCID containing their content. This can be looked up on chain to find the SP, and then the flow from the previous scenario can occur. If for some reason the client does not know the PieceCID there content is in, there's no reasonable way to find the data.

The other option, as described in option 2 in the issue description would be:

• Indexes are generated for the original unsealed copy
• The unsealed copy is deleted
• The indexers are notified that the indexes for this deal are now sealed
• The client queries the indexers and discovers that SP Alice has the content, and that the indexes are sealed
• The client triggers unsealing via one of the above mentioned options
• The unsealed copy is recreated
• Local indexes are automatically "repaired" as needed.
  • The client already knows the SP, so they could retrieve at this point
• Indexers are notified that the indexes for the deal are now unsealed (along with any potential index updates)
  • The client can query the indexers to find the unsealed index location if they havent already retrieved

[LaurenSpiegel]

If for some reason the client does not know the PieceCID there content is in, there's no reasonable way to find the data.

This strikes me as a problem we should be thinking through. For example, if I know a content CID is stored with an SP for archival or backup purposes, I might not know the Piece CID but would still want to find it, have it unsealed and retrieve it.

Rather than deleting references to sealed data, shouldn't we consider keeping CID to Piece mappings and noting whether there is a sealed copy and unsealed copy?

[jacobheun]

Rather than deleting references to sealed data, shouldn't we consider keeping CID to Piece mappings and noting whether there is a sealed copy and unsealed copy?

This is option 2 mentioned in the description. It has better UX properties for clients, but definitely adds some complexities. Let's walk through the lifecycle of what that would look like in more depth.

Quickly though to clarify one thing (which we should ensure #689 covers):

• Not all clients will need their data indexed (announced or not) and SP's may wish to charge different rates for indexed or non indexed data. This may be especially true in the future when making smaller deals becomes more feasible.
  • As such, a client should have the ability to specify whether or not their data should be indexed locally when making a deal. This is separate from whether or not is should be announced.

The Lifecycle of Indexing w/ retained indexes for sealed data

🚧 Indexing a new Deal

This is mostly the same as option 1 in the description but there are some additional things we need to think about.

• If a deal is marked for indexing, we need to ensure indexes are generated before unsealed copies are/can be deleted. We'll need to look at the storage pipeline to get better clarity around this - If we generate the indexes too early we may need to delete them if the deal fails later, if we attempt to generate them too late the unsealed copy may get deleted.
  • As soon as the unsealed copy is gone, we lose our ability to index until we unseal.
  • It might be worth establishing a clear signal in the deal whether it is "cold storage" or not. If it is cold storage and the client wants it indexed, we could automatically handle clean up of the unsealed sector. I'm not sure if the "fast retrieval" flag would be sufficient for this.

🚧 Deletion of an unsealed copy

• Check if a sealed copy of the data exists:
  • If there is no sealed copy, delete indexes and announce the deletion.
  • If there is still a sealed copy, update indexers that the indexes are sealed.

🚧 A sector is unsealed

• When an unseal happens we should register for a "repair" of the indexes as an integrity check.
• Update indexers that the indexes are unsealed.

🚧 Expiration of a sector

• No action is needed. We shouldn't need to care until a sector is actually removed.

🚧 Removal of a sector

If the sector is removed but an unsealed copy is still available we need to determine the expected behavior here. As @dirkmc mentioned above, storage and retrieval could be negotiated separately, in which case we should keep indexes around as long as there is still an unsealed copy. So the logic would be:

• Check if an unsealed copy still exists:
  • If there is an unsealed copy, do nothing.
  • If there is not an unsealed copy, delete indexes and announce to the network.

🚧 Repairing Indexes

Same as option 1 above, however, integrity of indexes for sealed only copies would not be able to be checked without unsealing the data. Assuming appropriate replication/backup of the PieceDirectory once it's released, this should be a minimal risk.

[rvagg]

fwiw I'm fine with either option from a retrieval perspective as long as I can count on either either of these things being true (which map to the 2 options): (1) having some basic assurance (not rock solid of course) that the SPs the indexer tells me have a CID have an unsealed copy, or (2) that I can count on that FastRetrieval flag to mean something (it's also graphsync specific metadata btw, maybe it shouldn't be?) because then I can trivially filter it out. I'm currently in that code for lassie & autoretrieve right now and I'd love to be able to plumb it through as a filter.

I suspect though, if we go with option 1, that it's going to be harder to put sealed data back in to the indexers at a future date once we have sealed->unsealed request/retrieval UX sorted out because by that time the assumption will be baked in to the indexer data that it's sealed so we have a compounded UX problem of ensuring that requests to the indexer API or responses from it account for sealed status. However, if we go with option 2 today, then FastRetrieval means something and we can bake that into our retrieval code ignore sealed data entirely, the indexers have a ton more data from the network (and we get juicy metrics).

Maybe we should have a cursory discussion about unsealed request/retrieval UX to cover that base? Maybe it ends up being a simple matter that we can put on a roadmap and know it's not going to be a massive job. (e.g. a new endpoint that take a retrieval-like proposal that I can ping with my unsealing request and it returns a status code and a % of unsealed progress [so I can ping it repeatedly to see how it's going]).