Internal storage of binary objects in CloudKit

[gerfarfel]

Is it the case that all BLOB records are stored as CKAsset objects in the CloudKit database? CloudKit supports storing binary data internally as BYTES or externally as ASSET, and the docs for CKAsset specifically mention using it "when a field’s value is more than a few kilobytes in size". It seems to me that not supporting internal storage of small binary objects in the CloudKit database disincentivizes their use altogether.

[mbrandonw]

Hi @gerfarfel,

Is it the case that all BLOB records are stored as CKAsset objects in the CloudKit database?

Yeah, that is correct.

We can certainly make it so that we don't package blobs up into CKAssets if it's less than a few kilobytes, but just not sure what the problem is of always using a CKAsset. Do you know of any?

[mbrandonw]

Hi @gerfarfel, we would be open to taking this kind of change. Are you interested in contributing a PR for it? We can answer any questions you might have.

[gerfarfel]

I'm happy to hear you're open tho the change. It's not for lack of interest, but I'm not going to be able to put in the required time for the next couple of months. I'll try to circle back at some point if no one else takes it on. Thanks again.

[mbrandonw]

Hi @gerfarfel, I looked into this a bit this morning and I will say it's a bit more delicate than it seems at first. The main problem is that by default we store all fields in encryptedValues, but an exception is made for CKAssets because they can only be stored in non-encrypted values. Pair that with the fact that setting a value for a key in both encrypted and non-encrypted values leads to an immediate crash in CKRecord means that we run the risk of crashing everyone's app if not careful.

We have to be able to handle the situation that one app sets a binary blob that is under the threshold, and so stored in encrypted values, and then another devices comes along and sets a blob greater than the threshold, which puts a CKAsset in the non-encrypted values. And things even get more complicated when thinking about how some apps out there could be running an old version of SQLiteData without this behavior.

All to say that we do want to support this, but it isn't going to be the quick win I thought it would be.

[johankool]

@mbrandonw Looks to me that you are trying to make it too fancy by letting it depend on the blob size whether to store it as CKAsset vs. as data. If it is just a part of the definition, something like an @Column(assetBacked: false|true) annotation I think it would cover many use cases.

[mbrandonw]

If it is just a part of the definition, something like an @Column(assetBacked: false|true) annotation I think it would cover many use cases.

This isn't really possible because @Column is a StructuredQueries thing that is only concerned with SQL. It doesn't know anything about SQLiteData or iCloud. It would need a different approach.

Also posted this in Slack and repeating here for posterity:

one workaround, albeit not ideal, would be to make a custom representation for Data so that it is stored as a base64 encoded string:

@Table struct Model {
  @Column(as: Data.Base64Encoded.self)
  var data: Data
}

That would allow you to use it as Data in swift, but in SQL it would be turned into a base64 encoded string so that it would sidestep any of the CKAsset stuff.