Atomic writes in Zarr v3?

[shoyer]

Zarr v2 took care in DirectoryStore to use atomic writes by renaming files into place, which ensures that writes to disk are either complete or not written at all:

zarr-python/zarr/storage.py

Lines 1146 to 1161 in f1978dd

	# write to temporary file
	# note we're not using tempfile.NamedTemporaryFile to avoid restrictive file permissions
	temp_name = file_name + "." + uuid.uuid4().hex + ".partial"
	temp_path = os.path.join(dir_path, temp_name)
	try:
	self._tofile(value, temp_path)
	# move temporary file into place;
	# make several attempts at writing the temporary file to get past
	# potential antivirus file locking issues
	retry_call(os.replace, (temp_path, file_path), exceptions=(PermissionError,))
	finally:
	# clean up if temp file still exists for whatever reason
	if os.path.exists(temp_path): # pragma: no cover
	os.remove(temp_path)

I don't see any similar logic in Zarr v3, which just seems to using Python's file interface (via open):

zarr-python/src/zarr/storage/_local.py

Lines 52 to 55 in c9509ee

	with path.open("r+b") as f:
	f.seek(start)
	# write takes any object supporting the buffer protocol
	f.write(value.as_buffer_like())

Was this an intentional omission? I am concerned about files potentially ending up partially written due to interrupted writes.

[d-v-b]

good question, I don't know the answer but maybe @jhamman does

[rabernat]

I'm almost certain it was not intentional. The storage backends were rewritten from scratch, ported over from zarrita. I wasn't even aware of that detail of the v2 implementation.

[shoyer]

OK, great. I am send a PR.

[shoyer]

In practice, I think this is not an issue for many underlying object stores, because they guarantee atomic writes already. At least this is the case for Google Cloud Storage. But I suspect this may be an issue with our network storage systems, so it's probably worth fixing anyways -- and also just so new Zarr store writers know what they need to think about.

[d-v-b]

im excited to see how you test this!