Skip to content

Commit 2ce07c0

Browse files
authored
Merge pull request #1130 from mraspaud/feature-datatype-note
Add note about datatype in custom reader documentation
2 parents c35e8c0 + ebbf2f1 commit 2ce07c0

1 file changed

Lines changed: 19 additions & 0 deletions

File tree

doc/source/dev_guide/custom_reader.rst

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -506,6 +506,25 @@ a convenience and are not required to read these formats. In many cases using
506506
the :func:`xarray.open_dataset` function in a custom file handler is a much
507507
better idea.
508508

509+
.. note::
510+
Be careful about the data types of the datasets your reader is returning.
511+
It is easy to let the data be coerced into double precision floats (`np.float64`). At the
512+
moment, satellite instruments are rarely measuring in a resolution greater
513+
than what can be encoded in 16 bits. As such, to preserve processing power,
514+
please consider carefully what data type you should scale or calibrate your
515+
data to.
516+
517+
Single precision floats (`np.float32`) is a good compromise, as it has 23
518+
significant bits (mantissa) and can thus represent 16 bit integers exactly,
519+
as well as keeping the memory footprint half of a double precision float.
520+
521+
One commonly used method in readers is :meth:`xarray.DataArray.where` (to
522+
mask invalid data) which can be coercing the data to `np.float64`. To ensure
523+
for example that integer data is coerced to `np.float32` when
524+
:meth:`xarray.DataArray.where` is used, you can do::
525+
526+
my_float_dataarray = my_int_dataarray.where(some_condition, np.float32(np.nan))
527+
509528
One way of implementing a file handler is shown below:
510529

511530
.. code:: python

0 commit comments

Comments
 (0)