-
Notifications
You must be signed in to change notification settings - Fork 11
Open
Description
Computing the mean and std of large frames (e.g. ~16 million points) can suffer from floating point error when they are performed using float32 values. Both mean and std can be passed dtype=np.float64 to perform these computations using float64 instead and thereby retain better accuracy. One place where this could be added is here:
Line 398 in 5ca049f
| self.noise_mean, self.noise_std = np.mean(clipped_data), np.std(clipped_data) |
but perhaps it would be good to add a dtype=np.float64 parameter to the enclosing function so that the user could specify a different type if desired.
Metadata
Metadata
Assignees
Labels
No labels