GeneratorRecording outline

[h-mayorquin]

Hi, so I am writing here the outline of the features that the GeneratorRecording should have as discussed on today's meeting. The purpose of this issue is to discuss, describe and agree what we want ant to outline the features that we will be adding to it.

The general idea, as described in #1391, is to have a generator whose memory footpring is fully controlled. This is very useful for testing purposes and specifically decoupling reading from writing bugs on high-memory loads. Plus, if we add more functionality, as discussed below, maybe this function can be used in more general scenarios.

Ther are at least two important properties:

1. Consistency with signature: get_traces(start_frame, end_frame, segment_index) output is totally determined by the parameters on the signature.
2. Array-like behavior: get_traces(start_frame, end_frame) == get_traces(start_frame,end_frame + 10)[start_frame:end_frame, :]

Notablye that the naive solution of:

def _random_traces(self, start_frame: int, end_frame: int) -> np.ndarray:

    unique_seed_hash = (start_frame, end_frame, self.seed)
    local_rng = np.random.default_rng(seed=unique_seed_hash)
    traces = local_rng.random(size=(end_frame - start_frame, self.num_channels), dtype=self.dtype)
    
    return traces

Has property 1 but not property 2.

Right now, #1391 has a solution that accomplishes these two properties to this by returning a "deterministic" function with period output that looks like this:

The general shape is controlled by a random seed and it produces channels with different period, phase and amplitude. I was thinking in using to test detect_peaks scenarios when I wrote that.

That said, we would like to have more functionality to this function in the form of modes so we can support a wider variety of cases. The following outline describes them.

• Add capabilities to generate pure noise which also has property 2 above.
• Add capabilites to combine the "deterministic" output of the current solution with noise on top.
• Add simply sinusoidal behavior for LFP like scenarios.

[h-mayorquin]

For the pure noise scenario the idea is to generate a basic block of purse noise basic_trace = np.random.rand(some_small_number_of_frame, number_of_channels) and then use the frames with a modulus operation as a periodic tile to generate a noisy trace that has properties 1 and 2 above. Something like:

basic_trace[start_frame % size:end_frame % size]

Two possible concerns are that it imposes a minimal size on the trace (which I don't think is a big deal) and that the statistical properties of the noise may be strange with periodicity.

[samuelgarcia]

I think I wanted to say somethign like this

seed = (init_seed, segment_index, start_frame % size, end_frame % size +1) 
rng = np.random.default_rng(seed)
length = (end_frame - start_frame) rounded to next size
traces_with_border = rng.rand(some_small_number_of_frame, number_of_channels)
local_traces = traces_with_border[margin removed]

[DradeAW]

Following my comment in #1581
I think adding the option for white_noise on top of pure_noise could be beneficial.

It's fairly easy to implement: instead of each point in time following a uniform distribution, it follows a Gaussian distribution.

Generating white noise can be useful for testing. As it contains all the frequencies equally, you can test stuff like filtering and make sure it does what you expect it to do ...
For example in #1424 tests, I generate a white noise recording to make sure all of the frequencies are reconstructed correctly.

[alejoe91]

otherwise I think that pure_noise could be gaussian noise, which is a better model for experimental traces. I don't see a point of generating uniformly distributed noise. @samuelgarcia?

[h-mayorquin]

Yes, I was thinking that having pure_noise being gaussian noise probably makes sense.

As a reminder, two use cases that were discussed are:

1. To use this inside the current generate_recording.
2. To generate LFP-like signals.

Can those be done with Gaussian noise?

I would be against having two noise modes. I agree that the functionality will be simpy to implement but I think it is harder to mantain. When I designed the two current modalities I was very carefully to avoid calls to malloc as numpy sometimes makes them under the hood without making you aware. This is to say tha the function has a high mainteannce costs and I would like to diminish the surface areas to 1-3 modes. Power users ca combine those method as they please with ad-hoc post-processing or other tricks.

[alejoe91]

Not the LFP, but the underlying noise of AP signals can be as a first approx be considered gaussian.

[h-mayorquin]

I think that the objectives outlined here have been achieved. If resources allow, we will come back to the LFP case in the context of the hybrid generation project which will probably provide a better context for developing this.