Failing parameter recovery on truncated data

[ddgpalmer]

Hi there, I posted this a couple of weeks ago, but accidentally posted it to Issues instead of Discussions, so I'm moving it here.

I have a dataset for an experiment where the response window timed out at a fixed time, which means that on approximately 15% of trials, response and rt are unrecorded. I want to use a DDM to model participant responses, but am a little confused as to whether the missing or deadline conditions of HSSM is correct to use for this.

As a test, I simulated a dataset using HSSM, and modelled it four ways. First, I used the whole dataset. Second, I truncated it by deleting trials for which the rt was above the 85th percentile. Third, a version where those long trials were marked as -999 and the argument missing=True was passed to HSSM, and a version with an added column, deadline filled with the value of the 85th percentile and deadline=True passed to HSSM. I used the CPN and OPN supplied by @digicosmos in this thread https://github.com/lnccbrown/HSSM/issues/173.

The parameter estimates for each of these models are summarized in this Forest plot. You can see that the estimates from missing_data and deadline are essentially no different to those when I just delete the long responses. Code to reproduce this is below. I'd appreciate any advice as to whether I'm using the package wrong or whether this is a bug.

import hssm
import matplotlib.pyplot as plt
import arviz as az


# Simulate data
v_true, a_true, z_true, t_true = [0.5, 1.5, 0.5, 0.5]
dataset = hssm.simulate_data(
    model="ddm",
    theta=[v_true, a_true, z_true, t_true],
    size=1000,
)

truncated_del = dataset[dataset["rt"] < 3.9]  # Approximate 85% quantile

truncated_missing = dataset.copy()
truncated_missing.loc[truncated_missing["rt"] >= 3.9, "rt"] = -999
truncated_missing["deadline"] = 3.9

truncated_deadline = dataset.copy()
truncated_deadline.loc[truncated_missing["rt"] == -999, "rt"] = 3.9


# Create and sample models
full_model = hssm.HSSM(
    model="ddm",
    data=dataset,
)

truncated_model = hssm.HSSM(
    model="ddm",
    data=truncated_del,
)

missing_model = hssm.HSSM(
    model="ddm",
    data=truncated_missing,
    missing_data=-999,
    loglik_missing_data=r"C:\Users\ddgpa\Downloads\ddm_cpn (1).onnx",
)
deadline_model = hssm.HSSM(
    model="ddm",
    data=truncated_missing,
    deadline="deadline",
    loglik_missing_data=r"C:\Users\ddgpa\Downloads\ddm_opn.onnx",
)

full_idata = full_model.sample()
truncated_idata = truncated_model.sample()
missing_idata = missing_model.sample()
deadline_idata = deadline_model.sample()


# Plot
fig, ax = plt.subplots(figsize=(9, 5))
az.plot_forest(
    [full_idata, truncated_idata, missing_idata, deadline_idata],
    model_names=["Full", "Simple", "Missing", "Deadline"],
    combined=True,
    ax=ax,
)

[AlexanderFengler]

Hi @ddgpalmer,

as a starter, just want to make sure you are using the correct version of HSSM, 0.2.10.
The most recent release includes a few improvements on the OPN, CPN front.

Let me try to run your example I will report back.

[AlexanderFengler]

Ok, here are results from running this on my end.

Note 1:

Tiny changes in the code so don't forget to copy the code snippet directly.

Note 2:

Here is a link to a google drive folder with the opn and cpn I used. We are lagging a bit with making these more comprehensively available through huggingface, it needs one bigger push.
opn_cpn files

Note 3:

From what I understand about the scenario you are testing for, the correct setup should be to use an OPN with deadline. The CPN is more useful for a Go-Nogo situation.

Note 4:

You might find this preprint of ours interesting omitting omissions

import hssm
import matplotlib.pyplot as plt
import arviz as az


# Simulate data
v_true, a_true, z_true, t_true = [0.5, 1.5, 0.5, 0.5]
dataset = hssm.simulate_data(
    model="ddm",
    theta=[v_true, a_true, z_true, t_true],
    size=1000,
)

truncated_del = dataset[dataset["rt"] < 3.9]  # Approximate 85% quantile

truncated_missing = dataset.copy()
truncated_missing.loc[truncated_missing["rt"] >= 3.9, "rt"] = -999
truncated_missing["deadline"] = 3.9

truncated_deadline = dataset.copy()
truncated_deadline.loc[truncated_missing["rt"] == -999, "rt"] = 3.9


# Create and sample models
full_model = hssm.HSSM(
    model="ddm",
    data=dataset,
)

truncated_model = hssm.HSSM(
    model="ddm",
    data=truncated_del,
)

missing_model = hssm.HSSM(
    model="ddm",
    data=truncated_missing,
    missing_data=-999.,
    loglik_missing_data="test_cpn/ddm_cpn_batch.onnx",
)

deadline_model = hssm.HSSM(
    model="ddm",
    data=truncated_missing,
    missing_data=-999.,
    deadline="deadline",
    loglik_missing_data="test_opn/ddm_opn_batch.onnx",
)

full_idata = full_model.sample(sampler="nuts_numpyro", chain_method="vectorized")
truncated_idata = truncated_model.sample(sampler="nuts_numpyro", chain_method="vectorized")
missing_idata = missing_model.sample(sampler="nuts_numpyro", chain_method="vectorized")
deadline_idata = deadline_model.sample(sampler="nuts_numpyro", chain_method="vectorized")


# Plot
fig, ax = plt.subplots(figsize=(9, 5))
az.plot_forest(
    [full_idata, truncated_idata, missing_idata, deadline_idata],
    model_names=["Full", "Simple", "Missing", "Deadline"],
    combined=True,
    ax=ax,
)

[ddgpalmer]

Hi @AlexanderFengler, thanks for taking the time to look into this and respond. I've updated to the latest version of hssm, and when I run the code there is now a difference between the methods, with the deadline looking much closer to the true value. The estimate isn't as good as yours though, which I presume is due to the older opn file I'm still using. I can't access the folder you linked for the latest cpn and opn, so I've sent an access request. It appears from your results that using them will get me to the result I'm looking for.

Thanks also for the preprint, which is very helpful. It's surprising that this issue has been ignored for so long given how significant its effect is on estimate accuracy.

[AlexanderFengler]

Access granted @ddgpalmer. Thank you for your patience.

[ddgpalmer]

It's working beautifully now. Thanks for your help 😃.