Use of a functools.partial not supported in cost.UnbinnedNLL?

[GWeberJ]

With 2.4.0 (February 10, 2021) it is claimed that "Minuit now supports functions wrapped with functools.partial, by @jnsdrtlf".

However, with iminuit-2.31.1 I get the following error:

File C:\ProgramData\anaconda3\envs\E134\Lib\site-packages\iminuit\minuit.py:2653, in _make_init_state(pos2var, args, kwds)
2650 nargs = len(kwds)
2652 if len(pos2var) != nargs:
-> 2653 raise RuntimeError(
2654 f"{nargs} values given for {len(pos2var)} function parameter(s)"
2655 )
2657 state = MnUserParameterState()
2658 for i, x in enumerate(pos2var):

RuntimeError: 4 values given for 6 function parameter(s)

Relevant lines of code:
def pdf(x, A, B, C, D, data_x, data_y):
...
if name == 'main':
...
pdf_now = partial(pdf, data_x=x_axis, data_y=ref_hist)
c = cost.UnbinnedNLL(test60, pdf_now)

Am I doing something wrong or is this really a problem of iminuit?

Thank you very much!

[GWeberJ]

Ok, found the reason:

Case 1:
def pdf(x, A, B, C, D, data_x, data_y):
...

pdf_now = partial(pdf, data_x=x_axis, data_y=ref_hist)
sig = signature(pdf_now)
str(sig)
Out[17]: '(x, A, B, C, D, *, data_x=array([...], shape=(14999,)), data_y=array([...], shape=(14999,)))'

Case 2:
def pdf(data_x, data_y, x, A, B, C, D):
...

pdf_now = partial(pdf, x_axis, ref_hist)
sig = signature(pdf_now)
str(sig)
Out[21]: '(x, A, B, C, D)'

Only in Case 2, the parameters that are fixed by partial(func, ...) are removed from the signature. It would be really nice, if during parsing the function signature iminuit could handle this (at least to me) unintuitive behaviour of partial().

More information on this: https://stackoverflow.com/questions/71402387/the-rationale-of-functools-partial-behavior

[3j14]

I think the Stack Overflow thread that you linked sums it up perfectly: When using partial(pdf, data_x=x_axis, data_y=ref_hist), you turn data_x and data_y into keyword arguments. This means that the function signature becomes equivalent to the following:

def pdf(x, A, B, C, D, data_x=x_axis, data_y=ref_hist):
    # ...

So there is really no difference to using partial(pdf, data_x=x_axis, data_y=ref_hist) and defining the method as above. For both instances, you will get the same error.

Internally, Minuit uses iminuit.util.describe to parse the function signature -- which treats keyword arguments the same as positional arguments.

So while I agree that this behavior is counter intuitive, I would argue that there is nothing we can do about it as keyword arguments are treated the same as positional arguments in Minuit. partial does not remove the arguments if you specify them as a keyword argument. You can achieve this behavior e.g. by using lambda:

def pdf(x, A, B, C, D, data_x, data_y):
    # ...

c = cost.UnbinnedNLL(test60, lambda x, A, B, C, D: pdf(x, A, B, C, D, data_x=x_axis, data_y=ref_hist))

Though this is very verbose. Instead, you can use *args but you will lose the automatic detection of parameter names:

c = cost.UnbinnedNLL(test60, lambda *args: pdf(*args, data_x=x_axis, data_y=ref_hist))

[HDembinski]

I am inclined to agree. We can't treat keyword arguments with and without default values different in describe, because it is not obvious that those should be ignored. Zen of Python: "When facing ambiguity, refuse the temptation to guess."

However, I also agree that the behavior of partial is quite surprising. Maybe we can special-case this when partial is used. The lambda workaround also works, of course.