BUG: creating Categorical from pandas Index/Series with "object" dtype infers string

[niruta25]

• closes BUG?: creating Categorical from pandas Index/Series with "object" dtype infers string #61778
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

Preserve object dtype for categories when constructing Categorical from pandas objects

This PR fixes an inconsistency in how pandas infers the dtype of categories when constructing a Categorical from different input types:

When constructing a Categorical from a pandas Series or Index with dtype="object", the categories' dtype is now preserved as object.
When constructing from a NumPy array with dtype="object" or a raw Python sequence, pandas continues to infer the most specific dtype for the categories (e.g., str if all elements are strings).
This change brings the behavior of Categorical in line with how Series and Index handle dtype preservation, making the API more consistent and predictable.

Example

pd.options.future.infer_string = True

ser = pd.Series(["foo", "bar", "baz"], dtype="object")
idx = pd.Index(["foo", "bar", "baz"], dtype="object")
arr = np.array(["foo", "bar", "baz"], dtype="object")
pylist = ["foo", "bar", "baz"]

cat_from_ser = pd.Categorical(ser)
cat_from_idx = pd.Categorical(idx)
cat_from_arr = pd.Categorical(arr)
cat_from_list = pd.Categorical(pylist)

# Series/Index with object dtype: preserve object dtype
assert cat_from_ser.categories.dtype == "object"
assert cat_from_idx.categories.dtype == "object"

# Numpy array or list: infer string dtype
assert cat_from_arr.categories.dtype == "str"
assert cat_from_list.categories.dtype == "str"

Documentation and release notes have been updated.
Closes: #61778

[niruta25]

@jbrockmendel Regarding this bug, the change to always preserve object dtype for categories when constructing a Categorical from a pandas Series or Index with dtype="object" is a behavioral change that affects a wide range of pandas internals and user-facing APIs. Hence I am seeing a lot of failures.

I see two ways to resolve without changing overall behavior.

1. Only Preserve object Dtype When All Elements Are Not Strings

• If the input is a pandas Series/Index with dtype="object", only preserve object dtype for categories if not all elements are strings.
• If all elements are strings, allow inference to str (the current behavior).

2. Add a Keyword Argument to Categorical (e.g., preserve_object_dtype=False)

• Add an explicit option to the Categorical constructor to preserve the object dtype for categories.
• Default to the current behavior, but allow users to opt in to preservation.

Let me know your thoughts.

[jbrockmendel]

this will probably go in tests.arrays.categorical.test_constructors or something similar

[jbrockmendel]

these imports go at the top of the file

[jbrockmendel]

use tm.option_context for this

[niruta25]

done

[jbrockmendel]

why is this removed?

[niruta25]

I felt the comments were redundant as TypeError already explain it clearly and also new logic is added to detect if the input values is a pandas Series or Index with "object" dtype, and then force the categories to use object dtype.

[niruta25]

Although I do not have any strong preference, I am happy to add it back.

[niruta25]

@jbrockmendel Regarding this bug, the change to always preserve object dtype for categories when constructing a Categorical from a pandas Series or Index with dtype="object" is a behavioral change that affects a wide range of pandas internals and user-facing APIs. Hence I am seeing a lot of failures.

I see two ways to resolve without changing overall behavior.

1. Only Preserve object Dtype When All Elements Are Not Strings

• If the input is a pandas Series/Index with dtype="object", only preserve object dtype for categories if not all elements are strings.
• If all elements are strings, allow inference to str (the current behavior).

2. Add a Keyword Argument to Categorical (e.g., preserve_object_dtype=False)

• Add an explicit option to the Categorical constructor to preserve the object dtype for categories.
• Default to the current behavior, but allow users to opt in to preservation.

Let me know your thoughts.

@jbrockmendel Thank you for your comments. I have addressed them. Although I doubt it would help address all the failing tests. Please let me know your thoughts. Any preference from above two options.

[niruta25]

@jbrockmendel Regarding this bug, the change to always preserve object dtype for categories when constructing a Categorical from a pandas Series or Index with dtype="object" is a behavioral change that affects a wide range of pandas internals and user-facing APIs. Hence I am seeing a lot of failures.
I see two ways to resolve without changing overall behavior.

1. Only Preserve object Dtype When All Elements Are Not Strings

• If the input is a pandas Series/Index with dtype="object", only preserve object dtype for categories if not all elements are strings.
• If all elements are strings, allow inference to str (the current behavior).

2. Add a Keyword Argument to Categorical (e.g., preserve_object_dtype=False)

• Add an explicit option to the Categorical constructor to preserve the object dtype for categories.
• Default to the current behavior, but allow users to opt in to preservation.

Let me know your thoughts.

@jbrockmendel Thank you for your comments. I have addressed them. Although I doubt it would help address all the failing tests. Please let me know your thoughts. Any preference from above two options.

I tried out the 1st way and all the test cases are passing. Let me know your thoughts.