v1.0 InferencePool API Review #1173

capri-xiyue · 2025-07-16T19:06:41Z

What type of PR is this?
/kind api-change

What this PR does / why we need it:
This PR is a diff of /apis from alpha (main branch) to v1.0 (release-1.0 branch).

Note: This PR is purely to facilitate review, it is not intended to merge.

Changes:

group change from "inference.networking.x-k8s.io" to "inference.networking.k8s.io"
Change the InferencePool.Selector from map[string]string to a struct for future flexibility in v1. See Upgrade the inferencePool selector to a struct from a map. #1330
Simplify EndpointPickerConfig(remove the whole inline struct)
TargetPortNumber int32 to become TargetPorts []Port see feat: TargetPortNumber int32 to become TargetPorts []Port #1354

/assign @robscott

k8s-ci-robot · 2025-07-16T19:06:50Z

Hi @capri-xiyue. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

capri-xiyue · 2025-07-16T19:07:01Z

/hold
Don't merge as it is just for api review.

netlify · 2025-07-16T19:07:12Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`6c02159`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/687811ca8fb1b50008233e1f
😎 Deploy Preview	https://deploy-preview-1173--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

capri-xiyue · 2025-07-16T20:53:02Z

/hold, this should not get merged

robscott · 2025-07-17T06:11:54Z

Note: This PR is NOT intended to merge, it is entirely for the purpose of API review.

/cc @aojea @danwinship @thockin

thockin · 2025-08-14T23:13:01Z

apix/v1/inferencepool_types.go

+	// associated with the InferencePool, and the status of the InferencePool with respect to
+	// each parent.
+	//
+	// A maximum of 32 Gateways will be represented in this list. When the list contains


I do not understand the comment about "kind: Status, name: default" -- where are those fields? They are not in PoolStatus.

I also don't understand why you need a canary entry to signal "nothing to see" -- why is an empty list not sufficient? I find the imposition on the controllers to be very awkward.

This was copied from Gateway API. Generally k8s users don't seem to treat the absence of status as a bad thing, so they may not realize their InferencePool is not being used/referenced.

This is particularly useful in Gateways where you made a typo on gatewayClass and nothing picks up/implements your Gateway. Having a baseline signal that a Gateway is waiting for a controller to pick it up is useful.

A similar pattern is useful for Routes if you make a typo referring to a Gateway - there's no controller empowered to warn you of this since each Gateway controller only populates status on Routes that are attached to their Gateway. If there's a reference to a nonexistent Gateway, there's nothing to warn you about that other than some kind of default status.

With that said, this doesn't seem quite as useful on InferencePool. If someone makes a typo when pointing a route to an InferencePool, they'll get a warning in the status of that Route. This status can be useful in indicating that the InferencePool has not actually been implemented yet, but an empty list may suffice here.

cc @danehans

I do like the explicitness of the current approach, but I don't have a strong opinion here.

I'd make it as resolved for now seems the majority is ok with the current approach. Feel free to re-open it if further discussion is needed.

I find this very brittle and awkward. Doesn't this argue instead for something like an MAP which detects an empty list and inserts the canary?

xref kubernetes-sigs/gateway-api#3738 for adding a similar default status condition to HTTPRoute.

As a quick update from Slack - we discussed this and decided to remove the defaulting for now. In the corresponding Gateway issue we've been looking into MAP as a potential solution as well. Although it's not clear that will work as we'd like, I think it makes sense to hold off on this until we have more time to look into a MAP-based solution.

Can I resolve this with the agreement to remove the defaulting for now?

#1427 is removing the defaulting.

apix/v1/inferencepool_types.go

capri-xiyue · 2025-08-19T16:25:26Z

/reopen

k8s-ci-robot · 2025-08-19T16:35:07Z

@capri-xiyue: Reopened this PR.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

thockin

I am LGTM except for a few nits

apix/v1/inferencepool_types.go

thockin · 2025-08-21T22:33:46Z

apix/v1/inferencepool_types.go

+	// +listType=map
+	// +listMapKey=type
+	// +kubebuilder:validation:MaxItems=8
+	// +kubebuilder:default={{type: "Accepted", status: "Unknown", reason:"Pending", message:"Waiting for controller", lastTransitionTime: "1970-01-01T00:00:00Z"}}


Is this right? I don't understand it

I will further update it once #1427 gets merged.

#1427 removes the status condition defaulting.

JoelSpeed · 2025-08-22T15:12:44Z

I lost the comment thread, but I believe I saw something along the lines of:

A decision has been made to stick to pointers for optional structs, but pointers only when required for scalars

If that's the case, you can set KALs config for optionalfields.omitzero.policy: Forbid and it will behave as you intend.

Omitzero support is only for structs in KAL right now, and with it set to forbid, it will force the omitempty route with a pointer for all optional structs

danehans · 2025-08-22T17:35:50Z

FYI #1427 is a PR under review that implements several API changes based on feedback from API reviewers here and discussions among the community, maintainers, etc.

danehans · 2025-08-22T17:47:33Z

Given it is consistent, I am satisified. Implementations MUST use RV preconditions. If there is a v2, I would suggest to inline the Ref here and use those fields as the map key.

@thockin the challenge I see with inlining is that no field by itself can be used as the map key:

type ParentReference struct {
	// Group is the group of the referent API object. When unspecified, the referent is assumed
	// to be in the "gateway.networking.k8s.io" API group.
	//
	// +optional
	// +kubebuilder:default="gateway.networking.k8s.io"
	Group *Group `json:"group,omitempty"`

	// Kind is the kind of the referent API object. When unspecified, the referent is assumed
	// to be a "Gateway" kind.
	//
	// +optional
	// +kubebuilder:default=Gateway
	Kind *Kind `json:"kind,omitempty"`

	// Name is the name of the referent API object.
	//
	// +required
	Name ObjectName `json:"name,omitempty"`

	// Namespace is the namespace of the referenced object. When unspecified, the local
	// namespace is inferred.
	...
	// +optional
	Namespace *Namespace `json:"namespace,omitempty"`
}

All fields are needed to ensure the uniqueness of a parent.

capri-xiyue · 2025-08-22T22:49:38Z

I lost the comment thread, but I believe I saw something along the lines of:
A decision has been made to stick to pointers for optional structs, but pointers only when required for scalars
If that's the case, you can set KALs config for optionalfields.omitzero.policy: Forbid and it will behave as you intend.

Omitzero support is only for structs in KAL right now, and with it set to forbid, it will force the omitempty route with a pointer for all optional structs

I tried it, with omitzero forbid, https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/1438/files#diff-8d7db842a7673f66c7db001de9fe07ce67b67e21d8cbce12394fde46eea1e5b7R36 I still need ignore kubeapi linter otherwise it will also show optionalfields: field Kind does not allow the zero value. The field does not need to be a pointer. (kubeapilinter) Kind *Kind `json:"kind,omitempty" for optional pointer struct where zero value is not allowed

k8s-ci-robot · 2025-08-22T23:38:39Z

@capri-xiyue: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gateway-api-inference-extension-test-unit-main	`33c8e89`	link	true	`/test pull-gateway-api-inference-extension-test-unit-main`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

robscott · 2025-08-22T23:42:26Z

@thockin @aojea I believe we've addressed all your feedback, thanks! PTAL

JoelSpeed · 2025-08-23T07:39:10Z

I still need ignore kubeapi linter otherwise it will also show optionalfields: field Kind does not allow the zero value. The field does not need to be a pointer. (kubeapilinter) Kind *Kind `json:"kind,omitempty" for optional pointer struct where zero value is not allowed

Kind is not a struct, it is a string. You have a non-zero minimum length marker on kind so why do we need it to be a pointer? The string being empty implies to the go client that it wasn't present when admitted

JoelSpeed · 2025-08-23T07:40:53Z

apix/v1/inferencepool_types.go

+	// Status defines the observed state of the InferencePool.
+	//
+	// +optional
+	//nolint:kubeapilinter // status should not be a pointer.


Why not? It's the same as any other struct

We followed the existing k8s convention. See https://github.com/kubernetes/api/blob/master/batch/v1/types.go#L84

JoelSpeed · 2025-08-23T07:41:27Z

apix/v1/inferencepool_types.go

+	//
+	// +optional
+	// +kubebuilder:default=Service
+	//nolint:kubeapilinter // ignore kubeapilinter here as we want to use pointer for optional struct.


It's a string, not a struct, this should not be ignored

Same as above, moved the discussion to https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/1173/files#r2295450243

JoelSpeed · 2025-08-23T07:42:45Z

apix/v1/inferencepool_types.go

+	// unspecified (defaults to "Service").
+	//
+	// +optional
+	//nolint:kubeapilinter // ignore kubeapilinter here as we want to use pointer for optional struct.


Not a struct, it's an integer. I know we wanted to not allow 0, but this isn't the right way to except this. Add an exception to the golang ci lint config with the explicit error message.

Right now you're ignoring all possible KAL issues and that might mask other problems with the field

zero means all ports by convention, we don't want to use zero to indicate not set, therefore we want to add a exception here. I will change the comment as it is not accurate

JoelSpeed · 2025-08-23T07:44:53Z

apix/v1/inferencepool_types.go

+	//
+	// +optional
+	// +kubebuilder:default="FailClose"
+	//nolint:kubeapilinter // ignore kubeapilinter here as we want to use pointer for optional struct.


Not a struct, it's a string. Enum validation prevents the empty string, this doesn't need to be a pointer.

Go clients can check if empty to assert whether the string was provided or not

Do we consider such type EndpointPickerFailureMode string as a built-in type string or a struct(In reality, it is neither a struct nor a built-in type) But maybe more close to a string when it comes to serialization/deserialization? cc @robscott @kfswain As we are going to cut the RC, if we decide to make a last-minute change, I can send a PR for the fix.

#1444 is ready to merge if we want to change it to non-pointer for such named type.

EndpointPickerFailureMode

This is a string, it's definitely not a struct (that would be type Foo struct)

JoelSpeed · 2025-08-23T07:46:16Z

apix/v1/inferencepool_types.go

+	//
+	// +optional
+	// +kubebuilder:default=Gateway
+	//nolint:kubeapilinter // ignore kubeapilinter here as we want to use pointer for optional struct.


Is a string, not a struct. Empty string isn't valid, should still be a non-pointer

Same as above, moved the discussion to https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/1173/files#r2295450243

JoelSpeed · 2025-08-23T07:47:12Z

apix/v1/inferencepool_types.go

+	// documentation for details: https://gateway-api.sigs.k8s.io/api-types/referencegrant/
+	//
+	// +optional
+	//nolint:kubeapilinter // ignore kubeapilinter here as we want to use pointer for optional struct.


Is a string, not a struct, empty namespace is not a valid choice, does not need to be a pointer

Same as above, moved the discussion to https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/1173/files#r2295450243

JoelSpeed · 2025-08-23T07:49:51Z

Further to my comment above, did we decide all optional fields should be pointers, or, only optional structs? Cc @robscott

I thought the latter, but the exceptions I'm seeing imply the former

Either decision can be configured in KAL so we shouldn't need any exceptions apart from the port number where a specific decision was made to ignore the guidance about the zero value

k8s-ci-robot assigned robscott Jul 16, 2025

k8s-ci-robot added kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 16, 2025

k8s-ci-robot requested review from kfswain and nirrozenbaum July 16, 2025 19:06

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 16, 2025

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jul 16, 2025

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 16, 2025

capri-xiyue closed this Jul 16, 2025

capri-xiyue force-pushed the capri-xiyue/capri-xiyue-v1-api-review branch from 5317094 to 4ffb5f6 Compare July 16, 2025 20:36

k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 16, 2025

capri-xiyue reopened this Jul 16, 2025

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 16, 2025

capri-xiyue closed this Jul 16, 2025

capri-xiyue force-pushed the capri-xiyue/capri-xiyue-v1-api-review branch from 77371fe to 4ffb5f6 Compare July 16, 2025 20:53

k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 16, 2025

capri-xiyue reopened this Jul 16, 2025

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 16, 2025

danehans added this to the v1.0 (InferencePool GA) milestone Aug 14, 2025

updated

f4ccc68

thockin reviewed Aug 14, 2025

View reviewed changes

liggitt reviewed Aug 15, 2025

View reviewed changes

apix/v1/inferencepool_types.go Show resolved Hide resolved

aojea reviewed Aug 18, 2025

View reviewed changes

apix/v1/inferencepool_types.go Show resolved Hide resolved

capri-xiyue mentioned this pull request Aug 18, 2025

fix: updated listtype for targetports #1401

Merged

updated api

0050467

k8s-ci-robot closed this in #1401 Aug 19, 2025

kfswain reopened this Aug 19, 2025

updated targetports listtype

80dee7b

capri-xiyue mentioned this pull request Aug 20, 2025

fix(cleanup): change the naming to be endpointpickerref #1420

Merged

updated to use endpointpickerref

9981a82

danehans mentioned this pull request Aug 21, 2025

InferencePool: Add Approval CRD Annotation #1424

Open

capri-xiyue mentioned this pull request Aug 21, 2025

fix(cleanup) added pointer usage for failure mode #1428

Closed

danehans mentioned this pull request Aug 21, 2025

Refactors InferencePool v1 Status and FailureModes #1427

Open

thockin reviewed Aug 21, 2025

View reviewed changes

capri-xiyue mentioned this pull request Aug 22, 2025

fork daneyon's change - Refactors InferencePool v1 Status and FailureModes #1438

Closed

robscott mentioned this pull request Aug 22, 2025

Final bits of v1.0 API cleanup #1441

Merged

final clean up

33c8e89

JoelSpeed reviewed Aug 23, 2025

View reviewed changes

v1.0 InferencePool API Review #1173

Are you sure you want to change the base?

v1.0 InferencePool API Review #1173

Conversation

capri-xiyue commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jul 16, 2025

Uh oh!

capri-xiyue commented Jul 16, 2025

Uh oh!

netlify bot commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

capri-xiyue commented Jul 16, 2025

Uh oh!

robscott commented Jul 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danehans Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

capri-xiyue commented Aug 19, 2025

Uh oh!

k8s-ci-robot commented Aug 19, 2025

Uh oh!

thockin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoelSpeed commented Aug 22, 2025

Uh oh!

danehans commented Aug 22, 2025

Uh oh!

danehans commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

capri-xiyue commented Aug 22, 2025

Uh oh!

k8s-ci-robot commented Aug 22, 2025

Uh oh!

robscott commented Aug 22, 2025

Uh oh!

JoelSpeed commented Aug 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

capri-xiyue commented Jul 16, 2025 •

edited

Loading

netlify bot commented Jul 16, 2025 •

edited

Loading

danehans Aug 20, 2025 •

edited

Loading

danehans commented Aug 22, 2025 •

edited

Loading

capri-xiyue Aug 23, 2025 •

edited

Loading