Add aks deployment

[vofish]

• Added AKS support for Gemma-2b
• Selected NVIDIA T4 GPU for deployment, as L4 GPUs are not available in Azure.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vofish
Once this PR has been reviewed and has the lgtm label, please assign jjk-g for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• serving-catalog/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[huaxig]

FYI, the output of kustomize build core/deployment/vllm/gemma-2b/aks:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: gemma-2b-vllm-inference-server
  name: gemma-2b-vllm-service
spec:
  ports:
  - port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    app: gemma-2b-vllm-inference-server
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: gemma-2b-vllm-inference-server
  name: gemma-2b-vllm-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gemma-2b-vllm-inference-server
  template:
    metadata:
      labels:
        ai.gke.io/inference-server: vllm
        ai.gke.io/model: gemma-2b
        app: gemma-2b-vllm-inference-server
        examples.ai.gke.io/source: blueprints
    spec:
      containers:
      - args:
        - --model=$(MODEL_ID)
        - --tensor-parallel-size=1
        command:
        - python3
        - -m
        - vllm.entrypoints.openai.api_server
        env:
        - name: MODEL_ID
          value: google/gemma-2b
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              key: hf_api_token
              name: hf-secret
        image: vllm/vllm-openai:latest
        name: inference-server
        ports:
        - containerPort: 8000
          name: metrics
        readinessProbe:
          failureThreshold: 60
          httpGet:
            path: /health
            port: 8000
          periodSeconds: 10
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            nvidia.com/gpu: 1
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
      nodeSelector:
        kubernetes.azure.com/accelerator: nvidia
      volumes:
      - emptyDir:
          medium: Memory
        name: dshm

[huaxig]

I'm a bit unclear on how this GPU patch restricts the deployment to only use T4-type GPU instances. Looking at the t4.yaml file, the nodeSelector is set to kubernetes.azure.com/accelerator: nvidia. This seems to select any node with an NVIDIA GPU, rather than specifically targeting T4 instances.

Could you clarify how the T4 type is enforced?

Also, have you tested this on AKS? If so, did you use a manual node pool (perhaps one provisioned with only T4 instances) or an automatic node pool? If it was an automatic node pool, how is this resource request bound specifically to T4 GPUs?