Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Jul 15, 2025

This PR implements a complete nuclio serverless function that wraps the AOC fruit detector for seamless integration with CVAT (Computer Vision Annotation Tool). The solution enables automatic fruit detection and annotation in CVAT workflows while maintaining the existing ROS2-based fruit detector architecture.

Architecture Overview

The implementation follows a two-container design:

  1. Fruit Detector Container: The existing ROS2 node performing fruit detection using Detectron2 MaskRCNN
  2. Nuclio Function Container: A new lightweight serverless function that bridges CVAT and the fruit detector

Communication Flow

CVAT → HTTP POST → Nuclio Function → ROS2 /camera/image_raw → Fruit Detector
                                 ←                        ← /fruit_info ←
CVAT ← HTTP Response ← Nuclio Function ← ROS2 Subscription

Key Components

Core Implementation (nuclio_function/)

  • main.py: Main nuclio function handler implementing CVAT-compatible HTTP interface
  • ros2_bridge.py: ROS2 communication bridge managing image publishing and result subscription
  • cvat_converter.py: Converter transforming AOC FruitInfoArray messages to CVAT annotation format
  • Dockerfile: Multi-stage container build for nuclio deployment

Deployment Infrastructure

  • function.yaml: Nuclio function configuration with resource limits and triggers
  • deploy.sh: Automated deployment script for Kubernetes/Docker environments
  • docker-compose.yml: Local development environment setup

Testing & Validation

  • validate.py: Syntax and logic validation without ROS2 dependencies
  • test_function.py: End-to-end integration testing with visualization
  • health_check.py: Production health monitoring utility

Documentation & Examples

  • README.md: Comprehensive deployment and usage instructions
  • CVAT_INTEGRATION.md: Step-by-step CVAT integration guide
  • cvat_model_config.json: Example CVAT model configuration
  • IMPLEMENTATION_SUMMARY.md: Technical architecture documentation

Message Format Handling

The implementation correctly processes the rich AOC message structure:

# Input: FruitInfoArray with detailed fruit detection data
FruitInfoMessage:
  - Basic detection: bbox, confidence, fruit_type
  - Ripeness analysis: ripeness_category, ripeness_level  
  - Physical properties: area, volume, weight
  - Quality metrics: fruit_quality, occlusion_level
  - Botanical classification: pomological_class, edible_plant_part

# Output: CVAT-compatible annotations
[
  {
    "confidence": 0.95,
    "label": "strawberry_ripe", 
    "points": [x1, y1, x2, y2],
    "type": "rectangle",
    "attributes": {
      "ripeness_level": 0.85,
      "quality": "High",
      "area": 1234.5
    }
  }
]

Key Features

Serverless Architecture: Production-ready nuclio function with auto-scaling
ROS2 Integration: Uses /camera/image_raw and /fruit_info topics as specified
Rich Metadata: Preserves all AOC detection attributes in CVAT format
Error Handling: Robust timeout management and graceful degradation
Development Support: Mock classes and validation for testing without ROS2
Documentation: Complete deployment guides and integration examples

Usage

Deployment

cd nuclio_function
./deploy.sh

CVAT Integration

  1. Deploy the nuclio function
  2. Add model in CVAT admin panel with function endpoint
  3. Configure fruit detection labels (strawberry, tomato, etc.)
  4. Use automatic annotation in CVAT tasks

Testing

# Validate implementation
python3 validate.py

# Test deployed function
python3 test_function.py --url http://function-endpoint --image test.jpg

# Health check
python3 health_check.py http://function-endpoint

Compliance with Requirements

  • ✅ Separate container implementation for nuclio function
  • ✅ ROS2 communication via /camera/image_raw/fruit_info topics
  • ✅ CVAT serverless interface compatibility
  • ✅ Proper handling of FruitInfoArray and FruitInfoMessage structures
  • ✅ RGB-only processing (no depth images required)
  • ✅ Nuclio deployment from Dockerfile
  • ✅ Comprehensive documentation and examples

This implementation provides a production-ready bridge between the AOC fruit detector and CVAT, enabling seamless automatic annotation workflows for agricultural computer vision tasks.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@Copilot Copilot AI changed the title [WIP] I want to wrap the fruit detector as a nuclio serverless function to be used in CVAT. create a new branch with that implementation. Research how to implement this research https://docs.nuclio.io/en/stable/tasks/deploy-functions-from-dockerfile.html and ... Implement Nuclio serverless function for CVAT integration with AOC Fruit Detector Jul 15, 2025
@Copilot Copilot AI requested a review from marc-hanheide July 15, 2025 09:44
Copilot finished work on behalf of marc-hanheide July 15, 2025 09:44
Copy link
Member

@marc-hanheide marc-hanheide left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will have to wait until #34 is merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants