Skip to content

reference.md

Latest commit

 

History

History
646 lines (413 loc) · 13.6 KB

reference.md

File metadata and controls

646 lines (413 loc) · 13.6 KB
Raw

UnrealCV Runtime MCP Tool Reference

The UnrealCV Runtime MCP server currently exposes 8 toolsets and 36 tools.

1. UnrealCV Toolset

Low-level access to the existing UnrealCV command system.

unrealcv.list_cmd

Lists registered UnrealCV commands and their descriptions.

Use this tool to discover legacy commands that have not yet been wrapped as dedicated MCP tools.

unrealcv.describe_command

Returns the description and usage information for one UnrealCV command template.

Key input:

  • command: UnrealCV command or command template to inspect.

unrealcv.exec

Executes a raw command through the UnrealCV command dispatcher.

Key input:

  • command: Complete UnrealCV command string, such as vget /objects.

Use dedicated MCP tools when available because they provide structured schemas and results.

2. Scene Toolset

Provides runtime scene discovery, perception, inspection, capture, and spatial queries.

scene.overview

Returns an overview of the scene around the main camera.

The result includes:

  • Camera pose
  • Nearby actors
  • Actor distances
  • View-direction signals
  • Occlusion and visibility information

Key inputs:

  • radius: Search radius in centimeters.
  • max_actors: Maximum number of actors returned.

scene.list_actors

Lists actors in the current Game or PIE world.

Supports:

  • Runtime-name filtering
  • Class-path filtering
  • Actor-tag filtering
  • Pagination
  • Stable name ordering

Key inputs:

  • name_contains: Case-insensitive actor-name substring.
  • class_contains: Case-insensitive class-path substring.
  • tag: Exact Actor tag, matched case-insensitively.
  • offset: Pagination offset.
  • limit: Maximum number of actors returned.

Returned actor names can be passed to inspection, camera, character, and spawn tools.

scene.perceive

Returns a compact structured perception snapshot.

The result can include:

  • Map and agent state
  • Nearby objects
  • Navigation reachability
  • Line-of-sight information
  • Radial collision rays

Key inputs:

  • radius: Perception radius in centimeters.
  • max_objects: Maximum nearby objects.
  • max_rays: Maximum radial collision rays.

scene.inspect_actor

Inspects one runtime actor.

The result includes:

  • Runtime name and class
  • Location, rotation, and scale
  • World-space bounds
  • Tags
  • Owner
  • Components

Key input:

  • actor: Exact runtime actor name.

scene.capture_view

Captures the current lit scene and saves it as a PNG in the system temporary directory.

Supported capture sources:

  • mvrc: Main viewport backbuffer; follows the main window camera and usually provides the best visual quality.
  • mqrc: Independently configurable movie-quality renderer; may require warmup.

Key inputs:

  • camera_id: UnrealCV camera ID.
  • source: mvrc or mqrc.
  • width: Requested image width.
  • height: Requested image height.

The result includes the saved file path, dimensions, renderer class, MIME type, and byte count.

scene.raycast_down

Traces vertically downward from a world coordinate.

The result includes:

  • Whether a blocking surface was hit
  • Impact location
  • Surface normal
  • Hit actor and component
  • Physical material
  • Trace distance
  • Basic walkability classification

Key inputs:

  • x, y, z: Trace origin in world-space centimeters.
  • trace_length_cm: Maximum downward trace distance.

scene.check_region

Checks whether an axis-aligned world-space region is clear.

The result includes:

  • Whether the region is clear
  • Intersecting actors
  • Actor classes
  • Whether intersecting actors were generated by the MCP workflow

Key inputs:

  • center_x, center_y, center_z
  • extent_x, extent_y, extent_z

3. Perception Toolset

Provides native agent-oriented scene perception.

perception.snapshot

Returns structured scene state without requiring image recognition.

The result can include:

  • Agent pose
  • Nearby objects and bounds
  • Line of sight
  • Navigation reachability
  • Radial collision rays
  • Occupancy capability metadata

Key inputs:

  • radius_cm: Radius around the main camera.
  • max_objects: Maximum number of nearby objects.
  • max_rays: Maximum number of radial rays.

4. Camera Toolset

Controls the main runtime camera and actor-following behavior.

camera.get_pose

Returns the current main camera state:

  • Location
  • Rotation
  • Forward direction
  • Field of view

camera.set_pose

Moves the main camera to an exact world-space pose.

Key inputs:

  • location: {x, y, z} in centimeters.
  • rotation: {x, y, z}, mapped to pitch, yaw, and roll in degrees.

camera.look_at

Points the main camera at a world-space target.

The camera can rotate in place or move to a new location first.

Key inputs:

  • target: World-space point to look at.
  • location: Optional new camera location.

camera.set_view_preset

Places the camera relative to a target and aims it at that target.

Supported presets include:

  • top_down
  • bottom_up
  • x_positive
  • x_negative
  • y_positive
  • y_negative
  • Axis-aligned 45-degree downward views

Key inputs:

  • target: Center of the view.
  • preset: View direction preset.
  • distance_cm: Distance between camera and target.

camera.frame_bounds

Frames an actor using its world-space bounds and the main camera field of view.

Key inputs:

  • actor: Exact runtime actor name.
  • preset: View direction.
  • margin: Framing-distance multiplier.

camera.follow_actor

Makes an actor the first local PlayerController's view target and activates one explicitly named CameraComponent.

Automatic camera inference is intentionally unsupported.

Required inputs:

  • actor: Exact runtime actor name.
  • camera_component: Exact, case-sensitive CameraComponent name.

Optional input:

  • blend_time_seconds: View transition duration.

For the tested SmartLocomotion character:

  • First-person camera: FPS Camera
  • Third-person camera: TPS Camera

This tool changes the view target but does not possess the actor or transfer movement input.

camera.detach

Stops following a separate actor and restores the PlayerController's currently possessed Pawn as its view target.

Optional input:

  • blend_time_seconds: Transition duration back to the possessed Pawn.

This tool does not:

  • Change possession
  • Move actors
  • Destroy actors
  • Select a CameraComponent

After detaching, use camera.set_pose or camera.look_at for free camera movement.

5. Assets Toolset

Searches and inspects spawnable Unreal assets.

assets.search

Searches for:

  • Static Meshes
  • Skeletal Meshes
  • Actor Blueprints

The first query for a path reads AssetRegistry metadata without loading Blueprint classes. Later queries reuse a path-based cache.

Key inputs:

  • path: Asset root, such as /Game/SmartLocomotion.
  • search_spec: Case-insensitive substring matched against asset name, type, and path.
  • offset: Pagination offset.
  • limit: Maximum returned entries.
  • refresh: Force a fresh AssetRegistry query and replace the path cache.

The result includes:

  • Matching assets
  • Total count
  • Pagination state
  • Cache-hit status
  • AssetRegistry scan time
  • Filtering time

assets.get_info

Loads one spawnable asset and returns:

  • Asset type
  • Object path
  • Bounding-box metadata
  • Size description
  • Bounds source and confidence

Key input:

  • asset_path: Full Unreal asset object path.

assets.get_bounds

Returns local bounding-box and size metadata for a spawnable asset.

Key input:

  • asset_path: Full Unreal asset object path.

6. Spawn Toolset

Spawns, places, validates, and destroys runtime actors.

spawn.from_asset

Spawns a Static Mesh, Skeletal Mesh, or Actor Blueprint asset.

Supports:

  • Explicit transform
  • Custom runtime name
  • Ground settling
  • Placement annotation
  • Overlap rejection
  • Placement rollback

Key inputs:

  • asset_path: Spawnable asset path.
  • location: Initial world location.
  • rotation: Optional pitch, yaw, and roll.
  • name: Optional unique actor name.
  • settle: none, simple, bounds, or sweep.
  • height_offset_cm: Height above detected ground.
  • reject_overlaps: Whether invalid overlapping placement should be rolled back.

spawn.get_overlaps

Lists workflow-generated actors whose world-space bounds overlap a target actor.

Key input:

  • actor: Exact runtime actor name.

spawn.validate_placement

Validates an existing generated actor.

Checks:

  • Generated-object overlap
  • Ground contact
  • Overall placement validity

Key input:

  • actor: Exact runtime actor name.

spawn.settle_to_ground

Moves an existing actor onto the first valid surface below it.

Key inputs:

  • actor: Exact runtime actor name.
  • mode: simple, bounds, or sweep.
  • height_offset_cm: Final height above the detected surface.

spawn.destroy

Destroys a runtime actor.

This can also be used to roll back failed generated placements.

Key input:

  • actor: Exact runtime actor name.

7. Blueprint Toolset

Discovers and invokes reflected Blueprint and native Unreal functions.

blueprint.inspect_api

Discovers callable reflected functions on:

  • Runtime objects
  • Components
  • Class default objects

The result includes:

  • Exact function name
  • Display name
  • Declaring class
  • Function flags
  • Parameter names
  • C++ parameter types
  • Input/output direction

Key inputs:

  • target: Runtime object name, component name, class:<class path>, or cdo:<class path>.
  • name_filter: Optional case-insensitive function-name filter.

Use this tool before blueprint.call when a function signature is unknown.

blueprint.call

Invokes a reflected BlueprintCallable or native UFunction using structured JSON arguments.

Key inputs:

  • target: Runtime object, component, class, or class default object.
  • function: Exact reflected function name.
  • arguments: JSON object keyed by exact reflected parameter names.

The result includes output parameters and return values as structured JSON.

8. Character Toolset

Provides a unified workflow for character spawning, physics, controllers, navigation, movement, and diagnostics.

character.spawn_ready

Spawns a controllable character from an Actor Blueprint.

Default asset:

/Game/SmartLocomotion/Blueprints/BP_Character.BP_Character

The asset path can be replaced for other character Blueprints.

Key inputs:

- location: Initial world location.
- rotation: Optional pitch, yaw, and roll.
- actor_name: Unique runtime name.
- asset_path: Character Blueprint asset path.
- ensure_ai_controller: Immediately create and bind an AIController.

For free-fall initialization, spawn above the ground and call
character.set_physics.

### character.set_physics

Enables or disables a character Blueprint's physics/free-fall mode.

For the default character, this calls set_phy with 1 or 0.

The wrapper supports Blueprint parameters represented as either booleans or
numeric values.

Key inputs:

- actor: Exact character actor name.
- enabled: Enable or disable physics.
- function: Physics function name; defaults to set_phy.

Recommended sequence:

1. Spawn above clear ground.
2. Enable physics.
3. Wait for the Z position to stabilize.
4. Disable physics.
5. Begin navigation.

### character.ensure_ai_controller

Ensures that a Pawn has an AIController.

Behavior:

- Reuses an existing AIController.
- Replaces a non-AI controller when necessary.
- Otherwise spawns /Script/AIModule.AIController.
- Possesses the target Pawn.

Key inputs:

- actor: Pawn or Character name.
- controller_name: Name assigned to a newly spawned controller.

### character.get_controller

Inspects the controller currently possessing a Pawn.

The result includes:

- Whether a controller exists
- Controller object name
- Controller class
- Whether it is an AIController

Key input:

- actor: Pawn or Character name.

### character.nav_area

Gets or sets the character Blueprint's internal navigation sampling area.

This tool does not create or resize the world's Recast NavMesh.

Key inputs:

- actor: Character name.
- action: get or set.
- x_min, x_max
- y_min, y_max
- z_min, z_max

After setting the area, call get and confirm is_valid before random
navigation.

### character.navigate

Controls the default character's Blueprint navigation API.

Supported actions:

- to_goal: Navigate to a specific world-space destination.
- random: Start random navigation.
- sample_goal: Generate a navigation candidate without moving.
- stop: Stop AI movement.

Key inputs:

- actor: Character name.
- action: Navigation action.
- target: Destination for to_goal.
- range_cm: Search radius for random navigation.
- loop: Continue selecting random destinations.
- radius_min_cm: Minimum sample radius.
- radius_max_cm: Maximum sample radius.

Actual path following requires:

- A valid NavMesh
- A valid navigation area
- An AIController
- A reachable destination

### character.move_pulse

Calls the character Blueprint's set_move(rot, Speed) function.

Key inputs:

- actor: Character name.
- yaw_degrees: World-space movement heading.
- speed_cm_s: Movement speed in centimeters per second.

Repeat the tool at a controlled interval when the Blueprint requires
continuous movement commands.

Use character.movement_state to verify displacement.

### character.movement_state

Returns detailed locomotion diagnostics for an ACharacter.

The result includes:

- Actor location
- Velocity
- Movement mode
- Walking and falling state
- Maximum movement speed
- Current acceleration
- Pending movement input
- Last movement input
- Whether movement input is ignored
- Controller information

Use this tool when navigation or movement commands return successfully but
the character does not visibly move.