Copilot: Collect metadata and add to message before sending

[profpylons]

Allow copilot clients to send contextual information with a message about the state of the client.

eg.

        (window as any).setChainlitMetadata({
          questionnaireState: {
            questionId,
            questionnaire,
          }
        });

The function setChainlitMetadata() is available before the widget is mounted in order that the first question asked has the state/metadata available immediately. This bypasses the difficulty of listening for when the widget is mounted.

In the backend this can read out and used to format queries to the downstream LLM

@cl.on_message
async def main(message: cl.Message):

    if cl.context.session.client_type == "copilot":
        print("Copilot detected.")

    if message.metadata:
        print(f"Message type: '{message.type}', sent with metadata: '{message.metadata}'")
        # Do something with different prompts based on the question passed

[profpylons]

FYI previously we'd been using system_message as recommended by the docs.

@cl.on_message
async def main(message: cl.Message):
    if cl.context.session.client_type == "copilot":
        print("Copilot detected.")

    # Check for received system message that tells us to use guidance
    # from the chainlit widget copilot. This just sets the session state
    # and awaits a user message to process the query.
    if message.type == "system_message":
        # Store guidance metadata in the user session
        cl.user_session.set(SESS_CLIENT_STATE, message.metadata)
        print(f"Setting Client App State via system message: {message.metadata}")

        # Don't show the message to the user, this is a system message
        await message.remove()
        return

    # Determine guidance source and extract questionnaire information
    print(f"Fetching Client App State from user session: {cl.user_session.get(SESS_CLIENT_STATE)}")
    copilot_guidance = cl.user_session.get(SESS_CLIENT_STATE)

    # Update the prompts for different questions

But as we could not listen for amount event this meant the first question asked by a user could not be preceded by setting this metadata.

[asvishnyakov]

@profpylons Could you elaborate a bit? I think the docs are talking about window_message, which is a way to send data through an iframe, not a system message.

---

https://docs.chainlit.io/api-reference/window-message

[profpylons]

With an iFrame, yes, that seems to be the way. Using the copilot. the docs recommend system_message.

There are two drawbacks to this message sending.

1. The sendChainlitMessage() method is not responsive until the popover has been loaded. So the copilot cannot be updated with the state of the client UI whilst the widget is collapsed. Opening the widget does not fire an event to preload the state from the client UI, so the first message sent to the chainlit backend is always unaware of the state.
2. In any case, the Copilot UI does not distinguish system_message from user_message and renders the content before firing the backend event. So when sending state information, it is necessary to update the system_message with message.remove() after receipt causing noticeable flicker in the widget. (see example code above).

I considered adding an event to fire when the copilot is mounted. But this just requires extra plumbing and tracking in the client app.

In any case, I think it is still a good idea to send the state in metadata rather than the with system_message to prevent the UI flicker and reduce the need for strange session management hula-hoops to track the value in the backend.

This led me to here: Keep the necessary state locally in the widget and pass it with each user_message from the Copilot. It leads to:

1. Simpler & more stateless backend (not tracking client state in session variables)
2. No additional event listener in the client to pre-seed the widget after mounting the popover. Just send every state change.
3. Removal of the UI flicker

[github-actions]

This PR is stale because it has been open for 14 days with no activity.