feat: Chat completion supports multi-modal input (images and text). Text-to-speech supports streaming. (AEGHB-963)

[icyqwq]

Text-to-speech supports streaming.

Description

1. Chat completion supports multi-modal input (images and text).
2. Text-to-speech supports streaming.
3. 🚨 Includes a breaking change: the message function in chat completion now includes a new type parameter to specify the type of multi-modal input.

Related

Testing

Chat completion

// text
result = chatCompletion->message(chatCompletion, "text", "tell me a joke", false);
// image
result = chatCompletion->message(chatCompletion, "image_url", "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg", false);

Streamed tts

audioSpeech->speechStream(audioSpeech, "Sample text", [](const uint8_t *data, size_t len) {
    if (len == 0) {
        return;
    }
    size_t bytes_written;
    i2s_write(I2S_NUM, data, len, &bytes_written, portMAX_DELAY);
    i2s_zero_dma_buffer(I2S_NUM);
  });

---

Checklist

Before submitting a Pull Request, please ensure the following:

• 🚨 This PR does not introduce breaking changes.
• All CI checks (GH Actions) pass.
• [ x ] Documentation is updated as needed.
• [ x ] Tests are updated or added as necessary.
• [ x ] Code is well-commented, especially in complex areas.
• [ x ] Git history is clean — commits are squashed to the minimum necessary.

[CLAassistant]

All committers have signed the CLA.

[github-actions]

Warnings

⚠️

Some issues found for the commit messages in this PR: the commit message "feat: Chat completion supports multi-modal input (images and text).": summary should not end with a period (full stop) body must have leading blank line Please fix these commit messages - here are some basic tips: follow Conventional Commits style correct format of commit message should be: <type/action>(<scope/component>): <summary>, for example fix(esp32): Fixed startup timeout issue allowed types are: change,ci,docs,feat,fix,refactor,remove,revert,test sufficiently descriptive message summary should be between 20 to 72 characters and start with upper case letter avoid Jira references in commit messages (unavailable/irrelevant for our customers) TIP: Install pre-commit hooks and run this check when committing (uses the Conventional Precommit Linter).

👋 Hello icyqwq, we appreciate your contribution to this project!

---

Click to see more instructions ...

This automated output is generated by the PR linter DangerJS, which checks if your Pull Request meets the project's requirements and helps you fix potential issues.

DangerJS is triggered with each push event to a Pull Request and modify the contents of this comment.

Please consider the following:
- Danger mainly focuses on the PR structure and formatting and can't understand the meaning behind your code or changes.
- Danger is not a substitute for human code reviews; it's still important to request a code review from your colleagues.
- Resolve all warnings (⚠️ ) before requesting a review from human reviewers - they will appreciate it.
- To manually retry these Danger checks, please navigate to the Actions tab and re-run last Danger workflow.

Review and merge process you can expect ...

We do welcome contributions in the form of bug reports, feature requests and pull requests.

1. An internal issue has been created for the PR, we assign it to the relevant engineer.
2. They review the PR and either approve it or ask you for changes or clarifications.
3. Once the GitHub PR is approved we do the final review, collect approvals from core owners and make sure all the automated tests are passing.
- At this point we may do some adjustments to the proposed change, or extend it by adding tests or documentation.
4. If the change is approved and passes the tests it is merged into the default branch.

Generated by 🚫 dangerJS against fbe2376

[lijunru-hub]

@leeebo Please add tag openai

[leeebo]

Hi @icyqwq thanks for your contribution, we will review it ASAP!

[Copilot]

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

[Copilot]

Copilot wasn't able to review any files in this pull request.

Files not reviewed (3)

• components/openai/OpenAI.c: Language not supported
• components/openai/include/OpenAI.h: Language not supported
• components/openai/test_apps/main/test_openai.c: Language not supported

[lijunru-hub]

@icyqwq Since the OPENAI component has been modified, could you rebase onto the latest esp-iot-solution? This will allow us to proceed with this MR.

[lijunru-hub]

Additionally, it seems that the image URL string is "input_image". Could you double-check?

[leeebo]

sha=fbe2376255fd8e14987718e93fcfc3ad260e332d