Summarize actions to sentence + icon

[abrichr]

Feature request

When visualizing recordings, we would like to collapse each top level action into a one to two sentence description in a systematized way, ideally with an icon (eg prompt for a segment).

Motivation

Understanding recordings, humans and models.

[abrichr]

/bounty $100

[algora-pbc]

💎 $100 bounty • OpenAdaptAI

Steps to solve:

1. Start working: Comment /attempt #741 with your implementation plan
2. Submit work: Create a pull request including /claim #741 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to OpenAdaptAI/OpenAdapt!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🔴 @IvanLauLinTiong	Jun 16, 2024, 6:46:11 PM	WIP
🟢 @onyedikachi-david	Jul 5, 2024, 7:19:13 PM	WIP
🟢 @vishwamartur	Nov 17, 2024, 3:36:41 PM	#922

[IvanLauLinTiong]

/attempt #741

Options

• Cancel my attempt

[shashank40]

@abrichr can you please explain the issue a bit more in detail?

[abrichr]

@shashank40 from the README, here is a screenshot of the dashboard:

Here is a screenshot of Microsoft's recent updates to PowerAutomate:

Notice the one-sentence action summaries. We would like to implement the same type of thing, i.e. convert an action dict, window dict, and screenshot into a sentence and an icon. You can use the prompt adapter for this, e.g.

summary = prompt_adapter.prompt("Please provide a one sentence summary of the following contents: ...")

I expect the prompt should live in a .j2 file. You will need another prompt to generate an icon.

[onyedikachi-david]

/attempt #741

Algora profile	Completed bounties	Tech	Active attempts	Options
@onyedikachi-david	2 bounties from 1 project	JavaScript, Shell		Cancel attempt

[vishwamartur]

/attempt #741

Options

• Cancel my attempt