Make a summarizing reply bot

[mlissner]

Reply bots are a neat idea that I've seen in a few places. Basically, they sit around and wait for other bots to post things, and then automatically reply with some additional context.

These are fun because:

• If you like what the reply bot does, you can follow it and always see the original post with the reply threaded below.
• If you don't like the bot, you can block or mute it.
• If you think it's fine, you can click each post and the reply will be there for you to see, when you want to see it.

In this case, what I think we should do is create a bot on Bluesky that replies to all of our bots with an AI summary of the document.

A few thoughts:

1. The AI summaries might get expensive, so maybe they come out of the sponsorship budget. At a minimum, we should be able to turn them on or off for a particular bot.

2. We're working on AI summaries as part of a grant, so this could be an early exploration of that, or it could simply take the lessons from that and build on them (or it could just do a basic job, and that's fine too, it's a social media bot after all).

3. The bot could have a handle like @autosummarizer.bots.law.

Architecturally, the easiest way to do this is probably to use rq to create a summarizing task after making each post. The summarizer task could take the ID of the post, grab the post from Bluesky, download the PDF linked in the post (if available), and then summarize it. Alternatively, if we already have the link to the post, maybe we skip all that and just do the reply instantly (that'd be nicer, if we can!).

[warent]

Since the bot creating the original post is already integrated with the social media, has its own account, etc. How would we feel about decoupling summary functionality from bot/social functionality?

e.g.

1. Place summary functionality into its own service
2. Bot requests and receives a summary
3. Bot replies to its own thread

This has benefits like:

• Allowing other services to build on top of the summary feature
• Decreasing dependencies / complexity

The Summary service could be synchronous, or asynchronous by receiving a "context" which it would include in the reply

[mlissner]

Interesting. Are you imagining that it'd have it's own DB and everything, or that it'd be more like a microservice that would get the POST with the context, do it's thing, and then store in the DB we already have?

Architecture aside, I think it's worth using a different account for this so that muting/blocking, etc. work without having to mute/block the original bot.

[warent]

Are you imagining that it'd have it's own DB and everything, or that it'd be more like a microservice that would get the POST with the context, do it's thing, and then store in the DB we already have?

Either way is fine, I defer to you on that. It might be easier and safer if we spin up it's own DB, and in the future could do some migration/merge if you ever decided to. I usually use Postgres and Mongodb if either of those work for you.

Makes sense, a separate bot can still work with this and I'd be happy to help set it up. So the BigCases bot (and any other authorized bot) can talk to the AutoSummary bot, which calls the Summary service and posts the social media reply. Sound good?

[mlissner]

I'd say using the same DB (we use postgres) is easier and cheaper than standing up a new one. To do that, I think we'd just need a new column in the Post table, probably, and we might even be able to abuse the Channel table to host the details of this bot.

I think if we're using the same DB, a full-blown service probably isn't needed, if I'm understanding what you mean by that. Would a service be a micro-service via HTTP or is this more of, like, a java thing where a service is a fancy name for a class? Because if we're using the same DB, I think the flow could be:

• Get a webhook event from CourtListener
• Do a post (make thumbnails, etc)
• Enqueue a task in rq to reply in 10s.
• The task then:
  • uses whatever context it has to call out to Claude (or whatever);
  • saves the summary to the Post table;
  • replies to the original post.

---

An unrelated thought...we get updates from the court sometimes that are too long to fit into a post, and we use an image to hold the excess text. For example:

https://bsky.app/profile/bigcases.bots.law/post/3lipouxjot22p

We could probably repurpose that code for any summary that's longer than 180 characters. (Scope creep alert, I know.)

[warent]

When you're talking about the DB, do you mean the same DB that the Bot itself uses? i.e. https://github.com/freelawproject/bigcases2/blob/main/ARCHITECTURE.md#database

A big thing is that I want is to access summary data in separate apps as well. In an ideal world, Court Listener actually provides AI summary fields in its API responses--something I would definitely use.

If summaries only live within the context of a Bot, it seems like we wouldn't be able to easily utilize that data externally; it would be tightly coupled / locked down into the Bot (unless I'm misunderstanding.)

That's where the idea of a "Service" comes into this: a microservice / HTTP service (since it isn't likely to be embedded as part of Court Listener) which provides an API to generate and store summaries for generalized access, which can be used by a Bot and more.

Let me know if this makes sense and is still close to what you were envisioning / wanting!

---

As an aside, maybe another part of what we need to get aligned on is what exactly is meant by "Summary". In my mind, it means:

• Full summary: Full legal document(s) translated into an easily consumable format (a la news article, blog post, etc)
• Short summary: ~180 character headline version for social media

[warent]

(Apologies for the confusion, when you said "DB we already have" I was thinking you meant Court Listener's main database, not the Bot's database)

[warent]

I've put together a quick MVP of this concept here: https://github.com/warent/summary.law

The idea is that we can generate summaries based on case number / documents. For example:

curl -X POST -H "Content-Type: application/json" -d '{"pacerDocumentIds": ["045011178496", "045011178497"]}' http://localhost:3000/v1/summary/277129

POST request to the endpoint with the pacer case number and pacer document IDs.

Which results in this:
https://summary.law/api/v1/summary/277129

The idea now would be that a Bot would utilize this to fetch/generate document summaries.

[mlissner]

When you're talking about the DB, do you mean the same DB that the Bot itself uses?

Yes.

A big thing is that I want is to access summary data in separate apps as well. In an ideal world, Court Listener actually provides AI summary fields in its API responses--something I would definitely use.

Yeah, architecturally it's a bit weird to have the bot making AI summaries that it just posts into the social media ether, and in a few months we actually plan to start doing AI summaries of filings in CourtListener itself. I didn't want to lump that into this feature because once CL does something, it has to do it really well, and the critical path for that feature is lengthy:

1. Make a document categorizer that can say that a filing is a motion to dismiss, or a complaint, or something boring.

2. Using the new categorizations of the documents, decide which to summarize.

3. Develop custom AI prompts for different types of filings, so that you can get the gist of each based on what that category of document usually is about.

So that's going to take a bit of time.

If summaries only live within the context of a Bot, it seems like we wouldn't be able to easily utilize that data externally; it would be tightly coupled / locked down into the Bot (unless I'm misunderstanding.)

Yeah, I think getting the summaries out of bots.law would involve either making an API for it (kind of a big deal, and not a feature we really want on bots.law, I think) or scraping the summaries from the posts (kind of lame).

That's where the idea of a "Service" comes into this: a microservice / HTTP service (since it isn't likely to be embedded as part of Court Listener) which provides an API to generate and store summaries for generalized access, which can be used by a Bot and more.

I guess the worry/question I have on this is:

1. If we're relying on it in bots.law, might it go down, go away, etc? We've had bad luck with that in the past.

2. If the service is just making an API call to claude, it seems kind of unnecessary — like calling an API to call an API. Feels like bots.law should just call claude directly.

The MVP is cool and impressive though!

As an aside, maybe another part of what we need to get aligned on is what exactly is meant by "Summary". In my mind, it means:

Yeah, I hadn't really thought about that too much. I could see just a short one being generated for social media with a link to a long one (or in a thumbnail?) or I could see just a short one...I don't know, some room to play here, I guess.

---

Summing up (uh, no pun intended), my gut is that relying on a service for this is going to be less reliable for bots.law, so maybe it's not the way to go. Maybe the smarter approach is you run the/a reply bot in your own infra — You'd get the summaries you made that way, and FLP wouldn't worry about uptime or whatever. You'd also, clearly, be able to make it a lot faster than figuring out bots.law, Python, etc. And maybe down the road, when CL has fancy summaries, you could consider switching to them to save money, who knows?