Unredacted text

[valimikayilov]

Hi,

thanks for working on something like this it is a great idea,

I was curious why is it asking directly llm, instead of getting the logprob of each token and using a topk set to 1, LLM can find the next word?

KR.

[P1ayer-1]

This was my exact thought after watching the video. I was hoping someone else suggested it.

To add to this: Instead of doing top_k = 1, I think it would work better to sample multiple candidate tokens and then score them using the probability of the known continuation.

For example, if the prompt is:

He travelled from California on a ____ to the Bahamas

We already know the tokens after the blank. So for each candidate token we could compute the log-probability of the following tokens given that candidate, and pick the token that maximizes the joint likelihood of the full sequence.

In other words, choose the word that makes the entire known sequence most probable, rather than just the next token.

I’m on mobile right now, sorry if this isn’t that descriptive.

[rhowardstone]

Wouldn't that often require inordinate context though? Sometimes the answer to what's likely underneath is just not in that document, sometimes it's in others, or mixed between 1,000+ pages in a single document. Maybe I'm not understanding? This sounds clever

EDIT: Claude says:

The interesting recoveries aren't local - they require cross-referencing other documents in the corpus. A logprob approach would work for formulaic fills but miss the cases where the redacted content is only identifiable from external context.

[P1ayer-1]

I think we might be talking about slightly different failure modes.

The continuation scoring idea I mentioned is only meant to improve local reconstruction. It doesn't require additional context; it just evaluates which candidate makes the existing right-hand tokens most probable.

Building on the top-k idea mentioned earlier, one option could be to generate several candidate tokens and then re-rank them based on how well they predict the known continuation.

Instead of selecting the token with the highest P(w | left_context), you'd score candidates by something like:

P(w | left_context) * P(right_context | left_context + w)

In other words, choose the candidate that makes the entire known sequence most likely, not just the next token.

That should help for cases where the correct word is locally constrained by grammar or phrasing.

I agree it wouldn’t solve the cross-document cases you mentioned, though. Those would need retrieval from the wider corpus.

[rhowardstone]

Interesting, I wonder if you'd manage some kind of wider-corpus abilities by adding some context to this before the page/doc text? For me getting the text with exactly the locations of the "[redacted]" markers in relation to the extracted text is main problem in trying to test this. I have a bad version of it.. I suppose it could auto-render for people in the browser, then they could alter the redaction text's locations to clean it up manually, and it would re-render with a shelf of candidate words that can appear underneath the bars, with confidence levels shown for top K candidates? Or, do you know of any good models that can run on modest hardware to get this done?