[hma] Inconsistent information on lookups

[Pezmc]

Request: Add hash data to /m/lookup or match data to /h/raw_lookup

In the current API design:

• /h/hash returns raw hashes:

{
  "pdq": "facd8bcb2a49bcebdec1985298d5fe84bcd006c187c598c720c3c087b3fdb318",
  "md5": "d41d8cd98f00b204e9800998ecf8427e",
  "phash": "a1b2c3d4e5f6789a"
}

• /m/raw_lookup returns only match distances:

{
  "matches": [0]
}

• /m/lookup returns match and bank info - but not the hash values themselves:

{
  "EXAMPLE_BANK": {
    "pdq": [
      {
        "bank_content_id": 123,
        "distance": "0"
      },
      {
        "bank_content_id": 456,
        "distance": "5"
      }
    ],
    "md5": [
      {
        "bank_content_id": 789,
        "distance": "0"
      }
    ]
  }
}

---

Problem

In our deployment, we’re following the HMA model where hasher, matcher, and curator run as separate services. This breaks compatibility with /h/lookup, which expects both hashing and matching on the same instance.

To persist both the hashes and the bank matches for reporting in our app, we’re forced to:

1. Call /h/hash to get the hashes.
2. Call /m/lookup to get the matches
3. Enable hashing in our matcher to enable /m/lookup, which defeats service separation.

---

Feature Request

We’d like to be able to fully decouple hasher and matcher services, and still retrieve both hashes and matches. I propose two possible solutions:

• Option A: Extend /m/lookup to include the generated hash values alongside the match data.

  • Pros: Keeps all relevant info in one call.
  • Cons: Couples hashing and matching.

• Option B: Extend /h/raw_lookup to include the same bank match info as /m/lookup.

  • Pros: Enables strict service separation - hasher hashes, matcher matches.
  • Cons: Modifying an existing endpoint format

---

Summary

Right now, the only way to get both hashes and bank match data is via /m/lookup, which implicitly requires combining the hasher and matcher roles.

We’d appreciate a way to retrieve both hash and match info while preserving the ability to scale the hasher and matcher independently.

[Dcallies]

Hey @Pezmc , thanks for the detailed feature request. The intent is for it to be possible to fully decouple hashing and matching instances (many platforms may move the hashing code directly to the web servers, since it tends to be CPU heavy, and hashes are much smaller than the full media).

/m/raw_lookup should be returning the banked_content_id as well, and not just distance, sounds like a bug or mistake on our end.

We had another request that was very similar: #1681, and I think this specific request in an offline discussion.

The only reason why we wouldn't include the hash by default is that the indices themselves don't always store the original hash, and so you need to look it up by bank_content_id in a db call.

I propose we do the following:

1. Ensure raw_lookup also includes the banked_content_id... otherwise it's kind of useless
2. Add a parameter that signals the caller of a match wants the matching hash back in the same request, which saves one rountdrip calling
3. Ensure that a GET /bank_content_id exists that returns the same information (haven't read the code to check yet).

[julietshen]

@Dcallies @Pezmc I'll try to piece it together from the issue trail, but let me know what still needs to be done to solve this