[FEATURE REQUEST] simple integration with python

[jakobhuss]

Hi,

I wish there was a simple way to integrate kingfisher in python (and other languages). Perhaps using python bindings or if kingfisher could have a json api where I could post files/text for scanning.

I tried running a separate process on a small file:

time kingfisher scan - -f jsonl < .env

On my machine this takes ~3s. My guess is that the startup has a lot of overhead which makes it infeasible to use that method for fast scanning of many text strings/files from python.

Thank you for a great piece of software

[mickgmdb]

@jakobhuss - Thanks for the kind words and for the detailed report!

You’re right about the ~2-3s startup cost. On each run Kingfisher compiles all active patterns into a Vectorscan/Hyperscan database before scanning anything, and that one-time setup dominates runtime for small inputs. Once it's running, the actual scanning is extremely fast, but the current cli design pays that cost every process invocation.

I’m looking into a few options to make this easier to integrate from Python/other languages: a long-running mode or JSON API you can send text/files to, and better reuse of the compiled database (instead of rebuilding it on every run) to cut the startup delay. In the meantime, the best workaround is to keep a single Kingfisher process alive (or container running) and feed multiple scans through it, rather than spawning a new process per file/string.

Thank you for the feature request --- I am going to test out a few solutions and determine which will offer the best performance!

[mickgmdb]

@jakobhuss — one thing you can do today is scan a large number of files in a single run. This is where Kingfisher really shines. It builds a vectorscan database up front (which takes ~2–3 seconds), and once that’s done, scanning is often an order of magnitude faster than other regex engines and secret-scanning tools.

In practice, this means you can scan hundreds of gigabytes very quickly. The tradeoff is that if you’re scanning a single ~10KB file, you’ll still pay that ~2 second startup cost.

[jakobhuss]

I was thinking to build this in to a api for LLM prompts so i could protect against sending secrets directly to the LLM providers. I know that it is very fast when scanning on disk.

[mickgmdb]

@jakobhuss - Let me think about this. I've got some ideas I want to try out that might help fill this gap.