[Store] Load, split and normalize documents

[chr-hertel]

A common use-case for populating a vector store is to load the content as documents from a source and manipulate them while doing so, e.g. splitting into multiple chunks, normalizing the language, or summarizing.

I'd guess that In the end the LoaderInterface should spit out a iterable<TextDocument> that is then handed over to the Embedder (rename to Indexer?)

[chr-hertel]

While thinking about it, from a design point of view, I'd like to have a pipeline pattern:

$indexer = new Indexer($platform, $model, $store);
$loader = new LoaderImplementation(...);
$pipeline = new TransformationPipeline(
    new TextNormalizer($languagePattern),
    new TextSplitter(length: 1000, overlap: 50),
    new Chunker(chunk: 100, sleep: 10),
);

$indexer->index($pipeline($loader));

[OskarStark]

Super flexible, nice 👍🏻

[chr-hertel]

Parts of it where done with #57, but more Loader and Transformer implementations + docs would be great