feat: reuse doc/chunk/emb results, augment chunks, web service

[kemingy]

• chunk augmentation for context & query
• doc summarization
• more semantic chunker
• prepare for dense/sparse/keyword embedding (not supported by vectorchord)]
• reuse the doc/chunk/emb
• augment with chunk-context/chunk-query/doc-summary
• web service

[Copilot]

Copilot reviewed 6 out of 10 changed files in this pull request and generated no comments.

Files not reviewed (4)

• vechord/config.py: Evaluated as low risk
• vechord/extract.py: Evaluated as low risk
• vechord/pipeline.py: Evaluated as low risk
• test.py: Evaluated as low risk

Comments suppressed due to low confidence (4)

vechord/augment.py:49

• The method name 'augment_chunks' should be renamed to 'augment_context' to match the abstract method name in 'BaseAugmenter'.

def augment_chunks(self, chunks: list[str]) -> list[str]:

vechord/augment.py:59

• The method name 'augment_queries' should be renamed to 'augment_query' to match the abstract method name in 'BaseAugmenter'.

def augment_queries(self, chunks: list[str]) -> list[str]:

vechord/embedding.py:18

• [nitpick] The method name 'vectorize_doc' might be confusing. Consider renaming it to 'vectorize' for consistency with the original method name.

def vectorize_doc(self, text: str) -> np.ndarray:

vechord/embedding.py:21

• Ensure that the new method 'vectorize_query' is covered by tests to validate its functionality.

def vectorize_query(self, text: str) -> np.ndarray:

[Copilot]

Copilot reviewed 6 out of 10 changed files in this pull request and generated no comments.

Files not reviewed (4)

• vechord/config.py: Evaluated as low risk
• vechord/extract.py: Evaluated as low risk
• vechord/pipeline.py: Evaluated as low risk
• test.py: Evaluated as low risk

[VoVAllen]

What does current interface look like?

[kemingy]

What does current interface look like?

Almost the same as before. Although the interface can support dense/sparse/keywords, vectorchord doesn't.

[VoVAllen]

Can you add an example.py and we merge this PR first? @kemingy

[kemingy]

Can you add an example.py and we merge this PR first? @kemingy

Sure.

[kemingy]

@VoVAllen shall we merge this PR?

[VoVAllen]

Let's merge it first.