Examples: Answer questions about a PDF document - #132
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Load the European patent EP0666666B1 from its canonical URL, split each page into fragments, index them in `CrateDBVectorStore`, and query the result. Document retrieval over a PDF is the shape most people arrive looking for, and the existing vector example reads a plain text file. The specification is published in English, German and French, so the program asks the same question in each of the three, which the embeddings answer from one shared index. `pypdf` joins the test group because the suite executes every example.
f0f37c7 to
2340569
Compare
| fragments = [] | ||
| for page in loader.load(): | ||
| fragments += text_splitter.create_documents([page.page_content]) | ||
| return fragments |
There was a problem hiding this comment.
This loop drops all the metadata PyPDFLoader gives you. create_documents() builds fresh Documents from raw strings, so every one of the 102 fragments comes out with metadata == {}, so no source, no page, no page_label.
| fragments = [] | |
| for page in loader.load(): | |
| fragments += text_splitter.create_documents([page.page_content]) | |
| return fragments | |
| return text_splitter.split_documents(loader.load()) |
Mostly flagging this because it's an example, and "which page did this answer come from?" is the first thing anyone asks of a PDF RAG pipeline. Losing page makes that unanswerable. Bonus: three lines become one :)
| """ | ||
|
|
||
| # Define resource loader. | ||
| loader = PyPDFLoader(RESOURCE_URL) |
There was a problem hiding this comment.
PyPDFLoader fetches remote URLs through a bare requests.get(...) with no timeout. Since the test suite runs every example, a stalled CDN would hang the CI job until GitHub's 6-hour limit.
The other two network examples both use timeout=10 (vector_search.py and document_loader.py)
It's not a blocker but a simple fix could prevent future pipeline issues.
| # Embed each fragment, and load them into the vector store. | ||
| vector_store = CrateDBVectorStore.from_documents( | ||
| documents=documents, | ||
| embedding=OpenAIEmbeddings(), |
There was a problem hiding this comment.
from_documents() writes in langchain collection as default, it's not a problem on CI because it drops the DB on every test but locally, all examples goes on same collection (eg. vector_search.py) small fix can be good to show users adding custom collection.
| embedding=OpenAIEmbeddings(), | |
| embedding=OpenAIEmbeddings(), | |
| collection_name="pdf_example", |
2 participants
Closes #130. Supersedes the PDF half of #17.
What it does
Loads the European patent EP0666666B1 straight from its canonical URL, splits each page into 500-character fragments, indexes them in
CrateDBVectorStore, and runs three queries against the result. 13 pages become 102 fragments.The specification is published in English, German and French, so the three queries are the same question in each of those languages, answered from one shared index. That is a property of the document rather than decoration, and it shows something the plain-text example cannot.
Where it came from
The working code is c3cba55 under #17, not the later
examples-2branch, whereget_documents()was left mid-experiment with three stacked unreachable returns and a read of the PDF from the working directory.Rebuilt on current main from there: the docstring no longer claims to use
state_of_the_union.txtor to be about Hugging Face, the branch'spyproject.tomlstanza is dropped entirely (it pinnedlangchain-openai <0.3andpytest <9), and the script header asks for 3.10 rather than 3.9. Embeddings stay on OpenAI, so this costs CI one PDF fetch and nothing else. A key-free path is #131.Verification
The suite executes every file under
examples/, sopypdfjoins the test group. Running the program's ownget_documents()and store calls against acrate/crate:nightlycontainer, with only the embedding provider swapped out (no key on this machine), loads, splits, indexes and searches end to end.ruff check,ruff format --diffandmypy .are clean, andpdf.pynow behaves like the other OpenAI examples under pytest: it needs the key CI supplies.One thing to decide
PyPDFLoadercomes fromlangchain_community, which emits a sunset warning on import pointing at standalone integration packages. There is nolangchain-pypdfon PyPI to move to. Shipping here and revisiting seems right, but it is a choice rather than an oversight.