[eaa0ae3] by @thiswillbeyourgithub, 2 hours ago:
enh: pass the version of wdoc to langfuse too
Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/llm.py
wdoc/utils/misc.py
wdoc/wdoc.py

[b72a101] by @thiswillbeyourgithub, 2 hours ago:
enh: add a unique langfuse session_id by launch of wdoce
Signed-off-by: thiswillbeyourgithub
[email protected]

wdoc/utils/llm.py

[6d62ed4] by @thiswillbeyourgithub, 2 hours ago:
enh: use langchain's callback for langfuse instead of litellm's
Signed-off-by: thiswillbeyourgithub
[email protected]

README.md
wdoc/utils/llm.py
wdoc/wdoc.py

Contributors

thiswillbeyourgithub

Assets 2

10 Oct 23:00

thiswillbeyourgithub

2.3.1

71c64ca

Release 2.3.1

Commits since last release

[cd131b4] by @thiswillbeyourgithub, 12 minutes ago:
fix: wrong attribute name used for PersistDict
Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/customs/compressed_embeddings_cache.py

[e8d8d55] by @thiswillbeyourgithub, 6 hours ago:
docs: add FAQ question to rate limiting
Signed-off-by: thiswillbeyourgithub [email protected]

README.md

[1e1446e] by @thiswillbeyourgithub, 6 hours ago:
enh: add yet another failsafe for private mode
Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

[f48e879] by @thiswillbeyourgithub, 6 hours ago:
docs: mention langfuse as a feature
Signed-off-by: thiswillbeyourgithub [email protected]

README.md

[5c8b6aa] by @thiswillbeyourgithub, 6 hours ago:
fix: langfuse import
Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

[89b9532] by @thiswillbeyourgithub, 8 hours ago:
new: force using litellm backend if langfuse enabled and not private
Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/llm.py
wdoc/wdoc.py

[f9af1e6] by @thiswillbeyourgithub, 8 hours ago:
enh: add check that private == is_private
Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/llm.py

[fd41ef8] by @thiswillbeyourgithub, 8 hours ago:
new: enable langfuse callbacks if the right env variables are provided
Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/wdoc.py

[0e40624] by @thiswillbeyourgithub, 8 hours ago:
enh: add a flags is_private
Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/flags.py

[2ae4451] by @thiswillbeyourgithub, 8 hours ago:
feat: --private now disables langfuse callbacks just in case
Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/docs/USAGE.md
wdoc/utils/llm.py
wdoc/wdoc.py

[f25c007] by @thiswillbeyourgithub, 8 hours ago:
new: setup metadata tags for traces to use with langfuse etc
Signed-off-by: thiswillbeyourgithub [email protected]

wdoc/utils/llm.py
wdoc/wdoc.py

[c8b3038] by @thiswillbeyourgithub, 26 hours ago:
forgot to bump version of thefiche

scripts/TheFiche/TheFiche.py

[d194c73] by @thiswillbeyourgithub, 29 hours ago:
enh: better formulation of source_id

wdoc/utils/prompts.py
wdoc/utils/tasks/query.py

[48968ea] by @thiswillbeyourgithub, 29 hours ago:
enh: combiner prompt

wdoc/utils/prompts.py

[9b5e706] by @thiswillbeyourgithub, 29 hours ago:
fix: logseq lines starting with > must be escaped

scripts/TheFiche/TheFiche.py

[1167bdb] by @thiswillbeyourgithub, 34 hours ago:
fix: boundaries of sup_tags and sub_tags

scripts/TheFiche/TheFiche.py

[f55219f] by @thiswillbeyourgithub, 8 hours ago:
docs: mention that dollar_limit only applies to summaries and embeddings, not queries
Signed-off-by: thiswillbeyourgithub [email protected]

README.md
wdoc/docs/USAGE.md

Contributors

thiswillbeyourgithub

Assets 2

09 Oct 01:37

thiswillbeyourgithub

2.3.0

d99bd0a

Release 2.3.0

Commits since last release

[d99bd0a] by @thiswillbeyourgithub, 4 minutes ago:
bump version 2.2.1 -> 2.3.0

bumpver.toml
setup.py
wdoc/wdoc.py

[4a8f0a9] by @thiswillbeyourgithub, 44 minutes ago:
enh: updated shell completions

shell_completions/wdoc_completion.bash
shell_completions/wdoc_completion.fish
shell_completions/wdoc_completion.zsh

[f445717] by @thiswillbeyourgithub, 44 minutes ago:
fix: onlinepdfloader is slow slow its better to just try downloading the file first

wdoc/docs/USAGE.md
wdoc/utils/loaders.py

[09202fd] by @thiswillbeyourgithub, 48 minutes ago:
fix: typo

wdoc/utils/embeddings.py

[8fd0682] by @thiswillbeyourgithub, 3 hours ago:
feat: add argument --oneoff to avoid prompting continuously

wdoc/docs/USAGE.md
wdoc/wdoc.py

[4137319] by @thiswillbeyourgithub, 3 hours ago:
prompt: tell each llm the name of the others

wdoc/utils/prompts.py

[e15c736] by @thiswillbeyourgithub, 4 hours ago:
move my sql_dict code to its own library
https://github.com/thiswillbeyourgithub/PersistDict

setup.py
wdoc/utils/customs/compressed_embeddings_cache.py
wdoc/utils/customs/fix_llm_caching.py
wdoc/utils/customs/sql_dict.py

[06f9420] by @thiswillbeyourgithub, 4 hours ago:
gitignore

.gitignore

[8a234ac] by @thiswillbeyourgithub, 5 hours ago:
remove docstring

wdoc/utils/customs/sql_dict.py

[75d43e1] by @thiswillbeyourgithub, 5 hours ago:
enh: pass verbose argument to the custom caching

wdoc/utils/customs/compressed_embeddings_cache.py
wdoc/utils/customs/fix_llm_caching.py
wdoc/utils/embeddings.py
wdoc/wdoc.py

[425e699] by @thiswillbeyourgithub, 6 hours ago:
update gitignore

.gitignore

[12241c4] by @thiswillbeyourgithub, 6 hours ago:
fix: forbid to use call if setitem was used

wdoc/utils/customs/sql_dict.py

[7db4387] by @thiswillbeyourgithub, 6 hours ago:
better documentation

wdoc/utils/customs/sql_dict.py

[9e74454] by @thiswillbeyourgithub, 6 hours ago:
rename example folder to scripts

README.md
scripts/AnkiFiltered/AnkiFilteredDeckCreator.py
scripts/NtfySummarizer/NtfySummarizer.py
scripts/NtfySummarizer/README.md
scripts/NtfySummarizer/TOPICS.EXAMPLE
scripts/NtfySummarizer/script.sh
scripts/TheFiche/README.md
scripts/TheFiche/TheFiche.py

[633a222] by @thiswillbeyourgithub, 6 hours ago:
typo

README.md

[debbf21] by @thiswillbeyourgithub, 6 hours ago:
added to the readme

README.md

[ba68b44] by @thiswillbeyourgithub, 6 hours ago:
better code to reset the cache if another python process modified it

wdoc/utils/customs/sql_dict.py

[aa083b4] by @thiswillbeyourgithub, 7 hours ago:
minor: password management

wdoc/utils/customs/sql_dict.py

[782b200] by @thiswillbeyourgithub, 8 hours ago:
upadte code to use my sql_dict

wdoc/utils/customs/compressed_embeddings_cache.py
wdoc/utils/customs/fix_llm_caching.py
wdoc/utils/embeddings.py

[5e03fb5] by @thiswillbeyourgithub, 8 hours ago:
new: might as well create my own sql_dict for fun today

wdoc/utils/customs/sql_dict.py

[9f8b4fd] by @thiswillbeyourgithub, 8 hours ago:
reduce concurrency to 15

wdoc/docs/USAGE.md
wdoc/utils/env.py

[07bf694] by @thiswillbeyourgithub (aider), 23 hours ago:
feat: Add timeout argument to SQLite connections

wdoc/utils/customs/fix_llm_caching.py

[e42f9c4] by @thiswillbeyourgithub (aider), 23 hours ago:
feat: add SQLITE3_CONNECTION_TIMEOUT variable with default of 30

wdoc/utils/customs/fix_llm_caching.py

[c2ccc6d] by @thiswillbeyourgithub, 26 hours ago:
set default max concurrency for llms to 20

wdoc/docs/USAGE.md
wdoc/utils/env.py

[dc6b162] by @thiswillbeyourgithub, 29 hours ago:
fix: forgot to check if max_top_k was set, crashing in some cases

wdoc/utils/tasks/query.py
wdoc/wdoc.py

[670bb8b] by @thiswillbeyourgithub, 34 hours ago:
minor: bump version

wdoc/utils/customs/fix_llm_caching.py

[4ab313f] by @thiswillbeyourgithub, 35 hours ago:
fix: add a maybe needed commits

wdoc/utils/customs/fix_llm_caching.py

[a40a100] by @thiswillbeyourgithub, 35 hours ago:
enh: add vacuum and compression

wdoc/utils/customs/fix_llm_caching.py

[349ba90] by @thiswillbeyourgithub, 35 hours ago:
minor: add a method to get the columns in the db

wdoc/utils/customs/fix_llm_caching.py

[52cc97a] by @thiswillbeyourgithub, 35 hours ago:
minor: rename cache to storage

wdoc/utils/customs/fix_llm_caching.py

[8cca75d] by @thiswillbeyourgithub, 2 days ago:
untested: add expiration date to the sql cache

wdoc/utils/customs/fix_llm_caching.py
wdoc/utils/embeddings.py

[b409642] by @thiswillbeyourgithub, 2 days ago:
enh: add timestamp to the sql cache

wdoc/utils/customs/fix_llm_caching.py

[615c967] by @thiswillbeyourgithub, 2 days ago:
minor

examples/TheFiche/TheFiche.py

[b26694a] by @thiswillbeyourgithub, 2 days ago:
fix: modify md to always add a leading dash before ints in enumeration
this avoids issues with markdown

examples/TheFiche/TheFiche.py

[ef30912] by @thiswillbeyourgithub, 2 days ago:
untested: rewrite how sqlite caching is done to reduce memory footprint by using it like a dict basically

wdoc/utils/customs/compressed_embeddings_cache.py
wdoc/utils/customs/fix_llm_caching.py

[6876ef2] by @thiswillbeyourgithub, 2 days ago:
enh: make all instance of the same db use the same cache

wdoc/utils/customs/fix_llm_caching.py

[1e0cb6c] by @thiswillbeyourgithub, 2 days ago:
new: print costs when using import_mode

wdoc/wdoc.py

[f6c6b6d] by @thiswillbeyourgithub, 2 days ago:
enh: display the number of cid and nid found per document, to help troubleshoot issues

examples/AnkiFiltered/AnkiFilteredDeckCreator.py

[16da77f] by @thiswillbeyourgithub, 2 days ago:
fix: in some situation not all nids were returned

examples/AnkiFiltered/AnkiFilteredDeckCreator.py

[9aac581] by @thiswillbeyourgithub, 2 days ago:
fix: must return int not str

examples/AnkiFiltered/AnkiFilteredDeckCreator.py

[2e89612] by @thiswillbeyourgithub, 2 days ago:
tell the rephraser to take into account instructions to the other llm

wdoc/utils/prompts.py

[cd665a9] by @thiswillbeyourgithub, 2 days ago:
typo

wdoc/utils/prompts.py
wdoc/utils/tasks/query.py

[dcd2654] by @thiswillbeyourgithub, 2 days ago:
untested: make the compressed embeddings cache actually use the sqlite3 backend

wdoc/utils/customs/compressed_embeddings_cache.py
wdoc/utils/embeddings.py

[094d762] by @thiswillbeyourgithub, 2 days ago:
enh: when using joblib only send the relevant batch

wdoc/utils/embeddings.py

[4fdaec0] by @thiswillbeyourgithub, 2 days ago:
enh: use pickle instead of dill because its much slower

wdoc/utils/customs/fix_llm_caching.py

[b54cbe6] by @thiswillbeyourgithub, 2 days ago:
enh: add BEGIN execute

wdoc/utils/customs/fix_llm_caching.py

[501901a] by @thiswillbeyourgithub, 2 days ago:
enh: always run .clear()

wdoc/utils/customs/fix_llm_caching.py

[5a63f3e] by @thiswillbeyourgithub, 2 days ago:
enh: use try finally block to close the connection

wdoc/utils/customs/fix_llm_caching.py

[52edeb3] by @thiswillbeyourgithub, 2 days ago:
enh: set sqlite3's check_same_thred to False now

wdoc/utils/customs/fix_llm_caching.py

[2d8aa9c] by @thiswillbeyourgithub, 2 days ago:
enh: keep locks opened less time

wdoc/utils/customs/fix_llm_caching.py

[9c85df8] by @thiswillbeyourgithub, 3 days ago:
make it easy to turn on or off check_same_thread

wdoc/utils/customs/fix_llm_caching.py

[cd20832] by @thiswillbeyourgithub, 3 days ago:
fix: chmod dir was not applied if dir already existed

wdoc/utils/customs/compressed_embeddings_cache.py

[c801061] by @thiswillbeyourgithub, 3 days ago:
fix: regex check arg has to be checked outside the memoize

wdoc/utils/customs/compressed_embeddings_cache.py

[2c5a3a3] by @thiswillbeyourgithub, 3 days ago:
enh: removed function _get_full_path that is not needed anymore

wdoc/utils/customs/compressed_embeddings_cache.py

[b180eef] by @thiswillbeyourgithub, 3 days ago:
enh: no need to create the directory every time we set a key

wdoc/utils/customs/compressed_embeddings_cache.py

[e6c9186] by @thiswillbeyourgithub, 3 days ago:
enh: add an arg to disable regex checking for keys

wdoc/utils/customs/compressed_embeddings_cache.py

[aa2cc87] by @thiswillbeyourgithub, 3 days ago:
enh: memoized hash checker for keys

wdoc/utils/customs/compressed_embeddings_cache.py

[2360cac] by @thiswillbeyourgithub, 3 days ago:
enh: no more try except as they are slow in python and not needed now that we check the db _z

wdoc/utils/customs/compressed_embeddings_cache.py

[9cde74c] by @thiswillbeyourgithub, 3 days ago:
enh: precompile regex check for hash keys

wdoc/utils/customs/compressed_embeddings_cache.py

[f527cdc] by @thiswillbeyourgithub, 3 days ago:
add versionning for db to anticipate breaking changes

wdoc/utils/customs/compressed_embeddings_cache.py
wdoc/utils/customs/fix_llm_caching.py

[44e7c4c] by @thiswillbeyourgithub, 3 days ago:
minor

wdoc/utils/customs/fix_llm_caching.py

[8b4e9fe] by @thiswillbeyourgithub, 3 days ago:
untested: remove unused kwargs in self.clear() that were not used as kwargs are slow

wdoc/utils/customs/fix_llm_caching.py

[636e62c] by @thiswillbeyourgithub, 3 days ago:
fix: use a threading lock for caching databases

wdoc/utils/customs/fix_llm_caching.py

[f9b47d0] by @thiswillbeyourgithub, 3 days ago:
rename a chain for clarificaiton

wdoc/wdoc.py

[224eaba] by @thiswillbeyourgithub, 3 days ago:
mention each new name in the readme

README.md

[84f1ba0] by @thiswillbeyourgithub, 3 days ago:
prompts: give name to each model

wdoc/utils/prompts.py

[e5ac221] by @thiswillbeyourgithub, 3 days ago:
prompts: tell to add [doubtful] source if really unsure

wdoc/utils/prompts.py

[fdfd966] by @thiswillbeyourgithub, 3 days ago:
fix: wdoc is...

Contributors

thiswillbeyourgithub

Assets 2

04 Oct 22:15

thiswillbeyourgithub

2.2.1

8be37ae

Release 2.2.1

Commits since last release

[d42f0b5] bump version 2.2.0 -> 2.2.1
[5e66cd3] enh: eval llm now applies a threshold
[201c55e] fix: forgot to add the progress bars for task=search
[c961386] example: ankifilteredeck: new: bump version
[e5f2062] example: ankifilteredeck: new: by default set the LLM to the strong llm
[32bb450] new: retry with cache disabled if fails
[05c705d] fix: actually use caching inside the evaluate function just in case for partial results
[930b18d] example: ankifilteredeck: new: allow setting a task
[e150aeb] fix: auto increasing top_k could fail in some cases
[a154260] docs: mention the better way to call parse_file
[f0e2fc2] new: turn 'wdoc parse' or 'wdoc parse_file' into 'wdoc_parse_file'
[020dfe4] example: ankifiltereddeci: bump version to 1.0
[f0b3fb9] example: ankifiltereddeci: remove top_k arg

Assets 2

04 Oct 18:24

thiswillbeyourgithub

2.2.0

0894463

Release 2.2.0

Commits since last release

[0894463] bump version 2.1.8 -> 2.2.0
[78ab333] better import
[4e490a4] rename WDoc to wdoc
[d72b238] rename summary.py to summarize.py
[1a1437e] docs: mention that faiss score fn modification changes the range of similarity
[299f420] typo
[c7efbdd] fix: actually relevancy is a similarity so goes from -1 to +1
[cc152d1] fix: imports were broken
[751baa0] docs: simpler install url
[4042226] update zsh completion
[8e92beb] new: add shell completions for bash and fish
[b54e1fb] docs: update completion
[515a273] fix: generating completions
[4054576] revert: forget idea about lazy loading batch loadere
[f0c384a] untested: use lazy loading for batchloader
[6e3a05b] fix: arg parsing
[3762bb3] fix: imports
[ec0b8bb] minor: fix
[e37350b] breaking: ename cache folders
[4fbb396] docs: fix incorrect help command
[3465a2b] enh: check env variable case insensitively
[193e595] major: rename project from WDoc to wdoc Lowercase is better and more humble
[19006a5] fix: crash when no arguments given
[e7a9e57] docs: remove a print
[cd75e07] docs: clarification
[39dbc3e] minor: rework the way init.py of subdirs are used
[6e7aa2f] enh: when calling --help, use lazy loading for faster documentation
[a536df5] feat: add a new env variable to enable import tricks for a way faster import time
[c25aeb4] fix: entrypoints It turns out that the way I set my entry points made my init called
twice, which was adding an extra delay when calling --help and probably
other circumstances. Thanks to claude-sonnet I was able to figure out
how to refactor init.py and main.py to a more standard usage.
Some changes will probably be fixed later as they create bugs as it
required changing a lot of moving parts
[be6b4c6] docs: dont mention lazy imports as a feature
[7c1e8cd] docs: add top level docstring to env
[cc94309] docs: better parse_file documentation
[afedd6c] minor: set debug and verbose flag as arg to wdoc if parse_file called
[272e179] minor: md_printer should dedent message
[d8b4b53] minor: use Literal for some env types
[b6891b0] docs: typo
[d20d3df] minor: use Literal for typing of task
[646db3c] todo: mention to start using unit test
[81c0cb6] minor: init phrasing
[8136606] potentially better main
[5df363e] simpler and faster way to load flags
[213bc02] minor: docstring for wdoc_parse_file
[d79a140] bump version 2.1.7 -> 2.1.8
[10446d9] fix: better way to catch exceptions when using filetype online_media
[82caa42] fix: the exception cause was not opened in the debugger
[6657124] minor: better exception printer
[8f251a7] minor: info print
[a076e50] minor: dont change sharedmem if not multithreading anyway
[d141787] docs: clarificaiton
[001ddd3] new: if using wdoc_parse_file set loading_failure to crash by default
[18b20bf] minor: add comment
[5fd87c7] add new crash earlier if all documents failed to load
[ba7947b] fix: debug in parse_file now also opens debugger like in wdoc
[e4f7a97] fix: debug and verbose args were missing in parse_file
[6461252] set max token for semantic batching to 500 by default instead of 750