Feat/model recommendation wizard - #1
Merged
Merged
Conversation
Nine-question interactive wizard using questionary. Collects use case, vision input, optional sample text for ratio estimation, token ratio, context length, model source, cache hit ratio, Arena threshold, and provider subset. All output in English.
Filters records through BlacklistFilter, ArenaFilter, and preference-based inline filters. Returns Best Value (lowest weighted price), Best Quality (highest Arena score), and Balanced (40% price + 60% quality rank aggregation) recommendations. Fewer than 3 survivors returns a single Best Value pick.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ness check Restructures `llmcost/cli.py` as a subcommand dispatcher. Bare `llmcost` and `llmcost price` retain existing behavior. `llmcost recommend` runs the interactive wizard, auto-refreshes pricing data if cache is older than 30 days, and renders Rich recommendation panels. Extends testpaths to include recommender test suite. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds _PRICE_OPTIONS constant and Q8 question to the wizard with options No limit / $10 / $25 / $50 / $75 per M tokens (default: No limit). Engine filters out records whose weighted price exceeds prefs.max_price. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ndations z-ai is OpenRouter's namespace for Zhipu AI models; the direct zhipu/* records are already present after the scraper runs. Without this filter, z-ai/glm-5.1 and zhipu/glm-5.1 both enter the scoring pool, inflating the pool size and making Zhipu appear twice in results. Mirrors the existing filter in pricing CLI. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move filter chain logic from cli.py and engine.py into a new RecordFilter builder (filters/pipeline.py) and centralize cache loading + arena-score hydration into loader.py. Both the price CLI and the recommend wizard now share the same pipeline, eliminating duplicated filtering logic and cache-management code. - Add llmcost/pricing/filters/pipeline.py: fluent RecordFilter builder - Add llmcost/pricing/loader.py: load_records() with auto-refresh - Simplify cli.py and engine.py to delegate to these shared modules - Add display_filter_summary() in recommender/display.py - Move CN_PROVIDERS constant to pipeline.py (shared) - Change Q8 default max price from "No limit" to "$10/M" Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Consolidate USE_CASE_GROUPS, descriptions, and ratio defaults into a single UseCaseDef dataclass registry (single source of truth) - Add vision_default and default_context_length per use case - Q1.5 vision default pre-set from registry; Q3 context length default pre-set from registry with safe fallback - Q1.6 sample blends 1:1 with use-case preset (source: 'blended') - Q5 (cache hit ratio) removed — value comes directly from registry - Q7 provider checkboxes pre-filtered based on Q4 model source - Add 11 new use cases across QA, code, conversation, reasoning groups - Fix display.py ratio label truncation (int → :g format) - Update tests to match new wizard flow and preset values
…wizard polish - RecordFilter: add has_cache_pricing() and require_arena_score() builder methods - Engine: rank by $/kArena (value_ratio) instead of raw weighted $/M; expose value_ratio on Recommendation; filter by require_cache_pricing pref - Wizard: add cache pricing question (default Yes); remove Q-number prefixes; default use case changed to Hermes - Display: show $/kArena alongside $/M in recommendation panels - Tests: update engine/wizard tests for new filter logic and question strings
…ive sample prompting
- Wizard: replace fixed $/M price options with SOTA model selector (claude-opus-4-7 default); stores direct_id in max_price_model - Engine: _resolve_max_price() looks up SOTA model weighted price at runtime; falls back to max_price if model not found or None - Engine: add exclude_redundant_pinned() to _filter() chain to match CLI behaviour (fixes gemini-3.1-pro-preview-customtools leaking through)
- Add $/image column to pricing table for models billed per image - Mark Weighted$/M with * for hybrid models (excludes per-image cost) - Add footnote explaining * when any per-image model is displayed - Add exclude_per_image_pricing() filter to RecordFilter - Add Recommendation.value_ratio field to engine dataclass
…iration_date, tokenizer from OpenRouter
…ixes - UseCaseDef: required_parameters (OR filter) + preferred_parameters (rank boost) - RecordFilter.has_required_parameters(): OR logic, None supported_parameters = keep - Engine: ScoredCandidate dataclass, _compute_combined() 5:4:1 weights, debug_candidates() - display: render_debug_candidates() sorted by $/kArena - CLI: llmcost recommend --debug prints full candidate ranking - Fix _is_redundant_pinned: catch date suffix after -preview (e.g. -preview-09-2025)
… review - cache.py: atomic write via .tmp + os.replace(), JSONDecodeError guard in load() - loader.py: reload from cache after save() to return full merged set - engine.py: warn when max_price_model not found; fix _preferred_score(None)→0.5; cache scored results in recommend() to avoid redundant work in debug_candidates(); fix TypeError when all value_ratios are None in _select_tiers - wizard.py: fix _SOTA_MODELS/default label to use dotted IDs (claude-opus-4.7); fix required_parameters comments to describe OR semantics correctly - display.py: import CN_PROVIDERS/IMAGE_USE_CASES instead of redefining; fix max_price row to show model name; fix debug table title; add ?ctx marker for models with unknown context length when min_context_length is active - cli.py: non-interactive --use-case flag for agent/script usage; enable --help
- test_cache: incremental merge preserves old records, overwrites by id, corrupt JSON returns empty (REL-1 regression) - test_filters: has_required_parameters OR logic, supported_parameters=None passthrough, _is_redundant_pinned preview+date fix (T2/T3) - test_engine: _preferred_score None→0.5, debug_candidates caching (T4/T5), required/preferred param propagation (T7); fix rationale string concat bug that dropped "(only N model(s)...)" from the non-None branch - test_wizard: fix stale claude-opus-4-7 → claude-opus-4.7 assertions
…urations Add llmcost recommend usage examples (interactive and --use-case), update all llmcost price command examples, and add a recommended configurations section covering workload types, quality tiers, and provider focus patterns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.