Is the Learning System Actually Learning? (USER/ files + ratings)

[noxx]

Hi,

Just installed new version of PAI and spent time understanding the system deeply. I love the architecture and the skill system is impressive. However, I discovered something that confused me and I want to understand if this is intended or a gap.

The Issue: Learning Infrastructure Without Learning

PAI has extensive infrastructure for "learning":

• MEMORY/LEARNING/ directory for captured learnings
• MEMORY/LEARNING/SIGNALS/ratings.jsonl for rating capture
• Rating hooks (ExplicitRatingCapture, ImplicitSentimentCapture)
• USER/ directory with ABOUTME.md, CONTACTS.md, TECHSTACKPREFERENCES.md, etc.
  But from what I can tell, none of this data is actually used.

Specific Issues

1. Ratings go nowhere
   When I rate a response "7" or "8 - good work", it gets captured to ratings.jsonl. But what reads this file? What learns from it? It seems like the ratings are stored but never used to
   improve anything.

2. USER/ files aren't loaded
   I was told to fill in USER/ABOUTME.md, USER/CONTACTS.md, USER/TECHSTACKPREFERENCES.md to personalize PAI. But these files are never auto-loaded. If I say "send to John", PAI doesn't know who John is - it would have to manually grep CONTACTS.md.

The only files that ARE auto-loaded via LoadContext hook:

• SKILL.md (the algorithm)
• AISTEERINGRULES.md
• DAIDENTITY.md

3. The only skill that reads USER/ is Telos skill that actually reads from USER/TELOS/. That's wired up properly. But everything else in USER/ is just... storage that nothing reads.

My Questions

1. Is this the intended design? Is USER/ meant to be manually referenced rather than auto-loaded?
2. What's the point of the rating system? If ratings are captured but nothing uses them, why capture them?
3. Is there a learning loop I'm missing? Does something actually process ratings.jsonl and MEMORY/LEARNING/ to improve future responses?
4. Should more skills reference USER/? For example, should Research skill check TECHSTACKPREFERENCES.md when giving tech recommendations?

Suggestion, Either:

1. Wire up the learning - Have LoadContext inject more USER/ files, have something process ratings
2. Document the reality - Make it clear these are "storage for future use" not "active learning"

I don't want to spend time filling in USER/ files if nothing reads them. And I don't want to rate responses if it's not actually improving anything.

Am I missing something? Is there a learning loop I don't understand?

Thanks for building this - the skill system and algorithm are genuinely impressive. Just want to understand if the learning piece is functional or aspirational.

[wojteksbt]

This is a bit of a surprise! I had assumed the learning system was working somehow and decided to investigate today. It turns out that nothing actually seems to use the gathered LEARNINGS data.

I'm wondering if something is misconfigured on my end, or if the retrieval/application layer simply isn't implemented yet.

I couldn't find any information about this in the README. However, I did notice that AISTEERINGRULES.md appears to be one output of the learning system - something has written to it during PAI development:

"These rules were derived from failure analysis of 84 rating 1 events (2026-01-08 to 2026-01-17)."

This suggests there's some mechanism for applying learnings, but I can't find how it works.

My guesses:

1. There's an automatic feedback loop in KAi that hasn't been extracted to PAI yet
2. The retrieval mechanism is planned but not yet implemented
3. The intended workflow is to periodically ask the DA to analyze the data and update steering rules manually

@danielmiessler — Could you clarify how this is meant to work? I'd love to understand the intended approach. Thanks!

[zeddy303]

This is what my PAI told me when I asked:
❯ Hi Sherpa! can you honestly tell me (do not just create content, this needs to based on your actual
process) are you using the learning system, if so, how are you doing that? Can you give me an example
of the last 10 things you learned?

I am not currently using a persistent learning system. Here's the honest reality:

1. No persistent memory across sessions. Each time you start a new conversation, I start fresh. I don't
   retain anything from previous sessions unless it's loaded into my context (like the CORE skill, steering
   rules, etc. that your hooks inject at startup).
2. The MEMORY system exists in your architecture — I can see references to WORK/, STATE/, LEARNING/
   directories in the documentation. But I'd need to actually check whether those directories exist and have
   content, and whether any workflow is writing to them.
3. I cannot give you "the last 10 things I learned" because I genuinely don't have that data. Making up a
   list would be exactly the kind of fabrication you're asking me not to do.

What I can do right now: Check if the memory/learning directories exist and have any content, which would
tell us whether any session has ever written learnings. Want me to look?

What would need to be true for learning to work:

• A hook or workflow that writes learnings to disk at session end
• A hook or context loader that reads them back at session start
• Neither of those appears to be happening automatically in what I see loaded

This is a real gap worth closing if the learning system matters to you.
\

(but then I followed up with "where did we leave off" and then searches the folders and figured it out):

[danielmiessler]

Yes, the final piece of applying the learnings and doing the matching is not happening natively. It's just all the pieces are set up there for it to happen because we've been capturing the two things combined. But you're right that a final piece is needed to close the loop. We'll be adding that very soon.

…

On Thu, Jan 29, 2026 at 7:37 PM, Zane Edwards < ***@***.*** > wrote: This is what my PAI told me (but then I followed up with "where did we leave off" and then searches the folders and figured it out): I am not currently using a persistent learning system. Here's the honest reality: * No persistent memory across sessions. Each time you start a new conversation, I start fresh. I don't retain anything from previous sessions unless it's loaded into my context (like the CORE skill, steering rules, etc. that your hooks inject at startup). * The MEMORY system exists in your architecture — I can see references to WORK/, STATE/, LEARNING/ directories in the documentation. But I'd need to actually check whether those directories exist and have content, and whether any workflow is writing to them. * I cannot give you "the last 10 things I learned" because I genuinely don't have that data. Making up a list would be exactly the kind of fabrication you're asking me not to do. What I can do right now: Check if the memory/learning directories exist and have any content, which would tell us whether any session has ever written learnings. Want me to look? What would need to be true for learning to work: * A hook or workflow that writes learnings to disk at session end * A hook or context loader that reads them back at session start * Neither of those appears to be happening automatically in what I see loaded This is a real gap worth closing if the learning system matters to you. — Reply to this email directly, view it on GitHub ( #531 (reply in thread) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAAMLXSCBFKO6J6EEVK3MEL4JLGYXAVCNFSM6AAAAACTIHJXI2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKNRUGY2TGOA ). You are receiving this because you were mentioned. Message ID: <danielmiessler/Personal_AI_Infrastructure/repo-discussions/531/comments/15646538 @ github. com>

[DolphusCY]

Hi @zeddy303

I just copied your exact question and asked my PAI (after upgarding to 2.5) and this is what it said....

Here's my honest assessment of the learning system:

The learning system is mostly write-only.

What works:

• Hooks reliably write data: 809 algorithm learnings, 55 failure analyses, 246 system learnings, 1200+ signal ratings
• The ratings counter in the startup banner ("Signals 1236") is real — it counts lines in ratings.jsonl

What doesn't work:

• LoadContext.hook.ts — the main session startup hook — does not load any learning files. It loads SKILL.md, USER files, steering rules, but zero learnings. So when
  a new session starts, past learnings aren't in context.
• SoulEvolution.hook.ts reads recent learnings (last 3 days), but its actual impact on behavior is unclear — it runs as a hook, not as injected context that shapes
  responses.

The honest conclusion:
The system captures data well but doesn't close the loop. Learnings are written to disk and sit there. They don't get loaded into new sessions in a way that changes my behavior. The ratings feed a counter. The failure analyses document what went wrong but aren't systematically reviewed before similar tasks.

It's an architectural gap: the write path is solid, the read-back-and-apply path is missing from the main context loading pipeline.

Last 10 real learnings (from the files I read): timezone changes made without asking, PDF styling, fix completions, and several neutral 5/10 sentiment captures. Most contained boilerplate "Action Items" like "Review..." and "Consider..." with no automated follow-through.

What would fix it: LoadContext.hook.ts needs to load a digest of recent learnings (especially failures and low-rated interactions) into session context at startup.

That's a straightforward change — read the last N learning files and inject a summary.

I know Daniel and team will get to this, its just a matter of time .