Skip to content

lol #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 81 commits into
base: main
Choose a base branch
from
Open

lol #4

wants to merge 81 commits into from

Conversation

Butanium
Copy link
Collaborator

No description provided.

mntss and others added 30 commits October 14, 2024 20:59
@mntss
Fix BatchTopKSAE training
4aea538
@adamkarvonen
Add a simple end to end test
fe54b00
@adamkarvonen
Rename input to inputs per nnsight 0.3.0
9ed4af2
@adamkarvonen
Complete nnsight 0.2 to 0.3 changes
807f6ef
@adamkarvonen
Fix frac_alive calculation, perform evaluation over multiple batches
dc30720
@adamkarvonen
Obtain better test results using multiple batches
067bf7b
@adamkarvonen
Add early stopping in forward pass
05fe179
@adamkarvonen
Change save_steps to a list of ints
f1b9b80
@adamkarvonen
Check for is_tuple to support mlp / attn submodules
d350415
@adamkarvonen
Merge pull request #26 from mntss/batchtokp_aux_fix
2ec1890 
Fix BatchTopKSAE training
@adamkarvonen
Merge pull request #30 from adamkarvonen/add_tests
c4eed3c 
Add end to end test, upgrade nnsight to support 0.3.0, fix bugs
@adamkarvonen
Ensure activation buffer has the correct dtype
d416eab
@adamkarvonen
Fix JumpReLU training and loading
552a8c2
@adamkarvonen
Begin creation of demo script
712eb98
@adamkarvonen
Modularize demo script
dcc02f0
@adamkarvonen
Track threshold for batchtopk, rename for consistency
32d198f
@adamkarvonen
Track thresholds for topk and batchtopk during training
b5821fd
@adamkarvonen
Remove demo script and graphing notebook
57f451b
@adamkarvonen
Add option to normalize dataset activations
81968f2
@adamkarvonen
Fix topk bfloat16 dtype error
488a154
@adamkarvonen
Add bias scaling to topk saes
484ca01
@adamkarvonen
Use the correct standard SAE reconstruction loss, initialize W_dec to…
8b95ec9 
… W_enc.T
@adamkarvonen
Also scale topk thresholds when scaling biases
efd76b1
@adamkarvonen
Merge pull request #31 from saprmarks/add_demo
67a7857 
Add option to normalize dataset, track thresholds for TopK SAEs, Fix Standard SAE
@adamkarvonen
Remove leftover variable, update expected results with standard SAE i…
9687bb9 
…mprovements
@adamkarvonen
Track lr decay implementation
f0bb66d
@adamkarvonen
Clean up lr decay
e0db40b
@adamkarvonen
Add sparsity warmup for trainers with a sparsity penalty
911b958
@adamkarvonen
Merge pull request #32 from saprmarks/add_sparsity_warmup
a11670f 
Add sparsity warmup
@adamkarvonen
Standardize learning rate and sparsity schedules
a2d6c43
adamkarvonen and others added 30 commits January 3, 2025 20:21
@adamkarvonen
Add an option to pass LR to TopK trainers
8316a44
@adamkarvonen
Save state dicts to cpu
3c5a5cd
@adamkarvonen
Add torch autocast to training loop
832f4a3
@adamkarvonen
Disable autocast for threshold tracking
17aa5d5
@adamkarvonen
Also update context manager for matroyshka threshold
65e7af8
@adamkarvonen
By default, don't normalize Gated activations during inference
52b0c54
@adamkarvonen
Import trainers from correct relative location for submodule use
8363ff7
@adamkarvonen
Make sure x is on the correct dtype for jumprelu when logging
c697d0f
@adamkarvonen
Remove experimental matroyshka temperature
6c2fcfc
@adamkarvonen
Normalize decoder after optimzer step
200ed3b
@adamkarvonen
Standardize and fix topk auxk loss implementation
0af1971
@adamkarvonen
Merge pull request #36 from saprmarks/aux_loss_fixes
34eefda 
Aux loss fixes, standardize decoder normalization
@adamkarvonen
Make sure to detach reconstruction before calculating aux loss
db2b564
@adamkarvonen
Add citation
77f2690
@adamkarvonen
Fix incorrect auxk logging name
784a62a
@adamkarvonen
Fix matryoshka spelling
aa45bf6
@adamkarvonen
Use torch.split() instead of direct indexing for 25% speedup
505a445
@adamkarvonen
simplify matryoshka loss
43421f5
@chanind
feat: pypi packaging and auto-release with semantic release
0ff8888
@saprmarks
Merge pull request #37 from chanind/pypi-package
a711efe 
feat: pypi packaging and auto-release with semantic release
0.1.0
07975f7 
Automatically generated by python-semantic-release
@adamkarvonen
Add a pytorch activation buffer, enable model truncation
944edd1
@adamkarvonen
Add better dataset generators
c644ccd
@adamkarvonen
Add optional backup step
17a41c7
@adamkarvonen
add activault buffer implementation
fe9d8c7
@adamkarvonen
Merge pull request #42 from saprmarks/mixed_datasets
d639166 
Add mixed datasets, model truncation, new activation buffers
@andyrdt
assert right padding for remove_bos logic
c7b2527
@andyrdt
mask out bos activations
59abc88
@adamkarvonen
Merge pull request #44 from andyrdt/andyrdt/bos_bugfix
b2bb4b5 
safer remove_bos logic
@canrager
add handling of device and dtype to IdentityDict
61ac634
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@Butanium @mntss @adamkarvonen @chanind @saprmarks @andyrdt @canrager