Performance issues initializing shell with hundreds of abbreviations

[qadzek]

First of all, thank you for developing this wonderful tool. I use zsh-abbr all the time. According to grep --count '^abbr' $ABBR_USER_ABBREVIATIONS_FILE, I currently have 461 abbreviations. I even have an abbreviation to edit my ABBR_USER_ABBREVIATIONS_FILE!

However, it feels like zsh-abbr has become a victim of its own success. The time to first prompt feels quite long.

I'm using v6.2.1 on a 2020 ThinkPad laptop (32 GB RAM) for the following tests. On a slow VPS, things are significantly worse.

Running /usr/bin/time zsh -i -c exit with a blank ABBR_USER_ABBREVIATIONS_FILE:

0.17user 0.31system 0:00.43elapsed 111%CPU (0avgtext+0avgdata 9460maxresident)k
0inputs+48outputs (0major+31682minor)pagefaults 0swaps

With almost 500 abbreviations:

0.50user 1.15system 0:01.57elapsed 104%CPU (0avgtext+0avgdata 9436maxresident)k
0inputs+72outputs (1major+121043minor)pagefaults 0swaps

Here are my zprof results:

num  calls                time                       self            name
-----------------------------------------------------------------------------------
 1)  460        1020.46     2.22   66.29%    936.18     2.04   60.82%  _abbr:util_add
 2)  461        1178.47     2.56   76.56%    111.63     0.24    7.25%  abbr
 3)    1         104.63   104.63    6.80%    104.50   104.50    6.79%  _zsh_nvm_lazy_load
 4) 2791          44.10     0.02    2.87%     44.10     0.02    2.87%  _abbr_debugger
 5)    1        1217.49  1217.49   79.09%     39.00    39.00    2.53%  _abbr_load_user_abbreviations:load
 6)  461        1059.70     2.30   68.84%     34.05     0.07    2.21%  _abbr:add
 7)   10        1472.94   147.29   95.69%     26.55     2.66    1.72%  .zinit-load-plugin
 8)  460          41.34     0.09    2.69%     26.03     0.06    1.69%  _abbr:util_check_command
 9)   18          21.00     1.17    1.36%     21.00     1.17    1.36%  .zinit-diff-parameter
10)   18          18.20     1.01    1.18%     18.20     1.01    1.18%  .zinit-diff-functions
11)    1        1246.41  1246.41   80.97%     14.92    14.92    0.97%  _abbr_init
12)    1          15.00    15.00    0.97%     13.46    13.46    0.87%  (anon) [/home/qadzek/.cache/p10k-instant-prompt-qadzek.zsh:3]
13)  460          21.05     0.05    1.37%     13.09     0.03    0.85%  _abbr:util_sync_user
14)    1          19.07    19.07    1.24%     13.04    13.04    0.85%  compinit
15)  460          17.55     0.04    1.14%     12.75     0.03    0.83%  _abbr:util_log_unless_quiet

I'm wondering if these results are to be expected?
According to this formula, 500 abbreviations should only add about 200ms, but that doesn't seem to be the case for me.
Is there anything I can do to speed things up? The only thing that seems to help is deleting a large number of abbreviations.

[olets]

Thanks for sharing. Have you tried running the benchmarks? The zsh-abbr performance docs you linked say how… almost. Looking at it now, I see the docs don't say to run that from the zsh-abbr directory. So start with cd $ABBR_SOURCE_PATH.

[olets]

Oh it also requires Docker (fwiw I install that by installing Orbstack. I'm not affiliated with Orbstack).

Here's a blank Notion template you can use to do your personal calculations https://oletsdev.notion.site/221c6fd61c57800bb570dea27737aa2c. Start with the "copy" button in the upper right.

[qadzek]

Thanks for the helpful instructions and template. Here are my benchmarks: https://encouraging-salesman-96f.notion.site/223f005c2c4980668100fc47434cef58?v=223f005c2c498152a7af000c24891988 I'm not sure why your results are significantly better.

The vast majority of my abbreviations are multi-stage abbreviations, which is a super handy feature. For example:

abbr 'd'='docker'
abbr 'docker c'='docker container'
abbr 'docker container a'='docker container attach'

I have no idea how feasible this is, but I imagine performance could be improved if only the first abbreviation is loaded initially. When the first abbreviation expands, the second one would be loaded, and so on.

[olets]

I currently have 461 abbreviations

You might hold the record for most abbreviations! I have around 200, but a lot of that is aspirational — I would guess I use around 70 with any regularity. Cool that it's so useful for you.

Here are my benchmarks

Thanks for sharing those!

Hard to say with so few machines to compare, but we may just be seeing the difference between your 2020 ThinkPad and my 2020 M1 MBP.

I have no idea how feasible this is, but I imagine performance could be improved if only the first abbreviation is loaded initially. When the first abbreviation expands, the second one would be loaded, and so on.

I don't think this is in the cards. Beyond any parsing complications, it would take a different storage approach. But it's an interesting idea! If anyone reading this wants to take it on, let's talk here.

The vast majority of my abbreviations are multi-stage abbreviations, which is a super handy feature.

Glad to hear this feature is getting use!

In your example abbr 'docker container a'='docker container attach', do you expand a to something else in other contexts? If not, could you abbr a='docker container attach'? Ideally zsh-abbr's performance wouldn't be a factor in the abbreviations you choose… but something to think about.

 

//

Another thing that might be helpful is per-directory abbreviations. There's one solution for that in #167. Of course not helpful if you use the same commands everywhere.

 

//

If you come up with a workaround please let us know

[qadzek]

You might hold the record for most abbreviations!

I definitely went a little overboard. Some abbreviations are used daily, while others were added just in case they might be useful in the future, almost as documentation.

In your example abbr 'docker container a'='docker container attach', do you expand a to something else in other contexts? If not, could you abbr a='docker container attach'?

This wouldn't work for me, as the letter a has indeed different meanings in different contexts. For example:

abbr 'a'='apt'
abbr 'apt a'='apt autoremove'
abbr 'g'='git'
abbr 'git a'='git add'

If you come up with a workaround please let us know

I think I might use the following as a workaround to keep my main ABBR_USER_ABBREVIATIONS_FILE short.

1. Add an abbreviation to my ABBR_USER_ABBREVIATIONS_FILE to load abbreviations related to a particular application on demand:

abbr 'ab d'='source ~/dotfiles/zsh/zsh-abbr/docker.zsh'

2. That docker.zsh file contains session scoped abbreviations, for example:

abbr --session --quiet 'docker c'='docker container'
abbr --session --quiet 'docker container a'='docker container attach'
abbr --session --quiet 'docker container e'='docker container exec'

[5A6F65]

• I opened a PR perf(_abbr:util_add): optimize execution time #179 to solve this issue, welcome to try.

[5A6F65]

This is just a simple mitigation, but I think it is effective enough

[olets]

@qadzek this change is released in v6.3.2. It may improve performance for you

[qadzek]

@5A6F65 Thanks a lot for this optimization!
For some reason, the benchmarks return roughly the same results as before updating to v6.3.2.
However, running /usr/bin/time zsh -i -c exit shows a major improvement: 0.50user 1.15system 0:01.57elapsed (before) versus 0.25user 0.38system 0:00.58elapsed (after). The zprof results have improved a lot too, and opening a terminal feels snappier.