Memory issues with large number of sources

[tanderson92]

I (and some collaborators) am observing strange (incorrect) memory usage with FMM3D and large numbers of sources. The following Julia MWE demonstrates the problem.

using FMM3D

M = 11_000_000

sources = rand(3, M)

charges = rand(M)

vals = @time lfmm3d(1.0e-8, sources; targets = sources, pg = 1, charges = charges)

@show vals.pot |> sum

On my system, with 64GB of RAM, M=11_000_000 executes properly while utilizing <11GB of memory, while M=14_000_000 OOMs. Between these values of M I observe increasingly implausible amounts of memory being required.

[xuanzhaogao]

The memory cost of FMM3D doesn’t scale linearly with the number of particles; instead, it scales with the number of tree boxes. When the tree is one level deeper, the number of tree boxes increases by approximately 7 to 8 times. This should explain the problem you’re encountering.

See the report below:

Conclusion
The memory jump from ~11 GB to ~64 GB is dominated by two large buffers that scale linearly with the number of tree boxes:

• rmlexp (double precision multipole/local expansion workspace)
• mexp (complex plane-wave expansions, 16 bytes/element)

When M increases from 11M to 14M, nboxes jumps 7.3×, which multiplies those buffers and drives the peak memory.

---

Measured allocations (instrumented run)

M = 11,000,000

• nboxes = 37,521
• rmlexp (double) = 2.74 GB
• mexp (complex) = 5.86 GB
• list arrays (int64) = 0.075 GB
• pgboxwexp (complex) = 0.037 GB
• gboxmexp (complex) = 0.068 GB
• gboxwexp (complex) = 0.040 GB

Approx total of just these: ~8.8 GB
With sources/targets/charges/temps → ~11 GB (matches observation)

---

M = 14,000,000

• nboxes = 274,153
• rmlexp (double) = 19.98 GB
• mexp (complex) = 42.82 GB
• list arrays (int64) = 0.90 GB
• pgboxwexp (complex) = 0.50 GB
• gboxmexp (complex) = 0.93 GB
• gboxwexp (complex) = 0.04 GB

Approx total of just these: ~65 GB
(matches ~64 GB observed)

---

Root Cause
The large increase in nboxes at 14M causes the plane-wave expansion buffers (mexp) and multipole/local workspace (rmlexp) to explode in size. These two dominate total memory usage.

[xuanzhaogao]

You can change the number of particles in a leaf node to reduce the tree’s level by calling lfmm3d_ndiv and hfmm3d_ndiv in Fortran. However, it appears that the Julia wrapper doesn’t provide these functions.
Maybe we should expose this opinion in the future.

[lu1and10]

Thanks @ArrogantGao , it makes sense. @ahbarnett , your suggestion helps, thanks.

[ahbarnett]

So a suggestion for Thomas and other users is to increase the number of particles per box ``ndiv``, rebalancing to greater direct cost while keeping the tree RAM down. As Xuanzhao says, seems to be not yet available as a user parameter. You have to hack ``Helmholtz/hndiv.f`` for now. Note the MATLAB interface provides a model for how the Julia wrapper could expose as ``opts.ndiv`` PS Xuanzhao - pretty good answer by codex with ChatGPT 5.2 :) How much context does reading the codebase take? Did it insert memory-tracking statements and recompile?

…

On Wed, Feb 4, 2026 at 3:34 PM Libin Lu ***@***.***> wrote: *lu1and10* left a comment (flatironinstitute/FMM3D#68) <https://urldefense.com/v3/__https://github.com/flatironinstitute/FMM3D/issues/68*issuecomment-3849527525__;Iw!!DSb-azq1wVFtOg!QZAtSsI7oUCugCtp2qsfr68UdUvuyaezMlXqZouKRFIp4DLOhVP24mvfoZH06bSU3cFimkSSlaUS4Rzu8fk_dyKqn0uHnwBi$> Thanks @ArrogantGao <https://urldefense.com/v3/__https://github.com/ArrogantGao__;!!DSb-azq1wVFtOg!QZAtSsI7oUCugCtp2qsfr68UdUvuyaezMlXqZouKRFIp4DLOhVP24mvfoZH06bSU3cFimkSSlaUS4Rzu8fk_dyKqnzutahor$> , it makes sense. @ahbarnett <https://urldefense.com/v3/__https://github.com/ahbarnett__;!!DSb-azq1wVFtOg!QZAtSsI7oUCugCtp2qsfr68UdUvuyaezMlXqZouKRFIp4DLOhVP24mvfoZH06bSU3cFimkSSlaUS4Rzu8fk_dyKqn-0vHl5I$> , your suggestion helps, thanks. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/flatironinstitute/FMM3D/issues/68*issuecomment-3849527525__;Iw!!DSb-azq1wVFtOg!QZAtSsI7oUCugCtp2qsfr68UdUvuyaezMlXqZouKRFIp4DLOhVP24mvfoZH06bSU3cFimkSSlaUS4Rzu8fk_dyKqn0uHnwBi$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ACNZRSRYQEXSWHTR4KGO27D4KJJU7AVCNFSM6AAAAACTWKPQMSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTQNBZGUZDONJSGU__;!!DSb-azq1wVFtOg!QZAtSsI7oUCugCtp2qsfr68UdUvuyaezMlXqZouKRFIp4DLOhVP24mvfoZH06bSU3cFimkSSlaUS4Rzu8fk_dyKqn90lZwq5$> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- *-------------------------------------------------------------------~^`^~._.~' |\ Alex Barnett Center for Computational Mathematics, Flatiron Institute | \ http://users.flatironinstitute.org/~ahb 646-876-5942

[xuanzhaogao]

So a suggestion for Thomas and other users is to increase the number of
particles per box ndiv, rebalancing to greater direct cost while
keeping the tree RAM down.
As Xuanzhao says, seems to be not yet available as a user parameter.
You have to hack Helmholtz/hndiv.f for now.

Yes, there are two different functions, may need to hack and recompile locally.

Note the MATLAB interface provides a model for how the Julia wrapper could
expose as opts.ndiv

Then the c functions should be already there, just needed to be exported into julia.

PS Xuanzhao - pretty good answer by codex with ChatGPT 5.2 :)
How much context does reading the codebase take? Did it insert
memory-tracking statements and recompile?

The context used is not very large. It inserted print functions to display the number of tree boxes required and recompiled the code locally XD

[tanderson92]

I can hack my Julia wrappers locally to include this parameter for now; thanks Alex and Xuanzhao for pinpointing the issue and a workaround of sorts.

I did find this behavior (the large jumps in workspace sizes) rather surprising. I would naiively have thought that you could do a lot more creation/destruction of these workspaces as one traverses the tree and not do the preallocation up front (so long as one is not re-using any of the data structures across fmm runs, which I understand to not be possible).