LD_PRELOAD Causes Error on Deployment

[scottwater]

I am investing using this buildpack.

Durning my deployment to heroku, the build is rejected do to an error (cannot open shared object file).

Some of the output (same error over and over) is below.

Any suggestions for a next step? Any other info I could provide?

Thanks,
Scott

kickoff:jemal g push staging jemal:master
Counting objects: 12, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (11/11), done.
Writing objects: 100% (12/12), 931 bytes | 0 bytes/s, done.
Total 12 (delta 9), reused 2 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Deleting 1 files matching .slugignore patterns.
remote: -----> jemalloc app detected
remote: -----> Vendoring binaries
remote: Fetching https://github.com/mojodna/heroku-buildpack-jemalloc/releases/download/4.2.1/jemalloc-4.2.1-1.tar.gz
remote: -----> Configuring build environment
remote: -----> Building runtime environment
remote: -----> Ruby app detected
remote: -----> Compiling Ruby/Rails
remote: -----> Using Ruby version: ruby-2.3.3
remote: -----> Installing dependencies using bundler 1.13.6
remote: Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
remote: ERROR: ld.so: object '/app/vendor/jemalloc/lib/libjemalloc.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

[sethfitz]

Good catch.

Are you setting LD_PRELOAD via heroku config?

If so, change /app/vendor/jemalloc/lib/libjemalloc.so.1 to /app/vendor/jemalloc/lib/libjemalloc.so (the former doesn't exist, despite what the docs and jemalloc.sh say).

If not, set that instead of using jemalloc.sh.

[sethfitz]

Actually, I'm wrong. /app/vendor/jemalloc/lib/libjemalloc.so.1 does exist in the tarball. I don't know what's going on here.

Which stack are you using?

[scottwater]

@mojodna I am using the cedar-14 stack with Ruby 2.3.3. I had set the LD_PREPOLOAD via Heroku config.

I had originally thought it was a typo as well, but I connected via bash and was able to verify the libjemalloc.so.1 does exist.

Exact commands executed:
heroku config:set LD_PRELOAD=/app/vendor/jemalloc/lib/libjemalloc.so.1

heroku config -s | grep LD_PRELOAD
LD_PRELOAD='/app/vendor/jemalloc/lib/libjemalloc.so.1'

Please let me know if I can provide any other details/info.

Thanks,
Scott

[sethfitz]

I was able to reproduce the error by running heroku run bash against a newly-created app with LD_PRELOAD set (prior to pushing any commits). That was because the app hadn't been built yet, and the dyno only contained the cedar-14 runtime. After pushing a commit containing an empty index.html, I re-ran heroku run bash and things worked swimmingly.

Try reconnecting to the dyno via bash and run (and report back):

ldd /app/vendor/jemalloc/lib/libjemalloc.so.1

Output should be (and there should be no error when connecting):

~ $ ldd /app/vendor/jemalloc/lib/libjemalloc.so.1
	linux-vdso.so.1 =>  (0x00007ffc5dffb000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f53da66d000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f53da2a8000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f53daacb000)

[scottwater]

I get the following:

~ $ ldd /app/vendor/jemalloc/lib/libjemalloc.so.1
linux-vdso.so.1 => (0x00007ffdb438c000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f652d6b6000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f652d2f1000)
/lib64/ld-linux-x86-64.so.2 (0x00007f652db14000)

I did a bit more testing. On a brand new/empty rails project (4.2.7.1), I get the error ERROR: ld.so: object '/app/vendor/jemalloc/lib/libjemalloc.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file, but the build is still deployed.

In my app, looking through more of the output I also see a JSON:ParseError (see http://d.pr/n/2bZd)

I tried to play around with some of the gem versions of sprockets (3.7.0) and sprocket-rails (3.2.0), but I have not been able to narrow down the issue.

Thanks,
Scott

[sethfitz]

Can you share access with me so I can do some poking around? mojodna / seth at mojodna dot net

[czj]

Hi @mojodna I'm having the same issue with a Heroku app :

ERROR: ld.so: object '/app/vendor/jemalloc/lib/libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

I'm using the same buildpack/configuration on Scalingo without any specific issues. If you want, I could add you on the open-source project we run with this lib on Heroku.

[9mm]

Curious about this if it still exists, I am wanting to use this in production on Heroku, rails 5, ruby 2.4.1, I'm hesitant to pull the trigger though.

[nateberkopec]

Let's try and get it fixed if it is an issue.

Can everyone using LD_PRELOAD as an env variable (like in your "config settings" in Heroku or whatever) in this thread please try removing that and just use jemalloc.sh in your Procfile?

i.e.:

web: jemalloc.sh bundle exec puma -C config/puma.rb

[9mm]

I havent tried using it yet, but I could try now. I did have one other question, do you know how I might do this if I'm using pgbouncer? It's bin/start-pgbouncer-stunnel below

web: bin/start-pgbouncer-stunnel bundle exec puma -C config/puma.rb
clock: bundle exec clockwork config/clock.rb
worker: bundle exec que ./config/environment.rb

[nateberkopec]

@9mm I think jemalloc.sh bin/start-pgbouncer-stunnel bundle exec puma -C config/puma.rb should work but I am no bash master.

[havenwood]

I ran into this issue a while back setting LD_PRELOAD but just prefixing jemalloc.sh is now working for me.

[knightq]

It works with the script, but it makes the Procfile strongly environment-dependent.
This "solution" should be considered only a workaround, not a fix to the original issue problem, IMHO.

[brian-kephart]

I too was unable to get LD_PRELOAD to work. Using jemalloc.sh in the Procfile did work, and reduced my overall memory usage significantly, but increased my swap memory usage. Some days, after nearly a full day's uptime, I'd see ~17MB of swap memory when my total memory usage was under 70%. With the stock allocator, those numbers are more like 2-6MB at 80-85% usage.

I bring it up here in case this sort of misbehavior is the result of having the script prepended only to processes in the Procfile, rather than all commands run on the dyno, making the Procfile option less viable. Has anyone else seen this?

This behavior was consistent on both the Cedar-14 and Heroku-16 stacks, running Rails 5.0 with Ruby 2.4.1 on a Hobby web dyno.