Starving nginx of File-Handles
This post covers starving nginx of file-handles, demonstrating a tool I wrote for exhuasting file-handles.
The program works by open()ing files until it can’t open any more.
Then provides the option to close() files, one at a time... thus freeing a single file-handle at a time, allowing the user to test a targeted program’s behavior against the number of available file-handles.
First off, to run these experiments I need to run nginx as root, that is, I can’t start nginx on the command line as a regular user.
user@debian:~$ /usr/sbin/nginx nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2019/07/07 17:01:48 [warn] 1335#1335: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1 2019/07/07 17:01:48 [emerg] 1335#1335: open() "/var/log/nginx/access.log" failed (13: Permission denied)
And yes, nginx is usually managed as a systemd service (on systemd systems), but like I said, in order to run these tests, I need to start it on the command line.
But root has the privilege to use file-handles in excess of fs.file-max. So what we need, is a root shell, but where fs.file-max is applied.
The way to do this is to get a root shell, but without the "CAP_SYS_ADMIN" Linux capability... among other things, the "CAP_SYS_ADMIN" capability allows the user to exceed fs.file-max.
Getting a shell without this capability can be acheived with
# setpriv --bounding-set -sys_admin bash
This allows me to achieve my goal of starving nginx of file-handles, while still having every other rootly power.
I’m using nginx version: nginx/1.10.3
The [n-m] numbers indicate the range of available file-handles resulting in the given errors.
[000-001]
bash: start_pipeline: pgrp pipe: Too many open files in system bash: /usr/sbin/nginx: Too many open files in system
[002-003]
/usr/sbin/nginx: error while loading shared libraries: libdl.so.2: cannot open shared object file: Error 23
[004-004]
/usr/sbin/nginx: error while loading shared libraries: libpthread.so.0: cannot open shared object file: Error 23
[005-005]
/usr/sbin/nginx: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: Error 23
[006-006]
/usr/sbin/nginx: error while loading shared libraries: libpcre.so.3: cannot open shared object file: Error 23
[007-007]
/usr/sbin/nginx: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: Error 23
[008-008]
/usr/sbin/nginx: error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file: Error 23
[009-009]
/usr/sbin/nginx: error while loading shared libraries: libz.so.1: cannot open shared object file: Error 23
[010-010]
/usr/sbin/nginx: error while loading shared libraries: libc.so.6: cannot open shared object file: Error 23
[011-011]
nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (23: Too many open files in system)
[012-012]
nginx: [emerg] getpwnam("www-data") failed (23: Too many open files in system) in /etc/nginx/nginx.conf:1
[013-014]
nginx: [emerg] getpwnam("www-data") failed in /etc/nginx/nginx.conf:1
[015-015]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_auth_pam_module.so" failed (/usr/share/nginx/modules/ngx_http_auth_pam_module.so: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-auth-pam.conf:1
[016-016]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_auth_pam_module.so" failed (libpam.so.0: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-auth-pam.conf:1
[017-017]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_auth_pam_module.so" failed (/usr/share/nginx/modules/ngx_http_auth_pam_module.so: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-auth-pam.conf:1
[018-019]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_auth_pam_module.so" failed (libpam.so.0: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-auth-pam.conf:1
[020-020]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_auth_pam_module.so" failed (libaudit.so.1: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-auth-pam.conf:1
[021-021]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_auth_pam_module.so" failed (libcap-ng.so.0: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-auth-pam.conf:1
[022-023]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_dav_ext_module.so" failed (libexpat.so.1: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-dav-ext.conf:1
[024-024]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_geoip_module.so" failed (/usr/share/nginx/modules/ngx_http_geoip_module.so: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-geoip.conf:1
[025-026]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_geoip_module.so" failed (libGeoIP.so.1: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-geoip.conf:1
[027-028]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libgd.so.3: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[029-029]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libm.so.6: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[030-030]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libpng16.so.16: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[031-031]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libfontconfig.so.1: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[032-032]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libfreetype.so.6: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[033-033]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libjpeg.so.62: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[034-034]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libXpm.so.4: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[035-035]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libX11.so.6: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[036-036]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libtiff.so.5: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[037-037]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libwebp.so.6: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[038-038]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libxcb.so.1: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[039-039]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (liblzma.so.5: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[040-040]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libjbig.so.0: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[041-041]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libXau.so.6: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[042-042]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libXdmcp.so.6: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[043-043]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (libbsd.so.0: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[044-044]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_image_filter_module.so" failed (librt.so.1: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:1
[045-045]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_upstream_fair_module.so" failed (/usr/share/nginx/modules/ngx_http_upstream_fair_module.so: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-upstream-fair.conf:1
[046-046]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_xslt_filter_module.so" failed (/usr/share/nginx/modules/ngx_http_xslt_filter_module.so: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:1
[047-048]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_xslt_filter_module.so" failed (libxml2.so.2: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:1
[049-050]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_xslt_filter_module.so" failed (libxslt.so.1: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:1
[051-051]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_xslt_filter_module.so" failed (libicui18n.so.57: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:1
[052-052]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_xslt_filter_module.so" failed (libicuuc.so.57: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:1
[053-053]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_xslt_filter_module.so" failed (libicudata.so.57: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:1
[054-054]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_xslt_filter_module.so" failed (libgcrypt.so.20: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:1
[055-055]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_xslt_filter_module.so" failed (libstdc++.so.6: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:1
[056-056]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_xslt_filter_module.so" failed (libgcc_s.so.1: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:1
[057-057]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_http_xslt_filter_module.so" failed (libgpg-error.so.0: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:1
[058-058]
nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_stream_module.so" failed (/usr/share/nginx/modules/ngx_stream_module.so: cannot open shared object file: Too many open files in system) in /etc/nginx/modules-enabled/50-mod-stream.conf:1
[059-059]
nginx: [emerg] socket() 0.0.0.0:80 failed (23: Too many open files in system)
[060-060]
nginx: [emerg] socket() [::]:80 failed (23: Too many open files in system)
[061-061]
nginx: [alert] mmap(MAP_ANON|MAP_SHARED, 1280) failed (23: Too many open files in system)
[062-062]
nginx: [emerg] open("/dev/null") failed (23: Too many open files in system)
[063-063]
nginx: [emerg] open() "/run/nginx.pid" failed (23: Too many open files in system)
[064-064] No message on terminal. Master process running. No response when trying to load homepage.
root 1492 0.0 0.1 159532 1620 ? Ss 13:47 0:00 nginx: master process /usr/sbin/nginx
/var/log/nginx/error.log:
2019/07/03 13:47:54 [emerg] 1493#1493: epoll_create() failed (23: Too many open files in system) 2019/07/03 13:47:54 [alert] 1492#1492: worker process 1493 exited with fatal code 2 and cannot be respawned
[065-065] No message on terminal. Master and worker process started. No response when trying to load homepage.
root@debian:/var/log/nginx# ps aux | grep -i nginx root 1507 0.0 0.1 159532 1620 ? Ss 13:53 0:00 nginx: master process /usr/sbin/nginx www-data 1508 0.0 0.3 159868 3156 ? S 13:53 0:00 nginx: worker process
/var/log/nginx/error.log:
2019/07/03 13:53:34 [emerg] 1508#1508: eventfd() failed (23: Too many open files in system) 2019/07/03 13:53:59 [crit] 1508#1508: accept4() failed (23: Too many open files in system) 2019/07/03 13:54:00 [crit] 1508#1508: accept4() failed (23: Too many open files in system) 2019/07/03 13:54:00 [crit] 1508#1508: accept4() failed (23: Too many open files in system) 2019/07/03 13:54:01 [crit] 1508#1508: accept4() failed (23: Too many open files in system) [... snip ...]
[066-066] No message on terminal. Master and worker process started.
root@debian:/var/log/nginx# ps aux | grep -i nginx root 1523 0.0 0.1 159532 1636 ? Ss 13:56 0:00 nginx: master process /usr/sbin/nginx www-data 1524 0.0 0.3 159868 3400 ? S 13:56 0:00 nginx: worker process
/var/log/nginx/error.log:
2019/07/03 13:56:37 [crit] 1524#1524: accept4() failed (23: Too many open files in system) 2019/07/03 13:56:37 [crit] 1524#1524: accept4() failed (23: Too many open files in system) 2019/07/03 13:56:38 [crit] 1524#1524: accept4() failed (23: Too many open files in system) 2019/07/03 13:56:38 [crit] 1524#1524: accept4() failed (23: Too many open files in system) 2019/07/03 13:56:39 [crit] 1524#1524: accept4() failed (23: Too many open files in system) [... snip ...]
[067-067] No message on terminal. Master and worker process started. Loading homepage results in: “500 Internal Server Error”
root@debian:/var/log/nginx# ps aux | grep -i nginx root 1538 0.0 0.1 159532 1624 ? Ss 13:57 0:00 nginx: master process /usr/sbin/nginx www-data 1539 0.0 0.3 159868 3308 ? S 13:57 0:00 nginx: worker process
/var/log/nginx/error.log:
2019/07/03 13:58:34 [crit] 1539#1539: *1 open() "/var/www/html/index.nginx-debian.html" failed (23: Too many open files in system), client: 10.0.2.2, server: _, request: "GET / HTTP/1.1", host: "127.0.0.1:8080"
[068-999] No message on terminal. Master and worker process started. Landing page successfully loads: “Welcome to nginx!”