nginx hogs cpu when proxying large files

I have a server where nginx is used as frontend for Apache. nginx serves static content and Apache serves PHP pages. This is a common setup.

Today I migrated stuff to a new server and needed to copy a 7GB database file to another server. I figured HTTP would be fastest way to do it. Unfortunatelly, DNS change already went thought so I could not serve the file on the static domain nginx was configured for.

I thought nevermind, placed file under one of those domains handled by Apache and started the download. It was going fine at 11MB/s for some time. However, soon it started to crawl at 850KB/s. I suspected network problems, but everything else was running fine. I looked at process list and whoa, nginx using 99% of CPU. Because of this single download, the server was brought to its knees and no other client could even get a simple Hello World page.

I stopped the download at the client side and nginx soon recovered (not restart needed). Then I edited /etc/hosts and place the old IP address of static domain and continued the download (wget -c). It finished few minutes later with 11MB/s average.

Tweet to @mbabuskov Milan Babuškov, 2011-12-01