Monitoring Nginx with New Relic

I've had my New Relic account for some time, using it to monitor multiple VPSes and the applications that run on them. Since switching from PHP-based Wordpress to Ghost, which is run on node.js, I've been putting off monitoring both node.js and Nginx, primarily due to lack of time.

It seems that this was probably a wise choice. Setting up an app monitor for node.js was easy enough, but the Nginx monitor was a completely different matter.

When I apt-get a file for install on my Ubuntu VPS, I don't always check the repo it's coming from as closely as I should, especially when I'm doing a quick-and-dirty installation of something on-the-fly. Doing this may lead you to install an app on your box that is from a third-party instead of the original developer. The drawback comes in when you try to get support from the actual developer and you get so far until you're told they don't support third-party repos.

This is exactly what happened when I tried to get support for the installation of the newrelic_nginx_agent daemon. My problem was that I couldn't get the stub to receive any data. Where the problem lies is in the description of the process from several of the blogs I found online stating to add the stub code to your nginx configuration file.

To fully explain, I'll need to back up a bit. The dichotomy between an installation of Nginx on Debian/Ubuntu Linux and other distros is quite glaring. When you run a simple sudo apt-get install nginx you are not getting the original developer's repo for Nginx. Despite running Nginx for some time as my web server under Ubuntu, I had never had a problem until I tried to get support for the installation of the monitoring daemon.

For the record, the nginx configuration file that all instructions will point you to is located at /etc/nginx/nginx.conf. Sometimes you'll find instructions that will have you modify the default config file under sites-available or sites-enabled. This can be confusing and since the code snippet you need to add looks like something that would go under /sites-available/default, it's extremely misleading.

Here was my process for correcting this massive clusterfuck:

  1. Obtain your license key and download the .tar.gz package.
  2. Unpack the directory on your server in the directory of your choice (/etc, /root).
  3. You may need to run bundle install to obtain any Ruby pre-requisites the daemon relies upon.

Editing the newrelic_plugin.yml file

Use vi to edit the config/newrelic_plugin.yml.in file and rename it to newrelic_plugin_yml. Be sure to enter the license key that you should have previously obtained during the download portion of the process. You'll also need to modify the the Agent Configuration to your specific details. Here's a copy of what I have:

agents:  
  nginx_status_agent:
      -
        instance_name: nginx
        status_url: http://localhost:8080/nginx_stub_status

The instance name is the name the data will display with on New Relic.

Status_url is the internal URL that the daemon will use to test. I put mine on port 8080 so that it wouldn't conflict with main http port 80.

Adding the Stub to nginx.conf

The next step in the process is to add the necessary code for the stub in your /etc/nginx/nginx.conf file. Below is what I've added to my file and from what I can tell this stub is pretty standard, requiring no customization.

    server {

    listen 127.0.0.1:8080; #IPv4
    server_name localhost;
    location /nginx_stub_status {
        stub_status on;
        allow 127.0.0.1;
        deny all;
        }
    }

It took me a bit to figure out this goes under nginx.conf and not my /sites-available/default config file. The information I found online didn't make this distinction and it caused me about 30 minutes of headache.

Instructions for Debian-based distros will emplore you to use the symbolic link method between sites-available and sites-enabled to reduce the amount of configuration files you need to edit (especially helpful if you manage more than one site with your Nginx install). From the information I've read, it seems that the default config for other distros is to use a .conf file under /etc/nginx/conf.d. This was a large part of the confusion. When most of the instructions I found instructed me to add code to the nginx configuration file, it leads you to believe it's located under /etc/nginx/conf.d. In my case, since I was using a symbolic link between available and enabled, it was the config file simply named default. Learn from my mistake: edit the /etc/nginx/nginx.conf file.

Final Steps

The final few steps should be to test the Nginx configuration with /usr/sbin/nginx -t. If you receive the following output, you can service nginx restart.

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok  
nginx: configuration file /etc/nginx/nginx.conf test is successful  

The most important step is to remember to test the stub to ensure it's listening.

To fire up the daemon with standard output on the command line, cd into the directory you expanded the .tar.gz file (in my case, it was /root/newrelic_nginx_agent) and run ./newrelic_nginx_agent. If you receive output back ending with INFO: Gathered 2 statistics from 1 components, you know the stub is receiving data and passing it on to New Relic.

If you get any kind of error or it tells you it gathered 0 stats, you can run netstat -anp | grep 8080 to see if there's anything listening to port 8080. You should see output to the effect of:

tcp  0    0    127.0.0.1:8080     0.0.0.0:*       LISTEN 1931/nginx  
tcp  0    0    127.0.0.1:8080     127.0.0.1:54314 TIME_WAIT  -  

A null return indicates that something somewhere went wrong and you'll need to retrace your steps to correct the problem. The problem I had when nothing was listening to port 8080 was solved when I added the stub code to the nginx.conf file. Restarting all the affected services corrected this problem. As long as all your standard outputs come back normal, you should see the Nginx plugin in your New Relic menu:

Happy monitoring!

Update: If you’re interested in knowing how I solved my woes with the third-party Nginx repository I was pulling from, you can read about that here.