Varnish + Apache and HTTPS

If you’ve been keeping up with web programming and web technology you’ll have heard people talk about something called Nginx. Nginx is a reverse proxy that also doubles up as a web-server. It’s pretty fast. So I’ve been told – I actually haven’t used it but the benchmarks look exciting. Unfortunately I’m not ready to make that switch, Apache has served me (pun intended?) well over the years, it’s the grand father of web servers. It’s not the fastest by any means but it’s the most mature and feature packed out there. Where Apache specifically is let down is serving static content along side processing PHP pages, the PHP Apache module bogs down the whole run time of Apache*, and makes it a bit inefficient when processing static requests. When I say static requests, I’m talking content that doesn’t change – images, stylesheets, javascript and icons.

So what can we do to increase Apache’s serving of static content, and let Apache deal with generating the pages dynamically with PHP?

Enter Varnish. Varnish is according to their website:

Varnish is a web application accelerator.

Technically speaking it is a reverse proxy cache. I’ll explain…

When you think of proxies you might think back to your office or school internet system which had a proxy filtering out the naughty content from the web. Well proxies also served another purpose – to cache content for the network. This means every time someone goes to Google on the network it doesn’t have to make a request through to the Google servers, it can use it’s cached version instead, speeding the whole process up. So I guess by definition a reverse proxy works the other way around. Rather than caching content from the web it instead caches the content from your website, and sends it back to your visitor’s browser rather than let the web server handle the request.

Varnish does exactly this. What makes Varnish particularly exciting is the way it caches the data. It does it in memory. For those that don’t know, memory is faster than reading from your local hard-disk. It why other tools like Memcached are soo popular also. So when you think about it; Varnish is quite a simple concept. I’ve particularly noticed a decrease in page load times on legacy websites where there are a lot of images and CSS sprites. When I say noticeable, I mean a couple of seconds – which in page load times is quite significant.

 

Installing for standard HTTP

To install Varnish you can look at the documents here, but essentially it’s a simple Apt-Get on a Ubuntu server.

The confusing part is getting your head around the port forwarding. You can’t have 2 things bound to the same port. Something has to give and since we need to hit Varnish rather than Apache from our web browser, Varnish wins the coin toss for port 80. This means we need to run Apache on a different port number. The default configuration on Varnish, is to serve content from Varnish from port 8080 and talk to the Apache web server running on port 80. We need to reverse this situation.

If you’re running Ubuntu you need to look at your:

/etc/apache2/ports.conf

Change this to use 8080 instead of port 80.

Then you’ll want to update all your VirtualHost configurations for Apache to use port 8080. The line should look like this

<VirtualHost *:8080>

Now you’ll need to change Varnish to run on port 80. Varnish’s configuration is stored in:

/etc/vagrant/default.vcl

In side here we’ll need to configure the back ends. In Varnish’s terminology backend means your Apache web server. You’ll need to have something like this:

backend default{
   .host = "127.0.0.1";
   .port = "80";
}

This only tells Varnish where to find Apache, we now need to bind Varnish to port 80 so our web browser can talk to Varnish. This is stored in:

/etc/default/varnish

In side here is a line that begins with

"DAEMON_OPTS="-a :8080 \"

This line contains the parameters that Vagrant is launched with at system boot and when you call the service program. Here we need to change the :8080 to just :80.

This should be enough configuration to get going with Varnish in a production environment. You’ll now want to restart both Apache and Varnish via

sudo service apache2 restart && sudo service varnish restart

Using Varnish with HTTPS

One draw back with Varnish is that is doesn’t understand SSL encrypted requests. This means HTTPS is alien to it, it doesn’t know what is inside each packet because it’s encrypted, and therefore it can’t figure out if the content being passed between is suitable for it to cache.

So to circumvent this limitation we can introduce Pound. Pound is a reverse proxy similar to Varnish except it’s focus is more on load balancing – directing traffic to the right places. Pound like Varnish is pretty simple to setup on Ubuntu:

sudo apt-get install pound

Like I said Pound is intended to route requests to the various places it needs to. Think of it like Apache rewrite rules that redirect requests to servers rather than pages/scripts. We’ll need Pound to send requests to Varnish when it receives them, and for Varnish to forward the request onto Apache or use it’s own cache. Again for this to work we need to bind Pound to port 80 for our web browser, and since we’re dealing with HTTPS/SSL we’ll need to bind to 443 also.

Once installed Pound’s configuration lives in:

/etc/pound/pound.cfg

We’ll need to look for/create a block of configuration that looks like:

ListenHTTP
	Address 127.0.0.1 ## this needs to be the external IP address of the server.
	Port	80        ## this needs to be 80 for web browsers to connect to.

	## allow PUT and DELETE also (by default only GET, POST and HEAD)?:
	xHTTP		0

	Service
		BackEnd
			Address	127.0.0.1
			Port	8080       ## this needs to be our Varnish port number for HTTP connections
		End
	End
End

Here we’ll need to change the first Address directive from 127.0.0.1 to the external IP address of the server (the one your domain resolves to) this way Pound can handle requests coming to your domain(s). The port number then needs to be changed to 80 so it’s listening to HTTP traffic. The backend configuration block is going to point to our Varnish installation. We’ll configure that in just a second, but we can assume that port number is going to be 9080. Now because we’re going to be dealing with SSL we need to include an extra configuration block to handle incoming HTTPS requests to Pound. The block should look something like this:

ListenHTTPS
	HeadRemove "X-Forwarded-Proto"
	AddHeader "X-Forwarded-Proto: https"
	Address 127.0.0.1 ## this needs to be the external IP address of the server.
	Port 443          ## this needs to be 443 to listen for HTTPS connections from browsers
	xHTTP		0
	Cert "/etc/apache2/ssl/pound.pem" ## this needs to be your SSL certificate
	Service
		Backend
			Address 127.0.0.1
			Port 9443 ## this needs to be the Varnish HTTPS port number
		End
	End
End

As you can see above we’ve got some extra configuration there. We’ve added in two new pieces of configuration at the top, which remove the X-Forwarded-Proto header from the request and then replaces it with another one telling Varnish this is a HTTPS request. This is particularly important, like I said earlier Varnish can’t understand SSL encrypted connections, so the connection between Pound and Varnish will be over HTTP unencrypted. We can get away with this as the connection between the user’s browser and the server is encrypted, its only where it’s going over the loopback interface of the server it’ll be unencrypted.

The other thing we’ve added is our SSL certificate. This is a pem file which took me ages to figure out how to do. I’m not an SSL expert and don’t fully understand certification. Basically the pem file is a concatenation of your certificate (crt file) and your key. Doing this step wrong will result in Pound failing start with the error message “SSL_CTX_use_PrivateKey_file Error“. You need to concatenate the files in the right order for it to work, and make sure your key doesn’t have it’s password on the file. This thread over at the Pound mailing list has some information on this, but for reference you should have it in this order as detailed here. see: http://www.project-open.org/en/howto_pound_https_configuration

Again the backend block needs to refer to our Varnish configuration that is going to handle/forward our HTTPS connections. Here we’ve assumed it’s 9443.

Next we’ll update our Varnish configuration so it’s listening on a different port from Pound. Open:

/etc/default/varnish

You’ll notice we’ve basically put both port 80 and 433 in the 9000 range, this is just a convention so we know 9000 ports are what Varnish is listening on. So we’ll need to change the :80 or :8080 to :9080. We’ll also want Varnish to handle (spoof) HTTPS requests, so we’ll need it to listen on another port for our SSL traffic – this port is 9443. Adding a second port number can be done by using a comma separator. Make sure you don’t forget the colon in-front of the port number – I spent an hour pulling my hair out trying to diagnose this earlier. The configuration should look like this:

"DAEMON_OPTS="-a :9080,:9443 \"

Next we’ll need to make sure Apache isn’t bound to port 443, as we now need this port to go to Pound and then to Varnish. Again open:

/etc/apache2/ports.conf

You’ll need to change 443 to something else, it’s worth checking port 80 isn’t being used as well. We’re going to use the 8000′s here so HTTP traffic should be 8080 and HTTPS should be 8443. Again we’ll need to change over all our VirtualHost directives to listen for the new HTTPS and HTTP ports.

<VirtualHost *:8443>

Now we need to make sure Varnish’s backend reflect our new Apache port numbers, as we only configured what Varnish is listening on before not where it’s forwarding it’s requests. So lets open (again):

/etc/varnish/default.vcl

We’ve already got our HTTP backend defined, now we need to define a new backend for HTTPS requests. Ours will look like this:

backend default_ssl{
    .host = "127.0.0.1";
    .port = "8443";
}

Although we’ve got 2 backends we need to make sure Varnish knows which one to use for which connection it receives. We need to add this small block of logic which defines which backend to use based on the port number it receives the connection on:

sub vcl_recv {
  # Set the director to cycle between web servers.
  if (server.port == 9443) {
    set req.backend = default_ssl;
  }
  else {
   set req.backend = default;
  }
}

We’ll also need to make sure our default’s backend is up to date too, it should be forwarding to 8080.

At this point we should be done with configuring Pound, Varnish and Apache. The only thing left to do is tweak our Apache VirtualHost configuration so it’s speaking plain unencrypted HTTP to Varnish. Remember we sent through that X-Forwarded-Proto header? Well this is where that comes in handy! In our VirtualHost configuration we can comment out our SSL directives, as we don’t want to encrypt anything going through Varnish as Pound will do that for us!

Next however we need to spoof the fact this is a HTTPS connection. This is important because although your browser is visiting the site in HTTPS, Apache won’t know it’s HTTPS and serve all images and CSS as HTTP. This causes all kinds of warnings and sirens in the browser and the result is it breaks the page. We therefore need to add this following directive our side the VirtualHost:

SetEnvIf X-Forwarded-Proto "^https$" HTTPS=on

Our final VirtualHost Configuration should look like:

SetEnvIf X-Forwarded-Proto "^https$" HTTPS=on
<VirtualHost *:8443>
    ServerName www.somesite.com
    DocumentRoot /var/www/site

#   SSLEngine On
#   SSLCertificateFile /etc/apache2/ssl/certificate.crt
#   SSLCertificateKeyFile /etc/apache2/ssl/certificate.key
#   SSLCertificateChainFile /etc/apache2/ssl/certificate.pem
#   SSLVerifyClient None

    <Directory /var/www/site>
        AllowOveride None ## Disable .htaccess
    </Directory>
    
&lt/VirtualHost>

One last thing to do before starting Pound. You’ll need to enable to configuration from the service configuration. You need to add setup=1 in:

/etc/default/pound

Now everything should be configured you just need to reboot all your services.

sudo service apache2 restart
sudo service varnish restart
sudo service pound restart

Problems and Troubleshooting

  • If you have a problem starting Pound because of SSL_CTX_use_PrivateKey_file you’ll need to check your certificate/pem file you’ve put in your Pound configuration.
  • If when you visit your website you see “Service is unavailable, please try again later” this means Pound couldn’t talk to Varnish, make sure you check your Pound configuration matches the ports listening in Varnish. See /etc/pound/pound.cfg and /etc/default/varnish.
  • If you get a 503 error when visiting your site, this is coming from Varnish, and usually means that Varnish is talking to Apache but Apache is sending back encrypted stuff when Varnish is expecting plain old HTTP. Make sure you remove your SSL directives from the VirtualHost and put the SetEnvIf directive on the X-Forwarded-Proto header
  • To diagnose problems, it might be useful to visit your website directly via Apache’s new ports to ensure Apache is working properly. Visit http://www.somesite.com:8443/.
  • Varnish comes with it’s own logging system that stores stuff in-memory, you just need to run
    varnishlog

    on the command line

  • Pound also logs stuff but it goes to syslog. So you’ll want to
    tail /var/log/syslog
  • Over course you always have your Apache logs at
    tail /var/log/apache2/access_log
  • Another tip is to use Curl to see what Headers are being sent in the request.
     curl -v http:/www.somesite.com/

I hope the above article helps someone trying to get a LAMP setup working with HTTPS over Varnish and Pound. It took me a few hours of playing around before I got everything all setup perfectly. It’s definitely worth the effort to get this set-up working as it means your website is much more scalable and configurable.

Thanks
Adam.

*feel free to correct me on that statement.

12 thoughts on “Varnish + Apache and HTTPS

  1. Adam Malone says:

    I’ve just been linked to this blog by a commenter on a Varnish/Pound article over on my site (https://www.adammalone.net/post/why-pound-awesome-front-varnish).

    My understanding is that you’re utilising different ports with both Pound and Varnish as an alternate to utilising different hash keys within Varnish to deal with mixed mode requests. Have you considered playing with vcl_hash at all or did you run into problems that I should likely be cautious of!

  2. Victor Augusto says:

    Have you tried this method with mod_spdy? When i run apache directly over https, mod_spdy works fine, but as soon as i put varnish+pound in the mix, https still works fine, but mod_spdy stop working, i think its because the # SSLEngine On

  3. Matt says:

    Thanks for the write-up. I’ve been looking how to use varnish and https for drupal.

  4. poltz says:

    There is a typo in the vcl congif, port 9443 should be 8443

  5. poltz says:

    Another one is port 433 — search for “433″

  6. Greg says:

    Pound is handling the ssl decryption at the outer most layer so why do we need to maintain distinct routing internally?

    • Adam says:

      This is true, we don’t really need to have 2 Varnish backends Pound can send everything to the one backend and should be able to handle it.

  7. ADDISON says:

    I would like to find out an technically explanation for why keeping SSL module enable in Apache.

    Well I set up correctly Pound in front of Varnish for dealing with SSL. In Apache I did the following changes:

    - removed 443 from listen in ports.conf
    - commented all SSL statement in virtual host file and apache2.conf like SSLProtocol and SSLCipherSuite)

    Everything works as expected with SSL protocol. Because Apache was absolved from any SSL tasks I wanted to disable SSL module. Once disable the whole configuration isn’t working. I had to enable it again.

    Why is need SSL module still needed in Apache2 when Pound is dealing with SSL certificate and ciphers and passing a unsecured connection through Varnish to Apache?

  8. sss says:

    in “/etc/default/pound”

    you said “setup=1″,
    should it be “startup=1″ ?

  9. Dan says:

    Thanks very much for writing this article, I needed Varnish to handle Apache and Node back-ends with SSL requirements, this is ideal!

  10. Ash Glover says:

    Hi Adam
    Thanks for this great info, Pound looks very good!

    One issue i’m having is with Varnish, i’ve set it up as you’ve outlined here, with the port numbers 9080 and 9443:

    varnishd -a :9080,:9443 -T XXX.X.X.X:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,4G

    but when i start up varnish, it fails with:

    varnishd[4908]: Error: Unknown protocol ‘:9443′

    any idea why this might be the case? Do i need to “tell” it what port 9443 is?

    Ash

  11. You have a typo in the varnish default.vlc file where you used “vagrant” instead of “varnish”. Common muscle memory mistake for me too.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>