Varnish - An Introduction

General Information and History of Varnish

Varnish combines the functionalities of a reverse proxy, HTTP load balancer, and cache for data delivered via HTTP.
Development started around 2006 with version 0.9 as the first release and has now reached version 5.1.

Varnish is available both as a free Community Edition (BSD license) and an Enterprise Edition with various additional features for HA, SSL, and Web GUI, for example. With fastly, there is even a CDN provider that is built entirely on Varnish.

The Community Edition cannot terminate HTTPS requests, which is why an SSL terminator such as Nginx or Caddy is required to forward requests to Varnish as HTTP requests. Naturally, an existing appliance or other available infrastructure can also be used for this purpose. There will be no SSL support in the free version, a decision which is explained in detail by Poul-Henning Kamp, one of the lead developers.

Internally, Varnish operates according to the following simplified principle (details):

1. A request is accepted by varnishd and assigned an ID
2. This request is forwarded to a worker thread using the current VCL
3. The request is processed accordingly and sent to the backend if no cached entry is available
4. The backend’s response is received by varnishd, also assigned an ID, and processed according to the VCL
5. The response is then sent from varnishd to the client

Requests via HTTPS follow the path through the SSL proxy accordingly.

More on the term VCL in the corresponding section later.

Why Varnish?

There are several alternatives to Varnish, including Nginx, Apache, and HAProxy. As usual, personal requirements, preferences, and, of course, expertise are the deciding factors.

However, Varnish is specialized in its field and offers several features that are difficult to implement in the alternatives.

Examples include the seamless switching of configurations at runtime (in both directions), extensive debugging options in the request flow, and flexible configuration via VCL.

Configuring a large number of redirects or URL manipulations in Nginx, for instance, can quickly become confusing and may only be maintainable with the appropriate expertise.

Ultimately, as is so often the case, the project requirements and respective possibilities must be examined in advance.

Installation

Varnish is included in a specific version in the repositories of most Linux distributions.

An overview of the available versions for manual installation can be found here.

More detailed explanations regarding installation can be found here

Configuration

Varnish consists of several components, which must also be configured separately.

varnishd

The daemon that receives and processes requests. The general configuration of the daemon is handled either within the systemd service file or in a separate file, such as /etc/defaults/varnish. This is where settings like the number of worker threads, RAM limits, or daemon ports are configured.

VCL (Request Processing)

The processing of requests or the configuration of backends and their health checks are performed in a separate configuration. This is a VCL file (initially default.vcl, which is loaded when the service starts, e.g., under /etc/varnish/default.vcl). VCL is a Varnish-internal scripting language that describes to the service how to handle requests. This is automatically compiled into C code upon loading and made available to the varnishd daemon. It can also be changed or completely replaced at runtime.

More on this later.

Varnishncsa

The logging daemon for Varnish, similar to a web server’s access log. This runs independently of varnishd and must therefore be started separately. Configuration is performed either in the corresponding systemd service file or a separate file such as /etc/defaults/varnishncsa. The log format can also be adjusted to match the familiar Apache format, for example.

VCL

VCL (Varnish Configuration Language) is a Varnish-internal language used to control request handling.
This is defined in several sub-functions with the vcl_ prefix (built-ins).
You can, of course, add your own functions to the configuration, but these must be called within the corresponding built-ins.
Some configurations, such as the definition of backends, take place outside of the built-ins. More on this in the example below.
A complete VCL reference can be found here, along with a detailed description of the built-ins here.

A VCL configuration can be very extensive. Instead of including and commenting on it in its entirety here, please refer to
the example VCL by Matthias Geniar.

The configuration listed there includes basic entries such as backend and ACL definitions, as well as several additional functions and request manipulations (WebSocket support, URI manipulation, ESI, and more). Of course, this is only a general configuration. For production purposes, you should check which requirements are needed and how they are to be met.

Not included in that example, but a common configuration: If Nginx is used as an SSL terminator, it can simultaneously function as an emergency backend to deliver a general maintenance page, for instance, should none of the regular backends be available. This would prevent error messages from being displayed to visitors.

VMODs

Varnish offers the possibility to integrate custom C code within the VCL. However, this is very cumbersome, which is why VMODs exist. These are additional modules implemented in C that can extend the functions of Varnish or its VCL.
Varnish provides some standard modules, such as std or director, directly with the installation, but there are also many already published modules that extend functionality.

A list of the most common modules can be found here.

At this point, it is important to ensure that these are compatible with the installed Varnish version. Some of the most popular ones are likely GeoIP and BasicAuth.

A description of how to implement your own VMODs can be found here

CLI and other Binaries

The following is an overview of the most common binaries delivered with the Varnish package that simplify administration and/or debugging.

varnishadm

An interactive CLI that allows you to jump into a running Varnish instance to, for example, load new VCLs, get an overview of the storages used, view the varnishd configuration, or stop the instance. The CLI can also be used as a “one-shot command” to check a new VCL for compatibility. To do this, simply run “varnishd -C -f /etc/varnishd/default.vcl” (or the path to the file to be checked). If there are no problems, the VCL translated into C code is output; otherwise, a corresponding error message is displayed.
This way, you can always ensure whether the new configuration works at all. Functional errors are, of course, not checked here.

varnishlog

Varnishlog is a very powerful tool that allows you to view the current request flow from Varnish in real time. Since a large amount of data can be generated here, it is recommended to call varnishlog manually only for debugging purposes. It lists all request headers, timestamps, Varnish function calls, and methods. This makes it very easy to examine a specific request through its entire path from reception to the sending of the response.

Below are a few examples of commands:

• Logs all requests for the Host header example.com
  varnishlog -q ‘ReqHeader ~ “Host: example.com”‘
• Logs the URL and referer for all requests where the Host header matches (www\.)?example\.com.
  varnishlog -q ‘ReqHeader ~ “Host: (www\.)?example\.com”‘ -i “ReqURL” -I “ReqHeader:Referer:”

varnishstat

Displays all important internal data of the corresponding Varnish instance in a top-like overview. This includes, for example, the total number of requests since startup, backend availability over the last few minutes, cache hits/misses, and much more. There is also a one-shot function that outputs a snapshot of the data, which can be used for monitoring purposes.

varnishtop

This is a top-like overview of the varnishlog. It displays the entries most frequently appearing in the log in descending order of occurrence.

An example:

3.00 ReqHeader Cookie:
3.00 ReqUnset Cookie:
0.75 ReqURL /
0.75 BerespProtocol HTTP/1.1
0.75 VCL_return fetch
0.75 VCL_return deliver
0.75 ReqHeader Host: localhost

Further Topics

The possibilities with Varnish are very diverse, and covering them all in one article would exceed the scope. If you have any questions or requests regarding specific topics that should be explored in more depth in an article, please leave them in the comments.