Bot Events on Fastly VCL

If you run a Fastly VCL (Delivery) service and don’t want to stand up a Compute service, you can still see who is crawling your site. A VCL snippet builds one JSON line per request and a log-streaming endpoint ships it to Supertab, where it is classified the same way as events from the SDK. The snippet runs in vcl_deliver, after the response has already gone to the visitor. It makes no network calls, adds no origin fetches, and does not affect caching.

This gives you analytics only — no CAP enforcement. Requests are observed and labelled, never blocked or challenged. Acting on a license needs the SDK in a Compute service; see Connect on Fastly. A VCL service can also serve your RSL license without the SDK — see Serving the license on VCL.

What gets recorded

Per request: timestamp, a Fastly request id, client IP, user agent, path, method, Accept-Language, host, response status, country, network (ASN and operator), the Sec-Fetch-* headers, HTTP Message Signature headers when present, which standard headers were sent, whether a cookie was sent, and three measurements of the query string. Deliberately not recorded — meaning never written to the log line Supertab receives. Your traffic is served exactly as before, so anything below still reaches your own origin as part of the normal request:

Not recorded	What is kept instead
Cookie values	Whether a `Cookie` header was present — a true/false. The contents are never read.
The query string	Its length, its parameter count, and whether it matches known exploit patterns. The snippet reads the query, derives those three numbers in your POP, and never writes the characters to the log line.
The `Referer` value	Only that a referer was sent, as a name in the header list. A referer URL can carry another site’s query string.
Any other header value	Header names only. `Authorization`, for example, is recorded as present, never as its value.
Request and response bodies	Nothing.

Before you begin

You need permission to edit and activate the service, and three values:

Value	Where to get it
Website URN	View Website Details in your dashboard, next to your API keys
S3 access key	Sent to you by Supertab
S3 secret key	Sent to you by Supertab

Set it up

Clone the active version

Open the service, click Edit configuration, and clone the active version. Everything below happens on that draft and goes live only when you activate it.

Create the VCL snippet

Go to VCL Snippets → Create snippet:

Setting	Value
Name	`bot_event`
Type	`Regular`
Subroutine	`deliver (vcl_deliver)`
Priority	`100`

The subroutine must be deliver. The snippet reads the response status, which does not exist in recv — the default — so choosing recv fails to compile.

Paste the snippet

Paste the code below into the snippet’s VCL field.

Replace YOUR_WEBSITE_URN with your Website URN. It appears once, marked by a ==== banner near the top of the snippet. Keep the %22 quotes around it.Paste the whole value including the urn: prefix — the placeholder is the entire URN, not just the id:urn:stc:merchant:system:3f9a1c72-8b04-4e15-9c7d-2a6f0e4b1d83Getting this wrong raises no error. The snippet compiles, log files are written, and your dashboard stays empty.

# --- Supertab bot-event logging -----------------------------------------------
# Builds one JSON line describing this request into a request header,
# X-Bot-Event; the "bot_events" log endpoint writes out that header and nothing
# else. It runs after the response has gone to the visitor, makes no network
# calls, and does not affect caching.
#
# It must run in "deliver" — it reads the response status, which does not exist
# earlier in the request lifecycle.
#
# VCL strings cannot hold a literal double quote, so every quote the JSON needs
# is written as %22. Long strings ({"..."}) appear only where a % must pass
# through untouched. One statement per line — VCL cannot continue one.
#
# If you use shielding, this VCL runs at both the edge and the shield POP. The
# check below builds the line only at the edge, so requests are not counted
# twice.

# Never log a value that came from the client: the log endpoint runs on the
# shield leg too, where this header still holds whatever the caller sent.
unset req.http.X-Bot-Event;

if (fastly.ff.visits_this_service == 0) {
  declare local var.line   STRING;
  declare local var.ip     STRING;
  declare local var.hdrs   STRING;
  declare local var.qs     STRING;
  declare local var.pcount INTEGER;

  # Record the client IP in one consistent form: IPv4 written as IPv6-mapped
  # (::ffff:1.2.3.4), IPv6 as-is, "::" when there is no address.
  set var.ip = client.ip;
  if (var.ip ~ {"^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$"}) {
    set var.ip = "::ffff:" var.ip;
  }
  if (var.ip == "") { set var.ip = "::"; }

  # Which standard headers the visitor sent — NAMES ONLY, never values. VCL
  # cannot list headers, so each is checked individually. The list is broad on
  # purpose: a header not checked here is never recorded. Which headers a client
  # sends is one of the strongest tells of a real browser versus software
  # imitating one.
  set var.hdrs = "";
  if (req.http.Accept)                    { set var.hdrs = var.hdrs "%22accept%22,"; }
  if (req.http.Accept-Encoding)           { set var.hdrs = var.hdrs "%22accept-encoding%22,"; }
  if (req.http.Accept-Language)           { set var.hdrs = var.hdrs "%22accept-language%22,"; }
  if (req.http.Authorization)             { set var.hdrs = var.hdrs "%22authorization%22,"; }
  if (req.http.Cache-Control)             { set var.hdrs = var.hdrs "%22cache-control%22,"; }
  if (req.http.Cookie)                    { set var.hdrs = var.hdrs "%22cookie%22,"; }
  if (req.http.DNT)                       { set var.hdrs = var.hdrs "%22dnt%22,"; }
  if (req.http.From)                      { set var.hdrs = var.hdrs "%22from%22,"; }
  if (req.http.If-Modified-Since)         { set var.hdrs = var.hdrs "%22if-modified-since%22,"; }
  if (req.http.If-None-Match)             { set var.hdrs = var.hdrs "%22if-none-match%22,"; }
  if (req.http.Origin)                    { set var.hdrs = var.hdrs "%22origin%22,"; }
  if (req.http.Pragma)                    { set var.hdrs = var.hdrs "%22pragma%22,"; }
  if (req.http.Priority)                  { set var.hdrs = var.hdrs "%22priority%22,"; }
  if (req.http.Range)                     { set var.hdrs = var.hdrs "%22range%22,"; }
  if (req.http.Referer)                   { set var.hdrs = var.hdrs "%22referer%22,"; }
  if (req.http.Sec-CH-UA)                 { set var.hdrs = var.hdrs "%22sec-ch-ua%22,"; }
  if (req.http.Sec-CH-UA-Mobile)          { set var.hdrs = var.hdrs "%22sec-ch-ua-mobile%22,"; }
  if (req.http.Sec-CH-UA-Platform)        { set var.hdrs = var.hdrs "%22sec-ch-ua-platform%22,"; }
  if (req.http.Sec-Fetch-Dest)            { set var.hdrs = var.hdrs "%22sec-fetch-dest%22,"; }
  if (req.http.Sec-Fetch-Mode)            { set var.hdrs = var.hdrs "%22sec-fetch-mode%22,"; }
  if (req.http.Sec-Fetch-Site)            { set var.hdrs = var.hdrs "%22sec-fetch-site%22,"; }
  if (req.http.Sec-Fetch-User)            { set var.hdrs = var.hdrs "%22sec-fetch-user%22,"; }
  if (req.http.Signature)                 { set var.hdrs = var.hdrs "%22signature%22,"; }
  if (req.http.Signature-Agent)           { set var.hdrs = var.hdrs "%22signature-agent%22,"; }
  if (req.http.Signature-Input)           { set var.hdrs = var.hdrs "%22signature-input%22,"; }
  if (req.http.TE)                        { set var.hdrs = var.hdrs "%22te%22,"; }
  if (req.http.Upgrade-Insecure-Requests) { set var.hdrs = var.hdrs "%22upgrade-insecure-requests%22,"; }
  if (req.http.User-Agent)                { set var.hdrs = var.hdrs "%22user-agent%22,"; }
  if (req.http.X-Requested-With)          { set var.hdrs = var.hdrs "%22x-requested-with%22,"; }

  # ===========================================================================
  #  YOUR_WEBSITE_URN below is the only edit this snippet needs. Replace it
  #  with your Website URN — the whole value, starting "urn:" — keeping the
  #  %22 quotes around it.
  # ===========================================================================
  set var.line = "{%22merchant_system_urn%22:%22YOUR_WEBSITE_URN%22";
  set var.line = var.line ",%22schema_version%22:3";
  set var.line = var.line ",%22source_cdn%22:%22fastly_vcl%22";
  set var.line = var.line ",%22status_source%22:%22observed%22";
  set var.line = var.line ",%22client_ip_source%22:%22cdn_declared%22";

  # REQUIRED — do not remove these four.
  # They describe a licence-enforcement decision. This configuration makes no
  # such decision (it only observes), so they are fixed values saying exactly
  # that: no token was checked, nothing was blocked, enforcement is off.
  #
  # They look optional and are not. A record that leaves any of them out is
  # rejected in full and discarded — you would see traffic being logged and no
  # data ever arriving.
  set var.line = var.line ",%22has_token%22:false";
  set var.line = var.line ",%22token_outcome%22:%22not_validated%22";
  set var.line = var.line ",%22final_action%22:%22allow%22";
  set var.line = var.line ",%22enforcement_mode%22:%22disabled%22";

  # The format is a long string so that % reaches strftime untouched.
  set var.line = var.line ",%22timestamp%22:%22" strftime({"%Y-%m-%dT%H:%M:%S"}, time.start);
  set var.line = var.line "." time.start.msec_frac "Z%22";

  # Text fields. These are always present, and empty when the header was absent.
  set var.line = var.line ",%22request_id%22:%22" req.xid "%22";
  set var.line = var.line ",%22client_ip%22:%22" var.ip "%22";
  set var.line = var.line ",%22user_agent%22:%22" json.escape(req.http.User-Agent) "%22";
  set var.line = var.line ",%22path%22:%22" json.escape(req.url.path) "%22";
  set var.line = var.line ",%22method%22:%22" req.method "%22";
  # The Referer value is deliberately not recorded — a referer URL can carry
  # another site's query string. Whether one was sent is still noted, as a name
  # in the header list above. The field itself must stay present and empty.
  set var.line = var.line ",%22referer%22:%22%22";
  set var.line = var.line ",%22accept_language%22:%22" json.escape(req.http.Accept-Language) "%22";
  set var.line = var.line ",%22host%22:%22" json.escape(req.http.Host) "%22";

  # The status actually served to the visitor. Available because this runs in
  # deliver, after the response exists.
  set var.line = var.line ",%22status_code%22:" resp.status;

  # --- Optional fields.
  # When a value is not available these MUST be null, never an empty string. An
  # empty string would be read as a real answer rather than "not known", which
  # silently changes how the request is classified.
  if (client.geo.country_code && client.geo.country_code != "--") {
    set var.line = var.line ",%22request_country%22:%22" client.geo.country_code "%22";
  } else {
    set var.line = var.line ",%22request_country%22:null";
  }

  if (client.as.number > 0) {
    set var.line = var.line ",%22request_asn%22:" client.as.number;
  } else {
    set var.line = var.line ",%22request_asn%22:null";
  }

  if (client.as.name && client.as.name != "unknown") {
    set var.line = var.line ",%22as_organization%22:%22" json.escape(client.as.name) "%22";
  } else {
    set var.line = var.line ",%22as_organization%22:null";
  }

  if (req.http.Sec-Fetch-Mode) {
    set var.line = var.line ",%22sec_fetch_mode%22:%22" json.escape(req.http.Sec-Fetch-Mode) "%22";
  } else {
    set var.line = var.line ",%22sec_fetch_mode%22:null";
  }

  if (req.http.Sec-Fetch-Site) {
    set var.line = var.line ",%22sec_fetch_site%22:%22" json.escape(req.http.Sec-Fetch-Site) "%22";
  } else {
    set var.line = var.line ",%22sec_fetch_site%22:null";
  }

  if (req.http.Sec-Fetch-Dest) {
    set var.line = var.line ",%22sec_fetch_dest%22:%22" json.escape(req.http.Sec-Fetch-Dest) "%22";
  } else {
    set var.line = var.line ",%22sec_fetch_dest%22:null";
  }

  if (req.http.Sec-Fetch-User) {
    set var.line = var.line ",%22sec_fetch_user%22:%22" json.escape(req.http.Sec-Fetch-User) "%22";
  } else {
    set var.line = var.line ",%22sec_fetch_user%22:null";
  }

  # HTTP Message Signature headers (RFC 9421), recorded exactly as sent. Only
  # well-behaved agents that cryptographically sign their requests send these;
  # they are how such an agent proves it is who it claims to be.
  if (req.http.Signature-Agent) {
    set var.line = var.line ",%22signature_agent%22:%22" json.escape(req.http.Signature-Agent) "%22";
  } else {
    set var.line = var.line ",%22signature_agent%22:null";
  }

  if (req.http.Signature-Input) {
    set var.line = var.line ",%22signature_input%22:%22" json.escape(req.http.Signature-Input) "%22";
  } else {
    set var.line = var.line ",%22signature_input%22:null";
  }

  if (req.http.Signature) {
    set var.line = var.line ",%22signature%22:%22" json.escape(req.http.Signature) "%22";
  } else {
    set var.line = var.line ",%22signature%22:null";
  }

  if (req.http.Cookie) {
    set var.line = var.line ",%22has_cookies%22:true";
  } else {
    set var.line = var.line ",%22has_cookies%22:false";
  }

  set var.line = var.line ",%22header_names%22:[" regsub(var.hdrs, ",$", "") "]";

  # Query string: measurements only. The query string itself is never recorded —
  # only how long it was, how many parameters it had, and whether it looks like
  # an attack. The characters are discarded here and never leave your POP.
  set var.qs = req.url.qs;
  if (var.qs == "") {
    set var.line = var.line ",%22query_length%22:0";
    set var.line = var.line ",%22query_param_count%22:0";
    set var.line = var.line ",%22query_suspicious%22:false";
  } else {
    # `+` is concatenation in VCL, not addition — arithmetic needs the `+=` form.
    set var.pcount = std.strlen(regsuball(var.qs, "[^&]", ""));
    set var.pcount += 1;
    set var.line = var.line ",%22query_length%22:" std.strlen(var.qs);
    set var.line = var.line ",%22query_param_count%22:" var.pcount;
    # Known exploit patterns, checked against the raw and URL-decoded query.
    if (var.qs ~ {"(?i)(\.\./|union select|<script|onerror=|/etc/passwd)"} || urldecode(var.qs) ~ {"(?i)(\.\./|union select|<script|onerror=|/etc/passwd)"}) {
      set var.line = var.line ",%22query_suspicious%22:true";
    } else {
      set var.line = var.line ",%22query_suspicious%22:false";
    }
  }

  set var.line = var.line "}";

  # The log endpoint reads this request header. It is set in deliver, after the
  # origin fetch has already happened, so it is never sent to your origin.
  set req.http.X-Bot-Event = var.line;

}

Create the logging endpoint

Go to Resources → Log streaming → Create endpoint → Amazon S3:

Setting	Value
Name	`bot_events`
Placement	`Format Version Default`
Log format	`%{req.http.X-Bot-Event}V`
Access method	`User credentials`
Access key / Secret key	provided by Supertab
Bucket name	`lpeu-prod-connect-bot-events`
Path	`bot-events/`
Domain	`s3.eu-central-1.amazonaws.com`
Period	`60` while setting up

Under Advanced options, set Log line format to Blank and leave compression off.Four of these are easy to get wrong, and none of them raise an error:

Log format replaces the prefilled JSON template entirely — delete it. The snippet already builds the JSON; this field only emits it.
Log line format must be Blank. The Classic default prepends a syslog header and corrupts every line.
Compression must be off, or the connector won’t match the *.log objects.
Path must be exactly bot-events/, trailing slash included — not a subfolder, not your company name.

Use Period 60 while setting up so you find out quickly whether it works, then raise it to 900 (15 minutes) once we’ve confirmed data is arriving. The longer interval batches events into fewer, larger files.

Activate

Activate the version. The snippet and the logging endpoint go live together.

Confirm it’s working

Send some traffic, then allow 15 minutes before checking. Logging configuration takes a few minutes to reach every POP, Fastly writes one file per Period, and it buffers per cache node — so events arrive as a trickle rather than all at once. Requests served before the log endpoint reaches a POP are lost rather than delayed. Then open your dashboard. Crawlers that hit your site in that window appear there, classified by operator. If it’s still empty, work through the table below.

Troubleshooting

Misconfigurations here fail quietly rather than erroring, so work down the list in order.

Symptom	Check
Nothing after 15 minutes	Was the version activated? A cloned draft serves nothing.
Still nothing	If your service uses custom VCL, does `main.vcl` still contain the `#FASTLY deliver` and `#FASTLY log` macros? Fastly inserts snippets and the log statement at those markers.
Still nothing	Log line format `Blank`, compression off, Path exactly `bot-events/`, Bucket exactly `lpeu-prod-connect-bot-events`.
Still nothing	Access key and secret pasted without whitespace. Fastly does not validate them.
Files are written but your dashboard is empty	The URN in the snippet — it must be your full Website URN, with no duplicated `urn:stc:merchant:system:` prefix.
Data arrives but looks wrong	Send us one example request and what you expected. Don’t edit the snippet’s field values to compensate — several look optional and are not, and removing one causes the whole record to be discarded.

Every request is logged, including cache hits. With shielding enabled the line is built only at the edge POP, so requests are not counted twice.

Connect on Fastly

Serve your RSL license and enforce CAP on Fastly, with Compute or by chaining VCL to a validator.

Deploy at the Edge

CDN-agnostic guide covering RSL serving, CAP enforcement, and robots.txt.

Introduction

Guides

Licensing

Reference

What gets recorded

Before you begin

Set it up

Confirm it’s working

Troubleshooting

Connect on Fastly

Deploy at the Edge

​What gets recorded

​Before you begin

​Set it up

​Confirm it’s working

​Troubleshooting

​Related Docs

Connect on Fastly

Deploy at the Edge

What gets recorded

Before you begin

Set it up

Confirm it’s working

Troubleshooting

Related Docs