> ## Documentation Index
> Fetch the complete documentation index at: https://connect-docs.supertab.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Bot Events on Fastly VCL

> Measure and classify bot traffic from a Fastly VCL service — no Compute service, no SDK.

If you run a Fastly **VCL (Delivery)** service and don't want to stand up a Compute service, you can still see who is crawling your site. A VCL snippet builds one JSON line per request and a log-streaming endpoint ships it to Supertab, where it is classified the same way as events from the SDK.

The snippet runs in `vcl_deliver`, after the response has already gone to the visitor. It makes no network calls, adds no origin fetches, and does not affect caching.

<Note>
  This gives you analytics only — **no CAP enforcement**. Requests are observed and labelled, never blocked or challenged. Acting on a license needs the SDK in a Compute service; see [Connect on Fastly](/reference/fastly/connect-on-fastly). A VCL service can also serve your RSL license without the SDK — see [Serving the license on VCL](/reference/fastly/connect-on-fastly#serving-the-license-on-vcl).
</Note>

## What gets recorded

Per request: timestamp, a Fastly request id, client IP, user agent, path, method, `Accept-Language`, host, response status, country, network (ASN and operator), the `Sec-Fetch-*` headers, HTTP Message Signature headers when present, which standard headers were sent, whether a cookie was sent, and three measurements of the query string.

Deliberately **not** recorded — meaning never written to the log line Supertab receives. Your traffic is served exactly as before, so anything below still reaches your own origin as part of the normal request:

| Not recorded                    | What is kept instead                                                                                                                                                                                   |
| :------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Cookie values**               | Whether a `Cookie` header was present — a true/false. The contents are never read.                                                                                                                     |
| **The query string**            | Its length, its parameter count, and whether it matches known exploit patterns. The snippet reads the query, derives those three numbers in your POP, and never writes the characters to the log line. |
| **The `Referer` value**         | Only that a referer was sent, as a name in the header list. A referer URL can carry another site's query string.                                                                                       |
| **Any other header value**      | Header *names* only. `Authorization`, for example, is recorded as present, never as its value.                                                                                                         |
| **Request and response bodies** | Nothing.                                                                                                                                                                                               |

## Before you begin

You need permission to edit and activate the service, and three values:

| Value             | Where to get it                                                   |
| :---------------- | :---------------------------------------------------------------- |
| **Website URN**   | **View Website Details** in your dashboard, next to your API keys |
| **S3 access key** | Sent to you by Supertab                                           |
| **S3 secret key** | Sent to you by Supertab                                           |

## Set it up

<Steps>
  <Step title="Clone the active version">
    Open the service, click **Edit configuration**, and clone the active version. Everything below happens on that draft and goes live only when you activate it.
  </Step>

  <Step title="Create the VCL snippet">
    Go to **VCL Snippets** → **Create snippet**:

    | Setting        | Value                   |
    | :------------- | :---------------------- |
    | **Name**       | `bot_event`             |
    | **Type**       | `Regular`               |
    | **Subroutine** | `deliver (vcl_deliver)` |
    | **Priority**   | `100`                   |

    <Warning>
      The subroutine must be `deliver`. The snippet reads the response status, which does not exist in `recv` — the default — so choosing `recv` fails to compile.
    </Warning>
  </Step>

  <Step title="Paste the snippet">
    Paste the code below into the snippet's VCL field.

    <Warning>
      **Replace `YOUR_WEBSITE_URN` with your Website URN.** It appears once, marked by a `====` banner near the top of the snippet. Keep the `%22` quotes around it.

      Paste the **whole** value including the `urn:` prefix — the placeholder is the entire URN, not just the id:

      `urn:stc:merchant:system:3f9a1c72-8b04-4e15-9c7d-2a6f0e4b1d83`

      Getting this wrong raises no error. The snippet compiles, log files are written, and your dashboard stays empty.
    </Warning>

    ```vcl theme={null}
    # --- Supertab bot-event logging -----------------------------------------------
    # Builds one JSON line describing this request into a request header,
    # X-Bot-Event; the "bot_events" log endpoint writes out that header and nothing
    # else. It runs after the response has gone to the visitor, makes no network
    # calls, and does not affect caching.
    #
    # It must run in "deliver" — it reads the response status, which does not exist
    # earlier in the request lifecycle.
    #
    # VCL strings cannot hold a literal double quote, so every quote the JSON needs
    # is written as %22. Long strings ({"..."}) appear only where a % must pass
    # through untouched. One statement per line — VCL cannot continue one.
    #
    # If you use shielding, this VCL runs at both the edge and the shield POP. The
    # check below builds the line only at the edge, so requests are not counted
    # twice.

    # Never log a value that came from the client: the log endpoint runs on the
    # shield leg too, where this header still holds whatever the caller sent.
    unset req.http.X-Bot-Event;

    if (fastly.ff.visits_this_service == 0) {
      declare local var.line   STRING;
      declare local var.ip     STRING;
      declare local var.hdrs   STRING;
      declare local var.qs     STRING;
      declare local var.pcount INTEGER;

      # Record the client IP in one consistent form: IPv4 written as IPv6-mapped
      # (::ffff:1.2.3.4), IPv6 as-is, "::" when there is no address.
      set var.ip = client.ip;
      if (var.ip ~ {"^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$"}) {
        set var.ip = "::ffff:" var.ip;
      }
      if (var.ip == "") { set var.ip = "::"; }

      # Which standard headers the visitor sent — NAMES ONLY, never values. VCL
      # cannot list headers, so each is checked individually. The list is broad on
      # purpose: a header not checked here is never recorded. Which headers a client
      # sends is one of the strongest tells of a real browser versus software
      # imitating one.
      set var.hdrs = "";
      if (req.http.Accept)                    { set var.hdrs = var.hdrs "%22accept%22,"; }
      if (req.http.Accept-Encoding)           { set var.hdrs = var.hdrs "%22accept-encoding%22,"; }
      if (req.http.Accept-Language)           { set var.hdrs = var.hdrs "%22accept-language%22,"; }
      if (req.http.Authorization)             { set var.hdrs = var.hdrs "%22authorization%22,"; }
      if (req.http.Cache-Control)             { set var.hdrs = var.hdrs "%22cache-control%22,"; }
      if (req.http.Cookie)                    { set var.hdrs = var.hdrs "%22cookie%22,"; }
      if (req.http.DNT)                       { set var.hdrs = var.hdrs "%22dnt%22,"; }
      if (req.http.From)                      { set var.hdrs = var.hdrs "%22from%22,"; }
      if (req.http.If-Modified-Since)         { set var.hdrs = var.hdrs "%22if-modified-since%22,"; }
      if (req.http.If-None-Match)             { set var.hdrs = var.hdrs "%22if-none-match%22,"; }
      if (req.http.Origin)                    { set var.hdrs = var.hdrs "%22origin%22,"; }
      if (req.http.Pragma)                    { set var.hdrs = var.hdrs "%22pragma%22,"; }
      if (req.http.Priority)                  { set var.hdrs = var.hdrs "%22priority%22,"; }
      if (req.http.Range)                     { set var.hdrs = var.hdrs "%22range%22,"; }
      if (req.http.Referer)                   { set var.hdrs = var.hdrs "%22referer%22,"; }
      if (req.http.Sec-CH-UA)                 { set var.hdrs = var.hdrs "%22sec-ch-ua%22,"; }
      if (req.http.Sec-CH-UA-Mobile)          { set var.hdrs = var.hdrs "%22sec-ch-ua-mobile%22,"; }
      if (req.http.Sec-CH-UA-Platform)        { set var.hdrs = var.hdrs "%22sec-ch-ua-platform%22,"; }
      if (req.http.Sec-Fetch-Dest)            { set var.hdrs = var.hdrs "%22sec-fetch-dest%22,"; }
      if (req.http.Sec-Fetch-Mode)            { set var.hdrs = var.hdrs "%22sec-fetch-mode%22,"; }
      if (req.http.Sec-Fetch-Site)            { set var.hdrs = var.hdrs "%22sec-fetch-site%22,"; }
      if (req.http.Sec-Fetch-User)            { set var.hdrs = var.hdrs "%22sec-fetch-user%22,"; }
      if (req.http.Signature)                 { set var.hdrs = var.hdrs "%22signature%22,"; }
      if (req.http.Signature-Agent)           { set var.hdrs = var.hdrs "%22signature-agent%22,"; }
      if (req.http.Signature-Input)           { set var.hdrs = var.hdrs "%22signature-input%22,"; }
      if (req.http.TE)                        { set var.hdrs = var.hdrs "%22te%22,"; }
      if (req.http.Upgrade-Insecure-Requests) { set var.hdrs = var.hdrs "%22upgrade-insecure-requests%22,"; }
      if (req.http.User-Agent)                { set var.hdrs = var.hdrs "%22user-agent%22,"; }
      if (req.http.X-Requested-With)          { set var.hdrs = var.hdrs "%22x-requested-with%22,"; }

      # ===========================================================================
      #  YOUR_WEBSITE_URN below is the only edit this snippet needs. Replace it
      #  with your Website URN — the whole value, starting "urn:" — keeping the
      #  %22 quotes around it.
      # ===========================================================================
      set var.line = "{%22merchant_system_urn%22:%22YOUR_WEBSITE_URN%22";
      set var.line = var.line ",%22schema_version%22:3";
      set var.line = var.line ",%22source_cdn%22:%22fastly_vcl%22";
      set var.line = var.line ",%22status_source%22:%22observed%22";
      set var.line = var.line ",%22client_ip_source%22:%22cdn_declared%22";

      # REQUIRED — do not remove these four.
      # They describe a licence-enforcement decision. This configuration makes no
      # such decision (it only observes), so they are fixed values saying exactly
      # that: no token was checked, nothing was blocked, enforcement is off.
      #
      # They look optional and are not. A record that leaves any of them out is
      # rejected in full and discarded — you would see traffic being logged and no
      # data ever arriving.
      set var.line = var.line ",%22has_token%22:false";
      set var.line = var.line ",%22token_outcome%22:%22not_validated%22";
      set var.line = var.line ",%22final_action%22:%22allow%22";
      set var.line = var.line ",%22enforcement_mode%22:%22disabled%22";

      # The format is a long string so that % reaches strftime untouched.
      set var.line = var.line ",%22timestamp%22:%22" strftime({"%Y-%m-%dT%H:%M:%S"}, time.start);
      set var.line = var.line "." time.start.msec_frac "Z%22";

      # Text fields. These are always present, and empty when the header was absent.
      set var.line = var.line ",%22request_id%22:%22" req.xid "%22";
      set var.line = var.line ",%22client_ip%22:%22" var.ip "%22";
      set var.line = var.line ",%22user_agent%22:%22" json.escape(req.http.User-Agent) "%22";
      set var.line = var.line ",%22path%22:%22" json.escape(req.url.path) "%22";
      set var.line = var.line ",%22method%22:%22" req.method "%22";
      # The Referer value is deliberately not recorded — a referer URL can carry
      # another site's query string. Whether one was sent is still noted, as a name
      # in the header list above. The field itself must stay present and empty.
      set var.line = var.line ",%22referer%22:%22%22";
      set var.line = var.line ",%22accept_language%22:%22" json.escape(req.http.Accept-Language) "%22";
      set var.line = var.line ",%22host%22:%22" json.escape(req.http.Host) "%22";

      # The status actually served to the visitor. Available because this runs in
      # deliver, after the response exists.
      set var.line = var.line ",%22status_code%22:" resp.status;

      # --- Optional fields.
      # When a value is not available these MUST be null, never an empty string. An
      # empty string would be read as a real answer rather than "not known", which
      # silently changes how the request is classified.
      if (client.geo.country_code && client.geo.country_code != "--") {
        set var.line = var.line ",%22request_country%22:%22" client.geo.country_code "%22";
      } else {
        set var.line = var.line ",%22request_country%22:null";
      }

      if (client.as.number > 0) {
        set var.line = var.line ",%22request_asn%22:" client.as.number;
      } else {
        set var.line = var.line ",%22request_asn%22:null";
      }

      if (client.as.name && client.as.name != "unknown") {
        set var.line = var.line ",%22as_organization%22:%22" json.escape(client.as.name) "%22";
      } else {
        set var.line = var.line ",%22as_organization%22:null";
      }

      if (req.http.Sec-Fetch-Mode) {
        set var.line = var.line ",%22sec_fetch_mode%22:%22" json.escape(req.http.Sec-Fetch-Mode) "%22";
      } else {
        set var.line = var.line ",%22sec_fetch_mode%22:null";
      }

      if (req.http.Sec-Fetch-Site) {
        set var.line = var.line ",%22sec_fetch_site%22:%22" json.escape(req.http.Sec-Fetch-Site) "%22";
      } else {
        set var.line = var.line ",%22sec_fetch_site%22:null";
      }

      if (req.http.Sec-Fetch-Dest) {
        set var.line = var.line ",%22sec_fetch_dest%22:%22" json.escape(req.http.Sec-Fetch-Dest) "%22";
      } else {
        set var.line = var.line ",%22sec_fetch_dest%22:null";
      }

      if (req.http.Sec-Fetch-User) {
        set var.line = var.line ",%22sec_fetch_user%22:%22" json.escape(req.http.Sec-Fetch-User) "%22";
      } else {
        set var.line = var.line ",%22sec_fetch_user%22:null";
      }

      # HTTP Message Signature headers (RFC 9421), recorded exactly as sent. Only
      # well-behaved agents that cryptographically sign their requests send these;
      # they are how such an agent proves it is who it claims to be.
      if (req.http.Signature-Agent) {
        set var.line = var.line ",%22signature_agent%22:%22" json.escape(req.http.Signature-Agent) "%22";
      } else {
        set var.line = var.line ",%22signature_agent%22:null";
      }

      if (req.http.Signature-Input) {
        set var.line = var.line ",%22signature_input%22:%22" json.escape(req.http.Signature-Input) "%22";
      } else {
        set var.line = var.line ",%22signature_input%22:null";
      }

      if (req.http.Signature) {
        set var.line = var.line ",%22signature%22:%22" json.escape(req.http.Signature) "%22";
      } else {
        set var.line = var.line ",%22signature%22:null";
      }

      if (req.http.Cookie) {
        set var.line = var.line ",%22has_cookies%22:true";
      } else {
        set var.line = var.line ",%22has_cookies%22:false";
      }

      set var.line = var.line ",%22header_names%22:[" regsub(var.hdrs, ",$", "") "]";

      # Query string: measurements only. The query string itself is never recorded —
      # only how long it was, how many parameters it had, and whether it looks like
      # an attack. The characters are discarded here and never leave your POP.
      set var.qs = req.url.qs;
      if (var.qs == "") {
        set var.line = var.line ",%22query_length%22:0";
        set var.line = var.line ",%22query_param_count%22:0";
        set var.line = var.line ",%22query_suspicious%22:false";
      } else {
        # `+` is concatenation in VCL, not addition — arithmetic needs the `+=` form.
        set var.pcount = std.strlen(regsuball(var.qs, "[^&]", ""));
        set var.pcount += 1;
        set var.line = var.line ",%22query_length%22:" std.strlen(var.qs);
        set var.line = var.line ",%22query_param_count%22:" var.pcount;
        # Known exploit patterns, checked against the raw and URL-decoded query.
        if (var.qs ~ {"(?i)(\.\./|union select|<script|onerror=|/etc/passwd)"} || urldecode(var.qs) ~ {"(?i)(\.\./|union select|<script|onerror=|/etc/passwd)"}) {
          set var.line = var.line ",%22query_suspicious%22:true";
        } else {
          set var.line = var.line ",%22query_suspicious%22:false";
        }
      }

      set var.line = var.line "}";

      # The log endpoint reads this request header. It is set in deliver, after the
      # origin fetch has already happened, so it is never sent to your origin.
      set req.http.X-Bot-Event = var.line;

    }
    ```
  </Step>

  <Step title="Create the logging endpoint">
    Go to **Resources** → **Log streaming** → **Create endpoint** → **Amazon S3**:

    | Setting                     | Value                           |
    | :-------------------------- | :------------------------------ |
    | **Name**                    | `bot_events`                    |
    | **Placement**               | `Format Version Default`        |
    | **Log format**              | `%{req.http.X-Bot-Event}V`      |
    | **Access method**           | `User credentials`              |
    | **Access key / Secret key** | provided by Supertab            |
    | **Bucket name**             | `lpeu-prod-connect-bot-events`  |
    | **Path**                    | `bot-events/`                   |
    | **Domain**                  | `s3.eu-central-1.amazonaws.com` |
    | **Period**                  | `60` while setting up           |

    Under **Advanced options**, set **Log line format** to `Blank` and leave compression off.

    Four of these are easy to get wrong, and none of them raise an error:

    * **Log format** replaces the prefilled JSON template entirely — delete it. The snippet already builds the JSON; this field only emits it.
    * **Log line format** must be `Blank`. The `Classic` default prepends a syslog header and corrupts every line.
    * **Compression** must be off, or the connector won't match the `*.log` objects.
    * **Path** must be exactly `bot-events/`, trailing slash included — not a subfolder, not your company name.

    <Note>
      Use `Period` `60` while setting up so you find out quickly whether it works, then raise it to `900` (15 minutes) once we've confirmed data is arriving. The longer interval batches events into fewer, larger files.
    </Note>
  </Step>

  <Step title="Activate">
    Activate the version. The snippet and the logging endpoint go live together.
  </Step>
</Steps>

## Confirm it's working

Send some traffic, then allow **15 minutes** before checking. Logging configuration takes a few minutes to reach every POP, Fastly writes one file per `Period`, and it buffers per cache node — so events arrive as a trickle rather than all at once. Requests served before the log endpoint reaches a POP are lost rather than delayed.

Then open your dashboard. Crawlers that hit your site in that window appear there, classified by operator. If it's still empty, work through the table below.

## Troubleshooting

Misconfigurations here fail quietly rather than erroring, so work down the list in order.

| Symptom                                       | Check                                                                                                                                                                                                 |
| :-------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Nothing after 15 minutes                      | Was the version **activated**? A cloned draft serves nothing.                                                                                                                                         |
| Still nothing                                 | If your service uses custom VCL, does `main.vcl` still contain the `#FASTLY deliver` and `#FASTLY log` macros? Fastly inserts snippets and the log statement at those markers.                        |
| Still nothing                                 | **Log line format** `Blank`, compression off, **Path** exactly `bot-events/`, **Bucket** exactly `lpeu-prod-connect-bot-events`.                                                                      |
| Still nothing                                 | Access key and secret pasted without whitespace. Fastly does not validate them.                                                                                                                       |
| Files are written but your dashboard is empty | The URN in the snippet — it must be your full Website URN, with no duplicated `urn:stc:merchant:system:` prefix.                                                                                      |
| Data arrives but looks wrong                  | Send us one example request and what you expected. Don't edit the snippet's field values to compensate — several look optional and are not, and removing one causes the whole record to be discarded. |

<Note>
  Every request is logged, including cache hits. With shielding enabled the line is built only at the edge POP, so requests are not counted twice.
</Note>

***

## Related Docs

<CardGroup cols={2}>
  <Card title="Connect on Fastly" icon="bolt" href="/reference/fastly/connect-on-fastly">
    Serve your RSL license and enforce CAP on Fastly, with Compute or by chaining VCL to a validator.
  </Card>

  <Card title="Deploy at the Edge" icon="shield" href="/guides/deploy-cdn">
    CDN-agnostic guide covering RSL serving, CAP enforcement, and robots.txt.
  </Card>
</CardGroup>