mokhtar/nxdns
1
0
Fork 0
Code Pull Requests Actions Packages Releases 13 Activity

milestone 13 discrepancies: redact credentials from urls in logs, metrics and cli output

Browse Source
This commit is contained in:
mokhtar
2026-08-07 00:45:17 +02:00
parent 1ff727feb8
commit 8c3328562e
39 changed files with 5734 additions and 510 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/safe_url.zig
+947
View File
@@ -0,0 +1,947 @@
//! The one redaction an operator-supplied string passes through before it
//! reaches a log line or an operator-facing diagnostic.
//!
//! It lives at the root rather than under `filter/` because both the blocklist
//! manager and the configuration validator print urls, and a `config/` module
//! importing `filter/` would be the wrong dependency direction. It takes bytes
//! and returns bytes: no `Io`, no allocator, no failure mode.
const std = @import("std");
/// The longest text one `SafeUrl` or one `QuotedText` prints, ahead of the `...`
/// that marks a truncation. An operator-supplied url or name has no length
/// limit and a log line must have one. A `QuotedText`'s two quotes fall outside
/// the count: they are the delimiter rather than the text, and a truncated name
/// still closes the one it opened.
///
/// The count is of printed characters, not of source bytes: an escape sequence
/// costs the two or four characters it prints, not the one byte it stands for.
/// A string of control characters therefore truncates early instead of printing
/// several times its length.
///
/// This is deliberately a second number rather than a reuse of the blocklist
/// manager's `max_url_len`, which sizes the inline url copy inside
/// `SourceStatus`. The two answer different questions — how wide one log line
/// may be, and how large a status value a caller may keep is — and they agree
/// on 255 only because that width suits both. Neither may be changed on the
/// other's account.
pub const max_len: usize = 255;
/// The only form of a url that may be written to a log. `format` prints the
/// scheme, the host and the port; it drops the userinfo, the path, the query
/// and the fragment; it escapes every control character; and it bounds the
/// result at `max_len`.
///
/// Where the line falls, and why it falls there. A blocklist source url is
/// operator-supplied and nothing on the way in restricts it to a bare public
/// path: `?apikey=…`, a signed url whose signature is a query parameter,
/// `https://user:pass@host/list.txt` and `https://host/d/<token>/hosts.txt` all
/// validate, import and store. Userinfo, path, query and fragment are the four
/// components where a credential can legally live, and a log line outlives the
/// process — journald and the configured log file keep it, and neither is as
/// protected as the database row the url came from. The rule against writing a
/// secret to a log is absolute, so all four go.
///
/// Dropping the path costs the url the one job it used to do here: telling two
/// sources on one host apart. That job was never the url's. A caller names the
/// source it reports on from the source's own identity — `filter/manager.zig`
/// prints the row id and the name beside this — and an identity read from the
/// row cannot leak what the url holds.
///
/// What it still prints, deliberately: the scheme, the host and the port,
/// because an operator reading a failure has to know where the source points;
/// and, for a url that names no scheme at all, whatever precedes the first `/`
/// once the userinfo is removed, because nothing in such a string distinguishes
/// a host from anything else and that text is the closest thing to one. A
/// string that names a scheme without a `//` after it gets no such treatment,
/// whether or not it holds an `@` — see `redact`, where the text after such a
/// scheme is a path segment under at least one reading and is never printed.
///
/// The host is therefore the one place a credential still survives this, and it
/// is not hypothetical: NextDNS identifies an account by a profile id, which its
/// DoH url carries in the path — `https://dns.nextdns.io/abcd12`, dropped here —
/// and its DoT url carries in the hostname — `tls://abcd12.dns.nextdns.io`,
/// kept. Nothing can be done about the second without printing no host at all,
/// which would leave every line unactionable. An operator whose provider puts a
/// secret in the hostname has published it to every resolver on the way to it
/// long before it reaches this function.
///
/// It scans and never parses, so it cannot fail. `error.BadUrl` — the error
/// `std.Uri.parse` raises — is one of the failures reported through here, so
/// the inputs a parser refuses are exactly the inputs this must still redact.
/// Only `://` introduces an authority, though: a run of one separator, of three
/// or more, or of two that are not both `/`, leaves text that RFC 3986 reads as a
/// path and WHATWG may read as a host, and `redact` prints no host it cannot
/// settle. A `\` still ends an authority and still anchors the scheme scan — see
/// `isSeparator` — it just does not open one.
///
/// Scanning a string a parser refuses means some of those strings have more
/// than one reading, and the readings disagree about which side of an `@` the
/// host is on. `https://lists.example?token=prefix@hunter2` is one: RFC 3986
/// ends the authority at the `?`, so the host is `lists.example` and `hunter2`
/// is part of an api key; read the `@` as a userinfo delimiter instead and the
/// host is `hunter2`. No scan resolves that, and for a while this one chose the
/// second reading and printed `https://hunter2` — a query-string secret printed
/// as a host, which is the exact leak this file exists to stop. `authority` is
/// therefore `null` on such a url and `format` prints
/// `(ambiguous authority omitted)` in place of a host. Withholding the
/// authority is always available and never wrong; choosing a side is wrong half
/// the time, and the half it is wrong in is the half that leaks.
pub const SafeUrl = struct {
/// The scheme without its `:`, or empty when the url carries no scheme
/// delimiter.
scheme: []const u8,
/// The authority without its userinfo — the host and, when present, the
/// port — or `null` when the url admits two readings of where the host is.
/// Empty and `null` are different answers: empty says the url names no
/// authority, `null` says it names one this cannot resolve.
authority: ?[]const u8,
/// Unquoted. A caller that prints this inside a delimiter of its own must
/// use `redactQuoted` instead, or handle the delimiter itself the way
/// `web/metrics.zig` does for a Prometheus label value.
///
/// This renders `scheme://authority` canonically; it does not quote the
/// input's own syntax. A scheme is always followed by `://`, so
/// `tls:\\host` prints `tls://` and `https://?apikey=…` prints `https://`,
/// though only the second contained a `//`. The `://` marks where a host
/// would go, and the operator's remedy is the same either way: the url names
/// no host this can print. Nothing about which bytes separated the scheme
/// from the rest survives redaction, and nothing should — the input is not
/// reproducible from this and is not meant to be.
pub fn format(self: SafeUrl, w: *std.Io.Writer) std.Io.Writer.Error!void {
return self.write(w, .none);
}
/// A truncation ends the value and returns, so a caller that has written an
/// opening delimiter still gets to write the closing one.
fn write(self: SafeUrl, w: *std.Io.Writer, delimiter: Delimiter) std.Io.Writer.Error!void {
var budget: usize = max_len;
if (!try writeEscaped(w, self.scheme, &budget, delimiter)) return w.writeAll("...");
if (self.scheme.len != 0 and !try writeEscaped(w, "://", &budget, delimiter))
return w.writeAll("...");
const authority = self.authority orelse ambiguous_authority;
if (!try writeEscaped(w, authority, &budget, delimiter)) return w.writeAll("...");
}
};
/// A redacted url in the form a caller may print inside a log line or a
/// diagnostic sentence, where it needs a delimiter to keep a host from running
/// into the words around it.
///
/// **`format` writes the quotes**, for the reason `QuotedText` states: a
/// delimiter a caller adds is a delimiter the value can close. Redaction does
/// not make that go away. A `'` is neither a component separator nor a control
/// character, so it survives the scan into the authority — `https://ho'st/x`
/// redacts to `https://ho'st`, and inside a caller's own `'…'` that reads as
/// `'ho'` followed by loose text. The quote and its escape therefore live here,
/// together, and a caller adds none of its own.
///
/// A caller sizing a fixed buffer needs `max_len + 3` for the value, as
/// `SafeUrl` does, plus the two quotes.
pub const QuotedUrl = struct {
url: SafeUrl,
pub fn format(self: QuotedUrl, w: *std.Io.Writer) std.Io.Writer.Error!void {
try w.writeByte('\'');
try self.url.write(w, .single_quote);
try w.writeByte('\'');
}
};
/// What `format` prints where a host would go when it cannot say which text is
/// the host. It is prose rather than a placeholder host because an operator has
/// to read it as a statement about the line and not as an address: `https://`
/// alone already means "this url names no authority", and the two call for
/// different actions.
///
/// It spends the same printing budget the authority would have, so the bound
/// `web/metrics.zig` sizes its buffer against still holds.
///
/// An operator can write a source url whose redaction is this same text, since
/// a url that is not a url prints as itself. That collision costs nothing: it
/// makes one line say less about a source than it could, and it cannot make a
/// credential read as a host, which is the direction that matters.
const ambiguous_authority = "(ambiguous authority omitted)";
/// An operator-supplied string that is not a url — a blocklist source name — in
/// the only form it may be written to a log. It holds no credential by design,
/// so nothing is dropped from it; it comes out of a database row the same way a
/// url does, so a control character in it can forge a log line the same way,
/// and `format` escapes and bounds it for that reason alone.
///
/// **`format` writes the quotes.** A caller printing a name inside a log line
/// has to delimit it, or a name with a space in it runs into the words around
/// it; and a delimiter a caller adds is a delimiter the name can close. `ads'
/// (https://decoy.example) --` inside a caller's quotes produces a line naming a
/// url no source has. So the quote and its escape live in one place, here,
/// where they cannot drift apart. A caller adds none of its own.
pub const QuotedText = struct {
text: []const u8,
pub fn format(self: QuotedText, w: *std.Io.Writer) std.Io.Writer.Error!void {
var budget: usize = max_len;
try w.writeByte('\'');
if (!try writeEscaped(w, self.text, &budget, .single_quote)) try w.writeAll("...");
try w.writeByte('\'');
}
};
/// The `SafeUrl` of `url`. Both fields borrow from `url`, which every caller
/// holds for the length of the call it prints in.
pub fn redact(url: []const u8) SafeUrl {
const first_sep = std.mem.indexOfAny(u8, url, separators) orelse url.len;
const first_colon = std.mem.indexOfScalar(u8, url, ':') orelse url.len;
// A scheme delimiter is a colon immediately before the first separator of
// the whole string. Anchoring on the first separator is what keeps `a/b:/c`
// from reading as a scheme. It anchors on a `\` too, so a url pasted with
// backslashes still has its scheme recognised and echoed, even though a `\`
// no longer opens an authority.
var scheme: []const u8 = "";
var rest = url;
var delimited = false;
if (first_colon + 1 == first_sep and isScheme(url[0..first_colon])) {
scheme = url[0..first_colon];
var after = first_sep;
while (after < url.len and isSeparator(url[after])) after += 1;
// Exactly `//` introduces an authority, and nothing else does. One
// separator leaves an absolute path — RFC 3986 reads `https:/hunter2` as
// the path `/hunter2`, WHATWG reads `hunter2` as the host — and three or
// more is an empty authority to the first and a host to the second.
//
// A run of two that is not two slashes depends on the scheme. WHATWG
// converts a `\` to a `/` only for a *special* scheme, so `https:\\host`
// is contested the same way, while `tls:\\host` — `tls` is not special —
// has no reading at all under which `host` is a host. RFC 3986 gives `\`
// no meaning anywhere.
//
// So the two answers are different answers, and `authority` carries the
// difference: `null` where the readings disagree, empty where they agree
// the url names no authority. Either way the path segment stays out of
// the log, which is the property that matters; this decides only what the
// line then claims about it.
//
// An earlier revision accepted any number of separators, to keep
// `https:/user:pass@host/list` from printing its userinfo. That reason
// expired when the `/` cut moved ahead of the userinfo lookup: the
// authority of a url with no `://` now ends at its first `/`, so it holds
// no userinfo to print. Withholding it is both safe and the honest
// answer.
if (after - first_sep != 2 or url[first_sep] != '/' or url[first_sep + 1] != '/')
return .{ .scheme = scheme, .authority = contestedOrAbsent(scheme, url[after..]) };
rest = url[after..];
delimited = true;
}
// A network-path reference names an authority and no scheme (RFC 3986
// §4.2). Without this the `/` cut below lands at byte zero, the authority is
// empty, and the line reports nothing at all — including for
// `//user:pa55@lists.example/x`, where the host is not in doubt.
// Exactly `//` here too, and for the same reason. RFC 3986 reads an authority
// after `//` and nothing else — `///hunter2/x` is an empty authority and the
// path `/hunter2/x`, and `\\hunter2\x` is a path outright. WHATWG resolves a
// reference against a base, and against a special-scheme base its
// ignore-slashes state reads `hunter2` as the host in both. A run that is not
// exactly `//` is therefore contested, not settled, and it is withheld rather
// than reported as an authority the url does not have.
//
// A run of one is settled: both readings make it a path. So is any run whose
// candidate authority is empty — `\\?\C:\lists\hosts.txt` ends the authority
// at its `?` under the reading that looks for one, so neither finds a host
// and there is nothing to contest.
if (!delimited) {
var run: usize = 0;
while (run < rest.len and isSeparator(rest[run])) run += 1;
if (run >= 2) {
if (candidateAuthority(rest[run..]).len == 0)
return .{ .scheme = scheme, .authority = "" };
if (run != 2 or rest[0] != '/' or rest[1] != '/')
return .{ .scheme = scheme, .authority = null };
rest = rest[2..];
delimited = true;
}
}
// The authority ends at the first `/`. Everything from there on is path,
// query or fragment, and none of the three is printed — so an `@` after
// that slash is not userinfo by any reading, and no longer needs to be one
// to stay out of the log.
const authority = rest[0 .. std.mem.indexOfScalar(u8, rest, '/') orelse rest.len];
// A scheme with no separator after it is an opaque path under RFC 3986 and,
// for a special scheme, a host under WHATWG — so `https:hunter2` is either a
// path segment, which is where a token lives, or a host. The disagreement is
// the whole of the evidence, exactly as it is for `https:a@hunter2`, and this
// check runs before the `@` lookup so both readings reach it.
//
// No exception is made for a suffix that looks like a port. An earlier
// revision took the digits in `localhost:8080` for one, which also let
// `https:123456` through, whose digits are an opaque path and as much a token
// as any other text. Neither is resolved now.
//
// Note what the two cases are not. `https:` is a WHATWG special scheme, so
// `https:hunter2` is contested — an opaque path to RFC 3986, a host to
// WHATWG. `localhost:` is not special, so `localhost:8080` is a scheme and an
// opaque path to *both*, and what an operator meant by it — a host and a
// port — is a reading no parser offers. It is withheld all the same, because
// the text after the colon is a path segment under every reading and a path
// segment is never printed.
//
// So the two get the same treatment and different answers: `https:hunter2` is
// withheld as contested, `localhost:8080` as naming no authority at all.
//
// An `IP:port` is unaffected: a leading digit fails the scheme production, so
// `10.0.0.2:8080` and `[::1]:853` resolve.
if (!delimited) {
const trimmed = cut(authority);
const colon = std.mem.indexOfScalar(u8, trimmed, ':') orelse trimmed.len;
if (colon != trimmed.len and isScheme(trimmed[0..colon])) return .{
.scheme = scheme,
.authority = contestedOrAbsent(trimmed[0..colon], trimmed[colon + 1 ..]),
};
}
// With no `@` there is no userinfo to remove and no side to choose. A `?`,
// a `#` or a `\` ends the authority; each of the three ends it under every
// reading of the text before it.
const at = std.mem.lastIndexOfScalar(u8, authority, '@') orelse
return .{ .scheme = scheme, .authority = cut(authority) };
// A `?`, a `#` or a `\` in front of that `@` makes the authority ambiguous,
// and the two readings put the host on opposite sides of the `@`. On
// `https://lists.example?token=prefix@hunter2` the text after it is an api
// key; on `https://user:pa55?@host` the text before it is a password. Both
// readings are available on both urls and nothing in either string tells
// them apart, so neither side may be printed.
if (std.mem.indexOfAny(u8, authority[0..at], "?#\\") != null)
return .{ .scheme = scheme, .authority = null };
// An `@` is a userinfo delimiter only inside an authority. The scan knows one
// is there in exactly three cases: a scheme delimiter introduced it, a `//`
// did, or the string names no scheme at all, where this file's rule is that
// the text before the first `/` is the closest thing to a host. The remaining
// case — a scheme with no separator after it — was withheld above, before the
// `@` was looked for, because it is ambiguous whether or not an `@` is in it.
return .{ .scheme = scheme, .authority = cut(authority[at + 1 ..]) };
}
/// `text` up to the first `?`, `#` or `\`, each of which ends an authority. A
/// `\` is here rather than in `redact`'s `/` cut because the `/` cut runs before
/// the userinfo is located and a `\` before an `@` is not a separator under
/// every reading — `https://user:pa55\@host` is ambiguous, not hierarchical.
fn cut(text: []const u8) []const u8 {
return text[0 .. std.mem.indexOfAny(u8, text, "?#\\") orelse text.len];
}
/// The `QuotedText` of `text`, which it borrows for the length of the call that
/// prints it. The result prints its own quotes; see `QuotedText`.
pub fn quoteText(text: []const u8) QuotedText {
return .{ .text = text };
}
/// The `redact` of `url`, quoted.
///
/// The rule for choosing between the two, so it does not have to be rediscovered
/// per call site: **use this whenever anything follows the url on the line.** A
/// redacted authority can still hold a space, a `:` and a `'`, so unquoted it can
/// impersonate whatever comes next — `upstream {f} failed: {t}` with a url whose
/// authority is `ok failed: Timeout` reports a failure that did not happen.
///
/// `redact` is for the two cases where that cannot arise: the url ends the line,
/// or the caller owns the escaping for a delimiter of its own, as
/// `web/metrics.zig` does for a Prometheus label value.
///
/// The result prints its own quotes; see `QuotedUrl`.
pub fn redactQuoted(url: []const u8) QuotedUrl {
return .{ .url = redact(url) };
}
/// What ends a url's components. `/` is RFC 3986's. `\` is here because the
/// strings this scan exists for are the ones a parser refuses:
/// `https:\\dns.nextdns.io\abcd12` carries an account identifier after a `\`,
/// and a scan that read only `/` printed the whole of it.
///
/// It ends a component; it does not open an authority. What that url *names* is
/// contested — path text to RFC 3986, which gives `\` no meaning, and a host to
/// WHATWG, which reads `\` as `/` for a special scheme — so `redact` withholds
/// it. The `\` still matters here because both readings agree the account
/// identifier after it is not part of any host.
/// `null` when the readings disagree about whether `after` holds a host, empty
/// when they agree it holds none. Both withhold; they differ in what the line
/// claims, and `SafeUrl.authority` documents that as a real distinction.
///
/// Known imprecision, in the safe direction. Three shapes return `null` where
/// both readings in fact find no host, so the line says "could not resolve"
/// where "names none" is the truth:
///
/// - `https:/user@/x` — emptiness is decided before the userinfo is removed,
/// so `user@` counts as a candidate host when the host after it is empty.
/// - `\path@hunter2` — a leading run of one is settled as a path under both
/// readings, but reaches the late-delimiter rule instead of returning here.
/// - `file:secret` — `file` is in `special_schemes`, but WHATWG gives it its
/// own parsing states in which that input is a local path with no host.
///
/// Each prints *less* than it could, never more; none prints the path segment.
/// Fixing them means modelling more of two standards for inputs no accepted
/// configuration can hold: every url this program takes carries `http`, `https`,
/// `tls`, `udp` or `tcp` and a `//`, so all three shapes reach a line only as
/// something the validator is already rejecting by field path.
///
/// Only a WHATWG special scheme reads an authority out of text a `//` did not
/// introduce, so only a special scheme can disagree with RFC 3986 here. And
/// nothing is contested when the reading that looks for a host finds none —
/// `https:` alone names no authority under either.
fn contestedOrAbsent(scheme: []const u8, after: []const u8) ?[]const u8 {
if (candidateAuthority(after).len == 0) return "";
return if (isSpecialScheme(scheme)) null else "";
}
/// The host a WHATWG-style reading would take out of `after`, used only to tell
/// an empty one from a non-empty one.
fn candidateAuthority(after: []const u8) []const u8 {
return cut(after[0 .. std.mem.indexOfScalar(u8, after, '/') orelse after.len]);
}
/// WHATWG's special schemes: the ones whose urls it reads an authority into
/// without a `//`, and whose backslashes it converts to slashes.
///
/// This is a closed set fixed by the URL Standard, not a list of what this
/// program supports. That is the difference between it and the known-scheme list
/// an earlier revision removed: this one cannot go stale when nxdns learns a new
/// transport, because it never described nxdns in the first place.
const special_schemes = [_][]const u8{ "ftp", "file", "http", "https", "ws", "wss" };
fn isSpecialScheme(scheme: []const u8) bool {
for (special_schemes) |special| {
if (std.ascii.eqlIgnoreCase(scheme, special)) return true;
}
return false;
}
const separators = "/\\";
fn isSeparator(c: u8) bool {
return std.mem.indexOfScalar(u8, separators, c) != null;
}
/// Whether `text` is a url scheme: an ASCII letter followed by letters, digits,
/// `+`, `-` and `.` — RFC 3986's production, which is what a scheme delimiter
/// has to look like before the text in front of it may be dropped as one.
fn isScheme(text: []const u8) bool {
if (text.len == 0) return false;
if (!std.ascii.isAlphabetic(text[0])) return false;
for (text[1..]) |c| {
if (std.ascii.isAlphanumeric(c)) continue;
if (c == '+' or c == '-' or c == '.') continue;
return false;
}
return true;
}
/// Writes `text` with every control character escaped, spending at most
/// `budget` printed characters and never splitting an escape sequence. Returns
/// whether the whole of `text` was written.
///
/// The control escapes are the ones `platform/logging.zig` uses on a whole log
/// message, byte for byte, and that is deliberate. `delimiter` adds the one
/// escape that sink has no reason to make.
///
/// Two layers escape the same bytes here and neither is redundant. Do not
/// delete this one on the grounds that the log sink already covers it: that
/// sink covers every `std.log` line and nothing else, and a redacted url does
/// not only reach a sink. `config/validate.zig` builds `Problem.message` as an
/// allocated string; `web/handlers/mutations.zig` prints that message into the
/// body of the 400 from `POST /api/blocklists`, and `cli.zig` prints it to the
/// stdout of an interactive `nxdns check`. Control bytes baked into that string
/// reach an HTTP response and an operator's terminal, neither of which any log
/// sink is in a position to escape. A url has to arrive safe rather than be
/// made safe by whatever it is written to.
///
/// The cost is that a url inside a log line is escaped twice: `\n` in the row
/// prints as `\\n` in journald and as `\n` on stdout. One notation across both
/// is what keeps that legible.
///
/// A `\` is escaped for the same reason it is there: `\n` in the output then
/// means the byte this function replaced and `\\n` means two characters an
/// operator typed. It is what makes `\'` unambiguous as well.
fn writeEscaped(
w: *std.Io.Writer,
text: []const u8,
budget: *usize,
delimiter: Delimiter,
) std.Io.Writer.Error!bool {
const hex = "0123456789abcdef";
for (text) |byte| {
var hex_buf: [4]u8 = undefined;
const escape: ?[]const u8 = switch (byte) {
'\\' => "\\\\",
'\n' => "\\n",
'\r' => "\\r",
'\t' => "\\t",
'\'' => if (delimiter == .single_quote) "\\'" else null,
0x00...0x08, 0x0b, 0x0c, 0x0e...0x1f, 0x7f => blk: {
hex_buf = .{ '\\', 'x', hex[byte >> 4], hex[byte & 0x0f] };
break :blk &hex_buf;
},
else => null,
};
if (escape) |seq| {
if (budget.* < seq.len) return false;
try w.writeAll(seq);
budget.* -= seq.len;
} else {
if (budget.* < 1) return false;
try w.writeByte(byte);
budget.* -= 1;
}
}
return true;
}
/// The character the caller of `writeEscaped` wraps the escaped text in, which
/// is therefore the one character beyond the control set that has to be escaped
/// inside it. `.none` is a value nothing wraps: a `SafeUrl` is printed bare, and
/// escaping a `'` in a host would say a `'` there means something it does not.
const Delimiter = enum { none, single_quote };
// ---------------------------------------------------------------------------
// tests
// ---------------------------------------------------------------------------
const testing = std.testing;
fn expectRedacted(expected: []const u8, url: []const u8) !void {
var buf: [8 * max_len]u8 = undefined;
try testing.expectEqualStrings(expected, try std.fmt.bufPrint(&buf, "{f}", .{redact(url)}));
}
test "redact drops the query, the fragment and the userinfo" {
// The credential shapes a source url can hold: an api key in the query, a
// signed url whose signature is a query parameter, and userinfo.
try expectRedacted(
"https://lists.example",
"https://lists.example/hosts.txt?apikey=s3cr3t",
);
try expectRedacted(
"https://cdn.example",
"https://cdn.example/l/hosts.txt?Expires=1700000000&Signature=abc123&Key-Pair-Id=K2",
);
try expectRedacted(
"https://lists.example",
"https://user:pa55@lists.example/hosts.txt",
);
try expectRedacted(
"https://lists.example:8443",
"https://token@lists.example:8443/hosts.txt?t=1#frag",
);
}
test "redact drops a credential carried in a path segment" {
// The path is a place a token lives — a per-subscriber download url is the
// common shape — and the rule against writing a secret to a log does not
// bend for the component it sits in.
try expectRedacted(
"https://lists.example",
"https://lists.example/download/token/hunter2/hosts.txt",
);
// NextDNS: the path segment is the account identifier, so this is the shape
// the rule exists for rather than an invented one.
try expectRedacted("https://dns.nextdns.io", "https://dns.nextdns.io/abcd12");
try expectRedacted("https://dns.nextdns.io", "https://dns.nextdns.io/abcd12/mydevice");
// Its DoT form puts the same id in the hostname, where redaction cannot
// reach it. Pinned so the limit stays visible rather than being discovered.
try expectRedacted("tls://abcd12.dns.nextdns.io", "tls://abcd12.dns.nextdns.io");
try expectRedacted("https://lists.example", "https://lists.example/hunter2");
// An `@` in the path is not userinfo, and no longer has to be told apart
// from it: the path goes either way.
try expectRedacted("https://lists.example", "https://lists.example/@who/hosts.txt");
}
test "redact keeps everything an operator needs to know where a source points" {
try expectRedacted("https://lists.example", "https://lists.example/hosts.txt");
try expectRedacted("http://10.0.0.2:8080", "http://10.0.0.2:8080/a/b.txt");
try expectRedacted("https://lists.example", "https://lists.example?apikey=s3cr3t");
// No host: what is left still names the scheme.
try expectRedacted("https://", "https://?apikey=s3cr3t");
}
test "redact holds on the malformed urls a parser refuses" {
// These reach the log through `error.BadUrl`, so scanning has to hold where
// `std.Uri.parse` gives up. A scheme delimiter of one separator, of three,
// and of none at all: each once carried the userinfo into the log, because
// the scan looked for `://` and found no authority without it.
//
// Each is now withheld rather than resolved. An earlier revision read the
// text after a run of any length as the authority, which drops the userinfo
// on these four but prints the path segment of `https:/hunter2` as a host.
// Only a run of exactly two says an authority is there; what these hold after
// one, or after three, is a path to RFC 3986 and a host to WHATWG. The
// property the line asserts is unchanged — no userinfo reaches the log — and
// it now holds by withholding rather than by resolving.
try expectRedacted("https://(ambiguous authority omitted)", "https:/user:hunter2@host/list");
try expectRedacted("https://(ambiguous authority omitted)", "https:///user:hunter2@host/list");
try expectRedacted("HTTPS://(ambiguous authority omitted)", "HTTPS:/user:hunter2@host/list");
try expectRedacted("https://(ambiguous authority omitted)", "https:\\user:hunter2@host\\list");
// A separator of none at all is withheld for the same reason.
try expectRedacted("(ambiguous authority omitted)", "https:user:hunter2@host/list");
try expectRedacted("", "?apikey=s3cr3t");
try expectRedacted("not a url", "not a url");
try expectRedacted("", "");
// A colon that is not a scheme delimiter does not make one.
try expectRedacted("lists.example", "lists.example/a:/b");
try expectRedacted("", "/download/token/hunter2/hosts.txt");
}
test "redact treats a backslash as a hierarchical separator" {
// A url typed or pasted with backslashes reaches these lines through
// `error.BadUrl`, so the scan has to cut on one. NextDNS again, because the
// path segment it carries is the whole account identifier.
// A `\` ends an authority but never opens one. WHATWG converts it to a `/`
// only for a special scheme, and RFC 3986 gives it no meaning at all, so none
// of these names a host that both readings agree on — and `tls:\\host` has no
// reading at all that makes it one. The account identifier stays out of the
// line either way, which is the property this test is for.
try expectRedacted("https://(ambiguous authority omitted)", "https:\\\\dns.nextdns.io\\abcd12");
try expectRedacted("https://(ambiguous authority omitted)", "https:/\\dns.nextdns.io\\abcd12");
try expectRedacted("https://(ambiguous authority omitted)", "https:\\/dns.nextdns.io\\abcd12");
try expectRedacted("https://(ambiguous authority omitted)", "https:\\dns.nextdns.io\\abcd12");
try expectRedacted("tls://", "tls:\\\\abcd12.dns.nextdns.io");
// A token in a backslash path goes the way a token in a `/` path goes.
try expectRedacted(
"https://lists.example",
"https://lists.example\\download\\token\\hunter2\\hosts.txt",
);
// The `\\` does not open an authority, so this is reported by its scheme
// alone. What the assertion is really for is that the token in front of the
// `@` does not reach the line, and withholding delivers that at least as
// well as resolving did.
try expectRedacted(
"https://(ambiguous authority omitted)",
"https:\\\\token@lists.example:8443\\hosts.txt?t=1#frag",
);
// A backslash separator does not make a scheme out of a colon that is not
// one, exactly as a `/` does not.
try expectRedacted("lists.example", "lists.example\\a:\\b");
try expectRedacted("", "\\download\\token\\hunter2\\hosts.txt");
}
test "redact resolves a backslash authority only where one reading survives" {
// A `\` after the userinfo ends the authority under the WHATWG reading and
// is an illegal host byte under RFC 3986's, so both readings agree that
// nothing after it is a host. Cutting there prints less than either.
try expectRedacted("https://host", "https://user@host\\list");
// But only once a `//` has established that an authority is there at all. A
// leading `\\` does not, so the userinfo is withheld with everything else
// rather than cut out of a host that was never settled.
try expectRedacted(
"https://(ambiguous authority omitted)",
"https:\\\\user:hunter2@host\\list",
);
// A `\` in front of the `@` is the ambiguous shape instead, and used to
// print the text on one side of it: `https:\\lists.example\path@evil` gave
// `https://evil`. See "redact omits an authority it cannot resolve".
try expectRedacted(
"https://(ambiguous authority omitted)",
"https:\\\\lists.example\\path@evil",
);
// A Windows path is not a url and leaves nothing that names a host. The
// line it appears on still carries the source's row id and name.
try expectRedacted("", "\\\\?\\C:\\lists\\hosts.txt");
}
test "redact omits an authority it cannot resolve" {
// The shape this exists for: a query parameter whose value holds an `@`.
// The text after that `@` is the query, which is where an api key lives, and
// a scan that read it as the end of a userinfo printed the key where the
// host goes — having already dropped the real host.
try expectRedacted(
"https://(ambiguous authority omitted)",
"https://lists.example?token=prefix@hunter2",
);
try expectRedacted(
"https://(ambiguous authority omitted)",
"https://lists.example?user=a@b.example&key=hunter2",
);
try expectRedacted(
"https://(ambiguous authority omitted)",
"https://lists.example#f@hunter2",
);
try expectRedacted(
"https://(ambiguous authority omitted)",
"https:\\\\lists.example\\p@hunter2",
);
// The mirror image, which the previous ordering fixed and this keeps fixed:
// the text before the `@` is a credential just as often, so neither side may
// be printed. Both of these once printed `user:pa55` as the host.
try expectRedacted(
"https://(ambiguous authority omitted)",
"https://user:pa55?@host/x",
);
try expectRedacted(
"https://(ambiguous authority omitted)",
"https://user:pa55#@host/x",
);
try expectRedacted(
"https://(ambiguous authority omitted)",
"https://user:pa55\\@host/x",
);
try expectRedacted(
"https://(ambiguous authority omitted)",
"https:\\\\user:pa55?@host\\x",
);
// A url naming a scheme without a separator after it has no authority by
// RFC 3986 and one by the WHATWG parser, so the `@` in it is a path
// separator under the first reading and a userinfo delimiter under the
// second. `hunter2` is a path segment or a host and no scan can say which.
try expectRedacted("(ambiguous authority omitted)", "https:a@hunter2");
try expectRedacted("(ambiguous authority omitted)", "https:user:hunter2@host/list");
// `user` is not one of WHATWG's special schemes, so both readings make this a
// scheme and an opaque path and neither finds a host. It names no authority
// rather than one that cannot be resolved, and the password is withheld by
// the same rule either way.
try expectRedacted("", "user:pa55@lists.example/hosts.txt?apikey=s3cr3t");
// A url with no scheme at all prints the marker on its own, which is still
// not a url and still not a host.
try expectRedacted("(ambiguous authority omitted)", "lists.example?token=a@hunter2");
}
test "redact tells an omitted authority apart from an absent one" {
// Three outcomes an operator has to be able to tell apart, because the
// action each calls for differs: a host, no host, and a host this cannot
// name. Only the third withholds anything.
try expectRedacted("https://lists.example", "https://lists.example/hosts.txt");
try expectRedacted("https://", "https://?apikey=s3cr3t");
try expectRedacted("https://(ambiguous authority omitted)", "https://?u=a@hunter2");
try expectRedacted("lists.example", "lists.example/hosts.txt");
try expectRedacted("", "?apikey=s3cr3t");
try expectRedacted("(ambiguous authority omitted)", "?u=a@hunter2");
}
test "redact still resolves an authority whose delimiters follow the userinfo" {
// The ambiguity is an `@` after a `?`, a `#` or a `\`, not an `@` at all.
// Where the delimiters fall the way a url puts them, the host is not in
// doubt and withholding it would cost an operator the line's whole point.
try expectRedacted("https://lists.example", "https://user@lists.example?apikey=s3cr3t");
try expectRedacted("https://lists.example", "https://user:pa55@lists.example#frag");
try expectRedacted("https://lists.example:8443", "https://token@lists.example:8443\\hosts.txt");
try expectRedacted("https://lists.example", "https://user@lists.example/x?u=a@b");
// No `@` in the authority: a `?` still ends it, and nothing is ambiguous.
try expectRedacted("https://lists.example", "https://lists.example?apikey=s3cr3t");
}
test "redact escapes the control characters that would forge a log line" {
// A row can be written by a path that does not validate as strictly as the
// config validator, so the manager prints whatever the column holds. A
// newline in it would end the line and start one of the operator's
// choosing.
try expectRedacted(
"https://lists.example\\n2026-01-01 ERROR forged",
"https://lists.example\n2026-01-01 ERROR forged/hosts.txt",
);
try expectRedacted("https://a\\rb", "https://a\rb/x");
try expectRedacted("https://a\\tb", "https://a\tb/x");
try expectRedacted("https://a\\x00b", "https://a\x00b/x");
try expectRedacted("https://a\\x7fb", "https://a\x7fb/x");
// An ESC would reach a terminal as a control sequence on the one path the
// log sink does not cover: `cli.zig` prints a diagnostic to stdout.
try expectRedacted("https://a\\x1bb", "https://a\x1bb/x");
// A literal backslash ends the authority, so `redact` cannot print one at
// all. The doubling that keeps an escape sequence unambiguous is exercised
// where a backslash does survive: `quoteText`, below.
try expectRedacted("https://a", "https://a\\nb/x");
}
test "redact bounds the line it prints at max_len" {
const long_host = "h" ** (2 * max_len);
var buf: [8 * max_len]u8 = undefined;
const printed = try std.fmt.bufPrint(&buf, "{f}", .{redact("https://" ++ long_host ++ "/x")});
try testing.expectEqualStrings(("https://" ++ long_host)[0..max_len] ++ "...", printed);
// The bound counts what is printed, so an escape cannot spend four
// characters of a log line per byte of url.
const control_host = "\n" ** max_len;
const escaped = try std.fmt.bufPrint(&buf, "{f}", .{redact("https://" ++ control_host ++ "/x")});
try testing.expect(escaped.len <= max_len + 3);
try testing.expect(std.mem.endsWith(u8, escaped, "..."));
try testing.expect(!std.mem.containsAtLeast(u8, escaped, 1, "\n"));
// A scheme long enough on its own truncates inside the scheme rather than
// printing it whole and starting on the host.
const long_scheme = "s" ** (2 * max_len);
const truncated = try std.fmt.bufPrint(&buf, "{f}", .{redact(long_scheme ++ ":/host")});
try testing.expectEqualStrings(long_scheme[0..max_len] ++ "...", truncated);
}
test "quoteText escapes and bounds an operator-supplied name" {
var buf: [8 * max_len]u8 = undefined;
try testing.expectEqualStrings(
"'ads and trackers'",
try std.fmt.bufPrint(&buf, "{f}", .{quoteText("ads and trackers")}),
);
try testing.expectEqualStrings(
"'ads\\n2026-01-01 ERROR forged'",
try std.fmt.bufPrint(&buf, "{f}", .{quoteText("ads\n2026-01-01 ERROR forged")}),
);
// A name cannot close the quote around it and write what follows as though
// it were another field of the line.
try testing.expectEqualStrings(
"'ads\\' (https://decoy.example) --'",
try std.fmt.bufPrint(&buf, "{f}", .{quoteText("ads' (https://decoy.example) --")}),
);
// Nor by escaping the escape: a `\` before the quote is doubled first, so
// `\'` in the output is this function's and never the operator's.
try testing.expectEqualStrings(
"'ads\\\\\\' (https://decoy.example) --'",
try std.fmt.bufPrint(&buf, "{f}", .{quoteText("ads\\' (https://decoy.example) --")}),
);
// Nor by running past `max_len`: the truncation closes the quote too.
const long_name = "n" ** (2 * max_len);
try testing.expectEqualStrings(
"'" ++ long_name[0..max_len] ++ "...'",
try std.fmt.bufPrint(&buf, "{f}", .{quoteText(long_name)}),
);
try testing.expectEqualStrings(
"'" ++ "\\'" ** (max_len / 2) ++ "...'",
try std.fmt.bufPrint(&buf, "{f}", .{quoteText("'" ** max_len)}),
);
}
test "redact withholds a scheme whose separator is missing, with or without an @" {
// RFC 3986 reads `hunter2` as an opaque path and a path segment is where a
// token lives; WHATWG inserts the missing `//` for a special scheme and reads
// it as the host. An earlier revision withheld this only when an `@` was
// present, so the plainer shape printed the path whole.
try expectRedacted("(ambiguous authority omitted)", "https:hunter2");
try expectRedacted("(ambiguous authority omitted)", "https:a@hunter2");
try expectRedacted("(ambiguous authority omitted)", "https:123456");
// Only a WHATWG special scheme can disagree with RFC 3986 here, because only
// a special scheme reads an authority out of text no `//` introduced. For
// every other scheme both readings say the same thing — a scheme and an
// opaque path — so these name no authority rather than an unresolved one.
// Withheld either way; what differs is what the line then claims.
try expectRedacted("", "mailto:ops@example.com");
try expectRedacted("", "localhost:8080/hosts.txt");
try expectRedacted("", "localhost:8080@evil/x");
// An `IP:port` is unaffected, because a leading digit fails the scheme
// production and there is nothing to disagree about.
try expectRedacted("10.0.0.2:8080", "10.0.0.2:8080/hosts.txt");
try expectRedacted("[::1]:853", "[::1]:853/x");
try expectRedacted("lists.example", "lists.example/hosts.txt");
}
test "redact takes exactly two separators as an authority delimiter" {
// One separator leaves an absolute path: RFC 3986 reads `https:/hunter2` as
// the path `/hunter2`, WHATWG reads `hunter2` as the host. Three or more is
// an empty authority to the first and a host to the second. An earlier
// revision accepted any run and printed the path segment as the host.
try expectRedacted("https://(ambiguous authority omitted)", "https:/hunter2");
try expectRedacted("https://(ambiguous authority omitted)", "https:///hunter2");
try expectRedacted("localhost://", "localhost:/hunter2");
try expectRedacted("https://lists.example", "https://lists.example/hosts.txt");
// The userinfo this tolerance was introduced to protect is protected by
// withholding instead. The `/` cut runs ahead of the userinfo lookup now, so
// the authority of a url with no `://` ends before any `@` it holds.
try expectRedacted("https://(ambiguous authority omitted)", "https:/user:pass@host/list");
}
test "redact resolves a network-path reference instead of reporting nothing" {
// The `/` cut lands at byte zero here, so without the `//` branch every one
// of these redacted to the empty string and the line named no source at all.
try expectRedacted("lists.example", "//lists.example/hosts.txt");
// Exactly two. A longer run is contested — an empty authority to RFC 3986,
// and `lists.example` as the host to WHATWG resolving against a
// special-scheme base — so it is withheld rather than reported as a url that
// names no authority. Those are different answers and this type keeps them
// apart.
try expectRedacted("(ambiguous authority omitted)", "///lists.example/x");
// The authority is not in doubt, so the userinfo is dropped rather than the
// whole of it withheld.
try expectRedacted("lists.example", "//user:pa55@lists.example/hosts.txt");
// An ambiguous one is still withheld: the branch says where the authority
// starts, not that every reading of it is settled.
try expectRedacted("(ambiguous authority omitted)", "//a?b@c");
// A single leading `/` is a path, not an authority, and still names none.
try expectRedacted("", "/path/only");
}
test "the shapes redact over-withholds on print less, never more" {
// The three imprecisions `contestedOrAbsent` documents. Each says "could not
// resolve" where both readings in fact find no host, so each prints less than
// it could. Pinned because the failure that matters is the other direction:
// if one of these ever starts naming a host, this test says so.
try expectRedacted("https://(ambiguous authority omitted)", "https:/user@/x");
try expectRedacted("https://(ambiguous authority omitted)", "https:/user:hunter2@/x");
try expectRedacted("(ambiguous authority omitted)", "\\path@hunter2");
try expectRedacted("(ambiguous authority omitted)", "file:secret");
try expectRedacted("file://(ambiguous authority omitted)", "file:\\secret");
// `file` still resolves where a `//` settles it, which is why the imprecision
// is in the classification and not in the scan.
try expectRedacted("file://host", "file://host/secret");
}
test "redactQuoted writes its own quotes and closes them in every exit" {
var buf: [8 * max_len]u8 = undefined;
try testing.expectEqualStrings(
"'https://lists.example'",
try std.fmt.bufPrint(&buf, "{f}", .{redactQuoted("https://lists.example/hosts.txt")}),
);
// The omitted-authority value is prose with spaces in it, which is the case
// the quotes exist for: unquoted it runs into the words of the sentence
// around it.
try testing.expectEqualStrings(
"'https://(ambiguous authority omitted)'",
try std.fmt.bufPrint(&buf, "{f}", .{redactQuoted("https://lists.example?token=prefix@hunter2")}),
);
// A truncation returns early, and the closing quote still has to be written
// or the rest of the line reads as part of the value.
const long_host = "h" ** (2 * max_len);
try testing.expectEqualStrings(
"'https://" ++ ("h" ** (max_len - "https://".len)) ++ "...'",
try std.fmt.bufPrint(&buf, "{f}", .{redactQuoted("https://" ++ long_host ++ "/x")}),
);
}
test "a redacted authority cannot close the quote a caller would have added" {
var buf: [8 * max_len]u8 = undefined;
// A `'` is neither a component separator nor a control character, so it
// survives redaction into the authority. `redact` leaves it, which is why a
// caller may not supply the quotes itself.
try testing.expectEqualStrings(
"https://ho'st",
try std.fmt.bufPrint(&buf, "{f}", .{redact("https://ho'st/x")}),
);
try testing.expectEqualStrings(
"'https://ho\\'st'",
try std.fmt.bufPrint(&buf, "{f}", .{redactQuoted("https://ho'st/x")}),
);
// The forging shape, whole: an operator-supplied url that ends the value and
// writes what follows as though the line had said it.
try testing.expectEqualStrings(
"'https://ho\\' is fine; upstreams[9] '",
try std.fmt.bufPrint(&buf, "{f}", .{redactQuoted("https://ho' is fine; upstreams[9] /x")}),
);
// The escape-the-escape shape that `QuotedText` has to defend against does
// not arise here, and not because the escaper is different — it is the same
// one. A `\` ends an authority, so it never reaches the value to be doubled.
// This is asserted rather than assumed: it is the property that makes a
// single `\` in the output always this file's and never the operator's.
try testing.expectEqualStrings(
"'https://ho'",
try std.fmt.bufPrint(&buf, "{f}", .{redactQuoted("https://ho\\'st/x")}),
);
}