Rules of Engagement for Black-Box Product Audits

I've been auditing microsaas and what are likely vibecoded products for the last couple of weeks. I came up with a simple rubric and process that performs a simple evaluation. In this post I'll go over the method.

What the collector is allowed to touch

Everything I score comes from publicly available sources, the app's own network calls, response headers, public pages, public DNS, and whatever I see after signing up like a normal user. Probing, fuzzing, enumeration and auth bypass are all out of bounds; I'm not touching users' data.

Anyone can re-run the same fetch and get the same file.

#!/usr/bin/env bash
# seams-collect — black-box evidence collector for a public web product.
#
# Collects only what a normal visitor's browser would see: public pages, the
# shipped bundle, response headers, and public DNS. It does NOT probe, fuzz,
# enumerate, or authenticate.

set -euo pipefail

HOST="${1:?usage: seams-collect <host>}"
OUT="evidence/$HOST/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUT"

UA="Mozilla/5.0 (compatible; seams-audit/1.0)"
CURL=(curl -sS -L --max-time 30 -A "$UA")

"${CURL[@]}" -D "$OUT/headers.txt" -o "$OUT/index.html" "https://$HOST/"

I'm trying to do this as ethically and transparently as possible hence the user agent. The script collects; it doesn't score. Scoring stays human, and while subjective, it is based on the evidence collected.

On the ethics, any real vulnerabilities go to the operator privately, and get published once they're fixed or after a stated window runs out.

Scoring the evidence

My first pass at all five audits was garbage. I read the marketing pages, formed an impression, and wrote down numbers. I'm sure this will be controversial, as most of the sites I've score so far are not doing great, so I'm making sure to keep things reproducible and copies of all the evidence.

Nothing gets scored without evidence. If I can't find evidence, the score is Unknown. If I don't run the check, the score is Unmeasured.

evidence/example.com/
├── 20260803T171718Z/          # collect
│   ├── summary.txt            # the run, as printed
│   ├── index.html             # the served HTML
│   ├── headers.txt
│   ├── meta.txt               # title, description, OG/Twitter, canonical
│   ├── dns.txt                # A, NS, MX
│   ├── bundle.js              # first-party entry bundle, if one exists
│   ├── og-image.bin           # the social card, actually fetched
│   └── emails.txt             # addresses published on /, /privacy, /terms
├── 20260803T173722Z-render/   # Lighthouse + 375/768/1280 screenshots
├── 20260803T181905Z-policy/   # legal surface, extracted
└── 20260803T182117Z-probe/    # error states on paths that don't exist

Anything collected gets a UTC timestamp.

Unknown and Unmeasured

Unknown — I can't tell from outside whether your authorization is enforced server-side. It scores exactly half the dimension's score. Unmeasured — I didn't run the check. That says nothing about your product at all. It scores nothing and blocks the band.

Here is the classification in full:

Confirmed     directly observed (bundle line, request, response,
              screenshot, measured metric)      → scores normally
Inferred      strongly implied by observable
              shape, stated as inference        → scores normally, never as fact
Unknown       opaque from outside to anyone     → exactly 50% of the weight
Unmeasured    the check was not run             → no score; blocks the band
Human-check   needs a person on a real machine  → no score; blocks the band

For example:

# Two audits of the same product, same 11-point dimension.
# Auditor A did the work and hit a wall. Auditor B never opened the terminal.

A: B2 Data Access = Unknown    → 5.5 / 11   (probed the public page; no client-DB
                                             calls in the bundle; still can't see
                                             whether row policies are enforced)

B: B2 Data Access = "unknown"  → 5.5 / 11   (did not look)

                                  ^^^ identical output.

Scope and Commercial Model

Because I'm essentially auditing a black box, there are limits. Some dimensions, such as server-side authorization or row-level database policies, cannot be fully verified from the outside. In this public audit series, these are marked Unknown.

I do offer a complete version of the audit as a service which includes source code reviews to fill the Unknowns.

The weights

Here is the rubric I used for the first five audits. The total is 100, split 55/45 between product and engineering. Each dimension has a weight, and each finding is scored as Confirmed, Inferred, Unknown or Unmeasured.

Part A — Product & Growth ............................ 55
  A1  Time-to-Value & Onboarding ..................... 15
  A2  Trust & Commercial Clarity ..................... 16
  A3  Coherence — The Seams .......................... 8
  A4  Positioning & Copy ............................. 8
  A5  Discoverability ................................ 8

Part B — Engineering & Risk .......................... 45
  B1  Secrets & Exposure ............................. 10
  B2  Data Access & Authorization .................... 11
  B3  Payments Integrity ............................. 9
  B4  Cost & Scale ................................... 5
  B5  Operability & Data Model ....................... 4
  B6  Delivery, Performance & Accessibility .......... 6

Part A is the product side, and easier to verify without source access. Part B is the engineering side, and ideally I can't verify most of the elements without source access, however I have already ran into instances of sites running with their secrets exposed and databases open to the public.

As I do more audits, I will refine the weights and the scoring rubric. The goal is to have a consistent, reproducible method that can be applied across different products and services.

Mail posture

Mail is pretty important and often overlooked by vibecoded products. The rubric includes a check for mail posture, which is the configuration of the domain's DNS records related to email delivery and security.

$ dig +short example.com MX
# empty — every published @example.com address bounces

$ dig +short example.com TXT | grep spf
# empty — the domain is spoofable

$ dig +short _dmarc.example.com TXT
"v=DMARC1; p=none;"
# monitor-only; does NOT block spoofed mail

The collector runs the same lookups and prints them next to the addresses it scraped off the public pages.

=== DNS & MAIL POSTURE ===
A:      203.0.113.10
NS:     dns1.registrar-servers.com. dns2.registrar-servers.com.
MX:
  BROKEN  no MX record — any @example.com address published on this site BOUNCES
  MISSING SPF   — domain is spoofable
  WEAK    DMARC: "v=DMARC1; p=none;"
          p=none is monitor-only — it does NOT block spoofed mail
  absent  MTA-STS (minor)

=== EMAIL ADDRESSES PUBLISHED ===
      9 hello@example.com
  BROKEN  hello@example.com → example.com has NO MX. This address bounces.

I'm also looking for unreacheable addresses published on the site. The collector scrapes the homepage, privacy policy and terms of service for any email addresses, and checks whether they are reachable.

p=none scores weak. It's a reporting policy; receivers are told to do nothing.dmarc Counting it as DMARC-present is the exact substitution this rubric exists to catch.

Bands and caps

A number alone doesn't really tell you anything. The bands are a simple way to communicate the overall health of the product, and the caps are there to prevent a high score from being misleading.

Solid    80–100   shippable, fix the long tail
Fragile  60–79    works today, one growth spurt from trouble
At Risk  40–59    material exposure, act now
Broken    0–39    a damn shame!

# Caps — published, because they are the reason a high score means anything.
cap 1: any Confirmed Critical in Secrets or Data Access ....... total ≤ 39
cap 2: a Broken Time-to-Value ................................. total ≤ 59

Cap 1 needs no defence. Cap 2 is the one people will call unfair: an app nobody can get value from doesn't get a passing grade for being well built, useless products that nobody wants should be considered at risk

Structure of the Audit

Each audit is presented in a consistent format:

What it is: A straight, one-paragraph description of the product.
What works: Specific, verifiable strengths. This section is included for a complete picture; an audit that only lists negatives is incomplete.
Findings: Each finding is tied to an evidence file and a confidence marker.
The generalizable rule: The transferable lesson from the audit.
Fix priority: A ranked list of what to address first.
Disclosure: A statement on the audit's scope and commercial model.

The findings section (3) never includes unfixed vulnerabilities or details that could be weaponized. The primary output is the rule (4) — the score is a snapshot, but the rule is durable.

Known Limitations

The engineering scores (Part B) from the initial audits have clustered in a narrow mid-range. This may indicate a limitation of the black-box method; external audits may not be able to finely discriminate on internal engineering quality.

This is a potential finding about the methodology itself. As more audits are completed, this will either become a clear conclusion or the variance will increase. The method is designed to be transparent about its own limits.