Stop Treating Well-Known URIs as URL Shorteners

The /.well-known/ path prefix is the web’s ultimate double-edged sword. To some, it is a clean, standardized way to publish site-wide metadata like security policies or password-change endpoints. To others, it is a convenient digital junk drawer—a place to dump configuration files and custom protocol endpoints without having to design a proper discovery mechanism.

Defined originally in RFC 5785 and updated by RFC 8615, the /.well-known/ prefix was designed to solve a very specific problem: allowing clients to discover site-wide metadata before making a request, without risking path collisions with the origin's own resources. But as Mark Nottingham—co-author of the specification and the current Designated Expert for the IANA registry—recently warned, developers are increasingly abusing this space.

If you are thinking about defining a new well-known URI for your protocol or application, you need to understand the architectural trade-offs, the operational pitfalls of multi-tenancy, and how to implement routing without breaking your server.

The Litmus Test: Site-Wide Metadata vs. Lazy Protocol Design

Before you register a path in the IANA well-known URI registry, you must ask a fundamental question: Is this resource actually describing the entire origin?

Well-known URIs work best when a client already knows the target site and needs to efficiently discover something about the site as a whole. The classic conceptual ancestor is robots.txt (which predates the standard but inspired it). A crawler needs to know access policies for the entire origin; checking headers on every single sub-resource would be an operational disaster. Similarly, /.well-known/change-password works because a password change is an origin-wide action.

An anti-pattern occurs when developers use a well-known URI as a glorified URL shortener. This usually happens when a custom protocol is designed to only carry a hostname rather than a full URL. To bridge the gap, the protocol designers register a well-known URI to "fill in the rest" of the path.

This introduces a severe architectural constraint: the 1:1 service-to-domain lock-in.

If your protocol dictates that the service configuration must live at example.com/.well-known/my-service, you can never host more than one instance of that service on that domain. If a future deployment requires splitting tenants, routing to multiple backends, or running parallel staging environments, you are stuck. You have traded protocol flexibility for a marginally shorter payload.

The rule of thumb is simple: If your protocol can carry a full URL, carry the full URL. Do not use a well-known URI just to make your protocol feel more "official."

The Operational Reality of "Fuzzy Origins"

Even when a well-known URI is the correct tool, the assumption that a "site" is a single, easily queried boundary rarely holds true in production. This manifests as the "fuzzy origin" problem, particularly around subdomains and redirects.

If a client starts an interaction at login.example.com, where should it look for the well-known URI?

Advertisement

Serverless Inference by DigitalOcean 55+ models, every modality. One API key, one bill.

1. Should it query login.example.com/.well-known/...?
2. Should it strip the subdomain and query the apex domain at example.com/.well-known/...?
3. Should it follow HTTP redirects if the subdomain redirects to an external identity provider?

For non-web protocols that merely leverage HTTP as a transport layer, forcing metadata onto the apex domain can be an operational nightmare. The team managing the apex domain (example.com) is rarely the same team managing the specific application subdomain (api.example.com). Forcing application-specific metadata onto the apex requires cross-team coordination, complex routing rules, and increases the blast radius of configuration errors.

The Multi-Tenant and Shared-Hosting Nightmare

Another major pitfall is using well-known URIs for content-specific metadata.

On shared hosting platforms or multi-tenant applications (e.g., legacy example.com/~username/ paths or modern SaaS platforms routing via paths), the /.well-known/ directory is controlled exclusively by the root domain administrator.

If your protocol relies on a well-known URI to discover metadata about specific user content, you have effectively locked out individual tenants from customizing that metadata. To support them, the system administrator must build complex proxying infrastructure to parse incoming requests and dynamically route /.well-known/ queries to the correct tenant's backend.

If you need to attach metadata to specific resources or tenants, do not centralize it. Use HTTP response headers or embed the metadata directly within the resource representation itself.

Developer Guide: Implementing and Routing Safely

If you have weighed the trade-offs and determined that a well-known URI is indeed the correct architectural choice, you must implement it defensively.

Because /.well-known/ is a frequent target for vulnerability scanners and automated bots looking for exposed configuration files (like security.txt or ACME challenge tokens), your routing configuration must be explicit and secure.

Here is how to configure an Nginx server block to handle a specific registered well-known URI while explicitly denying access to unhandled or sensitive hidden files within that directory:

server {
    listen 443 ssl;
    server_name example.com;

    # Explicitly allow your registered well-known URI
    location = /.well-known/your-custom-metadata {
        default_type application/json;
        alias /var/www/metadata/config.json;
        add_header Access-Control-Allow-Origin "*";
        add_header Cache-Control "public, max-age=3600";
    }

    # Allow Let's Encrypt / ACME challenges if using certbot
    location /.well-known/acme-challenge/ {
        root /var/www/letsencrypt;
    }

    # Block access to everything else under .well-known to prevent scanning
    location /.well-known/ {
        return 404;
    }

    # Prevent access to hidden files generally, but ensure it doesn't conflict
    location ~ /\.(?!well-known) {
        deny all;
    }
}

The Registration Process

If you are writing a specification that defines a new well-known URI, you must submit a registration template to IANA. Under RFC 8615, this registry operates under the "Expert Review" policy.

Your registration request will be reviewed by a designated expert (such as Mark Nottingham) who will evaluate whether your proposed URI conforms to the guidelines. To pass review, your specification must:

• Define the path segment clearly (matching the segment-nz production in RFC 3986).
• Specify the alignment and scoping of the metadata (i.e., how clients should handle subdomains and redirects).
• Detail the security considerations, specifically addressing whether the metadata could expose hidden capabilities or sensitive system configurations.

The Verdict

Well-known URIs are an architectural compromise. They exist because the stateless, distributed nature of the web occasionally demands a predictable shortcut. But shortcuts should be used sparingly. If your protocol can carry a full, explicit URL, do not delegate that responsibility to a hardcoded path on an origin server. Keep your protocols flexible, your domains decoupled, and your /.well-known/ directory clean.