I ended up putting together a couple of Caddyfiles for the Caddy server. The project really wants users to choose automatic HTTPS, but it wasn’t a good fit for using with a load balancer and auto-scaling. Surprisingly, it is willing to contact the production Let’s Encrypt API even without an email address.
Anyway, I have a few pieces here to cover:
- The easiest way to ignore HTTPS, and returning a fixed error message on HTTP
- Mysterious error messages, relating to the way Caddy chooses certificates and configuration blocks
- Using the tls directive to get self-signed certificates for a public name, for using HTTPS between Caddy and a load balancer which does not validate certificates
- How to use trusted_proxies—and a bit more trickery—so that, using HTTP behind a load balancer, PHP knows the client’s connection is HTTPS
- How to compose multiple URL mappings, and using a front controller for PHP apps
Aside from the above links, Caddy provides more conceptual documentation at the Automatic HTTPS article.
My end goal is to get FrankenPHP running, but as it is built on Caddy, I wanted to understand that part before moving ahead.
The Easiest Way to Ignore HTTPS
Configuring the HTTP port without a hostname—i.e. using :80 as the address—will use HTTP instead of
HTTPS for that block. For example, we didn’t want to encourage accidentally using HTTP for the endpoint and
getting a redirect every single request ever made, so the Apache site replies with a basic error message.
Replicating that in a Caddyfile looks like:
:80 {
header Content-Type "application/json"
respond `{"error":"TLS required"}` 403
}
Note that this is on the HTTP port with no hostname, so Caddy skips HTTPS. It’s also a static response, so nothing can go wrong with HTTP/HTTPS detection.
That was neat, but ended up being irrelevant. The AWS ALB (Application Load Balancer) supports a “fixed response”, allowing us to do the same thing in a Listener Rule without even contacting Caddy. (ALB imposes a 1 KiB limit on the response size, but that’s plenty for our needs here.)
Mysterious Errors
Alongside the above, I configured the main app on the HTTPS port. This gave me an immense amount of trouble.
If I had localhost as the server block, I couldn’t connect to api.example.com. curl would report very
similar to:
OpenSSL/3.0.11: error:0A000438:SSL routines::tlsv1 alert internal error
Looking in the logs with all the debug flags turned on, I could also see Caddy producing an “internal error (592)”, which was equally unclear.
What this is: Caddy doesn’t think it has a certificate configured for the hostname used in the curl
command. If I switched it to using localhost, then it all went fine, because Caddy thought it was serving
localhost and that’s what I was asking for.
I found out that I could get the TLS connection to work if I used fallback_sni, but then I had a
new problem. Caddy would respond to everything on api.example.com with a 200 OK status and a 0-byte body.
What that is: if Caddy doesn’t think it is configured to serve anything for the requested host, then its
default response is the empty 200 reply. In my case, there was still no api.example.com server block, so
although fallback_sni allowed Caddy to make a secure connection, it wouldn’t serve anything over it.
Using the tls Directive
The correct thing to do was get rid of fallback_sni and configure the server to use internal TLS. As in, I
started with:
# WRONG CONFIG:
{
auto_https disable_redirects
fallback_sni localhost
}
# [:80 with a fixed response was here]
localhost {
…
}
This needed to be:
{
auto_https disable_redirects
}
# [:80 with a fixed response was here]
api.example.com {
tls internal
…
}
This asks Caddy to generate a self-signed certificate for the public name, the same way localhost does
automatically. (That default is what led me down the path of trying to make localhost work as a server
name. Oops.)
Everything appeared to work at that point, I sent it to my boss, and we had a chat about the overall architecture. It seems he would rather use HTTP, and trust the cloud’s private network. Back to the drawing board!
How to use trusted_proxies
Since our load balancer was now handling rejection of HTTP requests, it was fairly easy to change the site
configuration back to HTTP. Just put it on :80 and remove the hard-won tls internal line. And since
auto_https doesn’t apply to this port anyway, get rid of that, too.
# WRONG FOR A LOAD BALANCER BACKEND:
:80 {
…
}
Except I started getting an HTTP→HTTPS redirect. Much debugging later, I found out that our framework (based on Symfony Components) was seeing the request as plain HTTP. Thus, the stack decided that it needed to issue that redirect.
Caddy was dropping all incoming X-Forwarded-* headers (from nginx, which emulates ALB in my testbed), and
setting its own. Since Caddy was serving on HTTP, that’s what it filled in. The real problem is that it
didn’t know it could trust those headers. I had to tell it who was allowed to send them:
# STILL INCOMPLETE:
{
servers {
trusted_proxies static private_ranges
}
}
:80 {
…
}
It improved the situation according to phpinfo(), but it didn’t work yet. The Symfony Components still
didn’t accept the requests as secure, and therefore kept redirecting. The problem was, Web server was now
sending REMOTE_ADDR as the IP extracted from X-Forwarded-For. This made HttpFoundation skip checking proxy
headers, and look to HTTPS instead. However, it wasn’t set, because the connection into Caddy was plaintext
HTTP.
This problem is actually familiar. On our Apache instances, we have a SetEnvIf rule to
establish HTTPS=on if the connection’s IP address is from one of our valid private IP ranges and
X-Forwarded-Proto indicates HTTPS. (I’m not at work to look up the exact mechanism, but this may be paired
with the %{CONN_REMOTE_ADDR} variable established by mod_remoteip. Naturally, that module also trusts the
same proxies.)
Duplicating that setup with Caddy looks something like this:
{
servers {
trusted_proxies static private_ranges
}
}
:80 {
@trusted_https {
header X-Forwarded-Proto https
client_ip private_ranges
}
vars @trusted_https client_https "on"
vars client_https "off"
# Debian "selected alternative" path
php_fastcgi unix//run/php/php-fpm.sock {
env HTTPS {vars.client_https}
…
}
…
}
In this example, I threw in private_ranges, but this should be replaced in both places with the CIDR where
the load balancer actually resides.
Composing multiple URL mappings
Suppose we run several user-visible APIs like OAuth, a simple New Contract Entry, some Customer Data Reports, and a Full System API. Further, suppose most of them are under one framework and directory, while the Full API is under another (but in the same repository.) What we end up with is a config like this:
# [global options skipped]
:80 {
php_fastcgi unix//run/php/php-fpm.sock
file_server
# static files for the OAuth Authorization Flow
handle_path /static/* {
root /app/public
encode
}
# Full API - front controller, PATH_INFO
handle_path /full/* {
root /app/conway/public
rewrite index.php{path}
}
# Basic/OAuth APIs
handle {
root /app/srv
rewrite front.php{path}
}
}
This enables PHP and file serving on all of the branches, then starts configuring the URL space. Using
handle_path like that removes the URL prefix, so a request for URL path /static/auth.css will become
/app/public/app.css in the filesystem. The static component is removed.
Both frameworks conveniently run everything via a PHP front controller that expects the actual URL to be
given through CGI PATH_INFO. Therefore, we can take any URL that came in and prepend the front controller.
Requests to URL path /full/63/contracts will be served by /app/conway/public/index.php with a path info of
/63/contracts. URL query parameters are even preserved by these specific rewrite rules.
Closing Notes
It took me a long time to put this all together, but understanding the tools is a necessary step toward building bigger systems with them. Besides, how much time have I put into Apache over the last 25 years? How much into nginx? This is surely a drop in the bucket.
No comments:
Post a Comment