Podman's pasta networking breaks host.containers.internal compatibility
Here's my setup in a nutshell.
I've got a Plex quadlet running on the host network because Plex is finicky about networks. If I run it on a bridge network, Plex The Service thinks my server is "remote" and needs a "relay connection" even if I'm trying to stream media on my LAN. And now that Plex The Service charges a subscription to stream remotely, that's not an option for me.
I'm using Caddy as a reverse proxy for the various services on my server. For everything except Plex, that means the quadlets live on a custom bridge network called "proxy". That makes the Caddy config file easy as I can route traffic to containers by name. Plex is the exception since it lives on the host network. For Plex, plex.local:80 routes to host.containers.internal:32400. host.containers.internal is the neutral equivalent to host.docker.internal (which is also supported as of writing this).
Unfortunately this setup broke at some point between the end of February and today, when I noticed the issue. I probably updated Podman and didn't notice the change. I'm currently running Podman 5.4.2.
After a bunch of debugging, the issue appears to be a change in default networking from slirp4netns to pasta. The only way to get back the preferred behavior is to set this setting in ~/.config/containers/containers.conf.
# ~/.config/containers/containers.conf
[network]
default_rootless_network_cmd = "slirp4netns"
Then recreate the containers and everything should work again.
The difference is in how host.containers.internal is resolved. With pasta, it appears to be resolved to an unclaimed synthetic link-local address, 169.254.1.2. Because nothing is actually claiming that address, traffic sent to it goes out through the host kernel and the connection times out waiting for a response that never comes. There appears to be ways to work around this, but only with pasta's default network, not with a custom network like my "proxy" network.
With slirp4netns on the other hand, host.containers.internal gets resolved to the actual host machine's IP address, which is exactly the behavior I expect.
This was a frustrating regression to work around. It took me several hours to fix. I spent some time afterwards reading up on the differences between pasta and slirp4netns and I understand why pasta can be considered a big performance win for many cases... except for cases where you need a container to be able to reach the host machine. If I had a nickel for every time I've had to use host.containers.internal, I'd have two nickels, which isn't a lot. But it's weird that it happened twice. Right, Stefano Brivio?