Host config defaults

[thomas-zahner]

Collect useful real-world host-specific configurations and configure them by default. This is a follow-up feature of #1929. This also addresses the idea of this comment and this discussion.

This feature is comparable to quirks but specifically for HostConfigs. I've started in 6b4b61f, which can be used as a starting point.

One difficulty to consider is the merging/precedence of the different options. I guess our host specific "quirk" defaults should take precedence over user specified global defaults, but not over user-specified host defaults.

[mre]

One difficulty to consider is the merging/precedence of the different options. I guess our host specific "quirk" defaults should take precedence over user specified global defaults, but not over user-specified host defaults.

agreed

[thomas-zahner]

Here is a table for known host specific restrictive user-agent quirks. When adding these by default it will improve UX out of the box for lychee. I suspect there are many websites witch such behaviour, additions to this table are welcome. Also the websites probably tend to accept different common headers, probably the ones used by common web browsers.

Website	User-Agent
www.gnu.org	Mozilla/5.0
developers.redhat.com	curl/8.4.0