Unsetting file-name-handler-alist around a `load` call prevents any
change to this variable from surviving that file's evaluation (e.g. by
packages loaded therein). Since the user's config files are loaded with
this macro, this affects users' configs, which is unacceptable.
Since this optimization is already done in early-init.el, we can get
away with being more selective here.