Previously, we'd implicitly turn off future notices about unknown extra conf
files if we already raised one exception about it. This breaks when the user
ends up not receiving the "unknown extra conf, load?" message.
Now we only turn off the notice as a result of the user saying "don't load this"
so that if the first request fails to reach them, they'll get a second (and
third etc) request about it.
Fixes#615
b1d71bbb91