Fix includeconfigsource handling of multiple file writes #11950

Job	Run time
setup-environment	1m 45s
generate-metrics	5m 23s
gofmt	5m 30s
coverage	9m 45s
cross-compile (binaries-darwin_amd64)	5m 7s
lint	15m 22s
test	16m 18s
tidy	2m 25s
cross-compile (binaries-darwin_arm64)	4m 56s
cross-compile (binaries-linux_amd64)	5m 6s
cross-compile (binaries-linux_arm64)	5m 7s
cross-compile (binaries-windows_amd64)	5m 0s
cross-compile (binaries-linux_ppc64le)	5m 3s
	1h 26m 47s

Provide feedback