Quote from Ted in Cleaning House
Ted: Wait, Barney and James are two years apart. If Sam only knew Loretta for a few months, he couldn't possibly be Barney's father. Robin: [texting] He's also quite the detective.