There seems to be a competition between San Diego (who is sending 80 athletes) and Orange County, for the king of the Olympics:
If Orange County was a nation it would have ranked among the top 10 in gold medals at each of the past two Summer Olympics. At the 2004 Games in Athens, Orange County athletes won as many golds (nine) as Great Britain, or one more than Brazil and Spain combined. Four years later, O.C. athletes brought home 19 medals, as many as Ethiopia, the Czech Republic and Argentina combined.
Athletes with O.C. ties also produced two of the most iconic moments of the 2008 Beijing Games. Irvine’s Jason Lezak kept Michael Phelps’ bid for a record eight gold medals alive in the 4×100-meter freestyle relay with what has been called as the greatest anchor ever. Phelps later edged Serbia’s Milorad Cavic, a Tustin High grad, by a mere hundredth of a second to win the 100-meter butterfly to equal Mark Spitz’s then-Olympic record of seven golds.
In London, Orange County athletes could put up even bigger numbers.
A record 79 O.C. athletes will compete in the 2012 Olympic Games in London, more than double the 31 who participated in the Athens Games just eight years ago. And unlike some other Olympic hotbeds like Kenya’s Rift Valley or Australia’s Gold Coast, Orange County’s Olympic success is not limited to just one sport. In London, O.C. athletes could win gold medals in as many as nine sports.
Source: OC Register - For Olympics, Orange County has become a powerhouse
If you add in the athletes from Los Angeles then 1 out of 3, or even half, of all Olympic athletes hail from Southern California.