There are only 2 Sith because about 1000 years before the Battle of Yavin there was a huge war between the Jedi and the Sith. At the time the Sith, called the Brotherhood of Darkness, were a large order that rivaled that of the Jedi. During the Battle of Ruusan nearly all the combatants on both sides were destroyed. Darth Bane was one of the few who survived the battle and it was Bane who instituted the Rule of Two based on the teachings of a Holocron he found that had been left behind by Darth Revan.
It was also the Battle of Ruusan that led to the Republic having no military until the formation of the Grand Army of the Republic at the outset of the Clone Wars.
The Rule of Two poses a threat because the Sith spent nearly a 1000 years growing stronger, mastering the dark side of the Force in ways the Jedi never could during the time. The Sith only cared about growing strong enough to one day finally the defeat the Jedi in revenge for every slight throughout the history of the Sith. The Jedi spent that time worrying more about their code than wondering where the Sith had disappeared to. The Jedi might have stood chance if any one other than Obi-Wan knew how to be a true Jedi. All the ridiculous rules they enforced based on their code did them more harm than good, especially since they followed a variation of the initial code. If only they still followed the original code they might have avoided the whole mess with Anakin and being wiped out by the Sith.