From http://en.wikipedia.org/wiki/Technological_singularity:

In 1965, I. J. Good first wrote of an “intelligence explosion”, suggesting that if machines could even slightly surpass human intellect, they could improve their own designs in ways unforeseen by their designers, and thus recursively augment themselves into far greater intelligences. The first such improvements might be small, but as the machine became more intelligent it would become better at becoming more intelligent, which could lead to a cascade of self-improvements and a sudden surge to superintelligence (or a singularity).

The implications for human society are interesting:

In 2009, leading computer scientists, artificial intelligence researchers, and roboticists met at the Asilomar Conference Grounds near Monterey Bay in California to discuss the potential impact of the hypothetical possibility that robots could become self-sufficient and able to make their own decisions. They discussed the extent to which computers and robots might be able to acquire autonomy, and to what degree they could use such abilities to pose threats or hazards. Some machines have acquired various forms of semi-autonomy, including the ability to locate their own power sources and choose targets to attack with weapons. Also, some computer viruses can evade elimination and have achieved “cockroach intelligence.” The conference attendees noted that self-awareness as depicted in science-fiction is probably unlikely, but that other potential hazards and pitfalls exist.^[8]

Some experts and academics have questioned the use of robots for military combat, especially when such robots are given some degree of autonomous functions.^[9] A United States Navy report indicates that, as military robots become more complex, there should be greater attention to implications of their ability to make autonomous decisions.^[10][11]

The Association for the Advancement of Artificial Intelligence has commissioned a study to examine this issue,^[12] pointing to programs like the Language Acquisition Device, which can emulate human interaction.