The Dosadi experiment - Frank Herbert

In the far future, humans and aliens have joined together in a galaxy-wide federation of intelligent species: The Consentiency, but its existence is now threatened by the discovery of a secret, illegal and deadly experiment on the planet Dosadi.
There the froglike Gowachin have created a brutal society: 850 million beings, human and Gowachin, are confined in an area of 40 square kilometres. But the experiment in overcrowding is getting out of control: the beings that have bred on Dosadi are so tough and resourceful that they are capable of breaking out of their monstrous prison and overrunning the galaxy.