The legendary Z Sword was long encased in a mountain on the scared world of the Supreme Kai's. It was believed to be a weapon of unimaginable power and each of the Kai's would take their turn at trying to retrieve it from the stone.
It was finally pulled from the mountain by Gohan, a mere mortal chosen by the Supreme Kai to attempt to wield it to defeat Majin Buu. The sword itself was extremely heavy and they believed wielding it properly would unleash it's true power. When the sword broke during training, the Elder Kai was released from within it. He had been trapped there for millenia, and now free he was able to unleash Gohan's true power potential - the true value of the sword.