The original timber castle at Crickhowell was rebuilt in stone by Sir Grimbald Pauncefote in 1272. He married a Turberville heiress and was governor of the castle. The castle was siezed by Roger Mortimer. Then in 1402 it was returned to Sir John Pauncefote who re-fortified it under royal command. It failed to withstand attacks of Owain Glyndwr which reduced it to a ruin. During the reign of Elizabeth I, the stone fortress which was then standing on the motte was declared uninhabitable.