Situated on low-lying ground across a tributary of the River Wye, Eardisley was said to be heavily wooded at the time of Domesday, when held of Roger de Lacy by one Robert, who had some kind of defended residence there; he may well have been Robert de Baskerville. Certainly Eardisley had come into the hands of the Baskerville family by the twelfth century and a castle there is first heard of in 1183, perhaps built on the site of the Domesday fortification. It was the principal residence of the family by the late thirteenth century, although for a period seems to have been directly in the hands of its overlords, the Bohun Earls of Hereford.

The Domesday settlement was not associated with any hundred and paid no tax, which suggests that it was essentially perceived as a defensive base. The later town must similarly be understood as established to support the castle. Immediately east of the castle was a twelfth-century church, doubtless serving the castle community as much as the village – for not until 1272 did an ageing Walter de Baskerville seek royal licence to construct an oratory within the castle walls. A north-south through-road passed the east side of the churchyard and a short street ran westwards off it, along the north side of the churchyard, to the castle. Early settlement is likely to have been around the through-road in the area of the church, bounded to the north by the course of the Wye tributary, presumably forded or even bridged at the through-road.

What looks like a planned extension to this settlement lies just north of open land (probably meadow) flanking this watercourse. There the through-road is lined with burgage-type plots, seemingly also served by back lanes. This extension may coincide with a young Walter de Baskerville receiving licence for a Wednesday market and July fair in 1225; that this was during Henry III's minority probably explains why the licence had to be reissued in 1233. In 1292 Richard de Baskerville had to account, in a quo warranto proceeding, for his administration of the assize of bread and ale, usually associated either with market operation or with frankpledge, and other judicial jurisdiction; he defended that his family had held these jurisdictions in the time of Richard I and as far back as the Conquest, suggesting that he felt that the assize was more associated with tenure of the manor than possession of a market licence. Although the family spread from Herefordshire into Cheshire and later into other counties and Wales, it is not known to be connected with any other market foundations.

A possible market site has been identified as a triangular space where the through-road widened out a little and was joined by a road from the west – a road from the east meeting the through-road slightly further south may also have existed but was not settled in the Middle Ages. This postulated market space lay towards the northern end of that block of burgage plots along the west side of through-road, though it is possible such plots were also laid out along the south side of the road coming from the west. A stream that ran down the north side of that road and through the marketplace would have served for drainage. Blocks of plots around the marketplace are shorter than blocks in the southern part of the urban area, which may point to provision for different needs of settlers or to separate phases of planned development.

Eardisley was exposed to the instability of the region. In 1263 it was severely damaged by the Welsh and in 1278 the sheriff had to take on defence of the castle. By 1374 the castle was neglected and decaying, though when Glendower's rebellion threatened in 1403 the king seemed to think it might still be made defensible. The architecture of surviving fourteenth- and fifteenth-century houses suggests some continued prosperity; but later buildings are of lower quality, and Eardisley's decline had reduced it the condition of a village by the post-medieval period.