A borough with sixteen burgesses was recorded by Domesday. Its foundation, along part of a through-road, followed the erection of a castle there by William Fitz-Osbern, but by 1086, following the forfeiture of the rebellious Earl Roger (1075), William's brother-in-law, Ralph de Tosny (d. 1102), was lord; so the identity of the town-founder is uncertain. Clifford castle was built on waste land, atop a small hill with a steep cliff dropping down to the River Wye, to the north, near a fording point. Tosny may have rebuilt the castle in stone and his heirs continued to hold the manor until around the mid-twelfth century, when Roger de Tosny's steward, Walter de Clifford, who had married Roger's sister, took control of the manor and successfully resisted claims by the Tosny heirs, perhaps thanks to the influence of Walter's daughter, known to history as 'The Fair Rosamund', over Henry II. However, in Henry III's reign Walter's namesake grandson strengthened the castle and twice had to uphold his claim by force against royal forces, in the 1230s and 1250s; though unsuccessful on either occasion, he was able subsequently to regain royal favour.

At the period between the Conquest and Domesday, when the borough is presumed to have been founded, no market licence was required. Despite that, a royal grant of Wednesday market and August fair were obtained in 1261 by Walter III Clifford; we must understand this in the context of Walter winning back royal acceptance of his claim to Clifford, not based on heredity, so that his right to any market or fair now required a licence in the eyes of royal law. The licence would help reinforce his claim to the manor, giving it tacit royal approval. Yet there was more at work than simply Walter's political rehabilitation, for at the same time as he acquired the Clifford licence, he also obtained grant of a market and fair for the Sussex manor of Findon, which he held through marriage into the de Braose family; a later member of that family claimed – perhaps in response to a challenge (unsuccessful) against Findon's market from the borough of Steyning – that the market had existed since the Conquest.

The licence for Clifford may perhaps also have been intended to revitalize the town, which could have suffered when the king had besieged the castle in 1233, although the town was not immediately adjacent to the castle, but to its north on level ground below the cliffs descending to the river. Growth of the town is suggested by the fact that 200 houses were said to have burned down as the result of a border raid in 1368. Half a century earlier, castle and town had come into the hands of the Mortimers, who, having many estates, made only sporadic effort to keep the castle in repair, and no effort to increase the number of fairs held at Clifford.

The town was established around a through-road heading south to Hay-on-Wye (Wales). A secondary route ran west off this to a ford. Their junction seems to have opened out into a wider space that may have served as marketplace. Plots thought to have been part of the borough lie on either side of the through-road, north of the marketplace, while the castle was south-west of the marketplace. Clifford's economy was probably founded on the needs of the castle and its lords, and on trade in the wool from sheep farms of the region. However, the borough has left little trace in the records and by the close of the Middle Ages had declined to become a village without a regular market.