Pembridge is placed along a single through-road, connecting Leominster and Kington, and paralleling the course of the River Arrow, to north of the settlement. Domesday records a manorial village in the hands of Alured of Marlborough, also lord of Ewyas Harold; this village was likely the nucleus of the later town. It is possible there may have been a small castle there at the time of Domesday (which tended to ignore such structures, as they were drains on, rather than sources of, revenue), or one may have been built by Ralph de Pembrugge who, in the late eleventh and early twelfth centuries, held the manor from William I de Braose, as part of the Honour of Radnor that William had carved out of Wales by conquest. That the Pembrugge family were tenants of Braose overlords is also indicated by Ralph acting as witness to several grants and charters of Philip de Braose, and by reference to William VI de Braose as lord of Pembridge in the reign of Henry III.

The castle at Pembridge perhaps comprised little more than a motte topped by a keep, with manor-house adjacent, surrounded by a moat; a stone keep was still visible to a seventeenth-century visitor, although now has disappeared. Further evidence supporting the presence of a castle is that the church – situated immediately north of the castle/manor-house enclosure, incorporating a twelfth-century capital, and with a detached bell-tower dated to the early twelfth century – was dedicated to St. Mary, a common association with a church tied to a castle. The entrance to the manorial enclosure would have been on its west side; a holloway led from there to the church. Pembridge Castle is mentioned in a document of 1222, but the situation is complicated by the fact that in 1208 the family had acquired Newland Castle and given it the Pembridge name; this Pembridge Castle was over twenty-five miles away from the settlement of Pembridge, near Welsh Newton (on the Herefordshire-Monmouthshire border); who built this castle is uncertain, but Matilda de St. Valery, wife of William III de Braose, and a later Ralph de Pembrugge (ca.1135) have been put forward as candidates. This Pembridge Castle came to be the chief seat of the family, which seems to have had several branches, so that the genealogy of the family is somewhat nebulous.

Nonetheless the manor of Pembridge presumably represents the earliest seat of the family, from which it took its name. By 1203 Henry de Pembrugge held five knight's fees in Herefordshire, one presumably being Pembridge. Henry was succeeded by son Ralph in 1211, though at this period the Braose lands were in the king's hands. Overlordship of Pembridge, through the Honour of Radnor, was in a weakened state (due to the minority of unmarried Braose heiresses) at the time that a later Henry de Pembrugge, who had inherited by 1230, obtained in 1239 licence for a Monday market and October fair at Pembridge manor. The following year this Henry issued a charter establishing a liber burgus there. Interestingly, at some point in the 1230s Ralph had married the widow of Stephen de Ebroicis, Isabel de Cantilupe, who already had familiarity with market foundations through both her birth and marriage families, though whether she influenced Ralph's plans we cannot know. This was not the first association of the Pembrugge and Cantilupe families, for at the death of one Ralph de Pembrugge, ca. 1216, William de Cantilupe sued Reginald de Braose for custody of the heir (another Ralph), though Reginald defeated the suit by arguing that not he but the late Ralph's half-brother Henry had custody; William apparently then sued Henry, for in 1222 the sheriff was ordered to see that Pembridge Castle was delivered to William.

There seems to have been a string of men named Henry de Pembrugge in this period, most probably fathers and sons, and one of them was escheator of Worcestershire in 1250/1, then sheriff of Herefordshire in 1255. Around this time he married Elizabeth Gamages and through her inheritance came into possession of the Welsh manors of Boughrood and Trewern in Elfael, probably also part of the Honour of Radnor; in 1335 Edward de Pembrugge would obtain licence for a market and fair at Boughrood – perhaps a renewal, for the Bishop of St. David's was said to have been granted such rights there in the late thirteenth century. The same Henry assisted Roger de Mortimer of Wigmore – who had become his landlord in 1247 through marriage to a Braose heiress – in the defence of the Marches against Llewellyn ap Gruffydd and, after Roger's defeat in 1262, Henry was ordered to defend his castle in the Marches (whether one of the Pembridge castles or that at Boughrood is not specified), and later to defend Mortimer from his enemies, though it is not evident he was very energetic in that regard. For Henry appears to have supported the Montfortian party in the civil war and, though he survived Evesham, subsequently continued defiant, was imprisoned in Wigmore Castle (1265), and his estates delivered into Mortimer's hands. Mortimer, correctly anticipating that this disgrace might be reversed in time and the estates restored, took advantage of his possession of Henry and of his sons to extort from him a number of documents, and acknowledgement of the same in the king's court, securely conveying Pembridge to Mortimer.

Though Henry's like-named son in 1267 tried to regain Pembridge at law, Mortimer had arranged matters too well, and the legal challenge was defeated. After this the family seems to have counted Pembridge lost, though they remained lords of Pembridge Castle at Welsh Newton. Henry junior sought to rebuild family fortunes in other directions; around 1270 he married Orabilia de Harcourt, whose descent from the de la Zouche family brought lordship of Tong (Salop., originally held of Braose overlords) into the Pembridge family. In 1271 Henry and Orabilia acquired licence for a market and fair there; descendants rebuilt the castle, turning it into a residence, and built a church there.

The borough at Pembridge, although one of the latest such foundations in its county, shows signs of having prospered; that a number of buildings dating to the fourteenth century have survived suggest it was not among the places devastated during the Glendower rebellion. It was one of the nine Herefordshire market towns to survive to the close of the Middle Ages, although declining in the post-medieval period, perhaps slowly losing ground to its competitor at Kington; in 1610 it still seemed to have enough potential for James I to grant the manor and borough to the Earl of Essex, though that perception proved mistaken.

Any earlier prosperity was despite the changes in lordship and no subsequent efforts to maintain a licensed market at Pembridge. Once Roger de Mortimer had secured his hold on Pembridge there is no evidence he, or any of his successors, renewed its market licence. This is surprising, considering the amount of effort Roger put into securing possession of Pembridge. There may have been bad blood between the two families, so that the Mortimer motivation was simply revenge, or perhaps even a desire to prevent Pembridge's market becoming too competitive with other Honour of Radnor markets. The Mortimer lords showed no great interest in Pembridge, often including it among the lands held by their dowagers, although Pembridge would come into royal possession in the person of Edward IV, a Mortimer descendant. Yet there is the slightest hint, from the quo warranto proceedings of 1292, that perhaps the borough component of Pembridge had been gifted to the dean and chapter of Hereford cathedral, for they were called to justify their administration of the assizes of bread and ale at Pembridge. On the other hand it may simply have been that the Mortimers, being a power in the region, did not feel it necessary to take out a licence for Pembridge. There was nothing to prevent a market from continuing to take place there, unless a challenge were issued by some neighbouring licensed market. In theory lack of a licence could have obstructed the collection of market tolls at Pembridge, but Mortimer will was not easily gainsaid; that lack would not have prevented the manorial justice system from continuing to deal with, and profit from, legal disputes stemming from commercial transactions.

Pembridge's moated castle and manor-house – whose function was remembered in the name of its replacement building, Court House Farm – stood in a central position on the south side of the settlement, with the churchyard immediately north of it. The original entrance of the enclosure would have been on its west side. A holloway led from the bailey to the church and on to the main road; it is probable that the castle-church combination was the focus for early settlement in the Domesday manor. Past the north end of the churchyard ran the through-road. On the north-west side of the churchyard this road broadened out where joined by the lane coming from the manorial enclosure, and there lay the marketplace, roughly triangular in shape; this marketplace could been a feature of the pre-urban settlement, or have been carved out of the churchyard at the time the borough was founded. The stretches of road on either side of the marketplace were later known as West Street and East Street. Along both sides of East Street and the northern side of West street we can distinguish a layout of burgages – probably the borough established 1239; this is less evident on that stretch of the southern side of West Street closest to the marketplace, and it appears the earlier village may have lain there. Running northwards off East Street, just east of the marketplace, Bridge Street headed towards a crossing of the Arrow; the name of the town suggests such a crossing, and presumably a track leading to it, to have long been in existence, but Bridge Street seems to have been a service route, rather than a focus for intensive settlement.

That no additional residential streets appear to have come into being during the Middle Ages, the apparent absence of back lanes and relative scarcity of buildings in the rears of burgage plots, as well as the fact that street frontages of the early burgages were relatively generous, all tend to indicate that even during its more prosperous period Pembridge was not attracting the volume of new blood that would have put pressure on land. There is nothing that points to it having been a market town of any great note.