Bromyard arose on the gentle slope of the Frome valley. The River Frome skirted the site on east and south, with fords on both sides. From before the time of Domesday, throughout the Middle Ages, Bromyard was the administrative centre of a large manor held by the Bishop of Hereford, who had a palace there by the thirteenth century and presumably a manor-house earlier; in this building the bishop's bailiff would have presided over the local court, and rents and possibly market tolls would have been payable. A minster church known from ca. 840 was replaced by a parish church in the late twelfth century. A borough is thought to have been founded there by Bishop Capella in the 1120s, though the first reference extant to burgesses is not until a survey of episcopal lands ca. 1288, when there were 255 burgage tenants living in seven streets, each paying rents of 12d for a burgage plot, or 6d. for a half-size plot; 26 selds are mentioned, each rented for 4d. annually, and held by persons who included a fishmonger, a tailor, and four mercers. Surnames also suggest the presence of weavers, dyers, and a blanket-maker, and evidence the leather industry, workers and dealers in metal goods, and of course the food services industry.

The absence of a market grant to Bromyard suggests one likely existed at Bromyard before Capella's bishopric; at a later period it was held on Mondays. It was left to a successor to obtain a licence for a two-day fair in May, in 1218, probably once the volume of commerce at Bromyard's market had developed enough to show the need. In 1288 the fair profits were estimated at 20s; toll revenues were rolled up with court perquisites at £13, a figure that, despite being an estimation, suggests the market was doing moderately well. An extent made in 1404 of bishopric manors and other lands in Herefordshire found that the Monday market at the borough of Bromyard produced about 40s. from tolls, while the fair tolls and its other profits yielded 50s.

The core of medieval Bromyard was evidently the church, with the episcopal residence south of it. Domesday Book points to barely sixty residents, yet even this modest number made Bromyard one of the larger communities in the lightly populated county. Early settlement is likely to have been around the church and along a through-road that passed slightly east of church and manor-house; this was a long looping road – its central portion, now Church Street, probably that referred to in the 1288 survey as Old Street – which entered the site via the two fords at north-east and south-east corners of the site. The northern part of the road went on to Stourton-on-Severn, while the southern part connected to Worcester. South of the church, and just east of the main gate to the bishop's palace, the road opened westwards to form a largely square space, still today known as Market Square; this must have been the site of the medieval market, for cartographic evidence shows regular-sized property plots aligned around that square and along streets running off it, suggesting the marketplace to have been the core of the planned town. The north-east and south-east corners of this marketplace were entrances for the through-road; off the south-west corner what is now Broad Street/High Street – wider in the Middle Ages than it is today – runs westerly on a north-looping course, paralleled by another street (Rowberry) running from the north-west corner of the marketplace, and with burgage plots stretching between the two streets. The High Street was referred to in the survey as Novus Vicus; it and Rowberry terminate at Cruxwell Street, which runs north-eastwards to the churchyard and is seen in the 1288 survey as vico de Crokeswalle, one of the three most heavily populated streets (along with Old Street and New Street). Slightly less heavily populated was the vico de Stonhulle, believed to be Pump Street/Tower Hill, running southwards off the High Street. Whether Cruxwell, Pump and the further stretches of Rowberry and High Streets were part of the original planned town or represent an early phase (pre-1285) of expansion of urban settlement we cannot be certain.

By the time of the 1288 survey, settlement had spread over an extensive area. That the palace was no longer used by the bishops after 1356 does not appear to have had a significantly adverse impact on the local economy. Not that Bromyard was ever a town of great wealth; although its borough status was recognized by the government of Edward I, which invited it to send representatives to parliament, the burgesses soon begged off on the grounds they could not afford to pay the expenses involved. Nonetheless, Bromyard was one of a minority of medieval towns in Herefordshire to survive as a market town into the post-medieval period, and experienced a decline only in relatively modern times: by the time that Duncumb wrote his history of the county (1812) the town had shrunk, both in terms of the number of houses and of residents with burgess status, to below its medieval level, and the episcopal palace had been converted to the vicarage.