Brown is an earth colour in the most literal sense, and the standard pigments are named after where the earth came from.
Umber is a clay containing iron and manganese oxides, named for Umbria. Sienna is a similar earth named for the Tuscan city. Both are sold raw and burnt, roasting deepening and warming the colour, and the four together, raw and burnt umber and raw and burnt sienna, formed the backbone of the warm neutral range in European painting for centuries. They are cheap, permanent and reliable, which is a rare combination.
Bistre was made from beech soot and used as a wash for drawing. Van Dyck brown, a peat or lignite earth, is rich and notoriously unstable, and paintings that relied on it have darkened badly.
And then there is mummy brown, which was made from ground Egyptian mummies. This is not a legend. It was manufactured and sold by colourmen in Europe from the sixteenth century into the twentieth, made from bituminous material and human and animal remains taken from Egyptian burials, and painters used it as a transparent brown glaze. Some of them, on learning what they had been working with, buried their remaining tubes. Supply and disquiet ended it in the twentieth century.
The colour's symbolic associations, humility, poverty and the monastic habit, come from undyed and cheaply dyed wool rather than from any of these pigments, and they run alongside the pigment history without touching it.