The Buddhist hundred and eight is usually explained with a formula: six senses, times three valences of pleasant, unpleasant and neutral, times two of internal and external, times three of past, present and future. That formula is a later reconstruction and it is not the canonical derivation.
The Pali canon supplies one directly. The Tanha Sutta at Anguttara Nikaya 4.199 sets out eighteen craving-verbalisations dependent on what is internal and eighteen dependent on what is external, giving thirty-six, and then takes those thirty-six over past, present and future to give a hundred and eight. The text does the arithmetic explicitly. Anyone citing the hundred and eight defilements has a canonical source to cite, and it is worth citing rather than the popular formula.
Related enumerations appear across Buddhist traditions, and the number is put to liturgical use: a hundred and eight prostrations are a standard practice in Korean Seon, and the joya no kane, the striking of the temple bell a hundred and eight times on New Year's Eve in Japan, is explicitly understood as dispelling the hundred and eight afflictions.
The arithmetic of the number itself is elegant and genuine. A hundred and eight is 2 x 2 x 3 x 3 x 3, which can be written as one to the first power times two squared times three cubed, an unusually tidy factorisation that has been noticed for a long time. It is also divisible by 1, 2, 3, 4, 6, 9, 12, 18, 27, 36 and 54, which is why it subdivides so conveniently for practice: a hundred and eight, fifty-four, thirty-six, twenty-seven, and nine are all workable rounds.