The oldest sustained analysis of the syllable is also the shortest text in the Upanishadic corpus, and it does something unexpected with it.
The Mandukya Upanishad runs to twelve verses. It takes the syllable apart into three sounds, a, u, m, and matches each to a state of consciousness: a to waking, in which awareness is turned outward; u to dream, turned inward; m to deep dreamless sleep, in which there is no division into knower and known. It then names a fourth, turiya, which is not a fourth sound but the silence the three resolve into, and identifies that with the self.
The scheme is not decorative. It is an argument that the structure of a sound models the structure of consciousness, and it is the basis of the commentarial tradition that follows, particularly Gaudapada's verses on the text and Shankara's commentary on those.
A quite different account comes from the Chandogya Upanishad, which opens by identifying the syllable with the udgitha, the sung portion of the Vedic sacrifice, and builds its case from ritual rather than from psychology. That older ritual layer is where the syllable actually comes from: it functions in the Brahmanas as a formula of assent and response in liturgy, comparable in position to amen, before it becomes an object of philosophical attention.
The spelling in English books varies. Older works usually write Aum, reflecting the analysis into three; most modern works write Om, reflecting the sound. Neither is more correct; the Sanskrit is a single character.