The Book of Mormon names its own writing system directly, in a passage its narrator Moroni includes near the record's close. He explains that the Nephites, the people whose history the record covers, had not preserved Hebrew, their ancestors' original language, in its pure form; instead, he writes, they had altered it according to their own manner of speech over the centuries since Lehi's family left Jerusalem around 600 BCE. What they wrote the record in instead, he says, was a form of Egyptian likewise reformed and altered by their own usage, learned because Lehi himself had been taught Egyptian, and passed down and adapted across the generations that followed him in the new land.
Moroni is explicit that this was a matter of practicality rather than preference: Hebrew, he writes, took up more space to convey the same meaning than the modified Egyptian characters did, and since the plates were a fixed, heavy medium with limited room, compressing centuries of history onto them required the more economical script. He adds a line of resignation rather than confidence, that if the plates had been large enough to allow it they would have written in Hebrew, and that no other people on earth would know how to read Reformed Egyptian besides the record's own writers and whoever God chose to give the gift of translating it.