Branches
Why are the first four tales called the “branches of the mabinogi”? This is easy, because that’s what the tales call themselves! Here’s the opening of Manawydan fab Llŷr on folio 182v in the manuscript of the Red Book of Hergest:

Above the beginning of the story, the scribe has written in red, “llyma y dryded geinc or mabinogi”, meaning “here’s the third branch of the mabinogi”. The word “geinc” (modern “cainc”) means literally “branch” but metaphorically a portion or episode of a larger tale. All four of the branches end with the formula, “ac uelly teruyna y geing honn [yma] or mabinogi”, meaning “and thus ends this branch [here] of the mabinogi”.
(Note on orthography: in medieval spelling the sound /v/ was written “u”, but in modern spelling it is written “f”, so that “uelly” corresponds to modern “felly” meaning “thus, so” and “teruyna” is related to modern “terfyn” meaning “limit, boundary”.)
Mabinogion
How did these eleven tales come to be collectively known as the “Mabinogion”? The word itself is mysterious since it only appears at the end of Pwyll: the other three branches have “mabinogi”. This must mean that it is a scribal error of some kind. Sebastian Rider-Bezerra reports the following theory:
The form Mabinogi, which occurs at the end of the Second, Third and Fourth Branches, is already plural. The error made in pluralizing an already-plural form can be explained by a scribe’s error, at the end of the First Branch in the Red Book of Hergest, transposing the plural ending of a word directly above mabinogi, to inadvertently create the more famous title “Mabinogion”.
Sebastian Rider-Bezerra (2011). ‘The Mabinogion Project: A Brief History of the Mabinogion’. University of Rochester.
Here’s the ending of Pwyll and the beginning of Branwen on folio 179v in the manuscript of the Red Book of Hergest:

In this manuscript, the scribe wrote out the ending formula for Pywll in black, “Ac uelly y teruyna y geing honn or mabynnogy”, and the on the next line wrote the opening formula for Branwen in red: “llyma yr eil geinc or mabinogi” meaning, “here’s the second branch of the mabinogi”. At the end of the line in red, the letters “on” appear in black, but boxed off in red.
Rider-Bezerra’s explanation is the scribe lost track of where they were, and copied the “on” from “honn” on the previous line. This doesn’t seem very plausible in the Hergest document, but perhaps the error was in the exemplar, and the Hergest scribe has simply copied the error, but indicated their dissatisfaction with it by boxing off the erroneous “on”. (If “honn” had been spelled “hon” in the exemplar, a common variant, then it would be easier to imagine how the error might have been made.)
However the error occurred, the suffix “-ion” is a common plural ending in Welsh, for example “cantorion” (singers), so “mabinogion” has been interpreted as representing a plural form of “mabinogi”. Sebastian Rider-Bezerra says that this interpretation originated with William Owen Pughe:
William Owen Pughe (1759–1835)’s best-known work, the Geiriadur Cynmraeg a Saesoneg (Dictionary of Welsh and English), a Welsh–English dictionary including words both modern and archaic, was published in 1793. In this Dictionary the following entry is given for “mabinogion”: “mabinogi, s. m. pl -ion (mabinawg): Juvenility; juvenile instruction; the amusement of youth, the title of some ancient tales.”
Sebastian Rider-Bezerra (2011). ‘The Mabinogion Project: A Brief History of the Mabinogion’. University of Rochester.
You can see an 1873 edition of Pughe’s dictionary on the Internet Archive, where the entry for “mabinogi” is on page 311. Pughe’s definition was based on the observation that “mabinogi” starts with “mab” meaning “boy”. (But this has long been considered doubtful: see below.)
Based on his theory about the word’s meaning, Pughe planned to publish a translation of all the medieval Welsh tales, not just the four branches, under the title “Mabinogion”. This project was not completed, but his work is thought to have influenced Charlotte Guest (1812–1895) whose translation of the stories into English, the first complete translation to appear in print, was published under the title The Mabinogion in seven volumes from 1838–1845. The name has stuck ever since.
Meaning of mabinogi
If “mabinogi” does not mean “children’s story” as suggested by Pughe, what does it mean? Unfortunately, there is no consensus.
It has long been recognized that word contains the regular Welsh word for ‘son, boy’, mab. It was thought, therefore, that the tales had something to do with youth, either tales for boys, perhaps for their edification, or apprentice tales for those learning the story-telling art. Alternatively, it was noticed that mabinogi translates Latin infantia in a fourteenth-century aprocryphal gospel of the boyhood of Jesus. On the basis of the French form of the word, enfance, it was thought that the tales were histories of the birth, boyhood deeds, later feat of arms, of certain heroes. The difficulty in all of these guesses is that they fit none of the four branches of the mabinogi, nor do they fit the four branches as a whole.
Patrick K. Ford (2019). The Mabinogi and Other Medieval Welsh Tales, p. 2. University of California Press.
Eric Hamp published a theory (‘Mabinogi’, Transactions of the Honorable Society of Cymmrodorion, 1974–1975, pp. 243–249) that the word “mabinogi” meant the collection of stories relating to the Brythonic god Maponos. The trouble with this theory is that Maponos does not appear in the four branches, and so Hamp’s theory is that the four branches we have are only part of a larger mythological cycle in which Pryderi fab Pwyll, a character who appears in all four extant branches, was the father of Maponos. (Note: I have not read Hamp’s paper, but only secondary sources like Ford and Fulk, so I may be misrepresenting it.)
Robert Fulk recently criticised Hamp’s theory on etymological grounds, and proposed an alternative:
It has been some forty years since the present writer was first struck by the phono- logical resemblance of the name Mabinogi to Old Irish Mac ind Óc ‘The Young Son’, a name that appears frequently in the critical literature on the Four Branches. It is an epithet of Óengus, son of the Dagda and Boann, often regarded in the literary commentary as the Irish equivalent of Maponos.
Robert D. Fulk (2019). ‘The derivation of the name Mabinogi’. Studia Celtica 53, p. 50.
The difficulty with Fulk’s theory, however, is the same as with Hamp’s: Mac ind Óc no more appears in the four branches than does Maponos.