It might have been a college statute at the time, or possibly just a "gentlemen's agreement".
Cambridge and Oxford colleges are strange, anachronistic institutions, quite unlike any other organisation of higher education in the UK (or probably anywhere else, with the possible exception of institutions in other countries which are specifically modelled on Oxbridge colleges). They are not subject-based like departments or faculties, and they play little role in the teaching or research aspects of the university, but many of them are among the oldest and most prestigious institutions in the country, as well as being significant landowners worth hundreds of millions of pounds. These assets are usually managed by the board of Fellows (college members, professors and the like) presided over by the Master.
Each college has its own Statutes and Ordinances which govern the running and affairs of the college. I haven't checked all of these for all Cambridge colleges, let alone all the revisions to them through the centuries (if such a thing is even publicly recorded anywhere), but I did check the one for Christ's College, where C. P. Snow was a fellow and on which the college featuring in The Masters may be based:
Whenever the office of Master falls vacant the Fellows shall as soon as may be elect a person to fill the office. All Fellows, including those granted or deemed to have been granted leave of absence by the Council, shall be entitled to vote and no person shall be elected unless a majority of those entitled to vote have voted in favour of him.
The Master shall take no part in the pre-election of his successor. If there is no Master and the last holder of that office is a Fellow, he shall take no part in the election of his successor and he shall not count as a Fellow for the purposes of this Chapter.
The Governing Body may by Ordinance make such additional rules as it deems necessary concerning the procedure to be followed at meetings called for the purpose of the election or pre-election of a Master.
-- Christ's College Cambridge Statutes
At this particular college, for whatever reason, the outgoing Master is not allowed to cast a vote for the new one. It's also worth noting that additional rules may be added if deemed necessary (including potentially a rule preventing candidates from voting for themselves).
Another interesting case to consider is the Mastership election in St. Catharine's College in 1861, which created something of a scandal in the college:
The latter part of the 19th century was dominated by what became known as the 'Robinson Vote'. In 1861, when Philpott was made Bishop, the election for his successor took place, with Charles Robinson and Francis Jameson standing for the position. The actual facts of the case are obscure, but Jameson voted for Robinson, presumably expecting Robinson to vote for him, but Robinson voted for himself, enabling him to win. The episode was well-known throughout the University and caused the reputation of the College to suffer for a good number of years.
-- St. Catharine's College website
It seems unclear whether the natural or expected thing to do was for both candidates to vote for themselves or for them to vote for each other. Wikipedia says "Jameson naturally voted for Robinson, however Robinson voted for himself"; British History Online says "Jameson and another voted for Robinson, two for Jameson, but Robinson, following an old tradition of the College, voted for himself, and so secured the election." In any case, it seems likely that this was the very incident Snow had in mind when writing The Masters, given the notoriety of this scandal in the university. The St. Catharine's College magazine explicitly compares the Robinson scandal with The Masters:
The most infamous controversial election is the one which blighted the College’s reputation for half a century and for which St Catharine’s for a time was unjustifiably notorious. Jones and the Quincentenary Essays discuss the 1861 election at length – worth reading if only to compare with CP Snow’s The Masters. [...] The two junior Fellows had decided to vote for Robinson, but, the evening before the election, they had a discussion and changed their minds. They therefore voted for Jameson; the other three, including Jameson, voted for Robinson, despite Robinson having previously repeatedly urged Jameson that he should vote for himself as was the custom in such elections. For some time all was sweetness and light until Jameson belatedly discovered Robinson’s marriage, upon which he made a great scandal while Robinson was away on honeymoon, broadcasting around that Robinson won the election only because he voted for himself. Robinson offered to stand down, but even Jameson agreed that the election should stand. Jones (who was elected a Fellow towards the end of Robinson’s Mastership and so had first-hand experience of the effect) writes ‘The scandal caused ... was probably the greatest disaster that ever happened to any college...’. Robinson was Master for nearly fifty years and his presence a constant reminder of the ‘Robinson vote.’
The conclusion (Wikipedia notwithstanding) appears to be that voting for oneself was usual in such elections, and Jameson perhaps made a mistake. However, this incident may have led to ordinances against voting for oneself in other college elections: no other college would want a scandal like St. Catharine's had.
Another case, this time from Gonville and Caius College:
Three scrutinies were held. At the first two. Guest obtained exactly half the votes. At the third he was induced to give his own vote for himself, which secured the requisite majority, and he was duly elected.
-- "Biographical history of Gonville and Caius college, 1349-1897; containing a list of all known members of the college from the foundation to the present time, with biographical notes
The incoming Master "was induced" to vote for himself, having not done so in the first two rounds of voting. It seems that voting for oneself, while not against the rules, was at least not just the natural and expected thing to do.
From Trinity College, with also a note about non-collegiate university practices:
And he, this deponent, lastly maketh oath, that as no clause in the statute about Royal Professorships does forbid any elector to vote for himself, if he pleases to be a candidate; for the constant practice in like cases both in Trinity College and the University for near twenty years, that this deponent hath been Master there, and, as he believes, for time immemorial, doth warrant the said voting for one's self. It being notoriously known that the eight Seniors of the said College, who, with the Master, are electors into all the College offices, do yearly some of them vote for themselves for the offices of Vice-master, of Senior Dean, and of Senior Bursar. And in the said University, whenever an University living, or a place of member of Parliament, or yearly the University lectureships of rhetoric, &c. do fall vacant, the candidates for any of the said places are allowed to have a right, and, as occasion requires, are accustomed to vote for themselves.
-- The Life of Richard Bentley, D.D., Master of Trinity College
Again, although it was permitted to vote for oneself, someone felt the need to write a lengthy paragraph to justify that it was indeed permitted. It seems that some Fellows might have felt it inappropriate (ungentlemanly?) to vote for oneself.
From Oxford this time, Balliol College:
Best's side argued that the custom of private and secret voting in the Chapel was not an inflexible constraint. The Visitor himself, they pointed out, had been elected at a meeting convened for the purpose by the Master in his Lodging, because he had 'taken physick and was fearfull of venturing out'. Several other instances were cited to show that the College was free to use its discretion in matters not explicitly covered by the Statutes. Their case was voluminous, but weak, lacking examples of elections which had involved real decisions by voting. A second line of attack was aimed at Quick. By some oversight he had failed to take the Oath of Allegiance, and it was maintained that his Fellowship was therefore void, and his vote worthless. Leigh's team entered fully into the spirit of the thing. Adherence to ancient custom was strongly defended. Was it in order for Best to vote for himself? Was Lux compos mentis? If not, was his vote valid? And so on.
-- Balliol College: A History
Among many other nice points of college regulations, the issue was raised of whether or not a candidate was allowed to vote for himself.
TL;DR: although I haven't found any Oxbridge college statutes or ordinances, past or present, which specifically disallow voting for oneself in Mastership elections, doing so was often considered controversial. So, it's plausible based on reality either that the fictional college of Snow's The Masters imposed ordinances disallowing it, or that the candidates in this particular story had a "gentlemen's agreement" not to vote for themselves.