Yes, the device of the good and the bad angel had definitely been used before, for example by Christopher Marlowe in The Tragical History of Dr. Faustus. Marlowe's plays are generally hard to date and Doctor Faustus was probably written between 1587 and 1589, while The Merchant of Venice was presumably written between 1596 and 1599, i.e. several years after Marlowe's death.
In Doctor Faustus, the good and the evil angel already appear in the very first scene:
Good Angel: O Faustus, lay that damned book aside
And gaze not on it, lest it tempt thy soul,
And heap God's heavy wrath upon thy head!
Read, read the Scriptures—that is blasphemy.
Evil Angel: Go forward, Faustus, in that famous art
Wherein all nature's treasure is contained;
Be thou on earth as Jove is in the sky,
Lord and commander of these elements.
They come back in scene 5:
Good Angel: Faustus, repent! Yet God will pity thee.
Evil Angel: Thou art a spirit; God cannot pity thee.
And again in scene 6:
Evil Angel: Too late.
Good Angel: Never too late, if Faustus can repent.
Evil Angel: If thou repent, devils shall tear thee in pieces.
Good Angel: Repent, and they shall never raze thy skin.
(I am quoting from an edition that has only scene divisions, no acts.)
However, the origin of the device is older and was already used in the so-called morality plays, e.g. the fifteenth-century The Castle of Perseverance. In the text available at the Robbins Libary, University of Rochester, they are listed as "bonus angelus" ("good angel") and "malus angelus" ("bad angel").
I am aware of two sources for this motif:
- The parable of the choice of Heracles/Hercules by the Greek philosopher Prodicus. In this parable, two women appear to Hercules: one who is voluptuous and who promises him the shortest road to the enjoyment of pleasure, and one of more dignified beauty who urges him to follow the longer and more arduous path of noble deeds. Hercules chooses virtue over vice.
- The Christian concept of the guardian angel, which may have some sort of demon as its counterpart. The concept of guardian angel was never formulated by the Church but can be found in the writings of one of the church fahers, St. Jerome: "how great the dignity of the soul, since each one has from his birth an angel commissioned to guard it" (Commentary on Matthew, xviii, lib. II)). (See Catholic.com.) The "bad angel" presumably derives from the angels that were cast out of heaven together with Satan (Revelation 12:7-9).
Edit, 22.12.2020: I have added a link to the follow-up question about the dating of Doctor Faustus.