Hamlet refers to the improvisation of clowns, rather than of actors in general. His reasoning is explicit: he doesn't want the audience to laugh. The play has a point ("some necessary question of the play"), and he doesn't want it lost on the audience just to satisfy some clown's "pitiful ambition" (to be noticed, attract attention, and possibly patronage).
Shakespeare incorporated clowns into most of his tragedies, including Hamlet. The Gravedigger scene is an excellent example of black comedy, as is the Porter in Macbeth. While Hamlet's "The Mousetrap" doesn't appear to have any comic scenes (we see what is, presumably an abbreviated form of it), it's important to Hamlet that the clowns not change the tone of the play.
Clowning had a strong improv tradition, inherited from Commedia dell'Arte. They would start with a plot outline, but the actual text would be improvised (and it was sometimes called "commedia improvviso"). Shakespeare and his contemporaries often followed old, well-known stories, and wrote specific dialogue for them: it wasn't improvisation, but it reflects the way audiences were coming to see a production rather than a story. The would usually already know the story: Hamlet goes out his way to clue them in with a dumbshow (mime) before The Mousetrap.
There is thus some tension between the clowns, who see it as their job to make people laugh, and the playwright who has a dramatic goal in mind. The clowns can play with the tension created by the dramatic scenes, but if they break it entirely, then the purpose of the play is lost.
Making people laugh is always a hit-and-miss operation. All theater requires actors to feel out what is working for this audience, but comedies get explicit feedback in the form of laughter. The laughter is contagious: it communicates not just to the actors, but to other audience members. (Having one person laugh in an audience is a huge help: a "shill" can be the difference between audiences who are slightly amused and audiences having the time of their lives.) So a comic actor has plenty of motivation, and expertise, to play up to what's working today -- which may be totally different from what worked yesterday.
(Or at the matinee. Matinee audiences are particularly problematic, and actors today often dread them. They have a tendency to pull out all the stops just to get something. For Shakespeare, of course, all performances at the Globe were matinees.)
So I would view Hamlet's talk of improvisation to refer specifically to comedy, and possibly just to this performance of this show. The "Advice to the Players" speech is often taken as coming directly from the author, and there's certainly good reason to take it more generally (especially, as I said, since Hamlet's play doesn't seem to have any clowns). So it may well also be Shakespeare telling Will Kemp to tone it down.