Book was set in the 1950s about a boy working for a racist store owner. The main character's brother was killed in a hit-and-run which sent his father into a deep depression. The store owner he works for is extremely racist and prejudiced, mocking his customers behind their backs. It's also heavily implied he abuses his wife and daughter. Eventually, the boy meets a Holocaust survivor who lost his family in WW2. He doesn't speak much and tips his hat to everyone who passes by him. He learns that the man is creating a wooden model of his old village. When he tells his boss about the man, his boss threatens him with expulsion from school unless he destroys the man's model.
This is Tunes for Bears to Dance To by Robert Cormier.
12-year-old Henry Cassavant moves with his parents to a new town to escape from the memories of Henry's older brother Diltz, who was hit and killed by a car. Henry contributes to his family by working at a grocery store for Mr. Hairston, a deceptive old man who makes rude and racist comments about the townsfolk that would walk by his store.
Henry follows the old man to an art center, where he meets him in person. From George Graham, the supervisor of the center, Henry learns the old man, Mr. Levine, is a Holocaust survivor who lost his family to the SS. Mr Levine goes to the art center every day to carve out a model of his old hometown, complete with carvings of all the people he had lost, including his wife and children.
Mr. Hairston tells Henry that he will let him keep the job and he will get his brother's headstone on one condition: he must destroy Mr. Levine's model village.
It's an exact match to what you describe, I don't have much more to add to this answer. How I found it: the winning search query was
1950s novel boy shop owner "holocaust survivor" lost family "model of his village" destroy. The exact quotes were a risk, as I didn't know if those exact phrases would be used even if all the details matched, but luckily it led me to this page where the book is described by the following blurb:
Eleven-year-old Henry Cassavant works in a grocery store for a racist grocer in a neighborhood of immigrants. His employer's racist comments disturb Henry, who has recently befriended a Holocaust survivor, Mr. Levine, who works tirelessly each day at the city's arts and crafts center, carving an intricate model of his village which was destroyed during World War II. When the village is finished, Mr. Hairston coerces Henry to destroy the village, threatening to fire him, to have his mother fired from her job, and to ensure that neither of them will be employed by any other merchants in town.
(second half of the plot summary removed because of spoilers, for anyone else who may read this)