I remember reading this perhaps during elementary school.
While this wasn't the entire plot of the book, one of its major points was a mother who slept with a black man and had a black son. She and her other (white) son bully and starve the black boy sometimes, and she's ashamed of him.
Near the end of the book, the town finds the black son's remains days after he dies, after which they go and kill the mother and white son.