(I remember borrowing a physical copy of the book at the library, but I never got to finish it before returning it.) The name is Hispanic, and in the novel some key scenes happen early on:
Most of your description sounds like One Hundred Years of Solitude by Gabriel Garcia Marquez. This book follows the history of the Buendía family in the fictional village of Macondo in Colombia - there's the Hispanic connection you were looking for, and "everyone has the same name" because they're the same family. In fact, as well as the same surname, many of the first names are also repeated from one generation to the next, especially Aureliano, José, and Arcadio.
A kid runs away to join the circus.
This must be the second José Arcadio, first son of the original patriarch. See below.
The dad becomes a reclusive inventor, while living in the same house as the rest of his family.
That would be the original José Arcadio Buendía, patriarch of the family and founder of the village Macondo. He takes an interest in alchemy thanks to the inspiration of the gypsy Melquiades. From Wikipedia:
For years the town is solitary and unconnected to the outside world with the exception of the annual visit of a band of gypsies, who show the townspeople technology such as magnets, telescopes and ice. The leader of the gypsies, Melquiades, maintains a close friendship with José Arcadio Buendía who becomes increasingly withdrawn, obsessed with investigating the mysteries of the universe presented to him by the gypsies. Ultimately he is driven insane, speaking only in Latin, and is tied to a chestnut tree by his family for many years until his death.
A different kid attempts to have relations with a gypsy stuck in the form of a frog, when the same circus comes back to town.
If my identification is right, then there are two details you've got wrong here. First, this kid is not a different kid but the same one as in your first bullet point. Second, the gypsy is not literally stuck in the form of a frog: she's described metaphorically as a "frog" due to her thin, weak body, but she's a human girl. The magic/fantasy elements in the book are not that explicit! From the novel itself (emphasis mine), showing José Arcadio's encounter with the gypsy and his subsequent departure with the gypsy caravan (not technically a circus):
José Arcadio and the gypsy girl did not witness the decapitation. They went to her tent, where they kissed each other with a desperate anxiety while they took off their clothes. The gypsy girl [...] was a languid little frog, with incipient breasts and legs so thin that they did not even match the size of José Arcadio's arms, but she had a decision and a warmth that compensated for her fragility. [...]
José Arcadio's companion asked them to leave them alone. And the couple lay down on the ground, close to the bed. The passion of the others woke up José Arcadio's fervor. On the first contact the bones of the girl seemed to become disjointed with a disorderly crunch like the sound of a box of dominoes, and her skin broke out into a pale sweat and her eyes filled with tears as her whole body exhaled a lugubrious lament and a vague smell of mud. But she bore the impact with a firmness of character and a bravery that were admirable. José Arcadio felt himself lifted up into the air toward a state of seraphic inspiration […] It was Thursday. On Saturday night, José Arcadio wrapped a red cloth around his head and left with the gypsies.
Opinion section: I never managed to finish this book either, but not because I had to return it to the library. It's pretty grim in many ways: the family starts off relatively normal, but as the story evolves, we see assassinations, mysterious deaths, incest and other sexual aberrations, and even gruesome deaths of children. The story isn't even saved by having a clear plot that makes you want to read on and find out what happens next. There's no underlying storyline: it's rather a series of more or less disjointed vignettes of different people in the same family and what they do with their lives.
