According to The Free Dictionary, the word "entail" means:
To abridge, settle, or limit succession to real property. An estate whose succession is limited to certain people rather than being passed to all heirs.
In real property, a fee tail is the conveyance of land subject to certain limitations or restrictions, namely, that it may only descend to certain specified heirs.
Basically, this was a way of preventing the land from being sold or inherited outside of the family.
The fact that the land was entailed evidently caused him some financial difficulty because he couldn't sell the land to raise cash. It would also mean that he wouldn't be able to borrow money against it, because the fact that he wasn't allowed to sell it would mean that he also couldn't use it as collateral. (That's why the book that says that the land that wasn't entailed was mortgaged to the hilt).
The book implies that he had consulted Atticus Finch on the entailment, which seems to suggest that he was actively trying to find a way to sell the land (or borrow against it) because he needed the money.
Another example of a similar legal arrangement is in Pride and Prejudice, where the land was entailed in such a way that only male family members could inherit it, so all of the Bennet daughters were being deprived of their sole source of income.
