I think the implication is that Von Bork is so knowledgeable that he can guess just which minister's country house is meant, and he has also been a guest at that house.
If I am correct, this not an idiomatic expression at all, but is meant literally: Van Bork has been at that house. I must admit that I do not have any clear evidence of this, but I think it is more logical than assuming that he is speaking metaphorically about a somewhat simiansimilar experience. After all the essence of Holmesian adventure is the ability (mostly of Holmes, but not always) to come to correct conclusions on what seems quite insufficient evidence.