The "Great Silence" has been discussed here before; I agree it's an interesting topic.
Personally, I believe the best answer to Fermi's question is probably something like this: intelligent life is rare to begin with, and species willing to expend resources to travel or even communicate over interstellar distances rarer still. So there just aren't that many explorers / would-be empire builders whizzing around out there. Even if something like the Alcubierre warp drive (http://whywontgodhealamputees.com/forums/index.php/topic,24931.0.html
) is possible, the universe is just so damn big
that the "nearest" interstellar civilization is probably a looooooooooooooooooooong way off.
Add that we haven't been asking the question and really looking for more than a few decades, and maybe it isn't as much of a paradox as we think it is...