I won't do any justice to Krauss or the science of his argument, but I figured I should at least attempt a summary.
Basically, 'nothing' can't exist very long. Even in empty space, or even an area devoid of anything (including empty space) quantum fluctuations are still going to occur. Occasionally, those fluctuations produce a small blip of matter and anti matter which very quickly annihilate each other and 'nothing' returns. It is possible, though, if there is just a bit of non uniformity in that blip of particle generation, for the entire system to expand rapidly and give rise to a whole new universe.
All the cosmologists out there are welcome to strangle me now.
Also, the big bang theory makes certain specific and testable predictions. One of those is the ratio of lighter elements to heavier ones. Our current big bang models predict near perfectly the exact ratios we see. If our theory of the big bang isn't (at least mostly) correct, this is the biggest coincidence in the history of coincidences, and then some.
We're on the right track.