Indeed. Whether we're dealing with the actual universe or the equivalent of a computer program like the Matrix, it's basically communicated to us through our senses. Those senses may not be especially good, but they're consistent - if several people all look at a scene, they will all see the same scene.
Compare this to the imagination, which is pretty close to what a "world produced by consciousness" would be like. If I imagine something, say an imaginary world, it is extremely unlikely that someone else's imaginary world will be anything like my own without an external framework (for example, if we both imagined Tatooine from Star Wars). And even then, there will be dramatic differences between the two.
I only see two ways to produce a consistent "imagined world". One is for it to be the product of only one mind, with everyone else a figment of that mind; the other being that there's some kind of mechanism for each individual consciousness to share what it produces. Neither adequately explains the universe as it exists.
If the universe is simply imagined, then there's no reason for it to stay consistent and produce, say, a coherent timeline. When I imagine things, I seldom concern myself with the kind of linearity that we see in the actual world. I jump around depending on what I'm thinking about. I don't even imagine stuff from the same continuity a lot of times - it depends on what I'm actually thinking about the time.
And sharing information between consciousnesses gets to be problematic. How would this system determine which consciousness was 'dominant' for a given place or time? How would it ensure that they were all consistent with each other, so as to prevent continuity problems (like magic working in one place and science another, and a third where both worked)? How would it share information between consciousnesses to begin with, given that the 'world' wouldn't be a valid way to communicate information?
Like it or not, the idea of us experiencing a reality that exists whether we're here to see it or not just makes more sense than any other.