I'm more looking at the issue from a socioeconomic stand point. Resource constraints/allocations are a different issue and should be considered separate from population rates and further by geography and category (energy, food, building materials).
Considering population rates, it depends on the country/society under consideration. For modern post-agrarian, industrialized developed countries (US, Canada, most of Europe, etc) it is critical to maintain (on average) replacement level reproduction rates. Failure to replenish/maintain workers places sever strain on services and production and can lead to economic stagnation/collapse.
We need look no further than Detroit to see what happens to an area when the population decreases. No money for public services, fire departments, utilities, schools. The remaining residents struggle, property values fall, crime sky-rockets.