I don’t see it as just semantics at all. If you choose to not help, you choose to allow pain to continue. You become part of the problem.
Oh, so it's no different than walking up to someone begging for money and kicking them in the head to rob them of what little money they managed to accumulate?
[1]Callous indifference is not the same thing as intentionally causing harm. You have a point that it can contribute (often seriously) to a problem, and it is something that needs to be addressed, but I don't think it helps to try to lump the two together. You can't deal with callous indifference by using the same methods you use to deal with intentional harm. So it's not just a semantic difference, it's the fact that the two are dissimilar and shouldn't be treated the same way.
There may be a difference in scope but there is no difference in intent. The intent is “my religion is right and I have the right to do what I want to someone because of it”.
There is both a difference in scope and a difference in intent, which I'll explain by analogy.
Let's take bullying
[2]. Someone who suffers from this has to deal with active harassment - frequently by a few individuals, and infrequently by others - as well as a form of callous indifference - people who say "just ignore them" and other platitudes. Both of these are harmful to the person being bullied, but there's no question that the former is far harder to deal with. The main problem with the latter is that it can contribute to the overall problem despite whatever the person might actually intend
[3].
Now, it's not the same thing with religion, but it's close enough to count. People who think they're doing something helpful because of their religion may be misguided or simply wrong, and may actually be doing more harm than good with their efforts. But it is generally not an issue of them intentionally and deliberately trying to do harm to the people they're trying to help, it is an issue of not understanding that their attempts to help are actually not very helpful or even possibly harmful. And I don't think it helps to reinterpret someone's actions in the worst possible light, anymore than it would to do so in the best possible light. The best thing to do is to simply be accurate - to explain why something thought to be helpful was actually harmful, without unduly castigating the person doing it.
Railing about Mother Teresa being sadistic and manipulative isn't going to get through to people convinced that she was saintly and beatific any more than my railing at well-meaning people who give seriously bad advice to those being bullied would show them that their advice was harmful and why.