I think at least one chance for them to clarify their position, or genuinely answer to critique should be given. If it's clear that it's just a drive-by low-effort comment, or if they just have an axe to grind, only then should they be downvoted. If they are serial offenders, they typically get shadow-banned.

However, I disagree with you on a second point. If you comment, don't downvote at all. Your words make your opinions known and downvoting feels both disrespectful to your conversational partner and arrogant. Like voting for yourself for prom king.

And I think hiding others' scores is a very important part HN's culture - making up/down votes is direct communication with author, not "virtue signalling". And authors themselves have no way to compare them with others. (Unlike Reddit, where large negative score often invite even more downvotes from "regulars")