You can turn on a feature that will not auto-add invites to your calendar if sent from "unknown senders". The downside (significant IMO) is that "unknown senders" is based on Google Address Book + Other Contacts (aka people you email with). So if you're texting with someone, using a Calendly-style link, or pretty much any other non-email method of communication to schedule the meeting, it can easily be lost. It will still end up in your inbox, but you will have to accept the invite there for it to appear anywhere in Google Calendar.

Context: I'm the cofounder of Reclaim.ai.