https://en.wikipedia.org/wiki/Dyck_language

So to generalise your point you need a counter for each transition to a different type of opener.

So (([])) needs only 2 counters, not 3.

You could constrain it further if certain types of openers are only valid in certain cases so you could exclude certain types of transitions.

EDIT:

([)] could indeed be handled by just additionally tracking the current open type. (([]]) is a better example, as it shows that to handle deeper nesting you need additional pieces of data that will grow at some rate (at most by the number of opens, possibly lower depending on which types can validly appear within which types)

  bool balanced(const string& text, const string& open, const string& close) {
    size_t length = text.size(), brackets = open.size();
    assert(close.size() == brackets);
    stack<char> buffer;
    for (size_t index = 0; index < length; ++index) {
      char ch = text[index];
      for (size_t slot = 0; slot < brackets; ++slot) {
        if (ch == open[slot])
          buffer.push(ch);
        else if (ch == close[slot]) {
          if (buffer.empty() || buffer.top() != open[slot])
            return false;
          buffer.pop();
        }
      }
    }
    return buffer.empty();
  }

EDIT: It gets complicated if you need to count multiple different types of openers. In that case I think you need the stack, at least unless there are constraints on which openers can occur within others - you at the very least need to know which closer you're looking for right now, but if you can't deduce what is outside, you obviously then need to keep track of it.

In practice, of course, we'll generally use a stack because it's just pointless to make life harder by not using one for this.