The preconditions are that you have domain experts or business people interested and willing to engage in writing or reviewing these tests. Unless you have this and it's something that those people are going to sustain when the going gets tough you're just making writing tests harder for no real benefit.
- write specs in a cucumber fashion
- write parsers to go from cucumber to whatever objects / data you expect
- finally write your tests
In practice when people try to use it for anything semi-complex they inevitably end up either writing very vague tests or repetitive tests, neither of which are of much interest to stakeholders.
I've used docs generated from hitchstory for having conversations with stakeholders. I've also shared semi-human readable unit and e2e tests - especially when it's, say, JSON snippets in an API with technical stakeholders.
Unfortunately, except for a few domains where specs can be expressed very concisely, cucumber isn't suitable for that.
Going from "Step 1: Do X. Expected result: Y. Step 2: Do Z. Expected result: Q" to a big long "sentence" describing these steps joined with "and" (without the expected results being linked to each step) is strictly worse in just about every way. The actual automated tests are still being written the same way they always were, the descriptions of them are just WAY harder to understand.
It's a two way street. I'd highly recommend never implementing it unless product is the one driving it, and actually makes it a priority.
To me it’s just another framework to learn - a failed abstraction. It’s always introduced by idealistic devs who regularly jerks off to conference talks.
Even the US-born engineers cant always agree on how to communicate product ideas!
Just like all the other no-code inventions out there, they fail to reckon with the fact that essential complexity isn’t a problem with programming language syntax.
If readable tests are the goal, I think the time is much better spent cleaning up regular tests and writing well-named helper methods. Especially in Ruby you can get pretty close to tests that can be deciphered on a high level by non-programmers. That said, I've never had a PM or other non-programmer actually want to do that.
Capybara has this problem as well as do other web acceptance tests. Maybe not every time a new matcher but frequent enough. Mostly because of the high variability of HTML paths. You have to design your HTML in a way (use lots of [title] tags and standard HTML) if you want to avoid it.
Given /^a nice new bike$/ do
expect(bike).to be_shiny
end
have never actually tried it for that reason - it just seems worse in every way
If your job is polishing a turd, make the turd shine like a cucumber. I have seen countless hours wasted on making Cuke/Gherkin "pretty" instead of accomplishing anything for the business.
Engineers' responsibility begins at the "code behind" the gherkin layer. Someone needs to enforce this.
Sadly, IMO because gherkin looks like structured language (i.e "code"), and because that's what engineers do, they end up doing it. Either they tricked themselves into thinking they should, or the product team tacitly assumed it was a code thing.
If you're an engineer and you're writing gherkin, you should consider this a "code smell" and consider ditching it. Go for a spec tool one layer closer to your main domain. i.e. just write normal unit & integration tests.
I'm not saying "don't do BDD" or "don't do acceptance tests". You should do that! I'm saying "gherkin should allow product owners/managers to take ownership of reporting on how well the app meets their expectations". But most typically, they don't get it, or they don't want to do it.
I recently worked somewhere where we had dedicated devs-in-test. They wrote most of the e2e tests for the application. They wrote it in Gherkin, which meant that they _also_ wrote the JS/Selenium tests behind them. So they were basically writing two sets of scripts for each test. At some point they explored using a 3rd party cross-device testing platform (lambda test, maybe?) who had their own in-house test script that looked like Gherkin, so it needed a JS/TS code-behind to actually run the tests. So they did it all again!! Such a waste.
Specifically I try to make a nice string representation of state, and then tests match off that state.
So for instance I don't want to say expect(x.length).toBe(0) (one syntax, Cucumber goes further in that syntactic direction). Because if the length is 1 then 1!=0 is a very opaque failure. Or if you do expect(x[0].attr).toBe("y") then you've tested one thing but you have to have another test for list length, other attributes, etc. Often you'll leave the others out, expecting them to be ok, but sometimes they aren't...
Investing in a good, stable, matchable string representation will give you a great tool for any number of tests, and it's not just a great tool to verify but also to debug your tests.
Cucumber was all the rage in 2008, IIRC..
Cucumber tries to solve the problem of turning customer requirements into 'real code'. In exchange for that worthwhile benefit, it asks you to implement the most terrible, reg-ex based spaghetti code imaginable.
The problem is that it doesn't solve the original problem AT ALL. And then you are left with terrible reg-ex driven spaghetti code. Like the Jamie Zawinski saying, "now you have two problems".
The lesson here is that software development processes have to pass the 'human nature' test.
The software industry has largely abandoned waterfall development because it just doesn't work well in practice. It doesn't work because people don't know perfectly what they want before they build it. Agile processes usually are much more efficient because they are more closely aligned to how humans solve problems in the real world.
Cucumber suffers from the same issue of being disconnected with reality. In theory, you can find a customer who can give you perfectly written use cases and you can cut-and-paste those into your cukes. In practice, that never, ever works. So let's all stop wasting our time pretending it was a good idea now that it has been shown to not work.
It's quite funny that now with AI-based no/low code the same thing is attempted again. Even if the regexes might disappear, it's even more text, with even less structure (assuming anyone checked those prompts into git in the first place)
I'd never ask them to write them, but I will often write a spec/test based upon their two sentence jira and then screenshare and walk them through my interpretation to get feedback early (i.e. before ive wasted time building the wrong thing).
Cucumber/gherkin is awful at this of course, and the regex thing was a terrible idea but it's not the only tool.
The idea that tests should be split into a specification layer and execution layer is a good one that should have taken off by now.
There is a fundamental reason it hasn't:
An actual specification layer isn't any simpler than the execution layer. That's a programmer's fallacy.
What has taken off, and is part of virtually every software project, is a loose, natural language specification, which hints at "more or less" what the stakeholders are imagining. The idea that you can close the gap between this and a complete specification in a way that all stakeholders can still digest is the fantasy of cucumber. Or any other tool that attempts it.
You can't solve the problem in that way. Because, from a high-level stakeholder's perspective, the whole point of the people below them (whether programmers or UX designers or anyone else) is to fill in those details in a way that matches their own hazy expectations, at least well enough.
The point of separation of concerns isnt to keep the simple layer separate from the complex one. It's to simplify the whole thing by only addressing one concern per layer.
Unit tests are often a pain in the ass to read because they are a mess of implementation and specification details. No separation.
>You can't solve the problem in that way. Because, from a high-level stakeholder's perspective, the whole point of the people below them (whether programmers or UX designers or anyone else) is to fill in those details in a way that matches their own hazy expectations
If you crystallize the hazy expectations on their behalf and skip feeding it back to them then you will often find out that "thats not what i meant" after the code is complete.
Those mistakes are expensive and avoidable.
> The point of separation of concerns isnt to keep the simple layer separate from the complex one. It's to simplify the whole thing by only addressing one concern per layer.
With some caveats, I think this is just a fiction. There can be some value in having high-level tests and low-level tests, but not because it removes complexity. It can help with focus and priority. Which is a problem that can be solved in many ways.
> If you crystallize the hazy expectations on their behalf and skip feeding it back to them then you will often find out that "thats not what i meant" after the code is complete.
But this is exactly what they want you to do, and do well enough that "that's not what i meant" is not a big problem. They certainly don't want to read cucumber tests as a way of ensuring you're on the same page. They will tolerate rough, incremental prototypes, per the old agile advice, and this is probably still the best way of solving the communication gap problem.
Only now instead of this developing brittle generated tests, it will instead be used by the llm as guidance to generate the actual code and tests.
Before people jump down my throat, I know we are nowhere near that today and I promise I'm not pitching this to my leadership because they would gobble it up too fast.
But for us engineers, I think there is an interesting space for thinking of llms as akin to garbage collection, a feature that allows us to abstract to a slightly higher level of thought. Yes we still need to know how to check under the hood, but this is looking like the right level of precision-flexibility ratio that llms thrive in
The idea behind it was that Behavior-Driven Development might be a great idea, but Gherkin was a pain to work with. LLMs bridge that gap now:
Business people don't really care about this stuff. Over the years I realize more and more that we engineers are naive in thinking that business side is concerned with edge cases and race conditions.
Just to give an analogy software people might get better, if you come to a lawyer because, say, you want to buy a house, you are not going to sit down with them and say "given I want to buy a house, when the seller hides water damage costing over $2000, I get to walk away from the deal". You just hope the lawyer is good and will protect you from various edge cases. You have a lot more to deal with than just closing paperwork. You probably are thinking about renovating, moving, getting inspection, etc.
Businesses are just like that with engineering. They don't want to sit down and meticulously analyze every possible edge case. They have other things to do. Especially when stakes are not that high. Most of these errors can probably be resolved with a phone call and a database edit.
I think this is probably for the best. A good engineer will make sure you're standing on a solid ground, and ask the right questions at the right time. They wouldn't need this amount of hand holding. Leave business time to focus on making deals, connections, organizing the whole operation to move forward, etc. Let them give you vague requirements, and crystallize them yourself. It's way better than a micro-managing business that thinks they know exactly how everything should be.
P.S. Also, I'm not sure why everyone is so hung up on regex = bad, it's not like switching to an AST-based language would've made anything better here. Regex is fine imo, just the entire concept isn't.
It's better to (1) write the test, (2) then later once all that's done, extract documentation in a human-friendly format.
It creates unnecessary work for something that's never to be seen by real stakeholders.
There is a new trend these days using LLM that is similar to Cucumber: Spec Driven Development using AI. You'd be left disappointed again.
Once you write it, you can use the sentence for building other tests if you design it for being flexible and variable, i.e. design well enough your code.
Given I'm connected as ___ with password ___.
When I click on the element __.
Then the element ___ is present.
Then the element ___ is present.
[...]
For critical use case, it was enough. For more that's a bad idea: it take way too much maintenance cost.
On a technical level, cucumber is also at odds with the need for a test suite to be easily maintainable. What I mean is that each test (especially e2e tests) will want to do some setup/initialization. This is usually expressed as step definitions.
Over time, an undisciplined team may write several slightly different but effectively identical step definitions. They may also combine multiple steps into bigger steps because usually a spec writer doesn't want to exhaustively define every piece of setup, they just want to write "Given all 200 pieces of input data and mocks are magically in the right place..."
I was able to wrangle the specs into composability using Rule and Background blocks, but at that point we were just programming tests with a shitty layer over the actual code.
headcanon•9h ago
baal80spam•9h ago
threetonesun•8h ago