While I can’t really comment on how good this specific implementation is simple diff based testing frameworks like this dont get enough press. My first job involved working on an internal programming language. The test suite was just a ton of statements that were executed top down and their output was compared against a single file. Yes that’s a bit absurd but it worked remarkably well. If they’d bothered to add a bit more structure around it I think it would have been perfect.
sestep•1h ago
Agreed, this is the default testing methodology I reach for. Other methodologies are useful in some situations, but those are the minority.
tom_•16m ago
It's not absurd at all (in my view). A test checks that some obtained result matches the expected result - and if that obtained result is something that got printed out and redirected to a file, and that expected result is something that was produced the same way from a known good run (that was determined to be good by somebody looking at it with their eyes), and the match is performed by comparing the two output files... then there you go.
This is how basically all of the useful tests I've written have ended up working. (Including, yes, tests for an internal programming language.) The language is irrelevant, and the target system is irrelevant. All you need to be able to do is run something and capture its output somehow.
(You're not wrong to note that the first draft basic approach can still be improved. I've had a lot of mileage from adding stuff: producing additional useful output files (image diffs in particular are very helpful), copying input and output files around so they're conveniently accessible when sizing up failures, poking at test runner setup so it scales will with core count, more of the same so that it's easy to re-run a specific problem test in the debugger - and so on. But the basic principle is always the same: does actual output match expected output, yes (success)/no (fail).)
mmastrac•52m ago
I wrote https://github.com/mmastrac/clitest because I needed a more complex testing harness for CLI tests that does something similar. It's not exactly the same, but it's definitely in the same universe.
One-file-per testcase like `tc` does works, but it tends to fall apart a bit at large scale in my experience.
roxolotl•1h ago
sestep•1h ago
tom_•16m ago
This is how basically all of the useful tests I've written have ended up working. (Including, yes, tests for an internal programming language.) The language is irrelevant, and the target system is irrelevant. All you need to be able to do is run something and capture its output somehow.
(You're not wrong to note that the first draft basic approach can still be improved. I've had a lot of mileage from adding stuff: producing additional useful output files (image diffs in particular are very helpful), copying input and output files around so they're conveniently accessible when sizing up failures, poking at test runner setup so it scales will with core count, more of the same so that it's easy to re-run a specific problem test in the debugger - and so on. But the basic principle is always the same: does actual output match expected output, yes (success)/no (fail).)