The article's approach seems super ad-hoc, leaving you to have to think hard, do all the work, and make all the mistakes.
If you were to go down the other path, you might try dividing and conquering the problem. An arbitrary Pair<A,B> is trivially constructed from an arbitrary A and an arbitrary B. So if you can generate a string, and a number, you could generate a User full of number and string fields. If your generate function accepts a number describing how complex a string to make, then you can also choose how complicated to make your User. That's all shrinking needs to be. Repeatedly trying smaller Ns while the problem still happens (the problem being one of your unit tests - not an additional "interestingness" test you need to write.)
You'll probably way more likely to hit boundary cases by using the structure of the input and making interesting variations that way, rather than hoping you can permute the right bytes from the CLI.
It's a tremendously economical way to test compilers. For a modest and finite investment in testing infrastructure I get an unlimited number of tests. Over the years I've run many billions of test inputs on various Common Lisp implementations, although I'm mostly focusing on sbcl these days. When a bug is found the input quickly reduces to a something small that usually immediately tells the developers where the problem is (usually but not always something introduced recently.)
I also have a testing harness that cobbles together usually erroneous Lisp code and sees if the compiler blows up (the sbcl compiler as designed must never throw an error condition even on erroneous input.) This exploits a corpus of public Common Lisp code, combining and mutating the code in various ways.
I'm not sure if that's an article failure (that I didn't want to read a whole ton of text and C code details), or a success (as it got me interested in the topic). I guess both?
It's answered pretty early on:
>> Test-case reducers try to reduce the length of an input
If that still doesn't answer the question, try this extension:
>> Test-case reducers try to reduce the length of an [error causing or interesting] input
Also, do test case reducers work on integers or other numbers? What about reducing some other complexity? Is this for developing unit tests or just debugging?
Imagine being handed a sheet of 10 equations and being told "1 of these equations is wrong." Now imagine that someone came in and erased 8 of the correct equations - they just saved you a bunch of time.
I'd love to get some feedback if anyone's interested.
I was also wondering: is there a UI while reduction happens? I've found Shrink Ray's UI improvements in the last year to be much more useful than I first expected: not just because it gives me something to look at, but because it really helps me understand if reduction is on the right path or not. [Some of the new Shrink Ray extras like being able to rewind reduction to a past point and to skip passes are also really useful too.]
https://dlang.org/blog/2020/04/13/dustmite-the-general-purpo...
Created by Vladimir Panteleev
I really appreciated the humour in this article. I've always wanted to use creduce, but it looks like Shrink Ray is easier to set up and get running (pip install, set up a harness, run)
Similarly, narrowing test cases to the smallest case that reproduces a particular behaviour so you're only actually testing a very targeted thing will make the test suite faster and also make it easier to fix tests which break because they exercise a very narrow path.
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaẞ
Boy I sure do wonder what character could possibly be causing issues there. whereas without shrinking it might instead spit out something like
ЁЂЃЄЅІЇЈЉЊЋЌЍЎЏАБВГДẞЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмноп
Which makes finding the offending character a lot harder.
There's no C in there? It seems to be Python and shell scripts.
sigbottle•5d ago
On one project, through a variety of circumstances, dead code elimination was straight up not working, but we wanted to show the theoretical improvement of some approach - but we couldn't figure out why at the moment (we did spend a whole week chasing down the root cause after - maybe worth in hindsight...).
We were doing it by hand at one point, but someone suggested using CReduce for shrinking the code. Definitely was an interesting test-iterate loop...