Why I dont like Hypothesis


Hypothesis & property based testing are some of the new darlings of modern development. Many people love them. I do not. In fact I despise Hypothesis. Or perhaps I should say, I despise the way people use Hypothesis, treating it like it is a unittest framework when it isn’t.

Let me explain why. Here is a pattern I have followed many times:

Now what?

Sometimes Hypothesis has taken too long to build strategies and I need to add another commit simplifying the hypothesis test. Sometimes a test has failed. Is it a real failure? I don’t know, I have never worked on that area of the code. The correct thing to do is to raise a ticket, but we all know that usually people just click ‘retry’ on the CI server.

But the point is I shouldn’t have to do any of that. Hypothesis shouldn’t interrupt my workflow. Hypothesis is bad because it breaks the rules of unittesting.

Rules of Testing:

Let us examine the unit test FIRST rules that are defined in the book Clean Code written by Uncle Bob Martin.

And no just because Hypothesis allows you to add a decorator to re-run a failed test - that is not the same thing as being repeatable. A repeatable test needs to always do the same thing in different environments.

I once attended a talk by the creator of hypothesis DRMacIver, he is definitely a smart guy and I directly asked him why we should use hypothesis when it seems to break so many rules of unit testing. His answer was that different people have different ideas of what constitutes a unit test. It is an interesting answer, but I think it is wrong.

But I still think Hypothesis can be used for good

Despite all this I feel there is a place somewhere for Hypothesis. A separate build that is allowed to fail following the ‘push on yellow’ philosophy of google. Hypothesis is great at finding edge cases.

But please for the sake of your sanity and mine don’t use Hypothesis in unit tests. Ever.