Follow-up in #12389, where we had to fix our machinery, and where @astrofrog noted it should be possible to do this just with pytest now.