Apparently it was noise on my switch lines. Even though I have shielded motor lines and shielded switch lines ans everything is properly grounded, filtering the switch input helped.
For the record, I put 100R in series with the switch line and 100n capacitors to ground after the 100R. This cured the noise issue. The extra protection does not hurt either. (You might want to add a note to the wiki snd maybe, if there ever is going to be v8…)