Paper Notes: Behavioral Testing of NLP Models with CheckList
last updated 2026-09-17
CheckList is a notable paper in part because it won the best overall paper award at ACL 2020. The core idea is to use automatically constructed unit tests to evaluate model robustness, without knowledge of model internals (black-box testing, via behavioral testing).
For example: if we want to evaluate the robustness of a sentiment detector (positive/negative/neutral classifier), we can generate test cases that include negation phrases:
I can't say I recommend the food.–>negativeI didn't love the flight.–>negative
Both of these are generated from a template:
I {NEGATION PHRASE} {POSITIVE VERB} the {NOUN}.
After running a few hundred of these through our sentiment detector, the accuracy will tell us how robust we are to negation.