The term problem plays was coined by critic F.S. Boas in Shakespeare and his Predecessors (1896). The three problem plays are Troilus and Cressida, Measure for Measure and Alls Well that Ends Well.
The problem plays are characterised by their complex and ambiguous tone, which shifts violently between dark, psychological drama and more straightforward comic material.
All's Well That Ends Well and Measure for Measure have happy endings that seem awkward, artificial and perfunctory. Troilus and Cressida ends with the death of the Trojan hero and the separation of the lovers.
The problem plays are also known as dark comedies. Critics have suggested that this sequence of plays marked a psychological turning point for Shakespeare, during which he lost interest in the romantic comedies he had specialized in and turned towards the darker worlds of his tragedies. Some critics also apply this term to the plays Winter's Tale, Timon of Athens and Merchant of Venice.
The term has also been applied to other odd plays from various points in his career, as the term has always been somewhat vaguely defined it is not accepted by all critics.
I enjoy the term because it does give a fourth level to sort Shakespeare's plays into. There is the histories, the comedies, the tragedies, and then the problem plays or dark comedies.