There's an entire category of films my husband has written off as personal poison. He calls them "bad-choice movies," and he hates the genre so much that he won't go see a film if it even looks like it fits the description. The example that defined his personal hate-genre is Very Bad Things, Peter Berg's 1998 black comedy about a bunch of yahoos who go to Vegas for a bachelor party, accidentally kill a stripper (ha ha on her, I guess?), and try to cover it up, with disastrous results that just keep creating bigger and even more insoluble problems. A bad-choice movie is defined by the horrible corners the characters paint themselves into, and how each successive poor judgment call just leads to a worse set of options. It's about escalation, and the horror that comes from eventually running out of choices.

Very Bad Things is a manic, lowbrow comedy, but bad-choice films come from all sectors of cinema. Arthouse dramas like Frozen River and Wendy and Lucy fit the bill, as do thrillers like A Simple Plan, any action movie involving schemers or scammers, and quite a few Coen brothers films. And this year's Toronto International Film Festival seems to be racking them up pretty rapidly. And no wonder: bad choices create drama, and drama creates exciting movies. (With all apologies to my husband, I don't share his particular prejudice against this species of film.) The problem comes when characters' choices are so bad that they're mystifying, when they bounce you out of the film because you're too busy wondering "Why would a human being do that?" to stay engaged with the plot.

I had that reaction to Ben Wheatley's Free Fire, a manic crime dramedy about an arms deal gone terribly, terribly wrong. Wheatley most recently directed the J.G. Ballard adaptation High-Rise, in which an entire fancy apartment complex turns on itself in an orgy of class-based murder and madness. Free Fire completely loses High-Rise's chilly calm, and reverts back to a more manic form of storytelling as seen in his early films like Kill List. It takes place almost entirely within a single filthy, run-down factory, where a group of IRA representatives (including Michael Smiley and Cillian Murphy) are out to buy a load of automatic rifles from a sleazy South African arms merchant (Sharlto Copley) and his partner (Babou Ceesay), with middleman Ord (Armie Hammer), fixer Justine (Brie Larson), and various hired help looking on. Then a preexisting squabble between two of those helpers breaks out into violence, and soon, everyone in the cast is hunkered down behind one crumbling support beam or another, wildly shooting at everyone else. Imagine the final face-off in Reservoir Dogs with five times as many people, a lot of improvised cover, and a minimal build-up before the fireworks.

The film's freewheeling action is meant to be fun, with over-the-top machismo causing most of the early problems. But Free Fire's fatal flaw is that none of the action makes much sense. There are too many players in the game, not enough of them have clear personalities, and above all, there's just no good reason for them to start shooting and keep shooting once it becomes clear that everyone's armed, angry, and dangerous. Wheatley gets some comic mileage out of the confusion — "I can't remember what side I'm on!" one pathetic mook yells in the middle of everything — but any remotely reasonable person will want to shout "What are you idiots thinking?" at least a dozen times during the film.

The cast is smaller and the action is clearer in Dog Eat Dog, the latest from writer-director Paul Schrader. But the decision-making is just as distractingly terrible and difficult to fathom. Nicolas Cage stars as Troy, a small-time criminal trying to get back on his feet with a kidnapping scheme. Along for the ride are two buddies, hot-tempered, erratic Mad Dog (Willem Dafoe) and their calmer muscle Diesel (Christopher Matthew Cook, whose weary tolerance of the other two is the film's highlight). The film opens with Mad Dog murdering a former lover and her child — a sequence played for manic comedy, largely because both his victims are overweight — and the story just gets more ghoulish from there.

There are plenty of crime movies about comically incompetent bunglers, but the gag starts to wear thin if there's a body count, involved. Troy talks and talks and talks to the audience, explaining his motives and methods in a grating wash of chatter. But he never becomes the soulful character he's intended to be, because his choices make everyone around him suffer, and he's indifferent to that suffering. (He also goes far beyond unreliable-narrator territory, into the point of seeming delusional: his lengthy insistence that Diesel is a borderline genius, for instance, is contradicted by nearly every action the character takes.)

Dog Eat Dog is tonally bizarre — at one point, a badly botched mission is followed by a celebratory party that feels like it belongs in a different movie — and its protagonists are so awful that the film eventually feels like it's celebrating the violent, sadistic cops who brutalize one of the crew. The whole thing wraps up with a fantasy sequence in which Cage gets to air out his best Humphrey Bogart impression. But even in Troy's fevered dreams, innocent people die for no reason, because people do stupid and poorly justified things. If the film held together better as a story, the bad choices might at least seem thematic. Instead, the theme seems to be "How can we alienate the audience as much as possible?"

Paul Verhoeven is a career provocateur who's never cared about alienating his audience; he sets out to titillate and unsettle them at the same time, usually by exploring the worst extremes of human behavior, either for comic or horrific effect. To long-time Verhoeven watchers, the fact that Elle opens with protagonist Michèle Leblanc (Isabelle Huppert) being violently raped by a masked home-invader might barely even register on the outrage scale. It's what she does after the rape that digs deep under the skin. After a childhood scarred by publicity and police, due to an infamous father — Verhoeven saves the precise details of his crime for late in the film — Michèle has no interest in dealing with the law. So she calmly sweeps up everything the invader smashed, washes off the blood, and continues her orderly, well-appointed life. It's a surprising decision, but it's well-justified within the context of the film.

But after that, Elle wanders into increasingly murky territory, as Michèle investigates her rapist and pursues a physical and emotional relationship with him. Fetishism and masochism work directly into her choices, and so does the unsatisfying nature of her other relationships, with the callously entitled husband of her best friend and business partner, and with her ex-husband, who's moved on to date a beautiful yoga instructor. Michèle is by turns jealous and possessive, dismissive and cold, obsessed with punishing her rapist and coolly intrigued by him. And much of Elle is about the attempt to connect all these pieces into a single cogent portrait of a deeply complicated and fascinating anti-heroine.

Verhoeven and screenwriter Philippe Djian based Elle on a novel, which explains many of its biggest problems. These characters seem intensely internal, but cinema can only explore them through their actions, which are sometimes incomprehensible. Michèle's behavior is often the sexual-drama equivalent of horror-movie characters splitting up to wander into dark tool-sheds or cobwebbed basements alone and unarmed, in search of stray slashers. But in spite of Elle's queasier exploitation aspects — an expected Verhoeven staple — and its why-would-anyone-do-that questions, it's a mesmerizing movie. The initial procedural mystery becomes profoundly uncomfortable, as Michèle sizes up every man in her life for rapist potential, and the audience realizes how many suspects there are. But it's also compelling, as she navigates her already-knotty relationships with the resentful male staff at her video game company, her possessive lovers and mysterious new friends, and various other people in her life. At one point, Elle begins to suggest that Michèle's real choices are to withdraw in fear from all men, or bull her way confidently through, taking charge of what she wants sexually, emotionally, and domestically. Naturally, she heads in the direction of drama. Whether there's logic in that or not, it's what a good movie demands.