I'll preface this by saying I'm a very new GM, so I'm fairly sure I made the wrong ruling here, but as the saying goes, the best ruling is the one that keeps the game going. I was running the first session of Abomination Vaults last weekend and something came up that I'm not sure how to handle RAW.
Basically, a PC wanted to use the Lie activity in order to talk their way out of a fight. They were fighting a group of Mitflits, and had already used Recall Knowledge to know that Mitflits are easy to manipulate. The PC speaks Undercommon, and they are a Bard, but they want their character to be friendly and therefore didn't like the idea of using Demoralize or Coerce. They instead tried to use the Lie activity. They told one of the Mitflits that the party was sent by their boss, and therefore they should let them go. They rolled a critical success on the Lie. Now I needed to decide what to do with that.
This was a weird situation, because the rules don't really say what lying can accomplish. All it says is that the target believes the lie. Given that the combat had already started, and the PCs had already attacked some of the Mitflits with lethal attacks, it seemed impossible to me that this Mitflit would actually stop fighting (ie. go from hostile to unfriendly), even if he believed the lie. The lie was convincing in terms of how it was worded, but the situation in which it was spoken made it kinda unbelievable. But then again, Mitflits are supernaturally easy to manipulate and bully. Ultimately I ruled that the targeted Mitflit would become Frightened 1, as though the Bard had succeeded on a Demoralize. My reasoning being that the Mitflit believes the lie, but also believes he should continue fighting the PCs. This cognitive dissonance in his mind shocked him and distracted him from the fight at hand.
I'm curious if any GMs out there have any tips on what the best way to handle this would have been.
It sounds like you handled it fine. This feels like it sort of falls in between Coerce (Using the threat of the mitflit's boss. Your boss sent us, so you need to do X, or else the boss will be mad.) and the player attempting to Make an Impression. Generally speaking each of those take a minute to accomplish, so are hard to do in combat. So it would very much be a GM decision. Another option would to have the Mitflit flee for a round or even attempt to run back to the boss to check (which could bring reinforcements). But with all that said, you rewarded the player's idea and critical success, even if it isn't exactly what the player was hoping for. So, well done!