Ask Experienced Devs

In my experience the lean coffee format or something like a 3 column "keep doing" "stop doing" "start doing" style retro is just not enough to understand the interactions between items or to come up with a small number of high quality experiments.

I fully agree with the other poster who mentioned starting to gather metrics so that you can measure experiment success, just make sure that the metrics you're gathering aren't vanity metrics (or at least that you're aware of how they can be gamed and keeping an eye out for it when designing experiments).

When I run retrospectives they generally follow a four step process. Brainstorm items to discuss, filter those items to the ones with the greatest chance of creating an impact, brainstorm ways to create an impact around those items, filter those items and form them into one or two experiments.

To put this in more concrete terms:

1. Brainstorm items to discuss

This might at first look a lot like the first step of your lean coffee format, spend no more than 5 minutes throwing up any ideas you have. Start with a timer of 2-3 minutes adding 1 minute as you near the end of the timer if there is still activity. Spend no more than about 30 seconds per item making sure the team understands what was meant by that item and doing an initial filtering round of affinity mapping or grouping related items.

2.Filter those items to the ones with the greatest chance of creating an impact

Now we spend a few minutes discussing each of the groups of items, maybe 2-5 minutes per group exploring the impact of these items, interactions between them, and maybe touching on potential solutions. Finally, we either vote or otherwise filter down to one or two groups to bring to the next step.

3. Brainstorm ways to create an impact around those items

This is another brainstorming step so we move fast again to reduce internal filters, spend no more than 5 minutes but starting with a timer of 2-3 minutes to propose experiments, these don't need to be well formed but should roughly follow the format of "if we do X, we expect Y". Similar to the previous step you can spend no more than 30 seconds per item making sure it's understood and getting related experiments.

4. Filter those items and form them into one or two experiments

Finally we discuss each of the groups, possibly combining the proposed experiments or coming up with new ones during the discussion. For each group you will likely come up with 1 proposed experiment. If at the end of this process you have only one experiment great! Get an active thumbs up from the group to try it and make sure it's visible in your team space (on your stand-up board maybe?) so that it doesn't get lost. If you have more proposed experiments do a round of voting to get down to at most two that the group will agree to try. This limiting Experiments In Progress is very similar to limiting Work In Progress, we do it to improve the chances that any one item gets delivered (or experiment implemented) in a relatively short timeframe.

For each of these steps there are all kinds of activities you could do to keep things fresh, check out Retromat and Liberating Structures for ideas.