After getting angry at Lua for bugs reappearing after fixing them, and otherwise having issues with making it interoperable with my own language of choice (D), I decided to roll out my own VM out, with some enhanced optional features compared to regular Lua (type safety, static arrays on the stack, enhanced metatables, etc.), and also allowing the potential of other scripting languages to be ported to the VM. As I already have crafted a VM for a programmable MIDI format (M2: Docs / Implementation ; bit incomplete as of now), I thought it'll be easy, just don't optimize this time entirely around preallocation (M2 has 128 not entirely general purpose registers per pattern (musical thread) + 128 shared registers + shared arrays), add stack handling, add heap handling, add more types, etc.
Thus I begun working on PingusVM, to contribute to the problem of ever growing number of standards.
However, as time went by, I had to realize (just like every time I've added another engine feature) that it's way more complicated, especially as I have realized mid-development that I had a lot of oversight on design. Not only that, I have a lot of other projects, such as the game engine I've originally intended the VM for. My main issue is, potential candidates either lack integer support, or are very bloated (I don't need a second XML DOM parser, etc). Lua kind of worked, but after I fixed an issue (which was hard as every tutorial implied you just wanted to load a file directly instead of having the ability of loading the text directly) a very similar one reappeared, while following every tutorial possible. Others would lead me to introduce some C-style build system, as they would need "hard linking" to my engine, and that "hard linking" is why I had to halt development of my image library, as I would have needed to introduce a build system into my project for the LZW codec (required for GIF files).
I don't have enough experience yet with WASM; ultimately, it's bytecode, right? So the same rules apply. Note I didn't make an exception for dynamically linked programs; the more components in your stack that are variable, the more instability you have.
But it sounds like you're doing a sort of plug-in system, and that involves instability by design. Using WASM seems far easier that implementing a bespoke Lua VM.