Syntia uses a Monte Carlo Tree Search based approach to program syntheses to learn program semantics from input/output examples. While synthesizing semantics from input/output examples only works for simple targets, it is extremely robust to common obfuscation schemes. As a consequence it is highly useful to deobfuscate the custom instruction sets from common VM based obfuscators such as Themida or VMProtect.
The paper describing Syntia was published at USENIX Security 2017. A recording of the presentation and the slides can be found here: