Deduplicate instruction prefixes when disassembling

Previously, the opcode lookup table would encode every possible permutation of
allowable prefixes for each instruction as a separate path. This is expensive
in both space and time, as observed in #40. The new approach taken in this
patch, as described in `Note [x86_64 disassembly]` in `Flexdis86.Disassembler`,
is to only encode the VEX prefixe and opcode bytes in the lookup table, leaving
out all other forms of prefix bytes entirely. Instead, disassembly will start
by eagerly parsing as many prefix bytes as possible, proceeding to parse opcode
bytes after the first non-prefix byte is encountered. After identifying the
possible instructions from the opcode, we will then narrow down exactly which
instruction it is by validating them against the set of parsed prefixes.

As noted in `Note [x86_64 disassembly]`, we had to add some special cases for
`nop`-like instructions—namely, `endbr32`, endbr64`, `pause`, and `xchg`—to
avoid some prefix byte–related ambiguity. The new handling for `xchg` is
more accurate than it was before, so this patch fixes #42 as a side effect.
This patch also addresses part (1) of #40 in that it should reduce the amount
of memory usage that the lookup table takes, although there is potentially more
work to be done (see part (2) of #40).

data/optable.xml

@@ -5209,6 +5209,22 @@
     <instruction>
         <mnemonic>nop</mnemonic>
         <def>
+            <!--
+              Note that this opcode is really three instructions in one:
+
+              * nop (opcode 0x90)
+              * pause (opcode 0xf3 0x90)
+              * xchg ax,ax (opcode 0x66 0x90)
+
+              Unfortunately, the 0xf3 byte is also used as a REP prefix, and
+              the 0x66 byte is used as an OSO prefix, which poses some
+              challenges for the way that we parse instruction prefixes.
+              (See Note [x86_64 disassembly] in Flexdis86.Disassembler.) As
+              a result, we instead parse all of these instructions with the
+              opcode 0x90 and check the prefixes after the fact to disambiguate
+              among the three.
+            -->
+            <pfx>rep oso</pfx>
             <opc>90</opc>
         </def>
        <def>
@@ -5246,43 +5262,6 @@
             <opc>0f 1f</opc>
             <opr>M</opr>
         </def>
-<!--  TODO: this is a dumb hack to deal with multi-byte nops
-            get rid of it when general prefix repetition is handled -->
-        <def>
-            <pfx>rexr rexx rexb</pfx>
-            <opc>66 66 2e 0f 1f</opc>
-            <opr>M</opr>
-        </def>
-        <def>
-            <pfx>rexr rexx rexb</pfx>
-            <opc>66 66 66 2e 0f 1f</opc>
-            <opr>M</opr>
-        </def>
-        <def>
-            <pfx>rexr rexx rexb</pfx>
-            <opc>66 66 66 66 2e 0f 1f</opc>
-            <opr>M</opr>
-        </def>
-        <def>
-            <pfx>rexr rexx rexb</pfx>
-            <opc>66 66 66 66 66 2e 0f 1f</opc>
-            <opr>M</opr>
-        </def>
-        <def>
-            <pfx>rexr rexx rexb</pfx>
-            <opc>66 66 66 66 66 66 2e 0f 1f</opc>
-            <opr>M</opr>
-        </def>
-        <def>
-            <pfx>rexr rexx rexb</pfx>
-            <opc>66 66 66 66 66 66 66 2e 0f 1f</opc>
-            <opr>M</opr>
-        </def>
-        <def>
-            <pfx>rexr rexx rexb</pfx>
-            <opc>66 66 66 66 66 66 66 66 2e 0f 1f</opc>
-            <opr>M</opr>
-        </def>
     </instruction>
 
     <instruction>
@@ -5637,13 +5616,6 @@
         </def>
     </instruction>
 
-    <instruction>
-        <mnemonic>pause</mnemonic>
-        <def>
-            <opc>f3 90</opc>
-        </def>
-    </instruction>
-
     <instruction>
         <mnemonic>pavgb</mnemonic>
         <def>
@@ -8386,11 +8358,11 @@
             <opc>87</opc>
             <opr>Ev Gv</opr>
         </def>
-        <def>
-            <pfx>oso</pfx>
-            <opc>66 90</opc>
-            <opr>R0v rAX</opr>
-        </def>
+        <!--
+          The opcode-0x90 version of xchg is missing here because it is included
+          as a part of the nop instruction, which also uses opcode 0x90. See
+          the comments surroudning nop for more details.
+        -->
         <def>
             <pfx>oso rexw rexb</pfx>
             <opc>91</opc>
@@ -9944,18 +9916,26 @@
       </def>
     </instruction>
 
-    <!-- These are CFI instructions coming to Intel processors -->
+    <!--
+      These are CFI instructions coming to Intel processors
+
+      Note that the /actual/ instruction definitions have F3 at the start
+      of the opcode. However, this byte is also used for REP prefixes, so
+      to avoid ambiguity, we remove the F3 from the opcode and instead make
+      it a required prefix. See Note [x86_64 disassembly] in
+      Flexdis86.Disassembler for more details.
+    -->
     <instruction>
       <mnemonic>endbr32</mnemonic>
       <def>
-        <opc>F3 0F 1E FB</opc>
+        <opc>0F 1E FB /reqpfx=F3</opc>
       </def>
     </instruction>
 
     <instruction>
       <mnemonic>endbr64</mnemonic>
       <def>
-        <opc>F3 0F 1E FA</opc>
+        <opc>0F 1E FA /reqpfx=F3</opc>
       </def>
     </instruction>
 

flexdis86.cabal

@@ -12,7 +12,7 @@ extra-source-files: data/optable.xml
 library
   default-language: Haskell2010
   build-depends:
-    base >= 4,
+    base >= 4.6,
     ansi-wl-pprint,
     binary,
     binary-symbols,