Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poorly formatted VerificationError when cryptol("...") mentions an unbound variable #1128

Closed
RyanGlScott opened this issue Mar 1, 2021 · 2 comments
Labels
subsystem: saw-remote-api Issues related to the SAW server and its RPC bindings

Comments

@RyanGlScott
Copy link
Contributor

I recently made a mistake when using the cryptol() function in the following setup:

// f.c
void f (int *p) {
    p[2] = p[0];
    p[3] = p[1];
}
# bug.py
import os
from pathlib import Path
from saw import *
from saw.llvm import Contract, cryptol, void
import saw.llvm_types as ty
import saw.connection as conn

dir_path = os.path.dirname(os.path.realpath(__file__))

server = os.environ.get('SAW_SERVER')
if not server:
    server = "cabal new-exec --verbose=0 saw-remote-api socket"
print("Running: " + server)
connect(server)
view(LogResults())

path = [dir_path, "f.bc"]
bcname = os.path.join(*path)
mod = llvm_load_module(bcname)

class FPointsToContract(Contract):
    def specification(self) -> None:
        x = self.fresh_var(ty.array(2, ty.i32), "x")
        p = self.alloc(ty.array(4, ty.i32))
        self.points_to(p, x, check_target_type = None)

        self.execute_func(p)

        self.points_to(p, cryptol("x # x"), check_target_type = None)
        # self.points_to(p, cryptol("{x} # {x}".format(x=x.name())), check_target_type = None)
        self.returns(void)

result = llvm_verify(mod, "f", FPointsToContract())

The commented-out # self.points_to(p, cryptol("{x} # {x}".format(x=x.name())), check_target_type = None) line is what I should have written, but I accidentally used the line above it instead. Unfortunately, the error message that galois-py-toolkit gives here doesn't make the mistake as obvious as it could be. To reproduce, run the following commands:

$ clang -g -c -emit-llvm -o f.bc f.c
$ cabal exec -- python bug.py
Running: cabal new-exec --verbose=0 saw-remote-api socket
⚠️  Failed to verify: lemma_FPointsToContract (defined at bug.py:34):
error: Cryptol error: RenamerErrors (FromModule (ModName "<interactive>")) [UnboundExpr (Located {srcRange = Range {from = Position {line = 1, col = 1}, to = Position {line = 1, col = 2}, source = ""}, thing = UnQual (Ident False "x")}) <NameDisp>,UnboundExpr (Located {srcRange = Range {from = Position {line = 1, col = 5}, to = Position {line = 1, col = 6}, source = ""}, thing = UnQual (Ident False "x")}) <NameDisp>]
CallStack (from HasCallStack):
  error, called at src/SAWServer/LLVMCrucibleSetup.hs:175:24 in saw-remote-api-0.1.0.0-inplace:SAWServer.LLVMCrucibleSetup
Traceback (most recent call last):
  File "bug.py", line 34, in <module>
    result = llvm_verify(mod, "f", FPointsToContract())
  File "/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/saw/__init__.py", line 396, in llvm_verify
    raise result.exception from None
  File "/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/saw/__init__.py", line 361, in llvm_verify
    conn.llvm_assume(module.server_name,
  File "/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/virtenv/lib/python3.8/site-packages/argo_client/interaction.py", line 163, in result
    return self.process_result(self._result_and_state_and_out_err()[0])
  File "/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/virtenv/lib/python3.8/site-packages/argo_client/interaction.py", line 148, in _result_and_state_and_out_err
    raise self.process_error(exception)
saw.exceptions.VerificationError: error: Cryptol error: RenamerErrors (FromModule (ModName "<interactive>")) [UnboundExpr (Located {srcRange = Range {from = Position {line = 1, col = 1}, to = Position {line = 1, col = 2}, source = ""}, thing = UnQual (Ident False "x")}) <NameDisp>,UnboundExpr (Located {srcRange = Range {from = Position {line = 1, col = 5}, to = Position {line = 1, col = 6}, source = ""}, thing = UnQual (Ident False "x")}) <NameDisp>]
CallStack (from HasCallStack):
  error, called at src/SAWServer/LLVMCrucibleSetup.hs:175:24 in saw-remote-api-0.1.0.0-inplace:SAWServer.LLVMCrucibleSetup
🛑  The goal failed to verify.

Notice that a raw VerificationError is dumped to the screen unformatted. Compare this to the error that you'd experience in a similar SAWScript program:

bc <- llvm_load_module "f.bc";

let i32 = llvm_int 32;

let f_points_to_contract = do {
      x <- llvm_fresh_var "x" (llvm_array 2 i32);
      p <- llvm_alloc (llvm_array 4 i32);
      llvm_points_to_untyped p (llvm_term x);
      llvm_execute_func [p];
      llvm_points_to_untyped p (llvm_term {{ z # z }});
};

llvm_verify bc "f" [] false f_points_to_contract abc;
$ cabal run exe:saw -- bug.saw
Up to date


[16:02:46.757] Loading file "/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/bug.saw"
[16:02:46.773] Stack trace:
"llvm_verify" (/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/bug.saw:13:1-13:12):
"f_points_to_contract" (/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/bug.saw:13:29-13:49):
Cryptol error:
[error] at /home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/bug.saw:10:46--10:47
    Value not in scope: z
[error] at /home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/bug.saw:10:50--10:51
    Value not in scope: z
@RyanGlScott
Copy link
Contributor Author

Some further digging reveals that this is at least partially saw-remote-api's fault, since this is where the error: Cryptol error: RenamerErrors portion of the error message comes from:

    getSetupVal (_, cenv) (CryptolExpr expr) = LLVMCrucibleSetupM $
      do res <- liftIO $ getTypedTermOfCExp fileReader (biSharedContext bic) cenv expr
         -- TODO: add warnings (snd res)
         case fst res of
           Right (t, _) -> return (CMS.anySetupTerm t)
           Left err -> error $ "Cryptol error: " ++ show err -- TODO: report properly

@RyanGlScott
Copy link
Contributor Author

RyanGlScott commented Mar 3, 2021

After some experimentation, the following saw-remote-api hack makes the error look slightly prettier:

diff --git a/saw-remote-api/src/SAWServer/CryptolExpression.hs b/saw-remote-api/src/SAWServer/CryptolExpression.hs
index 5af4d95c..b52b8e5f 100644
--- a/saw-remote-api/src/SAWServer/CryptolExpression.hs
+++ b/saw-remote-api/src/SAWServer/CryptolExpression.hs
@@ -6,8 +6,10 @@ module SAWServer.CryptolExpression
  ( getCryptolExpr
  , getTypedTerm
  , getTypedTermOfCExp
+ , CryptolModuleException(..)
  ) where
 
+import Control.Exception (Exception)
 import Control.Lens ( view )
 import Control.Monad.IO.Class ( MonadIO(liftIO) )
 import qualified Data.ByteString as B
@@ -15,7 +17,7 @@ import qualified Data.Map as Map
 import Data.Maybe (fromMaybe)
 
 import Cryptol.Eval (EvalOpts(..))
-import Cryptol.ModuleSystem (ModuleRes, ModuleInput(..))
+import Cryptol.ModuleSystem (ModuleError, ModuleInput(..), ModuleRes, ModuleWarning)
 import Cryptol.ModuleSystem.Base (genInferInput, getPrimMap, noPat, rename)
 import Cryptol.ModuleSystem.Env (ModuleEnv, meSolverConfig)
 import Cryptol.ModuleSystem.Interface (noIfaceParams)
@@ -108,3 +110,10 @@ runInferOutput out =
     InferFailed nm warns errs ->
       do typeCheckWarnings nm warns
          typeCheckingFailed nm errs
+
+data CryptolModuleException = CryptolModuleException
+  { cmeError    :: ModuleError
+  , cmeWarnings :: [ModuleWarning]
+  } deriving Show
+
+instance Exception CryptolModuleException
diff --git a/saw-remote-api/src/SAWServer/JVMCrucibleSetup.hs b/saw-remote-api/src/SAWServer/JVMCrucibleSetup.hs
index 16e3121c..a71241dc 100644
--- a/saw-remote-api/src/SAWServer/JVMCrucibleSetup.hs
+++ b/saw-remote-api/src/SAWServer/JVMCrucibleSetup.hs
@@ -13,6 +13,7 @@ module SAWServer.JVMCrucibleSetup
   , compileJVMContract
   ) where
 
+import Control.Exception (throw)
 import Control.Lens ( view )
 import Control.Monad.IO.Class ( MonadIO(liftIO) )
 import Control.Monad.State
@@ -67,7 +68,7 @@ import SAWServer.Data.Contract
                argumentVals, postVars, postConds, postAllocated, postPointsTos,
                returnVal) )
 import SAWServer.Data.SetupValue ()
-import SAWServer.CryptolExpression (getTypedTermOfCExp)
+import SAWServer.CryptolExpression (CryptolModuleException(..), getTypedTermOfCExp)
 import SAWServer.Exceptions ( notAtTopLevel )
 import SAWServer.OK ( OK, ok )
 import SAWServer.TopLevel ( tl )
@@ -179,24 +180,20 @@ interpretJVMSetup fileReader bic cenv0 ss = evalStateT (traverse_ go ss) (mempty
         Val x -> return x -- TODO add cases for the server values that
                           -- are not coming from the setup monad
                           -- (e.g. surrounding context)
-    getSetupVal (_, cenv) (CryptolExpr expr) = JVMSetupM $
-      do res <- liftIO $ getTypedTermOfCExp fileReader (biSharedContext bic) cenv expr
-         -- TODO: add warnings (snd res)
-         case fst res of
-           Right (t, _) -> return (MS.SetupTerm t)
-           Left err -> error $ "Cryptol error: " ++ show err -- TODO: report properly
+    getSetupVal env (CryptolExpr expr) =
+      do t <- getTypedTerm env expr
+         return (MS.SetupTerm t)
     getSetupVal _ _sv = error $ "unrecognized setup value" -- ++ show sv
 
     getTypedTerm ::
       (Map ServerName ServerSetupVal, CryptolEnv) ->
       P.Expr P.PName ->
       JVMSetupM TypedTerm
-    getTypedTerm (_, cenv) expr = JVMSetupM $
-      do res <- liftIO $ getTypedTermOfCExp fileReader (biSharedContext bic) cenv expr
-         -- TODO: add warnings (snd res)
-         case fst res of
-           Right (t, _) -> return t
-           Left err -> error $ "Cryptol error: " ++ show err -- TODO: report properly
+    getTypedTerm (_, cenv) expr = JVMSetupM $ liftIO $
+      do (res, warnings) <- getTypedTermOfCExp fileReader (biSharedContext bic) cenv expr
+         case res of
+           Right (t, _) -> return t -- TODO: Report warnings
+           Left err -> throw $ CryptolModuleException err warnings
 
     resolve env name =
        case Map.lookup name env of
diff --git a/saw-remote-api/src/SAWServer/LLVMCrucibleSetup.hs b/saw-remote-api/src/SAWServer/LLVMCrucibleSetup.hs
index 2d76b992..ced0f238 100644
--- a/saw-remote-api/src/SAWServer/LLVMCrucibleSetup.hs
+++ b/saw-remote-api/src/SAWServer/LLVMCrucibleSetup.hs
@@ -18,6 +18,7 @@ module SAWServer.LLVMCrucibleSetup
   , compileLLVMContract
   ) where
 
+import Control.Exception (throw)
 import Control.Lens ( view )
 import Control.Monad.State
     ( evalStateT,
@@ -72,7 +73,7 @@ import SAWServer.Data.Contract
     ( PointsTo(..), Allocated(..), ContractVar(..), Contract(..) )
 import SAWServer.Data.LLVMType (JSONLLVMType, llvmType)
 import SAWServer.Data.SetupValue ()
-import SAWServer.CryptolExpression (getTypedTermOfCExp)
+import SAWServer.CryptolExpression (CryptolModuleException(..), getTypedTermOfCExp)
 import SAWServer.Exceptions ( notAtTopLevel, cantLoadLLVMModule )
 import SAWServer.OK ( OK, ok )
 import SAWServer.TrackFile ( trackFile )
@@ -182,23 +183,19 @@ interpretLLVMSetup fileReader bic cenv0 ss =
         Val x -> return x -- TODO add cases for the named server values that
                           -- are not coming from the setup monad
                           -- (e.g. surrounding context)
-    getSetupVal (_, cenv) (CryptolExpr expr) = LLVMCrucibleSetupM $
-      do res <- liftIO $ getTypedTermOfCExp fileReader (biSharedContext bic) cenv expr
-         -- TODO: add warnings (snd res)
-         case fst res of
-           Right (t, _) -> return (CMS.anySetupTerm t)
-           Left err -> error $ "Cryptol error: " ++ show err -- TODO: report properly
+    getSetupVal env (CryptolExpr expr) =
+      do t <- getTypedTerm env expr
+         return (CMS.anySetupTerm t)
 
     getTypedTerm ::
       (Map ServerName ServerSetupVal, CryptolEnv) ->
       P.Expr P.PName ->
       LLVMCrucibleSetupM TypedTerm
-    getTypedTerm (_, cenv) expr = LLVMCrucibleSetupM $
-      do res <- liftIO $ getTypedTermOfCExp fileReader (biSharedContext bic) cenv expr
-         -- TODO: add warnings (snd res)
-         case fst res of
-           Right (t, _) -> return t
-           Left err -> error $ "Cryptol error: " ++ show err -- TODO: report properly
+    getTypedTerm (_, cenv) expr = LLVMCrucibleSetupM $ liftIO $
+      do (res, warnings) <- getTypedTermOfCExp fileReader (biSharedContext bic) cenv expr
+         case res of
+           Right (t, _) -> return t -- TODO: report warnings
+           Left err -> throw $ CryptolModuleException err warnings
 
     resolve env name =
        case Map.lookup name env of
diff --git a/saw-remote-api/src/SAWServer/TopLevel.hs b/saw-remote-api/src/SAWServer/TopLevel.hs
index 8121edd6..4d314abd 100644
--- a/saw-remote-api/src/SAWServer/TopLevel.hs
+++ b/saw-remote-api/src/SAWServer/TopLevel.hs
@@ -2,14 +2,17 @@
 {-# LANGUAGE ScopedTypeVariables #-}
 module SAWServer.TopLevel (tl) where
 
-import Control.Exception ( try, SomeException )
+import Control.Exception ( try, SomeException(..) )
 import Control.Lens ( view, set )
 import Control.Monad.State ( MonadIO(liftIO) )
+import Data.Typeable (cast)
 
 import SAWScript.Value ( TopLevel, runTopLevel )
 
 import qualified Argo
+import CryptolServer.Exceptions (cryptolError)
 import SAWServer ( SAWState, sawTopLevelRO, sawTopLevelRW )
+import SAWServer.CryptolExpression (CryptolModuleException(..))
 import SAWServer.Exceptions ( verificationException )
 
 tl :: TopLevel a -> Argo.Command SAWState a
@@ -19,8 +22,11 @@ tl act =
          rw = view sawTopLevelRW st
      liftIO (try (runTopLevel act ro rw)) >>=
        \case
-         Left (e :: SomeException) ->
-           Argo.raise (verificationException e)
+         Left e@(SomeException e')
+           |  Just (CryptolModuleException err warnings) <- cast e'
+           -> Argo.raise (cryptolError err warnings)
+           |  otherwise
+           -> Argo.raise (verificationException e)
          Right (res, rw') ->
            do Argo.modifyState $ set sawTopLevelRW rw'
               return res

With that patch, the error becomes:

Running: cabal new-exec --verbose=0 saw-remote-api socket
Traceback (most recent call last):
  File "bug.py", line 34, in <module>
    result = llvm_verify(mod, "f", FPointsToContract())
  File "/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/saw/__init__.py", line 381, in llvm_verify
    raise err from None
  File "/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/saw/__init__.py", line 346, in llvm_verify
    stdout = res.stdout()
  File "/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/virtenv/lib/python3.8/site-packages/argo_client/interaction.py", line 167, in stdout
    return self._result_and_state_and_out_err()[2]
  File "/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/virtenv/lib/python3.8/site-packages/argo_client/interaction.py", line 148, in _result_and_state_and_out_err
    raise self.process_error(exception)
  File "/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/saw/commands.py", line 8, in process_error
    return exceptions.make_saw_exception(ae)
  File "/home/rscott/Documents/Hacking/Haskell/sandbox/saw-script/galois-py-toolkit/saw/exceptions.py", line 42, in make_saw_exception
    raise ae
argo_client.interaction.ArgoException: [error] at :1:1--1:2 Value not in scope: x
[error] at :1:5--1:6 Value not in scope: x {'warnings': [], 'source': 'module name <interactive>', 'errors': ['[error] at :1:1--1:2 Value not in scope: x', '[error] at :1:5--1:6 Value not in scope: x']}
🔶 No goal results logged.

Perhaps there's a more elegant way to propagate the Cryptol module errors up to the top level, but I couldn't think of one.

@pnwamk pnwamk transferred this issue from GaloisInc/galois-py-toolkit Mar 12, 2021
@pnwamk pnwamk added the subsystem: saw-remote-api Issues related to the SAW server and its RPC bindings label Mar 12, 2021
@mergify mergify bot closed this as completed in 06be727 Mar 16, 2021
mergify bot added a commit that referenced this issue May 19, 2021
Adapt to cryptol PR #1128 "persist-solver2"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
subsystem: saw-remote-api Issues related to the SAW server and its RPC bindings
Projects
None yet
Development

No branches or pull requests

2 participants