Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge nano tests #69

Merged
merged 9 commits into from
Oct 19, 2023
Merged

Merge nano tests #69

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7f85236
Tests: copy tests from nanos, nanosp and nanox in a nano folder
spalmer25 Oct 18, 2023
7b4594a
Tests: remove device from expect_section_content arguments
spalmer25 Oct 18, 2023
99bfa62
Tests: introduce generic functions
spalmer25 Oct 18, 2023
4307fc5
Tests: use expect_section_content for operation fields
spalmer25 Oct 18, 2023
549ce9b
Tests: remove device for the generation
spalmer25 Oct 18, 2023
6324e18
Tests: add an until_failure
spalmer25 Oct 18, 2023
3347691
CI/Makefile: Update CI and Makefile to the new nano folder
spalmer25 Oct 18, 2023
18bf477
Tests: remove the legacy nano(s/sp/x) basic tests folder
spalmer25 Oct 18, 2023
d41d0f7
Tests: Relocate device-specific tests to their own folder
spalmer25 Oct 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 18 additions & 52 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,13 @@ jobs:

- name: Run test
run: |
./tests/integration/run_test_local.sh -F -m ${{ matrix.device }} \
./tests/integration/${{ matrix.device }}
emturner marked this conversation as resolved.
Show resolved Hide resolved
if [ "${{ matrix.device }}" = "stax" ]; then
./tests/integration/run_test_local.sh -F -m ${{ matrix.device }} \
./tests/integration/${{ matrix.device }}
else
./tests/integration/run_test_local.sh -F -m ${{ matrix.device }} \
./tests/integration/nano ./tests/integration/${{ matrix.device }}
fi

- name: Upload results
uses: actions/upload-artifact@v3
Expand All @@ -114,74 +119,35 @@ jobs:

- name: Create dirs
run: |
mkdir -p tests/samples/micheline/nanos
mkdir -p tests/samples/operations/nanos

mkdir -p tests/samples/micheline/nanosp
mkdir -p tests/samples/operations/nanosp

mkdir -p tests/samples/micheline/nanox
mkdir -p tests/samples/operations/nanox
Copy link
Collaborator

@emturner emturner Oct 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merging the generated tests makes sense I think 👍🏻 . In which case it might be worth generating some more? but then splitting them between the three devices potentially.

e.g. could generated 1500, and run 500 on each device

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to run the same tests on each device to make sure that a test fails on a specific device and if we wanted to run more tests, we'd have to do it at a different time from the merge requests CI.
What do you think about it?

Copy link
Collaborator Author

@spalmer25 spalmer25 Oct 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you're up for it, I'd like to associate a specific outcome with this change.
#72

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that makes sense - let's defer until after we submit the app

mkdir -p tests/samples/micheline/nano
mkdir -p tests/samples/operations/nano

- name: Generate
run: |
export PATH=/home/opam/.opam/4.14/bin:$PATH

dune exec ./tests/generate/generate.exe micheline 500 \
nanos tests/samples/micheline
dune exec ./tests/generate/generate.exe operations 500 \
nanos tests/samples/operations

dune exec ./tests/generate/generate.exe micheline 500 \
nanosp tests/samples/micheline
nano tests/samples/micheline
dune exec ./tests/generate/generate.exe operations 500 \
nanosp tests/samples/operations \

dune exec ./tests/generate/generate.exe micheline 500 \
nanox tests/samples/micheline
dune exec ./tests/generate/generate.exe operations 500 \
nanox tests/samples/operations
nano tests/samples/operations

- name: Unit tests
run: |
export PATH=/home/opam/.opam/4.14/bin:$PATH

make -C tests/unit

- name: Upload results (nanos, micheline)
uses: actions/upload-artifact@v3
with:
name: nanos_samples_micheline
path: tests/samples/micheline/nanos
- name: Upload results (nanos, operations)
uses: actions/upload-artifact@v3
with:
name: nanos_samples_operations
path: tests/samples/operations/nanos

- name: Upload results (nanosp, micheline)
uses: actions/upload-artifact@v3
with:
name: nanosp_samples_micheline
path: tests/samples/micheline/nanosp

- name: Upload results (nanosp, operations)
uses: actions/upload-artifact@v3
with:
name: nanosp_samples_operations
path: tests/samples/operations/nanosp

- name: Upload results (nanox, micheline)
- name: Upload results (nano, micheline)
uses: actions/upload-artifact@v3
with:
name: nanox_samples_micheline
path: tests/samples/micheline/nanox
name: nano_samples_micheline
path: tests/samples/micheline/nano

- name: Upload results (nanox, operations)
- name: Upload results (nano, operations)
uses: actions/upload-artifact@v3
with:
name: nanox_samples_operations
path: tests/samples/operations/nanox
name: nano_samples_operations
path: tests/samples/operations/nano

integration_tests_samples:
needs: [build_app, generate_samples_unit_tests, build_docker_integration_tests]
Expand Down Expand Up @@ -213,7 +179,7 @@ jobs:
- name: Download samples
uses: actions/download-artifact@v3
with:
name: ${{ matrix.device }}_samples_${{ matrix.type }}
name: nano_samples_${{ matrix.type }}
path: tests

- name: Test
Expand Down
22 changes: 15 additions & 7 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,8 @@ clean:
$(DOCKER_RUN_APP_BUILDER) make -C app mrproper
$(DOCKER_RUN_APP_OCAML) bash -c "make -C /app/tests/generate clean && cd /app && rm -rf _build"

unit_tests: test/samples/micheline/nanos/samples.hex \
test/samples/operations/nanos/samples.hex \
unit_tests: test/samples/micheline/nano/samples.hex \
test/samples/operations/nano/samples.hex \
tests/unit/*.ml* \
tests/unit/*.[ch] \
tests/unit/dune \
Expand All @@ -79,29 +79,37 @@ unit_tests: test/samples/micheline/nanos/samples.hex \

RUN_TEST_DOCKER = ./tests/integration/run_test_docker.sh

integration_tests_basic_stax: app_stax.tgz \
app_stax_dbg.tgz \
tests/integration/* \
tests/integration/stax/*
$(RUN_TEST_DOCKER) stax tests/integration/stax

integration_tests_basic_%: app_%.tgz \
app_%_dbg.tgz \
tests/integration/* \
tests/integration/nano/* \
tests/integration/%/*
$(RUN_TEST_DOCKER) $* tests/integration/$*
$(RUN_TEST_DOCKER) $* tests/integration/nano tests/integration/$*

integration_tests_basic: integration_tests_basic_nanos \
integration_tests_basic_nanosp \
integration_tests_basic_nanox \
integration_tests_basic_stax

integration_tests_%: integration_tests_basic_% \
test/samples/operations/%/samples.hex \
test/samples/micheline/%/samples.hex \
test/samples/operations/nano/samples.hex \
test/samples/micheline/nano/samples.hex \
tests/integration/*.sh
$(RUN_TEST_DOCKER) $* \
tests/samples/micheline/$* \
tests/samples/operations/$*
tests/samples/micheline/nano \
tests/samples/operations/nano

integration_tests: tests/integration/*.sh \
integration_tests_nanos \
integration_tests_nanosp \
integration_tests_nanox \
integration_tests_basic_stax

test/samples/micheline/%/samples.hex: tests/generate/*.ml* \
tests/generate/dune \
Expand Down
6 changes: 2 additions & 4 deletions tests/generate/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
.PHONY: nanos nanosp nanox
nanos:
nanosp:
nanox:
.PHONY: nano stax
emturner marked this conversation as resolved.
Show resolved Hide resolved
nano:

../samples/micheline/%/samples.hex: % *.ml* dune Makefile
mkdir -p ../samples/micheline/$<
Expand Down
82 changes: 25 additions & 57 deletions tests/generate/gen_integration.ml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,29 +30,14 @@ let start_speculos ppf mnemonic =
Format.fprintf ppf "start_speculos \"%a\"@." Tezos_client_base.Bip39.pp
mnemonic

module Device = struct
type t = Nanos | Nanosp | Nanox

let of_string_exn = function
| "nanos" -> Nanos
| "nanosp" -> Nanosp
| "nanox" -> Nanox
| d -> failwith @@ Format.sprintf "invalid device %s" d

let to_string = function
| Nanos -> "nanos"
| Nanosp -> "nanosp"
| Nanox -> "nanox"
end

let expect_full_text ppf l =
let pp_space ppf () = Format.fprintf ppf " " in
let pp_args = Format.pp_print_list ~pp_sep:pp_space shell_escape in
Format.fprintf ppf "expect_full_text %a@." pp_args l

let expect_section_content ppf ~device ~title content =
Format.fprintf ppf "expect_section_content %s %a %a@."
(Device.to_string device) shell_escape title shell_escape content
let expect_section_content ppf ~title content =
Format.fprintf ppf "expect_section_content %a %a@." shell_escape title
shell_escape content

module Button = struct
type t = Both | Right | Left
Expand Down Expand Up @@ -86,7 +71,9 @@ let send_async_apdus ppf async_apdus =
let expect_async_apdus_sent ppf () =
Format.fprintf ppf "expect_async_apdus_sent@."

let expected_home ppf () = Format.fprintf ppf "expected_home@."
let expect_exited ppf () = Format.fprintf ppf "expect_exited@."
let quit_app ppf () = Format.fprintf ppf "quit_app@."

let check_tlv_signature ppf ~prefix ~suffix pk message =
Format.fprintf ppf "check_tlv_signature %a %a %a %a@." pp_hex_bytes prefix
Expand All @@ -95,46 +82,28 @@ let check_tlv_signature ppf ~prefix ~suffix pk message =

(** Specific *)

let home ppf = function
| Device.Nanos -> expect_full_text ppf [ "ready for"; "safe signing" ]
| Device.Nanosp | Device.Nanox ->
expect_full_text ppf [ "Tezos Wallet"; "ready for"; "safe signing" ]

let expect_accept ppf = function
| Device.Nanos -> expect_full_text ppf [ "Accept?" ]
| Device.Nanosp | Device.Nanox ->
expect_full_text ppf [ "Accept?"; "Press both buttons to accept." ]

let expect_reject ppf = function
| Device.Nanos -> expect_full_text ppf [ "Reject?" ]
| Device.Nanosp | Device.Nanox ->
expect_full_text ppf [ "Accept?"; "Press both buttons to reject." ]

let expect_quit ppf () = expect_full_text ppf [ "Quit?" ]
let expect_accept ppf () = Format.fprintf ppf "expected_accept@."
let expect_reject ppf () = Format.fprintf ppf "expected_reject@."

let accept ppf device =
expect_accept ppf device;
let accept ppf () =
expect_accept ppf ();
Button.(press ppf Both)

let reject ppf device =
expect_reject ppf device;
Button.(press ppf Both)

let quit ppf () =
expect_quit ppf ();
let reject ppf () =
expect_reject ppf ();
Button.(press ppf Both)

type screen = { title : string; contents : string }

let make_screen ~title contents = { title; contents }

let expected_screen ppf ~device { title; contents } =
expect_section_content ppf ~device ~title contents
let expected_screen ppf { title; contents } =
expect_section_content ppf ~title contents

let go_through_screens ppf ~device screens =
let go_through_screens ppf screens =
List.iter
(fun screen ->
expected_screen ppf ~device screen;
expected_screen ppf screen;
Button.(press ppf Right))
screens

Expand Down Expand Up @@ -521,31 +490,30 @@ let tz3_signers =

let gen_signer = QCheck2.Gen.oneofl (tz1_signers @ tz2_signers @ tz3_signers)

let gen_expect_test_sign ppf ~device ~watermark bin screens =
let gen_expect_test_sign ppf ~watermark bin screens =
Format.fprintf ppf "# full input: %a@." pp_hex_bytes bin;
let signer = QCheck2.Gen.generate1 ~rand:Gen_utils.random_state gen_signer in
Format.fprintf ppf "# signer: %a@." Tezos_crypto.Signature.Public_key_hash.pp
signer.pkh;
Format.fprintf ppf "# path: %a@." Apdu.Path.pp signer.path;
start_speculos ppf signer.mnemonic;
home ppf device;
expected_home ppf ();
sign ppf ~signer ~watermark bin;
go_through_screens ppf ~device screens;
accept ppf device;
expect_async_apdus_sent ppf ()
go_through_screens ppf screens;
accept ppf ();
expect_async_apdus_sent ppf ();
quit_app ppf ()

let gen_expect_test_sign_micheline_data ~device ppf bin =
let gen_expect_test_sign_micheline_data ppf bin =
let screens =
let node = Gen_micheline.decode bin in
node_to_screens ~title:"Expression" ppf (Micheline.root node)
in
gen_expect_test_sign ppf ~device ~watermark:Gen_micheline.watermark bin
screens
gen_expect_test_sign ppf ~watermark:Gen_micheline.watermark bin screens

let gen_expect_test_sign_operation ~device ppf bin =
let gen_expect_test_sign_operation ppf bin =
let screens =
let op = Gen_operations.decode bin in
operation_to_screens op
in
gen_expect_test_sign ppf ~device ~watermark:Gen_operations.watermark bin
screens
gen_expect_test_sign ppf ~watermark:Gen_operations.watermark bin screens
26 changes: 10 additions & 16 deletions tests/generate/generate.ml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,9 @@

let () =
match Sys.argv with
| [| _; "micheline"; m; ("nanos" | "nanosp" | "nanox") as model; dir |] ->
let device = Gen_integration.Device.of_string_exn model in
| [| _; "micheline"; m; "nano"; dir |] ->
let m = int_of_string m in
let fp_hex = open_out @@ Format.sprintf "%s/%s/samples.hex" dir model in
let fp_hex = open_out @@ Format.sprintf "%s/nano/samples.hex" dir in
let ppf_hex = Format.formatter_of_out_channel fp_hex in
print_string "Generating Micheline samples";
Seq.iteri
Expand All @@ -30,19 +29,16 @@ let () =
print_string ".";
flush stdout;
Format.fprintf ppf_hex "%s@\n" txt;
let fp =
open_out (Format.asprintf "%s/%s/test_%03d.sh" dir model i)
in
let fp = open_out (Format.asprintf "%s/nano/test_%03d.sh" dir i) in
let ppf = Format.formatter_of_out_channel fp in
Gen_integration.gen_expect_test_sign_micheline_data ~device ppf bin;
Gen_integration.gen_expect_test_sign_micheline_data ppf bin;
close_out fp))
(Seq.take m Gen_micheline.hex);
Format.fprintf ppf_hex "%!";
print_newline ()
| [| _; "operations"; m; ("nanos" | "nanosp" | "nanox") as model; dir |] ->
let device = Gen_integration.Device.of_string_exn model in
| [| _; "operations"; m; "nano"; dir |] ->
let m = int_of_string m in
let fp_hex = open_out @@ Format.sprintf "%s/%s/samples.hex" dir model in
let fp_hex = open_out @@ Format.sprintf "%s/nano/samples.hex" dir in
let ppf_hex = Format.formatter_of_out_channel fp_hex in
print_string "Generating Operation samples";
Seq.iteri
Expand All @@ -54,20 +50,18 @@ let () =
print_string ".";
flush stdout;
Format.fprintf ppf_hex "%s@\n" txt;
let fp =
open_out (Format.asprintf "%s/%s/test_%03d.sh" dir model i)
in
let fp = open_out (Format.asprintf "%s/nano/test_%03d.sh" dir i) in
let ppf = Format.formatter_of_out_channel fp in
Gen_integration.gen_expect_test_sign_operation ~device ppf bin;
Gen_integration.gen_expect_test_sign_operation ppf bin;
close_out fp))
(Seq.take m Gen_operations.hex);
Format.fprintf ppf_hex "%!";
print_newline ()
| [| _; _; _m; "stax"; _dir |] ->
Format.eprintf "Only nanos|nanosp|nanox supported for now.@.";
Format.eprintf "Only nano supported for now.@.";
exit 1
| _ ->
Format.eprintf
"Usage: %s micheline <samples> <nanos|nanosp|nanox|stax> <dir>@."
"Usage: %s <micheline|operations> <samples> <nano|stax> <dir>@."
Sys.executable_name;
exit 1
6 changes: 3 additions & 3 deletions tests/integration/nanos/test_basic.sh → tests/integration/nano/test_basic.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
start_speculos "$seed"
sleep 0.2
expect_full_text "ready for" "safe signing"
expected_home
press_button right
expect_full_text "Settings"
press_button right
expect_full_text "Quit?"
press_button left
expect_full_text "Settings"
press_button left
expect_full_text "ready for" "safe signing"
expected_home
press_button both
expect_full_text "ready for" "safe signing"
expected_home
press_button right
expect_full_text "Settings"
press_button right
Expand Down
Loading
Loading