Try to build with OCaml 4.08 version #957

XVilka · 2019-06-19T04:41:37Z

With the new OCaml 4.08 release there are lot of nice optimizations and features out of the box, even without changing the current code. Worth to make sure it builds. For now lets see if everything passes OK.

XVilka · 2019-06-19T06:31:16Z

Strange error on 4.07 compiler:

- ocamldoc -html -d ../autodoc  -t "FrontC" cabs.ml cxml.ml cprint.ml ctoxml.ml frontc.ml
- ../autodoc/style.css: No such file or directory
- Fatal error: exception Failure("../autodoc/Cabs.html: No such file or directory")
- make[1]: *** [doc] Error 2

Also I don't get what is happening there - error message looks empty

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
+ /home/travis/.opam/opam-init/hooks/sandbox.sh "build" "./configure" "--prefix=/home/travis/.opam/4.07.0" "--with-cxx=`which clang++`" "--with-llvm-version=5.0.0git-929163d" "--with-llvm-config=/usr/local/clang-5.0.0/bin/llvm-config" "--mandir=/home/travis/.opam/4.07.0/man" "--enable-everything" "--enable-tests" "--disable-ida" "--disable-fsi-benchmark" "--disable-objdump" (CWD=/home/travis/.opam/4.07.0/.opam-switch/build/bap.master)
[ERROR] The compilation of bap failed at
        "/home/travis/.opam/opam-init/hooks/sandbox.sh build ./configure
        --prefix=/home/travis/.opam/4.07.0 --with-cxx=`which clang++`
        --with-llvm-version=5.0.0git-929163d
        --with-llvm-config=/usr/local/clang-5.0.0/bin/llvm-config
        --mandir=/home/travis/.opam/4.07.0/man --enable-everything
        --enable-tests --disable-ida --disable-fsi-benchmark
        --disable-objdump".
#=== ERROR while compiling bap.master =========================================#
# context     2.0.4 | linux/x86_64 | ocaml-base-compiler.4.07.0 | pinned(file:///home/travis/build/BinaryAnalysisPlatform/bap)
# path        ~/.opam/4.07.0/.opam-switch/build/bap.master
# command     ~/.opam/opam-init/hooks/sandbox.sh build ./configure --prefix=/home/travis/.opam/4.07.0 --with-cxx=`which clang++` --with-llvm-version=5.0.0git-929163d --with-llvm-config=/usr/local/clang-5.0.0/bin/llvm-config --mandir=/home/travis/.opam/4.07.0/man --enable-everything --enable-tests --disable-ida --disable-fsi-benchmark --disable-objdump
# exit-code   1
# env-file    ~/.opam/log/bap-3530-05296d.env
# output-file ~/.opam/log/bap-3530-05296d.out
<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build bap master
└─ 
╶─ No changes have been performed

XVilka · 2019-06-19T07:01:22Z

Blocked by alavrik/piqi#62 and camlp4/camlp4#151

XVilka · 2019-06-26T10:44:09Z

I checked the rest of the dependencies - they behave well for OCaml 4.08.0+, so once piqi is fixed, should be good.

XVilka · 2019-08-06T03:41:26Z

Note that camlp5 7.08 was released, and piqi moved to use it instead of camlp4. So please rerun the Travis, to check the build.

ivg · 2019-08-06T19:12:39Z

Our Travis infrastructure is still relying on opam 1.x so we don't see the updated piqi package. That only means, that we have to update our Travis CI scripts. Indeed, it doesn't make sense to test installability with 1.x

XVilka · 2019-08-12T06:36:09Z

I don't get the reason of 4.07 build failure:

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
+ /home/travis/.opam/opam-init/hooks/sandbox.sh "build" "./configure" "--prefix=/home/travis/.opam/4.07.0" "--with-cxx=`which clang++`" "--with-llvm-version=5.0.0git-929163d" "--with-llvm-config=/usr/local/clang-5.0.0/bin/llvm-config" "--mandir=/home/travis/.opam/4.07.0/man" "--enable-everything" "--enable-tests" "--disable-ida" "--disable-fsi-benchmark" "--disable-objdump" (CWD=/home/travis/.opam/4.07.0/.opam-switch/build/bap.master)
[ERROR] The compilation of bap failed at
        "/home/travis/.opam/opam-init/hooks/sandbox.sh build ./configure
        --prefix=/home/travis/.opam/4.07.0 --with-cxx=`which clang++`
        --with-llvm-version=5.0.0git-929163d
        --with-llvm-config=/usr/local/clang-5.0.0/bin/llvm-config
        --mandir=/home/travis/.opam/4.07.0/man --enable-everything
        --enable-tests --disable-ida --disable-fsi-benchmark
        --disable-objdump".
#=== ERROR while compiling bap.master =========================================#
# context     2.0.4 | linux/x86_64 | ocaml-base-compiler.4.07.0 | pinned(file:///home/travis/build/BinaryAnalysisPlatform/bap)
# path        ~/.opam/4.07.0/.opam-switch/build/bap.master
# command     ~/.opam/opam-init/hooks/sandbox.sh build ./configure --prefix=/home/travis/.opam/4.07.0 --with-cxx=`which clang++` --with-llvm-version=5.0.0git-929163d --with-llvm-config=/usr/local/clang-5.0.0/bin/llvm-config --mandir=/home/travis/.opam/4.07.0/man --enable-everything --enable-tests --disable-ida --disable-fsi-benchmark --disable-objdump
# exit-code   1
# env-file    ~/.opam/log/bap-5087-05296d.env
# output-file ~/.opam/log/bap-5087-05296d.out
<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build bap master
└─ 
╶─ No changes have been performed
The packages you requested declare the following system dependencies. Please
make sure they are installed before retrying:
    clang dejagnu libcurl4-gnutls-dev libgmp-dev libzip-dev llvm llvm-3.8-dev
    m4 time
'opam install bap -v' failed.
'unset TESTS; OPAMYES=1 export OPAMYES; set -uex; opam install bap "-v"' exited 31. Terminating with 31
The command "bash -ex .travis_install.sh" exited with 31.
cache.2
store build cache

ivg · 2019-08-12T20:53:27Z

The build fails, because the oasis file wasn't updated (it also limits the version of OCaml). @gitoleg revived his old branch and managed to build everything for 4.08. While he is running his tests, here is the status update.

The 4.07 is the latest version supported by core_kernel.v0.11.0, so in order to switch to 4.08 we need to switch to a newer version of core_kernel. As always, the switch is not backward compatible, so we can't support both v0.11 and v0.12 (especially, since they renamed a couple of libraries). Therefore, the solution is to switch to v0.12 and drop the support for 4.04, 4.05, and 4.06. So that now we will support only 4.07 and 4.08.

I'm totally ready to switch to 4.08+core.v0.12 even today in bap-2.0 branch. The main question is whether we should backport this to 1.x. Basically, the list of options that we have:

keep 1.6 on old compilers and update only 2.0
switch both 1.6 and 2.0 to the new compilers
fork 1.6 into 1.6.x, 1.7.0, and 2.0.0 with the latter two using v0.12 and the first still tracking the older compiler.

Opinions, other options?

XVilka · 2019-08-13T03:46:46Z

I think the third option is the sanest one, keeping one with the modern world, without the breaking old things.

ivg · 2019-08-13T16:59:53Z

I think the third option is the sanest one, keeping one with the modern world, without the breaking old things.

Yep, but this implies lots of work and less clear work. If we will pursue this path, then we need to keep code compatible with different versions of OCaml (including the deprecation and removal of Pervasives and Genarray.*.map_file, which requires us to add a few shim libraries as dependencies, and use them as indirections in our code).

XVilka · 2019-08-13T17:05:49Z

Then the least work path

…

On Wed, Aug 14, 2019, 12:59 AM Ivan Gotovchits ***@***.***> wrote: I think the third option is the sanest one, keeping one with the modern world, without the breaking old things. Yep, but this implies lots of work and less clear work. If we will pursue this path, then we need to keep code compatible with different versions of OCaml (including the deprecation and removal of Pervasives and Genarray.*.map_file, which requires us to add a few shim libraries as dependencies, and use them as indirections in our code). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#957?email_source=notifications&email_token=AABRT7K7CNJWXQ5MLD3JNN3QELSAVA5CNFSM4HZFZ3K2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4GJSPQ#issuecomment-520919358>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABRT7OPTWO7EUSHQTMBSZ3QELSAVANCNFSM4HZFZ3KQ> .

ivg · 2019-08-15T13:31:28Z

This branch is go to go into the master (after a merge or rebase), see the related issue for more discussion

XVilka · 2019-08-16T04:06:56Z

Rebased against the current master. Travis is drunk though. Sometimes it shows "Can't find the repository", sometimes - "can't find the build".

XVilka · 2019-08-16T06:17:30Z

I don't get why it shows while it was fixed:

# + /home/travis/.opam/4.07.0/bin/ocamlfind ocamlc -c -g -annot -bin-annot -short-paths -I lib/monads -I lib/bap_strings -I lib/bitvec -I lib/bitvec_sexp -I lib/bitvec_binprot -I lib/bitvec_order -I lib/knowledge -I lib/bap_core_theory -I lib/regular -I lib/graphlib -w +a-4-6-7-9-27-29-32..42-44-45-48-50-60 -package zarith -package sexplib0 -package ppx_jane -package parsexp -package ocamlgraph[...]
# File "lib/bap_lisp/bap_lisp__resolve.ml", line 115, characters 6-39:
# Error: This expression has type (var * 'weak2) list List.Or_unequal_lengths.t
#        but an expression was expected of type 'weak5 option
# Command exited with code 2.
# make: *** [build] Error 1

Even the line number is wrong

XVilka · 2019-08-16T06:19:03Z

Looks like Travis doesn't get the valid sources.

XVilka · 2019-08-19T05:10:40Z

Note, that the piqi switched to sedlex, so no need for camlp4 and camlp5 dependency! \o/
BAP is more future-proof now. Waiting for the release now: alavrik/piqi#61

XVilka · 2019-08-29T07:02:04Z

I think we also can drop these lines from the .travis-ocaml.sh:

  case "$OCAML_VERSION,$OPAM_VERSION" in
    3.12,1.2.2) OCAML_FULL_VERSION=3.12.1; brew install opam ;;
    4.00,1.2.2) OCAML_FULL_VERSION=4.00.1; brew install opam ;;
    4.01,1.2.2) OCAML_FULL_VERSION=4.01.0; brew install opam ;;
    4.02,1.2.2) OCAML_FULL_VERSION=4.02.3; brew install opam ;;
    4.02,1.3.0) OCAML_FULL_VERSION=4.02.3; brew install opam --HEAD ;;

https://github.com/BinaryAnalysisPlatform/bap/blob/master/.travis-ocaml.sh#L142
And other, similar cases too.

XVilka · 2019-09-13T09:23:19Z

There is still no answer from piqi author, but maybe it's possible to pin the repo for a while?

XVilka · 2019-09-29T06:41:21Z

Should be good to retry once ocaml/opam-repository#14947 is merged:

Changes

Piqi 0.6.15 (September 28, 2018)

Miscellaneous:

Switch from ulex to sedlex
Rely on opam for installing third-party dependencies
Bump minimal supported OCaml version from 4.02 to 4.03

P.S. We should try also the 4.09 too.

XVilka · 2019-10-02T06:08:59Z

Still the same error. Travis seems out of sync with the git and opam.

XVilka · 2019-10-11T07:44:12Z

So what do we do with this one?

gitoleg · 2019-10-15T19:19:53Z

@XVilka
We're currently working on the release of bap.2.0 that should happen before the end of this month. Right after that, we'll merge this PR.

XVilka · 2019-11-21T04:16:13Z

@ivg completely agree. One of the interesting features of the new library is a deriver plugin for Python accessors generation: https://github.com/janestreet/ppx_python

Since BAP has Python bindings, it might be quite useful for making them better.

ivg · 2020-01-23T16:18:10Z

lib/bap_future/bap_future.mli

-    module Args : Applicative.Args with type 'a arg := 'a t
+
+    module Args : sig
+      type 'a arg = 'a t


remove the arg type from the signature, it wasn't there before

ivg · 2020-01-23T16:56:02Z

lib/bap_primus/bap_primus_lisp_resolve.ml

  List.for_all ~f:(fun (v,x) ->
      f v.data.typ x)

+let zip x y =
+  let open List.Or_unequal_lengths in


There is no need for this open, as now the compiler will take the constructors from the right place.

ivg · 2020-01-23T17:06:00Z

lib/bap_types/bap_bitvector.ml

@@ -178,15 +178,15 @@ let pp_generic
    | `this x -> fprintf ppf "%s" x
    | `base -> pp_prefix ppf
    | `auto ->
-      if Bitvec_order.(x >= (min (word 10) base))
+      if Z.Compare.(x >= (Z.min (word 10) base))


wow, it was a bug before... not that it was breaking before, but at least the code as written was not the code as meant.

ivg · 2020-01-23T17:13:38Z

lib/bap_types/bap_ir.ml

-  let apply_map name get map skip blk ~f =
-    if List.mem ~equal:Polymorphic_compare.equal skip name
+  let apply_map ~equal name get map skip blk ~f =
+    if List.mem ~equal skip name


it is fine to use the polymorphic compare here

ivg

a couple of changes and we're good to go

restore ignore instead of Caml.ignore.
remove the _ arg type from the Args interface

As a general note, it is fine to use polymorphic compare, so this PR could be much easier to digest if we would just do open Poly.

File "oasis/common.setup.ml.in", line 1773, characters 22-40: Alert deprecated: module Stdlib.Pervasives Use Stdlib instead. If you need to stay compatible with OCaml < 4.07, you can use the stdlib-shims library: https://github.com/ocaml/stdlib-shims File "setup.ml", line 3467, characters 16-34: 3467 | Pervasives.compare o2 o1) ^^^^^^^^^^^^^^^^^^ Alert deprecated: module Stdlib.Pervasives Use Stdlib instead. If you need to stay compatible with OCaml < 4.07, you can use the stdlib-shims library: https://github.com/ocaml/stdlib-shims

Previously, we decided to disable the Jane Street code style checks as a fast solution, but it turned out that it is easier to adopt their style than to fight with it. To disable the checks, we used the _tags file, but since we have a global repository the tags file affected all subprojects, even those that do not use the ppx-jane preprocessor, thus breaking packages that do not explicitly depend on ppx-jane. Also, in BinaryAnalysisPlatform#957 we accidentally introduced a dependency on a newer version of `zartith` by using the `Compare` module. We will now use plaincompare et alas functions so that the old version of `zarith` could be still used with BAP.

Previously, we decided to disable the Jane Street code style checks as a fast solution, but it turned out that it is easier to adopt their style than to fight with it. To disable the checks, we used the _tags file, but since we have a global repository the tags file affected all subprojects, even those that do not use the ppx-jane preprocessor, thus breaking packages that do not explicitly depend on ppx-jane. Also, in #957 we accidentally introduced a dependency on a newer version of `zartith` by using the `Compare` module. We will now use plaincompare et alas functions so that the old version of `zarith` could be still used with BAP.

XVilka mentioned this pull request Jun 26, 2019

Deprecate ulex in favor of ulex-campl5 ocaml/opam-repository#14344

Closed

ivg mentioned this pull request Aug 6, 2019

update Travis CI infrastructure #965

Closed

XVilka force-pushed the patch-3 branch from f046d12 to 72cd85c Compare August 12, 2019 05:47

ivg mentioned this pull request Aug 15, 2019

Pervasives are deprecated - switch to stdlib-shims is suggested #969

Closed

XVilka mentioned this pull request Aug 15, 2019

FrontC documentation compilation error BinaryAnalysisPlatform/FrontC#5

Closed

XVilka force-pushed the patch-3 branch from 0ca5350 to a47ac93 Compare August 16, 2019 04:06

XVilka force-pushed the patch-3 branch from 9200ffc to 2a2d301 Compare August 28, 2019 10:48

XVilka force-pushed the patch-3 branch from 2a2d301 to e34fe4b Compare September 13, 2019 09:22

XVilka force-pushed the patch-3 branch from e34fe4b to ede6a02 Compare October 2, 2019 05:44

XVilka mentioned this pull request Nov 22, 2019

Use 4.07.1 for Xenial docker #972

Closed

gitoleg added 9 commits November 26, 2019 14:39

Merge remote-tracking branch 'upstream/master' into patch-3

4946640

removed polymorphic compare almost everywhere

9a92f5c

fixes few more lost polymorphic compare + warnings

f3c813e

fixed one more deprecated

20a4d89

Merge remote-tracking branch 'upstream/master' into patch-3

26a917d

updates testsuite submodule

5ad0855

revert changes in text-tags

5dbc104

remove new Format functions from bap_ir

c42a89b

updated testsuite/bap-veri/

3b95870

gitoleg force-pushed the patch-3 branch from fa7db13 to 3b95870 Compare December 10, 2019 19:54

gitoleg added 2 commits January 21, 2020 15:08

silenced a couple of warnings

7c186c9

reworked bap-future applicative interface

4386e70

gitoleg requested a review from ivg January 21, 2020 22:30

gitoleg added the ready for review label Jan 21, 2020

ivg reviewed Jan 23, 2020

View reviewed changes

ivg requested changes Jan 23, 2020

View reviewed changes

updated bap-future, replaced Caml.ignore to ignore

3bb5b74

ivg approved these changes Jan 23, 2020

View reviewed changes

gitoleg added 2 commits January 23, 2020 14:15

added travis 4.09 to travis

9bb1731

ivg approved these changes Jan 23, 2020

View reviewed changes

ivg merged commit 1181879 into BinaryAnalysisPlatform:master Jan 23, 2020

gitoleg mentioned this pull request Jan 27, 2020

enables the Jane Street code style #1038

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Try to build with OCaml 4.08 version #957

Try to build with OCaml 4.08 version #957

XVilka commented Jun 19, 2019

XVilka commented Jun 19, 2019

XVilka commented Jun 19, 2019

XVilka commented Jun 26, 2019

XVilka commented Aug 6, 2019

ivg commented Aug 6, 2019

XVilka commented Aug 12, 2019

ivg commented Aug 12, 2019 •

edited

Loading

XVilka commented Aug 13, 2019

ivg commented Aug 13, 2019

XVilka commented Aug 13, 2019 via email

ivg commented Aug 15, 2019

XVilka commented Aug 16, 2019 •

edited

Loading

XVilka commented Aug 16, 2019

XVilka commented Aug 16, 2019

XVilka commented Aug 19, 2019

XVilka commented Aug 29, 2019

XVilka commented Sep 13, 2019

XVilka commented Sep 29, 2019 •

edited

Loading

XVilka commented Oct 2, 2019

XVilka commented Oct 11, 2019

gitoleg commented Oct 15, 2019

XVilka commented Nov 21, 2019

ivg Jan 23, 2020

ivg Jan 23, 2020

ivg Jan 23, 2020

ivg Jan 23, 2020

ivg left a comment

Try to build with OCaml 4.08 version #957

Try to build with OCaml 4.08 version #957

Conversation

XVilka commented Jun 19, 2019

XVilka commented Jun 19, 2019

XVilka commented Jun 19, 2019

XVilka commented Jun 26, 2019

XVilka commented Aug 6, 2019

ivg commented Aug 6, 2019

XVilka commented Aug 12, 2019

ivg commented Aug 12, 2019 • edited Loading

XVilka commented Aug 13, 2019

ivg commented Aug 13, 2019

XVilka commented Aug 13, 2019 via email

ivg commented Aug 15, 2019

XVilka commented Aug 16, 2019 • edited Loading

XVilka commented Aug 16, 2019

XVilka commented Aug 16, 2019

XVilka commented Aug 19, 2019

XVilka commented Aug 29, 2019

XVilka commented Sep 13, 2019

XVilka commented Sep 29, 2019 • edited Loading

Changes

XVilka commented Oct 2, 2019

XVilka commented Oct 11, 2019

gitoleg commented Oct 15, 2019

XVilka commented Nov 21, 2019

ivg Jan 23, 2020

Choose a reason for hiding this comment

ivg Jan 23, 2020

Choose a reason for hiding this comment

ivg Jan 23, 2020

Choose a reason for hiding this comment

ivg Jan 23, 2020

Choose a reason for hiding this comment

ivg left a comment

Choose a reason for hiding this comment

ivg commented Aug 12, 2019 •

edited

Loading

XVilka commented Aug 16, 2019 •

edited

Loading

XVilka commented Sep 29, 2019 •

edited

Loading