Unify the buildbot testing of the 3 packages #1

ltratt · 2019-04-16T20:42:33Z

Surprisingly this worked first time:

http://bencher12.soft-dev.org:8010/#/builders/2/builds/123

I've checked the log and it really is executing all of the tests we want. I should have bought a lottery ticket instead...

vext01 · 2019-04-17T09:43:03Z

I should have bought a lottery ticket instead...

Wanna go halfs next time?

This LGTM, but we need to add a bors config file (derive from the existing one in yktrace?) and then enable bors on this repo. I can help with this, but let's get the existing BB config into a repo before we break it ;)

13: Yk copy good builds r=ltratt a=vext01 Successful builds are tarred up and placed in /opt/ykrustc-bin-snapshots on the build master. For now, store at most one archive, but we might change that in the future. I've tested something very close to this (just without the chmod line in the buildbot script). It *should* work fine 😸 ``` $ ls -al /opt/ykrustc-bin-snapshots/ total 188948 drwxrwxr-x 2 root buildbot_workers 4096 Apr 18 18:23 . drwxr-xr-x 8 root root 4096 Apr 18 11:32 .. -rwxrwxr-x 1 buildbot-worker4 buildbot-worker4 193469353 Apr 18 18:23 ykrustc-stage2-057d397c.tar.bz2 $ cd /tmp $ tar jxf /opt/ykrustc-bin-snapshots/ykrustc-stage2-057d397c.tar.bz2 $ cd ykrustc-stage2/ $ ls bin lib VERSION $ cat VERSION commit 057d397c982463d374f60e16f9b3863dc660e4f2 Author: Edd Barrett <[email protected]> Date: Thu Apr 18 12:04:20 2019 +0100 Kill [install]. $ ``` If you agree with this, then I suppose the next step is to try using the tarball in: ykjit/yk#1 Co-authored-by: Edd Barrett <[email protected]>

vext01 · 2019-04-23T15:27:03Z

This LGTM, but we need to add a bors config file (derive from the existing one in yktrace?) and then enable bors on this repo. I can help with this, but let's get the existing BB config into a repo before we break it ;)

So the buildbot config is in git now, to move this forward:

copy in a bors config (move the one from the yktrace subdir up, or here's one https://github.com/softdevteam/ykrustc/blob/master/bors.toml)
Enable bors on the repo (use https://github.com/apps/bors)
Adjust .buldbot.sh to use a ykrustc snapshot? (maybe as a separate PR, as the PR to make the snapshot isn't merged yet).

ltratt · 2019-04-23T15:29:19Z

Can I suggest you do those steps? You know each of them, and I don't, and so we'd probably do best to use your expertise. We can do that in this PR, and then ask someone else to review it, or we can merge this PR, and then you can open one doing those steps.

vext01 · 2019-04-23T15:31:59Z

I can do yeah.

ltratt · 2019-04-23T15:32:30Z

Thanks! Would you prefer doing it in this PR or a fresh one?

vext01 · 2019-04-23T15:33:25Z

Let's do the first two points in the above bullet list in this PR. But let me clear my PR backlog first.

ltratt · 2019-04-23T15:35:17Z

OK.

vext01 · 2019-04-23T16:01:27Z

bors ping

bors · 2019-04-23T16:01:28Z

pong

vext01 · 2019-04-23T16:02:47Z

bors try

vext01 · 2019-04-23T16:13:36Z

bors try-

The snapshot PR was merged, so we might aswell do that here too.

vext01 · 2019-04-23T16:52:56Z

Last commit is a stab at using the snapshot tarballs for testing, but it wont work until this is merged:
softdevteam/ykrustc#15

vext01 · 2019-04-24T14:34:00Z

bors try

vext01 · 2019-04-24T15:14:54Z

bors try-

vext01 · 2019-04-24T15:15:09Z

bors try

bors · 2019-04-24T15:16:03Z

try

Build failed

buildbot/buildbot-build-script

vext01 · 2019-04-24T15:38:31Z

Hrm, it appears cargo is still not installed...

ltratt · 2019-04-24T15:51:08Z

I think the problem is export PATH=ykrustc-stage2/bin:/usr/bin:/bin -- shouldn't the first entry be /opt/...?

vext01 · 2019-04-24T15:54:57Z

The path is correct, there's only a tarball in /opt and we've already untarred it.

I've done post-mortem on the buildbot worker and build no cargo binary anywhere, so my guess is that the extended tools are a dependency of cargo install.

Trying using that instead...

vext01 · 2019-04-27T14:17:33Z

bors try

bors · 2019-04-29T15:54:59Z

try

Build failed

buildbot/buildbot-build-script

vext01 · 2019-04-29T16:00:42Z

bors try

bors · 2019-04-29T16:02:18Z

try

Build failed

buildbot/buildbot-build-script

vext01 · 2019-04-29T16:05:04Z

bors try

bors · 2019-04-29T16:20:35Z

try

Build succeeded

buildbot/buildbot-build-script

vext01 · 2019-05-01T09:14:27Z

This is now ready for review, but since we've both pushed, we'd need a third-party. Assigning @jacob-hughes

jacob-hughes · 2019-05-01T11:27:10Z

LGTM. Looks like you had some fun getting buildbot to work :) Feel free to squash.

We now test all packages and use a ykrustc binary snapshot.

vext01 · 2019-05-01T13:43:20Z

splat

vext01 · 2019-05-01T13:43:55Z

[don't forget to not use the green button! I can disable that after this merge]

jacob-hughes · 2019-05-01T13:47:04Z

bors r+

1: Unify the buildbot testing of the 3 packages r=jacob-hughes a=ltratt Surprisingly this worked first time: http://bencher12.soft-dev.org:8010/#/builders/2/builds/123 I've checked the log and it really is executing all of the tests we want. I should have bought a lottery ticket instead... Co-authored-by: Laurence Tratt <[email protected]>

bors · 2019-05-01T14:00:49Z

Build succeeded

buildbot/buildbot-build-script

Convert the wiki to an mdbook.

I've slowly come to realise that calls to `yk_promote` could (but don't currently!) end up being in compiled in two different ways: 1. A literal call to `yk_promote_*`. 2. An inlined call to a side-band recording mechanism (e.g. `PTWRITE`). Because I hadn't teased these two apart in my mind, my previous attempt had kind-of tried to merge the two together in one API. It's now clear that was entirely misguided on my part. This commit can be seen as partly undoing my previous attempt: literal calls to `yk_promote_*` now record the data in `MTThread` which is a thread local. However, when tracing stops, the promotion data is now extracted from `MTThread` so it can safely be moved to another thread. That means we can now handle ykjit#1 above: but we don't yet have an API that can handle ykjit#2.

In essence, this is the first commit towards moving us to "reverse code generation by stealth". After JIT IR is optimised, this commit changes things to a single reverse pass (previously we did a single forward pass). During that pass we: 1. Identify `PtrAdd` instructions which can be inlined by `Load`/`Store`. 2. Implicitly do a second round of dead-code elimination: currently this can only DCE some (not all!) `PtrAdd`s (those inlined by ykjit#1 that aren't used elsewhere). This turns out to be a much nicer way of optimised `PtrAdd`s than our previous attempts: the code is neater (though that doesn't mean it's not involved!) and it's also more effective because the register allocator now has less things it thinks are alive. Overall, this speeds up big_loop by about 10% because of the pressure it relieves from the register allocator. There is a lot of boring churn in tests because this forces us to `black_box` lots of tests in the code generator. Honestly, we should always have done that, but we got with it until now.

Add workspace.

b75a3bd

ltratt assigned vext01 Apr 16, 2019

vext01 mentioned this pull request Apr 23, 2019

Yk copy good builds softdevteam/ykrustc#13

Merged

vext01 mentioned this pull request Apr 23, 2019

Perform SSA variable renaming. softdevteam/ykrustc#14

Closed

bors bot added a commit that referenced this pull request Apr 23, 2019

Try #1:

e5a19da

bors bot added a commit that referenced this pull request Apr 24, 2019

Try #1:

7f90f41

bors bot added a commit that referenced this pull request Apr 24, 2019

Try #1:

c2b9697

vext01 mentioned this pull request Apr 25, 2019

Implement bits necessary for SSA conversion in the compiler. #2

Closed

bors bot added a commit that referenced this pull request Apr 27, 2019

Try #1:

84062a8

bors bot added a commit that referenced this pull request Apr 29, 2019

Try #1:

2b53316

bors bot added a commit that referenced this pull request Apr 29, 2019

Try #1:

2db5f49

vext01 mentioned this pull request Apr 29, 2019

Really install cargo softdevteam/ykrustc#16

Merged

vext01 assigned jacob-hughes May 1, 2019

Reconfigure tests.

137d1e0

We now test all packages and use a ykrustc binary snapshot.

vext01 force-pushed the unify branch from e13e26d to 137d1e0 Compare May 1, 2019 13:43

bors bot merged commit 137d1e0 into master May 1, 2019

bors bot deleted the unify branch May 1, 2019 14:00

ptersilie added a commit to ptersilie/yk that referenced this pull request Aug 31, 2022

Address comments ykjit#1

3a22adb

ltratt added a commit to ltratt/yk that referenced this pull request Apr 11, 2023

Merge pull request ykjit#1 from vext01/initial

dfdad18

Convert the wiki to an mdbook.

vext01 mentioned this pull request Nov 17, 2023

callsite_off < return_off #908

Closed

ltratt mentioned this pull request Feb 16, 2024

Move data recorded with yk_promote into MTThread. #976

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unify the buildbot testing of the 3 packages #1

Unify the buildbot testing of the 3 packages #1

ltratt commented Apr 16, 2019

vext01 commented Apr 17, 2019

vext01 commented Apr 23, 2019

ltratt commented Apr 23, 2019

vext01 commented Apr 23, 2019

ltratt commented Apr 23, 2019

vext01 commented Apr 23, 2019

ltratt commented Apr 23, 2019

vext01 commented Apr 23, 2019

bors bot commented Apr 23, 2019

vext01 commented Apr 23, 2019

vext01 commented Apr 23, 2019

vext01 commented Apr 23, 2019

vext01 commented Apr 24, 2019

vext01 commented Apr 24, 2019

vext01 commented Apr 24, 2019

bors bot commented Apr 24, 2019

vext01 commented Apr 24, 2019

ltratt commented Apr 24, 2019

vext01 commented Apr 24, 2019

vext01 commented Apr 27, 2019

bors bot commented Apr 29, 2019

vext01 commented Apr 29, 2019

bors bot commented Apr 29, 2019

vext01 commented Apr 29, 2019

bors bot commented Apr 29, 2019

vext01 commented May 1, 2019

jacob-hughes commented May 1, 2019

vext01 commented May 1, 2019

vext01 commented May 1, 2019

jacob-hughes commented May 1, 2019

bors bot commented May 1, 2019

Unify the buildbot testing of the 3 packages #1

Unify the buildbot testing of the 3 packages #1

Conversation

ltratt commented Apr 16, 2019

vext01 commented Apr 17, 2019

vext01 commented Apr 23, 2019

ltratt commented Apr 23, 2019

vext01 commented Apr 23, 2019

ltratt commented Apr 23, 2019

vext01 commented Apr 23, 2019

ltratt commented Apr 23, 2019

vext01 commented Apr 23, 2019

bors bot commented Apr 23, 2019

vext01 commented Apr 23, 2019

vext01 commented Apr 23, 2019

vext01 commented Apr 23, 2019

vext01 commented Apr 24, 2019

vext01 commented Apr 24, 2019

vext01 commented Apr 24, 2019

bors bot commented Apr 24, 2019

try

Build failed

vext01 commented Apr 24, 2019

ltratt commented Apr 24, 2019

vext01 commented Apr 24, 2019

vext01 commented Apr 27, 2019

bors bot commented Apr 29, 2019

try

Build failed

vext01 commented Apr 29, 2019

bors bot commented Apr 29, 2019

try

Build failed

vext01 commented Apr 29, 2019

bors bot commented Apr 29, 2019

try

Build succeeded

vext01 commented May 1, 2019

jacob-hughes commented May 1, 2019

vext01 commented May 1, 2019

vext01 commented May 1, 2019

jacob-hughes commented May 1, 2019

bors bot commented May 1, 2019

Build succeeded