mypy_primer: add to CI (#9686)

This will run mypy_primer on mypy PRs. Changes on the open source
corpus are reported as comments on the PR.

We integrated this into typeshed CI; you can see examples here:
python/typeshed#3183
python/typeshed#4734

This might be a little slow. On typeshed this runs in 10 minutes, but
it's using a mypyc compiled wheel. It looks like it takes three minutes
to compile a MYPYC_OPT_LEVEL=0 wheel in our CI currently (for a ~2x
speedup), so that's probably worth it.

Co-authored-by: hauntsaninja <>

.github/workflows/mypy_primer.yml

@@ -0,0 +1,62 @@
+name: Run mypy_primer
+
+on:
+  # Only run on PR, since we diff against master
+  # pull_request_target gives us access to a write token
+  pull_request_target:
+    paths-ignore:
+    - 'docs/**'
+    - '**/*.rst'
+    - '**/*.md'
+    - 'mypyc/**'
+
+jobs:
+  mypy_primer:
+    name: Run mypy_primer
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          path: mypy_to_test
+          fetch-depth: 0
+          # pull_request_target checks out the PR base branch by default
+          ref: refs/pull/${{ github.event.pull_request.number }}/merge
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.8
+      - name: Install dependencies
+        run: |
+          python -m pip install -U pip
+          pip install git+https://github.com/hauntsaninja/mypy_primer.git
+      - name: Run mypy_primer
+        shell: bash
+        run: |
+          cd mypy_to_test
+          echo "new commit"
+          COMMIT=$(git rev-parse HEAD)
+          git rev-list --format=%s --max-count=1 $COMMIT
+          git checkout -b upstream_master origin/master
+          echo "base commit"
+          git rev-list --format=%s --max-count=1 upstream_master
+          echo ''
+          cd ..
+          ( mypy_primer --repo mypy_to_test --new $COMMIT --old upstream_master --mypyc-compile-level 0 -o concise | tee diff.txt ) || [ $? -eq 1 ]
+      - name: Post comment
+        uses: actions/github-script@v3
+        with:
+          github-token: ${{secrets.GITHUB_TOKEN}}
+          script: |
+            const fs = require('fs').promises;
+            try {
+              data = await fs.readFile('diff.txt', 'utf-8')
+              if (data.trim()) {
+                await github.issues.createComment({
+                  issue_number: context.issue.number,
+                  owner: context.repo.owner,
+                  repo: context.repo.repo,
+                  body: 'Diff from [mypy_primer](https://github.com/hauntsaninja/mypy_primer), showing the effect of this PR on open source code:\n```diff\n' + data + '```'
+                })
+              }
+            } catch (error) {
+              console.log(error)
+            }