Campaigns: 3.18 Tracking Issue

[mrnugget]

Plan

The main goals for this iteration are:

• Add support for Gitlab to campaigns (Adam)
• Implement the campaigns workflow as documented in update docs for new campaign flow #10921 (Erik, Thorsten)

Both tie back to our OKR-based overarching goal to release campaigns as a non-feature-flagged, non-beta feature and get N customers using it to create campaigns.

Implementing Gitlab support is the result of multiple customers asking for it. Implementing the new workflow is the result of usertesting with Sourcegraph colleagues.

The first one should increase the number of customers trying out campaigns, the second one should make it easier for all customers to get started with campaigns and actually create campaigns.

Availability

Period is from June 22nd to July 17th (20 days). Please write the days you won't be working and the number of working days for the period.

• @mrnugget: 17d (Not working 6-8th July )
• @eseliger: 20d
• @LawnGnome: 19d (Canada Day 1st July)

Workload

@LawnGnome: 13.00d

• Cache diffstat for external changesets #11075 0.5d
• Clearer error message for improper JSON escape sequence in src actions scope-query #10541 0.5d 🐛
  • Update jsonx dependency #233 🐛
• Support GitLab in campaigns #11586 12d
• campaigns: spurious Percy failures caused by the landing page video #12170 🐛
• campaigns: GitLab webhook support #12171 2d
• CODEOWNERS: campaigns should own docs #12218

@eseliger: 4.50d

• Frontend: New campaigns flow #10986 8d
• YAML support for action definitions #11589 1d
• Use paginated search in src-cli to fetch repositories for campaigns #11587 1d 🕵️
• Don't put campaigns marketing page on dotcom behind auth #11501 0.5d
• Allow previewing of changesets for unsupported code hosts #11193 1.5d
• Remove hard distinction between manual and patchset campaigns #10984 0.5d

@mrnugget

• Backend: New campaign flow #10985 12d
• Replace "automation" feature flag with "campaigns" toggle and remove configuration flags #10713 1d 🛠️
• Creating larger campaigns using src-cli fails with 413 #9202 0.5d 🐛
• Make the verbose-mode ("-v") UI the default UI for src action exec #224
• Enable Campaigns by default (with write-access only for admins) #11621
• Implement missing bits in CampaignSpec/ChangesetSpec resolvers #12247
• Delete expired CampaignSpecs and ChangesetSpecs #12210
• Implement ChangesetSpec resolver and validation of specs #12092
• New campaigns flow: implement moveCampaign mutation #12049
• Add a basic implementation of the applyCampaign mutation #11934
• Add createCampaignSpec/createChangesetSpec mutations #11813
• Add GraphQL resolvers for CampaignSpec/ChangesetSpec #11733
• Add CampaignSpecs & ChangesetSpecs tables, type definitions and campaigns.Store support #11711
• Check that $VERSION is in MAJOR.MINOR.PATCH format in release.sh #227

@ryanslade: 2.00d

• src actions exec can cause frontend to be OOMKilled #10540 2d 🐛

Legend

• 👩 Customer issue
• 🐛 Bug
• 🧶 Technical debt
• 🛠️ Roadmap
• 🕵️ Spike
• 🔒 Security issue
• Pull Request

[LawnGnome]

Weekly update: 2020-06-15 to 2020-06-19

Last week, I worked primarily on finishing out the work to cache diffstats for external changesets. While I didn't quite get this merged by the end of the week, it feels like it was extremely close, and should be merged early this week.

Additionally, I worked on surfacing the alerts we get from Sourcegraph when a search fails in src-cli, complete with shiny colours. Again, this didn't quite get merged due to time zones, but should be merged on Monday morning.

[mrnugget]

Weekly update: 2020-06-15 to 2020-06-19

Besides multiple interviews and grading coding exercises, my main focus was providing feedback on https://github.com/sourcegraph/sourcegraph/pull/10921 and planning 3.18.

This week I'll try to implement the API schema proposed in #10921 to try to see how the pieces fit together — in sourcegraph/sourcegraph and src-cli.

[mrnugget]

Weekly Team Update: 2020-06-15 to 2020-06-19

cc @nicksnyder

What work did the team do last week?

Finishing up the work for 3.17 and planning 3.18. The main goals for campaigns in 3.18 are:

• Add support for Gitlab to campaigns (Adam)
• Implement the campaigns workflow as documented in update docs for new campaign flow #10921 (Erik, Thorsten)

How does that work connect back to the team’s OKRs and roadmap?

Our main goal is to release campaigns as a non-feature-flagged, non-beta feature and get N customers using it to create campaigns.

Implementing Gitlab support is the result of multiple customers asking for it. Implementing the new workflow is the result of usertesting with Sourcegraph colleagues.

The first one should increase the number of customers trying out campaigns, the second one should make it easier for all customers to get started with campaigns and actually create campaigns.

For each OKR and roadmap item, is the team on track? The answer should be of the form:

I think we are at a slight risk of not reaching this goal in Q2 because we're taking half a step back to get the campaigns workflow right and then take two steps forward. The result is that we're at risk of reaching the goal this quarter, but our chances to reach it increase, because we're making it easier to use and create campaigns.

What did the team work on last week that wasn’t originally planned? Why?

Nothing that I'm aware of.

What pain is the team experiencing?

• Tech debt: We now have multiple "store" implementations in our codebase: db, campaigns.Store, repos.Store, codeintel.db. They're all following the same pattern, except db, but db is the only one with authz/authn support. The result is that in our campaigns package, we now use repos.Store, campaigns.Store and db.Repos. Confusing. I've already talked to Eric Fritz and Tomás about this and I want to do a mini-spike to see how much work it would be to merge all stores into one.
• Quinn in his role as PM is overloaded. But we're all aware of this, which is why we're looking for a dedicated campaigns PM, just wanted to mention this for completeness and because it came up in the sync.

[eseliger]

Weekly update: 2020-06-15 to 2020-06-19

Last week, I spent most time on interviews, planning 3.18 and investigating a performance issue that leads frontend to go OO.

This week I am going to finish up yaml support and clear out the small tasks I have assigned for this iteration, so I have plenty of room once the frontend work for the new campaigns flow can be started.

[mrnugget]

Weekly update: 2020-06-22 to 2020-06-28

Last week, after removing then-unneeded code, I started implementing the API and database layer for https://github.com/sourcegraph/sourcegraph/pull/10921. That already yielded a lot of feedback and insight into how we can structure and model certain things. Since the new campaigns workflow replaces a lot of code, @eseliger and I decided to use a single PR as a merge-target for all our new-workflow related PRs.

I've also dug a little bit into our dbtesting and other DB-testing-related packages to see if we can speed things up and/or unify them. Besides that: ton of interviews!

This week looks like it'll also have its fair share of interviews, but besides those my goal is to finish the database layer required for the new campaigns workflow. I also have some thoughts on how to build the reconciler/controller that turns changeset/campaign specs into "actual" things, and I might try to prototype some of that, although I won't finish it this week and there's still some question marks around how things will work in detail that we need to figure out first.

[eseliger]

Weekly update: 2020-06-22 to 2020-06-28

Last week

besides a couple of interviews, I finished yaml support in src cli, worked on frontend testing toward the frontend testing OKR by adding better serializers for enzyme and migrating the campaigns codebase to utilize those in this PR and cleaned-up and tested on edge-cases with the removal of the manual campaigns distinction here. I also started working on a prototype using search pagination in src-cli, and wrote my writeup of that today.

This week

I am going to work on enabling collecting changesets for unsupported codehosts and will look into the new GraphQL API and UI, and see what tasks we can derive from it, so I can make my placeholder issue #10986 more precise.

[LawnGnome]

Weekly update: 2020-06-22 to 2020-06-26

Last week was a good merging week for me: the PRs to cache external changeset diffstats and surfacing search alerts in src-cli both landed.

With those out of the way, I spent most of my week working on implementing GitLab support. The ticket with the scope of this is in #11586, and the draft PR is #11757. I made good progress, and the draft PR is operational (if poorly tested right now) for everything except changeset events and webhooks. Between those and improving the test coverage, I expect this will hit ready for review status late this week, assuming there are no unexpected icebergs.

I'm also trying to put a little effort in each week to improving the src-cli experience for campaign users: last week I implemented support for rendering pretty JSON errors. While campaigns will be moving to YAML soon, this also improves error messages for external service and site configuration updates, and provides a template for how I'd like to render YAML errors when the time comes as well.

This week, my focus will be on finishing the GitLab work and merging the src-cli JSON error change. If all goes well, this should allow me to pick up some more work this iteration!

[mrnugget]

Weekly Update 29 June 2020

cc @nicksnyder

What work did the team do last week?

We started the iteration by finishing some smaller tasks and then making progress on the big items in our tracking issue:

• Erik investigated the feasibility of paginated for use in campaigns, added YAML support to src-cli and added support for "unsupported" codehosts to campaigns.
• Adam improved the user-friendliness of src-cli error messages by giving precise information in JSON parse errors (by fixing issues and extending our jsonx library!) and started working on adding Gitlab support to campaigns.
• I (Thorsten) fixed a minor issue in src-cli, added more things to RFC179, and then started implementing the API and database layer for the new campaigns workflow.

How does that work connect back to the team’s OKRs and roadmap?

See the previous weekly update.

For each OKR and roadmap item, is the team on track? The answer should be of the form:

No big changes between last week's update and this one (the previous weekly update) except that Adam's thinks there's a chance that Gitlab support might be faster than we thought it would be and that he could start helping out with the new campaigns workflow earlier than we thought.

What did the team work on last week that wasn’t originally planned? Why?

Nothing major. Adam had to fix multiple things in jsonx to fix things in src-cli, but since the ticket itself was planned, I'm not sure this would fall under unplanned things. I invested some time to try to make our dbtesting package faster, still haven't finished it, but want to do a timeboxed hack session with Keegan on this.

What pain is the team experiencing?

• After the issue of tech debt was raised in Slack and last week's company update, I think we're all a bit more conscious of tech debt that has accumulated and try to fix it. That's not a real "pain", but of course no tech debt is better than tech debt.
• Same as last week: we're in need of a dedicated PM, because Quinn has too much on his plate. We're not at the point yet where we are blocked by lack of input from product side, but there are some open questions accumulating.
• I (Thorsten) am personally not 100% happy with having to keep a long-living branch (see my update here for context). Rebases are painful, yes, but on top of that: when I find something that I want to clean up or could improve (tech debt) the incentives are now aligned against me, because large sweeping changes would make the rebases and future merge harder and harder. Reviews would also be harder. And ever since Beyang's tweet about code reviews being bottlenecks I'm actively thinking about this.

[nicksnyder]

@mrnugget Thanks for the update! I realize that some of the original questions here were repetitive so I updated the docs in the handbook: https://about.sourcegraph.com/handbook/engineering/tracking_issues#progress-updates. So for future updates feel free to to a more informal prose based update based on what you think it is important to communicate about the team's progress.

I also updated the tracking issue template to inline the info that shouldn't change from week to week (what the team is working on and why). Code intel tracking issue is an example of what I am looking for: https://github.com/sourcegraph/sourcegraph/issues/11412

[mrnugget]

I realize that some of the original questions here were repetitive so I updated the docs in the handbook: https://about.sourcegraph.com/handbook/engineering/tracking_issues#progress-updates. So for future updates feel free to to a more informal prose based update based on what you think it is important to communicate about the team's progress.

Thank you! That makes sense. I still like the original questions you had (I now realised that they didn't outlive the original PR) and combined with the new ones I'll use those as prompts to start thinking about what we're doing, why and how it all fits together.

I also updated the tracking issue template to inline the info that shouldn't change from week to week (what the team is working on and why). Code intel tracking issue is an example of what I am looking for: #11412

Updated this tracking issue here to include a plan at the top too.

[mrnugget]

Weekly update: 2020-06-29 to 2020-07-03

I'm on a mini-vacation next week from Monday to Wednesday, so I'm posting this update today instead of on Monday.

This week I made progress on implementing the new campaigns workflow — no unforeseen problems, no huge pains, just slowly and steadily knocking out all the data layer and GraphQL resolver work. I updated the WIP PR's description with a huge braindump that contains everything I did and everything we need to do: https://github.com/sourcegraph/sourcegraph/pull/11675
Later today I'm giving @LawnGnome and @eseliger and overview of the current state so that they can, if possible, continue with that. The most important thing to solve is answering all the concept/API-schema related questions that are still open in @sqs' https://github.com/sourcegraph/sourcegraph/pull/10921 because while I can make further progress on the backend implementation, we're soon going to reach a point where the missing details will matter.

Next week, I'll continue doing what I did this week, except for only 2 days, since I'm out for 3 :)

[eseliger]

Weekly update: 2020-06-29 to 2020-07-03

Last week

I worked on making changesets for unsupported codehosts work in Sourcegraph and the src-cli. And finished up the spike work on triaging the paginated search API for campaigns usage. I also tinkered around with the new API and raised some questions which will be up for discussion in todays sync (write-up coming this week).

This week

I am going to work on clearing out the remaining questions for the campaigns workflow, and start implementation work where possible.

[LawnGnome]

Weekly update: 2020-06-29 to 2020-07-03

Around Canada Day, I focused mostly on GitLab last week. I discovered early in the week that the approvals API that I had been intending to use for review changeset events was insufficient to provide the historical information that we need to synthesise changeset events. Alas, the only fallback option is to parse strings from system "notes" (comments), which actually works, but feels fairly brittle.

The GitLab REST API also provided some smaller challenges in terms of retrieving labels, and its data model means that we're going to have to burn a decent amount of rate limiting quota to sync changesets. In the longer term, we can fix this by using GitLab's new GraphQL API, but it will likely be some time before our minimum supported GitLab version allows this.

This week, I hope to have the GitLab support ready for review by mid-week, and then hope to work more on the changeset-focused internal data model that we've been discussing within the team.

[eseliger]

Weekly Update July 6 2020

cc @nicksnyder

After Adam realized the work on GitLab support is probably going to be less than we estimated it to be, he chimed in last week on discussions around the new campaigns workflow.
We still have a lot of questions and underspecified requirements to fill in this list, which I am going to work on for the first half of the week. On Thursday, we are having a team sync to make a call whether we want to take a slightly different path, which we think will make the state of resources in the backend much more explicit. It's a tradeoff for complexity and getting campaigns right, and we decided to favor getting it right in todays sync. Hence, we don't have a definite release date for campaigns right now. We anticipated the August iteration in the past weeks, but decided it will be more valuable to get it right on day one of the GA version of campaigns, rather than rushing it.
Given that, we don't think that we will merge our tracking PR for this iteration. We still have Gitlab support and improvements to changeset tracking for unsupported codehosts on our plate for this iteration, so we still have some very valuable additions to the campaigns product anyways. In my opinion, the biggest unknown is time right now. It's likely that we are not merging in the new workflow this iteration, but rather incubate it for a little longer. We don't have a complete list of tasks to be done for the campaigns workflow revamp, and the scope of it changes daily while thinking about the use-cases and API design for it, so once we have that in place, work should return to a much more structured approach, getting rid of the big issues that have large estimates and no concrete tasks defined.

As the past weeks, we are still in need of a PM, because Quinn has too much on his plate and shouldn't need to be concerned of all the ongoings in campaigns.

---

@nicksnyder As this is my first time writing a team report, please feel free to ping me when you are missing context or I left questions unanswered, so I can follow-up on those.

[LawnGnome]

Weekly update: 2020-07-06 to 2020-07-10

Last week, I worked on getting the GitLab support ready for review. Unsurprisingly for anyone accustomed to software development, this has taken a bit longer than I optimistically thought. This was partly because I was dissatisfied with the test coverage in my prototype PR, so I spent more time improving that than expected.

At this stage, all features have been implemented except for webhook support. After discussing this in the campaign sync this morning, the plan for this week is to merge without webhook support before the release cut tomorrow, and then add webhook support as a cherry pick for 3.18 if possible.

We also spent time last week refining the design of the campaign implementation. I vaguely sketched out my take on the state diagram, and then Erik turned it into something that was actually useful. Teamwork!

I'm also talking to Laureen about making a (very) short video for the 3.18 release blog showing off GitLab support, so I expect that to take a bit of time towards the end of the week.

[eseliger]

Weekly update: 2020-07-06 to 2020-07-10

Last week

I worked on the API design for status and diff on campaigns and dug a bit into storybooks, where I started a short spike into using Chromatic (#12054) to share for fast feedback and added stories for a dependent component. Also found coverage reports annoyingly wrong, so dug into deploying a fix for that.

This week

I just finished #11528 up. Tomorrow I will be working on the final sketch out for the delta API. Afterwards I will either work on porting over Quinns UI components into our WIP branch or help out on the new workflow backend tasks. In addition, I am planning to spend some time on Adam's Gitlab PR, doing some extensive testing and reviewing there.

[daxmc99]

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.18 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly.
When in doubt, reach out!

Thank you

[mrnugget]

Weekly update: 2020-07-06 to 2020-07-10

3 days of vacation last week and two days of making as much progress as possible on implementing the new workflow in https://github.com/sourcegraph/sourcegraph/pull/11675. I implemented the last CRUD-heavy mutation that was missing, moveCampaign, and removed a lot of simple TODOs in resolvers.

I also discussed the two missing bits of API design with Erik last week and in today's sync: the "status API" and "preview/delta API" (see my comment here). The big questions marks around these two missing pieces are slowly turning into blockers, so in the sync we identified an owner for coming up with design (@eseliger) and a timeline (Erik and I will sync on that design on Wednesday).

That should remove the last unknown unknowns and help us better estimate how much we think we'll be able to do this iteration and what we need to defer to the next one. And if we need to defer something to next iteration (which I'm 90% sure we'll need to do), we'd be in a better position to plan and divide up the work.

Since we took a slight bet when deciding to implement the new workflow even though it wasn't 100% clear how all the pieces would fit together I'm very much looking forward to that.

[mrnugget]

Weekly update: 2020-07-13 to 2020-07-17

After sitting down and designing the rest of the GraphQL API that's necessary to implement the new campaigns workflow, we've made a lot of progress on a conceptual level. I'm now finally able to see all the things that are left to implement. It seems the unknown unknowns have been removed.

A big decision that helped with that was Quinn saying that we don't need to implement the "delta" API to release the new workflow. That not only cut the scope of the actual implementation work, but also removed a lot of questions around how the pieces in the API will fit together. We decided that we'll revisit the delta API once we have the new workflow working, merged and released.

After that I personally made some implementation progress by implementing a lot of the missing resolvers.

What's left to do: the new explicit changeset state machine, implementing the reconciler, supporting the new workflow in src-cli, and the frontend.

This week is planning week, which shouldn't yield a lot of surprises: most likely our goal will stay "Release the new campaigns workflow", except that all three of us will be able to help with that. We'll split up the work in a planning meeting this week.

Besides that I want to concentrate on the two big items: changeset state machine & reconciler. These go hand in hand and I anticipate a lot of thinking & writing code to happen this week.

[eseliger]

Weekly update: 2020-07-13 to 2020-07-17

Last week

I spent some time testing Gitlab support thoroughly. Afterwards, I spent most of my time on getting the API design finalized and then implemented placeholder resolvers for the API we came up with, so that the build is green again and we can work in parallel on frontend and backend. Since then, I started on cleaning up the frontend code from components we don't need anymore and added storybooks for some of the remaining components, so they can be developed on without the backend fully in place yet.

This week

I am going to continue work on the frontend parts of the new workflow. Once I gained a complete picture of the tasks left, I'm going to fill in all the required work items to completion in this issue. Tomorrow, we are going to meet for planning, so we can prioritize and estimate the effort of the remaining work items.

[LawnGnome]

Weekly update: 2020-07-13 to 2020-07-17

Last week, I finalised GitLab support, then moved on to GitLab webhook support. This work has gone pretty well: most of the issues that have been hit have been related to idiosyncrasies in GitLab's webhook API, especially the approval/unapproval flow being different to the normal one we use in syncing (webhook approval events come through as merge request events, but syncing approvals happens via notes) and GitLab using a different date/time format for some webhooks. Nevertheless, the GitLab webhook work is now feature complete.

This week, I'll be finishing the testing of GitLab webhooks, including building a story file for the site admin component, producing GitLab-related collateral for our 3.18 blog post, and then moving on to... well, whatever we decide at tomorrow's iteration planning meeting, but most likely some src-cli work to support the new campaign flow.