-
Notifications
You must be signed in to change notification settings - Fork 266
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Add continuous integration models (Projects and Builds) #1612
Conversation
c43d6c2
to
b7a5a2f
Compare
@eddycharly: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This feels like something that would be a good candidate for a TEP @vdemeester @bobcatfish @wlynch |
Agreed! This sounds similar to our ideas around generators. See https://github.com/tektoncd/experimental/tree/master/generators for some work we're doing in this space. |
Big +1 on this being a TEP and thanks for sharing @wlynch! As I said on the working group call today, it seems it's a sought-after feature in some way, shape, or form, and having the abstraction makes a late of sense. @eddycharly what do you think, perhaps we'd extend the dashboard to show the abstractions that do became part of the TEP, or your projects/builds become yours? Full disclosure, I used to work on a Kubernetes offering called Microclimate and we used the exact same terminology (we actually had CRDs), and that worked really well... unfortunately we didn't open-source that, but if you would like to know more please let me know and I'm sure we can make something awesome for the community to benefit from. |
One vague concern I have about this is that Tekton has been this far rather non-opinionated about how you setup you CI/CD pipelines. I'm a bit wary of taking a direction away from that. |
Good point, that's why the webhooks extension was an experimental thing really and was quite opinionated and limited in the type of pipelines you run - I think that may be the biggest thinking point actually... |
Thanks for your feedback. I’ve eared very little about the generators project but if I understand correctly it’s supposed to simplify writing pipelines. If so it would be complementary and allow to write CI spec in a simplified syntax (something like travis ?). Regarding the TEP suggestion, why not, it doesn’t have to be bound to the dashboard. In the end it’s just two new resource types and controllers. It was easy to start in the dashboard repo but it can be extracted very easily. What would be nice would be to be able to create Builds directly using a trigger template instead of going through a task run just for that. Any plan to support arbitrary resources in trigger templates ? |
I like the idea of keeping the core dashboard non-opinionated. It was originally envisioned as a Tekton analog of the Kubernetes dashboard, presenting a view of the live state of Tekton resources in the cluster. Developing a UI for a full CI/CD solution has a different set of design considerations as it needs to solve some very different problems. The current prototype should work well as a dashboard extension. A topic we've discussed a few times is making extensions more powerful, for example allowing them not only to add new pages to the dashboard, but also to allow them to integrate into / customise core pages. This wouldn't be needed for the current prototype but could open up some interesting possibilities. |
This has been raised a few times, see for example tektoncd/triggers#482 and some of the other issues linked to it. |
My dream would be to have a solution that captures both: or at least if the abstractions are more opinionated, at least have them be optional. I think getting a proposal that we're all happy with will probably take a while and there will be a lot to consider but I'd love to see the discussion get started!! If dashboard wants to go ahead with something in the meantime I could see that working too, as long as it's something that could be changed later if we create tekton-wide abstractions. @a-roberts @eddycharly do you think someone working on this feature would be able to give us an overview in the API working group one day soon? |
Not only pipelines, but all related resources (e.g. Triggers). The reason why we wanted to do this is for the same motivation as your reason for the
There's no technical limitation for creating other resources as a part of trigger templates. My understanding is that we restricted it because we were worried about the potential for abuse for creating arbitrary k8s resources. You can allow new clientsets using: https://github.com/tektoncd/triggers/tree/master/pkg/client/dynamic/clientset (see https://github.com/tektoncd/triggers/blob/d51f0f43612b394e2327bdabeeee020457b62f2d/cmd/eventlistenersink/main.go#L61 for how this is used). Another option is reconsidering the GVK restriction altogether and just trust EventListener service account RBAC roles to be a sufficient perimeter. /cc @gabemontero
This sounds like config as code! This is a problem we want to solve for Tasks/Pipelines in general.
Instead of defining a new wrapper type, my general preference would be to stick with PipelineRuns and solve the underlying root problem of config as code pipelines (since we know this is something we want to support for Pipeline/TaskRuns in general). This may benefit from Pipeline in Pipelines via Custom Tasks - https://github.com/tektoncd/community/blob/master/teps/0002-custom-tasks.md
+1. One thing we wanted to be careful about in Generators was to not make things GitHub specific (or even within GitHub, how can I configure OAuth vs GitHub Apps?) |
@bobcatfish i will try to join on next monday. no guarantee though as i'll be on vacation. |
Exactly, in-repo config, that was one of the most important point to address for me. Build strategy can be different from one branch to another and it should be the devs responsibility.
Same here, a
I tend to disagree on this one, having a wrapping type allows to add features like decorating pull requests, notifications, etc... easily. Those things do not really need to work with raw pipelineruns. It also allows easy aggregations/filtering (like filtering builds for a particular repo/branch).
I completely agree and understand this position. IMO it's not a vague concern but an essential point though. |
You might be interested in tektoncd/community#147! I recently proposed this to define how we can better annotate this type of data in Task/PipelineRuns. |
Interesting ! It looks like Task/PipelineRuns are getting responsible of a lot of things though. I like the idea of a more high level resource used to federate smaller resources that address a single responsibility. |
@eddycharly no hurry at all! vacations are important, plz don't feel any pressure to join, happy to wait till you're back :) |
@bobcatfish no pressure at all, I’ll join if I can don’t worry ;) |
Apologies for the delay (did not get to this before stepping away from work a little over a week ago). If we were to fall back / depend on service account RBAC, in addition to the EventListener's, I would also suggest the current Trigger SA association, as well as whatever falls out from the Multi Tenant EventListener and new Trigger CRD should also be considered as the authenticated party in object creation. Generally speaking, providing options wrt granularity (i.e. different controllers or different api server clients) on who can create whatever objects would be preferable.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
@eddycharly: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
Closing as discussed on recent Dashboard WG calls since this feature doesn't make sense as part of the core Dashboard application. This is still very useful and could be implemented as an extension or combined with the generators experimental project or other related efforts. |
Changes
This PR adds continuous integration models to the dashboard.
It does so by adding two new CRDs
Projects
andBuilds
.Those two new resources help bootstrapping the triggers and pipelines resources to setup continuous integration with pipeline definitions stored in the target repository.
A
Project
will setup:EventListener
to receive github webhooksTriggerBinding
to bind properties from the incoming webhookTriggerTemplate
to create aTaskRun
that will clone the repository read the pipeline definition file and instantiate aBuild
Ingress
pointing the event listener serviceA
Build
will setup:PipelineRun
implementing the pipeline definition that came from the repoAll those resources are correctly meshed together through owner references, they are reconciled and have statuses that aggregate the statuses from the resources they are responsible for.
I will make a demo on next WG meeting, i opened the PR if someone wants to give a look or try it out before Monday.
/cc @AlanGreene @a-roberts
A few screenshots:
Projects list
Builds list
Create project form
Submitter Checklist
These are the criteria that every PR should meet, please check them off as you
review them:
See the contribution guide
for more details.