Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

golangci added to verify target, fixes #746

Merged
merged 10 commits into from
Oct 15, 2021
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .golangci.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# See https://github.com/golangci/golangci-lint/blob/master/.golangci.example.yml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd trust the knative folks to have landed on a set of sensible defaults: https://github.com/knative-sandbox/sample-controller/blob/main/.golangci.yaml. WDTY?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started with those, but they had chosen a few that weren't a good idea. For example, one of them exhorted one to use make to pre-allocate slices. Which, while sometimes a good idea, can be a pretty dangerous change to make without a lot of testing as the semantics do change subtly.

run:
tests: false

timeout: 5m
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can remove the --timeout 5m arg in the Makefile now since we have this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do!


build-tags:
- aws

skip-dirs:
- tools
- website
- hack
- charts
- designs

linters:
enable:
- asciicheck
- errorlint
- gosec
- revive
- stylecheck
- tparallel
- unconvert
- unparam
disable:
- errcheck
- prealloc
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only field that diverges from kn IIUC

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my other comment about why prealloc isn't a good idea.


linters-settings:
gci:
local-prefixes: github.com/awslabs/karpenter
260 changes: 260 additions & 0 deletions go.sum

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions pkg/apis/provisioning/v1alpha4/provisioner_validation.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@ import (
"fmt"
"strings"

"github.com/awslabs/karpenter/pkg/utils/functional"
"github.com/awslabs/karpenter/pkg/utils/ptr"

v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/util/validation"
"knative.dev/pkg/apis"

"github.com/awslabs/karpenter/pkg/utils/functional"
"github.com/awslabs/karpenter/pkg/utils/ptr"
)

func (p *Provisioner) Validate(ctx context.Context) (errs *apis.FieldError) {
Expand Down
4 changes: 2 additions & 2 deletions pkg/cloudprovider/aws/ami.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ func (p *AMIProvider) Get(ctx context.Context, constraints *v1alpha1.Constraints
// Separate instance types by unique queries
amiQueries := map[string][]cloudprovider.InstanceType{}
for _, instanceType := range instanceTypes {
query := p.getSSMQuery(constraints, instanceType, version)
query := p.getSSMQuery(instanceType, version)
amiQueries[query] = append(amiQueries[query], instanceType)
}
// Separate instance types by unique AMIIDs
Expand Down Expand Up @@ -84,7 +84,7 @@ func (p *AMIProvider) getAMIID(ctx context.Context, query string) (string, error
return ami, nil
}

func (p *AMIProvider) getSSMQuery(constraints *v1alpha1.Constraints, instanceType cloudprovider.InstanceType, version string) string {
func (p *AMIProvider) getSSMQuery(instanceType cloudprovider.InstanceType, version string) string {
var amiSuffix string
if !instanceType.NvidiaGPUs().IsZero() || !instanceType.AWSNeurons().IsZero() {
amiSuffix = "-gpu"
Expand Down
39 changes: 39 additions & 0 deletions pkg/cloudprovider/aws/errors.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package aws

import (
"errors"

"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/awslabs/karpenter/pkg/utils/functional"
)

var (
// This is not an exhaustive list, add to it as needed
notFoundErrorCodes = []string{
"InvalidInstanceID.NotFound",
"InvalidLaunchTemplateName.NotFoundException",
}
)

// isNotFound returns true if the err is an AWS error (even if it's
// wrapped) and is a known to mean "not found" (as opposed to a more
// serious or unexpected error)
func isNotFound(err error) bool {
JacobGabrielson marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider making this public

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious why? I kinda got the impression public-ness was a signal that the function would be used outside of the aws package?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd agree on keeping this private. Limits the IDE-oopsy of importing this into another pkg that's not in aws cloud provider

var awsError awserr.Error
if errors.As(err, &awsError) {
return functional.ContainsString(notFoundErrorCodes, awsError.Code())
}
return false
}
4 changes: 2 additions & 2 deletions pkg/cloudprovider/aws/fake/ec2api.go
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,8 @@ func (e *EC2API) DescribeInstancesWithContext(_ context.Context, input *ec2.Desc
return e.DescribeInstancesOutput, nil
}
instances := []*ec2.Instance{}
for _, instanceId := range input.InstanceIds {
instance, _ := e.Instances.Load(*instanceId)
for _, instanceID := range input.InstanceIds {
instance, _ := e.Instances.Load(*instanceID)
instances = append(instances, instance.(*ec2.Instance))
}

Expand Down
19 changes: 7 additions & 12 deletions pkg/cloudprovider/aws/instance.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,22 +22,17 @@ import (

"github.com/avast/retry-go"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/aws/aws-sdk-go/service/ec2/ec2iface"
"github.com/awslabs/karpenter/pkg/apis/provisioning/v1alpha4"
"github.com/awslabs/karpenter/pkg/cloudprovider"
"github.com/awslabs/karpenter/pkg/cloudprovider/aws/apis/v1alpha1"
"knative.dev/pkg/logging"

"go.uber.org/multierr"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/sets"
)
"knative.dev/pkg/logging"

const (
EC2InstanceIDNotFoundErrCode = "InvalidInstanceID.NotFound"
"github.com/awslabs/karpenter/pkg/apis/provisioning/v1alpha4"
"github.com/awslabs/karpenter/pkg/cloudprovider"
"github.com/awslabs/karpenter/pkg/cloudprovider/aws/apis/v1alpha1"
)

type InstanceProvider struct {
Expand Down Expand Up @@ -101,7 +96,7 @@ func (p *InstanceProvider) Terminate(ctx context.Context, node *v1.Node) error {
if _, err = p.ec2api.TerminateInstancesWithContext(ctx, &ec2.TerminateInstancesInput{
InstanceIds: []*string{id},
}); err != nil {
if aerr, ok := err.(awserr.Error); ok && aerr.Code() == EC2InstanceIDNotFoundErrCode {
if isNotFound(err) {
return nil
}
return fmt.Errorf("terminating instance %s, %w", node.Name, err)
Expand Down Expand Up @@ -204,8 +199,8 @@ func (p *InstanceProvider) getOverrides(instanceTypeOptions []cloudprovider.Inst

func (p *InstanceProvider) getInstances(ctx context.Context, ids []*string) ([]*ec2.Instance, error) {
describeInstancesOutput, err := p.ec2api.DescribeInstancesWithContext(ctx, &ec2.DescribeInstancesInput{InstanceIds: ids})
if aerr, ok := err.(awserr.Error); ok && aerr.Code() == EC2InstanceIDNotFoundErrCode {
return nil, aerr
if isNotFound(err) {
return nil, err
}
if err != nil {
return nil, fmt.Errorf("failed to describe ec2 instances, %w", err)
Expand Down
3 changes: 1 addition & 2 deletions pkg/cloudprovider/aws/launchtemplate.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ import (
"strings"

"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/aws/aws-sdk-go/service/ec2/ec2iface"
"github.com/awslabs/karpenter/pkg/cloudprovider"
Expand Down Expand Up @@ -131,7 +130,7 @@ func (p *LaunchTemplateProvider) ensureLaunchTemplate(ctx context.Context, optio
LaunchTemplateNames: []*string{aws.String(name)},
})
// 3. Create LT if one doesn't exist
if aerr, ok := err.(awserr.Error); ok && aerr.Code() == "InvalidLaunchTemplateName.NotFoundException" {
if isNotFound(err) {
launchTemplate, err = p.createLaunchTemplate(ctx, options)
if err != nil {
return nil, fmt.Errorf("creating launch template, %w", err)
Expand Down
2 changes: 1 addition & 1 deletion pkg/controllers/allocation/binpacking/packer.go
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ func (p *packer) Pack(ctx context.Context, schedule *scheduling.Schedule, instan
key, err := hashstructure.Hash(packing, hashstructure.FormatV2, &hashstructure.HashOptions{SlicesAsSets: true})
if err == nil {
if mainPack, ok := packs[key]; ok {
mainPack.NodeQuantity += 1
mainPack.NodeQuantity++
mainPack.Pods = append(mainPack.Pods, packing.Pods...)
logging.FromContext(ctx).Infof("Incremented node count to %d on packing for %d pod(s) with instance type option(s) %v", mainPack.NodeQuantity, flattenedLen(packing.Pods...), instanceTypeNames(mainPack.InstanceTypeOptions))
continue
Expand Down
14 changes: 7 additions & 7 deletions pkg/controllers/allocation/controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,13 @@ import (
"fmt"
"time"

"github.com/awslabs/karpenter/pkg/apis/provisioning/v1alpha4"
"github.com/awslabs/karpenter/pkg/cloudprovider"
"github.com/awslabs/karpenter/pkg/controllers/allocation/binpacking"
"github.com/awslabs/karpenter/pkg/controllers/allocation/scheduling"
"github.com/awslabs/karpenter/pkg/utils/functional"
"go.uber.org/multierr"
"knative.dev/pkg/logging"

v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/types"
corev1 "k8s.io/client-go/kubernetes/typed/core/v1"
"k8s.io/client-go/util/workqueue"
"knative.dev/pkg/logging"
controllerruntime "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/builder"
"sigs.k8s.io/controller-runtime/pkg/client"
Expand All @@ -42,6 +36,12 @@ import (
"sigs.k8s.io/controller-runtime/pkg/predicate"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
"sigs.k8s.io/controller-runtime/pkg/source"

"github.com/awslabs/karpenter/pkg/apis/provisioning/v1alpha4"
"github.com/awslabs/karpenter/pkg/cloudprovider"
"github.com/awslabs/karpenter/pkg/controllers/allocation/binpacking"
"github.com/awslabs/karpenter/pkg/controllers/allocation/scheduling"
"github.com/awslabs/karpenter/pkg/utils/functional"
)

const (
Expand Down
3 changes: 2 additions & 1 deletion pkg/controllers/allocation/filter.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,8 @@ func (f *Filter) GetProvisionablePods(ctx context.Context, provisioner *v1alpha4
}
// 2. Filter pods that aren't provisionable
provisionable := []*v1.Pod{}
for _, p := range pods.Items {
for i := range pods.Items {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious why we'd switch to using i here rather than just ranging over the actual pod? We do the range over the object ignoring the index in other loops.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, I totally agree, and don't like this indexing style.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ellistarn that's right about the for loop gotcha - is there a different style you'd prefer?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bwagner5 the difference is that in other loops, the address of the range object isn't being taken (and there's no closure going on either), so the linter isn't warning about those.

p := pods.Items[i]
if err := f.isProvisionable(&p, provisioner); err != nil {
logging.FromContext(ctx).Debugf("Ignored pod %s/%s when allocating for provisioner %s, %s",
p.Name, p.Namespace, provisioner.Name, err.Error(),
Expand Down
7 changes: 2 additions & 5 deletions pkg/controllers/allocation/scheduling/constraints.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,13 +44,11 @@ func NewConstraints(ctx context.Context, constraints *v1alpha4.Constraints, pod
if err := generateLabels(constraints, pod); err != nil {
return nil, err
}
if err := generateTaints(constraints, pod); err != nil {
return nil, err
}
generateTaints(constraints, pod)
return constraints, nil
}

func generateTaints(constraints *v1alpha4.Constraints, pod *v1.Pod) error {
func generateTaints(constraints *v1alpha4.Constraints, pod *v1.Pod) {
taints := scheduling.Taints(constraints.Taints)
for _, toleration := range pod.Spec.Tolerations {
// Only OpEqual is supported. OpExists does not make sense for
Expand All @@ -77,7 +75,6 @@ func generateTaints(constraints *v1alpha4.Constraints, pod *v1.Pod) error {
}
}
constraints.Taints = taints
return nil
}

func generateLabels(constraints *v1alpha4.Constraints, pod *v1.Pod) error {
Expand Down
9 changes: 3 additions & 6 deletions pkg/controllers/allocation/scheduling/topology.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,7 @@ type Topology struct {
// Inject injects topology rules into pods using supported NodeSelectors
func (t *Topology) Inject(ctx context.Context, constraints *v1alpha4.Constraints, pods []*v1.Pod) error {
// 1. Group pods by equivalent topology spread constraints
topologyGroups, err := t.getTopologyGroups(pods)
if err != nil {
return fmt.Errorf("splitting topology groups, %w", err)
}
topologyGroups := t.getTopologyGroups(pods)
// 2. Compute spread
for _, topologyGroup := range topologyGroups {
if err := t.computeCurrentTopology(ctx, constraints, topologyGroup); err != nil {
Expand All @@ -58,7 +55,7 @@ func (t *Topology) Inject(ctx context.Context, constraints *v1alpha4.Constraints
}

// getTopologyGroups separates pods with equivalent topology rules
func (t *Topology) getTopologyGroups(pods []*v1.Pod) ([]*TopologyGroup, error) {
func (t *Topology) getTopologyGroups(pods []*v1.Pod) []*TopologyGroup {
topologyGroupMap := map[uint64]*TopologyGroup{}
for _, pod := range pods {
for _, constraint := range pod.Spec.TopologySpreadConstraints {
Expand All @@ -75,7 +72,7 @@ func (t *Topology) getTopologyGroups(pods []*v1.Pod) ([]*TopologyGroup, error) {
for _, topologyGroup := range topologyGroupMap {
topologyGroups = append(topologyGroups, topologyGroup)
}
return topologyGroups, nil
return topologyGroups
}

func (t *Topology) computeCurrentTopology(ctx context.Context, constraints *v1alpha4.Constraints, topologyGroup *TopologyGroup) error {
Expand Down
6 changes: 3 additions & 3 deletions pkg/controllers/node/controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,6 @@ import (
"context"
"fmt"

"github.com/awslabs/karpenter/pkg/apis/provisioning/v1alpha4"
"github.com/awslabs/karpenter/pkg/utils/result"

"go.uber.org/multierr"
v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/equality"
Expand All @@ -34,6 +31,9 @@ import (
"sigs.k8s.io/controller-runtime/pkg/manager"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
"sigs.k8s.io/controller-runtime/pkg/source"

"github.com/awslabs/karpenter/pkg/apis/provisioning/v1alpha4"
"github.com/awslabs/karpenter/pkg/utils/result"
)

// NewController constructs a controller instance
Expand Down
3 changes: 2 additions & 1 deletion pkg/controllers/node/emptiness.go
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,8 @@ func (r *Emptiness) isEmpty(ctx context.Context, n *v1.Node) (bool, error) {
if err := r.kubeClient.List(ctx, pods, client.MatchingFields{"spec.nodeName": n.Name}); err != nil {
return false, fmt.Errorf("listing pods for node %s, %w", n.Name, err)
}
for _, p := range pods.Items {
for i := range pods.Items {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm seeing this pattern in a lot of places now. Is this a lint recommendation? I'm not sure I agree that this is better. Is this supposed to protect against the pointer location changing out from under the local loop var?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only when it's closed over or its address is taken.

p := pods.Items[i]
if pod.HasFailed(&p) {
continue
}
Expand Down
7 changes: 4 additions & 3 deletions pkg/controllers/termination/controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,6 @@ import (
"fmt"
"time"

provisioning "github.com/awslabs/karpenter/pkg/apis/provisioning/v1alpha4"
"github.com/awslabs/karpenter/pkg/cloudprovider"
"github.com/awslabs/karpenter/pkg/utils/functional"
"golang.org/x/time/rate"
"knative.dev/pkg/logging"

Expand All @@ -34,6 +31,10 @@ import (
"sigs.k8s.io/controller-runtime/pkg/controller"
"sigs.k8s.io/controller-runtime/pkg/manager"
"sigs.k8s.io/controller-runtime/pkg/reconcile"

provisioning "github.com/awslabs/karpenter/pkg/apis/provisioning/v1alpha4"
"github.com/awslabs/karpenter/pkg/cloudprovider"
"github.com/awslabs/karpenter/pkg/utils/functional"
)

// Controller for the resource
Expand Down
13 changes: 6 additions & 7 deletions pkg/controllers/termination/terminate.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,18 @@ import (
"context"
"fmt"

v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
corev1 "k8s.io/client-go/kubernetes/typed/core/v1"
"knative.dev/pkg/logging"
"sigs.k8s.io/controller-runtime/pkg/client"

provisioning "github.com/awslabs/karpenter/pkg/apis/provisioning/v1alpha4"
"github.com/awslabs/karpenter/pkg/cloudprovider"
"github.com/awslabs/karpenter/pkg/scheduling"
"github.com/awslabs/karpenter/pkg/utils/functional"
"github.com/awslabs/karpenter/pkg/utils/injectabletime"
"github.com/awslabs/karpenter/pkg/utils/ptr"

"knative.dev/pkg/logging"

v1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
corev1 "k8s.io/client-go/kubernetes/typed/core/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
)

type Terminator struct {
Expand Down
3 changes: 2 additions & 1 deletion pkg/scheduling/taints.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ func (ts Taints) Has(taint v1.Taint) bool {

// Tolerates returns true if the pod tolerates all taints
func (ts Taints) Tolerates(pod *v1.Pod) (errs error) {
for _, taint := range ts {
for i := range ts {
taint := ts[i]
tolerates := false
for _, t := range pod.Spec.Tolerations {
tolerates = tolerates || t.ToleratesTaint(&taint)
Expand Down
Loading