Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

upgrade from 3.3.18 to 3.4.3 panic #11405

Closed
wrfly opened this issue Nov 28, 2019 · 4 comments · Fixed by #11413
Closed

upgrade from 3.3.18 to 3.4.3 panic #11405

wrfly opened this issue Nov 28, 2019 · 4 comments · Fixed by #11413
Labels
type/bug

Comments

@wrfly
Copy link
Contributor

wrfly commented Nov 28, 2019

I have a docker-compose with etcd 3.3.18, when I trying to replace the image version to 3.4.3, cannot auth the user I created under 3.3.18 and the etcd server paniced.

etcd3_1  | panic: runtime error: invalid memory address or nil pointer dereference
etcd3_1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa324ee]
etcd3_1  |
etcd3_1  | goroutine 4591 [running]:
etcd3_1  | go.etcd.io/etcd/auth.(*authStore).CheckPassword(0xc003b3f500, 0xc002dd0128, 0x4, 0xc002dd0140, 0x5, 0x0, 0x0, 0x0)
etcd3_1  |      /tmp/etcd-release-3.4.3/etcd/release/etcd/auth/store.go:347 +0x11e
etcd3_1  | go.etcd.io/etcd/etcdserver.(*EtcdServer).Authenticate(0xc000272b00, 0x1274bc0, 0xc0066b3560, 0xc006d0ee80, 0x102fa80, 0xf85b00, 0x128b680)
etcd3_1  |      /tmp/etcd-release-3.4.3/etcd/release/etcd/etcdserver/v3_server.go:410 +0xda
etcd3_1  | go.etcd.io/etcd/etcdserver/api/v3rpc.(*AuthServer).Authenticate(0xc000282440, 0x1274bc0, 0xc0066b3560, 0xc006d0ee80, 0xc000282440, 0x127ea00, 0xc002ff1d80)
etcd3_1  |      /tmp/etcd-release-3.4.3/etcd/release/etcd/etcdserver/api/v3rpc/auth.go:49 +0x55
etcd3_1  | go.etcd.io/etcd/etcdserver/etcdserverpb._Auth_Authenticate_Handler.func1(0x1274bc0, 0xc0066b3560, 0xffa160, 0xc006d0ee80, 0xcd2ff4, 0xc002ff1d80, 0xc0066b35c0, 0x3)
etcd3_1  |      /tmp/etcd-release-3.4.3/etcd/release/etcd/etcdserver/etcdserverpb/rpc.pb.go:4810 +0x86
etcd3_1  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1(0x1274bc0, 0xc0066b3560, 0xffa160, 0xc006d0ee80, 0x1f, 0xc0070c8a50, 0x58125e, 0xf356e0)
etcd3_1  |      /home/ec2-user/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:31 +0x111
etcd3_1  | github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1(0x1274bc0, 0xc0066b3560, 0xffa160, 0xc006d0ee80, 0xc006d0eec0, 0xc0070c8a00, 0xc0066b34d0, 0xc00213af20, 0x1ac16e8, 0xfb8040)
etcd3_1  |      /home/ec2-user/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:107 +0xad
etcd3_1  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1(0x1274bc0, 0xc0066b3560, 0xffa160, 0xc006d0ee80, 0x1ac2a00, 0xbf70059a8676d5ae, 0x8, 0x1aa2f00)
etcd3_1  |      /home/ec2-user/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 +0x99
etcd3_1  | go.etcd.io/etcd/etcdserver/api/v3rpc.newUnaryInterceptor.func1(0x1274bc0, 0xc0066b3560, 0xffa160, 0xc006d0ee80, 0xc006d0eec0, 0xc0070c8a00, 0xc002060a20, 0x4b5ac6, 0x5ddf9eea, 0xc00676e4cb)
etcd3_1  |      /tmp/etcd-release-3.4.3/etcd/release/etcd/etcdserver/api/v3rpc/interceptor.go:64 +0x10a
etcd3_1  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1(0x1274bc0, 0xc0066b3560, 0xffa160, 0xc006d0ee80, 0x48, 0x48, 0xc0070c8a00, 0x96e0f6)
etcd3_1  |      /home/ec2-user/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34 +0x99
etcd3_1  | go.etcd.io/etcd/etcdserver/api/v3rpc.newLogUnaryInterceptor.func1(0x1274bc0, 0xc0066b3560, 0xffa160, 0xc006d0ee80, 0xc006d0eec0, 0xc0070c8a00, 0x0, 0x0, 0x0, 0x0)
etcd3_1  |      /tmp/etcd-release-3.4.3/etcd/release/etcd/etcdserver/api/v3rpc/interceptor.go:71 +0xba
etcd3_1  | github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0x1274bc0, 0xc0066b3560, 0xffa160, 0xc006d0ee80, 0xc006d0eec0, 0xc006d0eee0, 0xc002060bd0, 0x580fda, 0xfeac00, 0xc0066b3560)
etcd3_1  |      /home/ec2-user/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:39 +0x153
etcd3_1  | go.etcd.io/etcd/etcdserver/etcdserverpb._Auth_Authenticate_Handler(0x102fa80, 0xc000282440, 0x1274bc0, 0xc0066b3560, 0xc0029ec240, 0xc0065fd2c0, 0x1274bc0, 0xc0066b3560, 0xc002dd0130, 0xd)
etcd3_1  |      /tmp/etcd-release-3.4.3/etcd/release/etcd/etcdserver/etcdserverpb/rpc.pb.go:4812 +0x158
etcd3_1  | google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000feb00, 0x1284d00, 0xc002448a80, 0xc001f67900, 0xc0065fd680, 0x1a95330, 0xc001f54680, 0x0, 0x0)
etcd3_1  |      /home/ec2-user/go/pkg/mod/google.golang.org/[email protected]/server.go:995 +0x466
etcd3_1  | google.golang.org/grpc.(*Server).handleStream(0xc0000feb00, 0x1284d00, 0xc002448a80, 0xc001f67900, 0xc001f54680)
etcd3_1  |      /home/ec2-user/go/pkg/mod/google.golang.org/[email protected]/server.go:1275 +0xda6
etcd3_1  | google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc002dd00b0, 0xc0000feb00, 0x1284d00, 0xc002448a80, 0xc001f67900)
etcd3_1  |      /home/ec2-user/go/pkg/mod/google.golang.org/[email protected]/server.go:710 +0x9f
etcd3_1  | created by google.golang.org/grpc.(*Server).serveStreams.func1
etcd3_1  |      /home/ec2-user/go/pkg/mod/google.golang.org/[email protected]/server.go:708 +0xa1
@wrfly wrfly changed the title update from 3.3.18 to 3.4.3 panic upgrade from 3.3.18 to 3.4.3 panic Nov 28, 2019
@wrfly
Copy link
Contributor Author

wrfly commented Nov 28, 2019 

version: '2'
services:

    etcd1:
        image: gcr.io/etcd-development/etcd:v3.4.3
        restart: always
        ports:
            - 23791:2379
            - 23801:2380
        network_mode: etcd_3313
        volumes:
            - etcd1:/data
            - ./cert:/cert
        environment:
            ETCD_NAME: infra1
            ETCD_DEBUG: "true"
            ETCD_INITIAL_ADVERTISE_PEER_URLS: http://172.30.0.1:23801
            ETCD_INITIAL_CLUSTER: infra3=http://172.30.0.1:23803,infra1=http://172.30.0.1:23801,infra2=http://172.30.0.1:23802
            ETCD_INITIAL_CLUSTER_STATE: new
            ETCD_DATA_DIR: /data
            ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
            ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
            ETCD_ADVERTISE_CLIENT_URLS: http://172.30.0.1:23791
            ETCD_AUTH_TOKEN: jwt,pub-key=/cert/jwt_RS256.pub,priv-key=/cert/jwt_RS256,sign-method=RS256



    etcd2:
        image: gcr.io/etcd-development/etcd:v3.4.3
        restart: always
        ports:
            - 23792:2379
            - 23802:2380
        network_mode: etcd_3313
        volumes:
            - etcd2:/data
            - ./cert:/cert
        environment:
            ETCD_NAME: infra2
            ETCD_DEBUG: "true"
            ETCD_INITIAL_ADVERTISE_PEER_URLS: http://172.30.0.1:23802
            ETCD_INITIAL_CLUSTER: infra3=http://172.30.0.1:23803,infra1=http://172.30.0.1:23801,infra2=http://172.30.0.1:23802
            ETCD_INITIAL_CLUSTER_STATE: new
            ETCD_DATA_DIR: /data
            ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
            ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
            ETCD_ADVERTISE_CLIENT_URLS: http://172.30.0.1:23792
            ETCD_AUTH_TOKEN: jwt,pub-key=/cert/jwt_RS256.pub,priv-key=/cert/jwt_RS256,sign-method=RS256


    etcd3:
        image: gcr.io/etcd-development/etcd:v3.4.3
        restart: always
        ports:
            - 23793:2379
            - 23803:2380
        network_mode: etcd_3313
        volumes:
            - etcd3:/data
            - ./cert:/cert
        environment:
            ETCD_NAME: infra3
            ETCD_DEBUG: "true"
            ETCD_INITIAL_ADVERTISE_PEER_URLS: http://172.30.0.1:23803
            ETCD_INITIAL_CLUSTER: infra3=http://172.30.0.1:23803,infra1=http://172.30.0.1:23801,infra2=http://172.30.0.1:23802
            ETCD_INITIAL_CLUSTER_STATE: new
            ETCD_DATA_DIR: /data
            ETCD_LOGGER: zap
            ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
            ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
            ETCD_ADVERTISE_CLIENT_URLS: http://172.30.0.1:23793
            ETCD_AUTH_TOKEN: jwt,pub-key=/cert/jwt_RS256.pub,priv-key=/cert/jwt_RS256,sign-method=RS256

volumes:
    etcd1:
    etcd2:
    etcd3:

@wrfly
Copy link
Contributor Author

wrfly commented Nov 28, 2019 •
edited
Loading

creating user and role with:

etcd-local is an alias for etcdctl --endpoints localhost:23791,localhost:23792,localhost:23793

etcd-local user add root:toor
etcd-local user add user:pass

etcd-local role add rw
etcd-local role grant-permission --prefix=true rw readwrite /
etcd-local user grant-role user rw

@jingyih
Copy link
Contributor

jingyih commented Dec 2, 2019

This is a bug. We should fix in master branch and backport to 3.4.

@YoyinZyc Do you want to give a try?

@YoyinZyc YoyinZyc mentioned this issue Dec 2, 2019
Merged
@davissp14
Copy link
Contributor

davissp14 commented Jan 23, 2020 •
edited
Loading

Any thoughts on when 3.4.4 will be released?

We are hitting #11378 in 3.3, and hitting this issue when we attempt to migrate to 3.4. :) . We are a little stuck until at least one of these are addressed.

cc:// @xiang90

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

auth: fix user.Options nil pointer YoyinZyc/etcd
3 participants
@davissp14 @wrfly @jingyih