Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade raft lib and fix group checksum #3085

Merged
merged 13 commits into from
Mar 3, 2019
Merged

Upgrade raft lib and fix group checksum #3085

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
9d0ae19
Update raft lib to use master
manishrjain Mar 2, 2019
49ee4a6
revert change in assign.go
manishrjain Mar 2, 2019
a7f269f
Stop if compilation fails (#3082)
Mar 2, 2019
ff0ef92
Remove network tests from blockade, so it can run faster.
manishrjain Mar 2, 2019
ec3f837
Revert "Prevent alphas from asking zero to serve tablets during queri…
manishrjain Mar 2, 2019
cda1888
fix up blockade
manishrjain Mar 2, 2019
2c4f6bb
Revert blockade changes
manishrjain Mar 2, 2019
506af50
Merge branch 'master' into mrjn/raft-master
manishrjain Mar 2, 2019
04f206c
Merge branch 'master' into mrjn/raft-master
manishrjain Mar 2, 2019
07cd362
Small changes to test blockade
manishrjain Mar 3, 2019
74589d3
Fix the issue where different Zeros would end up with different check…
manishrjain Mar 3, 2019
acae5ba
Remove debug changes
manishrjain Mar 3, 2019
ffe6de8
Remove unused var
manishrjain Mar 3, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions conn/node.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,14 @@ import (
"sync"
"time"

"github.com/coreos/etcd/raft"
"github.com/coreos/etcd/raft/raftpb"
"github.com/dgraph-io/badger/y"
"github.com/dgraph-io/dgo/protos/api"
"github.com/dgraph-io/dgraph/protos/pb"
"github.com/dgraph-io/dgraph/raftwal"
"github.com/dgraph-io/dgraph/x"
"github.com/golang/glog"
"go.etcd.io/etcd/raft"
"go.etcd.io/etcd/raft/raftpb"
"golang.org/x/net/context"
)

Expand Down Expand Up @@ -366,7 +366,7 @@ func (n *Node) BatchAndSendMessages() {
}

func (n *Node) doSendMessage(to uint64, pool *Pool, data []byte) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()

client := pool.Get()
Expand Down
4 changes: 2 additions & 2 deletions conn/node_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,12 @@ import (
"testing"
"time"

"github.com/coreos/etcd/raft"
"github.com/coreos/etcd/raft/raftpb"
"github.com/dgraph-io/badger"
"github.com/dgraph-io/dgraph/protos/pb"
"github.com/dgraph-io/dgraph/raftwal"
"github.com/stretchr/testify/require"
"go.etcd.io/etcd/raft"
"go.etcd.io/etcd/raft/raftpb"
"golang.org/x/net/context"
)

Expand Down
2 changes: 1 addition & 1 deletion conn/raft_server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,11 @@ import (
"sync"
"time"

"github.com/coreos/etcd/raft/raftpb"
"github.com/dgraph-io/dgo/protos/api"
"github.com/dgraph-io/dgraph/protos/pb"
"github.com/dgraph-io/dgraph/x"
"github.com/golang/glog"
"go.etcd.io/etcd/raft/raftpb"
)

type sendmsg struct {
Expand Down
12 changes: 6 additions & 6 deletions contrib/blockade/blockade.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ containers:
expose:
- 5080
- 6080
command: /gobin/dgraph zero --my=zero1:5080 --replicas 3 --idx 1 --bindall --expose_trace --logtostderr -v=2
command: /gobin/dgraph zero --my=zero1:5080 --replicas 3 --idx 1 --bindall --expose_trace --logtostderr -v=3
volumes:
# Note: Any environment variables must use the ${} syntax.
# ${GOPATH} works, $GOPATH does not.
Expand All @@ -33,7 +33,7 @@ containers:
expose:
- 5082
- 6082
command: /gobin/dgraph zero -o 2 --my=zero2:5082 --replicas 3 --peer=zero1:5080 --idx 2 --bindall --expose_trace --logtostderr -v=2
command: /gobin/dgraph zero -o 2 --my=zero2:5082 --replicas 3 --peer=zero1:5080 --idx 2 --bindall --expose_trace --logtostderr -v=3
volumes:
"${GOPATH}/bin": "/gobin"

Expand All @@ -47,7 +47,7 @@ containers:
expose:
- 5083
- 6083
command: /gobin/dgraph zero -o 3 --my=zero3:5083 --replicas 3 --peer=zero1:5080 --idx 3 --bindall --expose_trace --logtostderr -v=2
command: /gobin/dgraph zero -o 3 --my=zero3:5083 --replicas 3 --peer=zero1:5080 --idx 3 --bindall --expose_trace --logtostderr -v=3
volumes:
"${GOPATH}/bin": "/gobin"

Expand All @@ -61,7 +61,7 @@ containers:
expose:
- 8180
- 9180
command: /gobin/dgraph alpha --my=dg1:7180 --lru_mb=1024 --zero=zero1:5080 -o 100 --expose_trace --trace 1.0 --logtostderr -v=2
command: /gobin/dgraph alpha --my=dg1:7180 --lru_mb=1024 --zero=zero1:5080 -o 100 --expose_trace --trace 1.0 --logtostderr -v=3
volumes:
"${GOPATH}/bin": "/gobin"

Expand All @@ -76,7 +76,7 @@ containers:
- 8182
- 9182
start_delay: 8
command: /gobin/dgraph alpha --my=dg2:7182 --lru_mb=1024 --zero=zero1:5080 -o 102 --expose_trace --trace 1.0 --logtostderr -v=2
command: /gobin/dgraph alpha --my=dg2:7182 --lru_mb=1024 --zero=zero1:5080 -o 102 --expose_trace --trace 1.0 --logtostderr -v=3
volumes:
"${GOPATH}/bin": "/gobin"

Expand All @@ -91,7 +91,7 @@ containers:
- 8183
- 9183
start_delay: 16
command: /gobin/dgraph alpha --my=dg3:7183 --lru_mb=1024 --zero=zero1:5080 -o 103 --expose_trace --trace 1.0 --logtostderr -v=2
command: /gobin/dgraph alpha --my=dg3:7183 --lru_mb=1024 --zero=zero1:5080 -o 103 --expose_trace --trace 1.0 --logtostderr -v=3
volumes:
"${GOPATH}/bin": "/gobin"

Expand Down
16 changes: 9 additions & 7 deletions dgraph/cmd/zero/raft.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,25 +17,26 @@
package zero

import (
"bytes"
"errors"
"fmt"
"log"
"math"
"sort"
"strings"
"sync"
"time"

otrace "go.opencensus.io/trace"

"github.com/coreos/etcd/raft"
"github.com/coreos/etcd/raft/raftpb"
"github.com/dgraph-io/badger/y"
"github.com/dgraph-io/dgraph/conn"
"github.com/dgraph-io/dgraph/protos/pb"
"github.com/dgraph-io/dgraph/x"
farm "github.com/dgryski/go-farm"
"github.com/golang/glog"
"github.com/google/uuid"
"go.etcd.io/etcd/raft"
"go.etcd.io/etcd/raft/raftpb"
"golang.org/x/net/context"
)

Expand Down Expand Up @@ -245,11 +246,12 @@ func (n *node) handleTabletProposal(tablet *pb.Tablet) error {
// served by the group. If the tablets that a group is serving changes, and the Alpha does
// not know about these changes, then the read request must fail.
for _, g := range state.GetGroups() {
var buf bytes.Buffer
for name := range g.GetTablets() {
x.Check2(buf.WriteString(name))
preds := make([]string, 0, len(g.GetTablets()))
for pred := range g.GetTablets() {
preds = append(preds, pred)
}
g.Checksum = farm.Fingerprint64(buf.Bytes())
sort.Strings(preds)
g.Checksum = farm.Fingerprint64([]byte(strings.Join(preds, "")))
}
}()

Expand Down
13 changes: 13 additions & 0 deletions posting/oracle.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import (

"github.com/dgraph-io/dgraph/protos/pb"
"github.com/dgraph-io/dgraph/x"
"github.com/golang/glog"
)

var o *oracle
Expand Down Expand Up @@ -186,6 +187,18 @@ func (o *oracle) WaitForTs(ctx context.Context, startTs uint64) error {
}

func (o *oracle) ProcessDelta(delta *pb.OracleDelta) {
if glog.V(3) {
glog.Infof("ProcessDelta: Max Assigned: %d", delta.MaxAssigned)
glog.Infof("ProcessDelta: Group checksum: %v", delta.GroupChecksums)
for _, txn := range delta.Txns {
if txn.CommitTs == 0 {
glog.Infof("ProcessDelta Aborted: %d", txn.StartTs)
} else {
glog.Infof("ProcessDelta Committed: %d -> %d", txn.StartTs, txn.CommitTs)
}
}
}

o.Lock()
defer o.Unlock()
for _, txn := range delta.Txns {
Expand Down
4 changes: 2 additions & 2 deletions raftwal/storage.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@ import (
"math"
"sync"

"github.com/coreos/etcd/raft"
pb "github.com/coreos/etcd/raft/raftpb"
"github.com/dgraph-io/badger"
"github.com/golang/glog"
"go.etcd.io/etcd/raft"
pb "go.etcd.io/etcd/raft/raftpb"
"golang.org/x/net/trace"

"github.com/dgraph-io/dgraph/x"
Expand Down
4 changes: 2 additions & 2 deletions raftwal/storage_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,10 @@ import (
"reflect"
"testing"

"github.com/coreos/etcd/raft"
pb "github.com/coreos/etcd/raft/raftpb"
"github.com/dgraph-io/badger"
"github.com/stretchr/testify/require"
"go.etcd.io/etcd/raft"
pb "go.etcd.io/etcd/raft/raftpb"
)

func openBadger(dir string) (*badger.DB, error) {
Expand Down
0 vendor/github.com/coreos/etcd/LICENSE → vendor/go.etcd.io/etcd/LICENSE
View file
Open in desktop
File renamed without changes.
0 vendor/github.com/coreos/etcd/NOTICE → vendor/go.etcd.io/etcd/NOTICE
View file
Open in desktop
File renamed without changes.
19 changes: 19 additions & 0 deletions vendor/go.etcd.io/etcd/raft/OWNERS
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 7 additions & 6 deletions vendor/github.com/coreos/etcd/raft/README.md → vendor/go.etcd.io/etcd/raft/README.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 vendor/github.com/coreos/etcd/raft/design.md → vendor/go.etcd.io/etcd/raft/design.md
View file
Open in desktop
File renamed without changes.
8 changes: 4 additions & 4 deletions vendor/github.com/coreos/etcd/raft/doc.go → vendor/go.etcd.io/etcd/raft/doc.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

24 changes: 18 additions & 6 deletions vendor/github.com/coreos/etcd/raft/log.go → vendor/go.etcd.io/etcd/raft/log.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions ...thub.com/coreos/etcd/raft/log_unstable.go → vendor/go.etcd.io/etcd/raft/log_unstable.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion vendor/github.com/coreos/etcd/raft/logger.go → vendor/go.etcd.io/etcd/raft/logger.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading