Alpha crash while query after dropall

[sailorfeng]

If you suspect this could be a bug, follow the template.

• What version of Dgraph are you using?

1.0.15

• Have you tried reproducing the issue with latest release?

yes

• What is the hardware spec (RAM, OS)?

Distributor ID: Debian
Description: Debian GNU/Linux 9.2 (stretch)
Release: 9.2
Codename: stretch

• Steps to reproduce the issue (command/config used to run Dgraph).

It happened by accident. I've got the panic, but not so sure how to reproduct it.

alpha_1_1_6092b86da97f | I0710 08:30:21.396312       1 index.go:668] Deleting index for attr board_name and tokenizers []
alpha_1_1_6092b86da97f | I0710 08:30:21.396324       1 index.go:686] Rebuilding index for attr board_name and tokenizers [hash trigram]
alpha_1_1_6092b86da97f | I0710 08:30:21.413338       1 mutation.go:145] Done schema update predicate:"board_name" value_type:STRING directive:INDEX tokenizer:"hash" tokenizer:"trigram" 
alpha_1_1_6092b86da97f | I0710 08:30:21.413734       1 server.go:329] ALTER op: schema:"\n\ttype: string @index(hash) .\n\n\tboard: string @index(hash) .\n\tcontent: string .\n\tauthor: uid @reverse .\n\tpost_time: dateTime @index(day) .\n\treplies: uid @reverse @count .\n\tlike: uid @count .\n\tdiss: uid @count .\n\n\tuser_id: string @index(hash) .\n\tuser_name: string .\n\tuser_info: string .\n\n\tboard_name: string @index(hash, trigram) .\n"  done
alpha_1_1_6092b86da97f | W0710 08:30:22.537712       1 draft.go:914] Raft.Ready took too long to process: Timer Total: 798ms. Breakdown: [{sync 798ms} {disk 0s} {proposals 0s} {advance 0s}] Num entries: 1. MustSync: true
alpha_1_1_6092b86da97f | W0710 08:30:23.941738       1 draft.go:914] Raft.Ready took too long to process: Timer Total: 249ms. Breakdown: [{sync 249ms} {disk 0s} {proposals 0s} {advance 0s}] Num entries: 1. MustSync: true
zero_1_2d1c24b1e703 | W0710 08:30:24.355091       1 raft.go:707] Raft.Ready took too long to process: Timer Total: 364ms. Breakdown: [{sync 364ms} {disk 0s} {proposals 0s} {advance 0s}]. Num entries: 1. MustSync: true
alpha_1_1_6092b86da97f | I0710 08:30:24.370430       1 server.go:275] Received ALTER op: drop_all:true 
alpha_1_1_6092b86da97f | I0710 08:30:24.370476       1 server.go:602] Got Alter request from "10.252.50.43:61134"
alpha_1_1_6092b86da97f | 2019/07/10 08:30:24 Attribute board is not indexed.
alpha_1_1_6092b86da97f | 
alpha_1_1_6092b86da97f | github.com/dgraph-io/dgraph/x.Wrap
alpha_1_1_6092b86da97f |        /tmp/go/src/github.com/dgraph-io/dgraph/x/error.go:91
alpha_1_1_6092b86da97f | github.com/dgraph-io/dgraph/x.Check
alpha_1_1_6092b86da97f |        /tmp/go/src/github.com/dgraph-io/dgraph/x/error.go:41
alpha_1_1_6092b86da97f | github.com/dgraph-io/dgraph/worker.(*queryState).handleCompareFunction
alpha_1_1_6092b86da97f |        /tmp/go/src/github.com/dgraph-io/dgraph/worker/task.go:980
alpha_1_1_6092b86da97f | github.com/dgraph-io/dgraph/worker.(*queryState).helpProcessTask
alpha_1_1_6092b86da97f |        /tmp/go/src/github.com/dgraph-io/dgraph/worker/task.go:835
alpha_1_1_6092b86da97f | github.com/dgraph-io/dgraph/worker.processTask
alpha_1_1_6092b86da97f |        /tmp/go/src/github.com/dgraph-io/dgraph/worker/task.go:736
alpha_1_1_6092b86da97f | github.com/dgraph-io/dgraph/worker.ProcessTaskOverNetwork
alpha_1_1_6092b86da97f |        /tmp/go/src/github.com/dgraph-io/dgraph/worker/task.go:152
alpha_1_1_6092b86da97f | github.com/dgraph-io/dgraph/query.ProcessGraph
alpha_1_1_6092b86da97f |        /tmp/go/src/github.com/dgraph-io/dgraph/query/query.go:1938
alpha_1_1_6092b86da97f | runtime.goexit
alpha_1_1_6092b86da97f |        /usr/local/go/src/runtime/asm_amd64.s:1337
zero_1_2d1c24b1e703 | W0710 08:30:24.493940       1 pool.go:226] Connection lost with alpha_1:7080. Error: rpc error: code = Unavailable desc = transport is closing

1. dropall
2. set schema
3. query by another goroutine periodically, may happen between 1 & 2

Do the operations above for many times.

• Expected behaviour and actual result.

Don't crash.

[martinmr]

Thanks for your report. It seems like the following is happening.

1. Query needing an index for predicate board comes in. There are multiple checks but the first one passes.
2. Drop all happens, now the board predicate does not have an index.
3. handleCompareFunction is executed, it assumes that the board predicate has an index so the caller asserts that there's no error returned from this function. However, by this point this is not accurate.

@manishrjain I think I can fix this particular issue but I wonder if there's a more fundamental problem that could cause more issues down the road. If an alter happens while queries are running perhaps the queries should be cancelled to avoid getting into this kind of situations.