Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Races while running flock #3685

Closed
animesh2049 opened this issue Jul 18, 2019 · 14 comments
Closed

Races while running flock #3685

animesh2049 opened this issue Jul 18, 2019 · 14 comments
Labels
area/crash Dgraph issues that cause an operation to fail, or the whole server to crash. kind/bug Something is broken. priority/P0 Critical issue that requires immediate attention. status/accepted We accept to investigate/work on it.
Milestone
Dgraph v1.1

Comments

@animesh2049
Copy link
Contributor

If you suspect this could be a bug, follow the template.

  • What version of Dgraph are you using?
Dgraph version   : v1.0.12-rc3-494-g16663939
Commit SHA-1     : 16663939
Commit timestamp : 2019-06-07 10:28:44 +0530
Branch           : master
Go version       : go1.12.5

  • Have you tried reproducing the issue with latest release?
    Yes

  • Steps to reproduce the issue (command/config used to run Dgraph).
    Use this script to run flock. It's just pulling new master of dgraph and flock and running them along with flock client.

#!/bin/bash
set -x

RUN_TIME=7200
SLEEP_TIME=21600
DGRAPH_DIR=$HOME/go/src/github.com/dgraph-io/dgraph
FLOCK_DIR=$HOME/go/src/github.com/dgraph-io/flock
DOCKER_TMP_DIRS=/tmp/flock_test
OUTPUTS_DIR=$HOME/docker_outputs

while true
do
	cd $DGRAPH_DIR
	echo "Pulling dgraph fom git"
	git checkout master
	git pull
	
	cd $FLOCK_DIR
	echo "Pulling flock from git"
	git checkout master
	git pull
	
	cd $GOPATH/bin
	echo "Compiling dgraph and flock"
	go install -race github.com/dgraph-io/dgraph/dgraph
	go install github.com/dgraph-io/flock
	
	echo "Cleaning  temp directories"
	rm -rf $DOCKER_TMP_DIRS
	echo "Recreating temp directories"
	mkdir -p $DOCKER_TMP_DIRS/{working,testdata,data}
	mkdir -p $OUTPUTS_DIR
	chown -R flock.flock $DOCKER_TMP_DIRS
	chmod -R 777 $DOCKER_TMP_DIRS
	docker stop $(docker ps -a -q)
	docker rm $(docker ps -a -q)
	killall timeout

	echo "Running docker compose"	
	time_now=`date +%s` # Present timestamp
	timeout $RUN_TIME docker-compose -f $HOME/docker-compose.yml up >$OUTPUTS_DIR/$time_now 2>&1 &
	echo "Now sleeping while the dockers come up"
	sleep 120
	echo "Now starting flock"
	timeout $RUN_TIME $GOPATH/bin/flock -a ":9080,:9081,:9082" >$HOME/flock_outputs 2>&1 &
	echo "Now starting flock client"
	timeout $RUN_TIME go run $FLOCK_DIR/client/main.go -a ":9080,:9081,:9082" >$HOME/flock_client_output 2>&1 &
	
	sleep $RUN_TIME
	docker stop $(docker ps -a -q)
	docker rm $(docker ps -a -q)
	sleep $SLEEP_TIME
done

My docker-compose.yml file

# Auto-generated with: [/home/javier/dgraph-io/dgraph/compose/compose -u -d /var/tmp/flock_test/data/twitterer]
#
version: "3.5"
services:
  alpha1:
    image: dgraph/dgraph
    container_name: alpha1
    working_dir: /tmp/flock_test/working/alpha1
    labels:
      cluster: flock-cluster
    ports:
    - 8080:8080
    - 9080:9080
    volumes:
    - type: bind
      source: $GOPATH/bin/dgraph
      target: /gobin/dgraph
      read_only: true
    - type: bind
      source: /tmp/flock_test/testdata
      target: /tmp/flock_test/data
      read_only: false
    user: ${UID:-1000}
    command: /gobin/dgraph alpha -p /tmp/flock_test/data/alpha1/p -w /tmp/flock_test/data/alpha1/w -o 0 --my=alpha1:7080 --lru_mb=1024
      --zero=zero1:5080 --logtostderr -v=2 
  alpha2:
    image: dgraph/dgraph
    container_name: alpha2
    working_dir: /tmp/flock_test/working/alpha2
    depends_on:
    - alpha1
    labels:
      cluster: flock-cluster
    ports:
    - 8081:8081
    - 9081:9081
    volumes:
    - type: bind
      source: $GOPATH/bin/dgraph
      target: /gobin/dgraph
      read_only: true
    - type: bind
      source: /tmp/flock_test/testdata
      target: /tmp/flock_test/data
      read_only: false
    user: ${UID:-1000}
    command: /gobin/dgraph alpha -p /tmp/flock_test/data/alpha2/p -w /tmp/flock_test/data/alpha2/w -o 1 --my=alpha2:7081 --lru_mb=1024
      --zero=zero1:5080 --logtostderr -v=2 
  alpha3:
    image: dgraph/dgraph
    container_name: alpha3
    working_dir: /tmp/flock_test/working/alpha3
    depends_on:
    - alpha2
    labels:
      cluster: flock-cluster
    ports:
    - 8082:8082
    - 9082:9082
    volumes:
    - type: bind
      source: $GOPATH/bin/dgraph
      target: /gobin/dgraph
      read_only: true
    - type: bind
      source: /tmp/flock_test/testdata
      target: /tmp/flock_test/data
      read_only: false
    user: ${UID:-1000}
    command: /gobin/dgraph alpha -p /tmp/flock_test/data/alpha3/p -w /tmp/flock_test/data/alpha3/w -o 2 --my=alpha3:7082 --lru_mb=1024
      --zero=zero1:5080 --logtostderr -v=2 
  zero1:
    image: dgraph/dgraph
    container_name: zero1
    working_dir: /tmp/flock_test/working/zero1
    labels:
      cluster: flock-cluster
    ports:
    - 5080:5080
    - 6080:6080
    volumes:
    - type: bind
      source: $GOPATH/bin/dgraph
      target: /gobin/dgraph
      read_only: true
    - type: bind
      source: /tmp/flock_test/testdata
      target: /tmp/flock_test/data
      read_only: false
    user: ${UID:-1000}
    command: /gobin/dgraph zero -w /tmp/flock_test/data/zero1/zw -o 0 --idx=1 --my=zero1:5080 --replicas=3
      --logtostderr -v=2 --bindall 
  zero2:
    image: dgraph/dgraph
    container_name: zero2
    working_dir: /tmp/flock_test/working/zero2
    depends_on:
    - zero1
    labels:
      cluster: flock-cluster
    ports:
    - 5082:5082
    - 6082:6082
    volumes:
    - type: bind
      source: $GOPATH/bin/dgraph
      target: /gobin/dgraph
      read_only: true
    - type: bind
      source: /tmp/flock_test/testdata
      target: /tmp/flock_test/data
      read_only: false
    user: ${UID:-1000}
    command: /gobin/dgraph zero -w /tmp/flock_test/data/zero2/zw -o 2 --idx=2 --my=zero2:5082 --replicas=3
      --logtostderr -v=2 --peer=zero1:5080 
  zero3:
    image: dgraph/dgraph
    container_name: zero3
    working_dir: /tmp/flock_test/working/zero3
    depends_on:
    - zero2
    labels:
      cluster: flock-cluster
    ports:
    - 5083:5083
    - 6083:6083
    volumes:
    - type: bind
      source: $GOPATH/bin/dgraph
      target: /gobin/dgraph
      read_only: true
    - type: bind
      source: /tmp/flock_test/testdata
      target: /tmp/flock_test/data
      read_only: false
    user: ${UID:-1000}
    command: /gobin/dgraph zero -w /tmp/flock_test/data/zero3/zw -o 3 --idx=3 --my=zero3:5083 --replicas=3
      --logtostderr -v=2 --peer=zero1:5080 
  ratel:
    image: dgraph/dgraph
    container_name: ratel
    depends_on:
      - alpha1
    labels:
      cluster: flock-cluster
    ports:
    - 8000:8000
    command: dgraph-ratel
volumes: {}
  • Expected behaviour and actual result.
    Check comments for races.
@animesh2049
Copy link
Contributor Author

I am not sure if this is dgraph issue or flock issue.

^[[36;1malpha3    |^[[0m E0718 10:22:39.586322       1 draft.go:450] Applying proposal. Error: cannot add value with uid 43 to predicate mention because one of the existing values does not match this uid, either delete the existing values first or modify the schema to 'mention: [uid]'. Proposal: "mutations:<group_id:1 start_ts:58 edges:<entity:65 attr:\"author\" value_type:UID value_id:66 > edges:<entity:65 attr:\"created_at\" value:\"\\001\\000\\000\\000\\016\\324\\276g\\007\\000\\000\\000\\000\\020h\" value_type:DATETIME value_id:18446744073709551615 > edges:<entity:66 attr:\"description\" value:\"If you like role playing games or MMOs, sarcasm and occasional ranting I'm your Bear.\" value_type:STRING value_id:18446744073709551615 > edges:<entity:66 attr:\"friends_count\" value:\"\\352\\002\\000\\000\\000\\000\\000\\000\" value_type:INT value_id:18446744073709551615 > edges:<entity:65 attr:\"id_str\" value:\"1151799402710913024\" value_type:STRING value_id:18446744073709551615 > edges:<entity:65 attr:\"mention\" value_type:UID value_id:68 > edges:<entity:65 attr:\"mention\" value_type:UID value_id:67 > edges:<entity:65 attr:\"message\" value:\"RT @KarennChameleon: This totally happened. .@litedoctor \\342\\200\\230s 40 year old dream came true... https://t.co/9DdkIFOwM8\" value_type:STRING > edges:<entity:66 attr:\"profile_image_url\" value:\"http://pbs.twimg.com/profile_images/1541240204/BBBavatar_normal.jpg\" value_type:STRING > edges:<entity:65 attr:\"retweet\" value:\"\\000\" value_type:BOOL > edges:<entity:66 attr:\"screen_name\" value:\"BigBearButt\" value_type:STRING > edges:<entity:67 attr:\"screen_name\" value:\"KarennChameleon\" value_type:STRING > edges:<entity:68 attr:\"screen_name\" value:\"LiteDoctor\" value_type:STRING > edges:<entity:66 attr:\"user_id\" value:\"164498067\" value_type:STRING > edges:<entity:67 attr:\"user_id\" value:\"90015081\" value_type:STRING > edges:<entity:68 attr:\"user_id\" value:\"263492270\" value_type:STRING > edges:<entity:66 attr:\"user_name\" value:\"Big Bear Butt \\360\\237\\220\\273\" value_type:STRING > edges:<entity:67 attr:\"user_name\" value:\"Karen \\360\\237\\246\\216\" value_type:STRING > edges:<entity:68 attr:\"user_name\" value:\"Sean F\" value_type:STRING > > key:\"02-5623428966753316112\" index:77 ".

@animesh2049
Copy link
Contributor Author

Data Race 1

^[[36;1malpha3    |^[[0m ==================
^[[36;1malpha3    |^[[0m WARNING: DATA RACE 
^[[36;1malpha3    |^[[0m Read at 0x000002a068e0 by goroutine 336: 
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).preTraverse()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:525 +0x1d01
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.processNodeUids()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/outputnode.go:456 +0x3ad
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).toFastJSON()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/outputnode.go:505 +0x17b
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.ToJson()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/outputnode.go:49 +0x561
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).doQuery()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:677 +0x12fe
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).Query()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:565 +0xbb
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api._Dgraph_Query_Handler()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api/api.pb.go:1917 +0x2fc
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:972 +0x936
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleStream()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:1252 +0x1351
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:691 +0xac
^[[36;1malpha3    |^[[0m 
^[[36;1malpha3    |^[[0m Previous write at 0x000002a068e0 by goroutine 357: 
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).updateFacetMatrix()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1405 +0x3a0
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).updateUidMatrix()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1410 +0x3c
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).populateVarMap()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1450 +0x35c
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*Request).ProcessQuery()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:2724 +0x15c8
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*Request).Process()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:2757 +0x85
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).doQuery()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:656 +0xb35
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).Query()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:565 +0xbb
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api._Dgraph_Query_Handler()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api/api.pb.go:1917 +0x2fc
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:972 +0x936
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleStream()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:1252 +0x1351
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:691 +0xac
^[[36;1malpha3    |^[[0m 
^[[36;1malpha3    |^[[0m Goroutine 336 (running) created at:
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:689 +0xb8
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:421 +0x1622
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:461 +0x37a
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:687 +0x170
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:649 +0x50
^[[36;1malpha3    |^[[0m 
^[[36;1malpha3    |^[[0m Goroutine 357 (finished) created at:
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:689 +0xb8
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:421 +0x1622
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:461 +0x37a
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:687 +0x170
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:649 +0x50
^[[36;1malpha3    |^[[0m ==================

@animesh2049
Copy link
Contributor Author

animesh2049 commented Jul 18, 2019 •
edited
Loading

Data Race 2

^[[36;1malpha3    |^[[0m ==================
^[[36;1malpha3    |^[[0m WARNING: DATA RACE 
^[[36;1malpha3    |^[[0m Read at 0x000002a069e0 by goroutine 215: 
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/algo.MergeSorted()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/algo/uidlist.go:363 +0x141
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.ProcessGraph()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:2067 +0x3a09
^[[36;1malpha3    |^[[0m 
^[[36;1malpha3    |^[[0m Previous write at 0x000002a069e0 by goroutine 21:
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/algo.IntersectWith()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/algo/uidlist.go:156 +0x214
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).updateUidMatrix()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1420 +0x14b
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).populateVarMap()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1450 +0x35c
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*Request).ProcessQuery()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:2724 +0x15c8
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*Request).Process()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:2757 +0x85
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).doQuery()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:656 +0xb35
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).Query()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:565 +0xbb
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api._Dgraph_Query_Handler()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api/api.pb.go:1917 +0x2fc
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:972 +0x936
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleStream()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:1252 +0x1351
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:691 +0xac
^[[36;1malpha3    |^[[0m 
^[[36;1malpha3    |^[[0m Goroutine 215 (running) created at:
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.ProcessGraph()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:2213 +0x13e9
^[[36;1malpha3    |^[[0m 
^[[36;1malpha3    |^[[0m Goroutine 21 (finished) created at:
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:689 +0xb8
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:421 +0x1622
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:461 +0x37a
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:687 +0x170
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:649 +0x50
^[[36;1malpha3    |^[[0m ==================

@animesh2049
Copy link
Contributor Author

animesh2049 commented Jul 18, 2019 •
edited
Loading

Data Race 3

^[[31malpha2    |^[[0m ==================
^[[31malpha2    |^[[0m WARNING: DATA RACE
^[[31malpha2    |^[[0m Write at 0x000002a068e0 by goroutine 215:
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).updateFacetMatrix()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1405 +0x3a0
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).updateUidMatrix()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1410 +0x3c
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).populateVarMap()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1450 +0x35c
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*Request).ProcessQuery()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:2724 +0x15c8
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*Request).Process()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:2757 +0x85
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).doQuery()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:656 +0xb35
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).Query()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:565 +0xbb
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api._Dgraph_Query_Handler()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api/api.pb.go:1917 +0x2fc
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:972 +0x936
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleStream()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:1252 +0x1351
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:691 +0xac
^[[31malpha2    |^[[0m 
^[[31malpha2    |^[[0m Previous read at 0x000002a068e0 by goroutine 374: 
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).preTraverse()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:525 +0x1d01
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.processNodeUids()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/outputnode.go:456 +0x3ad
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).toFastJSON()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/outputnode.go:505 +0x17b
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.ToJson()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/outputnode.go:49 +0x561
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).doQuery()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:677 +0x12fe
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).Query()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:565 +0xbb
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api._Dgraph_Query_Handler()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api/api.pb.go:1917 +0x2fc
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:972 +0x936
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleStream()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:1252 +0x1351
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:691 +0xac
^[[31malpha2    |^[[0m 
^[[31malpha2    |^[[0m Goroutine 215 (running) created at:
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:689 +0xb8
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:421 +0x1622
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:461 +0x37a
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:687 +0x170
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:649 +0x50
^[[31malpha2    |^[[0m
^[[31malpha2    |^[[0m Goroutine 374 (running) created at:
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:689 +0xb8
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:421 +0x1622
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:461 +0x37a
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:687 +0x170
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:649 +0x50
^[[31malpha2    |^[[0m ==================

@animesh2049
Copy link
Contributor Author

animesh2049 commented Jul 18, 2019 •
edited
Loading

Data Race 4

^[[31malpha2    |^[[0m ==================
^[[31malpha2    |^[[0m WARNING: DATA RACE
^[[31malpha2    |^[[0m Write at 0x000002a069e0 by goroutine 215:
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/algo.IntersectWith()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/algo/uidlist.go:156 +0x214
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).updateUidMatrix()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1420 +0x14b
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).populateVarMap()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1450 +0x35c
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*Request).ProcessQuery()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:2724 +0x15c8
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*Request).Process()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:2757 +0x85
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).doQuery()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:656 +0xb35
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).Query()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:565 +0xbb
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api._Dgraph_Query_Handler()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api/api.pb.go:1917 +0x2fc
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:972 +0x936
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleStream()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:1252 +0x1351
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:691 +0xac
^[[31malpha2    |^[[0m
^[[31malpha2    |^[[0m Previous read at 0x000002a069e0 by goroutine 374:
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).preTraverse()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:449 +0x1f5a
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.processNodeUids()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/outputnode.go:456 +0x3ad
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).toFastJSON()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/outputnode.go:505 +0x17b
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/query.ToJson()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/outputnode.go:49 +0x561
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).doQuery()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:677 +0x12fe
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).Query()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:565 +0xbb
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api._Dgraph_Query_Handler()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api/api.pb.go:1917 +0x2fc
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:972 +0x936
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleStream()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:1252 +0x1351
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:691 +0xac
^[[31malpha2    |^[[0m
^[[31malpha2    |^[[0m Goroutine 215 (running) created at:
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:689 +0xb8
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:421 +0x1622
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:461 +0x37a
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:687 +0x170
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:649 +0x50
^[[31malpha2    |^[[0m
^[[31malpha2    |^[[0m Goroutine 374 (running) created at:
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:689 +0xb8
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:421 +0x1622
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:461 +0x37a
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:687 +0x170
^[[31malpha2    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1()
^[[31malpha2    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:649 +0x50
^[[31malpha2    |^[[0m ==================

@animesh2049 animesh2049 added kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit) labels Jul 18, 2019
@manishrjain
Copy link
Contributor

manishrjain commented Jul 18, 2019 •
edited
Loading

Very nice finds! Bugs and data races are always P0.

@danielmai
Copy link
Contributor

I am not sure if this is dgraph issue or flock issue.

E0718 10:22:39.586322 1 draft.go:450] Applying proposal. Error: cannot add value with uid 43 to predicate mention because one of the existing values does not match this uid, either delete the existing values first or modify the schema to 'mention: [uid]'.

This can be fixed in Flock by updating the schema for uid predicates to [uid] for Dgraph v1.1.

@danielmai danielmai mentioned this issue Jul 18, 2019
Merged
@campoy campoy added area/crash Dgraph issues that cause an operation to fail, or the whole server to crash. priority/P0 Critical issue that requires immediate attention. status/accepted We accept to investigate/work on it. and removed priority/P1 Serious issue that requires eventual attention (can wait a bit) labels Jul 19, 2019
@martinmr
Copy link
Contributor

why are you running flock with a version of Dgraph from a moth ago? At least that's what the checksum 16663939 is telling me and the line numbers in that commit make sense when looking at the stack traces.

I can take a look at them but we should be running dgraph on the latest version on master.

@martinmr
Copy link
Contributor

martinmr commented Jul 19, 2019 •
edited
Loading

Data Race 1

^[[36;1malpha3    |^[[0m ==================
^[[36;1malpha3    |^[[0m WARNING: DATA RACE 
^[[36;1malpha3    |^[[0m Read at 0x000002a068e0 by goroutine 336: 
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).preTraverse()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:525 +0x1d01
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.processNodeUids()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/outputnode.go:456 +0x3ad
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).toFastJSON()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/outputnode.go:505 +0x17b
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.ToJson()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/outputnode.go:49 +0x561
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).doQuery()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:677 +0x12fe
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).Query()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:565 +0xbb
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api._Dgraph_Query_Handler()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api/api.pb.go:1917 +0x2fc
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:972 +0x936
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleStream()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:1252 +0x1351
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:691 +0xac
^[[36;1malpha3    |^[[0m 
^[[36;1malpha3    |^[[0m Previous write at 0x000002a068e0 by goroutine 357: 
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).updateFacetMatrix()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1405 +0x3a0
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).updateUidMatrix()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1410 +0x3c
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*SubGraph).populateVarMap()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:1450 +0x35c
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*Request).ProcessQuery()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:2724 +0x15c8
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/query.(*Request).Process()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/query/query.go:2757 +0x85
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).doQuery()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:656 +0xb35
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/edgraph.(*Server).Query()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:565 +0xbb
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api._Dgraph_Query_Handler()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/dgo/protos/api/api.pb.go:1917 +0x2fc
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).processUnaryRPC()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:972 +0x936
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleStream()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:1252 +0x1351
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:691 +0xac
^[[36;1malpha3    |^[[0m 
^[[36;1malpha3    |^[[0m Goroutine 336 (running) created at:
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:689 +0xb8
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:421 +0x1622
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:461 +0x37a
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:687 +0x170
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:649 +0x50
^[[36;1malpha3    |^[[0m 
^[[36;1malpha3    |^[[0m Goroutine 357 (finished) created at:
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:689 +0xb8
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:421 +0x1622
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/internal/transport/http2_server.go:461 +0x37a
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).serveStreams()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:687 +0x170
^[[36;1malpha3    |^[[0m   github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1()
^[[36;1malpha3    |^[[0m       /home/flock/go/src/github.com/dgraph-io/dgraph/vendor/google.golang.org/grpc/server.go:649 +0x50
^[[36;1malpha3    |^[[0m ==================

preTraverse is doing:

			if len(pc.facetsMatrix) > idx && len(pc.facetsMatrix[idx].FacetsList) > 0 {

while updateFacetsMatrix is doing:

		sg.facetsMatrix[lidx].FacetsList = out

I am quoting the lines so that they are easier in different commits. The problem seems obvious but I am not sure how to address it. Currently, there's not any kind of synchronization happening inside the subgraphs. I mentioned this to Manish before and he said that was intentional as different subgraphs are meant to be processed by different goroutines. But the query codebase is pretty hairy so that's probably not the case everywhere (or it was at some point but not anymore due to more recent changes).

@animesh2049
Copy link
Contributor Author

why are you running flock with a version of Dgraph from a moth ago? At least that's what the checksum 16663939 is telling me and the line numbers in that commit make sense when looking at the stack traces.

I can take a look at them but we should be running dgraph on the latest version on master

@martinmr
I realized it later and then I ran flock on the latest master as well. Still these data races are there.

@pawanrawal
Copy link
Contributor

I looked into the data races in query package and couldn't see how would populateVarMap run concurrently with preTraverse. populateVarMap has to be called multiple times, i.e. it would be called every time a query is executed. The variables defined in the query are checked recursively and stored in req.vars.
After Process function has returned from the query package, we call ToJson from edgraph/server.go which calls preTraverse. Since ToJson is called after Process there shouldn't be a data race, unless, Process spawns a goroutine which is still running after it has returned.

@pawanrawal
Copy link
Contributor

pawanrawal commented Aug 5, 2019 •
edited
Loading

I was able to reproduce this. An easy way of reproducing this is to just start one alpha and one Zero and then run both flock and the flock client. With both of them running, you can typically observe the Data Race logs in < 5 mins. I believe its being caused by having package level variables in worker/task.go

dgraph/worker/task.go

Lines 48 to 51 in 58820c0

emptyUIDList pb.List
emptyFacetsList pb.FacetsList
emptyResult pb.Result
emptyValueList = pb.ValueList{Values: []*pb.TaskValue{}}
since this is the only state that is being shared between different HTTP requests. I am going to keep flock running for an hour to see if I still notice any race conditions after removing these package level variables.

@pawanrawal pawanrawal mentioned this issue Aug 5, 2019
Merged
@campoy campoy added this to the Dgraph v1.1 milestone Aug 6, 2019
pawanrawal added a commit that referenced this issue Aug 6, 2019
@pawanrawal
Add code comments and fix data races in query.go. (#3749)
8ba4a85 
- Fixes the race conditions mentioned in #3685 by removing global variables from `worker/task.go`. 
- I have also moved preTraverse and related functions to outputnode.go. 
- There is also a small optimization in ProcessGraph as we don't call fillVars with ParentVars if there are no more children of the current SubGraph node.
@pawanrawal pawanrawal mentioned this issue Aug 7, 2019
Closed
8 tasks
@pawanrawal
Copy link
Contributor

@animesh2049 confirmed that these races don't occur anymore. Closing this issue.

@animesh2049
Copy link
Contributor Author

I ran flock after @pawanrawal's PR. I didn't find any data race.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/crash Dgraph issues that cause an operation to fail, or the whole server to crash. kind/bug Something is broken. priority/P0 Critical issue that requires immediate attention. status/accepted We accept to investigate/work on it.
Milestone
Dgraph v1.1
Development

No branches or pull requests
6 participants
@martinmr @manishrjain @campoy @danielmai @pawanrawal @animesh2049