k6 dependencies update policy discussion

[mstoykov]

edit: updated table in comment to preserve history better

I make this issue to discuss whether we will be updating each dependency and on what ... schedule.

I would argue that while some dependencies are super important and later versions likely fix bugs and performance problems and it is worth our time updating them, others are either ones that we want to drop or are used in a single place for a single task and because of some specificity there is no point updating it regularly or at all.

In none of the cases, this means that a given library MUST be updated blindly or MUST NOT be updated if there turns out to be a good reason. Just that updating some libraries probably has more relevance than others and given our limited resources taking the time to update all of them constantly and fixing problems is probably not ... wanted.

Additionally, while I would prefer if some scripts are added to make this easier, I am definitely against this breaking the CI or preventing a release.

Goja

Goja is probably one of the bigger exceptions. It is what we use to run javascript and as any improvement can turn out to have a considerable either positive or negative impact on k6 we practically have to update it constantly. So even if something that we don't particularly care about is added/fixed we probably should update to it at least once each release cycle. In reality, there has been something useful more or less every month so we have been updating it constantly.

golang.org/x/*

While those libraries are not part of the golang stdlib they either get embed in it in different forms or are the natural extension. They also happen to have a branch for each release so my proposal is to just update to the new branch each time we update the golang version so we follow it. Or even just at the beginning of each release just in case.

Given their ubiquity in general I don't expect it will be breaking constantly. If anything not updating x/net has turned out to be problematic albeit because we were at an extreme not updating it ...

Others:

The list below is what it was in go.mod and I am listing the indirect just for ease of checking against the go.mod.

Post factum the categories are :

• never - we either don't care about any updates as we use it for something extremely specific and updating it won't help. Or the more common case it is something we want to drop.
• every release - things that we more or less need to update as we are very dependant on it
• 1-2 a year - things that we should be updating, but it is likely with minimal benefits and just doing it all the time has more chances of us running into problems then something else.

name	description	thoughts	proposed update period
github.com/Azure/go-ntlmssp	NTLM library	I am for updating it, but given our problems with testing it, I think that should wait	never for now 1-2 a year as we still need to update it
github.com/DataDog/datadog-go	datadogd/statsd library	we haven't had any problems, but it is probably easy to test	at least once a year probably be more with better tests
github.com/GeertJohan/go.rice	embed static files in go binaries	this should be replaceable by embed	we should drop it not update it
github.com/PuerkitoBio/goquery	like jquery	used in k6/html mostly, probably something that should be updated given that it has chances to introduce performance and compatibility fixes, but is probably somewhat risky	1-2 a year
github.com/Shopify/sarama	kafka library	we intend on dropping it so ....	never
github.com/Shopify/toxiproxy	indirect (probably by sarama) / introduce network problems	this could be useful but I prefer to first drop it with sarama and then figure out whether we will be using something like that. We currently don't vendor it as it is only used in tests	never
github.com/Soontao/goHttpDigestClient	digest authentication	this is easier to test than NTLM so we should probably be updating it	1-2 a year
github.com/andybalholm/brotli	brotli compression	possibly some performance improvement, so for updating every once in a while	1-2 a year
github.com/andybalholm/cascadia	indirect/ css selectors		never directly
github.com/daaku/go.zipexe	ndirect by go.rice		never
github.com/dlclark/regexp2	indirect by goja	probably worth updating every once in a while as I am not certain if goja will do it	goja should update it as dependency
github.com/dop251/goja	js VM	see part above	at least once a release
github.com/dustin/go-humanize	humanize numbers	used in stats for bytes and floats ... we can probably drop it and either way I don't think we need to update it	never - hopefully it will be dropped with #1806
github.com/eapache/go-resiliency	indirect by sarama / resiliency patterns		never
github.com/eapache/go-xerial-snappy	indirect by sarama / snappy		never
github.com/eapache/queue	indirect by sarama		never
github.com/fatih/color	color library	we use it, I doubt any updates to it are useful, but maybe something useful has been added	once a year just to keep up
github.com/gedex/inflector	indirect by jsonapi		never
github.com/ghodss/yaml	indirect by strvals which is used in kafka and lib/types/dns		never
github.com/gin-contrib/sse	indirect by gin		never
github.com/gin-gonic/gin	indirect by api2go		never
github.com/go-sourcemap/sourcemap	indirect by goja	we currently don't use sourcemaps, but probably can update it. I have no idea if it matters	with goja/ once a year
github.com/golang/protobuf	protobuf	used by grpc	update with grpc
github.com/golang/snappy	indirect by go-xerial-snappy		never
github.com/google/go-cmp	indirect by grpc		never it will be by grpc
github.com/gorilla/context	indirect by gorilla/mux used by api2go	we don't want any of those	never
github.com/gorilla/mux	indirect by api2go	we don't want any of those	never
github.com/gorilla/websocket	websocket library	we might want to change it but still worth it to be updated probably	1-2 per year
github.com/inconshreveable/mousetrap	indirect by cobra	it figures out whether on windows the command was called by double-clicking the icon - seems like there is no point in us updating this ....	never
github.com/influxdata/influxdb1-client	influxdb v1	we want to update to v2, but probably still support v1 ?!?	depends
github.com/jhump/protoreflect	grpc related		with grpc
github.com/julienschmidt/httprouter	HTTP router	our v1 rest API uses that and it doesn't need to	never
github.com/kardianos/osext	inderect by go.rice	we want to drop go.rice, so no updates	never
github.com/kelseyhightower/envconfig	config from env vars	we want to drop this	never
github.com/klauspost/compress	different compressions	we do use those and there could be performance benefits	1-2 per year
github.com/klauspost/cpuid	indirect by above		with dependency
github.com/kubernetes/helm	indirect for strvals in kafka and influxdb	I am for dropping this	never
github.com/labstack/echo	indirect by api2go		never
github.com/labstack/gommon	indirect by echo		never
github.com/mailru/easyjson	json marshalling	cloudapi uses for marshaling, possible performance improvement	1-2 per year
github.com/manyminds/api2go	api2go	used by the REST API for no good reason IMO	never
github.com/mattn/go-colorable	coloring output	given that it can probably fix stuff we should at least check it and update it once in a while. Unfortunately it also can break stuff	1-2 per year
github.com/mattn/go-isatty	checks whether something is tty	same as above	1-2 per year
github.com/mccutchen/go-httpbin	httpbin implementation	used in tests, we probably should be updating those once in a while, although I doubt anything new has been added	1-2 per year
github.com/mitchellh/mapstructure	map <-> struct mapping	used by kafka/influxdb and the compiler (not actually used ). This might be useful in other places, but likely will be dropped, and likely there are no reasons to update it if it works	never
github.com/nu7hatch/gouuid	uuid lib	used by digest lib and http-debug, probably fine to update, but also probably no benefit	1-2 year
github.com/onsi/ginkgo	indirect by api2go		never
github.com/oxtoacart/bpool	[]byte/Buffer pool	there could be some performance benefit updating it, also we probably should try sync.Pool again	1-2 per year
github.com/pierrec/lz4	indirect by sarama(kafka)		never
github.com/pierrec/xxHash	indirect by lz4		never
github.com/pkg/errors	errors handling	used through out k6, but I think we can just move the stdlib ?	never
github.com/pmezard/go-difflib	difflib	mostly used by testify and in k6 convert test	with testify/dependancy
github.com/rcrowley/go-metrics	indirect by sarama(kafka)	could be used by k6 directly as it seems useful, but currently is only dependency of sarama	never
github.com/serenize/snaker	snake case utility	used by k6/html and the "bridge", likely not a problem to update, but also probably not very useful	1-2 a year
github.com/sirupsen/logrus	logging library	not really interesting to update IMO it is also in maintenance mode	1-2 a year
github.com/spf13/afero	fs emulation	we have had problems with it and it's testing, we want to drop it and have so far fixed all the problems we had with it ... somehow	never
github.com/spf13/cobra	command line arguments	we do use this quite extensively, likely there are some fixes but it is probably risky given how bad we are at testing those	1-2 a year
github.com/spf13/pflag	flag parsing	same as above	-12 a year
github.com/stretchr/testify	testing library	we do use it quite a lot, I doubt anything but new methods will come from updates so I am fine updating it	1-2 a year
github.com/tidwall/gjson	xpath for json	used for k6/http , probably good idea to keep updating, wonder but we some review as this could be breaking changes	1-2 a year
github.com/tidwall/pretty	json prettifier	used by gjson and har converter, possibly dropped when we drop the har converter	with w/e gjson needs
github.com/ugorji/go	indirect by gin		never
github.com/urfave/negroni	HTTP middleware	used by the rest API	never
github.com/valyala/bytebufferpool	indirectly by the REST API	maybe instead of bpool?	never
github.com/valyala/fasttemplate	indirect by the REST API		never
github.com/zyedidia/highlight	syntax highlighter	used by cmd to highlight status reports from k6 stop/resume/stats/... ...	never ?
golang.org/x/crypto			each release
golang.org/x/net			each release
golang.org/x/sys			each release
golang.org/x/text			each release
golang.org/x/time			each release
golang.org/x/xerrors	indirect by grpc
google.golang.org/genproto	indirect by grpc
google.golang.org/grpc	grpc	used by 'k6/grpc', it is probably good idea to update it, but given lack of testing 🤷	1-2 a year ?
google.golang.org/protobuf	for grpc
gopkg.in/go-playground/assert.v1	indirect by something !??
gopkg.in/go-playground/validator.v8	indirect by gin
gopkg.in/guregu/null.v2	indirect by api2go
gopkg.in/guregu/null.v3	null library	this in general will not be "needed" once go generics land and until then I don't think it matters all that much	never ?
gopkg.in/yaml.v2	yaml	inderectly by gin and by the "status" highlighting code in `ui/dump.go'	never ?!?

[na--]

I agree with most of everything you wrote, with the following exceptions:

• github.com/Azure/go-ntlmssp - even if we don't have tests, we know we have bugs, so I think we should update this, just a bit more carefully and while actually reading the diff
• github.com/dlclark/regexp2 - goja has a go.mod since recently, so it will probably be updated when necessary and maybe we should treat it as other indirect dependencies and leave it for the upstream to manage
• github.com/dustin/go-humanize - we will drop it with Replace text summary generation with JS code #1806
• the TTY and color libraries will probably soon need to be updated or replaced for Fix TTY color support #1805 anyway

I also created and pinned an issue about our intention to slowly shift the kafka output to an xk6 extension: #1934

[mstoykov]

I updated the original comment with the proposed changes @na--

[mstoykov]

Updated table after updates:

name	description	thoughts	proposed update period
github.com/Azure/go-ntlmssp	NTLM library	I am for updating it, but given our problems with testing it, I think that should wait	it has practically no development, but we should probably update it every release or so
github.com/DataDog/datadog-go	datadogd/statsd library	we haven't had any problems, but it is probably easy to test	never currently as it adds too much new dependencies and we might want to change dependency in order to get tcp on top of udp support
github.com/GeertJohan/go.rice	embed static files in go binaries	this should be replaceable by embed	we should drop it not update it part of #1975
github.com/PuerkitoBio/goquery	like jquery	used in k6/html mostly, probably something that should be updated given that it has chances to introduce performance and compatibility fixes, but is probably somewhat risky	1-2 a year
github.com/Soontao/goHttpDigestClient	digest authentication	this is easier to test than NTLM so we should probably be updating it	1-2 a year
github.com/andybalholm/brotli	brotli compression	possibly some performance improvement, so for updating every once in a while	1-2 a year
github.com/dop251/goja	js VM	see part above	at least once a release
github.com/dustin/go-humanize	humanize numbers	used in stats for bytes and floats ... we can probably drop it and either way I don't think we need to update it	never - hopefully it will be dropped with #1806, part of #1975
github.com/fatih/color	color library	we use it, I doubt any updates to it are useful, but maybe something useful has been added	once a year just to keep up
github.com/golang/protobuf	protobuf	used by grpc	update with grpc
github.com/gorilla/websocket	websocket library	we might want to change it but still worth it to be updated probably	1-2 per year
github.com/influxdata/influxdb1-client	influxdb v1	we want to update to v2, but probably still support v1 ?!?	depends
github.com/jhump/protoreflect	grpc related		with grpc
github.com/k6io/xk6-output-kafka	kafka output		to be dropped in future version
github.com/kelseyhightower/envconfig	config from env vars	we want to drop this	never
github.com/klauspost/compress	different compressions	we do use those and there could be performance benefits	1-2 per year
github.com/kubernetes/helm	indirect for strvals in kafka and influxdb	I am for dropping this	never
github.com/mailru/easyjson	json marshalling	cloudapi uses for marshaling, possible performance improvement	1-2 per year
github.com/manyminds/api2go	api2go	used by the REST API for no good reason IMO	never
github.com/mattn/go-colorable	coloring output	given that it can probably fix stuff we should at least check it and update it once in a while. Unfortunately it also can break stuff	1-2 per year
github.com/mattn/go-isatty	checks whether something is tty	same as above	1-2 per year
github.com/mccutchen/go-httpbin	httpbin implementation	used in tests, we probably should be updating those once in a while, although I doubt anything new has been added	1-2 per year
github.com/mitchellh/mapstructure	map <-> struct mapping	used by kafka/influxdb and the compiler (not actually used ). This might be useful in other places, but likely will be dropped, and likely there are no reasons to update it if it works	never
github.com/nu7hatch/gouuid	uuid lib	used by digest lib and http-debug, probably fine to update, but also probably no benefit	1-2 year
github.com/oxtoacart/bpool	[]byte/Buffer pool	there could be some performance benefit updating it, also we probably should try sync.Pool again	1-2 per year
github.com/pmezard/go-difflib	difflib	mostly used by testify and in k6 convert test	with testify/dependancy
github.com/serenize/snaker	snake case utility	used by k6/html and the "bridge", likely not a problem to update, but also probably not very useful	1-2 a year
github.com/sirupsen/logrus	logging library	not really interesting to update IMO it is also in maintenance mode	we intend to replace it
github.com/spf13/afero	fs emulation	we have had problems with it and it's testing, we want to drop it and have so far fixed all the problems we had with it ... somehow	never
github.com/spf13/cobra	command line arguments	we do use this quite extensively, likely there are some fixes but it is probably risky given how bad we are at testing those	1-2 a year
github.com/spf13/pflag	flag parsing	same as above	-12 a year
github.com/stretchr/testify	testing library	we do use it quite a lot, I doubt anything but new methods will come from updates so I am fine updating it	1-2 a year
github.com/tidwall/gjson	xpath for json	used for k6/http , probably good idea to keep updating, wonder but we some review as this could be breaking changes	1-2 a year
github.com/tidwall/pretty	json prettifier	used by gjson and har converter, possibly dropped when we drop the har converter	with w/e gjson needs
github.com/zyedidia/highlight	syntax highlighter	used by cmd to highlight status reports from k6 stop/resume/stats/... ...	currently being dropped with #1975
golang.org/x/crypto			each release
golang.org/x/net			each release
golang.org/x/text			each release
golang.org/x/time			each release
google.golang.org/grpc	grpc	used by 'k6/grpc', it is probably good idea to update it, but given lack of testing 🤷	1-2 a year ?
google.golang.org/protobuf	for grpc
gopkg.in/guregu/null.v3	null library	this in general will not be "needed" once go generics land and until then I don't think it matters all that much	never ?
gopkg.in/yaml.v2	yaml	inderectly by gin and by the "status" highlighting code in `ui/dump.go'	never ?!?

[mstoykov]

I am closing this as it seems all the things needing discussing were discussed and we have also dropped most of the dependencies we wanted.

Going forward the above table can/should be used as a guide/suggestions on whether a dependency should be updated, but a per case care should be taken.