initial updates to simplify lambda packaging logic #1232

ryandeivert · 2020-04-09T00:51:51Z

to: @airbnb/streamalert-maintainers

Background

Our packaging logic is garbage. We try to do too much and it's prone to bugs.

Changes

Now using a singular package that is used across all lambda functions.
The single zip that is created will contain all streamalert code, the conf and any user defined folders needed, specified in the conf/globals.json file.

Testing

Updates to unit tests.
Adding some new tests for targets.

streamalert_cli/manage_lambda/deploy.py

tests/unit/streamalert_cli/manage_lambda/test_deploy.py

chunyong-lin

This is a good point you mention We try to do too much and it's prone to bugs. My only concern is the lambda package will be bigger and slows down the deployment after this PR. But it may be a good tradeoff. I have few comments, nothing major.

chunyong-lin · 2020-04-09T16:32:55Z

conf/global.json

+    ],
+    "third_party_libraries": [
+      "pathlib2==2.3.5",
+      "policyuniverse==1.3.2.1"


How about other third party libs 'cbapi', 'netaddr', 'pymsteams', 'requests'?
Aha, I see they are defined here

streamalert/streamalert_cli/manage_lambda/package.py

Lines 44 to 55 in c49050e

# Define a package dict to support pinning versions across all subclasses

REQUIRED_LIBS = {

'backoff==1.8.1',

'boto3==1.10.6',

'cbapi==1.5.4',

'google-api-python-client==1.7.11',

'jmespath==0.9.4',

'jsonlines==1.2.0',

'netaddr==0.7.19',

'requests==2.22.0',

'pymsteams==0.1.12',

}

What's the different adding third party libs in conf/global.json and streamalert_cli/manage_lambda/package.py?

the third_party_libraries is meant to allow users to inject dependencies that are NOT part of the core streamalert package.. meaning, for all of the user-configurable things like rules, publishers, matchers, schedule_queries, etc - dependencies not included as part of the core package should be included in third_party_libraries for installation at deploy time

does that make sense?

cbapi, pymsteams, etc are all dependencies that are needed by the streamalert package, so they are hard-coded here.

Not right now, but we should revisit these dependencies eventually. Like I don't think cbapi should be deployed as part of the base package, since it's only used by the alert processor.

disagree :) that is what we were doing and it's garbage. but a different discussion for another time.

I only agree with you if we adopt the concept of "subpackages" (https://packaging.python.org/guides/packaging-namespace-packages/), which could be very likely in the next iteration of this

streamalert/_vendored/README.rst

streamalert/apps/_apps/README.rst

streamalert_cli/_infrastructure/modules/tf_scheduled_queries/variables.tf

streamalert_cli/manage_lambda/deploy.py

streamalert_cli/manage_lambda/package.py

chunyong-lin

Ryxias · 2020-04-09T18:11:01Z

conf/global.json

+    ],
+    "third_party_libraries": [
+      "pathlib2==2.3.5",
+      "policyuniverse==1.3.2.1"


Not right now, but we should revisit these dependencies eventually. Like I don't think cbapi should be deployed as part of the base package, since it's only used by the alert processor.

* bumping version to 3.2.0 * migrating Athena function to use tf_lambda module (#1217) * rename of athena function * updating terraform generation code to use tf_lambda module * updating tf_athena module to remove lambda code * updates for packaging, rollback, and deploy * misc updates related to config path renaming, etc * removing no-longer-used method (athena is default) * addressing PR feedback * adding more granular time prefix to athena client * fixing duplicate resource issues (#1218) * fixing duplicate resource issues * fixing some other bugs in #1217 * fixing tf targets for athena deploy (#1220) * adding "--config-dir" flag to CLI to support specifying path for config files (#1224) * adding support for supplying path to config via CLI flag * misc touchups * updating publishers to accept configurable paths (#1223) * moving matchers outside of rules directory * updating rules for new matcher path * updating unit test for consistency * making publisher locations configurable * fixing typo * updating tf_lambda module to remove extra resources (#1225) * fixing rollback for all functions, removing 'all' flag for function deploys (#1222) * updating rollback functionality to include all funcs * updating tests to check for rollback of all funcs * updating docs * fixing tf cycle and index issue (#1226) * Add missing dependency (#1228) * Implements a v2 Lambda Output with AssumeRole (#1227) * First draft of aws-lambda-v2 * Tests * Fixup * Fixup * Fioxup * Fixup * fixup * adding terraform references for some buckets (#1229) * adding athena terraform references instead of literals * fixing tests * GitHub Actions (#1231) * port to github actions * remove travis * cover the 3.2 branch for now too * initial updates to simplify lambda packaging logic (#1232) * moving some precompiled files * initial revamp to packaging to remove multiple pacakges * taking out more trash * update scheduled queries module * updating deploy logic to suck garbage slightly less * updates to unit tests * addressing pr feedback * addressing PR feedback * small update to docs (#1233) Co-authored-by: Ryxias <[email protected]> Co-authored-by: Paul Kehrer <[email protected]>

ryandeivert added cli deployment labels Apr 9, 2020

ryandeivert added this to the 3.2.0 milestone Apr 9, 2020

blakemotl reviewed Apr 9, 2020

View reviewed changes

streamalert_cli/manage_lambda/deploy.py Outdated Show resolved Hide resolved

blakemotl reviewed Apr 9, 2020

View reviewed changes

tests/unit/streamalert_cli/manage_lambda/test_deploy.py Outdated Show resolved Hide resolved

ryandeivert force-pushed the ryandeivert-packaging-unfuck branch from f21059f to c49050e Compare April 9, 2020 01:17

chunyong-lin reviewed Apr 9, 2020

View reviewed changes

ryandeivert added 7 commits April 9, 2020 10:50

moving some precompiled files

5624796

initial revamp to packaging to remove multiple pacakges

84eb9bf

taking out more trash

78d9c7a

update scheduled queries module

82320d9

updating deploy logic to suck garbage slightly less

1e7d751

updates to unit tests

965d838

addressing pr feedback

08b86de

ryandeivert force-pushed the ryandeivert-packaging-unfuck branch from c49050e to 137ae9a Compare April 9, 2020 17:56

chunyong-lin approved these changes Apr 9, 2020

View reviewed changes

Ryxias approved these changes Apr 9, 2020

View reviewed changes

addressing PR feedback

b92fd56

ryandeivert force-pushed the ryandeivert-packaging-unfuck branch from 137ae9a to b92fd56 Compare April 9, 2020 18:47

ryandeivert merged commit abd2188 into release-3-2-0 Apr 9, 2020

ryandeivert deleted the ryandeivert-packaging-unfuck branch April 9, 2020 20:28

ryandeivert restored the ryandeivert-packaging-unfuck branch April 9, 2020 21:17

ryandeivert deleted the ryandeivert-packaging-unfuck branch April 9, 2020 21:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial updates to simplify lambda packaging logic #1232

initial updates to simplify lambda packaging logic #1232

ryandeivert commented Apr 9, 2020

chunyong-lin left a comment

chunyong-lin Apr 9, 2020

ryandeivert Apr 9, 2020

ryandeivert Apr 9, 2020

Ryxias Apr 9, 2020

ryandeivert Apr 9, 2020

ryandeivert Apr 9, 2020

chunyong-lin left a comment

Ryxias Apr 9, 2020

	# Define a package dict to support pinning versions across all subclasses
	REQUIRED_LIBS = {
	'backoff==1.8.1',
	'boto3==1.10.6',
	'cbapi==1.5.4',
	'google-api-python-client==1.7.11',
	'jmespath==0.9.4',
	'jsonlines==1.2.0',
	'netaddr==0.7.19',
	'requests==2.22.0',
	'pymsteams==0.1.12',
	}

initial updates to simplify lambda packaging logic #1232

initial updates to simplify lambda packaging logic #1232

Conversation

ryandeivert commented Apr 9, 2020

Background

Changes

Testing

chunyong-lin left a comment

Choose a reason for hiding this comment

chunyong-lin Apr 9, 2020

Choose a reason for hiding this comment

ryandeivert Apr 9, 2020

Choose a reason for hiding this comment

ryandeivert Apr 9, 2020

Choose a reason for hiding this comment

Ryxias Apr 9, 2020

Choose a reason for hiding this comment

ryandeivert Apr 9, 2020

Choose a reason for hiding this comment

ryandeivert Apr 9, 2020

Choose a reason for hiding this comment

chunyong-lin left a comment

Choose a reason for hiding this comment

Ryxias Apr 9, 2020

Choose a reason for hiding this comment