-
Notifications
You must be signed in to change notification settings - Fork 210
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SUSTAIN-632] Do not update the users table to contain the public key. The keys table should be the only source of truth #1383
Conversation
17e09c6
to
8c81ce7
Compare
34fb1ec
to
eeafa2f
Compare
p_admin users.admin%TYPE, | ||
p_id character(32)) RETURNS integer AS $update_user$ | ||
BEGIN | ||
IF p_public_key != 'thisisnotakey' THEN |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we just set to null/rely on null here instead of this value?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The reason we are not using null is - null in v0 api implies that the user wants to delete the key, hence the use of a sentinel value.
@@ -0,0 +1,14 @@ | |||
-- Revert keys_update_trigger |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The revert function should set it back to the previous definitons instead of dropping.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
right the current keys_update_trigger.sql does that, this is the revert for the file before making my changes.
END; | ||
$add_key$ LANGUAGE plpgsql; | ||
|
||
CREATE OR REPLACE FUNCTION update_key() RETURNS TRIGGER AS $update_key$ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
recommend rename to update_or_delete_default_key
since we also do direct updates to the keys table via other APIs, and it will delete the key if a new one is not provided.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Disregard, it was misnamed from the start and changing it now will complicate the sqitch bits unnecessarily.
@@ -1004,7 +1004,7 @@ create_object(insert_cookbook_artifact_version = QueryName, Args) when is_list(A | |||
{conflict, _Why} = Conflict -> Conflict | |||
end; | |||
create_object(QueryName, Args) when is_atom(QueryName), is_list(Args) -> | |||
sqerl:statement(QueryName, Args, count); | |||
sqerl:select(QueryName, Args, count); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you need to change this to select
? Context for concern: creation of most other object types is done via insert statements, and this would affect them as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We changed the insert statements to be a function in plpgsql, and those return in the form : {ok,[[{<<"add_user">>,<<>>}]]}
but select expects return in the form: https://github.com/chef/sqerl/blob/master/src/sqerl.erl#L94-L101
which is why the change from statement to select.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated as discussed!
@@ -80,3 +80,7 @@ node_policy_name_policy_group 2015-09-04T01:15:03Z Daniel DeLeo <ddeleo@lorentz. | |||
|
|||
users_email_functional_index [users] 2017-05-30T12:10:32Z Stephan Renatus <[email protected]> # Add functional index for index-based lower(email) lookups | |||
@users_email_functional_index 2017-05-30T13:27:15Z Stephan Renatus <[email protected]> # Add functional index for index-based lower(email) lookups | |||
|
|||
create_and_update_users 2017-08-23T22:53:21Z Prajakta Purohit <[email protected]> # Adding functions to create and update users by inserting a sentinel value in the public_key_columns | |||
keys_update_trigger [keys_update_trigger@users_email_functional_index] 2017-08-23T23:12:29Z Prajakta Purohit <[email protected]> # The insert and update triggers ignore rows with sentinel value for public_key. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The creation of this change as @users_email_functional_index
seems off to me since we want it to go in sequence after the previous one - and if sqitch tracks this as dependent on users_email_functional_index, it may not go in that order. Could you verify that this won't be a thing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is a direct change due to sqitch rework keys_update_trigger
but will make sure.
@@ -167,6 +166,7 @@ create(ObjectRec0, #context{server_api_version = ApiVersion, reqid = ReqId}, Act | |||
Fields = chef_object:fields_for_insert(ObjectRec2), | |||
case stats_hero:ctime(ReqId, {chef_sql, create_object}, | |||
fun() -> chef_sql:create_object(QueryName, Fields) end) of | |||
{ok,[[{<<"add_user">>,<<>>}]]} -> ok; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is very specific to the add_user query/function. If we are not able to get a return value consistent with other inserts (number of rows inserted), it's worth a closer look at epgsql+sqerl to understand the returned [[{binary(), binary()}]]
from stored proc calls. Would it be sufficient to match similar to the below w/ guards:
36ed4d0
to
c84eda4
Compare
p_external_authentication_uid users.external_authentication_uid%TYPE, | ||
p_recovery_authentication_enabled users.recovery_authentication_enabled%TYPE, | ||
p_serialized_object users.serialized_object%TYPE, | ||
p_admin users.admin%TYPE) RETURNS void AS $$ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should return an integer. use the sqerl:select with first_as_scalar, and the function can return into an integer variable. That will let the return format be {ok, 1}
RETURN NULL; -- result is ignored since this is an AFTER trigger | ||
END; | ||
BEGIN | ||
IF NEW.public_key != 'thisisnotakey' THEN |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
check if we should add a new change instead of reworking the existing change.
ELSE | ||
DELETE FROM keys WHERE id = p_id AND key_name = 'default'; | ||
END IF; | ||
END IF; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we really need this if block? it feels like the trigger still does the right thing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we still need this because we will not be inserting the key into the users table in all the new requests (after this release) so just leaving the trigger in will update 'thisisnotakey' into the keys table. This function does the right things and the trigger will only be useful for older versions of chef-server
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In SQL NULL returns NULL for all comparisons. So the DELETE will never fire above I think, but instead NULL != to 'thisisnotakey' will return NULL, which is falsy, and the inner if will never be reached.
CREATE OR REPLACE FUNCTION test(val text)
RETURNS text
AS $$
DECLARE
result text;
BEGIN
IF val != 'A' THEN
IF val IS NOT NULL THEN
result := 'NOT A, NOT NULL';
ELSE
result := 'NOT A, BUT NULL';
END IF;
ELSE
result := 'A';
END IF;
RETURN result;
END;
$$;
opscode_chef=# select test(NULL);
test
------
A
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably should have unit test coverage of some of these functions to insure they work as expected. bifrost does some testing of it's stored procedures; maybe that could be used as a model.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is an awesome catch! updated the cases to fall through to the delete operation when expected.
I will check the unit tests in bifrost and add something to test this functionality.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be helpful to have a comment somewhere explaining the migration implications of the way the old triggers and new code interact, and the careful thinking that had to be done to work this solution out.
hash_type, last_updated_by, created_at, updated_at, external_authentication_uid, | ||
recovery_authentication_enabled, serialized_object, admin, pubkey_version) VALUES | ||
($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, false, 0)">>}. | ||
<<"SELECT add_user($1, $2, $3, $4, 'thisisnotakey', 0, $5, $6, $7, $8, $9, $10, $11, $12, $13, false)">>}. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a nitpick for sure, but this_is_not_a_key might be more obvious at a glance.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done!
p_serialized_object users.serialized_object%TYPE, | ||
p_admin users.admin%TYPE) RETURNS void AS $$ | ||
BEGIN | ||
INSERT INTO users |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's probably worth explaining the relationship between this code block and the legacy triggers, and explaining when the triggers can go away.
ELSE | ||
DELETE FROM keys WHERE id = p_id AND key_name = 'default'; | ||
END IF; | ||
END IF; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In SQL NULL returns NULL for all comparisons. So the DELETE will never fire above I think, but instead NULL != to 'thisisnotakey' will return NULL, which is falsy, and the inner if will never be reached.
CREATE OR REPLACE FUNCTION test(val text)
RETURNS text
AS $$
DECLARE
result text;
BEGIN
IF val != 'A' THEN
IF val IS NOT NULL THEN
result := 'NOT A, NOT NULL';
ELSE
result := 'NOT A, BUT NULL';
END IF;
ELSE
result := 'A';
END IF;
RETURN result;
END;
$$;
opscode_chef=# select test(NULL);
test
------
A
ELSE | ||
DELETE FROM keys WHERE id = p_id AND key_name = 'default'; | ||
END IF; | ||
END IF; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably should have unit test coverage of some of these functions to insure they work as expected. bifrost does some testing of it's stored procedures; maybe that could be used as a model.
RETURN NULL; -- result is ignored since this is an AFTER trigger | ||
END; | ||
BEGIN | ||
IF NEW.public_key != 'thisisnotakey' THEN |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See the NULL comparision comment for create_and_update_users.sql above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated.
END; | ||
BEGIN | ||
IF NEW.public_key != 'thisisnotakey' THEN | ||
IF NEW.public_key IS NOT NULL THEN |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See the NULL comparision comment for create_and_update_users.sql above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated.
e1a5066
to
03c5d32
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the way this is heading. @chef/chef-server-maintainers do we have any opinion about whether to use sqitch revise as was done here, or to leave the old files alone and revise the functions in a new file. In the past chef server has tended to do the latter, but that can make it hard to figure out what is eventually installed on the system as you have to integrate a whole series of change files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good so far!
I'd definitely benefit from some documentation either in the commit messages, a comment, or in the HOW_STUFF_WORKS doc about the details of how this works and what the next steps are. For instance, I still find myself asking "wait, what do we need the triggers for at the moment?" when reading this PR.
I think we should add tests at least the following:
-
That the user never sees "this_is_not_a_key" in a response body.
-
What happens when the user deletes their 'default' key.
@@ -168,6 +167,7 @@ create(ObjectRec0, #context{server_api_version = ApiVersion, reqid = ReqId}, Act | |||
case stats_hero:ctime(ReqId, {chef_sql, create_object}, | |||
fun() -> chef_sql:create_object(QueryName, Fields) end) of | |||
{ok, 1} -> ok; | |||
{ok, 0} -> <<"Record not created '", ObjectRec2/binary, "'.">>; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The dialyzer spec on this function needs to be updated. Perhaps it would be good to use {error, <<"Record..._
here as you did below. I think this case would be pretty rare as I imagine if we can't create what we are trying to create, we would get an error back rather than a successful query?
WHERE id = p_id AND key_name = 'default'; | ||
INSERT INTO keys (id, key_name, public_key, key_version, created_at, expires_at) | ||
SELECT p_id, 'default', p_public_key, p_pubkey_version, now(), 'infinity'::timestamp | ||
WHERE NOT EXISTS (SELECT 1 FROM keys WHERE id = p_id AND key_name = 'default'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know this is how we are doing "upsert" in the original code, but I think this construction is racy. My guess is that concurrent user updates are rare enough that we can stay with this though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, I commented on this down below... if it's available on the Postgres version you're using, an ON CONFLICT
clause would be better for this.
Alternatively, using CTEs for this might be preferable, as you could contain the operations within a single statement.
You're also in PL/pgSQL, so you could also use exception handling, but that's more expensive.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately, I'm not sure we can drop support for 9.2 yet (we should check with product) which means no ON CONFLICT
. I think overall using CTE
s would also make some of this a bit more readable; Perhaps we can do that as a more easily reviewable follow-up improvement. (Getting a bit worried that there is a lot going on here as this is already code not many know well.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have added the new syntax as a comment in there - a TODO for the future!
bb170a8
to
a1496c5
Compare
p_external_authentication_uid, p_recovery_authentication_enabled, | ||
p_serialized_object, p_admin, p_pubkey_version) RETURNING 1) | ||
SELECT count(*) FROM createduser INTO inserteduser; | ||
RETURN inserteduser; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You don't need this indirection of using CTEs to always insert the literal value 1
into a variable.
Why does this function need to return anything if it's always returning 1
? Is the desire to indicate whether something was inserted?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yea - to make sure that the insert did happen.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
RE: You don't need this indirection of using CTEs to always insert the literal value 1 into a variable. - will update that but maybe as a second pass, if that is ok?
BEGIN; | ||
|
||
DROP FUNCTION IF EXISTS add_user(); | ||
DROP FUNCTION IF EXISTS update_user(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The deploy script implies that these functions existed in the system before, but this revert will not replace those older implementations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes - this is just safeguarding for the future, we don't expect the functions to actually be present in this iteration
WHERE id = NEW.id AND key_name = 'default'; | ||
INSERT INTO keys (id, key_name, public_key, key_version, created_at, expires_at) | ||
SELECT NEW.id, 'default', NEW.public_key, NEW.pubkey_version, now(), 'infinity'::timestamp | ||
WHERE NOT EXISTS (SELECT 1 FROM keys WHERE id = NEW.id AND key_name = 'default'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like you're trying to do an upsert? Can you use the ON CONFLICT
clause instead (see https://www.postgresql.org/docs/current/static/sql-insert.html)... not sure what version of PG y'all are on these days).
I saw this was being done above, as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added a comment can be changes once we drop support for postgres 9.2/9.3
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually maybe I will leave this one as is - since it is the revert of the update trigger.
@@ -0,0 +1,216 @@ | |||
CREATE OR REPLACE FUNCTION test_zzzzcreate_and_update_users() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why "zzzz"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wanted that test to run last - will edit.
-- v0 add_user tests -- | ||
RETURN QUERY SELECT is(add_user('11111111111111111111111111111111', | ||
'22222222222222222222222222222222', | ||
'aaaa', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For these dummy values, I'd recommend having them describe what they are as much as you can given any constraints that exist on the data.
So if "aaaa" is a username, use "test_user" instead, etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done!
RETURN QUERY SELECT results_eq( | ||
'"after update email v0"', | ||
ARRAY['[email protected]'], | ||
'after update email address for v0' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd recommend stating these messages in terms of the desired behavior you're going after, instead of the mechanics of what is happening. That makes it easier to figure out what the overall intention of the code is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done!
WHERE id = p_id AND key_name = 'default'; | ||
INSERT INTO keys (id, key_name, public_key, key_version, created_at, expires_at) | ||
SELECT p_id, 'default', p_public_key, p_pubkey_version, now(), 'infinity'::timestamp | ||
WHERE NOT EXISTS (SELECT 1 FROM keys WHERE id = p_id AND key_name = 'default'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, I commented on this down below... if it's available on the Postgres version you're using, an ON CONFLICT
clause would be better for this.
Alternatively, using CTEs for this might be preferable, as you could contain the operations within a single statement.
You're also in PL/pgSQL, so you could also use exception handling, but that's more expensive.
last_updated_by, created_at, updated_at, external_authentication_uid, | ||
recovery_authentication_enabled, serialized_object, admin, | ||
pubkey_version) | ||
VALUES (p_id, p_authz_id, p_username, p_email, 'this_is_not_a_key', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the intention is to have the keys
table be the source of truth for keys, why keep key-related information in the users
table at all? Is this step 1 of a broader refactoring?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yep - sort of a phase1 since on env like hosted we need to support older versions of chef-server.
a1496c5
to
8b64df3
Compare
39d017d
to
cf03c32
Compare
@@ -258,6 +258,7 @@ update(#chef_cookbook_version{org_id =OrgId} = Record, #context{reqid = ReqId} = | |||
update(ObjectRec, #context{reqid = ReqId}, ActorId) -> | |||
case stats_hero:ctime(ReqId, {chef_sql, do_update}, | |||
fun() -> chef_sql:update(ObjectRec, ActorId) end) of | |||
0 -> {error, <<"Record not updated '", ObjectRec/binary, "'.">>}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like the changes do_update will return {error, Reason}
- can we remove this branch from here?
Same question for create
, looks like the {ok. 0} return is handled there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes!
case sqerl:select(QueryName, UpdateFields, first_as_scalar, [update_user]) of | ||
{ok, 1} -> {ok, 1}; | ||
{ok, 0} -> {error, <<"Record not updated '", (term_to_binary(QueryName))/binary, "'.">>}; | ||
{ok, none} -> {ok, not_found}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the caller seems is expecting not_found
instead of {ok, not_found}
.
@@ -1,33 +1,43 @@ | |||
-- Deploy keys_update_trigger | |||
-- | |||
-- This keeps the keys table consistent when the clients and user tables update their keys | |||
-- | |||
-- We need these triggers to maintain compatibility with older versions ok chef-server. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/ok/of
INSERT INTO keys (id, key_name, public_key, key_version, created_at, expires_at) | ||
SELECT NEW.id, 'default', NEW.public_key, NEW.pubkey_version, now(), 'infinity'::timestamp | ||
WHERE NOT EXISTS (SELECT 1 FROM keys WHERE id = NEW.id AND key_name = 'default'); | ||
ELSE |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a port of the existing logic -- but it at always makes me say, "What what?" when I read it. Could you add a comment explaining why a null key causes a deletion? (v0 api behavior compatibility)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done!
cf03c32
to
8231a44
Compare
Signed-off-by: Marc A. Paradise <[email protected]>
291b13c
to
49d2dcd
Compare
The keys table should be the only source of truth New insert/updates to the users table will insert a sentinel value in the public_key column. The old values are left as is. Triggers will act only on the values in the users.public_key that are not the sentinel value. Signed-off-by: Prajakta Purohit <[email protected]>
49d2dcd
to
d137d5f
Compare
TODO:
knife create client
is the only way to create clients, this bug does not affect clients.http://wilson.ci.chef.co/job/chef-server-trigger-ad_hoc/10/downstreambuildview/