Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow setting createdAt and updatedAt during Parse.Object creation with maintenance key #8696

Merged
merged 17 commits into from
Sep 29, 2023

Conversation

westhom
Copy link
Contributor

@westhom westhom commented Jul 22, 2023

Pull Request

Issue

Closes: #7447

Approach

Allows client-specified createdAt and updatedAt during object creation when using maintenanceKey.

The benefits of this are discussed in the linked issue, but in summary this is really useful for backfilling data, where you want createdAt to match to pre-existing data.

Tasks

  • Add tests

@parse-github-assistant
Copy link

I will reformat the title to use the proper commit message syntax.

@parse-github-assistant parse-github-assistant bot changed the title feat: allow setting createdAt during object creation feat: Allow setting createdAt during object creation Jul 22, 2023
@parse-github-assistant
Copy link

Thanks for opening this pull request!

@westhom westhom changed the title feat: Allow setting createdAt during object creation feat: Allow setting createdAt during object creation Jul 22, 2023
@codecov
Copy link

codecov bot commented Jul 22, 2023

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (3d6d50e) 94.31% compared to head (244950b) 94.31%.
Report is 1 commits behind head on alpha.

❗ Current head 244950b differs from pull request most recent head 9b777e6. Consider uploading reports for the commit 9b777e6 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##            alpha    #8696      +/-   ##
==========================================
- Coverage   94.31%   94.31%   -0.01%     
==========================================
  Files         186      186              
  Lines       14770    14780      +10     
==========================================
+ Hits        13931    13940       +9     
- Misses        839      840       +1     
Files Coverage Δ
src/Options/Definitions.js 100.00% <ø> (ø)
src/Options/index.js 100.00% <ø> (ø)
src/RestWrite.js 94.95% <100.00%> (-0.08%) ⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@mtrezza mtrezza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This requires an option in Parse Server which is disabled by default. In most cases the developer has no control over the client, so if the client can set a custom createdAt field that means the field value cannot be trusted anymore as it may have been set by the client.

I also changed the PR title and included the field updatedAt as it seems to be set to the same manipulated date, so both fields cannot be trusted anymore. This needs to be disclosed as a warning in the Parse Server option documentation.

spec/rest.spec.js Outdated Show resolved Hide resolved
@mtrezza mtrezza changed the title feat: Allow setting createdAt during object creation feat: Allow setting createdAt during Parse.Object creation Jul 22, 2023
@mtrezza mtrezza changed the title feat: Allow setting createdAt during Parse.Object creation feat: Allow setting createdAt and updatedAt during Parse.Object creation Jul 22, 2023
@westhom
Copy link
Contributor Author

westhom commented Jul 22, 2023

so if the client can set a custom createdAt field that means the field value cannot be trusted anymore as it may have been set by the client

Good point. I can see why this change wants to be behind an option flag. In most cases you would only want to switch it on temporarily while backfilling anyway.

Do you have recommended option name? I can make that corresponding change.

@mtrezza
Copy link
Member

mtrezza commented Jul 22, 2023

An alternative could be to enable this only using the maintenanceKey, which was created for purposes such as data migration:

⚠️ This key is not intended to be used as part of a regular operation of Parse Server. This key is intended to conduct out-of-band changes such as one-time migrations or data correction tasks. Internal fields are not officially documented and may change at any time without publication in release changelogs. We strongly advice not to rely on internal fields as part of your regular operation and to investigate the implications of any planned changes directly in the source code of your current version of Parse Server.

This would have the advantage that objects with custom createdAt date can be created while the server is publicly accessible, without risking that someone may be creating objects with modified createdAt date while the Parse Server option is enabled. So it's also usable in a live environment.

I'm not sure we fully understand what it means to allow to manipulate these fields. createdAt and updatedAt are fields that - IIRC - are used internally in Parse client SDKs as part of some internal logic, e.g. to determine whether an object has been saved, etc. If the field is set manually (not by SDK or server), then we may introduce unintended side-effects. That's just my guess, I haven't looked into that.

That may be a good reason to restrict this functionality to the maintenanceKey. On the other hand createdAt and updatedAt may not exactly be "internal data" (which are fields that are completely inaccessible), so we'd need to refine the access scope definition.

@westhom westhom marked this pull request as draft July 29, 2023 20:23
@westhom
Copy link
Contributor Author

westhom commented Aug 5, 2023

@mtrezza Thanks for your feedback. I've updated the REST module to allow setting createdAt / updatedAt with a maintenance key, added additional tests, and added a new "Object timestamps" classification to the Access Scopes table in the readme.

Functional summary:

  • maintenanceKey can set createdAt on new objects
  • maintenanceKey can also set updatedAt on new objects if createdAt is also specified, and if updatedAt >= createdAt

(Setting updatedAt without createdAt felt too undefined as behavior on object creation.)

One last consideration: do we want to enable this maintenance feature on object update too?

@westhom westhom marked this pull request as ready for review August 5, 2023 17:49
@westhom
Copy link
Contributor Author

westhom commented Aug 5, 2023

https://github.com/parse-community/parse-server/blob/47022999a3a112b793dfa79ccbdb7149618d7a7c/README.md?plain=1#L361C1-L363

Under the first Scope column, Internal could be renamed Maintenance, since that scope can act on both internal and timestamps (non-internal) data now.

@mtrezza
Copy link
Member

mtrezza commented Aug 6, 2023

Actually we could say that it's modifying an internal key, as _created_at is the internal key and createdAt is the external read-only representation of it. Same with _updated_at for updatedAt and _id for objectId. But that may get too confusing for developers, so maybe we should change the scope to "internal and read-only fields".

Copy link
Member

@mtrezza mtrezza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • The scope names (1st column) are not descriptions but terms that reference a specific access scope, and I think we are referencing them in other places, so I'd suggest we keep them as they are.
  • For readers it may be unclear what "Object timestamps" are, so I've renamed this category to "Read only data", and added a footnote to spell out the individual fields. For example, objectId is also a read-only field, but not an "object timestamp"; if we allow to update it in the future, we can just amend the footnote.

Just as a side note: given that this is the maintenance key, we could also allow to write the createdAt and updatedAt fields for existing objects, not just on object creation. I just came across a use case for that. It's already possible using the databaseAdapter directly, but it would make more sense to allow the maintenance key to do it. Just saying this in case you have a use case for that and want to extend this PR.

README.md Outdated Show resolved Hide resolved
Signed-off-by: Manuel <[email protected]>
spec/rest.spec.js Outdated Show resolved Hide resolved
@westhom
Copy link
Contributor Author

westhom commented Aug 10, 2023

@mtrezza The specs have been rewritten and ready for review.

@mtrezza mtrezza requested a review from a team August 14, 2023 12:37
mtrezza
mtrezza previously approved these changes Aug 16, 2023
src/Options/Definitions.js Outdated Show resolved Hide resolved
src/Options/docs.js Outdated Show resolved Hide resolved
src/Options/index.js Outdated Show resolved Hide resolved
@mtrezza mtrezza requested a review from a team August 18, 2023 22:53
@mtrezza
Copy link
Member

mtrezza commented Aug 27, 2023

I'm rerunning the CI, but it seems to be passing apart from the flaky tests. It would be good if someone from @parse-community/server could review this before we merge.

@mtrezza
Copy link
Member

mtrezza commented Aug 30, 2023

@dplewis If you have some time, your opinion on this PR would be much appreciated.

Copy link
Member

@mtrezza mtrezza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@mtrezza mtrezza changed the title feat: Allow setting createdAt and updatedAt during Parse.Object creation feat: Allow setting createdAt and updatedAt during Parse.Object creation with maintenanceKey Sep 29, 2023
@mtrezza mtrezza changed the title feat: Allow setting createdAt and updatedAt during Parse.Object creation with maintenanceKey feat: Allow setting createdAt and updatedAt during Parse.Object creation with maintenance key Sep 29, 2023
@mtrezza mtrezza merged commit 77bbfb3 into parse-community:alpha Sep 29, 2023
parseplatformorg pushed a commit that referenced this pull request Sep 29, 2023
# [6.4.0-alpha.4](6.4.0-alpha.3...6.4.0-alpha.4) (2023-09-29)

### Features

* Allow setting `createdAt` and `updatedAt` during `Parse.Object` creation with maintenance key ([#8696](#8696)) ([77bbfb3](77bbfb3))
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 6.4.0-alpha.4

@parseplatformorg parseplatformorg added the state:released-alpha Released as alpha version label Sep 29, 2023
parseplatformorg pushed a commit that referenced this pull request Nov 16, 2023
# [6.5.0-beta.1](6.4.0...6.5.0-beta.1) (2023-11-16)

### Bug Fixes

* Context not passed to Cloud Code Trigger `beforeFind` when using `Parse.Query.include` ([#8765](#8765)) ([7d32d89](7d32d89))
* Parse Server option `fileUpload.fileExtensions` fails to determine file extension if filename contains multiple dots ([#8754](#8754)) ([3d6d50e](3d6d50e))
* Security bump @babel/traverse from 7.20.5 to 7.23.2 ([#8777](#8777)) ([2d6b3d1](2d6b3d1))
* Security upgrade graphql from 16.6.0 to 16.8.1 ([#8758](#8758)) ([71dfd8a](71dfd8a))

### Features

* Add `$setOnInsert` operator to `Parse.Server.database.update` ([#8791](#8791)) ([f630a45](f630a45))
* Add compatibility for MongoDB Atlas Serverless and AWS Amazon DocumentDB with collation options `enableCollationCaseComparison`, `transformEmailToLowercase`, `transformUsernameToLowercase` ([#8805](#8805)) ([09fbeeb](09fbeeb))
* Add context to Cloud Code Triggers `beforeLogin` and `afterLogin` ([#8724](#8724)) ([a9c34ef](a9c34ef))
* Allow setting `createdAt` and `updatedAt` during `Parse.Object` creation with maintenance key ([#8696](#8696)) ([77bbfb3](77bbfb3))
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 6.5.0-beta.1

@parseplatformorg parseplatformorg added the state:released-beta Released as beta version label Nov 16, 2023
parseplatformorg pushed a commit that referenced this pull request Nov 18, 2023
# [6.5.0-alpha.1](6.4.0...6.5.0-alpha.1) (2023-11-18)

### Bug Fixes

* Context not passed to Cloud Code Trigger `beforeFind` when using `Parse.Query.include` ([#8765](#8765)) ([7d32d89](7d32d89))
* Parse Server option `fileUpload.fileExtensions` fails to determine file extension if filename contains multiple dots ([#8754](#8754)) ([3d6d50e](3d6d50e))
* Security bump @babel/traverse from 7.20.5 to 7.23.2 ([#8777](#8777)) ([2d6b3d1](2d6b3d1))
* Security upgrade graphql from 16.6.0 to 16.8.1 ([#8758](#8758)) ([71dfd8a](71dfd8a))

### Features

* Add `$setOnInsert` operator to `Parse.Server.database.update` ([#8791](#8791)) ([f630a45](f630a45))
* Add compatibility for MongoDB Atlas Serverless and AWS Amazon DocumentDB with collation options `enableCollationCaseComparison`, `transformEmailToLowercase`, `transformUsernameToLowercase` ([#8805](#8805)) ([09fbeeb](09fbeeb))
* Add context to Cloud Code Triggers `beforeLogin` and `afterLogin` ([#8724](#8724)) ([a9c34ef](a9c34ef))
* Allow setting `createdAt` and `updatedAt` during `Parse.Object` creation with maintenance key ([#8696](#8696)) ([77bbfb3](77bbfb3))
* Upgrade Parse Server Push Adapter to 5.0.2 ([#8813](#8813)) ([6ef1986](6ef1986))
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 6.5.0-alpha.1

parseplatformorg pushed a commit that referenced this pull request Mar 1, 2024
# [6.5.0](6.4.0...6.5.0) (2024-03-01)

### Bug Fixes

* Context not passed to Cloud Code Trigger `beforeFind` when using `Parse.Query.include` ([#8765](#8765)) ([7d32d89](7d32d89))
* Improve PostgreSQL injection detection; fixes security vulnerability [GHSA-6927-3vr9-fxf2](GHSA-6927-3vr9-fxf2) which affects Parse Server deployments using a Postgres database ([#8960](#8960)) ([a6e6549](a6e6549))
* Parse Server option `fileUpload.fileExtensions` fails to determine file extension if filename contains multiple dots ([#8754](#8754)) ([3d6d50e](3d6d50e))
* Security bump @babel/traverse from 7.20.5 to 7.23.2 ([#8777](#8777)) ([2d6b3d1](2d6b3d1))
* Security upgrade graphql from 16.6.0 to 16.8.1 ([#8758](#8758)) ([71dfd8a](71dfd8a))

### Features

* Add `$setOnInsert` operator to `Parse.Server.database.update` ([#8791](#8791)) ([f630a45](f630a45))
* Add compatibility for MongoDB Atlas Serverless and AWS Amazon DocumentDB with collation options `enableCollationCaseComparison`, `transformEmailToLowercase`, `transformUsernameToLowercase` ([#8805](#8805)) ([09fbeeb](09fbeeb))
* Add context to Cloud Code Triggers `beforeLogin` and `afterLogin` ([#8724](#8724)) ([a9c34ef](a9c34ef))
* Allow setting `createdAt` and `updatedAt` during `Parse.Object` creation with maintenance key ([#8696](#8696)) ([77bbfb3](77bbfb3))
* Upgrade Parse Server Push Adapter to 5.0.2 ([#8813](#8813)) ([6ef1986](6ef1986))

### Performance Improvements

* Improved IP validation performance for `masterKeyIPs`, `maintenanceKeyIPs` ([#8510](#8510)) ([b87daba](b87daba))
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 6.5.0

parseplatformorg pushed a commit that referenced this pull request Mar 19, 2024
# [7.0.0](6.4.0...7.0.0) (2024-03-19)

### Bug Fixes

* CacheAdapter does not connect when using a CacheAdapter with a JSON config ([#8633](#8633)) ([720d24e](720d24e))
* Conditional email verification not working in some cases if `verifyUserEmails`, `preventLoginWithUnverifiedEmail` set to functions ([#8838](#8838)) ([8e7a6b1](8e7a6b1))
* Context not passed to Cloud Code Trigger `beforeFind` when using `Parse.Query.include` ([#8765](#8765)) ([7d32d89](7d32d89))
* Deny request if master key is not set in Parse Server option `masterKeyIps` regardless of ACL and CLP ([#8957](#8957)) ([a7b5b38](a7b5b38))
* Docker image not published to Docker Hub on new release ([#8905](#8905)) ([a2ac8d1](a2ac8d1))
* Docker version releases by removing arm/v6 and arm/v7 support ([#8976](#8976)) ([1f62dd0](1f62dd0))
* GraphQL file upload fails in case of use of pointer or relation ([#8721](#8721)) ([1aba638](1aba638))
* Improve PostgreSQL injection detection; fixes security vulnerability [GHSA-6927-3vr9-fxf2](GHSA-6927-3vr9-fxf2) which affects Parse Server deployments using a Postgres database ([#8961](#8961)) ([cbefe77](cbefe77))
* Incomplete user object in `verifyEmail` function if both username and email are changed ([#8889](#8889)) ([1eb95ae](1eb95ae))
* Parse Server option `emailVerifyTokenReuseIfValid: true` generates new token on every email verification request ([#8885](#8885)) ([0023ce4](0023ce4))
* Parse Server option `fileExtensions` default value rejects file extensions that are less than 3 or more than 4 characters long ([#8699](#8699)) ([2760381](2760381))
* Parse Server option `fileUpload.fileExtensions` fails to determine file extension if filename contains multiple dots ([#8754](#8754)) ([3d6d50e](3d6d50e))
* Security bump @babel/traverse from 7.20.5 to 7.23.2 ([#8777](#8777)) ([2d6b3d1](2d6b3d1))
* Security upgrade graphql from 16.6.0 to 16.8.1 ([#8758](#8758)) ([71dfd8a](71dfd8a))
* Server crashes on invalid Cloud Function or Cloud Job name; fixes security vulnerability [GHSA-6hh7-46r2-vf29](GHSA-6hh7-46r2-vf29) ([#9024](#9024)) ([9f6e342](9f6e342))
* Server crashes when receiving an array of `Parse.Pointer` in the request body ([#8784](#8784)) ([66e3603](66e3603))
* Username is `undefined` in email verification link on email change ([#8887](#8887)) ([e315c13](e315c13))

### Features

* Add `$setOnInsert` operator to `Parse.Server.database.update` ([#8791](#8791)) ([f630a45](f630a45))
* Add `installationId` to arguments for `verifyUserEmails`, `preventLoginWithUnverifiedEmail` ([#8836](#8836)) ([a22dbe1](a22dbe1))
* Add `installationId`, `ip`, `resendRequest` to arguments passed to `verifyUserEmails` on verification email request ([#8873](#8873)) ([8adcbee](8adcbee))
* Add `Parse.User` as function parameter to Parse Server options `verifyUserEmails`, `preventLoginWithUnverifiedEmail` on login ([#8850](#8850)) ([972f630](972f630))
* Add compatibility for MongoDB Atlas Serverless and AWS Amazon DocumentDB with collation options `enableCollationCaseComparison`, `transformEmailToLowercase`, `transformUsernameToLowercase` ([#8805](#8805)) ([09fbeeb](09fbeeb))
* Add context to Cloud Code Triggers `beforeLogin` and `afterLogin` ([#8724](#8724)) ([a9c34ef](a9c34ef))
* Add password validation via POST request for user with unverified email using master key and option `ignoreEmailVerification` ([#8895](#8895)) ([633a9d2](633a9d2))
* Add support for MongoDB 7 ([#8761](#8761)) ([3de8494](3de8494))
* Add support for MongoDB query comment ([#8928](#8928)) ([2170962](2170962))
* Add support for Node 20, drop support for Node 14, 16 ([#8907](#8907)) ([ced4872](ced4872))
* Add support for Postgres 16 ([#8898](#8898)) ([99489b2](99489b2))
* Allow `Parse.Session.current` on expired session token instead of throwing error ([#8722](#8722)) ([f9dde4a](f9dde4a))
* Allow setting `createdAt` and `updatedAt` during `Parse.Object` creation with maintenance key ([#8696](#8696)) ([77bbfb3](77bbfb3))
* Deprecation DEPPS5: Config option `allowClientClassCreation` defaults to `false` ([#8849](#8849)) ([29624e0](29624e0))
* Deprecation DEPPS6: Authentication adapters disabled by default ([#8858](#8858)) ([0cf58eb](0cf58eb))
* Deprecation DEPPS7: Remove deprecated Cloud Code file trigger syntax ([#8855](#8855)) ([4e6a375](4e6a375))
* Deprecation DEPPS8:  Parse Server option `allowExpiredAuthDataToken` defaults to `false` ([#8860](#8860)) ([e29845f](e29845f))
* Deprecation DEPPS9: LiveQuery `fields` option is renamed to `keys` ([#8852](#8852)) ([38983e8](38983e8))
* Node process exits with error code 1 on uncaught exception to allow custom uncaught exception handling ([#8894](#8894)) ([70c280c](70c280c))
* Switch GraphQL server from Yoga v2 to Apollo v4 ([#8959](#8959)) ([105ae7c](105ae7c))
* Upgrade Parse Server Push Adapter to 5.0.2 ([#8813](#8813)) ([6ef1986](6ef1986))
* Upgrade to Parse JS SDK 5 ([#9022](#9022)) ([ad4aa83](ad4aa83))

### Performance Improvements

* Improved IP validation performance for `masterKeyIPs`, `maintenanceKeyIPs` ([#8510](#8510)) ([b87daba](b87daba))

### BREAKING CHANGES

* The Parse Server option `allowClientClassCreation` defaults to `false`. ([29624e0](29624e0))
* A request using the master key will now be rejected as unauthorized if the IP from which the request originates is not set in the Parse Server option `masterKeyIps`, even if the request does not require the master key permission, for example for a public object in a public class class. ([a7b5b38](a7b5b38))
* Node process now exits with code 1 on uncaught exceptions, enabling custom handlers that were blocked by Parse Server's default behavior of re-throwing errors. This change may lead to automatic process restarts by the environment, unlike before. ([70c280c](70c280c))
* Authentication adapters are disabled by default; to use an authentication adapter it needs to be explicitly enabled in the Parse Server authentication adapter option `auth.<provider>.enabled: true` ([0cf58eb](0cf58eb))
* Parse Server option `allowExpiredAuthDataToken` defaults to `false`; a 3rd party authentication token will be validated every time the user tries to log in and the login will fail if the token has expired; the effect of this change may differ for different authentication adapters, depending on the token lifetime and the token refresh logic of the adapter ([e29845f](e29845f))
* LiveQuery `fields` option is renamed to `keys` ([38983e8](38983e8))
* Cloud Code file trigger syntax has been aligned with object trigger syntax, for example `Parse.Cloud.beforeDeleteFile'` has been changed to `Parse.Cloud.beforeDelete(Parse.File, (request) => {})'` ([4e6a375](4e6a375))
* Removes support for Node 14 and 16 ([ced4872](ced4872))
* Removes support for Postgres 11 and 12 ([99489b2](99489b2))
* The `Parse.User` passed as argument if `verifyUserEmails` is set to a function is renamed from `user` to `object` for consistency with invocations of `verifyUserEmails` on signup or login; the user object is not a plain JavaScript object anymore but an instance of `Parse.User` ([8adcbee](8adcbee))
* `Parse.Session.current()` no longer throws an error if the session token is expired, but instead returns the session token with its expiration date to allow checking its validity ([f9dde4a](f9dde4a))
* `Parse.Query` no longer supports the BSON type `code`; although this feature was never officially documented, its removal is announced as a breaking change to protect deployments where it might be in use. ([3de8494](3de8494))
@parseplatformorg
Copy link
Contributor

🎉 This change has been released in version 7.0.0

@parseplatformorg parseplatformorg added the state:released Released as stable version label Mar 19, 2024
@Jakub-Plan-d-k
Copy link

Hello, I would like to use this feature on my local server to set up test data. I have an algorithm, which relies on cretedAt relative to now.
I can see that I need to set up maintenanceKey in server config, which I did, but how do I pass it to Parse.Object.saveAll()? I have also updated to v 7.0.0

@mtrezza
Copy link
Member

mtrezza commented Mar 22, 2024

@Jakub-Plan-d-k

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
state:released Released as stable version state:released-alpha Released as alpha version state:released-beta Released as beta version
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow to set createdAt and updatedAt during data import(s)
4 participants