Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cases] Delete file API #153604

Merged
merged 29 commits into from
Apr 3, 2023
Merged

Conversation

jonathan-buttner
Copy link
Contributor

@jonathan-buttner jonathan-buttner commented Mar 23, 2023

This PR adds a new API for deleting a file within a case given the file id.

This API will retrieve the file saved object provided in the query and perform an authorization check using each file's file kind. It will also retrieve all the attachments associated with the files and perform an authorization check for each attachment. This api supports calling it with ids that only have the file saved objects and not the corresponding attachments. For the deletion sub privilege to work correctly, it must have access to updating the file saved objects. Therefore we also had to give the delete sub privilege all access to the file saved objects types.

This PR does not contain the logic for deleting all files when a case is deleted. That'll be completed in a separate PR.

Example request

POST /internal/cases/a58847c0-cccc-11ed-b071-4f11aa24310c/attachments/files/_bulk_delete
{
  "ids": ["clfr5sdky0001n811gjot7tv5", "clfr5sgru0002n8112t54bave"]
}

Example response

204

Notable changes

  • Refactored the delete all comments to leverage the bulk delete API from the saved object client
  • Updated the names of the api_integration users and roles to avoid clashing with the ones in cases_api_integration

@jonathan-buttner jonathan-buttner added release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Feature:Cases Cases feature v8.8.0 labels Mar 23, 2023
export const MAX_FILE_SIZE = 100 * 1024 * 1024; // 100 MiB

export const constructFilesHttpOperationTag = (owner: Owner, operation: HttpApiTagOperation) => {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to common/files/index.ts since they aren't really constants


export const CaseFileMetadataRt = rt.type({
// TODO: do we want this as an array?
caseId: rt.string,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd need this as an array if we wanted to support having a single file be associated with multiple cases (aka the file picker). I don't know if we'll want to do that, but just in case should we format this as an array for now?

// TODO: do we want this as an array?
caseId: rt.string,
// TODO: do we want this as an array?
owner: rt.array(rt.string),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking that we're in agreement on this being an array.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The owner also? In case we are deleting multiple files with different owners?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I don't think we have a use case currently for an array but we discussed it here: https://github.com/elastic/kibana/pull/152941/files#r1145115748

@@ -12,7 +12,7 @@ import { Role } from '../../../../cases_api_integration/common/lib/authenticatio
*/

export const secAllCasesOnlyDelete: Role = {
name: 'sec_all_cases_only_delete',
name: 'sec_all_cases_only_delete_api_int',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm appending api_int to these because I need to use these users/roles in the case_api_integration tests and I want to avoid collisions with the users/roles defined in cases_api_integration. I need to use these tests because the fixture users/roles aren't registered with the file kind.

}
};

export const deleteAllFiles = async ({
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is mostly used for clean up to remove the files once the tests are completed.

@@ -0,0 +1,136 @@
/*
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The deletion sub privilege is only available under a trial license so these tests are specifically for a user with the deletion sub privilege.

@jonathan-buttner jonathan-buttner marked this pull request as ready for review March 27, 2023 20:05
@jonathan-buttner jonathan-buttner requested review from a team as code owners March 27, 2023 20:05
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops-cases (Feature:Cases)

});

const [validFiles, invalidFiles] = partition(files, (file) => {
return CaseFileMetadataRt.is(file.data.meta) && file.data.meta.caseId === caseId;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cnasikas should we be more lenient here? For example lets say a user manually created a file and formatted the metadata incorrectly. If we are going to prevent deletion through the file management page, the user wouldn't have anyway to delete the file.

Maybe instead we just check that the metadata contains the caseId field and ignores the format of the owner since this function doesn't actually need it.

We can still enforce that the caseId has to be present and match the passed in caseId from the request.

@jonathan-buttner
Copy link
Contributor Author

@elasticmachine merge upstream

Copy link
Member

@cnasikas cnasikas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not checked the code yet but I think we should change the API to be POST /internal/cases/<case_ids>/attachments/files/_bulk_delete or POST /internal/cases/<case_ids>/files/_bulk_deletewith body { ids: [] }. The reason is that if the ids are a lot some proxies or gateway may trim the query params and lose some of the ids or even break an id in the middle making it invalid. Also, the _<whatever> is usually used in ES & Kibana to denote an operation and not a resource. Finally, we should put a limit to the number of files someone can delete with the API (you may have it already. I did not check 🙂 ).

@jonathan-buttner
Copy link
Contributor Author

I have not checked the code yet but I think we should change the API to be POST /internal/cases/<case_ids>/attachments/files/_bulk_delete

Great point. Will do!

Finally, we should put a limit to the number of files someone can delete with the API (you may have it already. I did not check 🙂 ).

Yep, it's currently set at 500 (I just did 5 times the max number of files on a case), happy to lower it though.

Copy link
Contributor

@semd semd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SecuritySolution changes LGTM

Copy link
Contributor

@kdelemme kdelemme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actionable-observability code change looks good to me !

const fileEntities = await getFileEntities(caseId, request.ids, fileService);

// It's possible for this to return an empty array if there was an error creating file attachments in which case the
// file would be present but the case attachment would not
Copy link
Contributor

@adcoelho adcoelho Mar 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// It's possible for this to return an empty array if there was an error creating file attachments in which case the
// file would be present but the case attachment would not

I'd like to think this won't happen based on the AddFile implementation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haha yeah let's hope not!

* to retrieve them all.
*/
const finder =
this.context.unsecuredSavedObjectsClient.createPointInTimeFinder<AttachmentAttributesWithoutRefs>(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't know this createPointInTimeFinder API so I was reading the saved_objects_client docs.

Considering we have the file ids references already are we really anticipating:

page through large sets of saved objects. We strongly recommend using this API for
any find queries that might return more than 1000 saved objects, [...]

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this is more of a safety measure. It's probably unlikely that we'd get more than 100 (or 50 if we lower the max number of files on a case). We could use the find api to do the same thing but it's limited to 10k results where as this will get all of them.

rt.identity
);

export const limitedArraySchema = <T extends rt.Mixed>(codec: T, min: number, max: number) =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks like an incredibly complicated way of checking an array size 😆

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like magic.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haha, yep this is exactly why Christos and I wanted to switch from io-ts to zod or anything else that isn't this convoluted. I did this as an example of what we're going to need to do for the road to versioned HTTP apis.


export const constructFileKindIdByOwner = (owner: Owner) => `${owner}FilesCases`;
export const MAX_FILES_PER_CASE = 100;
export const MAX_DELETE_FILES = MAX_FILES_PER_CASE * 5;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that the file deletion is slow what do you think about reducing the number to 50 (the maximum number of files shown on one page in the UI table)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds reasonable to me!


import { constructOwnerFromFileKind } from '.';

describe('files index', () => {
Copy link
Member

@cnasikas cnasikas Mar 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mind if you add one test for constructFilesHttpOperationTag and one for constructFileKindIdByOwner?

export type CaseFileMetadata = rt.TypeOf<typeof CaseFileMetadataForDeletionRt>;

export const constructFilesHttpOperationTag = (owner: Owner, operation: HttpApiTagOperation) => {
return `${owner}FilesCases${operation}`;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also put the FILE_KIND_DELIMITER here.

id: attachment.id,
owner: attachment.attributes.owner,
})),
...fileEntities,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to checking if the user can delete the attachments is this checking if he can also delete the files in the fileService?

And if so, is operation: Operations.deleteComment the correct one for that, isn't this Cases specific?

Copy link
Contributor Author

@jonathan-buttner jonathan-buttner Mar 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this checking if he can also delete the files in the fileService?

Sort of. The actual check for whether a user can delete a file service saved object is done within the saved object client. The way we give a user access to deleting the file service saved objects is when we implement the features within each plugin for example: https://github.com/elastic/kibana/blob/main/x-pack/plugins/cases/server/features.ts#L54

So if a user has a role that grants the All feature privilege within cases they will be authorized to delete file service saved objects. Technically that user could use the exposed saved object API to delete those file saved objects: https://www.elastic.co/guide/en/kibana/master/saved-objects-api-delete.html

The check we're doing here is one extra safety measure that ensures that the user does have access to the owner specified for that file saved object based on the file kind. It's unlikely that they wouldn't have access though because they'd have to manually create a file that has a reference to a case plugin outside of their access.

operation: Operations.deleteComment the correct one for that, isn't this Cases specific?

For now I think it's ok to leverage the deleteComment operation because we're defining the whole operation as a deleteComment. For example when you delete a case we specify the operation as delete case but in reality it's a combination of operations (deleting the case, comments, user actions, etc). If in the future we want to allow a user to delete a comment/attachment but not a file then we'll need to rethink this model and would likely need another operation here.

@@ -470,3 +477,16 @@ export const getCaseViewPath = (params: {

return `${basePath}${normalizePath(CASE_VIEW_PATH.replace(':detailName', caseId))}`;
};

export const partitionByCaseAssociation = (caseId: string, attachments: AttachmentSavedObject[]) =>
Copy link
Member

@cnasikas cnasikas Mar 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add some unit tests for the new functions? (I know there was no test for them. My fault.)

) => {
// it's possible that we're trying to delete a file when an attachment wasn't created (for example if the create
// attachment request failed)
const files = await pMap(fileIds, async (fileId: string) => fileService.getById({ id: fileId }), {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elastic/appex-sharedux Is it possible to expose a bulk get operation to improve performance?

});

await Promise.all([
pMap(request.ids, async (fileId: string) => fileService.delete({ id: fileId }), {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elastic/appex-sharedux Is it possible to expose a bulk delete operation to improve performance?

Copy link
Member

@cnasikas cnasikas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job!

supertest,
caseId: postedCase.id,
fileIds: ['abc'],
expectedHttpCode: 500,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not this be 404?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great point. The issue here was that the file service isn't returning the saved object error (which would have been a boom) it was wrapping it in its own FileNotFound error class. Our cases code only looks for booms to grab the status from. So since our code was encountering a non-boom it defaults to 500. I added a simple fix to do an error instanceof FileNotFound so we can get the 400. The file service does a similar check before it returns from their route.

});

it('returns a 400 when there are 501 ids being deleted', async () => {
const ids = Array.from(Array(501).keys()).map((item) => item.toString());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to change this to 51, right?

});
});

describe('rbac', () => {
Copy link
Member

@cnasikas cnasikas Mar 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add a test where we upload a file with a different file kind, not owned by cases (I think we can register a dummy one in our cases fixture plugin), with the exact same metadata (valid case ID) as expected by cases, attach the file to the case (only create an attachment), and see if we get a 403 when we try to delete it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great idea I'll add the test, we'll actually get a 400 bad request because we can't determine the owner. Does a 400 seem appropriate to you?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

400 is also fine 🙂.

@cnasikas cnasikas mentioned this pull request Apr 3, 2023
5 tasks
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
cases 565 567 +2

Adoption-tracked APIs that are not used anywhere

APIs whose adoption is being tracked and are still not being used. This number should tend to zero.

id before after diff
files 1 0 -1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
cases 364.3KB 364.3KB +4.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
cases 133.3KB 134.1KB +769.0B
Unknown metric groups

ESLint disabled line counts

id before after diff
securitySolution 432 435 +3

Total ESLint disabled count

id before after diff
securitySolution 512 515 +3

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jonathan-buttner jonathan-buttner merged commit 1e63515 into elastic:main Apr 3, 2023
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Apr 3, 2023
@jonathan-buttner jonathan-buttner deleted the cases-file-delete branch April 3, 2023 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting Feature:Cases Cases feature release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v8.8.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants