Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup S3 connection with integrations #1057

Merged
merged 97 commits into from
Oct 4, 2023
Merged

Setup S3 connection with integrations #1057

Show file tree
Hide file tree
Changes from 92 commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
c5cea8d
Stub catalog reader interface
Swiddis Aug 17, 2023
77ad89b
Add basic catalog functionality to new catalog reader
Swiddis Aug 17, 2023
676591d
Refactor validation logic with a deeper interface
Swiddis Aug 21, 2023
12c4bcf
Refactor validation logic with a deeper interface
Swiddis Aug 21, 2023
412480e
Remove redundant test.
Swiddis Aug 21, 2023
6d2bad1
Add tests for new validators
Swiddis Aug 21, 2023
c3dd111
Merge branch 'better-validation' into prototype-network-catalog
Swiddis Aug 21, 2023
55d6490
Make better failure mode for invalid objects
Swiddis Aug 21, 2023
c62d28b
Merge branch 'main' into prototype-network-catalog
Swiddis Aug 22, 2023
ba56fb8
Generalize Result type
Swiddis Aug 22, 2023
cb29208
Convert backend to use catalog reader (unstable)
Swiddis Aug 22, 2023
c9de36e
Repair tests for integrations class (unstable)
Swiddis Aug 22, 2023
37c7980
Refactor repository for new integration interface
Swiddis Aug 22, 2023
b331090
Fix outer repository and backend tests
Swiddis Aug 23, 2023
4b29e17
Add tests for sample data
Swiddis Aug 23, 2023
b35dfa8
Merge remote-tracking branch 'upstream/main' into prototype-network-c…
Swiddis Aug 23, 2023
1bef29e
Add CatalogReader JavaDocs
Swiddis Aug 23, 2023
a18cf47
Repair integrations builder
Swiddis Aug 23, 2023
047b14a
Remove extra commented test
Swiddis Aug 23, 2023
e65fe8b
Remove unnecessary log statement
Swiddis Aug 23, 2023
00ddb5b
Repair getSchemas behavior to return correct type
Swiddis Aug 23, 2023
8af7f4a
Add tests for getSchemas
Swiddis Aug 23, 2023
a31963d
Add tests for asset and sample data backend methods
Swiddis Aug 23, 2023
feeaef5
Break flyout validation methods out of constructing method
Swiddis Aug 24, 2023
b644d11
Add tests for extracted flyout methods
Swiddis Aug 24, 2023
614613a
Switch validation method to use ValidationResult
Swiddis Aug 24, 2023
d97b931
Merge branch 'prototype-network-catalog' into feature/integrations-setup
Swiddis Sep 6, 2023
c8b5eff
Merge branch 'refactor-integ-frontend' into feature/integrations-setup
Swiddis Sep 6, 2023
af78c27
Merge branch 'main' into prototype-network-catalog
Swiddis Sep 6, 2023
ab30ed0
Swap out flyout for hello-world setup page
Swiddis Sep 6, 2023
85361f8
Add basic step incrementing
Swiddis Sep 7, 2023
af21831
Add basic field skeleton for each step
Swiddis Sep 7, 2023
42dbb54
Add a cancel button
Swiddis Sep 7, 2023
bfa9d1a
Merge branch 'main' into prototype-network-catalog
Swiddis Sep 11, 2023
533e6c7
Add config type to developing form
Swiddis Sep 11, 2023
467488c
Flatten integration config
Swiddis Sep 11, 2023
670d229
Add sample data table modal
Swiddis Sep 12, 2023
392640d
Merge branch 'main' into feature/integrations-setup
Swiddis Sep 13, 2023
0a997a4
Add toggle for standard and advanced asset config
Swiddis Sep 13, 2023
8c7b0f2
Simplify imports
Swiddis Sep 13, 2023
616d44e
Merge remote-tracking branch 'upstream/main' into feature/integration…
Swiddis Sep 13, 2023
339fb09
Refactor major class names
Swiddis Sep 13, 2023
fae680f
(WIP) begin refactoring functionality into adaptor
Swiddis Sep 13, 2023
3096626
Finish migrating functionality to data adaptor
Swiddis Sep 13, 2023
5498724
Rename integration types for more clarity
Swiddis Sep 14, 2023
ee33a51
Merge branch 'main' into prototype-network-catalog
Swiddis Sep 14, 2023
9f97f5a
Merge branch 'prototype-network-catalog' into feature/integrations-setup
Swiddis Sep 15, 2023
b465b9f
Refactor component usage
Swiddis Sep 15, 2023
2af52f5
Connect forms to config state
Swiddis Sep 15, 2023
1687fbc
Fix filetype selector
Swiddis Sep 15, 2023
cb8dfd3
Merge branch 'main' into feature/integrations-setup
Swiddis Sep 18, 2023
1ee1b75
Merge remote-tracking branch 'upstream/main' into feature/integration…
Swiddis Sep 18, 2023
1049b13
Remove hardcoded name in path
Swiddis Sep 18, 2023
edd6bbe
Write one snapshot test
Swiddis Sep 18, 2023
1c95d41
Add more tests
Swiddis Sep 18, 2023
1760038
Fix test naming
Swiddis Sep 18, 2023
d4b27fe
Update obsolete snapshots
Swiddis Sep 18, 2023
4293290
Move integration creation helpers to own file
Swiddis Sep 18, 2023
36d65c6
Break out integration creation methods
Swiddis Sep 19, 2023
fe16007
Isolate more create_integration helpers
Swiddis Sep 22, 2023
dfadcb5
Simplify setup form
Swiddis Sep 22, 2023
af1471d
Add data source picker items
Swiddis Sep 22, 2023
9c9d800
Add better selector logic
Swiddis Sep 22, 2023
69e4978
Add queries for data sources
Swiddis Sep 25, 2023
038eefb
Switch from selector to combobox
Swiddis Sep 25, 2023
16745d1
Update snapshots
Swiddis Sep 25, 2023
1892b8f
Connect validation button to data source validation method
Swiddis Sep 26, 2023
ebdb008
Reimplement add integration button
Swiddis Sep 26, 2023
6230a65
Temporarily remove validate button
Swiddis Sep 27, 2023
b2fdb69
Merge remote-tracking branch 'upstream/main' into feature/integration…
Swiddis Sep 28, 2023
bbb137b
Add queries to integrations config
Swiddis Sep 28, 2023
be45f19
Merge branch 'main' into feature/integrations-setup
Swiddis Sep 29, 2023
764d028
Simplify dynamic table term selection
Swiddis Sep 29, 2023
28d761c
Remove unused validate code
Swiddis Sep 29, 2023
aaeddef
Undo wildcard import
Swiddis Sep 29, 2023
3ae20be
Switch from proxy to dataconnections endpoint
Swiddis Sep 29, 2023
0158cc3
Remove unused table fields
Swiddis Sep 29, 2023
3cb6c57
Switch dataconnections base to const
Swiddis Sep 29, 2023
dd1d763
Add console proxy to route constants
Swiddis Sep 29, 2023
3f2a79a
Update snapshots
Swiddis Sep 29, 2023
b1d9b09
Merge branch 'feature/integrations-setup' into setup-s3
Swiddis Sep 29, 2023
7325019
Add datasource toggling based on integration support
Swiddis Sep 29, 2023
ee6f063
Add basic querying to add-integration functionality
Swiddis Sep 29, 2023
5b0d5b3
Add loading screen (TODO fix tests)
Swiddis Oct 2, 2023
9a6dc3a
Move color to constants
Swiddis Oct 3, 2023
1e267c6
Move index name validation to constants and improve matching
Swiddis Oct 3, 2023
7d4854e
Move test constants to test constants
Swiddis Oct 3, 2023
2ab4a80
Merge branch 'feature/integrations-setup' into setup-s3
Swiddis Oct 3, 2023
4d5f491
Merge remote-tracking branch 'upstream/main' into setup-s3
Swiddis Oct 3, 2023
c7b9bf7
Remove mock query handler
Swiddis Oct 3, 2023
a17b757
Remove unused todo
Swiddis Oct 3, 2023
58c1dfd
Remove buggy query files
Swiddis Oct 3, 2023
d89c004
Fix faulty test
Swiddis Oct 3, 2023
97c02b2
Remove debug logging
Swiddis Oct 4, 2023
aaeb16d
Remove redundant copy complexity
Swiddis Oct 4, 2023
d1394a6
Add assets to default state
Swiddis Oct 4, 2023
d70b890
Fix progress updaters
Swiddis Oct 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 50 additions & 32 deletions ...omponents/integrations/components/__tests__/__snapshots__/setup_integration.test.tsx.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,27 +50,30 @@ exports[`Integration Setup Page Renders integration setup page as expected 1`] =
Object {
"connectionDataSource": "",
"connectionType": "index",
"displayName": "sample Integration",
"displayName": "[object Object] Integration",
Swiddis marked this conversation as resolved.
Show resolved Hide resolved
}
}
integration={
Object {
"assets": Object {
"savedObjects": Object {
"name": "sample",
"version": "1.0.1",
"name": Object {
"assets": Object {
"savedObjects": Object {
"name": "sample",
"version": "1.0.1",
},
},
"components": Array [
Object {
"name": "logs",
"version": "1.0.0",
},
],
"license": "Apache-2.0",
"name": "sample",
"type": "logs",
"version": "2.0.0",
},
"components": Array [
Object {
"name": "logs",
"version": "1.0.0",
},
],
"license": "Apache-2.0",
"name": "sample",
"type": "logs",
"version": "2.0.0",
"type": "",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you removing type here

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stopped filling a default value when the integration isn't loaded yet -- We only use the type when we go to integration adding so it's safe to keep empty until it's loaded.

}
}
updateConfig={[Function]}
Expand Down Expand Up @@ -141,7 +144,7 @@ exports[`Integration Setup Page Renders integration setup page as expected 1`] =
onBlur={[Function]}
onChange={[Function]}
onFocus={[Function]}
value="sample Integration"
value="[object Object] Integration"
>
<EuiFormControlLayout
fullWidth={false}
Expand All @@ -161,7 +164,7 @@ exports[`Integration Setup Page Renders integration setup page as expected 1`] =
onChange={[Function]}
onFocus={[Function]}
type="text"
value="sample Integration"
value="[object Object] Integration"
/>
</EuiValidatableControl>
<EuiFormControlLayoutIcons />
Expand Down Expand Up @@ -232,10 +235,12 @@ exports[`Integration Setup Page Renders integration setup page as expected 1`] =
options={
Array [
Object {
"disabled": true,
"text": "S3 Connection",
"value": "s3",
},
Object {
"disabled": true,
"text": "OpenSearch Index",
"value": "index",
},
Expand Down Expand Up @@ -273,12 +278,14 @@ exports[`Integration Setup Page Renders integration setup page as expected 1`] =
value="index"
>
<option
disabled={true}
key="0"
value="s3"
>
S3 Connection
</option>
<option
disabled={true}
key="1"
value="index"
>
Expand Down Expand Up @@ -719,29 +726,36 @@ exports[`Integration Setup Page Renders integration setup page as expected 1`] =
Object {
"connectionDataSource": "",
"connectionType": "index",
"displayName": "sample Integration",
"displayName": "[object Object] Integration",
}
}
integration={
Object {
"assets": Object {
"savedObjects": Object {
"name": "sample",
"version": "1.0.1",
"name": Object {
"assets": Object {
"savedObjects": Object {
"name": "sample",
"version": "1.0.1",
},
},
"components": Array [
Object {
"name": "logs",
"version": "1.0.0",
},
],
"license": "Apache-2.0",
"name": "sample",
"type": "logs",
"version": "2.0.0",
},
"components": Array [
Object {
"name": "logs",
"version": "1.0.0",
},
],
"license": "Apache-2.0",
"name": "sample",
"type": "logs",
"version": "2.0.0",
"type": "",
}
}
loading={false}
loadingProgress={0}
setLoading={[Function]}
setProgress={[Function]}
>
<EuiBottomBar>
<EuiPortal>
Expand Down Expand Up @@ -1243,10 +1257,12 @@ exports[`Integration Setup Page Renders the form as expected 1`] = `
options={
Array [
Object {
"disabled": true,
"text": "S3 Connection",
"value": "s3",
},
Object {
"disabled": false,
"text": "OpenSearch Index",
"value": "index",
},
Expand Down Expand Up @@ -1284,12 +1300,14 @@ exports[`Integration Setup Page Renders the form as expected 1`] = `
value="index"
>
<option
disabled={true}
key="0"
value="s3"
>
S3 Connection
</option>
<option
disabled={false}
key="1"
value="index"
>
Expand Down
11 changes: 11 additions & 0 deletions public/components/integrations/components/create_integration_helpers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,17 @@ type ValidationResult = { ok: true } | { ok: false; errors: string[] };
export interface IntegrationTemplate {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to properly merge this type with the config on the backend later -- Just need to deal with the effects of converting it into a module.

name: string;
type: string;
assets: {
savedObjects?: {
name: string;
version: string;
};
queries?: Array<{
name: string;
version: string;
language: string;
}>;
};
}

export const doTypeValidation = (
Expand Down
Loading
Loading