Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add custom filter function #632

Open
wants to merge 6 commits into
base: development
Choose a base branch
from
Open

feat: add custom filter function #632

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1868ebc
feat: add custom filter function
KevinRohlf Nov 20, 2024
c2819f9
chore: implements all filters to filters.config.js, move BoolFilterQu…
KevinRohlf Nov 22, 2024
401b690
feat: enhance filter functionality with dynamic query paths and impro…
KevinRohlf Dec 3, 2024
303f283
fix: update filter component keys
KevinRohlf Dec 5, 2024
302b545
fix: isSelected on top position
KevinRohlf Dec 5, 2024
581edd0
feat: add MUST_EXIST & MUSTS_EXITS_AND_NON_EMPTY filters for all ids
KevinRohlf Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 35 additions & 1 deletion filters.config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,38 @@
const FILTER_VALUES = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Move the FILTER_VALUES const into aquarius/index.ts

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made the const into an enum and added it to the other methods in the searchQuery.ts is this ok?

MUST_EXIST: 'MUST_EXIST',
moritzkirstein marked this conversation as resolved.
Show resolved Hide resolved
MUST_EXISTS_AND_NON_EMPTY: 'MUST_EXISTS_AND_NON_EMPTY'
//...
moritzkirstein marked this conversation as resolved.
Show resolved Hide resolved
}

module.exports = {
filters: [],
filters: [
{
id: 'gaiax',
label: 'Gaia-X Service',
type: 'filterList',
options: [
// a new filter value for a MUST_EXIST type could be added to handle this new functionality
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo(blocking): This seems to be a leftover. Remove it or enhance for clarity.

{
label: 'Service SD',
value: FILTER_VALUES.MUST_EXISTS_AND_NON_EMPTY,
queryPath:
'metadata.additionalInformation.gaiaXInformation.serviceSD.url'
},
{
label: 'Terms and Conditions',
value: FILTER_VALUES.MUST_EXIST,
queryPath:
'metadata.additionalInformation.gaiaXInformation.termsAndConditions'
},
// options can have their own queryPath defined that gets validated against the defined value
moritzkirstein marked this conversation as resolved.
Show resolved Hide resolved
{
label: 'Verified',
value: FILTER_VALUES.MUST_EXIST,
queryPath:
'metadata.additionalInformation.gaiaXInformation.serviceSd.isValid'
}
]
}
],
filterSets: {}
}
3 changes: 2 additions & 1 deletion src/@context/Filter.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ function FilterProvider({ children }: { children: ReactNode }): ReactElement {
const [filters, setFilters] = useState<Filters>({
accessType: [],
serviceType: [],
filterSet: []
filterSet: [],
gaiax: []
})
const [ignorePurgatory, setIgnorePurgatory] = useState<boolean>(true)
const [sort, setSort] = useState<Sort>({
Expand Down
16 changes: 16 additions & 0 deletions src/@types/aquarius/BaseQueryParams.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,30 @@
size?: number
}

interface boolFilter {
moritzkirstein marked this conversation as resolved.
Show resolved Hide resolved
bool: {
must: {
exists: {
field: string
}
}
must_not?: {
term: {
[key: string]: string
}
}
}
}

interface BaseQueryParams {
chainIds: number[]
// eslint-disable-next-line @typescript-eslint/no-explicit-any
nestedQuery?: any
esPaginationOptions?: EsPaginationOptions
sortOptions?: SortOptions
aggs?: any

Check warning on line 27 in src/@types/aquarius/BaseQueryParams.d.ts

View workflow job for this annotation

GitHub Actions / test (ubuntu-latest, 18) 

Unexpected any. Specify a different type
filters?: FilterTerm[]
bool?: boolFilter[]
moritzkirstein marked this conversation as resolved.
Show resolved Hide resolved
ignorePurgatory?: boolean
ignoreState?: boolean
showSaas?: boolean
Expand Down
51 changes: 50 additions & 1 deletion src/@utils/aquarius/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,18 @@
*/
type TFilterValue = string | number | boolean | number[] | string[]
type TFilterKey = 'terms' | 'term' | 'match' | 'match_phrase'
type Query = {
moritzkirstein marked this conversation as resolved.
Show resolved Hide resolved
must: {
exists: {
field: string
}
}
must_not?: {
term: {
[key: string]: string
}
}
moritzkirstein marked this conversation as resolved.
Show resolved Hide resolved
}

export function getFilterTerm(
filterField: string,
Expand All @@ -56,6 +68,39 @@
}
}

export function getFilter(...args: any[]) {

Check warning on line 71 in src/@utils/aquarius/index.ts

View workflow job for this annotation

GitHub Actions / test (ubuntu-latest, 18) 

Unexpected any. Specify a different type
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue(blocking): we should try not to use any as type here. If it is needed, we need to add a comment disabling eslint for this line and describing why the any type is used

let filters = []
if (typeof args[0] === 'object') {
args[0].forEach((arg) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion(blocking): In general prefer usage of for...of over forEach for readability and performance. More context: https://biomejs.dev/linter/rules/no-for-each/

This comment applies to all places in the code where forEach is being used.

const filter = arg.split('=')
filters = [...filters, filter]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion(blocking): move this code to a named function and reuse it to increase readability

})
} else {
const filter = args[0].split('=')
filters = [...filters, filter]
}

let filter: Query[] = []
filters.forEach((filterItem) => {
let query: Query = {
must: {
exists: { field: filterItem[0] }
}
}
if (filterItem[1] === 'MUST_EXISTS_AND_NON_EMPTY') {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion(blocking): Move FILTER_VALUES const into this file and reference it to prevent errors: FILTER_VALUES.MUST_EXISTS_AND_NON_EMPTY

query = {
...query,
must_not: {
term: { [filterItem[0] + '.keyword']: '' }
}
}
}
filter = [...filter, query]
})

return filter
}

export function parseFilters(
filtersList: Filters,
filterSets: { [key: string]: string[] }
Expand All @@ -77,6 +122,9 @@
? getFilterTerm(filterQueryPath[key], uniqueTags)
: undefined
}
if (key === 'gaiax') {
return undefined
}
if (filtersList[key].length > 0)
return getFilterTerm(filterQueryPath[key], filtersList[key])

Expand All @@ -90,7 +138,7 @@
const { whitelists } = addressConfig

const whitelistFilterTerms = Object.entries(whitelists)
.filter(([field, whitelist]) => whitelist.length > 0)

Check warning on line 141 in src/@utils/aquarius/index.ts

View workflow job for this annotation

GitHub Actions / test (ubuntu-latest, 18) 

'field' is defined but never used
.map(([field, whitelist]) =>
whitelist.map((address) => getFilterTerm(field, address, 'match'))
)
Expand Down Expand Up @@ -160,7 +208,8 @@
...(baseQueryParams.showSaas === false ? [saasFieldExists] : [])
]
}
}
},
...(baseQueryParams.bool || [])
]
}
}
Expand Down
66 changes: 56 additions & 10 deletions src/components/Search/Filter.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ interface FilterStructure {
options: {
label: string
value: string
queryPath?: string
}[]
}

Expand Down Expand Up @@ -114,10 +115,37 @@ export default function Filter({
router.push(urlLocation)
}

async function handleSelectedFilter(value: string, filterId: string) {
const updatedFilters = filters[filterId].includes(value)
? { ...filters, [filterId]: filters[filterId].filter((e) => e !== value) }
: { ...filters, [filterId]: [...filters[filterId], value] }
async function handleSelectedFilter(
value: { label: string; value: string; queryPath?: string },
moritzkirstein marked this conversation as resolved.
Show resolved Hide resolved
filterId: string
) {
let updatedFilters
if (value.queryPath) {
updatedFilters = filters[filterId].includes(
`${value.queryPath}=${value.value}`
)
? {
...filters,
[filterId]: filters[filterId].filter(
(e) => e !== `${value.queryPath}=${value.value}`
)
}
: {
...filters,
[filterId]: [
...filters[filterId],
`${value.queryPath}=${value.value}`
]
}
} else {
updatedFilters = filters[filterId].includes(value.value)
? {
...filters,
[filterId]: filters[filterId].filter((e) => e !== value.value)
}
: { ...filters, [filterId]: [...filters[filterId], value.value] }
}

setFilters(updatedFilters)

await applyFilter(updatedFilters[filterId], filterId)
Expand Down Expand Up @@ -176,16 +204,25 @@ export default function Filter({
<div key={filter.id} className={styles.filterType}>
<h5 className={styles.filterTypeLabel}>{filter.label}</h5>
{filter.options.map((option) => {
const isSelected = filters[filter.id].includes(option.value)
const isSelected =
filter.id === 'gaiax'
? filters[filter.id].includes(
`${option.queryPath}=${option.value}`
)
: filters[filter.id].includes(option.value)
return (
<Input
key={option.value}
key={
option.queryPath
? option.value + option.queryPath
: option.value
}
name={option.label}
type="checkbox"
options={[option.label]}
checked={isSelected}
onChange={async () => {
handleSelectedFilter(option.value, filter.id)
handleSelectedFilter(option, filter.id)
}}
/>
)
Expand Down Expand Up @@ -219,16 +256,25 @@ export default function Filter({
>
<div className={styles.compactOptionsContainer}>
{filter.options.map((option) => {
const isSelected = filters[filter.id].includes(option.value)
const isSelected =
filter.id === 'gaiax'
? filters[filter.id].includes(
`${option.queryPath}=${option.value}`
)
: filters[filter.id].includes(option.value)
return (
<Input
key={option.value}
key={
option.queryPath
? option.value + option.queryPath
: option.value
}
name={option.label}
type="checkbox"
options={[option.label]}
checked={isSelected}
onChange={async () => {
handleSelectedFilter(option.value, filter.id)
handleSelectedFilter(option, filter.id)
}}
/>
)
Expand Down
42 changes: 37 additions & 5 deletions src/components/Search/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { LoggerInstance } from '@oceanprotocol/lib'
import {
escapeEsReservedCharacters,
generateBaseQuery,
getFilter,
getFilterTerm,
parseFilters,
queryMetadata
Expand All @@ -16,6 +17,21 @@ import {
} from '../../@types/aquarius/SearchQuery'
import { filterSets, getInitialFilters } from './Filter'

export interface boolFilter {
bool: {
must: {
exists: {
field: string
}
}
must_not?: {
term: {
[key: string]: string
}
}
}
}

export function updateQueryStringParameter(
uri: string,
key: string,
Expand Down Expand Up @@ -43,15 +59,27 @@ export function getSearchQuery(
serviceType?: string | string[],
accessType?: string | string[],
filterSet?: string | string[],
showSaas?: boolean
showSaas?: boolean,
gaiax?: string | string[]
): SearchQuery {
text = escapeEsReservedCharacters(text)
const emptySearchTerm = text === undefined || text === ''
const filters: FilterTerm[] = []
const bool: boolFilter[] = []
let searchTerm = text || ''
let nestedQuery
if (tags) {
filters.push(getFilterTerm('metadata.tags.keyword', tags))
} else if (gaiax) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question(blocking): Is this query not compatible with tags or why is it only applied if tags is falsy?

const filter = getFilter(gaiax)
filter.forEach((term) => {
const query = {
bool: {
...term
}
}
bool.push(query)
})
} else {
searchTerm = searchTerm.trim()
const modifiedSearchTerm = searchTerm.split(' ').join(' OR ').trim()
Expand Down Expand Up @@ -120,8 +148,8 @@ export function getSearchQuery(
}

const filtersList = getInitialFilters(
{ accessType, serviceType, filterSet },
['accessType', 'serviceType', 'filterSet']
{ accessType, serviceType, filterSet, gaiax },
['accessType', 'serviceType', 'filterSet', 'gaiax']
)
parseFilters(filtersList, filterSets).forEach((term) => filters.push(term))

Expand All @@ -134,6 +162,7 @@ export function getSearchQuery(
},
sortOptions: { sortBy: sort, sortDirection },
filters,
bool,
showSaas
} as BaseQueryParams

Expand All @@ -154,6 +183,7 @@ export async function getResults(
serviceType?: string | string[]
accessType?: string | string[]
filterSet?: string[]
gaiax?: string | string[]
},
chainIds: number[],
cancelToken?: CancelToken
Expand All @@ -168,7 +198,8 @@ export async function getResults(
sortOrder,
serviceType,
accessType,
filterSet
filterSet,
gaiax
} = params

const showSaas =
Expand Down Expand Up @@ -199,7 +230,8 @@ export async function getResults(
sanitizedServiceType,
accessType,
filterSet,
showSaas
showSaas,
gaiax
)

const queryResult = await queryMetadata(searchQuery, cancelToken)
Expand Down
Loading