Admin UI: Publisher Search

[robredpath]

As a member of IATI Support,
I want to find publishers using the information I have available†
so that I can quickly discover the Registry situation for someone that I'm helping.

† Organisation name (e.g. "Open Data Services"), publishing status, presence of errors in data, org-id, country

In conversation with @cormachallinanderilinx we refined this to:

• A search box that can accept a name, org-id, or country, with the option of leaving it empty to see all results
• Checkboxes for "is approved", "has published", "has errors in data" per @siwhitehouse below this UI could be confusing and so we'd prefer to find a way of making a single control for this.

Acceptance criteria

• Publisher search should have the search box and checkboxes described above
• The free-text search should support some way of specifying both exact-string and fuzzy and/or wildcard matching
• Results should be presented in a table†† that is sortable††† by any column
• The results table should include all fields that are searchable as well as those currently included in the publisher list

††† I don't think that we should have both the table headings and the Order By: dropdown for sorting results. We should choose one. My preference would be for the table headings to sort the whole list.

2. Non-functional Criteria (Include availability, maintainability, performance, reliability, scalability, security, and usability criteria)

This search interface can be available to all users, apart from the ability to see unapproved publishers which should be restricted to logged-in Sysadmin users only.

EDIT: Update 2024-01-04 in line with discussions below

[siwhitehouse]

Thanks for raising this issue @robredpath. The main motivation for having this functionality is so that we can search across the whole set of publishers in one place, something that we don't have at the moment.

aiui the use of checkboxes as suggested makes it possible for a user to check "has published" without checking "is approved". I don't think a UI should allow this and it suggests to me that checkboxes are unsuitable here. Radio buttons with "all", "registered (unapproved)", "approved (unpublished)" and "published" or similar might be better.

In the publishers list,as it is currently implemented, the number of published datasets is shown. I think this is useful and would like to see it retained here, please.

What is the purpose of "presence of errors in data"? Is this a Boolean, numerical or some other type of field?

What is the logic for excluding unapproved publishers from non-Sysadmin users, please? At the moment sometimes people try to register the same organisation more than once. Not allowing people to check if their organisation is in the registration process is likely to increase the number of duplicate registrations that we see.

[robredpath]

Radio buttons with "all", "registered (unapproved)", "approved (unpublished)" and "published" or similar might be better.

I don't have strong feelings on this. I agree that the UI allowing the user to select (hopefully!) impossible combinations isn't ideal, but having a single control that selects based on a combination of fields doesn't feel great either. We could make some JavaScript to auto-select "is approved" if they select "is published"?

In the publishers list,as it is currently implemented, the number of published datasets is shown. I think this is useful and would like to see it retained here, please.

Agreed. Implicit - but should be explicit - is that there's no loss of display or other functionality as a result of this change.

What is the purpose of "presence of errors in data"? Is this a Boolean, numerical or some other type of field?

As I understand it, this is to make it easier to locate the correct publisher in a list where other search terms might lead to lots of results, and it stops someone having to click through a long list of publishers one-by-one to see if they have errors in their data. I would expect the control to be a Boolean, and the results to either be Boolean or numeric, depending on implementation considerations

What is the logic for excluding unapproved publishers from non-Sysadmin users, please?

This is a security consideration: if someone creates a publisher for some non-IATI-related purpose (such as to advertise their gambling website, or some illegal pursuit) then we don't want any content that they create, even whatever they entered into the publisher name field, to be displayed on the website until it's been reviewed.

Not allowing people to check if their organisation is in the registration process is likely to increase the number of duplicate registrations that we see.

Perhaps a mitigation to this might be to indicate if the search term appears in unapproved publishers, without actually listing them? Some carefully-crafted help text would be required to explain what was going on, however.

[siwhitehouse]

Radio buttons with "all", "registered (unapproved)", "approved (unpublished)" and "published" or similar might be better.

I don't have strong feelings on this. I agree that the UI allowing the user to select (hopefully!) impossible combinations isn't ideal, but having a single control that selects based on a combination of fields doesn't feel great either. We could make some JavaScript to auto-select "is approved" if they select "is published"?

I think what I am suggesting is a single control that allows someone to either search across all categories of publisher, or just one category. Let's move on and see what @cormachallinanderilinx suggests when implementing this.

snip

What is the purpose of "presence of errors in data"? Is this a Boolean, numerical or some other type of field?

As I understand it, this is to make it easier to locate the correct publisher in a list where other search terms might lead to lots of results, and it stops someone having to click through a long list of publishers one-by-one to see if they have errors in their data. I would expect the control to be a Boolean, and the results to either be Boolean or numeric, depending on implementation considerations

I'm still unclear about the use case for a member of IATI support to be using this. Should we be considering how this particular filter interacts, or doesn't, with http://dashboard.iatistandard.org/data_quality.html ?

What is the logic for excluding unapproved publishers from non-Sysadmin users, please?

This is a security consideration: if someone creates a publisher for some non-IATI-related purpose (such as to advertise their gambling website, or some illegal pursuit) then we don't want any content that they create, even whatever they entered into the publisher name field, to be displayed on the website until it's been reviewed.

Thanks. That makes sense.

Not allowing people to check if their organisation is in the registration process is likely to increase the number of duplicate registrations that we see.

Perhaps a mitigation to this might be to indicate if the search term appears in unapproved publishers, without actually listing them? Some carefully-crafted help text would be required to explain what was going on, however.

It feels tricky and something that will be difficult to implement. Let's see what options @cormachallinanderilinx can offer us.

[siwhitehouse]

#310 paraphrased here:

Add a Date Created column to this list. This will help us determine which are the newly created publishers that are waiting for approval.

Can this be added to the acceptance criteria, please?

[robredpath]

Thanks @siwhitehouse. I've updated the initial comment in line with our conversation here.

Two unresolved points, though:

What is the purpose of "presence of errors in data"? Is this a Boolean, numerical or some other type of field?

As I understand it, this is to make it easier to locate the correct publisher in a list where other search terms might lead to lots of results

I'm still unclear about the use case for a member of IATI support to be using this. Should we be considering how this particular filter interacts, or doesn't, with http://dashboard.iatistandard.org/data_quality.html ?

I'm not sure where this requirement came from. Maybe @IsabelBirds might know? I have no attachment to it, it's just in the Miro board so it's made its way here!

Not allowing people to check if their organisation is in the registration process is likely to increase the number of duplicate registrations that we see.

To clarify: would this make things worse, or is this the current situation (and so this just doesn't make things better)?

[IsabelBirds]

The error field was an idea to reduce the amount of digging we have to do to offer support.
Eg a an error count per activity from the validator.

Then if I'm already engaged with an org and can easily notice that they have errors, I can bring this up and offer support. This is likely to increase uptake and changes to data quality compared to contacting orgs out of the blue.

[cormachallinanderilinx]

For the is approved and has published and has errors.
After spending some time on this I'm not sure this the correct place, the publisher search page only returns approved publishers so their datasets can be viewed.
I think its probably a good idea to keep it like this.

This view is available for viewing pending publishers: https://www.iatiregistry.org/dashboard/mypublishers-pending
I add another tab (or similar) to show all that haven't published and possibly the publishers with errors (although this may not be as straight forward)
Then is is kept to the dashboard which is only available to admins.
there isn't a button to access the dashboard on the registry, so it probably makes sense to link that to the UI.

Just use this this ticket to improve the searching (fuzzy logic) and fix the sorting/?

[robredpath]

I think we can return to the user story to help us here:

As a member of IATI Support,
I want to find publishers using the information I have available†
so that I can quickly discover the Registry situation for someone that I'm helping.

The end state that we're trying to get to here is a situation where, when someone contacts IATI Support, we can quickly understand which Registry publisher(s) correspond to the person and/or organisation who has contacted us, and what the current state of them is.

Ideally, I think that would be part of the existing publisher search, because then there's just one place that you go to look for information about publishers. However, I think we're open to it being a separate admin tool if that's more straightforward in terms of implementation and security.

If the information is split across multiple tabs or multiple searches it becomes harder to use: at best you need to do the search multiple times, and it becomes very easy for people to either not know about or forget to use the other tabs.

Is that feasible: a tab in the dashboard which supports all the functionality that we've discussed here in a single view that's admin-only?

[cormachallinanderilinx]

yes, that sounds good to me

[robredpath]

Cool - I want to hear from @siwhitehouse before we proceed, though!

[cormachallinanderilinx]

Estimate 3 days

[siwhitehouse]

The error field was an idea to reduce the amount of digging we have to do to offer support. Eg a an error count per activity from the validator.

Then if I'm already engaged with an org and can easily notice that they have errors, I can bring this up and offer support. This is likely to increase uptake and changes to data quality compared to contacting orgs out of the blue.

I'm not clear still, my apologies. How could we show a per-activity error count when the search is at publisher level?

[siwhitehouse]

@robredpath I don't think we should have a single control for "is approved", "has published", "has errors in data".

I think we want to be able to filter by the three statuses that a publisher might be in: "registered (unapproved)", "approved (unpublished)" and "published". By default, a search should show all statuses. Either a single control, or a set of controls, should let us filter by status.

Separately, we want to be able to filter by whether a publisher has errors in its set of published files. That should show the number of errors, which we should be able to order on. @IsabelBirds have I specified what you have in mind here?

Thanks to @cormachallinanderilinx for the estimate.

[siwhitehouse]

I had misinterpreted @IsabelBirds latest comment.

What we would like is the mean average of errors per activity for the publisher as a column in the search results. That figure should also contain a link to the publisher's page on the IATI Validator, please.

[cormachallinanderilinx]

Is this the URL you would like included? https://validator.iatistandard.org/organisation/aiddata

Do you know is there a validator API that can be used to access this which will allow us to get a count of errors as we dont store the count?

To the best of my knowledge the validator only expose two APIs https://developer.iatistandard.org/api-details#api=iati-validator-v2&operation=get-pub-get-report

[robredpath]

Do you know is there a validator API that can be used to access this which will allow us to get a count of errors as we dont store the count?

The Validator API returns report.summary.critical which is a count of "critical" (i.e. structural validation) errors. We might also want to include report.summary.error (ruleset errors that contain "must", according to the validator docs) - any views @IsabelBirds @siwhitehouse ?

These are on a per-file basis; the way that we use CKAN in the Registry means that "file" and "dataset" are synonymous.

The pipeline that feeds the Validator starts with the Registry, so any file that exists on the Registry should have an entry in the Validator. There will be a time lag, I'm not sure what it is precisely, but it won't be long! @simon-20 or @odscjames might be able to advise.

Likewise, the Registry should know about updates to files first out of any of our systems. I'm not sure if there's an edge case where a file at an unchanged URL has been updated; again I hope that @simon-20 or @odscjames can advise on that.

We discussed on the call that this could result in a lot of API calls if the results page has a lot of publishers on, each of whom have a lot of datasets. Given that the Registry knows about changes to files first, it should be fine to cache results and invalidate the cache based on Registry / archiver updates. The API isn't actually as fast as I thought (I'm seeing 300-400ms response times); we can look into improving that but caching will likely be important. The API should support a reasonable number of concurrent queries, which would hopefully speed up total time to compile the list.

[robredpath]

@cormachallinanderilinx do you already have an IATI API key? We can help you get signed up and increase your access level once you're up and running if not.

[robredpath]

This gist is an example response for a file with several ruleset errors, but that is valid IATI data. The summary elements are at the end of the response.

[robredpath]

I have opened several issues against the Validator API repos for us to investigate whether we can make the Validator API more suitable for this use. Depending on complexity and how well this sits alongside other work we're doing on the Validator API, we may be able to make these changes very quickly, or not for several months.

The issues are:
Allow querying of multiple files at once
Speed up response
Allow users to request just particular elements of the response

[simon-20]

The pipeline that feeds the Validator starts with the Registry, so any file that exists on the Registry should have an entry in the Validator. There will be a time lag, I'm not sure what it is precisely, but it won't be long! @simon-20 or @odscjames might be able to advise.

I did a quick check, and there is a fair bit of variation. Over half of the datasets currently known about by the Datastore were validated within 30 minutes; but there is a long tail on this one, some can take a few hours, and if there is a problem--a publisher is flagged, for instance, for too much invalid data too quickly--then full validation may take much longer.

[cormachallinanderilinx]

Hi @robredpath yes I have an API key set up for some work we were looking into previously

[robredpath]

@odscjames could you get in touch with @cormachallinanderilinx via email and make sure that we know which is Derilinx' API key and that it has appropriately high limits? I want to make sure we're ahead of any rate limiting complications.

[siwhitehouse]

Do you know is there a validator API that can be used to access this which will allow us to get a count of errors as we dont store the count?

The Validator API returns report.summary.critical which is a count of "critical" (i.e. structural validation) errors. We might also want to include report.summary.error (ruleset errors that contain "must", according to the validator docs) - any views @IsabelBirds @siwhitehouse ?

We discussed this and we prefer to have them both included, please.

What about Warnings @robredpath ? Are they queryable through the API too?

These are on a per-file basis; the way that we use CKAN in the Registry means that "file" and "dataset" are synonymous.

So, is it possible to get a mean average of errors per activity then?

[robredpath]

We discussed this and we prefer to have them both included, please.

Is it more useful for them to be provided separately, or added together into one aggregate figure? I'm conscious that fixing a structural issue might then allow validation to proceed to the point where many warnings are triggered, so this number might appear to get worse as the data is actually improving.

What about Warnings @robredpath ? Are they queryable through the API too?

Yes, report.summary.warning provides that figure.

So, is it possible to get a mean average of errors per activity then?

@cormachallinanderilinx this one's for you!

[siwhitehouse]

@cormachallinanderilinx

I have a couple of comments about styling/layout.

Table settings

The table looks like it has fixed-width columns. Here is a screenshot of the top of the table when I search on 'development'

Could we configure the table display so that it avoids such text wrapping? From a fixed-width perspective, I think we could add width in the left-hand side columns from those on the right hand side. Ideally, the table would adapt to the display settings of the person's browsew/display settings. I don't know the possibilities and limitations to an approach like this though.

###Ordering by table header
I can still do this, but the text in the table header doesn't afford clicking. The sorting is on-page only and not across the whole of the returned data.

I would like ordering by table header to be clear to the user and for it to perform the same sorting as the dropdown i.e. across all of the returned data.

[cormachallinanderilinx]

@siwhitehouse
I don't understand your question. At the moment it looks to me that you no longer 'normalise' names by placing 'the's at the end of a name. I think that is good for the display, but I suspect we would still want to sort on the 'normalised' version. At the moment all organisations whose names begin with "The" are ordered using it, meaning they are all bunched together.
yes I have remove the normalization, so we will leave this until rob gives his opinion?

I have fixed the pagenation, still doing a bit of testing myself but looks good

On the table header clicks I will look at this now.

[robredpath]

@robredpath can you advise on best practice here, please?

I think that being clear about the normalisation and having it be consistent across the site is more important than whichever approach we choose - so, whatever we do elsewhere is what we should do here.

[siwhitehouse]

We discussed this on our call today. @cormachallinanderilinx will remove the sort from the column headers in the table, @siwhitehouse will check the pagination and then share this with the rest of IATI Support for feedback.

[siwhitehouse]

Pagination looks good now, thank you @cormachallinanderilinx

Is the API set up for the Staging instance? I'm asking because if I query

https://iatiregistry.org/api/action/organization_list?all_fields=true

then I get a list of organisations, but if I query

https://staging.iatiregistry.org/api/action/organization_list?all_fields=true

I get a 401 response. We'd like to be able to check the organisations in the "approval needed" state through the API and the UI.

[cormachallinanderilinx]

@siwhitehouse
Since it the staging URL the 401 suggests to me that you need to go to the staging site and put in the basic auth credentials.
If you are using python or a tool like postman you will also need to add the basic auth there

python example:
import requests
from requests.auth import HTTPBasicAuth
res = requests.post('https://staging.iatiregistry.org/api/action/organization_list?all_fields=true', auth=HTTPBasicAuth('user', 'password'))
print(res)

Postman:
You will need to set it as well, there should be a basic auth option under Authorization

[siwhitehouse]

@cormachallinanderilinx Thank you. Unfortunately, I am still receiving a

<Response [401]>

error when I follow your instructions.

I logged into my https://staging.iatiregistry.org/user/simonwhitehouse account and I created an API token. I then amended the code you posted above to include my username and API token. Running the code returns the 401.

I have just shared the code with you (via Deepnote) for you to troubleshoot.

I'd note that originally I was sending this as a get request without authentication, as per https://iatistandard.org/en/iati-tools-and-resources/iati-registry/iati-registry-api/publisher-endpoints/#ListPub

[cormachallinanderilinx]

Hi @siwhitehouse
This issue wasnt adding the API token, it was the Basic Auth

Thanks for sharing the Deepnote, I was able to fix it up there along with 1 or 2 small changes.

FYI the API will have paging (offset and limit), by default the limit is 20 so the next page will be:
https://staging.iatiregistry.org/api/action/organization_list?all_fields=true&offset=20&limit=20

You can also set a higher limit but response will be slower, example of 100 at a time:
https://staging.iatiregistry.org/api/action/organization_list?all_fields=true&offset=0&limit=100
To get the next 100 we set the offset to 100:
https://staging.iatiregistry.org/api/action/organization_list?all_fields=true&offset=100&limit=100

[siwhitehouse]

When logged in as https://staging.iatiregistry.org/user/simonwhitehouse I receive an internal server error when I click on the link to the last page (117). I also receive an internal server error when I select Order By "Created Ascending".

@cormachallinanderilinx I don't know why I am seeing these now when I didn't before. Can you investigate this, please? Let me know if you need any information from me.

[siwhitehouse]

When logged in as https://staging.iatiregistry.org/user/simonwhitehouse I see twenty publishers per page and (I assume) 116 pages return results. So, I would expect to see 2320-2340 publishers in the CSV download. I only see 1357.

This is my alternative check on the number of organisations appearing in the UI matching those in the database, as I don't have the coding skills to page through the API.

@cormachallinanderilinx I think the check here should be that the data in the CSV download matches that returned in the UI via the query in the URL. The API should also be consistent. This doesn't appear to be the case at the moment. Can you investigate this before we do any more testing, please? Happy to provide more information if you need it.

[cormachallinanderilinx]

Hi @siwhitehouse
Good catch, It looks like this may be an overall issue as the download is completely separate from actual publisher code.
I will work on building the publisher search code into the download functionality as well I think it complete makes sense to do this make sure they are the same.

[cormachallinanderilinx]

Hi @siwhitehouse
I just wanted to note I have done some work on aligning the downloads.
However, to properly align them will probably take another 1-2 days work.
It is working but it is very very slow and will result in a lot of Timeout Errors.
The reason is getting the GroupExtra details, the way we do it for the page load is a organization_show which is fine for 20 items on a page but for a couple of hundred its an issue.
Just want to check ye are with me continuing this work as part of this ticket or would ye rather it be done separate?

[siwhitehouse]

Hi @siwhitehouse Good catch, It looks like this may be an overall issue as the download is completely separate from actual publisher code. I will work on building the publisher search code into the download functionality as well I think it complete makes sense to do this make sure they are the same.

What do you mean by the "actual publisher code" here, please Cormac?

[siwhitehouse]

Hi @siwhitehouse I just wanted to note I have done some work on aligning the downloads. However, to properly align them will probably take another 1-2 days work. It is working but it is very very slow and will result in a lot of Timeout Errors. The reason is getting the GroupExtra details, the way we do it for the page load is a organization_show which is fine for 20 items on a page but for a couple of hundred its an issue. Just want to check ye are with me continuing this work as part of this ticket or would ye rather it be done separate?

Hi @cormachallinanderilinx

aiui we have two use cases for aligning the downloads:

1. As a check that the new Publisher search UI is returning a full and correct set of results
2. Because they should provide the same results to end users when this goes into live

I can't offer an opinion on the detail of how you propose to fix this, other than to say it looks like you are focusing on this end goal. It's fine to spend the time on this, so please do go ahead.

I have a couple of other observations:

1. The "State" column no longer appears in the table in Staging
2. The Downloads should also be updated to include the "State" column. I'm not sure if we have stated this explicitly before.

Finally, I'll leave it to you if you think it is better to set up a separate issue for aligning the Downloads. My preference would be for a new issue at this point.

[cormachallinanderilinx]

As a check that the new Publisher search UI is returning a full and correct set of results
and
What do you mean by the "actual publisher code" here, please Cormac?
The download CSV (even before these changes) were never developed to match the search functionality - the were developed as 2 seperate things.
The download button calls code that only downloads 'active' publishers who have published.

The "State" column no longer appears in the table in Staging
Can you check if you are logged in? this is hidden if you are logged out

[siwhitehouse]

What do you mean by the "actual publisher code" here, please Cormac? The download CSV (even before these changes) were never developed to match the search functionality - the were developed as 2 seperate things. The download button calls code that only downloads 'active' publishers who have published.

So, "actual publisher code" means the code that the UI uses to fetch and display publishers.

The "State" column no longer appears in the table in Staging Can you check if you are logged in? this is hidden if you are logged out

The staging website is still showing me an internal server error, but I expect you are right (D'oh)

[siwhitehouse]

@cormachallinanderilinx I have gone through this issue and I think the following have been discussed and have not yet been implemented:

1. error counts
2. removing the sorting from the table headers
3. a filter for the different publisher states
4. a country filter
5. word wrapping in tables
6. searching on multiple fields

You also said that:

“Country Search is a bit different as you cannot search on country name.
I just got an idea on this as im typing, I will look into somthing and update you when I check it out.”

but I don't see anything since then.

Could you give us an update on these before I share this with the team please?

[cormachallinanderilinx]

1. error counts (see below - causing slow page load)
2. removing the sorting from the table headers (done)
3. a filter for the different publisher states (done - Add a dropdown to show only approval_needed OR show all as I dont see anywhere in the issue that there should be a filter for showing active or only approval_needed but I think it will be very useful to show only approval_needed)
4. a country filter (done - drop down with a list of available countries)
5. word wrapping in tables (made some changes but when the cols on the right are made smaller it affects the header)
6. searching on multiple fields (what fields should be searchable as per the Acceptance Criteria A search box that can accept a name, org-id, or country so the search was implemented that if you search in the search box will will filter on all fields in the table as suggested to have a single control for searching. If we want to add a specific search for a field we will need more controls like Countries and State)

For error count I have been testing here and as you can see the page takes a long time to load: https://staging.iatiregistry.org/publisher/?q=&publisher_country=&state=&sort=created%20asc&page=3
one publisher has 35 datasets and another with 59, the rest roughly average about 2 which is think it a good test for production.
Reason being to get the count, first I need to get all the packages for each org that is loading.
Then on each package I need to run the IATI validator API: GET https://api.iatistandard.org/validator/report?name=
Which for which for this example it needs to run about 100 times and wait for a response as the page is loading, so in order for me to get a count it will reduce the page loading time a lot.
Unless there is an API where we can all errors for an org, im not sure there is a clean or out of the box way to get it.
I also looked at trying to download the CSV here but my key is invalid but again this will be for loops after reading in a CSV which I dont think is a great thing to do on a page load. https://api.iatistandard.org/vs/pvt/publishers/79d85709-2e20-4174-bc3d-d7179f6cc0eb/documents

[cormachallinanderilinx]

as per #466 I removed the error count
It was very slow for me but also seemed to causing some people gateway timeouts due to containing so many requests

[siwhitehouse]

Thank you @cormachallinanderilinx I suggest we keep the error count out and if it is needed I will raise a separate issue for it. This issue has multiple threads within it as is.

[siwhitehouse]

Going through the outstanding issues:

1. error counts
   We have agreed to remove the error counts. If the team want to revisit this we can do it separate to this issue.
2. removing the sorting from the table headers
   Done. Sorting is via dropdown.
3. a filter for the different publisher states
   Done. It's not how I'd imagined it might be, but I'm happy to share it with the team for their feedback.
4. a country filter
   Done, via dropdown and includes regions as well as countries.
5. word wrapping in tables
   Changes made.
6. searching on multiple fields
   Done.

I'm going to raise a separate issue for a bug I saw when testing the UI, but I'm keen to get these changes in front of the team and will be sharing them with them tomorrow. Thanks @cormachallinanderilinx

[cormachallinanderilinx]

Remove search when selecting from dropdown

[siwhitehouse]

@cormachallinanderilinx Are you ready to implement the changes, please?

[cormachallinanderilinx]

Hi @siwhitehouse im building the Image now.
When its build I was do some final testing and release to produciton

[cormachallinanderilinx]

PR: #462

[cormachallinanderilinx]

@siwhitehouse Changes are now on production