GSBPM Phase 8 Evaluate & Quality

[InKyungChoi]

Comments related to GSBPM Phase 8. Evaluate & Quality

• Should ‘Evaluate’ be represented as an overarching process to indicate its importance across the model? (UK)
• Although specific for each statistical operation, phase 8 could be perceived as belonging to the overarching process “Quality management”. (Portugal)
• Phase 8. Evaluate: this Phase is conceptually different from the others, it contains the steps of the PDCA-cycle. Monitoring is an activity performed throughout the statistical business process. We suggest the contents of this phase to be revised in a way to refer to the evaluation of the production processes. (Hungary)
• We found very difficult to map the analysis of the administrative data when they are collected. It includes the analysis of stability of data (compare variables and codes used with the ones from previous editions), the research of duplicates, the count of missing values and a first raw assessment of the coverage of the source with respect to the register population and/or to a statistical target variable (when possible). Our doubts were between 5.3, 8.2 and overarching quality management. At the end we decided for 8.2, because 5.3 seems more connected with the application of editing rules and quality management was too generic.... It would be useful to detail what is included in quality management overarching process (this would also simplify the distinction with GAMSO quality management) (Italy)
• We suggest adding a last sentence in the point 103 on the Evaluate phase to accommodate geospatial considerations: “Although geospatial production processes may be less well-established, their evaluation should also be managed to ensure quality and capacity building requirements.” (Portugal)
• We suggest reviewing the following sentences: (Phase 8) “This phase manages the evaluation of a specific instance…” and “Validate outputs comparing the statistics with previous cycles (if applicable)” (sub-process 6.2.). It seems that the second sentence should read “…with previous instances…” (Mexico)
• GeoGSBPM added in the introduction of the phase 8 "It is important to note that the extent and depth of evaluation depend on the knowledge and experience of staff about the process under the evaluation. As geospatial production processes may be less well-established and understood, a systematic quality framework and capacity building of staff may be required for a thorough evaluation." (GeoGSBPM)

[InKyungChoi]

See below for the updated GSBPM description (from action point: "to create two versions (1. Keep Phase Evaluate and change descriptions/order of OP; 2. Move quality elements into OP/GAMSO)" from Meeting March 28th)

II. The Model (to be moved to II. The Model, rather than its original place VI. Overarching Process)

Overarching activity

The GSBPM recognises several activities that are conducted across different phases throughout the production process. For example, quality is monitored and checked at various stages of the production. It may vary in its methods and objects on which the quality is measured (e.g., checking quality of product through quantitatively reviewing user feedback, checking the quality of imputation by calculating success rate), but this activity is often supported by a central unit in the organisation to maintain the consistent corporate policy and standard. In GSBPM, activities with a strong statistical component that apply throughout the production phases are called “overarching activities”.

Quality concerns organisations, products, sources and processes. In the present framework, the overarching activity "quality management" refers to the management of quality that takes place within the scope of the production process. For example, if the purpose of the business process is to produce enterprise demographic statistics, checking whether the quality of a register is good enough to meet the requirements is a part of the quality management activity of the corresponding production process while the managing quality of the register itself (e.g., checking coverage, making concepts homogenous) is not. Quality management activity at an institutional level (e.g. adoption of a Quality Policy or Quality Assurance Framework for the production of register-based statistics) or in areas that are not directly related to the production (e.g., quality of new employee onboarding programme) is considered in the GAMSO.

The main goal of quality management within the statistical business process is to understand and manage the quality of the statistical sources, processes and products. There is general agreement among statistical organisations that quality should be defined according to the ISO 9000-2015 standard: “The degree to which a set of inherent characteristics of an object fulfils requirements". Thus, quality is a complex and multi-faceted concept, usually defined in terms of several quality dimensions. The dimensions of quality that are considered most important depend on user perspectives, needs and priorities, which vary between processes and across groups of users.

In order to improve quality, quality management should be present throughout the business process and some of quality management activities are highlighted in GSBPM – for example, “Evaluate” phase can be considered as a quality management activity for the entire production process instance and “Produce dissemination products” sub-process of “Disseminate” phase contains the final quality check activity on the products before they are disseminated. Many statistical organisations have developed their own quality instructions according to GSBPM phases and sub-processes. A suggested list of quality indicators for the GSBPM which can be used as a reference or a checklist to identify gaps and/or duplication of work in the organisation can be found at the Quality Indicators for the GSBPM – for Statistics derived from Surveys and Administrative Data Sources.

[andreapetres]

I suggest to clarify the scope of each model first:

• if GAMSO is considered a generic activity model, it could contain all the activities relavant for statistical organizations, including activities related to production (this means the integration of generic activities from GAMSO and GSBPM into GAMSO). The activities may then be grouped by different criteria, as needed, e.g. by type, by level of detail, by their role in a process, etc. Please see example attached: 1_activity_model

• GSBPM, as a process model could focus more on how a statistical process is put together, using elements, activities from the GAMSO and mentioning their inputs and outputs, not only in GSIM terminology, but also in statistical context (e.g.: edit and impute uses raw microdata sets (Input = GSIM Data set) and produces a new version of the data set with edited/imputed data with relevant metadata (Output = GSIM Data set)). Please see example attached: _2.process_model

1_activity_model_20230303.pdf
2_process_model_20230223.pdf

[InKyungChoi]

Discussion from team meeting, see

• Meeting notes - March 7th, 2023 #6
• Meeting notes - March 28th, 2023 #7
• Meeting notes - April 18th, 2023 #10 (preliminary decision made)