dev/core#3414 Stop dropping temp table on finish of contact import job

[eileenmcnaughton]

Overview

dev/core#3414 Stop dropping temp table on finish of contact import job

Per https://lab.civicrm.org/dev/core/-/issues/3414 the intent is to leave the temp table in place after the job so that the output can be downloaded directly from the temp table. In addition it might be possible to allow viewing without downloading & leaves the potential for updating retrying.

Tracking the temp table through the user job, rather than the forms, also helps towards offline batch processing

Before

The import form process manages the deletion of the import temp tables

After

It is left to the clearTempTables - this clears them after 2 days from the scheduled job or immediately when a more active system flush is done

civicrm-core/CRM/Core/Config.php

Lines 360 to 376 in c187111

	$dao = new CRM_Core_DAO();
	$query = "
	SELECT TABLE_NAME as tableName
	FROM INFORMATION_SCHEMA.TABLES
	WHERE TABLE_SCHEMA = %1
	AND (
	TABLE_NAME LIKE 'civicrm_import_job_%'
	OR TABLE_NAME LIKE 'civicrm_report_temp%'
	OR TABLE_NAME LIKE 'civicrm_tmp_d%'
	)
	";
	// NOTE: Cannot find use-cases where "civicrm_report_temp" would be durable. Could probably remove.
	if ($timeInterval) {
	$query .= " AND CREATE_TIME < DATE_SUB(NOW(), INTERVAL {$timeInterval})";
	}

Technical details

If the scheduled job or occassional system flushes are happening (or not much importing is happening) the temp tables won't build up in the database, I might need to look at adding some sort of status check for if they do

The bigger picture is that ideally temp tables will be removed when the row is deleted from civicrm_user_job or when it's expires data passes - this would give more control - eg. to sysadmins who want to check in on imports

[civibot]

(Standard links)

• If this is your first pull-request for CiviCRM, please browse CONTRIBUTING.md for information about the development and testing processes.
• If you are reviewing this pull-request, you may wish to consult the test sites and the Review Standards (long template, short template).

[eileenmcnaughton]

@demeritcowboy this is also fairly simple - but is a pre-requisite for cleaning up some of the datasource code and for switching to downloading csvs based on the import results as in #23292

[Edzelopez]

Hi @eileenmcnaughton

I've tested this (on Monish's direction).

I used the hook_civicrm_import to determine the temp table name used for the import.

Before patch:

Temp table was destroyed as soon as the import was completed.

After patch:

Temp table was retained in the database, with all rows intact. Running cv api System.flush deleted the temp table.

[Edzelopez]

Hi @eileenmcnaughton

I've tested this (on Monish's direction).

I used the hook_civicrm_import to determine the temp table name used for the import.

Before patch:

Temp table was destroyed as soon as the import was completed.

After patch:

Temp table was retained in the database, with all rows intact. Running cv api System.flush deleted the temp table.

[eileenmcnaughton]

thanks @Edzelopez , @demeritcowboy