Initial Code for removing data provider dups #1652
Conversation
oblodgett
requested review from
abecerra,
markquintontulloch,
cmpich,
adamgibs and
VarunReddy1111
|
Choose to split the migrations into multiple files. Each file is basically its own transaction, which means putting everything into one transaction can run the server out of memory. This is the same reason for breaking up the deletes. They were MUCH faster doing them in batches vs one lump query. The single queries are run at the end just incase any were missed. Still need to run the GFF loads to test that this works. Will merge can happen after the testing is done locally. |
src/main/resources/db/migration/v0.37.0.24__remove_dataprovider_dups.sql
Show resolved
Hide resolved
|
Here are the timings on my local system... basically it ran ALL day. |
Initial code to restructure bulk load file history
|
Added to this is the reorg of BulkLoad -> BulkLoadFile -> BulkLoadFileHistory instead of both relationships being O2O there is not a BulkLoadFile - O2M -> BulkLoadFileHistory <- M2O - BulkLoadFile. This allows one file to be under multiple loads, and being able to split up loads into their parts. As also this PR does with the GFF loads and breaks them out into the 9 seperate loads per mod. This might be a bit excessive however it makes the loads run very fast. |
|
@markquintontulloch if you can look at this PR when you get a chance. |
| @@ -45,7 +45,7 @@ export const DataLoadsComponent = () => { | |||
| const [bulkLoadDialog, setBulkLoadDialog] = useState(false); | |||
| const [expandedGroupRows, setExpandedGroupRows] = useState(null); | |||
| const [expandedLoadRows, setExpandedLoadRows] = useState(null); | |||
| const [expandedFileRows, setExpandedFileRows] = useState(null); | |||
| //const [expandedFileRows, setExpandedFileRows] = useState(null); | |||
There was a problem hiding this comment.
Should that be removed? Or is that commenting out only temporary?
There was a problem hiding this comment.
ya the tables need to get reworked, in order to support to new structure.
| public APIResponse updateTranscripts(String dataProvider, String assembly, List<Gff3DTO> gffData) { | ||
| return gff3Executor.runLoadApi(dataProvider, assembly, gffData); | ||
| BulkLoadFileHistory history = new BulkLoadFileHistory(); |
There was a problem hiding this comment.
This looks like code very much like for the updateExons() method. Could that common code be centralized and being called?
No description provided.