Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TEST] Refactoring study's loader #893

Closed
wants to merge 5 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/libs/antares/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -469,6 +469,7 @@ set(SRC
study/memory-usage.cpp
study/importlogs.cpp
study/load.cpp
study/load-params.cpp
study/save.cpp
study/study.extra.cpp
study/version.h
Expand Down
185 changes: 185 additions & 0 deletions src/libs/antares/study/load-params.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
/*
** Copyright 2007-2018 RTE
** Authors: Antares_Simulator Team
**
** This file is part of Antares_Simulator.
**
** Antares_Simulator is free software: you can redistribute it and/or modify
** it under the terms of the GNU General Public License as published by
** the Free Software Foundation, either version 3 of the License, or
** (at your option) any later version.
**
** There are special exceptions to the terms and conditions of the
** license as they are applied to this software. View the full text of
** the exceptions in file COPYING.txt in the directory of this software
** distribution
**
** Antares_Simulator is distributed in the hope that it will be useful,
** but WITHOUT ANY WARRANTY; without even the implied warranty of
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
** GNU General Public License for more details.
**
** You should have received a copy of the GNU General Public License
** along with Antares_Simulator. If not, see <http://www.gnu.org/licenses/>.
**
** SPDX-License-Identifier: licenceRef-GPL3_WITH_RTE-Exceptions
*/

#include "study.h"

using namespace Yuni;


namespace Antares
{
namespace Data
{

bool Study::initializeInternalData(const StudyLoadOptions& options)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently, method initializeInternalData does 3 (at least) things :

  • it initializes parameters (it should be done by a method of class Parameters, or better : in the Parameters constructor, if possible)
  • it initializes the calendar (same remark as above, with class Date::Calendar)
  • it makes a check (should be moved with other checks : to be explored)

Each bullet should be separated from the others in a function / method
This split would have few consequences test-study-load-internal.cpp.

{

if (usedByTheSolver and not options.prepareOutput)
{
parameters.noOutput = true;
parameters.yearByYear = false;
parameters.timeSeriesToArchive = 0;
parameters.storeTimeseriesNumbers = false;
parameters.synthesis = false;
}

if (options.loadOnlyNeeded and !parameters.timeSeriesToGenerate)
// Nothing to refresh
parameters.timeSeriesToRefresh = 0;


// We can not run the simulation if the study folder is not in the latest
// version and that we would like to re-importe the generated timeseries
if (usedByTheSolver)
{
// We have time-series to import
if (parameters.timeSeriesToImport and (uint) header.version != (uint)versionLatest)
{
logs.error() << "Stochastic TS stored in input : study must be upgraded to "
<< Data::VersionToCStr((Data::Version)Data::versionLatest);
gotFatalError = true;
// it is useless to continue at this point
return false;
}
}

// calendar update
if (usedByTheSolver)
calendar.reset(parameters, /*force leapyear:*/ false);
else
calendar.reset(parameters);

calendarOutput.reset(parameters);

// In case hydro hot start is enabled, check all conditions are met.
// (has to be called after areas load and calendar building)
if (usedByTheSolver && !checkHydroHotStart())
return false;

// Reducing memory footprint
reduceMemoryUsage();
return true;

}

class SetHandlerAreas
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each method of class SetHandlerAreas is given a SingleSetOfAreas& set (which is a std::set<Area*>), and makes something with this argument.
Why not put the set in the class and rename the class setOfAreas ?
There would be some side effects to that change : it will bring some other changes in the code.
It remains to evaluate the size of these changes.

{
public:
SetHandlerAreas(AreaList& _areas) : areas(_areas)
{
}

void clear(Study::SingleSetOfAreas& set)
{
set.clear();
}

uint size(Study::SingleSetOfAreas& set) const
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uint size(Study::SingleSetOfAreas& set) const : I don't see the point of this function.
Instead of calling sets_of_areas_handler.size(set_of_areas), let's simply call sets_of_areas.size()

{
return (uint)set.size();
}

bool add(Study::SingleSetOfAreas& set, const String& value)
{
Area* area = AreaListLFind(&areas, value.c_str());
if (area)
{
set.insert(area);
return true;
}
return false;
}

bool add(Study::SingleSetOfAreas& set, const Study::SingleSetOfAreas& otherSet)
{
if (!otherSet.empty())
{
auto end = otherSet.end();
for (auto i = otherSet.begin(); i != end; ++i)
set.insert(*i);
}
return true;
}

bool remove(Study::SingleSetOfAreas& set, const String& value)
{
Area* area = AreaListLFind(&areas, value.c_str());
if (area)
{
set.erase(area);
return true;
}
return false;
}

bool remove(Study::SingleSetOfAreas& set, const Study::SingleSetOfAreas& otherSet)
{
if (!otherSet.empty())
{
auto end = otherSet.end();
for (auto i = otherSet.begin(); i != end; ++i)
{
set.erase(*i);
}
}
return true;
}

bool applyFilter(Study::SingleSetOfAreas& set, const String& value)
{
if (value == "add-all")
{
auto end = areas.end();
for (auto i = areas.begin(); i != end; ++i)
set.insert(i->second);
return true;
}

if (value == "remove-all")
{
set.clear();
return true;
}
return false;
}

private:
AreaList& areas;

}; // class SetHandlerAreas

void Study::initializeSetsData()
{
// Apply the rules
SetHandlerAreas handler(this->areas);
setsOfAreas.rebuildAllFromRules(handler);
// Write the results into the logs
setsOfAreas.dumpToLogs(logs);
}

} // namespace Data
} // namespace Antares
147 changes: 9 additions & 138 deletions src/libs/antares/study/load.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ namespace Antares
{
namespace Data
{

constexpr int MATRIX_BUFFER_4MB = 4 * 1024 * 1024;

bool Study::internalLoadHeader(const String& path)
{
// Header
Expand Down Expand Up @@ -93,7 +96,7 @@ bool Study::internalLoadFromFolder(const String& path, const StudyLoadOptions& o
inputExtensionCompatibility();

// Reserving enough space in buffer to avoid several calls to realloc
this->dataBuffer.reserve(4 * 1024 * 1024); // For matrices, reserving 4Mo
this->dataBuffer.reserve(MATRIX_BUFFER_4MB); // For matrices, reserving 4Mo
this->bufferLoadingTS.reserve(2096);
assert(this->bufferLoadingTS.capacity() > 0);

Expand All @@ -115,33 +118,6 @@ bool Study::internalLoadFromFolder(const String& path, const StudyLoadOptions& o
buffer.clear() << path << SEP << "layers" << SEP << "layers.ini";
loadLayers(buffer);

if (usedByTheSolver and not options.prepareOutput)
{
parameters.noOutput = true;
parameters.yearByYear = false;
parameters.timeSeriesToArchive = 0;
parameters.storeTimeseriesNumbers = false;
parameters.synthesis = false;
}

if (options.loadOnlyNeeded and !parameters.timeSeriesToGenerate)
// Nothing to refresh
parameters.timeSeriesToRefresh = 0;

// We can not run the simulation if the study folder is not in the latest
// version and that we would like to re-importe the generated timeseries
if (usedByTheSolver)
{
// We have time-series to import
if (parameters.timeSeriesToImport and (uint) header.version != (uint)versionLatest)
{
logs.error() << "Stochastic TS stored in input : study must be upgraded to "
<< Data::VersionToCStr((Data::Version)Data::versionLatest);
gotFatalError = true;
// it is useless to continue at this point
return false;
}
}

// This settings can only be enabled from the solver
// Prepare the output for the study
Expand All @@ -153,7 +129,7 @@ bool Study::internalLoadFromFolder(const String& path, const StudyLoadOptions& o
// -------------------------
// Getting the number of logical cores to use before loading and creating the areas :
// Areas need this number to be up-to-date at construction.
getNumberOfCores(options.forceParallel, options.maxNbYearsInParallel);
computeNumberOfCores(options);

#ifdef ANTARES_SWAP_SUPPORT
// In case of swap support, MC years parallel computing is forbidden.
Expand Down Expand Up @@ -222,23 +198,7 @@ bool Study::internalLoadFromFolder(const String& path, const StudyLoadOptions& o
}
}

// calendar update
if (usedByTheSolver)
calendar.reset(parameters, /*force leapyear:*/ false);
else
calendar.reset(parameters);

calendarOutput.reset(parameters);

// In case hydro hot start is enabled, check all conditions are met.
// (has to be called after areas load and calendar building)
if (usedByTheSolver && !checkHydroHotStart())
return false;

// Reducing memory footprint
reduceMemoryUsage();

return ret;
return initializeInternalData(options);
}

void Study::inputExtensionCompatibility()
Expand Down Expand Up @@ -317,92 +277,6 @@ bool Study::internalLoadBindingConstraints(const StudyLoadOptions& options)
return (!r and options.loadOnlyNeeded) ? false : r;
}

class SetHandlerAreas
{
public:
SetHandlerAreas(Study& study) : pStudy(study)
{
}

void clear(Study::SingleSetOfAreas& set)
{
set.clear();
}

uint size(Study::SingleSetOfAreas& set)
{
return (uint)set.size();
}

bool add(Study::SingleSetOfAreas& set, const String& value)
{
Area* area = AreaListLFind(&pStudy.areas, value.c_str());
if (area)
{
set.insert(area);
return true;
}
return false;
}

bool add(Study::SingleSetOfAreas& set, const Study::SingleSetOfAreas& otherSet)
{
if (!otherSet.empty())
{
auto end = otherSet.end();
for (auto i = otherSet.begin(); i != end; ++i)
set.insert(*i);
}
return true;
}

bool remove(Study::SingleSetOfAreas& set, const String& value)
{
Area* area = AreaListLFind(&pStudy.areas, value.c_str());
if (area)
{
set.erase(area);
return true;
}
return false;
}

bool remove(Study::SingleSetOfAreas& set, const Study::SingleSetOfAreas& otherSet)
{
if (!otherSet.empty())
{
auto end = otherSet.end();
for (auto i = otherSet.begin(); i != end; ++i)
{
set.erase(*i);
}
}
return true;
}

bool applyFilter(Study::SingleSetOfAreas& set, const String& value)
{
if (value == "add-all")
{
auto end = pStudy.areas.end();
for (auto i = pStudy.areas.begin(); i != end; ++i)
set.insert(i->second);
return true;
}

if (value == "remove-all")
{
set.clear();
return true;
}
return false;
}

private:
Study& pStudy;

}; // class SetHandlerAreas

bool Study::internalLoadSets()
{
// Set of areas
Expand All @@ -415,15 +289,12 @@ bool Study::internalLoadSets()
// Load the rules
if (setsOfAreas.loadFromFile(buffer))
{
// Apply the rules
SetHandlerAreas handler(*this);
setsOfAreas.rebuildAllFromRules(handler);
// Write the results into the logs
setsOfAreas.dumpToLogs(logs);
return true;
initializeSetsData();
return true;
}

logs.warning() << "Impossible to load the sets of areas";

return false;
}

Expand Down
Loading