Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: support k-parallelism diagonalization in lcao basis #4650

Merged
merged 111 commits into from
Jul 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
cbe0b6c
add a class parallel_k2d
hongriTianqi Jun 17, 2024
195bf48
merge develop
hongriTianqi Jun 24, 2024
9cec83b
move kpar init in input_conv
hongriTianqi Jun 24, 2024
38221f8
start ut of parallel_k2d
hongriTianqi Jun 24, 2024
14d9e41
update ut
hongriTianqi Jun 24, 2024
f430577
Merge branch 'develop' into parak
hongriTianqi Jun 24, 2024
d6f74dd
add pointer Pkpoints
hongriTianqi Jun 24, 2024
c297e13
update ut
hongriTianqi Jun 24, 2024
d6b438c
update ut
hongriTianqi Jun 24, 2024
038a57f
add P2D pointer
hongriTianqi Jun 24, 2024
c55feff
k2d communicate the production code
hongriTianqi Jun 25, 2024
c6d5496
prepare to get Hk and Sk
hongriTianqi Jun 25, 2024
33e633d
kpar 1 ok but 2 failed
hongriTianqi Jun 25, 2024
67b6d8e
collect and distribute hk sk
hongriTianqi Jun 25, 2024
a24c1fa
fix bug
hongriTianqi Jun 26, 2024
7316a96
correct collection and distribution of hk and sk
hongriTianqi Jun 26, 2024
cbff6cb
add initialization tag in parallel_k2d
hongriTianqi Jun 26, 2024
c3f6a2f
use init tag in hsolver
hongriTianqi Jun 26, 2024
64bae71
update k2d data structure
hongriTianqi Jun 26, 2024
833647f
k2d only save one k data
hongriTianqi Jun 26, 2024
845e306
update h s in operatorlcao and refactor code
hongriTianqi Jun 27, 2024
574c84d
require kpar>1
hongriTianqi Jun 28, 2024
b267e30
remove nkstot in k2d
hongriTianqi Jun 28, 2024
78993c5
set initialization state
hongriTianqi Jun 28, 2024
898fb13
fix error
hongriTianqi Jun 28, 2024
c803e39
correct ways of distributing
hongriTianqi Jul 1, 2024
7c309fd
remove unused codes
hongriTianqi Jul 1, 2024
e25574d
diago with scalapack seems ok
hongriTianqi Jul 1, 2024
3aefcc9
tried to fix bug
hongriTianqi Jul 1, 2024
23228e6
add check codes
hongriTianqi Jul 1, 2024
ead5cf1
check psi
hongriTianqi Jul 1, 2024
fa08e1f
run but stop run
hongriTianqi Jul 1, 2024
9367c80
run but results wrong
hongriTianqi Jul 2, 2024
e067e57
check ekb
hongriTianqi Jul 2, 2024
f475016
fix error
hongriTianqi Jul 2, 2024
17b2c8a
check ekb and psi
hongriTianqi Jul 2, 2024
d6e7fb3
fix nk error
hongriTianqi Jul 2, 2024
5982f4f
run but results wrong again
hongriTianqi Jul 2, 2024
3c0e04d
fix error
hongriTianqi Jul 2, 2024
8545576
update code
hongriTianqi Jul 2, 2024
5db13fa
first success
hongriTianqi Jul 3, 2024
9d12a02
elpa ok
hongriTianqi Jul 3, 2024
ef734d6
remove coments
hongriTianqi Jul 3, 2024
ae72130
simplify codes
hongriTianqi Jul 3, 2024
37e2d5e
Merge branch 'develop' into paraksubmit
hongriTianqi Jul 3, 2024
65c9295
update inputs
hongriTianqi Jul 3, 2024
1ba3055
add case test
hongriTianqi Jul 3, 2024
97e782e
update makefile
hongriTianqi Jul 3, 2024
1dd472c
cal_ncol_bands
hongriTianqi Jul 3, 2024
14a6c88
update test
hongriTianqi Jul 3, 2024
baf4227
rename local to pool
hongriTianqi Jul 3, 2024
33fad11
add comments
hongriTianqi Jul 3, 2024
ca901c5
update ut
hongriTianqi Jul 3, 2024
b3df301
fix compilation error
hongriTianqi Jul 3, 2024
41e456d
[pre-commit.ci lite] apply automatic fixes
pre-commit-ci-lite[bot] Jul 3, 2024
9a1be31
Merge branch 'develop' into paraksubmit
hongriTianqi Jul 4, 2024
8916a23
[pre-commit.ci lite] apply automatic fixes
pre-commit-ci-lite[bot] Jul 4, 2024
dfb3d6f
fix ut error
hongriTianqi Jul 4, 2024
379967a
Merge branch 'develop' into paraksubmit
hongriTianqi Jul 5, 2024
3507ebc
Merge branch 'develop' into paraksubmit
hongriTianqi Jul 5, 2024
8e11371
Merge branch 'develop' into paraksubmit
hongriTianqi Jul 8, 2024
bf38ada
refactor code the reduce memory
hongriTianqi Jul 8, 2024
849820a
update code
hongriTianqi Jul 9, 2024
d2570cb
merge develop
hongriTianqi Jul 11, 2024
3cd8241
release warning
hongriTianqi Jul 11, 2024
26a8262
move to a function
hongriTianqi Jul 11, 2024
5dc0b3a
add a small member function for pkpoints
hongriTianqi Jul 11, 2024
0361373
remove globalv
hongriTianqi Jul 11, 2024
43e5b08
add a getter
hongriTianqi Jul 11, 2024
52bfe74
update ut
hongriTianqi Jul 11, 2024
0430998
add getters
hongriTianqi Jul 11, 2024
d1332d3
adjust
hongriTianqi Jul 11, 2024
f35cd65
Merge branch 'develop' into parakf3
hongriTianqi Jul 11, 2024
6ab7f05
Merge branch 'develop' into parakf3
hongriTianqi Jul 12, 2024
484adb8
increase readability and fix ut error
hongriTianqi Jul 12, 2024
d6f1b55
[pre-commit.ci lite] apply automatic fixes
pre-commit-ci-lite[bot] Jul 12, 2024
d4f27de
record time and memory
hongriTianqi Jul 12, 2024
f404459
fix memory record
hongriTianqi Jul 12, 2024
6059dc8
fix bug
hongriTianqi Jul 14, 2024
fc238d8
Merge branch 'develop' into parakf3
hongriTianqi Jul 14, 2024
9faac04
fix compliation error
hongriTianqi Jul 14, 2024
7dbaa80
update
hongriTianqi Jul 14, 2024
0b448e1
fix error
hongriTianqi Jul 14, 2024
9875f05
fix error
hongriTianqi Jul 14, 2024
e8bf4e9
fix error
hongriTianqi Jul 14, 2024
a6d94b0
fix error
hongriTianqi Jul 14, 2024
15ddf3f
fix typo
hongriTianqi Jul 14, 2024
37a151a
refactor code
hongriTianqi Jul 14, 2024
3932967
modify timer
hongriTianqi Jul 14, 2024
23ecdff
Merge branch 'develop' into parakf3
hongriTianqi Jul 14, 2024
b1d8824
Merge branch 'develop' into parakf3
hongriTianqi Jul 15, 2024
4a188b5
update
hongriTianqi Jul 15, 2024
fdba3c0
[pre-commit.ci lite] apply automatic fixes
pre-commit-ci-lite[bot] Jul 15, 2024
ccbc332
fix
hongriTianqi Jul 15, 2024
c0e26aa
fix
hongriTianqi Jul 15, 2024
bf8d029
Merge branch 'develop' into parakf3
hongriTianqi Jul 16, 2024
fd3b521
Merge branch 'develop' into parakf3
hongriTianqi Jul 16, 2024
a155e00
add a warning
hongriTianqi Jul 16, 2024
5042a63
[pre-commit.ci lite] apply automatic fixes
pre-commit-ci-lite[bot] Jul 16, 2024
ec34a7f
fix error
hongriTianqi Jul 17, 2024
fcc175c
Merge branch 'develop' into parakf3
hongriTianqi Jul 17, 2024
47dd636
Merge branch 'develop' into parakf3
hongriTianqi Jul 17, 2024
b6f1b78
Merge branch 'develop' into parakf3
hongriTianqi Jul 18, 2024
91281ce
constrain range of k2d
hongriTianqi Jul 19, 2024
0fe9553
Merge branch 'develop' into parakf3
hongriTianqi Jul 19, 2024
0f2525b
fix error
hongriTianqi Jul 19, 2024
06753ba
Merge branch 'develop' into parakf3
hongriTianqi Jul 20, 2024
9a8d010
fix error
hongriTianqi Jul 20, 2024
f5afd01
modify code
hongriTianqi Jul 20, 2024
edfaa1e
fix error
hongriTianqi Jul 20, 2024
a57a5a7
[pre-commit.ci lite] apply automatic fixes
pre-commit-ci-lite[bot] Jul 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions source/Makefile.Objects
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,7 @@ OBJS_HSOLVER_LCAO=hsolver_lcao.o\
elpa_new_real.o\
elpa_new_complex.o\
utils.o\
parallel_k2d.o\

OBJS_HSOLVER_PEXSI=diago_pexsi.o\
pexsi_solver.o\
Expand Down
1 change: 1 addition & 0 deletions source/module_base/global_variable.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ bool out_mul = false; // qifeng add 2019/9/10

int NPROC = 1; ///< global number of process
int KPAR = 1; ///< global number of pools
int KPAR_LCAO = 1; ///< global number of pools for LCAO diagonalization only
int NSTOGROUP = 1;
int MY_RANK = 0; ///< global index of process
int MY_POOL = 0; ///< global index of pool (count in pool)
Expand Down
7 changes: 7 additions & 0 deletions source/module_base/global_variable.h
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,13 @@ extern int DCOLOR;
extern int GRANK;
extern int GSIZE;

//========================================================================
// EXPLAIN : Parallel information
// GLOBAL VARIABLES :
// NAME : KPAR_LCAO ( global number of pools for LCAO diagonalization only)
//========================================================================
extern int KPAR_LCAO;

//==========================================================
// EXPLAIN : readin file dir, output file std::ofstream
// GLOBAL VARIABLES :
Expand Down
6 changes: 6 additions & 0 deletions source/module_basis/module_ao/parallel_orbitals.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,11 @@ int Parallel_Orbitals::get_wfc_global_nbasis() const
#endif
}

int Parallel_Orbitals::get_nbands() const
{
return this->nbands;
hongriTianqi marked this conversation as resolved.
Show resolved Hide resolved
}

void Parallel_Orbitals::set_atomic_trace(const int* iat2iwt, const int &nat, const int &nlocal)
{
ModuleBase::TITLE("Parallel_Orbitals", "set_atomic_trace");
Expand Down Expand Up @@ -225,6 +230,7 @@ int Parallel_Orbitals::set_nloc_wfc_Eij(
{
ModuleBase::TITLE("Parallel_Orbitals", "set_nloc_wfc_Eij");
// for wavefuncton , calculate nbands_loc
this->nbands = N_A;
int end_id = 0;
int block = N_A / nb;
if (block * nb < N_A)
Expand Down
4 changes: 4 additions & 0 deletions source/module_basis/module_ao/parallel_orbitals.h
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,10 @@ class Parallel_Orbitals : public Parallel_2D
int get_col_size(int iat) const;
int get_row_size(int iat) const;

int get_nbands() const;

int nbands = 0;

/**
* @brief gather global indexes of orbitals in this processor
* get_indexes_row() : global indexes (~NLOCAL) of rows of Hamiltonian matrix in this processor
Expand Down
8 changes: 7 additions & 1 deletion source/module_cell/parallel_kpoints.h
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,12 @@ class Parallel_Kpoints
return startpro_pool[pool];
}

// get the maximum number of k-points in all pools
int get_max_nks_pool() const
{
return *std::max_element(nks_pool.begin(), nks_pool.end());
}

private:

int kpar = 0; // number of pools
Expand All @@ -76,7 +82,7 @@ class Parallel_Kpoints
void get_startk_pool(const int& nkstot);
void get_whichpool(const int& nkstot);

void set_startpro_pool(void);
void set_startpro_pool();
#endif
};

Expand Down
3 changes: 2 additions & 1 deletion source/module_cell/test/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -131,4 +131,5 @@ AddTest(
TARGET cell_index_test
LIBS ${math_libs} base device
SOURCES cell_index_test.cpp ../cell_index.cpp
)
)

21 changes: 21 additions & 0 deletions source/module_esolver/esolver_ks_lcao.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,27 @@ void ESolver_KS_LCAO<TK, TR>::before_all_runners(const Input_para& inp, UnitCell
this->pelec->fixed_weights(PARAM.inp.ocp_kb, GlobalV::NBANDS, GlobalV::nelec);
}

// 15) if kpar is not divisible by nks, print a warning
if (GlobalV::KPAR_LCAO > 1)
{
if (this->kv.get_nks() % GlobalV::KPAR_LCAO != 0)
{
ModuleBase::WARNING("ESolver_KS_LCAO::before_all_runners",
"nks is not divisible by kpar.");
std::cout << "\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
"%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
"%%%%%%%%%%%%%%%%%%%%%%%%%%" << std::endl;
std::cout << " Warning: nks (" << this->kv.get_nks() << ") is not divisible by kpar ("
<< GlobalV::KPAR_LCAO << ")." << std::endl;
std::cout << " This may lead to poor load balance. It is strongly suggested to" << std::endl;
std::cout << " set nks to be divisible by kpar, but if this is really what" << std::endl;
std::cout << " you want, please ignore this warning." << std::endl;
std::cout << "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
"%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
"%%%%%%%%%%%%\n";
}
}

ModuleBase::timer::tick("ESolver_KS_LCAO", "before_all_runners");
return;
}
Expand Down
Loading
Loading