Add condition param to autotune; Add feature engineering module #426

rayrayraykk · 2022-11-10T10:14:18Z

Main changes

condition param

The style are shown below:

   xgb_base.use  train.optimizer.lr  train.optimizer.num_of_trees  performance
0          True                 NaN                           5.0     0.59876
1         False            0.694859                           NaN     0.599826
2         False            0.876475                           NaN     0.600004

wandb

How to enable:
Install wandb and setup init configs for wandb
Run with wandb.use True:

python federatedscope/hpo.py --cfg federatedscope/autotune/baseline/fedex_vfl.yaml wandb.use True wandb.name_project vfl_demo wandb.name_user vfl_demo

Feature engineering

I implement Feature engineering as a wrapper for workers.

FederatedScope
├── federatedscope
│   ├── core           
│   |   ├── ...
│   |   ├── feature
│   |   |   ├── hFL
│   |   |   |   ├── ...
│   |   |   ├── vFL
│   |   |   |   ├── preprocess
│   |   |   |   |   ├── log_transform.py
│   |   |   |   |   ├── quantile_binning.py
│   |   |   |   |   ├── ...
│   |   |   |   ├── selection
│   |   |   |   |   ├── correlation_filter.py
│   |   |   |   |   ├── iv_filter.py
│   |   |   |   |   ├── ...

How to use:

python federatedscope/main.py --cfg federatedscope/vertical_fl/xgb_base/baseline/xgb_base_on_adult.yaml feat_engr.type vfl_instance_norm

joneswong · 2022-11-10T11:29:47Z

federatedscope/autotune/algos.py

@@ -57,7 +57,7 @@ def run(self):
                               server_class=get_server_cls(self._trial_cfg),
                               client_class=get_client_cls(self._trial_cfg),
                               config=self._trial_cfg.clone(),
-                               client_config=self._client_cfgs)
+                               client_configs=self._client_cfgs)


to avoid such kinds of mistakes, we need to improve the coverage of our unit test cases

Added in GitHub actions.

joneswong

good job! conditional search space is prevalent, and we really need it.

xieyxclack

The feature engineering module looks good to me!

xieyxclack · 2022-12-12T03:01:58Z

federatedscope/core/configs/cfg_data.py

@@ -64,6 +64,23 @@ def extend_data_cfg(cfg):
    cfg.data.quadratic.min_curv = 0.02
    cfg.data.quadratic.max_curv = 12.5

+    # feature engineer


It should be ''feature engineering''?

xieyxclack · 2022-12-12T03:03:43Z

federatedscope/core/configs/cfg_fl_setting.py

@@ -67,10 +67,10 @@ def extend_fl_setting_cfg(cfg):
    # ---------------------------------------------------------------------- #
    # Vertical FL related options (for demo)
    # ---------------------------------------------------------------------- #
+    cfg.vertical_dims = [5, 10]  # Avoid to be removed when `use` is False


Why we still need vertical_dims if vertical.use=False?

xgb.dim and vertical.dims are merged into one arg vertical_dims.

so what is this? I guess it is the input dim. If it is the case, why do we need it?

so what is this? I guess it is the input dim. If it is the case, why do we need it?

@qbc2016 Could you explain it briefly for me?

xieyxclack · 2022-12-12T03:21:29Z

federatedscope/core/feature/vfl/preprocess/instance_norm.py

+            self._init_data_related_var()
+            self.trigger_train_func(**self.kwargs_for_trigger_train_func)
+
+    # Bind method to instance


It looks good! @qbc2016 Maybe we can improve some modules in FL-Tree like this

joneswong · 2022-12-13T03:20:37Z

federatedscope/autotune/run.py

@@ -0,0 +1,14 @@
+def run_scheduler(scheduler, cfg, client_cfgs=None):


please provide docstring

joneswong · 2022-12-13T03:23:24Z

federatedscope/autotune/smac.py

@@ -16,21 +16,26 @@ def run_smac(cfg, scheduler, client_cfgs=None):

    def optimization_function_wrapper(config):


please supplement docstring for this method. It seems that understanding what its input is largely help figuring out what this method does.

joneswong · 2022-12-13T03:29:58Z

federatedscope/autotune/utils.py

+
+
+def log2wandb(trial, config, results, trial_cfg):
+    import wandb


As I have no idea about the overhead of repeatedly importing a module, I am wondering is it appropriate to do this? Have you benchmarked such a behavior? Here is a related thread: https://stackoverflow.com/questions/128478/should-import-statements-always-be-at-the-top-of-a-module

joneswong · 2022-12-13T03:38:45Z

federatedscope/core/feature/vfl/preprocess/instance_norm.py

+
+def wrap_instance_norm_server(worker):
+    """
+    This function is to perform instance norm vfl tabular data for server.


what do you mean by "perform instance norm vfl tabular data"?

joneswong · 2022-12-13T03:49:18Z

federatedscope/core/fed_runner.py

@@ -160,7 +164,7 @@ def _setup_server(self, resource_info=None, client_resource_info=None):
                wrap_nbafl_server
            wrap_nbafl_server(server)
        logger.info('Server has been set up ... ')
-        return server
+        return self.feat_engr_wrapper_server(server)


It is ok to inject those feature engineering procedures into an FL course in this way, but we shall change the instantiation of workers to a better (more general and unified) way. Actually, it is quite confusing to wrap a worker by feature engineering-related wrapper. Feature engineering is just a tiny step in an FL course, which doesn't change a worker significantly. One usual way to instantiate a worker from a collection of such pluggable behaviors is to use factory pattern I guess.

joneswong

approved.

joneswong

LGTM

rayrayraykk added 3 commits November 10, 2022 17:08

add yaml to run fedex + vFL

617a0ae

add condition param to support different model

2610730

remove print

df3d5ad

rayrayraykk added the enhancement New feature or request label Nov 10, 2022

rayrayraykk requested a review from joneswong November 10, 2022 10:14

Exclude ss yaml for test

70086de

joneswong reviewed Nov 10, 2022

View reviewed changes

joneswong previously approved these changes Nov 10, 2022

View reviewed changes

joneswong added the FedHPO FedHPO related label Nov 10, 2022

rayrayraykk and others added 2 commits November 11, 2022 11:01

Merge branch 'alibaba:master' into demo_vFL

0ab58fd

fix summarize_hpo_results()

1fe2d6e

rayrayraykk dismissed joneswong’s stale review via 1fe2d6e November 11, 2022 03:54

rayrayraykk and others added 8 commits November 11, 2022 12:12

optimize typo

1991c91

add wandb for autotune

7cd522a

fix minor bugs

d8ce865

fix bool

71211d9

remove print

85f2d8c

last commit

0f03ecb

merge master

62abf7d

Merge branch 'alibaba:master' into demo_vFL

e4e4537

rayrayraykk changed the title ~~Add condition param to autotune~~ [WIP]Add condition param to autotune; Add feature engineering module Nov 21, 2022

rayrayraykk added 3 commits November 21, 2022 17:19

add min-max scaling for vfl

6c8a78d

[WIP] add instance norm

bd99a09

add instance_norm for vFL

68511ff

rayrayraykk changed the title ~~[WIP]Add condition param to autotune; Add feature engineering module~~ Add condition param to autotune; Add feature engineering module Nov 22, 2022

rayrayraykk and others added 5 commits November 22, 2022 15:56

fix docstring

51b45cd

update docstring

c068d14

Merge branch 'alibaba:master' into demo_vFL

040f8b3

fix according to xgb updates

7e81d81

add interface for trigger_for_feat_engr

5ddee3b

rayrayraykk added 9 commits December 1, 2022 11:21

implement vfl binning

db04d5d

add comment for binning

78c0f82

add selection filter

e347954

fix minor bugs

a0287d4

fix minor bugs

0c68059

add corrcoef

9b7b4d8

keep style consistent

ac4177d

update yaml

f747b4c

add ss for instance norm

ef3004f

rayrayraykk requested a review from qbc2016 December 5, 2022 08:10

rayrayraykk added 3 commits December 5, 2022 16:12

rollback for hpo ss

9f6621b

WIP IV filter

e8d154b

fix iv filter

446975d

rayrayraykk changed the title ~~[WIP] Add condition param to autotune; Add feature engineering module~~ Add condition param to autotune; Add feature engineering module Dec 5, 2022

rayrayraykk requested a review from xieyxclack December 6, 2022 02:39

rayrayraykk added 2 commits December 8, 2022 17:40

Merge branch 'master' into demo_vFL

06d6ab6

Merge branch 'master' into demo_vFL

5b8a72e

xieyxclack reviewed Dec 12, 2022

View reviewed changes

Update cfg_data.py

42a3e51

joneswong reviewed Dec 13, 2022

View reviewed changes

federatedscope/autotune/run.py

@@ -0,0 +1,14 @@

def run_scheduler(scheduler, cfg, client_cfgs=None):

Copy link

Collaborator

joneswong Dec 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please provide docstring

joneswong reviewed Dec 13, 2022

View reviewed changes

joneswong previously approved these changes Dec 13, 2022

View reviewed changes

add docstring

47c256a

rayrayraykk dismissed joneswong’s stale review via 47c256a December 13, 2022 08:48

joneswong approved these changes Dec 13, 2022

View reviewed changes

joneswong merged commit a32a8f9 into alibaba:master Dec 13, 2022

joneswong mentioned this pull request Dec 13, 2022

[Discussion] Refactor worker class #472

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add condition param to autotune; Add feature engineering module #426

Add condition param to autotune; Add feature engineering module #426

rayrayraykk commented Nov 10, 2022 •

edited

Loading

joneswong Nov 10, 2022

rayrayraykk Nov 30, 2022

joneswong left a comment

xieyxclack left a comment

xieyxclack Dec 12, 2022

xieyxclack Dec 12, 2022

rayrayraykk Dec 12, 2022

joneswong Dec 13, 2022

rayrayraykk Dec 13, 2022

xieyxclack Dec 12, 2022

joneswong Dec 13, 2022

joneswong Dec 13, 2022

joneswong Dec 13, 2022

joneswong Dec 13, 2022

joneswong Dec 13, 2022

joneswong left a comment

joneswong left a comment

		@@ -0,0 +1,14 @@
		def run_scheduler(scheduler, cfg, client_cfgs=None):

		@@ -16,21 +16,26 @@ def run_smac(cfg, scheduler, client_cfgs=None):

		def optimization_function_wrapper(config):



		def log2wandb(trial, config, results, trial_cfg):
		import wandb

Add condition param to autotune; Add feature engineering module #426

Add condition param to autotune; Add feature engineering module #426

Conversation

rayrayraykk commented Nov 10, 2022 • edited Loading

Main changes

condition param

wandb

Feature engineering

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joneswong left a comment

Choose a reason for hiding this comment

xieyxclack left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joneswong left a comment

Choose a reason for hiding this comment

joneswong left a comment

Choose a reason for hiding this comment

rayrayraykk commented Nov 10, 2022 •

edited

Loading