Simplification not working with python 3.9

[thiborose]

Hi,

In a python 3.9 environment, I could not run the simplification script - I was having this error :

python scripts/simplify.py scripts/examples.fr --model-name muss_fr_mined
usage: simplify.py [-h] [--no-progress-bar] [--log-interval LOG_INTERVAL] [--log-format LOG_FORMAT]
                   [--tensorboard-logdir TENSORBOARD_LOGDIR] [--seed SEED] [--cpu] [--tpu] [--bf16] [--memory-efficient-bf16] [--fp16]
                   [--memory-efficient-fp16] [--fp16-no-flatten-grads] [--fp16-init-scale FP16_INIT_SCALE]
                   [--fp16-scale-window FP16_SCALE_WINDOW] [--fp16-scale-tolerance FP16_SCALE_TOLERANCE]
                   [--min-loss-scale MIN_LOSS_SCALE] [--threshold-loss-scale THRESHOLD_LOSS_SCALE] [--user-dir USER_DIR]
                   [--empty-cache-freq EMPTY_CACHE_FREQ] [--all-gather-list-size ALL_GATHER_LIST_SIZE]
                   [--model-parallel-size MODEL_PARALLEL_SIZE] [--checkpoint-suffix CHECKPOINT_SUFFIX]
                   [--checkpoint-shard-count CHECKPOINT_SHARD_COUNT] [--quantization-config-path QUANTIZATION_CONFIG_PATH] [--profile]
                   [--criterion {label_smoothed_cross_entropy,adaptive_loss,wav2vec,cross_entropy,sentence_ranking,label_smoothed_cross_entropy_with_alignment,composite_loss,sentence_prediction,ctc,nat_loss,masked_lm,legacy_masked_lm_loss,vocab_parallel_cross_entropy}]
                   [--tokenizer {nltk,space,moses}]
                   [--bpe {bert,byte_bpe,sentencepiece,bytes,subword_nmt,gpt2,fastbpe,hf_byte_bpe,characters}]
                   [--optimizer {adafactor,adamax,nag,sgd,lamb,adagrad,adadelta,adam}]
                   [--lr-scheduler {reduce_lr_on_plateau,inverse_sqrt,cosine,triangular,tri_stage,polynomial_decay,fixed}]
                   [--scoring {wer,sacrebleu,bleu,chrf}] [--task TASK] [--num-workers NUM_WORKERS]
                   [--skip-invalid-size-inputs-valid-test] [--max-tokens MAX_TOKENS] [--batch-size BATCH_SIZE]
                   [--required-batch-size-multiple REQUIRED_BATCH_SIZE_MULTIPLE]
                   [--required-seq-len-multiple REQUIRED_SEQ_LEN_MULTIPLE] [--dataset-impl DATASET_IMPL]
                   [--data-buffer-size DATA_BUFFER_SIZE] [--train-subset TRAIN_SUBSET] [--valid-subset VALID_SUBSET]
                   [--validate-interval VALIDATE_INTERVAL] [--validate-interval-updates VALIDATE_INTERVAL_UPDATES]
                   [--validate-after-updates VALIDATE_AFTER_UPDATES] [--fixed-validation-seed FIXED_VALIDATION_SEED]
                   [--disable-validation] [--max-tokens-valid MAX_TOKENS_VALID] [--batch-size-valid BATCH_SIZE_VALID]
                   [--curriculum CURRICULUM] [--gen-subset GEN_SUBSET] [--num-shards NUM_SHARDS] [--shard-id SHARD_ID]
                   [--distributed-world-size DISTRIBUTED_WORLD_SIZE] [--distributed-rank DISTRIBUTED_RANK]
                   [--distributed-backend DISTRIBUTED_BACKEND] [--distributed-init-method DISTRIBUTED_INIT_METHOD]
                   [--distributed-port DISTRIBUTED_PORT] [--device-id DEVICE_ID] [--local-rank LOCAL_RANK] [--distributed-no-spawn]
                   [--ddp-backend {c10d,no_c10d}] [--bucket-cap-mb BUCKET_CAP_MB] [--fix-batches-to-gpus] [--find-unused-parameters]
                   [--fast-stat-sync] [--broadcast-buffers] [--distributed-wrapper {DDP,SlowMo}] [--slowmo-momentum SLOWMO_MOMENTUM]
                   [--slowmo-algorithm SLOWMO_ALGORITHM] [--localsgd-frequency LOCALSGD_FREQUENCY] [--nprocs-per-node NPROCS_PER_NODE]
                   [--pipeline-model-parallel] [--pipeline-balance PIPELINE_BALANCE] [--pipeline-devices PIPELINE_DEVICES]
                   [--pipeline-chunks PIPELINE_CHUNKS] [--pipeline-encoder-balance PIPELINE_ENCODER_BALANCE]
                   [--pipeline-encoder-devices PIPELINE_ENCODER_DEVICES] [--pipeline-decoder-balance PIPELINE_DECODER_BALANCE]
                   [--pipeline-decoder-devices PIPELINE_DECODER_DEVICES] [--pipeline-checkpoint {always,never,except_last}]
                   [--zero-sharding {none,os}] [--path PATH] [--remove-bpe [REMOVE_BPE]] [--quiet] [--model-overrides MODEL_OVERRIDES]
                   [--results-path RESULTS_PATH] [--beam N] [--nbest N] [--max-len-a N] [--max-len-b N] [--min-len N]
                   [--match-source-len] [--no-early-stop] [--unnormalized] [--no-beamable-mm] [--lenpen LENPEN] [--unkpen UNKPEN]
                   [--replace-unk [REPLACE_UNK]] [--sacrebleu] [--score-reference] [--prefix-size PS] [--no-repeat-ngram-size N]
                   [--sampling] [--sampling-topk PS] [--sampling-topp PS] [--constraints [{ordered,unordered}]] [--temperature N]
                   [--diverse-beam-groups N] [--diverse-beam-strength N] [--diversity-rate N] [--print-alignment] [--print-step]
                   [--lm-path PATH] [--lm-weight N] [--iter-decode-eos-penalty N] [--iter-decode-max-iter N]
                   [--iter-decode-force-max-iter] [--iter-decode-with-beam N] [--iter-decode-with-external-reranker]
                   [--retain-iter-history] [--retain-dropout]
                   [--retain-dropout-modules RETAIN_DROPOUT_MODULES [RETAIN_DROPOUT_MODULES ...]]
                   [--decoding-format {unigram,ensemble,vote,dp,bs}]
simplify.py: error: argument --dataset-impl: invalid typing.Optional[fairseq.dataclass.utils.Choices] value: 'raw' 

Downgrading to python 3.7 and re running the setup fixed it. I'm reporting this since 20848f6 was intended to fix compatibility with 3.9, but it seems not to be working.

Anyhow, thanks for this great piece!

[louismartin]

Hi, thanks for raising this issue !

Are you sure that fairseq was at the 0.10.0 version as intended when you were in python 3.9 ? Sometimes it does not downgrades already installed libraries...
You should be able to force a downgrade of fairseq with pip install fairseq==0.10.0

[thiborose]

I checked at that moment with pip list, and fairseq had the expected version (0.10.0). (It was a fresh install)

[louismartin]

Ok that's weird :O
Let me check on my end as well.

[louismartin]

For future reference, we cannot install 0.10.2 on python 3.9 due to this bug: facebookresearch/fairseq#3535

[louismartin]

Yes I can reproduce the bug with python 3.9 and fairseq 0.10.0, I don't understand where it comes from but maybe the best thing is to downgrade to python < 3.9 for now until fairseq updates to a new version.