Features/#360, #433 PEPs configuration support

[KrGolovin]

See issues: #433, #360

[iromeo]

Completion list missing some variants (nested yaml items, e.g. 'key1') and allows non-identifiers (e.g. key3:key31) that leads to syntax errors in Snake file

Pipeline:

# pep1/config_360.yaml
pep_version: "2.0.0"
sample_table: ../config/sample_table.tsv
donor_table: ../config/donor_table.tsv

key1:
  key11: 
    "val11"
key2:
  - key21: "val21"
key3:key31: "val31"
key4.key41: "val41"
key5.key51:
  key52: "val52"
sample_modifiers:
  derive:
    attributes: [file_path]
    sources:
      source1: /data/lab/project/{organism}_{time}h.fastq
      source2: /path/from/collaborator/weirdNamingScheme_{external_id}.fastq
      
'not_ident: $$ ifier':
  sss: "ddd"
      
# ------ Snakefile --------
pepfile: "pep1/config_360.yaml"

print("--- All keys: ----")
print(pep.config)
print("--- key1: ----")
print(pep.config.key1)
print(pep.config.key1.key11)
print(pep.config['key1'])
print(pep.config['key1'].key11)
print(pep.config.key1['key11'])
print("--- key2: ----")
print(pep.config.key2)
print(pep.config.key2[0]['key21'])
print("--- key3: ----")
print(pep.config['key3:key31'])
# print(pep.config['key3']) # <- KeyError in line .. 'key3'
# print(pep.config.key3) # <- AttributeError in line .. 'key3'
print("--- key4: ----")
print(pep.config['key4.key41'])
# print(pep.config.key4) # <- AttributeError in line .. 'key4'
# print(pep.config.key4.key41) # <- AttributeError in line .. 'key4'
print("--- key5: ----")
print(pep.config['key5.key51'])
print(pep.config['key5.key51'].key52)
print("--- sample_modifiers: ----")
print(pep.config.sample_modifiers.derive)
print(pep.config['not_ident: $$ ifier'])

rule all:

Output:

$ snakemake --snakefile rule_360/rule_360.smk -c1 all  -n -q
--- All keys: ----
PathExAttMap
pep_version: 2.0.0
sample_table: /Users/romeo/work/snakecharm/snakemaek_examples/untitled10/pep1/../config/sample_table.tsv
donor_table: ../config/donor_table.tsv
key1:
  key11: val11
key2: 
 - {'key21': 'val21'}
key3:key31: val31
key4.key41: val41
key5.key51:
  key52: val52
--- key1: ----
PathExAttMap
key11: val11
val11
PathExAttMap
key11: val11
val11
val11
<...>
--- key2: ----
[{'key21': 'val21'}]
val21
--- key3: ----
val31
--- key4: ----
val41
--- key5: ----
PathExAttMap
key52: val52
val52
--- sample_modifiers: ----
PathExAttMap
attributes: 
 - file_path
sources:
  source1: /data/lab/project/{organism}_{time}h.fastq
  source2: /path/from/collaborator/weirdNamingScheme_{external_id}.fastq
  
PathExAttMap
sss: ddd

[iromeo]

it is reasonable to assume that getYamlFile(..) should return not any PsiFile? (e.g. PyFile, SmkFile, etc) but only YAMLFile or null.

So the signature of this method could be improoved.

[iromeo]

We've already discussed several times that low-level PSI API access is not recommended and better to use high level API instead. Especially when you deal with some 3-rd party PSI elements where parser impl could be changed at any time and such low-level code will stop working. Please find appropriate YAML PSI api and use it instead of low level API.

[iromeo]

The whole method 1) is using low level PSI access and parser impl details 2) duplicates some code from SmkPepfileReference reference 3) doesn't work properly if reference uses single quoting

Re-implement this using high-level PSI access (e.g. without LeafPsiElement, nextSibling, lastChild, text methods). Use the fact, that if everyting is ok, you already have SmkPepfileReference reference here and it is know how to resolve it's target, no need to copy resolve implementation here. Before commiting this change please show me new impl in Slack.

[iromeo]

Not all items are valid for completion list, e.g. key1:key2 is invalid option, because pep.config.key1:key2 is a syntax error. Make code smarter and add to completion list only valid identifiers

[iromeo]

1. The cache is organized as SmkPepConfigCollector object with private static field cache. This is memory leak & not threads safe. Should be changed.

2. pepfile, pepschema isn't in keyword comletion

3. seems reference completion after pepfile: "<>" is very slow.

[iromeo]

#360 is done, merged into master. #418 code was disabled in 9743246 commit