Regexdispatcher #1944

e1528532 · 2018-04-27T18:00:29Z

Purpose

Calculates regexes for checker plugins for the use with the typechecker, e.g. converting check/range=0-5000 to a regex representation of that range, also supports the enum checks currently.

Checklist

commit messages are fine ("module: short statement" syntax and refer to issues)
I added unit tests
I ran all tests locally and everything went fine
affected documentation is fixed
I added code comments, logging, and assertions (see doc/CODING.md)
meta data is updated (e.g. README.md of plugins)
release notes are updated (doc/news/_preparation_next_release.md)

@markus2330 based on the typechecker branch, the question here is how many things i want to support. I'd add the validation keyword as it more or less represents regexes directly but then lets see, i'd like to get back to writing soon. Basically only the src/plugins/regexdispatcher folder is really relevant and it will be easier to see once the prototype is merged to main.

e1528532 · 2018-06-04T15:13:04Z

@ingwinlu is there possible some cache involved in the docker container? Can i look into the workspace? This issue

setup: can't find source for Elektra/PluginProcess in

/home/jenkins/workspace/libelektra_PR-1944-NHTCZWJOQXIAMT5YFRVQPHMSUTCZBFPXHGIAGV2EUFI3EU7I6QNQ@2/src/bindings/haskell/src,

dist/build/autogen

is not reproducible on my local debian, the initial case issue should be fixed with the commit dcf7995.

ingwinlu · 2018-06-04T16:21:58Z

is there possible some cache involved in the docker container?

No, workspace is cleaned before and after every checkout

Can i look into the workspace?

The workspace is deleted completely for that test case.

This issue is not reproducible on my local debian

It is for me. Always fails.

Double check your last commit.
You have a typo.

Your commit:
haskell/src/Elektra/Pluginprocess.chs → ...ngs/haskell/src/elektra/PluginProcess.chs
The error:
can not find Elektra/PluginProcess

So maybe try renaming elektra/PluginProcess.chs to Elektra/PluginProcess.chs.

e1528532 · 2018-06-05T11:14:19Z

i see. since i use git on mac that doesn't really differentiate between cases i've totally overlooked that. i guess it was still cached somehow on my debian vm. good catch!

ingwinlu · 2018-06-05T11:43:22Z

good catch!

I will tell the build system the next time i log in.

markus2330 · 2018-06-05T14:00:18Z

Sorry, we both fixed the same problem. Please rebase.

…fix typechecker dependency delivery with sandbox, fix test order

…n the source folder

…done in haskell

…rocess

e1528532 · 2018-06-05T16:52:39Z

@markus2330 here you go finally

markus2330

Thank you! Nice PR but it is unclear for me what the Regexdispatcher is for!

markus2330 · 2018-06-05T17:09:30Z

src/plugins/regexdispatcher/README.md

+- infos/recommends =
+- infos/placements = presetstorage
+- infos/status = maintained experimental
+- infos/metadata =


Which metadata is used?

markus2330 · 2018-06-05T17:09:49Z

src/plugins/regexdispatcher/README.md

+
+## Introduction
+
+A plugin which generates regex-representations of specification keywords for the 


Input/output is unclear: which meta data is consumed, which is generated?

markus2330 · 2018-06-05T17:10:41Z

src/plugins/regexdispatcher/README.md

+
+A plugin which generates regex-representations of specification keywords for the 
+typechecker plugin before saving a configuration specification. Intended to be 
+mounted before the typechecker plugin.


"before" is misleading: the actual order need to be specified with placements or infos/ordering

markus2330 · 2018-06-05T17:11:24Z

src/plugins/regexdispatcher/README.md

+
+In order to generate regex representations for different specification keywords, mount a configuration specification along with this plugin. 
+
+`kdb mount <specification> spec/<path> <storage plugin to read the specification> regexdispatcher typechecker`


Why not let the typechecker depend on the regexdispatcher? Or is it usful to use the typechecker without regexdispatcher!

Please describe more clearly why the regexdispatcher is needed.

i will. basically the typechecker can also work without the regexdispatcher for certain metadata like check/long or check/validation or similar which do not need an intermediate translation process.
However something like check/enum/#1=A, check/enum/#2=B needs to be translated to the regex A|B, which the regexdispatcher handles to keep concerns separated.
Furthermore other people may implement similar plugins that translate their own metakeys to regexes (that don't necessarily have to be implemented in haskell) and so i concluded to keep those things separated from the typechecker plugin.

Yes, I assumed something like that. But the docu does not say it.

markus2330 · 2018-06-05T17:11:50Z

src/plugins/regexdispatcher/README.md

+* cabal, the haskell build system, usually bundled with ghc
+* augeas, which provides libfa utilized by this plugin
+
+## Limitations


Wrong section name, you describe features and not limitations here.

markus2330 · 2018-06-05T17:15:37Z

src/libs/pluginprocess/pluginprocess.c

+ * @param plugin a pointer to the plugin
+ * @param data the pointer to the data
+ */
+void elektraPluginProcessSetData (Plugin * handle, void * data)


How is this serialized and sent to the other process?

it is not, the proxied plugins just run in the child process and on linux it seems to share the memory between the parent and the child process. Thus if the child sets any plugin data directly, this would override the plugin data of the parent process, causing the pluginprocess library to fail.
Therefore i store it inside the pluginprocess data structure on the children's side to avoid this problem.

markus2330 · 2018-06-05T17:16:18Z

src/plugins/haskell/haskell.c.in

@@ -16,6 +16,11 @@
 #include <kdbpluginprocess.h>
 #include <stdio.h>

+// TODO:


Thank you for writing the TODO. Having a way to store plugin data is essential.

actually i've forgotten to remove this TODO, this is already implemented as i needed it to pass the linux builds.

Even better 👍

markus2330 · 2018-06-05T17:16:46Z

src/plugins/regexdispatcher/CMakeLists.txt

+find_package (fa)
+
+if (LIBFA_FOUND)
+	add_haskell_plugin (regexdispatcher MODULE "RegexDispatcher" SANDBOX_ADD_SOURCES "src/libs/typesystem/libfa")


how are the tests specified?

The tests are written in C

markus2330 · 2018-06-05T17:17:13Z

src/plugins/regexdispatcher/Elektra/Parsers.hs

+--
+-- @file
+--
+-- @brief Some parsers which are used in the translation process


Not clear to me what the parsers are for.

markus2330 · 2018-06-05T17:17:45Z

src/plugins/regexdispatcher/README.md

@@ -0,0 +1,36 @@
+- infos = Information about the regexdispatcher plugin is in keys below
+- infos/author = e1528532 <[email protected]>


Please write your name.

e1528532 · 2018-06-05T18:03:40Z

some very good points you've raised there especially regarding the ordering and the metakeys and the purpose.

markus2330 · 2018-06-05T18:43:30Z

doc/METADATA.ini

+usedby/plugins = regexdispatcher
+description = the generated regex representing the type of check/enum/# and check/enum/multi metakeys
+
+[elektra/spec/regex/check/range]


Some thoughts about the metadata name:

the elektra prefix does not give information, all meta data is for and from Elektra.

it should also not start with spec (do not be confused with the namespace).

And wouldn't it be better that we only have a single metadata and the regex is always merged together? Do you need the information of where the regex comes from?

What was the idea of giving this name?

you are right. as the regexdispatcher does all the preprocessing this essentially yields a check/validation keyword in the end all the time, so in that case i don't need further information where it came from i think. i will try that idea out.

e1528532 · 2018-06-08T15:43:38Z

@markus2330 as discussed in #160 these few checks are unlikely to every succeed and thus i'd suggest merging it now? as far as i know they don't check any haskell related things anyway.

markus2330 · 2018-06-08T20:13:36Z

thank you!

e1528532 added the work in progress label Apr 27, 2018

e1528532 force-pushed the regexdispatcher branch 4 times, most recently from 73f5c22 to 183d1c6 Compare May 29, 2018 13:26

e1528532 force-pushed the regexdispatcher branch from 2dac6d7 to dcf7995 Compare June 4, 2018 14:49

e1528532 added 19 commits June 5, 2018 16:03

haskell-cmake-improvements: add release note

aae608b

haskell-cmake-improvements: adjust the release notes to expected format

78fe8ba

typechecker-regex-prototype: improve/finish cmake packaging process, …

91c2807

…fix typechecker dependency delivery with sandbox, fix test order

typechecker-regex-prototype: improve build process; avoid artifacts i…

0546bd0

…n the source folder

typechecker-regex-prototype: require hint 0.7.0 or it fails on stretch

2ef3869

typechecker-regex-prototype: fix/unify sandbox handling

d1c8684

regexdispatcher: add first draft of the plugin

1657d17

regexdispatcher: finish range dispatcher

3c4f964

regexdispatcher: adjust typechecker to dispatched regexes

61e7d66

regexdispatcher: add dispatcher for enum checks

26aa49a

regexdispatcher: add travis test

4fb298b

regexdispatcher: fix compilation on linux

9c1e001

regexdispatcher: reformat cmake

9e72ede

regexdispatcher: mount along typechecker

4bf7de5

regexdispatcher: fix adjusted typechecker example

0583114

regexdispatcher: rebase fixies

58d972a

regexdispatcher: adjust cabal file

fccad52

regexdispatcher: prepare for new prelude loading implementation

3b241c9

regexdispatcher: add invoke bindings for alternative prelude handling

3b48a98

e1528532 added 12 commits June 5, 2018 16:03

regexdispatcher: update release notes

00c6af9

regexdispatcher: adjust ifKey and whenKey syntax to how its normally …

66c6de3

…done in haskell

regexdispatcher: add validation support

4d93782

regexdispatcher: correct order & minimize inversion results

5e39fda

regexdispatcher: small simplification

2bbf2c2

regexdispatcher: adjust rebase issue

f9ce3e0

regexdispatcher: add validation example to typechecker

98b38fd

regexdispatcher: remove ununsed function

efa7502

regexdispatcher: use raw pointers directly, figure out segfaults

0dd7dca

regexdispatcher: fix plugin data handling in combination with pluginp…

96be930

…rocess

regexdispatcher: correct case of file

176a63a

regexdispatcher: fix case ElektraInitiative#2

9bee5b1

e1528532 force-pushed the regexdispatcher branch from c81afb0 to 9bee5b1 Compare June 5, 2018 14:04

e1528532 added ready to merge and removed work in progress labels Jun 5, 2018

markus2330 suggested changes Jun 5, 2018

View reviewed changes

markus2330 removed the ready to merge label Jun 5, 2018

regexdispatcher: resolve the review comments

8c05f4a

markus2330 reviewed Jun 5, 2018

View reviewed changes

regexdispatcher: use check/validation metakey for generated regexes

0f75668

e1528532 mentioned this pull request Jun 7, 2018

optimizations-off + testshell_markdown_typechecker = unstable #2057

Closed

regexdispatcher: add usedby to check/validation metadata

e589848

e1528532 added the ready to merge label Jun 8, 2018

markus2330 approved these changes Jun 8, 2018

View reviewed changes

markus2330 merged commit 54e7c17 into ElektraInitiative:master Jun 8, 2018

e1528532 mentioned this pull request Jun 11, 2018

typechecker: build fails on travis #2015

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regexdispatcher #1944

Regexdispatcher #1944

e1528532 commented Apr 27, 2018 •

edited

Loading

e1528532 commented Jun 4, 2018

ingwinlu commented Jun 4, 2018 •

edited

Loading

e1528532 commented Jun 5, 2018

ingwinlu commented Jun 5, 2018

markus2330 commented Jun 5, 2018

e1528532 commented Jun 5, 2018

markus2330 left a comment

markus2330 Jun 5, 2018

markus2330 Jun 5, 2018

markus2330 Jun 5, 2018

markus2330 Jun 5, 2018

e1528532 Jun 5, 2018 •

edited

Loading

markus2330 Jun 5, 2018

markus2330 Jun 5, 2018

markus2330 Jun 5, 2018

e1528532 Jun 5, 2018

markus2330 Jun 5, 2018

e1528532 Jun 5, 2018

markus2330 Jun 5, 2018

markus2330 Jun 5, 2018

e1528532 Jun 5, 2018

markus2330 Jun 5, 2018

markus2330 Jun 5, 2018

e1528532 commented Jun 5, 2018

markus2330 Jun 5, 2018

e1528532 Jun 6, 2018

e1528532 commented Jun 8, 2018

markus2330 commented Jun 8, 2018


		## Introduction

		A plugin which generates regex-representations of specification keywords for the


		In order to generate regex representations for different specification keywords, mount a configuration specification along with this plugin.

		`kdb mount <specification> spec/<path> <storage plugin to read the specification> regexdispatcher typechecker`

		@@ -0,0 +1,36 @@
		- infos = Information about the regexdispatcher plugin is in keys below
		- infos/author = e1528532 <[email protected]>

Regexdispatcher #1944

Regexdispatcher #1944

Conversation

e1528532 commented Apr 27, 2018 • edited Loading

Purpose

Checklist

e1528532 commented Jun 4, 2018

ingwinlu commented Jun 4, 2018 • edited Loading

e1528532 commented Jun 5, 2018

ingwinlu commented Jun 5, 2018

markus2330 commented Jun 5, 2018

e1528532 commented Jun 5, 2018

markus2330 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

e1528532 Jun 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

e1528532 commented Jun 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

e1528532 commented Jun 8, 2018

markus2330 commented Jun 8, 2018

e1528532 commented Apr 27, 2018 •

edited

Loading

ingwinlu commented Jun 4, 2018 •

edited

Loading

e1528532 Jun 5, 2018 •

edited

Loading