Copy plugin (#223)

* ✨ initial prototype

* ✨ first working prototype that having relocated copy: into targets:. #165

* 📰 add missing unit test file

* ♿ deprecate copy: key word. #165

* 📰 add the missing file

* 🔨 code refactoring with TemplateTarget. replace all template tuples with TemplateTarget

* 💚 install pypi-mobans-pkg for testing

* 🔥 remove added test package: pypi-mobans-pkg

* 🐛 when new plugins are installed, call the lml scanner again so that newly installed packages are loaded, pypi-moban-pkg should be reloaded in the test, #165

* 🔨 refactor constants.py

* 🔥 remove previous fix for pypi-mobans-pkg not found. #165

* 📚 update copy action documentation and update integration tests 🔬

* 💄 minor update

* 🚜 relocating files

* 🚜 code refactoring on plugins.py

* 🚜 flatten concentrated files

* 🚜 fusing engines to template.py

* 🚜 relocate copier to plugins

* ✨ first working prototype: copy as template engine. #165

* 💚 fix unit test failure because of dependency

* 🔨 code refactoring, support template a directory as it used to, template-folder/a.jj2 become dest/a without jj2 suffix. it also means: copy-folder/me.copy becomes dest-folder/me but no effect if copy-folder/you, as it dest-folder/you

* 💚 try to fix broken unit tests in travis-ci

* 🐛 fix the bug where repo is not found if it is newly installed. #165

* 🔨 respect default template type

* 🔥 remove useless function as code evolves

* 🔨 code refactor magic copy label

* 🔬 test content forward engine, nickname copy engine. #165

* 🔥 remove useless report functions

* 🔥 remove unreachable test codes and remove unused code

* ✨ support group targets with template type, #165. And it is related to #62 with current restriction on template type as key. What will happen if template type as group key is unknown, atm it will be treated as jinja2 template type

* ✨ force template type regardless of individual template types. #165

* 💚 fix broken tests

* 🔬 more test coverage

* 🚜 rename docs directory

* 🔨 code refactoring

* 🔨 code refactoring on targets

* 🚜 re-organising file structures

* 🚜 re-organize repo related functionality

* 🚜 re-organize template target parsing functionality for moban file

* 📚 update documentation

* 📚 document 'copy' engine

* 🔨 black all source code

* 📚 update change log

* 💄 update gitignore

* 🔬 squeeze more test coverage from test code

* :micrscope: more test coverage

* 🔬 more tests on mobanfile targets

* 🔨 relocate test files

* 🔬 more tests on mobanfile targets

* 🚜 relocate test file

* 🔬 more tests

* 📚 update documentation index

* 📚 update change log

.gitignore

@@ -474,3 +474,14 @@ docs/level-9-moban-dependency-as-pypi-package/mytravis.yml
 docs/misc-1-copying-templates/misc-1-copying/
 docs/misc-1-copying-templates/test-dir/
 docs/misc-1-copying-templates/test-recursive-dir/
+docs/level-12-use-template-engine-extensions/a.output
+docs/level-13-any-data-override-any-data/a.output
+docs/level-13-any-data-override-any-data/b.output
+docs/level-15-copy-templates-as-target/target_in_short_form
+docs/level-15-copy-templates-as-target/target_without_template_type
+docs/level-15-copy-templates-as-target/test-dir/
+docs/level-15-copy-templates-as-target/test-recursive-dir/
+docs/level-16-group-targets-using-template-type/test-dir/
+docs/level-16-group-targets-using-template-type/test-recursive-dir/
+docs/level-17-force-template-type-from-moban-file/test-dir/
+docs/level-17-force-template-type-from-moban-file/test-recursive-dir/

.moban.cd/changelog.yml

@@ -1,6 +1,12 @@
 name: moban
 organisation: moremoban
 releases:
+- changes:
+  - action: Added
+    details:
+    - "`#165`: Copy as plugins"
+  date: unreleased
+  version: 0.4.0    
 - changes:
   - action: Added
     details:
@@ -11,7 +17,7 @@ releases:
     details:
     - "`#178`: UnboundLocalError: local variable 'target' referenced before assignment"
     - "`#169`: uses GitPython instead of barebone git commands"
-  date: 3.2.2019 
+  date: 03.02.2019
   version: 0.3.10
 - changes:
   - action: Updated

.moban.d/moban_gitignore.jj2

@@ -5,4 +5,15 @@ docs/level-9-moban-dependency-as-pypi-package/mytravis.yml
 docs/misc-1-copying-templates/misc-1-copying/
 docs/misc-1-copying-templates/test-dir/
 docs/misc-1-copying-templates/test-recursive-dir/
+docs/level-12-use-template-engine-extensions/a.output
+docs/level-13-any-data-override-any-data/a.output
+docs/level-13-any-data-override-any-data/b.output
+docs/level-15-copy-templates-as-target/target_in_short_form
+docs/level-15-copy-templates-as-target/target_without_template_type
+docs/level-15-copy-templates-as-target/test-dir/
+docs/level-15-copy-templates-as-target/test-recursive-dir/
+docs/level-16-group-targets-using-template-type/test-dir/
+docs/level-16-group-targets-using-template-type/test-recursive-dir/
+docs/level-17-force-template-type-from-moban-file/test-dir/
+docs/level-17-force-template-type-from-moban-file/test-recursive-dir/
 {% endblock %}

CHANGELOG.rst

@@ -1,7 +1,15 @@
 Change log
 ================================================================================
 
-0.3.10 - 3.2.2019
+0.4.0 - unreleased
+--------------------------------------------------------------------------------
+
+Added
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+#. `#165 <https://github.com/moremoban/moban/issues/165>`_: Copy as plugins
+
+0.3.10 - 03.02.2019
 --------------------------------------------------------------------------------
 
 Added

docs/README.rst

@@ -17,6 +17,9 @@ This section covers the use cases for moban. Please check them out individually.
 #. `Use template engine extensions`_
 #. `Any data overrides any data`_
 #. `Custom data loader`_
+#. `Copy templates as target`_
+#. `Group targets by template type`_
+#. `Force template type from moban file`_
 
 .. _Jinja2 command line: level-1-jinja2-cli
 .. _Template inheritance: level-2-template-inheritance
@@ -32,3 +35,6 @@ This section covers the use cases for moban. Please check them out individually.
 .. _Use template engine extensions: level-12-use-template-engine-extensions
 .. _Any data overrides any data: level-13-any-data-override-any-data
 .. _Custom data loader: level-14-custom-data-loader
+.. _Copy templates as target: level-15-copy-templates-as-target
+.. _Group targets by template type: level-16-group-targets-using-template-type
+.. _Force template type from moban file: level-17-force-template-type-from-moban-file

docs/index.rst

@@ -26,6 +26,12 @@ examples folder.
    level-9-moban-dependency-as-pypi-package/README.rst
    level-10-moban-dependency-as-git-repo/README.rst
    level-11-use-handlebars/README.rst
+   level-12-use-template-engine-extensions/README.rst
+   level-13-any-data-override-any-data/README.rst
+   level-14-custom-data-loader/README.rst
+   level-15-copy-templates-as-target/README.rst
+   level-16-group-targets-using-template-type/README.rst
+   level-17-force-template-type-from-moban-file/README.rst
 
 In pratice, the following use cases were found interesting to go along with.
 

docs/level-10-moban-dependency-as-git-repo/README.rst

@@ -1,7 +1,7 @@
 level 10: moban dependency as git repo
 ================================================================================
 
-Since the support to have a pypi package as dependency, the pro user of moban
+Since the support to have a pypi package as dependency, the moban pro user will
 find it more useful to have git repo so that the changes to static content
 could get propagate as it happens using git push and git pull.
 

docs/level-15-copy-templates-as-target/.moban.yml

@@ -0,0 +1,19 @@
+configuration:
+  template_dir:
+    - template-sources
+targets:
+  - output: simple.file.copy
+    template: file-in-template-sources-folder.txt
+    template_type: copy
+  - output: target_without_template_type
+    template: file_extension_will_trigger.copy
+  - target_in_short_form: as_long_as_this_one_has.copy
+  - output: "misc-1-copying/can-create-folder/if-not-exists.txt"
+    template: file-in-template-sources-folder.txt
+    template_type: copy
+  - output: "test-dir"
+    template: dir-for-copying
+    template_type: copy
+  - output: "test-recursive-dir"
+    template: dir-for-recusive-copying/**
+    template_type: copy

docs/level-15-copy-templates-as-target/README.rst

@@ -0,0 +1,82 @@
+Level 15: template copying becomes an action plugin in targets
+================================================================================
+
+With `.moban.yml`, you can copy templates to your destination. More information
+is documented in `misc-1-copying-template`. 
+
+Explicit syntax::
+
+    targets:
+       - output: explicit
+         template: template_file
+         template_type: copy
+
+
+Implicit syntax::
+
+    targets:
+       - output: explicit
+         template: template_file.copy
+
+
+Shorthand syntax::
+
+    targets:
+       - explicit: template_file.copy
+
+
+No implicit nor short hand syntax for the following directory copying unless
+you take a look at `force-template-type`. When you read
+`level-17-force-template-type-from-moban-file/README.rst`, you will find
+out more.
+
+
+Directory copying syntax::
+
+ 
+    targets:
+      - output: dest-dir
+        template: source-dir
+        template_type: copy
+   
+
+Recursive directory copying syntax::
+
+
+    targets:
+      - output: dest-dir
+        template: source-dir/**
+        template_type: copy
+
+
+Evaluation
+--------------------------------------------------------------------------------        
+
+Here is example moban file for copying::
+  
+    configuration:
+      template_dir:
+        - template-sources
+    targets:
+      - output: simple.file.copy
+        template: file-in-template-sources-folder.txt
+        template_type: copy
+      - output: target_without_template_type
+        template: file_extension_will_trigger.copy
+      - target_in_short_form: as_long_as_this_one_has.copy
+      - output: "misc-1-copying/can-create-folder/if-not-exists.txt"
+        template: file-in-template-sources-folder.txt
+        template_type: copy
+      - output: "test-dir"
+        template: dir-for-copying
+        template_type: copy
+      - output: "test-recursive-dir"
+        template: dir-for-recusive-copying/**
+        template_type: copy
+
+
+template copy does:
+
+
+#. copies any template inside pre-declared template directory to anywhere. moban will create directory if needed.
+#. copies any directory to anywhere. If "**" is followed, moban attempts to do recursive copying.

docs/level-15-copy-templates-as-target/misc-1-copying/can-create-folder/if-not-exists.txt

@@ -0,0 +1 @@
+test file

docs/level-15-copy-templates-as-target/template-sources/as_long_as_this_one_has.copy

@@ -0,0 +1 @@
+it is OK to have a short form, but the file to be 'copied' shall have 'copy' extension, so as to trigger ContentForwardEngine, 'copy' engine.

docs/level-15-copy-templates-as-target/template-sources/dir-for-copying/afile.txt

@@ -0,0 +1 @@
+dir for copying

docs/level-15-copy-templates-as-target/template-sources/dir-for-copying/sub_directory_is_not_copied/becuase_star_star_is_needed.txt

@@ -0,0 +1 @@
+Please look at .moban.yml

docs/level-15-copy-templates-as-target/template-sources/dir-for-recusive-copying/fileb.txt

@@ -0,0 +1 @@
+everything is copied

docs/level-15-copy-templates-as-target/template-sources/dir-for-recusive-copying/sub_directory_is_copied/because_star_star_is_specified.txt

@@ -0,0 +1 @@
+dest_directory: source_directory/**

docs/level-15-copy-templates-as-target/template-sources/file-in-template-sources-folder.txt

@@ -0,0 +1 @@
+test file

docs/level-15-copy-templates-as-target/template-sources/file_extension_will_trigger.copy

@@ -0,0 +1 @@
+file extension will trigger copy engine

docs/level-16-group-targets-using-template-type/.moban.yml

@@ -0,0 +1,9 @@
+configuration:
+  template_dir:
+    - template-sources
+targets:
+  - copy:
+    - simple.file.copy: file-in-template-sources-folder.txt
+    - "misc-1-copying/can-create-folder/if-not-exists.txt": file-in-template-sources-folder.txt
+    - "test-dir": dir-for-copying
+    - "test-recursive-dir": dir-for-recusive-copying/**

docs/level-16-group-targets-using-template-type/README.rst

@@ -0,0 +1,31 @@
+Level 16: group targets by their template type
+================================================================================
+
+Since moban version 0.4.0, you can group your targets with their template type.
+For example, with `copy` target, you can do the following things:
+
+
+Here is example moban file for copying::
+  
+    configuration:
+      template_dir:
+        - template-sources
+    targets:
+      - copy:
+        - simple.file.copy: file-in-template-sources-folder.txt
+        - "misc-1-copying/can-create-folder/if-not-exists.txt": file-in-template-sources-folder.txt
+        - "test-dir": dir-for-copying
+        - "test-recursive-dir": dir-for-recusive-copying/**
+
+More information is documented in `misc-1-copying-template`.
+
+
+template copy does:
+
+#. copies any template inside pre-declared template directory to anywhere. moban will create directory if needed.
+#. copies any directory to anywhere. If "**" is followed, moban attempts to do recursive copying.
+
+
+.. note::
+
+   The suffix `.copy` of `simple.file.copy` will be removed.

docs/level-16-group-targets-using-template-type/misc-1-copying/can-create-folder/if-not-exists.txt

@@ -0,0 +1 @@
+test file

docs/level-16-group-targets-using-template-type/template-sources/dir-for-copying/afile.txt

@@ -0,0 +1 @@
+dir for copying

docs/level-16-group-targets-using-template-type/template-sources/dir-for-copying/sub_directory_is_not_copied/becuase_star_star_is_needed.txt

@@ -0,0 +1 @@
+Please look at .moban.yml

docs/level-16-group-targets-using-template-type/template-sources/dir-for-recusive-copying/fileb.txt

@@ -0,0 +1 @@
+everything is copied

docs/level-16-group-targets-using-template-type/template-sources/dir-for-recusive-copying/sub_directory_is_copied/because_star_star_is_specified.txt

@@ -0,0 +1 @@
+dest_directory: source_directory/**

docs/level-16-group-targets-using-template-type/template-sources/file-in-template-sources-folder.txt

@@ -0,0 +1 @@
+test file

docs/level-17-force-template-type-from-moban-file/.moban.yml

@@ -0,0 +1,9 @@
+configuration:
+  template_dir:
+    - template-sources
+  force_template_type: copy
+targets:
+  - simple.file.copy: file-in-template-sources-folder.txt
+  - "misc-1-copying/can-create-folder/if-not-exists.txt": file-in-template-sources-folder.txt
+  - "test-dir": dir-for-copying
+  - "test-recursive-dir": dir-for-recusive-copying/**

docs/level-17-force-template-type-from-moban-file/README.rst

@@ -0,0 +1,26 @@
+Level 17: force template type
+================================================================================
+
+Since moban version 0.4.0, you can enforce all targets to use one and only one
+template type, regardless of their individual template types.
+
+
+Here is example moban file for copying::
+  
+    configuration:
+      template_dir:
+        - template-sources
+      force_template_type: copy
+    targets:
+      - simple.file.copy: file-in-template-sources-folder.txt
+      - "misc-1-copying/can-create-folder/if-not-exists.txt": file-in-template-sources-folder.txt
+      - "test-dir": dir-for-copying
+      - "test-recursive-dir": dir-for-recusive-copying/**
+
+More information is documented in `misc-1-copying-template`.
+
+
+template copy does:
+
+#. copies any template inside pre-declared template directory to anywhere. moban will create directory if needed.
+#. copies any directory to anywhere. If "**" is followed, moban attempts to do recursive copying.

docs/level-17-force-template-type-from-moban-file/misc-1-copying/can-create-folder/if-not-exists

@@ -0,0 +1 @@
+test file

docs/level-17-force-template-type-from-moban-file/misc-1-copying/can-create-folder/if-not-exists.txt

@@ -0,0 +1 @@
+test file

docs/level-17-force-template-type-from-moban-file/template-sources/dir-for-copying/afile.txt

@@ -0,0 +1 @@
+dir for copying

docs/level-17-force-template-type-from-moban-file/template-sources/dir-for-copying/sub_directory_is_not_copied/becuase_star_star_is_needed.txt

@@ -0,0 +1 @@
+Please look at .moban.yml

docs/level-17-force-template-type-from-moban-file/template-sources/dir-for-recusive-copying/fileb.txt

@@ -0,0 +1 @@
+everything is copied

docs/level-17-force-template-type-from-moban-file/template-sources/dir-for-recusive-copying/sub_directory_is_copied/because_star_star_is_specified.txt

@@ -0,0 +1 @@
+dest_directory: source_directory/**

docs/level-17-force-template-type-from-moban-file/template-sources/file-in-template-sources-folder.txt

@@ -0,0 +1 @@
+test file

docs/misc-1-copying-templates/README.rst

@@ -1,7 +1,12 @@
-Misc 1: copying templates
+Misc 1: copying templates - Deprecated since version 0.4.0
 ================================================================================
 
-With `.moban.yml`, you can copy templates to your destination.
+.. warning::
+   This chapter is kept for regression testing. If you have moban v0.4.0 or
+   above, please do not use the syntax here
+
+With `.moban.yml`, you can copy templates to your destination. Please be
+aware that it is not the same as 'cp', 'copy' commands you have experienced.
 
 
 Please be aware that, your templates and template folder have to be inside
@@ -23,7 +28,5 @@ Here is example moban file for copying::
 template copy does:
 
 
-#. copies any template inside pre-declared template directory to anywhere. moban will
-  create directory if needed.
-#. copies any directory to anywhere. If "**" is followed, moban attempts to do
-   recursive copying.
+#. copies any template inside pre-declared template directory to anywhere. moban will create directory if needed.
+#. copies any directory to anywhere. If "**" is followed, moban attempts to do recursive copying.