Fix any other regressions (#313)

* 🔥 remove appdirs and gitpython from moban dependency list

* 📚 add a use case: template from a s3 bucket

* ✨ add -d for development tracing. fix #185

* 🚜 do not log the exception but as info

* 📚 update documenation

* 🐛 moban -t url -c url -o moban.output failes

* 🔬 more tests

* 👕 fix format

* 📚 update change log and readme

* 🔥 remove unused imports

* 🔬 more tests

* 👕 fix coding style

* 📚 update package url link

* 💚 make travis tests pass

* 💚 make windows tests pass. an osfs was passed to multifs and the invalid characters on windows are different from the ones on linx. hence pypifs and gitfs2 url are seen as invalid on windows. PyFilesystem/pyfilesystem2#347

* 🔥 remove adapter folder

* 🐛 fix build failure on opensuse. resolves #275

* ✨ use released version of pypifs

* 📰 add pypifs as test dependency

.moban.cd/changelog.yml

@@ -5,6 +5,8 @@ releases:
       - action: Updated
         details:
           - "`#205`: support `pyFilesystem2 <https://pyfilesystem2.readthedocs.io/>`_"
+          - "`#185`: -d will enable moban application logging for development"
+          - "`#275`: fix moban 0.4.5 test failures on openSUSE Tumbleweed"
     date: tba
     version: 0.6.0
   - changes:

.moban.cd/moban.yml

@@ -26,8 +26,6 @@ dependencies:
   - lml>=0.0.9
   - appdirs>=1.4.3
   - crayons>= 0.1.0
-  - GitPython>=2.0.0
-  - git-url-parse>=1.2.2
   - fs>=2.4.6
   - jinja2-fsloader>=0.2.0
 description: Yet another jinja2 cli command for static text generation

.travis.yml

@@ -15,9 +15,9 @@ env:
   - MINREQ=0
 
 stages:
-  - test
   - lint
   - moban
+  - test
 
 .disable_global: &disable_global
   addons: false

CHANGELOG.rst

@@ -9,6 +9,10 @@ Updated
 
 #. `#205 <https://github.com/moremoban/moban/issues/205>`_: support
    `pyFilesystem2 <https://pyfilesystem2.readthedocs.io/>`_
+#. `#185 <https://github.com/moremoban/moban/issues/185>`_: -d will enable moban
+   application logging for development
+#. `#275 <https://github.com/moremoban/moban/issues/275>`_: fix moban 0.4.5 test
+   failures on openSUSE Tumbleweed
 
 0.5.0 - 14.07.2019
 --------------------------------------------------------------------------------

README.rst

@@ -35,6 +35,11 @@ any location: zip, git, pypi package, s3, etc. Please
 look at our issues. We have many more template engines and data format on the
 road map.
 
+Documentation
+=================================================================================
+
+All use cases are documented `here <http://moban.readthedocs.org/en/latest/#tutorial>`_
+
 Installation
 ================================================================================
 You can install it via pip:
@@ -95,9 +100,73 @@ moban.output will contain::
 
 Please note that data.yml will take precedence over environment variables.
 
-`the tutorial`_ has more use cases.
 
-.. _the tutorial: http://moban.readthedocs.org/en/latest/#tutorial
+Work with files in a git repo
+================================================================================
+
+Please install `gitfs2 <https://github.com/moremoban/gitfs2>`_::
+
+    $ pip install gitfs2
+
+
+And then you can do the following:
+
+.. code-block:: bash
+
+    $ moban -t 'git://github.com/moremoban/pypi-mobans.git!/templates/_version.py.jj2' \
+            -c 'git://github.com/moremoban/pypi-mobans.git!/config/data.yml' \
+            -o _version.py
+    Info: Found repo in /Users/jaska/Library/Caches/gitfs2/repos/pypi-mobans
+    Templating git://github.com/moremoban/pypi-mobans.git!/templates/_version.py.jj2 to _version.py
+    Templated 1 file.
+    $ cat _version.py
+    __version__ = "0.1.1rc3"
+    __author__ = "C.W."
+
+
+Work with files in a python package
+================================================================================
+
+Please install `pypifs <https://github.com/moremoban/pypifs>`_::
+
+    $ pip install pypifs
+
+
+And then you can do the following:
+
+.. code-block:: bash
+
+    $ moban -t 'pypi://pypi-mobans-pkg/resources/templates/_version.py.jj2' \
+            -c 'pypi://pypi-mobans-pkg/resources/config/data.yml' \
+            -o _version.py
+    Collecting pypi-mobans-pkg
+    ....
+    Installing collected packages: pypi-mobans-pkg
+    Successfully installed pypi-mobans-pkg-0.0.7
+    Templating pypi://pypi-mobans-pkg/resources/templates/_version.py.jj2 to _version.py
+    Templated 1 file.
+    $ cat _version.py
+    __version__ = "0.1.1rc3"
+    __author__ = "C.W."
+
+Work with S3 and other cloud based file systems
+================================================================================
+
+Please install `fs-s3fs <https://github.com/PyFilesystem/s3fs>`_::
+
+    $ pip install fs-s3fs
+
+.. code-block:: bash
+
+    $ moban -c s3://${client_id}:${client_secrect}@moremoban/s3data.yml \
+            -o 'zip://my.zip!/moban.output' {{hello}}
+    $ unzip my.zip
+    $ cat moban.output
+    world
+
+Where the configuration sits in a s3 bucket, the output is a file in a zip. The content of s3data.yaml is::
+
+    hello: world
 
 	
 Usage
@@ -151,58 +220,3 @@ With `--exit-code`:
 - 0 : no changes
 - 1 : has changes
 - 2 : error occured
-
-Built-in Filters
-================================================================================
-
-split_length
---------------------------------------------------------------------------------
-
-It breaks down the given string into a fixed length paragraph. Here is the syntax::
-
-    {% for line in your_string | split_length(your_line_with) %}
-    {{line}}
-    {% endfor %}
-
-It is used to keep changelog formatted in
-`CHANGELOG.rst.jj2 in pypi-mobans project <https://github.com/moremoban/pypi-mobans/blob/master/templates/CHANGELOG.rst.jj2#L15>`_
-
-github_expand
---------------------------------------------------------------------------------
-
-It expands simple hashtags into github issues. Here is the syntax::
-
-    {{ your_github_string | github_expand }}
-
-
-It makes it easy to mention github reference in change log in all projects. Here is
-the place it is applied:
-`CHANGELOG.rst.jj2 in pypi-mobans project <https://github.com/moremoban/pypi-mobans/blob/master/templates/CHANGELOG.rst.jj2#L15>`_
-
-
-Here is Grammar in the changelog.yml::
-
-    =============== ==============================
-    Syntax          Meaning
-    =============== ==============================
-    `#1`            moban issues 1
-    `PR#1`          moban pull request 1
-    `pyexcel#1`     other project issues 1
-    `pyexcel#PR#1`  other project pulll request 1
-    =============== ==============================
-
-More details can be found in `moban's changelog.yml <https://github.com/moremoban/moban/blob/master/.moban.cd/changelog.yml#L10>`_
-
-`repr`
---------------------------------------------------------------------------------
-
-Returns a single quoted string in the templated file
-
-
-Built-in Tests
-================================================================================
-
-`exists`
---------------------------------------------------------------------------------
-
-Test if a file exists or not

docs/extension.rst

@@ -16,12 +16,64 @@ Jinja2 Filter
 
 .. literalinclude:: ../moban/filters/repr.py
 
+split_length
+--------------------------------------------------------------------------------
+
+It breaks down the given string into a fixed length paragraph. Here is the syntax::
+
+    {% for line in your_string | split_length(your_line_with) %}
+    {{line}}
+    {% endfor %}
+
+It is used to keep changelog formatted in
+`CHANGELOG.rst.jj2 in pypi-mobans project <https://github.com/moremoban/pypi-mobans/blob/master/templates/CHANGELOG.rst.jj2#L15>`_
+
+github_expand
+--------------------------------------------------------------------------------
+
+It expands simple hashtags into github issues. Here is the syntax::
+
+    {{ your_github_string | github_expand }}
+
+
+It makes it easy to mention github reference in change log in all projects. Here is
+the place it is applied:
+`CHANGELOG.rst.jj2 in pypi-mobans project <https://github.com/moremoban/pypi-mobans/blob/master/templates/CHANGELOG.rst.jj2#L15>`_
+
+
+Here is Grammar in the changelog.yml::
+
+    =============== ==============================
+    Syntax          Meaning
+    =============== ==============================
+    `#1`            moban issues 1
+    `PR#1`          moban pull request 1
+    `pyexcel#1`     other project issues 1
+    `pyexcel#PR#1`  other project pulll request 1
+    =============== ==============================
+
+More details can be found in `moban's changelog.yml <https://github.com/moremoban/moban/blob/master/.moban.cd/changelog.yml#L10>`_
+
+`repr`
+--------------------------------------------------------------------------------
+
+Returns a single quoted string in the templated file
+
+
+Built-in Tests
+================================================================================
 
 Jinja2 Test
 *******************
 
 .. literalinclude:: ../moban/tests/files.py
 
+
+`exists`
+--------------------------------------------------------------------------------
+
+Test if a file exists or not
+
 Jinja2 Globals
 *******************
 

min_requirements.txt

@@ -6,7 +6,5 @@ jinja2==2.7.1
 lml==0.0.9
 appdirs==1.4.3
 crayons== 0.1.0
-GitPython==2.0.0
-git-url-parse==1.2.2
 fs==2.4.6
 jinja2-fsloader==0.2.0

moban/constants.py

@@ -70,6 +70,7 @@
 LABEL_DEST = "destination"
 LABEL_FORCE_TEMPLATE_TYPE = "force_template_type"
 LABEL_TEMPLATE_TYPES = "template_types"
+LABEL_DEBUG = "debug"
 
 # error messages
 ERROR_DATA_FILE_NOT_FOUND = "Both %s and %s does not exist"

moban/file_system.py

@@ -40,7 +40,7 @@ def wrapper(*args, **kwds):
             message = "Failed to open %s" % args[0]
             LOG.debug(message)
             reporter.report_error_message(message)
-            raise
+            raise exceptions.FileNotFound(args[0])
 
     return wrapper
 

moban/jinja2/engine.py

@@ -2,7 +2,7 @@
 import logging
 from importlib import import_module
 
-from moban import constants, exceptions
+from moban import constants, file_system
 from jinja2 import Template, Environment
 from lml.loader import scan_plugins_regex
 from lml.plugin import PluginInfo, PluginManager
@@ -79,6 +79,7 @@ def __init__(self, template_fs, options=None):
                 extension for extension in JINJA2_THIRD_PARTY_EXTENSIONS
             ],  # get a copy of this global variable
         )
+        self.template_loader = template_loader
         if options:
             if "extensions" in options:
                 extensions = options.pop("extensions")
@@ -117,7 +118,8 @@ def get_template(self, template_file):
         try:
             template = self.jj2_environment.get_template(template_file)
         except TemplateNotFound:
-            raise exceptions.FileNotFound("%s does not exist" % template_file)
+            content = file_system.read_unicode(template_file)
+            return Template(content)
         return template
 
     def get_template_from_string(self, string):

moban/main.py

@@ -11,6 +11,7 @@
 import sys
 import logging
 import argparse
+import logging.config
 
 from moban import plugins, reporter, constants, mobanfile, exceptions
 from moban.utils import merge
@@ -28,6 +29,12 @@ def main():
     parser = create_parser()
     options = vars(parser.parse_args())
     HASH_STORE.IGNORE_CACHE_FILE = options[constants.LABEL_FORCE]
+    if options[constants.LABEL_DEBUG]:
+        logging.basicConfig(
+            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
+            level=logging.DEBUG,
+        )
+
     moban_file = options[constants.LABEL_MOBANFILE]
     load_engine_factory_and_engines()  # Error: jinja2 if removed
     if moban_file is None:
@@ -126,6 +133,13 @@ def create_parser():
         action="version",
         version="%(prog)s {v}".format(v=__version__),
     )
+    parser.add_argument(
+        "-d",
+        action="store_true",
+        dest=constants.LABEL_DEBUG,
+        default=False,
+        help="to show debug trace",
+    )
     return parser
 
 

moban/mobanfile/templates.py

@@ -7,6 +7,7 @@
 
 def handle_template(template_file, output, template_dirs):
     log.info("handling %s" % template_file)
+
     template_file = file_system.to_unicode(template_file)
     multi_fs = file_system.get_multi_fs(template_dirs)
     if template_file.endswith("**"):

moban/plugins/template.py

@@ -3,6 +3,7 @@
 import logging
 
 from moban import utils, reporter, constants, exceptions, file_system
+from fs.errors import ResourceNotFound
 from lml.plugin import PluginManager
 from moban.hashstore import HASH_STORE
 from moban.deprecated import repo, deprecated
@@ -88,9 +89,12 @@ def render_to_file(self, template_file, data_file, output_file):
         template_file = file_system.to_unicode(template_file)
         data = self.context.get_data(data_file)
         template = self.engine.get_template(template_file)
-        template_abs_path = self.template_fs.geturl(
-            template_file, purpose="fs"
-        )
+        try:
+            template_abs_path = self.template_fs.geturl(
+                template_file, purpose="fs"
+            )
+        except ResourceNotFound:
+            template_abs_path = template_file
 
         flag = self.apply_template(
             template_abs_path, template, data, output_file
@@ -135,8 +139,9 @@ def apply_template(self, template_abs_path, template, data, output_file):
                         template_abs_path, output_file
                     )
             return flag
-        except exceptions.FileNotFound as e:
-            log.exception(e)
+        except exceptions.FileNotFound:
+            # the template is a string from command line
+            log.info("{} is not a file".format(template_abs_path))
             self.buffered_writer.write_file_out(output_file, rendered_content)
             return True
 

requirements.txt

@@ -6,7 +6,5 @@ jinja2>=2.7.1
 lml>=0.0.9
 appdirs>=1.4.3
 crayons>= 0.1.0
-GitPython>=2.0.0
-git-url-parse>=1.2.2
 fs>=2.4.6
 jinja2-fsloader>=0.2.0

rnd_requirements.txt

@@ -1,6 +1,5 @@
 https://github.com/moremoban/pyfilesystem2/archive/master.zip
 https://github.com/moremoban/moban-handlebars/archive/dev.zip
-https://github.com/moremoban/pypifs/archive/master.zip
 https://github.com/moremoban/gitfs2/archive/master.zip