Changing the Repository Structure

[tajmone]

EDIT: See Issue #148 for WIP status.

@AnssiR66, I suggest that before we merge the final release into master branch we revise the repository folders structure.

While working on the new wearer attribute I realized that after this feature is finally merged into the baseline dev branch we should reorganize the whole repository into a source folder and a distribution folder, where the former contains all the sources (AsciiDoc, raw examples and transcripts) and the latter the built and sanitized versions only.

This change in structure (i.e. a source and a distribution folder) will actually make things easier in terms of the toolchain, since these two folders will be specular in structure, one containing the source files, the other the converted and sanitized output, both having the same directory structure. It would be a more intuitive layout to work with, compared to the current structure (which doesn't provide for library sources sanitation, which is why it doesn't have a symmetric source/build layout). The new structure will probably look something like this:

├─ /_assets/     (toolchain assets, as before)
├─ /_assets_src/ (toolchain assets, as before)
├─ /lib_distro/
│    ├─ /extras/
│    │    ├─ /manual/
│    │    │    ├─ *.html (converted output)
│    │    │    └─ *.alan (sanitized sources)
│    │    └─ /tutorials/
│    │         ├─ *.html (converted output)
│    │         └─ *.alan (sanitized sources)
│    └─ /StdLib/
│         └─ *.i (sanitized sources)
├─ /lib_source/
│    ├─ /extras/
│    │    ├─ /manual/
│    │    │    ├─ *.asciidoc/*.adoc (ADoc sources)
│    │    │    └─ *.alan (raw sources with region tags and dev notes)
│    │    └─ /tutorials/
│    │         ├─ *.asciidoc/*.adoc (ADoc sources)
│    │         └─ *.alan (raw sources with region tags and dev notes)
│    └─ /StdLib/
│         └─ *.i (raw sources with region tags and dev notes)
└─ /tests/ (testsuite, as before)

The reason why this has suddenly become a compelling issue (although it has been on my mind for a long time) is because the recent work on the documentation is introducing various AsciiDoc region-comments into the library sources, which we might wish to strip away from the distribution files — they don't look nice, and make the code less readable for end users who wish to study it, but they are really useful to us in maintaining the documentation always aligned with the library sources.

I was also thinking that developers' annotations should use a special comment notation (e.g. comments starting in --!) so that they are also stripped away from the distribution library sources, since these are comments for the benefit of maintainers only (often mentioning bugs, or things to be done in the future, none of which are beneficial to end users).

Tracking Generated Contents?

I also believe that we should not track the distribution folder, since it's always built locally during development, and it would just consume space in the repository. But this choice largely depends on whether we wish to keep an online accessible version of the development-stage library files and HTML docs, or whether we should only focus on providing them via releases, via downloadable Zip archives that we attach to each release.

In practical terms, if we diced to ignore the whole distribution/build folder, it will mean that the repository will no longer track generated files like the HTML documents. We'll keep tracking tests transcripts, though, since we need to track their changes in order to see how changes to code affect the output (and catch any undesired side effects), but the tests folder is not part of the source/distribution structure, it's a separate folder.

If we end up ignoring the distribution folder, it will mean that the repository will no longer track any HTML file, so in order to visualize the documentation it becomes necessary to build the project locally — which now, thanks to the later adoption of Ruby Rake, has become a very simple operation, only requiring end users to type rake in the CDM/Shell. As for the accessing the latest (WIP) dev files of the library, only their unsanitized version would be available from the repository (e.g. by clicking on the "download" button on GitHub, after choosing a dev branch).

Personally, I think that once we've made our first public release of the new library (i.e. the new AsciiDoc toolchain, etc.), we probably won't have to worry about access to the HTML files — whereas right now their developer versions are all end users can rely on to study the new features. Chances that users will be in need to the development branch files are quite slim IMO, and in any case the new system is really easy to build locally: users don't need to understand anything about Ruby or Rake, they only need to install Ruby and Asciidoctor, and just type rake in the CMD, the rest is taken care of auto-magically.

So, ultimately it boils down to how us maintainers wish to go about it, since it would also mean that in order to peak at the latest dev-docs we'd need to build the whole project locally. Surely, having a direct Live HTML link to access the state of any document (from any branch) is more practical, especially when a maintainer is not tracking the latest developments (as it's often the case in many ALAN projects, for whatever reasons, where often the various contributors and maintainers work at alternate periods on the project).

The problem of tracking built files is mostly because of size bloat to the repository, but also because these files can introduce conflicts in rebase operations, etc. — in version controlled projects the general rule is to ignore any build artefacts, but there are exceptions to this rule (e.g. the transcripts in our tests/ folder, which are useful for detecting changes).

E.g. we don't strictly need to track generated transcripts used in the documents toolchain either, but at least tracking them in the library source folder will provide us some insights in their changes, which might be alarm bells to the fact that the original text might need to be looked into, when the detected changes are considerable.

Another consideration worth considering here is that the lib_distro/ folder would have a twofold function:

1. Checking at all times the integrity of the distro files.
2. Building the package Zip files to attach to every release.

As for the second point, we might want to do carry out this step manually (create the Zip files locally and upload them with each release), or we could do it automatically (via Travis CI or GitHub Actions) whenever a tagged release is created. Furthermore, we might consider creating two different distribution packages:

• Windows:
  • Use CRLF line endings in ALAN sources.
  • Provide .bat scripts (to build the examples).
• Linux/macOS —
  • Use LF line endings in ALAN sources.
  • Provide .sh scripts (to build the examples).

This last argument is also a reason why I don't think that storing the distribution files in the repository is particularly beneficial (aside from the HTML files), since any ALAN sources downloaded from GitHub will always have LF line endings, since GH uses Linux servers.

In any case, we can decide to track some files and omit others, or just track everything, according to whichever way seems more beneficial to us all.

We don't have to decide right away on this aspect of what to track or ignore — even if I change the repository structure in the coming week, we still have time until the library release to decide on what to ignore and what to track.

[AnssiR66]

All of that sounds good, and I agree with this proposed plan of action.

[tajmone]

New Structure in Practice

@AnssiR66, I've now been working on the new repo structure for a few days and I think that it's now ready to be merged into the main dev branch (which, BTW, was renamed from dev-2.2.0 to just dev, for reason that will be explained here).

I'm also buzzing @thoni56, since I'd like to probe if he has some tips on this, and to keep him updated on the upcoming library changes too.

While implementing the new structure I've had a chance to test various solutions, and ultimately ended up changing the directory tree structure initially proposed, as well as some of the original folder names, because practical usage lead me to re-evaluate the folders tree in more practical terms.

Definitely, having the library source and distribution folders share a symmetrical structure is a great improvement in many respects, and most important it's easier to manage and work with.

New Repo Structure

This is the actual new repository structure now:

├─ /_assets/     (toolchain assets, as before)
├─ /_assets_src/ (toolchain assets, as before)
├─ /lib_distro/
│    ├─ /docs/
│    │    ├─ /LibManual/
│    │    │    ├─ *.html (converted output)
│    │    │    └─ *.alan (sanitized sources)
│    │    └─ /ClothingGuide/
│    │         ├─ *.html (converted output)
│    │         └─ *.alan (sanitized sources)
│    └─ /StdLib/
│         └─ *.i (sanitized sources)
├─ /lib_source/
│    ├─ /docs/
│    │    ├─ /LibManual/
│    │    │    ├─ *.asciidoc/*.adoc (ADoc sources)
│    │    │    └─ *.alan (raw sources with region tags and dev notes)
│    │    └─ /ClothingGuide/
│    │         ├─ *.asciidoc/*.adoc (ADoc sources)
│    │         └─ *.alan (raw sources with region tags and dev notes)
│    └─ /StdLib/
│         └─ *.i (raw sources with region tags and dev notes)
├─ /lib-tests/ (testsuite, as before)
└─ /tbd/       (unprocessed library assets)

As you can see, there have been slight changes in the naming convention, compared to original proposal. These changes were motivated by how the package might develop in the future.

New Package Structure

So, here's the explanation of the new structure of the distribution package:

• <DistroFolder> — the actual name of the downloaded folder is subject to change, depending on whether it's downloaded via the WebUI, via the releases page, or if it's a Zip that we create for ueser.
  • /extras/ (TBD! coming soon) — adventure template, useful scripts, and other non-documentation assets.
  • /demos/ (TBD! coming soon) — sample adventure(s).
  • /docs/ — all library documentation will go in this folder (except the CHANGELOG and package document). The rule of thumb is that documents that are standalone and don't have examples that go with them should be in the root of docs/, whereas any multi-file docs or docs with examples will be stored in their own folder:
    • /ClothingGuide/ — guide + examples. Before it was inside tutorials/, but I reasoned that once the number of tutorials grows the folder would become cluttered by all the examples, making it hard to track which example belongs to which tutorial.
    • /LibManual/ — Manual + examples
    • QuickRef.html (TBD! coming soon)
    • QuickStart.html (TBD! coming soon)
  • /StdLib/ — library files (sanitized) + COPYING license.
  • CHANGELOG.html — I've moved the CHANGELOG to the root of the package, since it's not only about the library files but also deals with documentation, etc., so it's a "global" document.
  • README.html (TBD! coming soon) — Will act as a TOC, providing the package structure and links to its contents, along with a brief presentation of the current release, plus useful links, etc.

I think that the new folders structure makes sense both in term of the actual contents as well as in view of contents that will be added in the future.

Also, since now all documents are in HTML format, dynamic links between documents and to source files provide users with a quick and practical navigation system within the package. Since every PC has a browser, HTML is a good choice for documenting the package.

I think it's a good idea to try and preserve a consistent structure between new versions of the library, or at least keep it familiar, which is why I opted to create the docs/ subfolder and rename tutorials/ to ClothingGuide/, making it specific to that particular tutorial.

Organizing different types of contents into separate folders allows users to copy and paste assets they might want to reuse (e.g. the StdLib/ folder for their new adventure, or the template from extras/) without dragging along unneeded assets (e.g. no need to past the whole manual into an adventure working dir).

Of course, we can still change the folders structure or their names before release v2.2.0 — what matters is that now we have an organized symmetrical repository.

I'd like your opinion on the above choices, and if you think there are better ways to organize the various assets, and if there are some assets types I've overlooked, etc.

In any case, I'm ready to merge the new structure into dev/, I just want to take one more day to go over all the READMEs and see if there's some pending text polishes. As I said, there's still room for name changes even after merging. It's just that this is a rather big changes-set, so until it's merged into the dev baseline it would prevent working on other things (e.g. the manual).

Tracking Dilemma: Let's Track!

After thinking about the problem of whether to track in the repository the generated files or not, I've come to the conclusion that the benefits of tracking them outweigh the cons in this specific project. Here's why:

1. Not all files in the distro folder are generated from source, e.g. the license file and a few other assets (game template, etc.).

   Excluding the lib_distro/ from the repository (Git-ignoring it) would require us to copy such files from lib_source/ into lib_distro/ at build time, which is rather impractical — I've actually tried this approach during the first days, but I really disliked it.

2. Preserving the converted HTML docs allows anyone to view the latest version of any project document, including from dev branches.

   This is very practical for all ALAN developers who maintain other projects, since they might need to quickly peeks at the latest docs without having to clone the repository and build everything themselves.

   And it's also important for end users, since it allows them to access the latest version of the library documentation — after all, just like with the ALAN Manual, the StdLib Manual can be updated even within a same-library version, not just on dev branches, since documentation can always be improved upon, even within a same library version.

3. With the new system, and all files in lib_distro/ being tracked by Git, it's very easy for end users to download the library via the repository WebUI (Code > Download Zip menu), from any branch.

   I've actually tested this on the dev branch and works really great. So, users will always be able to download the latest dev snapshot of the library (code, docs and everything else), or even a specific dev branch for a WIP feature — which is really practical when we need beta-testers to provide us feedback on some WIP feature before we merge it, or if we have two different feature-branches for alternative implementation and need feedback on which one is better, before choosing which one to keep.

Having tried working with both approaches locally (ignore vs track), I quickly realized that tracking the distribution files is more practical. For example, now HTML documents are converted directly into their lib_distro/ mirror folder, whereas before they were first converted in their lib_source/ source folder (ignored by Git) and then copied to their distro folder. Although all of this was done automatically by Rake, it still felt redundant and lead to cluttered folders.

The new system is much cleaner and simpler:

• Every library file should naturally reside in lib_distro/, unless it needs to be build from source, in which case its source(s) will reside in lib_source/, but the final output will still be present in lib_distro/.
• Since end users are only interested in the contents of lib_distro/, they don't have to worry about any of this, since all required files will always be present in that folder.
• For us maintainers, on the other hand, the symmetrical structure works best since we have clearer expectations regarding certain file formats being build from sources in another format (e.g. HTML from AsciiDoc), and the clear division between the package sources and distro folders is tidier to work with, and the fact that files in lib_distro/ are tracked by Git spares us having to mentally keep track of what's actually in the repository and what's ignored — if we see a file in there, then it's there for end users too; i.e. WWSIWUG (What We See Is What Users Get), to paraphrase the famous WYSIWYG approach.

[AnssiR66]

Excellent work Tristano, and I agree with the choices you have made. I can't see any reason why we could not proceed the way you have laid out. It makes sense to me and feels clear and logical enough. If there is something I can do, let me know.

[tajmone]

I agree with the choices you have made.

Great!

Looking Into Unprocessed Library Assets

If there is something I can do, let me know.

After Now that I've squashed the new repo structure we should probably look at the unprocessed library assets which I've moved to the (new) tbd/ folder.

Some of them need minor updates, like:

• newgame.alan — only needs minor tweak and then be moved into extras/.
• QuickRef.txt — convert to AsciiDoc and update contents.
• QuickStart.docx/QuickStart.txt — convert to AsciiDoc and update contents.

whereas other assets are extremely outdated, to the point it might not be worth keeping them:

• readme.txt — we should probably simply create a new README, maybe reusing some text from here.
• mygame_import.i — the code is too outdated compared to the library changes, and I'm also not so sure it's even needed now that the library has been split into smaller modules (users can just copy and tweak the lib_messages.i module).

Finally, this test game probably needs updating the code, but I'm not sure whether it's worth the effort or not:

• testgame.alan — seems like a small demo.

If you could have a look at the problematic assets mentioned above, and share some thoughts on whether it's worth updating and keeping them, or if you think we should just create a new version from scratch using the new library, or whatever you suggest it's best to do.

Bear in mind that I've not used some of these assets in the past, so I'm not sure about what the intention behind their creation was, and since some might no longer compile with the current library your thoughts would be precious for me on this.

[AnssiR66]

We can certainly discard testgame.alan and mygame_import.i which by now are outdated. The testgame is rather messy, and something else could be created instead. The readme file can be rewritten, it shouldn't be too great a job to do.

[tajmone]

We can certainly discard testgame.alan and mygame_import.i which by now are outdated.

I agree, even though I have no insight on the testgame.alan file — but since it's not intuitive by a simple glance, I guess it means it's now out of synch with the new state of the package.

The testgame is rather messy, and something else could be created instead.

Would that be a real test or more of a sample adventure showing users the various library features and how to customize things? In term of tests, we have an entire new test suite, so our back is covered in that respect. As for a demo/sample adventure which is not just a small example (like those in the Library Manual files) but a full adventure written with the purpose of providing an example for end users to study (so, with lot's of comments in the source) I think it's a great idea and also something which is needed.

It could also be a good excuse for me to actually write a full adventure using the StdLib, and I'd like it even more if it become a "four hands work", i.e. we work together on it. Since this demo will have to updated when the library is updated (in case of breaking changes or new features), it also means that it can grow in time and doesn't have to be 100% complete for the next release. It's probably going to be the sort of free-form adventure where new stuff is added across updates, for the sake of demonstrating features, so the actual story and plot might have to be bent in favor of more practical needs.

The readme file can be rewritten, it shouldn't be too great a job to do.

Definitely. Once all the assets have found their final collocation in the package, and we're sure that files won't be renamed, created a README file is going to take little work.