Store grammar git version within license file when caching (github-li…

…nguist#4268)

* Add version when caching licenses

* Add approved grammar licenses to config.yml

* Add version and update cache of all licenses

All licenses should remain the same except where there is clearly a change in license

* Update release process with license caching deets

* Double check licensed cached before checking status

* Only need to explicitly approve grammars without license files

* Revert "Double check licensed cached before checking status"

This reverts commit c86cfd1.

This isn't needed now we're versioning out license files

* Change into submodule dir before getting SHA

* Update Licensed min version requirement

* Update license formats from Licensed 1.3.3 processing

* Remove tests covered by "licensed status" test

* 🔥 more redundant tests and associated methods

* Update hy.tmLanguage cached license

CONTRIBUTING.md

@@ -40,7 +40,7 @@ To add support for a new extension:
 
 1. Add your extension to the language entry in [`languages.yml`][languages], keeping the extensions in alphabetical and case-sensitive (uppercase before lowercase) order, with the exception of the primary extension; the primary extension should be first.
 1. Add at least one sample for your extension to the [samples directory][samples] in the correct subdirectory. We'd prefer examples of real-world code showing common usage. The more representative of the structure of the language, the better.
-1. Open a pull request, linking to a [GitHub search result](https://github.com/search?utf8=%E2%9C%93&q=extension%3Aboot+NOT+nothack&type=Code&ref=searchresults) showing in-the-wild usage.  
+1. Open a pull request, linking to a [GitHub search result](https://github.com/search?utf8=%E2%9C%93&q=extension%3Aboot+NOT+nothack&type=Code&ref=searchresults) showing in-the-wild usage.
   If you are adding a sample, please state clearly the license covering the code in the sample, and if possible, link to the original source of the sample.
 
 Additionally, if this extension is already listed in [`languages.yml`][languages] and associated with another language, then sometimes a few more steps will need to be taken:
@@ -57,12 +57,12 @@ We try only to add languages once they have some usage on GitHub. In most cases
 To add support for a new language:
 
 1. Add an entry for your language to [`languages.yml`][languages]. Omit the `language_id` field for now.
-1. Add a syntax-highlighting grammar for your language using: `script/add-grammar https://github.com/JaneSmith/MyGrammar`  
-  This command will analyze the grammar and, if no problems are found, add it to the repository. If problems are found, please report them to the grammar maintainer as you will not be able to add the grammar if problems are found.  
+1. Add a syntax-highlighting grammar for your language using: `script/add-grammar https://github.com/JaneSmith/MyGrammar`
+  This command will analyze the grammar and, if no problems are found, add it to the repository. If problems are found, please report them to the grammar maintainer as you will not be able to add the grammar if problems are found.
   **Please only add grammars that have [one of these licenses][licenses].**
 1. Add samples for your language to the [samples directory][samples] in the correct subdirectory.
-1. Generate a unique ID for your language by running `script/update-ids`.  
-1. Open a pull request, linking to a [GitHub search results](https://github.com/search?utf8=%E2%9C%93&q=extension%3Aboot+NOT+nothack&type=Code&ref=searchresults) showing in-the-wild usage.  
+1. Generate a unique ID for your language by running `script/update-ids`.
+1. Open a pull request, linking to a [GitHub search results](https://github.com/search?utf8=%E2%9C%93&q=extension%3Aboot+NOT+nothack&type=Code&ref=searchresults) showing in-the-wild usage.
   Please state clearly the license covering the code in the samples. Link directly to the original source if possible.
 
 In addition, if your new language defines an extension that's already listed in [`languages.yml`][languages] (such as `.foo`) then sometimes a few more steps will need to be taken:
@@ -100,7 +100,7 @@ Switching the source of a grammar is really easy:
 
     script/add-grammar --replace MyGrammar https://github.com/PeterPan/MyGrammar
 
-This command will analyze the grammar and, if no problems are found, add it to the repository. If problems are found, please report these problems to the grammar maintainer as you will not be able to add the grammar if problems are found.  
+This command will analyze the grammar and, if no problems are found, add it to the repository. If problems are found, please report these problems to the grammar maintainer as you will not be able to add the grammar if problems are found.
 
 **Please only add grammars that have [one of these licenses][licenses].**
 
@@ -142,7 +142,11 @@ If you are the current maintainer of this gem:
 
 1. Create a branch for the release: `git checkout -b release-vxx.xx.xx`
 1. Make sure your local dependencies are up to date: `script/bootstrap`
-1. If grammar submodules have not been updated recently, update them: `git submodule update --remote && git commit -a`
+1. If grammar submodules have not been updated recently, update them: `git submodule update --remote`. If any submodules are updated,
+    1. update the license cache: `script/licensed`
+    1. double check no problems found: `script/licensed status`
+    1. verify and fix any problems identified
+    1. commit all changes: `git commit -a`
 1. Ensure that samples are updated: `bundle exec rake samples`
 1. Ensure that tests are green: `bundle exec rake test`
 1. Build a test gem `GEM_VERSION=$(git describe --tags 2>/dev/null | sed 's/-/./g' | sed 's/v//') bundle exec rake build_gem`

github-linguist.gemspec

@@ -27,6 +27,6 @@ Gem::Specification.new do |s|
   s.add_development_dependency 'rake'
   s.add_development_dependency 'yajl-ruby'
   s.add_development_dependency 'color-proximity', '~> 0.2.1'
-  s.add_development_dependency 'licensed', '~> 1.1.0'
+  s.add_development_dependency 'licensed', '~> 1.3.0'
   s.add_development_dependency 'licensee'
 end

script/licensed

@@ -1,7 +1,5 @@
 #!/usr/bin/env ruby
 
-# TODO: push these changes to licensor gem
-
 require "bundler/setup"
 require "licensed/cli"
 require "optparse"
@@ -22,12 +20,19 @@ module Licensed
         !Dir.glob(@glob).empty?
       end
 
+      def submodule_version(directory)
+        return unless directory
+        Dir.chdir directory do
+          Licensed::Git.version(".")
+        end
+      end
+
       def dependencies
         Dir.glob(@glob).map do |directory|
-          puts "caching #{directory}"
           Licensed::Dependency.new(directory, {
             "type" => Filesystem.type,
-            "name" => File.basename(directory)
+            "name" => File.basename(directory),
+            "version" => submodule_version(directory)
           })
         end
       end

test/test_grammars.rb

@@ -3,53 +3,6 @@
 class TestGrammars < Minitest::Test
   ROOT = File.expand_path("../..", __FILE__)
 
-  # List of projects that are allowed without licenses
-  PROJECT_WHITELIST = [
-    "vendor/grammars/Sublime-Lasso",      # No license file
-    "vendor/grammars/blitzmax",           # No license file
-    "vendor/grammars/creole",             # License filename is not LICENSE(.*)?
-  ].freeze
-
-  HASH_WHITELIST = [
-    "4b4c1b4aff11b99e53091f95b24093655c0d9ed2", # Agda.tmbundle
-    "a92c0e4e644206078a158d663b9f033d0d0aefbf", # ant.tmbundle
-    "98eac91395c97ae090113e1b4527cae3228d563a", # bro-sublime
-    "220e011c8d686129e9c4163a7c655b9d64f61e59", # elixir-tmbundle
-    "55f431fb78e7b3cb6d18ef1042f70ebf4b8d838d", # factor
-    "b81acf2ba52d312754bf5055845a723123bda388", # FreeMarker.tmbundle
-    "ee77ce4cf9121bccc3e37ba6b98f8e7acd589aaf", # gap-tmbundle
-    "4cfc7ce12de920ccc836bbab2d748151d5ba7e38", # go-tmbundle
-    "01868f4a2476fad9abb8b7199ea1547d33cd48db", # hy.tmLanguage
-    "6c2e34d62c08f97a3e2ece3eedc65fbd99873ff4", # idl.tmbundle
-    "e68efca5a844aa78729cadcf42507013151e6605", # jflex.tmbundle
-    "39f092c726491ca6a02354dbc6c3a0920bb44d4c", # mako-tmbundle
-    "7821982b18bc35d6925cc16ece68d9c71f1fbba3", # moonscript-tmbundle
-    "806dae0a87f95409496223dfd9a45cc57c878aeb", # PHP-Twig.tmbundle
-    "0c216b112f3a4e6d5848128504d8378d8c7eee00", # r.tmbundle
-    "da39a3ee5e6b4b0d3255bfef95601890afd80709", # SCSS.tmbundle
-    "68539730d3cde34355f429f2267e265c1e030912", # smalltalk-tmbundle
-    "4b5f67a54532ca6e49ba44cd135a510a74712e07", # Stylus
-    "23d2538e33ce62d58abda2c039364b92f64ea6bc", # sublime-angelscript
-    "38ce879dc8b217a080caa2ff9e2241c9505256d6", # sublime-shen
-    "3df4ef028c6384b64bc59b8861d6c52093b2116d", # sublime-text-ox
-    "c18bcd787325167bf1405629259c091d49064541", # Std license in README.md of many TextMate grammars like abap.tmbundle
-  ].freeze
-
-  # List of allowed SPDX license names
-  LICENSE_WHITELIST = %w[
-    apache-2.0
-    bsd-2-clause
-    bsd-3-clause
-    isc
-    mit
-    mpl-2.0
-    public
-    textmate
-    unlicense
-    wtfpl
-    zlib
-  ].freeze
-
   def setup
     @grammars = YAML.load(File.read(File.join(ROOT, "grammars.yml")))
   end
@@ -89,59 +42,6 @@ def test_readme_file_is_in_sync
     assert_equal current_data, updated_data, "Grammar list is out-of-date. Run `script/list-grammars`"
   end
 
-  def test_submodules_have_recognized_licenses
-    unrecognized = submodule_licenses.select { |k,v| v.nil? && Licensee.project(k).license_file }
-    unrecognized.reject! { |k,v| PROJECT_WHITELIST.include?(k) }
-    message = "The following submodules have unrecognized licenses:\n* #{unrecognized.keys.join("\n* ")}\n"
-    message << "Please ensure that the project's LICENSE file contains the full text of the license"
-    assert_equal Hash.new, unrecognized, message
-  end
-
-  def test_submodules_have_licenses
-    unlicensed = submodule_licenses.select { |k,v| v.nil? }.reject { |k,v| PROJECT_WHITELIST.include?(k) }
-    message = "The following submodules don't have licenses:\n* #{unlicensed.keys.join("\n* ")}\n"
-    message << "Please ensure that the project has a LICENSE file, and that the LICENSE file contains the full text of the license"
-    assert_equal Hash.new, unlicensed, message
-  end
-
-  def test_submodules_have_approved_licenses
-    unapproved = submodule_licenses.reject { |k,v| LICENSE_WHITELIST.include?(v) ||
-                                                   PROJECT_WHITELIST.include?(k) ||
-                                                   HASH_WHITELIST.include?(v) }
-                                   .map { |k,v| "#{k}: #{v}"}
-    message = "The following submodules have unapproved licenses:\n* #{unapproved.join("\n* ")}\n"
-    message << "The license must be added to the LICENSE_WHITELIST in /test/test_grammars.rb once approved"
-    assert_equal [], unapproved, message
-  end
-
-  def test_whitelisted_submodules_dont_have_licenses
-    licensed = submodule_licenses.reject { |k,v| v.nil? }.select { |k,v| PROJECT_WHITELIST.include?(k) }
-    message = "The following whitelisted submodules have a license:\n* #{licensed.keys.join("\n* ")}\n"
-    message << "Please remove them from the project whitelist"
-    assert_equal Hash.new, licensed, message
-  end
-
-  def test_whitelisted_hashes_dont_have_licenses
-    used_hashes = submodule_licenses.values.reject { |v| v.nil? || LICENSE_WHITELIST.include?(v) }
-    unused_hashes = HASH_WHITELIST - used_hashes
-    message = "The following whitelisted license hashes are unused:\n* #{unused_hashes.join("\n* ")}\n"
-    message << "Please remove them from the hash whitelist"
-    assert_equal Array.new, unused_hashes, message
-  end
-
-  def test_submodules_whitelist_has_no_extra_entries
-    skip("Need to work out how to handle dual-licensed entities")
-    extra_whitelist_entries = PROJECT_WHITELIST - submodule_licenses.select { |k,v| v.nil? }.keys
-    not_present = extra_whitelist_entries.reject { |k,v| Dir.exist?(k) }
-    licensed = extra_whitelist_entries.select { |k,v| submodule_licenses[k] }
-
-    msg = "The following whitelisted submodules don't appear to be part of the project:\n* #{not_present.join("\n* ")}"
-    assert_equal [], not_present, msg
-
-    msg = "The following whitelisted submodules actually have licenses and don't need to be whitelisted:\n* #{licensed.join("\n* ")}"
-    assert_equal [], licensed, msg
-  end
-
   def test_submodules_use_https_links
     File.open(".gitmodules", "r") do |fh|
       ssh_submodules = []
@@ -162,35 +62,4 @@ def submodule_paths
     @submodule_paths ||= `git config --list --file "#{File.join(ROOT, ".gitmodules")}"`.lines.grep(/\.path=/).map { |line| line.chomp.split("=", 2).last }.reject { |path| path =~ /CodeMirror/ }
   end
 
-  # Returns a hash of submodules in the form of submodule_path => license
-  def submodule_licenses
-    @@submodule_licenses ||= begin
-      submodules = {}
-      submodule_paths.each { |submodule| submodules[submodule] = submodule_license(submodule) }
-      submodules
-    end
-  end
-
-  # Given the path to a submodule, return its SPDX-compliant license key
-  # If the license is unrecognized, return its hash
-  def submodule_license(submodule)
-    # Prefer Licensee to detect a submodule's license
-    project = Licensee.project(submodule, detect_packages: true, detect_readme: true)
-    return project.license.key if project.licenses.length == 1 && !project.license.pseudo_license?
-
-    # If we have more than one license, return the first one that isn't a
-    # pseudo-license (other or no-license), if any
-    if project.licenses.length > 1
-      first_real_license = project.licenses.reject{ |f| f.pseudo_license? }.first
-      return first_real_license.key unless first_real_license.nil?
-    end
-
-    # We know a license exists, but no method was able to recognize it.
-    # We return the license hash in this case, to uniquely identify it.
-    if project.license_file
-      return project.license_file.content_hash
-    elsif project.readme
-      return project.readme.content_hash
-    end
-  end
 end

vendor/licenses/config.yml

@@ -23,7 +23,9 @@ reviewed:
   rubygem:
     - mime-types
   grammar:
-    - Sublime-Lasso
+    - blitzmax        # No license file. License in README
+    - creole          # License filename is not LICENSE(.*) but rather "The MIT License (MIT)"
+    - Sublime-Lasso   # No license file. License in README
 
 sources:
   npm: false

vendor/licenses/grammar/ABNF.tmbundle.txt

@@ -1,6 +1,7 @@
 ---
 type: grammar
 name: ABNF.tmbundle
+version: 0042b3d7ee21b2f800a8f80d42761513f16d9a52
 license: apache-2.0
 ---
                                  Apache License
@@ -203,4 +204,4 @@ license: apache-2.0
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

vendor/licenses/grammar/Agda.tmbundle.txt

@@ -1,6 +1,7 @@
 ---
 type: grammar
 name: Agda.tmbundle
+version: 0ac22a3c6b231e620c7fa5bd8e83808d33c052ec
 license: bsd-3-clause
 ---
 Copyright (c) 2014 James Cook
@@ -31,4 +32,4 @@ OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
 STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
 ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
-POSSIBILITY OF SUCH DAMAGE.
+POSSIBILITY OF SUCH DAMAGE.

vendor/licenses/grammar/Alloy.tmbundle.txt

@@ -1,6 +1,7 @@
 ---
 type: grammar
 name: Alloy.tmbundle
+version: 0a1e9445864e74b3e93f3a383e1fe93ee6dcfb78
 license: apache-2.0
 ---
 Apache License
@@ -203,4 +204,4 @@ Apache License
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.

vendor/licenses/grammar/Assembly-Syntax-Definition.txt

@@ -1,6 +1,7 @@
 ---
 type: grammar
 name: Assembly-Syntax-Definition
+version: 3d82aa7847b04d39a8f6a6e2fcc8e7c702265b76
 license: mit
 ---
 The MIT License (MIT)
@@ -23,4 +24,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+SOFTWARE.

vendor/licenses/grammar/AutoHotkey.txt

@@ -1,6 +1,7 @@
 ---
 type: grammar
 name: AutoHotkey
+version: dc20ea3b615d902fc43045c1073353af5f9e0edf
 license: unlicense
 ---
 This is free and unencumbered software released into the public domain.
@@ -26,4 +27,4 @@ OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
 ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 OTHER DEALINGS IN THE SOFTWARE.
 
-For more information, please refer to <http://unlicense.org/>
+For more information, please refer to <http://unlicense.org/>

vendor/licenses/grammar/BrightScript.tmbundle.txt

@@ -1,6 +1,7 @@
 ---
 type: grammar
 name: BrightScript.tmbundle
+version: 905791b02bfc7256604bbc874f8c259713d62b12
 license: mit
 ---
 The MIT License (MIT)
@@ -23,4 +24,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+SOFTWARE.

vendor/licenses/grammar/ColdFusion.txt

@@ -1,6 +1,7 @@
 ---
 type: grammar
 name: ColdFusion
+version: dbe2f76038fe77fe90b4cc4a0651aa817f14f1a3
 license: mit
 ---
 The MIT License (MIT)
@@ -23,4 +24,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
+THE SOFTWARE.

vendor/licenses/grammar/Docker.tmbundle.txt

@@ -1,6 +1,7 @@
 ---
 type: grammar
 name: Docker.tmbundle
+version: 6e521ead6cd39013b8d8b95d6d2e2c9e5a548464
 license: mit
 ---
 The MIT License (MIT)
@@ -23,4 +24,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
+THE SOFTWARE.

vendor/licenses/grammar/EBNF.tmbundle.txt

@@ -1,6 +1,7 @@
 ---
 type: grammar
 name: EBNF.tmbundle
+version: 13efe9156b74c97c0d6b3d1fa46d4ad839fa80e5
 license: mit
 ---
 Copyright (C) 2012  by Arne Schroppe
@@ -21,4 +22,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
+THE SOFTWARE.

vendor/licenses/grammar/Elm.txt

@@ -1,6 +1,7 @@
 ---
 type: grammar
 name: Elm
+version: 6bbbca9ccd0bd3131575217b3780fa59d059fcba
 license: mit
 ---
 The MIT License (MIT)
@@ -22,4 +23,4 @@ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
 FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
 COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
 IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.