Identify Arduino libraries in Github repos' subfolders and more.. (no-html-encode)

[fpapadopou]

ChangeLog

• Added support for subfolder selection in multi-library repos, fetched from Github.
• Revamped the library addition page frontend.
• Added extra information to the ExternalLibrary Doctrine entity. These are the branch, inRepoPath, sourceUrl and notes properties. That is, a doctrine:schema:update will definitely be needed on the production server.
  Note for the author: Double check no other Entity properties will be affected before running the schema update
• Made the getLastCommit functionality fetch the last commit, depending on the selected repo branch and subfolder (if any).
• Proper handling of files to be stored to the library manager filesystem. Files are not UTF8-encoded any more on upload. Instead, they are encoded during fetching, and the original file is stored in tact in the filesystem.
• Used Github's blob API for file fetching, which supports file size up to 100 MB. This fixes the issue with the large library files, that made the library manager unable to fetch such libraries from Github.
• [removed] Made all returned text files HTML encoded (using htmlspecialchars), which should fix any issues with libraries containing HTML docs, and other such files that could break the JSON encoding.
• Fixed the download functionality, which would return all non-text files as UTF-8 encoded text sequences, which made no sense (e.g. mp3 files).
• Updated checkGithubUpdates method, made it use the branch and in-repo path (if available). This enables correctly getting the last commit for a specific branch (if a specific branch of the repo is used in eratosthenes), or for a specific path in the repo (if we've used a specific path of a multi-library repo).
• Moved external libraries out of the arduino library files repo and put them on separate parameters.
• Updated Symfony to version 2.3.33 (current 2.3 latest) and PHPUnit to version 4.8.14 (current 4.8 latest)
• Added initial scripts/data for fixture libraries & examples setup

BC Break

The checkGithubUpdates method was not designed to work in this context (with more specific information about Github libraries), thus it will not return correct results, if it returns any. In fact, this method, is that it is unused and needed to be re-written. fixed

Merge process

• Will need a composer install right after the update because Symfony vendor and its deps have beed updated in the composer.lock file
• Will need a schema update on the database, right after merging the new code (need to double check no data loss will take place during the update).
• Will need to add the two new parameters builtin_libraries and external_libraries. These two should point at the correct directories which contain the actual libraries files. Watch out for permissions issues
• (optional) It would be a good idea (will take time though), to double check metadata of existing Github libraries (branch & lastCommit) after merging.

For further investigation

Due to recursive calls to Github's API, adding a library from Github can take a long time. However, it's a pretty safe way of fetching code from Github. An example is UTFT library from (here)[https://github.com/mysensors/Arduino/tree/development/libraries/UTFT], which takes about 5 mins to get fetched on a local environment. Things could better though on an AWS server.

QA

Notes coming up

[fpapadopou]

Reviewed, all ok.