OSC 8 support for conhost and terminal (#7251)

<!-- Enter a brief description/summary of your PR here. What does it fix/what does it change/how was it tested (even manually, if necessary)? -->
## Summary of the Pull Request
Conhost can now support OSC8 sequences (as specified [here](https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda)). Terminal also supports those sequences and additionally hyperlinks can be opened by Ctrl+LeftClicking on them. 

<!-- Other than the issue solved, is this relevant to any other issues/existing PRs? --> 
## References
#204 

<!-- Please review the items on the PR checklist before submitting-->
## PR Checklist
* [X] Closes #204 
* [ ] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/Terminal) and sign the CLA
* [ ] Tests added/passed
* [ ] Documentation updated. If checked, please file a pull request on [our docs repo](https://github.com/MicrosoftDocs/terminal) and link it here: #xxx
* [ ] Schema updated.
* [ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

<!-- Provide a more detailed description of the PR, other things fixed or any additional comments/features here -->
## Detailed Description of the Pull Request / Additional comments
Added support to:

- parse OSC8 sequences and extract URIs from them (conhost and terminal)
- add hyperlink uri data to textbuffer/screeninformation, associated with a hyperlink id (conhost and terminal)
- attach hyperlink ids to text to allow for uri extraction from the textbuffer/screeninformation (conhost and terminal)
- process ctrl+leftclick to open a hyperlink in the clicked region if present

<!-- Describe how you validated the behavior. Add automated tests wherever possible, but list manual validation steps taken as well -->
## Validation Steps Performed
Open up a PowerShell tab and type
```PowerShell
${ESC}=[char]27
Write-Host "${ESC}]8;;https://github.com/microsoft/terminal${ESC}\This is a link!${ESC}]8;;${ESC}\"
```
Ctrl+LeftClick on the link correctly brings you to the terminal page on github

![hyperlink](https://user-images.githubusercontent.com/26824113/89953536-45a6f580-dbfd-11ea-8e0d-8a3cd25c634a.gif)

.github/actions/spell-check/dictionary/apis.txt

@@ -50,5 +50,6 @@ syscall
 tmp
 tx
 userenv
+wcstoui
 XDocument
 XElement

.github/actions/spell-check/dictionary/dictionary.txt

@@ -186988,6 +186988,7 @@ hyperleucocytotic
 hyperleukocytosis
 hyperlexis
 hyperlink
+hyperlinks
 hyperlinking
 hyperlipaemia
 hyperlipaemic

src/buffer/out/AttrRow.cpp

@@ -175,6 +175,23 @@ size_t ATTR_ROW::FindAttrIndex(const size_t index, size_t* const pApplies) const
     return runPos - _list.cbegin();
 }
 
+// Routine Description:
+// - Finds the hyperlink IDs present in this row and returns them
+// Return value:
+// - An unordered set containing the hyperlink IDs present in this row
+std::unordered_set<uint16_t> ATTR_ROW::GetHyperlinks()
+{
+    std::unordered_set<uint16_t> ids;
+    for (const auto& run : _list)
+    {
+        if (run.GetAttributes().IsHyperlink())
+        {
+            ids.emplace(run.GetAttributes().GetHyperlinkId());
+        }
+    }
+    return ids;
+}
+
 // Routine Description:
 // - Sets the attributes (colors) of all character positions from the given position through the end of the row.
 // Arguments:

src/buffer/out/AttrRow.hpp

@@ -41,6 +41,8 @@ class ATTR_ROW final
     size_t FindAttrIndex(const size_t index,
                          size_t* const pApplies) const;
 
+    std::unordered_set<uint16_t> GetHyperlinks();
+
     bool SetAttrToEnd(const UINT iStart, const TextAttribute attr);
     void ReplaceAttrs(const TextAttribute& toBeReplacedAttr, const TextAttribute& replaceWith) noexcept;
 

src/buffer/out/TextAttribute.cpp

@@ -112,6 +112,17 @@ std::pair<COLORREF, COLORREF> TextAttribute::CalculateRgbColors(const gsl::span<
     return { fg, bg };
 }
 
+// Method description:
+// - Tells us whether the text is a hyperlink or not
+// Return value:
+// - True if it is a hyperlink, false otherwise
+bool TextAttribute::IsHyperlink() const noexcept
+{
+    // All non-hyperlink text have a default hyperlinkId of 0 while
+    // all hyperlink text have a non-zero hyperlinkId
+    return _hyperlinkId != 0;
+}
+
 TextColor TextAttribute::GetForeground() const noexcept
 {
     return _foreground;
@@ -122,6 +133,15 @@ TextColor TextAttribute::GetBackground() const noexcept
     return _background;
 }
 
+// Method description:
+// - Retrieves the hyperlink ID of the text
+// Return value:
+// - The hyperlink ID
+uint16_t TextAttribute::GetHyperlinkId() const noexcept
+{
+    return _hyperlinkId;
+}
+
 void TextAttribute::SetForeground(const TextColor foreground) noexcept
 {
     _foreground = foreground;
@@ -174,6 +194,15 @@ void TextAttribute::SetColor(const COLORREF rgbColor, const bool fIsForeground)
     }
 }
 
+// Method description:
+// - Sets the hyperlink ID of the text
+// Arguments:
+// - id - the id we wish to set
+void TextAttribute::SetHyperlinkId(uint16_t id) noexcept
+{
+    _hyperlinkId = id;
+}
+
 bool TextAttribute::IsLeadingByte() const noexcept
 {
     return WI_IsFlagSet(_wAttrLegacy, COMMON_LVB_LEADING_BYTE);
@@ -336,6 +365,14 @@ void TextAttribute::SetDefaultBackground() noexcept
     _background = TextColor();
 }
 
+// Method description:
+// - Resets only the meta and extended attributes
+void TextAttribute::SetDefaultMetaAttrs() noexcept
+{
+    _extendedAttrs = ExtendedAttributes::Normal;
+    _wAttrLegacy = 0;
+}
+
 // Method Description:
 // - Returns true if this attribute indicates its background is the "default"
 //      background. Its _rgbBackground will contain the actual value of the
@@ -356,6 +393,6 @@ bool TextAttribute::BackgroundIsDefault() const noexcept
 //      requires for most erasing and filling operations.
 void TextAttribute::SetStandardErase() noexcept
 {
-    _extendedAttrs = ExtendedAttributes::Normal;
-    _wAttrLegacy = 0;
+    SetDefaultMetaAttrs();
+    _hyperlinkId = 0;
 }

src/buffer/out/TextAttribute.hpp

@@ -36,15 +36,17 @@ class TextAttribute final
         _wAttrLegacy{ 0 },
         _foreground{},
         _background{},
-        _extendedAttrs{ ExtendedAttributes::Normal }
+        _extendedAttrs{ ExtendedAttributes::Normal },
+        _hyperlinkId{ 0 }
     {
     }
 
     explicit constexpr TextAttribute(const WORD wLegacyAttr) noexcept :
         _wAttrLegacy{ gsl::narrow_cast<WORD>(wLegacyAttr & META_ATTRS) },
         _foreground{ s_LegacyIndexOrDefault(wLegacyAttr & FG_ATTRS, s_legacyDefaultForeground) },
         _background{ s_LegacyIndexOrDefault((wLegacyAttr & BG_ATTRS) >> 4, s_legacyDefaultBackground) },
-        _extendedAttrs{ ExtendedAttributes::Normal }
+        _extendedAttrs{ ExtendedAttributes::Normal },
+        _hyperlinkId{ 0 }
     {
         // If we're given lead/trailing byte information with the legacy color, strip it.
         WI_ClearAllFlags(_wAttrLegacy, COMMON_LVB_SBCSDBCS);
@@ -55,7 +57,8 @@ class TextAttribute final
         _wAttrLegacy{ 0 },
         _foreground{ rgbForeground },
         _background{ rgbBackground },
-        _extendedAttrs{ ExtendedAttributes::Normal }
+        _extendedAttrs{ ExtendedAttributes::Normal },
+        _hyperlinkId{ 0 }
     {
     }
 
@@ -112,8 +115,11 @@ class TextAttribute final
 
     ExtendedAttributes GetExtendedAttributes() const noexcept;
 
+    bool IsHyperlink() const noexcept;
+
     TextColor GetForeground() const noexcept;
     TextColor GetBackground() const noexcept;
+    uint16_t GetHyperlinkId() const noexcept;
     void SetForeground(const TextColor foreground) noexcept;
     void SetBackground(const TextColor background) noexcept;
     void SetForeground(const COLORREF rgbForeground) noexcept;
@@ -123,9 +129,11 @@ class TextAttribute final
     void SetIndexedForeground256(const BYTE fgIndex) noexcept;
     void SetIndexedBackground256(const BYTE bgIndex) noexcept;
     void SetColor(const COLORREF rgbColor, const bool fIsForeground) noexcept;
+    void SetHyperlinkId(uint16_t id) noexcept;
 
     void SetDefaultForeground() noexcept;
     void SetDefaultBackground() noexcept;
+    void SetDefaultMetaAttrs() noexcept;
 
     bool BackgroundIsDefault() const noexcept;
 
@@ -147,7 +155,8 @@ class TextAttribute final
                (_wAttrLegacy & META_ATTRS) == (other._wAttrLegacy & META_ATTRS) &&
                ((checkForeground && _foreground == other._foreground) ||
                 (!checkForeground && _background == other._background)) &&
-               _extendedAttrs == other._extendedAttrs;
+               _extendedAttrs == other._extendedAttrs &&
+               IsHyperlink() == other.IsHyperlink();
     }
 
     constexpr bool IsAnyGridLineEnabled() const noexcept
@@ -169,6 +178,8 @@ class TextAttribute final
     TextColor _background;
     ExtendedAttributes _extendedAttrs;
 
+    uint16_t _hyperlinkId;
+
 #ifdef UNIT_TESTING
     friend class TextBufferTests;
     friend class TextAttributeTests;
@@ -182,7 +193,7 @@ class TextAttribute final
 // 4 for _foreground
 // 4 for _background
 // 1 for _extendedAttrs
-static_assert(sizeof(TextAttribute) <= 11 * sizeof(BYTE), "We should only need 11B for an entire TextColor. Any more than that is just waste");
+static_assert(sizeof(TextAttribute) <= 13 * sizeof(BYTE), "We should only need 13B for an entire TextAttribute. We may need to increment this in the future as we add additional attributes");
 
 enum class TextAttributeBehavior
 {
@@ -196,7 +207,8 @@ constexpr bool operator==(const TextAttribute& a, const TextAttribute& b) noexce
     return a._wAttrLegacy == b._wAttrLegacy &&
            a._foreground == b._foreground &&
            a._background == b._background &&
-           a._extendedAttrs == b._extendedAttrs;
+           a._extendedAttrs == b._extendedAttrs &&
+           a._hyperlinkId == b._hyperlinkId;
 }
 
 constexpr bool operator!=(const TextAttribute& a, const TextAttribute& b) noexcept

src/buffer/out/textBuffer.cpp

@@ -34,7 +34,8 @@ TextBuffer::TextBuffer(const COORD screenBufferSize,
     _storage{},
     _unicodeStorage{},
     _renderTarget{ renderTarget },
-    _size{}
+    _size{},
+    _currentHyperlinkId{ 1 }
 {
     // initialize ROWs
     for (size_t i = 0; i < static_cast<size_t>(screenBufferSize.Y); ++i)
@@ -551,7 +552,10 @@ bool TextBuffer::IncrementCircularBuffer(const bool inVtMode)
     // to the logical position 0 in the window (cursor coordinates and all other coordinates).
     _renderTarget.TriggerCircling();
 
-    // First, clean out the old "first row" as it will become the "last row" of the buffer after the circle is performed.
+    // Prune hyperlinks to delete obsolete references
+    _PruneHyperlinks();
+
+    // Second, clean out the old "first row" as it will become the "last row" of the buffer after the circle is performed.
     auto fillAttributes = _currentAttributes;
     if (inVtMode)
     {
@@ -1185,6 +1189,46 @@ const COORD TextBuffer::_GetWordEndForSelection(const COORD target, const std::w
     return result;
 }
 
+void TextBuffer::_PruneHyperlinks()
+{
+    // Check the old first row for hyperlink references
+    // If there are any, search the entire buffer for the same reference
+    // If the buffer does not contain the same reference, we can remove that hyperlink from our map
+    // This way, obsolete hyperlink references are cleared from our hyperlink map instead of hanging around
+    // Get all the hyperlink references in the row we're erasing
+    auto firstRowRefs = _storage.at(_firstRow).GetAttrRow().GetHyperlinks();
+    if (!firstRowRefs.empty())
+    {
+        const auto total = TotalRowCount();
+        // Loop through all the rows in the buffer except the first row -
+        // we have found all hyperlink references in the first row and put them in refs,
+        // now we need to search the rest of the buffer (i.e. all the rows except the first)
+        // to see if those references are anywhere else
+        for (size_t i = 1; i != total; ++i)
+        {
+            const auto nextRowRefs = GetRowByOffset(i).GetAttrRow().GetHyperlinks();
+            for (auto id : nextRowRefs)
+            {
+                if (firstRowRefs.find(id) != firstRowRefs.end())
+                {
+                    firstRowRefs.erase(id);
+                }
+            }
+            if (firstRowRefs.empty())
+            {
+                // No more hyperlink references left to search for, terminate early
+                break;
+            }
+        }
+    }
+
+    // Now delete obsolete references from our map
+    for (auto hyperlinkReference : firstRowRefs)
+    {
+        RemoveHyperlinkFromMap(hyperlinkReference);
+    }
+}
+
 // Method Description:
 // - Update pos to be the position of the first character of the next word. This is used for accessibility
 // Arguments:
@@ -2142,6 +2186,7 @@ HRESULT TextBuffer::Reflow(TextBuffer& oldBuffer,
     {
         // Finish copying remaining parameters from the old text buffer to the new one
         newBuffer.CopyProperties(oldBuffer);
+        newBuffer.CopyHyperlinkMaps(oldBuffer);
 
         // If we found where to put the cursor while placing characters into the buffer,
         //   just put the cursor there. Otherwise we have to advance manually.
@@ -2207,3 +2252,104 @@ HRESULT TextBuffer::Reflow(TextBuffer& oldBuffer,
 
     return hr;
 }
+
+// Method Description:
+// - Adds or updates a hyperlink in our hyperlink table
+// Arguments:
+// - The hyperlink URI, the hyperlink id (could be new or old)
+void TextBuffer::AddHyperlinkToMap(std::wstring_view uri, uint16_t id)
+{
+    _hyperlinkMap[id] = uri;
+}
+
+// Method Description:
+// - Retrieves the URI associated with a particular hyperlink ID
+// Arguments:
+// - The hyperlink ID
+// Return Value:
+// - The URI
+std::wstring TextBuffer::GetHyperlinkUriFromId(uint16_t id) const
+{
+    return _hyperlinkMap.at(id);
+}
+
+// Method description:
+// - Provides the hyperlink ID to be assigned as a text attribute, based on the optional custom id provided
+// Arguments:
+// - The user-defined id
+// Return value:
+// - The internal hyperlink ID
+uint16_t TextBuffer::GetHyperlinkId(std::wstring_view params)
+{
+    uint16_t id = 0;
+    if (params.empty())
+    {
+        // no custom id specified, return our internal count
+        id = _currentHyperlinkId;
+        ++_currentHyperlinkId;
+    }
+    else
+    {
+        // assign _currentHyperlinkId if the custom id does not already exist
+        const auto result = _hyperlinkCustomIdMap.emplace(params, _currentHyperlinkId);
+        if (result.second)
+        {
+            // the custom id did not already exist
+            ++_currentHyperlinkId;
+        }
+        id = (*(result.first)).second;
+    }
+    // _currentHyperlinkId could overflow, make sure its not 0
+    if (_currentHyperlinkId == 0)
+    {
+        ++_currentHyperlinkId;
+    }
+    return id;
+}
+
+// Method Description:
+// - Removes a hyperlink from the hyperlink map and the associated
+//   user defined id from the custom id map (if there is one)
+// Arguments:
+// - The ID of the hyperlink to be removed
+void TextBuffer::RemoveHyperlinkFromMap(uint16_t id)
+{
+    _hyperlinkMap.erase(id);
+    for (const auto& customIdPair : _hyperlinkCustomIdMap)
+    {
+        if (customIdPair.second == id)
+        {
+            _hyperlinkCustomIdMap.erase(customIdPair.first);
+            break;
+        }
+    }
+}
+
+// Method Description:
+// - Obtains the custom ID, if there was one, associated with the
+//   uint16_t id of a hyperlink
+// Arguments:
+// - The uint16_t id of the hyperlink
+// Return Value:
+// - The custom ID if there was one, empty string otherwise
+std::wstring TextBuffer::GetCustomIdFromId(uint16_t id) const
+{
+    for (auto customIdPair : _hyperlinkCustomIdMap)
+    {
+        if (customIdPair.second == id)
+        {
+            return customIdPair.first;
+        }
+    }
+    return {};
+}
+
+// Method Description:
+// - Copies the hyperlink/customID maps of the old buffer into this one
+// Arguments:
+// - The other buffer
+void TextBuffer::CopyHyperlinkMaps(const TextBuffer& other)
+{
+    _hyperlinkMap = other._hyperlinkMap;
+    _hyperlinkCustomIdMap = other._hyperlinkCustomIdMap;
+}