Presentations/TRC-2023/Notebooks/Workflows-with-LLM-functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "38e15d7a",
   "metadata": {},
   "source": [
    "# Workflows with LLM functions\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this computational Markdown document we discuss and demonstrate the inclusion and integration of\n",
    "Large Language Model (LLM) functions into different types of Raku workflows.\n",
    "\n",
    "Since LLMs hallucinate results, it becomes necessary to manipulate their inputs, the outputs, or both. \n",
    "Therefore, having a system for managing, coordinating, and streamlining LLM requests, \n",
    "along with methods for incorporating these requests into the \"playgrounds\" of a certain programming language, \n",
    "would be highly beneficial.\n",
    "\n",
    "This is what the package \n",
    "[\"LLM::Functions\"](https://raku.land/zef:antononcube/LLM::Functions), [AAp1],\n",
    "aims to do in Raku and Raku's ecosystem. \n",
    "\n",
    "### Dynamic duo\n",
    "\n",
    "LLMs are celebrated for producing good to great results, but they have a few big issues. \n",
    "The content they generate can be inconsistent, prone to hallucination, and sometimes biased, making it unreliable.\n",
    "The form, or stylistic structure, may also vary widely, with a lack of determinism and sensitivity \n",
    "to hyperparameters contributing to challenges in reproducibility. \n",
    "Moreover, customization and debugging can be complex due to these inconsistencies. \n",
    "\n",
    "The lack of reliability and reproducibility in both content and form underscore\n",
    "the need for streamlining, managing, and transforming LLM inquiries and results.\n",
    "\n",
    "Raku, with its unique approach to text manipulation, not surprisingly complements LLMs nicely. \n",
    "While Raku might not be everyone's favorite language and has certain intricacies that take some getting used to, \n",
    "its strengths in handling text patterns are hard to ignore. ***Creating well-crafted pairings of Raku with LLMs \n",
    "can broaden Raku's adoption and utilization.***\n",
    "\n",
    "\"LLM::Functions\" establishes a (functional programming) connection between Raku's capabilities and the vast potential of LLMs. \n",
    "Ideally that promising LLM-Raku pairing is further strengthened and enriched into something that some might call a \"dynamic duo.\"\n",
    "\n",
    "**Remark:** For an example of a mature effort with the same mission (and naming, and design) see [SW1] and [WRIp1].\n",
    "\n",
    "**Remark:** And yes, for Mathematica or Wolfram Language (WL) it can be also said:\n",
    "*Creating well-crafted pairings of WL with LLMs can broaden WL's adoption and utilization.*\n",
    "WL, though, is much better positioned for integrating with multi-modal LLMs because of WL's\n",
    "ability to create and manipulate symbolic representation of different types of objects \n",
    "(audio, images, and video included), and WL's very advanced notebook technology.\n",
    "\n",
    "### Standard enhancements\n",
    "\n",
    "To enhance the pairing of Raku with LLMs, it is *also* essential to have:\n",
    "- LLM prompt repository with many well documented prompts\n",
    "- Polyglot parsing of dates, numbers, regular expressions, data formats, grammar specs, etc.  \n",
    "\n",
    "For an example of the former see the Wolfram Prompt Repository, [WRIr1].\n",
    "For examples of the latter see [AAp4], [MSp1, MSp2].\n",
    "\n",
    "**Remark:** I like the idea of having ready to apply \"power\" tokens like `<local-number>`\n",
    "provided by \"Intl::Token::Number\", [MSp1].\n",
    "\n",
    "**Remark:** For some reason the developer of \n",
    "\"Intl::Token::Number\", [MSp1], and \"Polyglot::Regexen\", [MSp2], prefers to make \n",
    "[Brainfuck](https://en.wikipedia.org/wiki/Brainfuck) \n",
    "[parsers](https://github.com/alabamenhu/PolyglotBrainfuck) and \n",
    "travel to Canada to [talk about it](https://www.youtube.com/watch?v=LSnkFfE7vPg)\n",
    "than making those packages ready to be used by \"LLM::Functions\", [AAp1], and \"Text::SubParsers\", [AAp4].\n",
    "\n",
    "### Interactivity is needed\n",
    "\n",
    "Generally speaking, using LLM functions in Raku (or Mathematica, or Python, or R) requires \n",
    "good tools for [Read Eval Print Loop (REPL)](https://en.wikipedia.org/wiki/Read–eval–print_loop).\n",
    "\n",
    "Notebooks are best for LLM utilization because notebooks offer an interactive environment where\n",
    "LLM whisperers, LLM tamers, neural net navigators, and bot wranglers can write code, run it, see the results, \n",
    "and tweak the code -- all in one place.\n",
    "\n",
    "Raku currently has (at least) two notebook solutions: \n",
    "1. [\"Jupyter::Kernel\"](https://raku.land/cpan:BDUGGAN/Jupyter::Kernel) with the [Jupyter framework](https://jupyter.org)\n",
    "2. [\"Text::CodeProcessing\"](https://raku.land/?q=Text%3A%3ACodeProcessing) \n",
    "and [\"RakuMode\" for Mathematica](https://resources.wolframcloud.com/PacletRepository/resources/AntonAntonov/RakuMode/), [AA2].\n",
    "\n",
    "Raku second best LLM-REPL solutions are those like \n",
    "[Comma's REPL](https://commaide.com/features) and \n",
    "[Emacs Raku Mode](https://github.com/Raku/raku-mode). \n",
    "\n",
    "\"Just\" using scripts is an option, but since LLM queries have certain time lag and usage expenses, it is not a good one:\n",
    "- We cannot see the intermediate results and adjust accordingly\n",
    "- Multiple (slow) executions would be needed to get desired results\n",
    "\n",
    "**Remark:** The very first version of this article was made with \"Text::CodeProcessing\" via Markdown execution (or weaving.)\n",
    "Then Comma's REPL was used, for extending and refining the examples. \"Jupyter::Kernel\" was also used\n",
    "for a few of the sections."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15c7fb02",
   "metadata": {},
   "source": [
    "### Article structure\n",
    "\n",
    "Here are sections of the article:\n",
    "\n",
    "- **General structure of LLM-based workflows**   \n",
    "  ... Formulating and visualizing the overall process used in all LLM workflow examples.\n",
    "- **Plot data**   \n",
    "  ... Plotting LLM-retrieved data.\n",
    "- **Normalizing outputs**   \n",
    "  ... Examples of how LLM-function outputs can be \"normalized\" using other LLM functions.\n",
    "- **Conversion to Raku objects**   \n",
    "  ... Conversion of LLM-outputs in Raku physical units objects.\n",
    "- **Chemical formulas**   \n",
    "  ... Retrieving chemical formulas and investigating them.\n",
    "- **Making (embedded) Mermaid diagrams**   \n",
    "  ... Straightforward application of LLM abilities and literate programming tools.\n",
    "- **Named entity recognition**  \n",
    "  ... How to obtain music album names and release dates and tabulate or plot them.\n",
    "- **Statistics of output data types**   \n",
    "  ... Illustration why programmers need streamlining solutions for LLMs.\n",
    "- **Other workflows**   \n",
    "  ... Outline of other workflows using LLM chat objects. (Also provided by \"LLM::Functions\".)\n",
    "\n",
    "**Remark:** Most of the sections have a sub-section titled \"Exercise questions\". \n",
    "The reader is the secondary target audience for those. The primary target are LLMs to respond to them.\n",
    "(Another article is going to discuss the staging and evaluating of those LLM answers.) \n",
    "\n",
    "### Packages and LLM access\n",
    "\n",
    "The following Raku packages used below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c2508f28",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:38.970145Z",
     "start_time": "2023-08-16T14:51:36.745765Z"
    },
    "incorrectly_encoded_metadata": ", results=hide"
   },
   "outputs": [],
   "source": [
    "use LLM::Functions;\n",
    "use Text::SubParsers;\n",
    "\n",
    "use Data::Reshapers;\n",
    "use Data::TypeSystem;\n",
    "use Data::Generators;\n",
    "use Data::Summarizers;\n",
    "use Data::ExampleDatasets;\n",
    "\n",
    "use Text::Plot;\n",
    "use JavaScript::D3;\n",
    "use Markdown::Grammar;\n",
    "\n",
    "use Physics::Unit;\n",
    "\n",
    "use Chemistry::Stoichiometry;\n",
    "\n",
    "use JSON::Fast;\n",
    "use HTTP::Tiny;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "100f8fb2",
   "metadata": {},
   "source": [
    "\"Out of the box\"\n",
    "[\"LLM::Functions\"](https://raku.land/zef:antononcube/LLM::Functions) uses\n",
    "[\"WWW::OpenAI\"](https://raku.land/zef:antononcube/WWW::OpenAI), [AAp2], and\n",
    "[\"WWW::PaLM\"](https://raku.land/zef:antononcube/WWW::PaLM), [AAp3].\n",
    "Other LLM access packages can utilized via appropriate LLM configurations.\n",
    "\n",
    "The LLM functions below use the LLM authorization tokens that are kept\n",
    "in the OS environment. See [AAp2] and [AAp3] for details how to setup LLM access.\n",
    "\n",
    "The Markdown document is executed (or \"woven\") with the CLI script of the package\n",
    "[\"Text::CodeProcessing\"](https://raku.land/zef:antononcube/Text::CodeProcessing), [AA5].\n",
    "\"Text::CodeProcessing\" has features that allow the woven documents to have render-ready \n",
    "Markdown cells, like, tables, Mermaid-JS diagrams, or JavaScript plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1d36f4b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": "require.config({\n     paths: {\n     d3: 'https://d3js.org/d3.v7.min'\n}});\n\nrequire(['d3'], function(d3) {\n     console.log(d3);\n});"
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%% javascript\n",
    "require.config({\n",
    "     paths: {\n",
    "     d3: 'https://d3js.org/d3.v7.min'\n",
    "}});\n",
    "\n",
    "require(['d3'], function(d3) {\n",
    "     console.log(d3);\n",
    "});"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaeee3b8",
   "metadata": {},
   "source": [
    "Here define a function that converts plain text tables into HTML tables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fa09be59",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "&to-html"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sub to-html($x) { md-interpret($x.Str.lines[1..*-2].join(\"\\n\").subst('+--','|--', :g).subst('--+','--|', :g), actions=>Markdown::Actions::HTML.new) }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f7d6906",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "## General structure of LLM-based workflows\n",
    "\n",
    "All systematic approaches of unfolding and refining workflows based on LLM functions, \n",
    "will include several decision points and iterations to ensure satisfactory results.\n",
    "\n",
    "This flowchart outlines such a systematic approach:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d3cf33d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "use WWW::MermaidInk;\n",
    "use Text::Plot;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2318334e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"data:image/png;base64,Tm90IEZvdW5k\">"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my $spec = q:to/ENDMM/;\n",
    "graph TD\n",
    "    A([Start]) --> HumanWorkflow[Outline a workflow] --> MakeLLMFuncs[\"Make LLM function(s)\"]\n",
    "    MakeLLMFuncs --> MakePipeline[Make pipeline]\n",
    "    MakePipeline --> LLMEval[\"Evaluate LLM function(s)\"]\n",
    "    LLMEval --> HumanAsses[\"Asses LLM's Outputs\"]\n",
    "    HumanAsses --> GoodLLMQ{\"Good or workable<br>results?\"}\n",
    "    GoodLLMQ --> |No| CanProgramQ{\"Can you<br>programmatically<br>change the<br>outputs?\"}\n",
    "    CanProgramQ --> |No| KnowVerb{\"Can you<br>verbalize<br>the required<br>change?\"}\n",
    "    KnowVerb --> |No| KnowRule{\"Can you<br>specify the change<br>as a set of training<br>rules?\"}\n",
    "    KnowVerb --> |Yes| ShouldAddLLMQ{\"Is it better to<br>make additional<br>LLM function(s)?\"}\n",
    "    ShouldAddLLMQ --> |Yes| AddLLM[\"Make additional<br>LLM function(s)\"]\n",
    "    AddLLM --> MakePipeline\n",
    "    ShouldAddLLMQ --> |No| ChangePrompt[\"Change prompt(s)<br>of LLM function(s)\"]\n",
    "    ChangePrompt --> ChangeOutputDescr[\"Change output description(s)<br>of LLM function(s)\"]\n",
    "    ChangeOutputDescr --> MakeLLMFuncs\n",
    "    CanProgramQ --> |Yes| ApplySubParser[\"Apply suitable (sub-)parsers\"]\n",
    "    ApplySubParser --> HumanMassageOutput[Program output transformations]\n",
    "    HumanMassageOutput --> MakePipeline\n",
    "    GoodLLMQ --> |Yes| OverallGood{\"Overall<br>satisfactory<br>(robust enough)<br>results?\"}\n",
    "    OverallGood --> |Yes| End([End])\n",
    "    OverallGood --> |No| DifferentModelQ{\"Willing and able<br>to apply<br>different model(s) or<br>model parameters?\"}\n",
    "    DifferentModelQ --> |No| HumanWorkflow\n",
    "    DifferentModelQ --> |Yes| ChangeModel[\"Change model<br>or model parameters\"]\n",
    "    ChangeModel --> MakeLLMFuncs\n",
    "    KnowRule --> |Yes| LLMExamFunc[Make LLM example function]\n",
    "    KnowRule --> |No| HumanWorkflow\n",
    "    LLMExamFunc --> MakePipeline \n",
    "ENDMM\n",
    "\n",
    "mermaid-ink($spec, format=>'md-image') ==> from-base64"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32721c3b",
   "metadata": {},
   "source": [
    "\n",
    "Here is a corresponding description:\n",
    "\n",
    "- **Start**: The beginning of the process.\n",
    "- **Outline a workflow**: The stage where a human outlines a general workflow for the process.\n",
    "- **Make LLM function(s)**: Creation of specific LLM function(s).\n",
    "- **Make pipeline**: Construction of a pipeline to integrate the LLM function(s).\n",
    "- **Evaluate LLM function(s)**: Evaluation of the created LLM function(s).\n",
    "- **Asses LLM's Outputs**: A human assesses the outputs from the LLM.\n",
    "- **Good or workable results?**: A decision point to check whether the results are good or workable.\n",
    "- **Can you programmatically change the outputs?**: If not satisfactory, a decision point to check if the outputs can be changed programmatically.\n",
    "  - *The human acts like a real programmer.*\n",
    "- **Can you verbalize the required change?**: If not programmable, a decision point to check if the changes can be verbalized.\n",
    "  - *The human programming is delegated to the LLM.*\n",
    "- **Can you specify the change as a set of training rules?**: If not verbalizable, a decision point to check if the change can be specified as training rules.\n",
    "  - *The human cannot program or verbalize the required changes, but can provide examples of those changes.*\n",
    "- **Is it better to make additional LLM function(s)?**: If changes can be verbalized, a decision point to check whether it is better to make additional LLM function(s), or it is better to change prompts or output descriptions. \n",
    "- **Make additional LLM function(s)**: Make additional LLM function(s) (since it is considered to be the better option.)  \n",
    "- **Change prompts of LLM function(s)**: Change prompts of already created LLM function(s).\n",
    "- **Change output description(s) of LLM function(s)**: Change output description(s) of already created LLM function(s).\n",
    "- **Apply suitable (sub-)parsers**: If changes can be programmed, choose, or program, and apply suitable parser(s) or sub-parser(s) for LLM's outputs.\n",
    "- **Program output transformations**: Transform the outputs of the (sub-)parser(s) programmatically.\n",
    "- **Overall satisfactory (robust enough) results?**: A decision point to assess whether the results are overall satisfactory.\n",
    "  - *This should include evaluation or estimate how robust and reproducible the results are.*\n",
    "- **Willing and able to apply different model(s) or model parameters?**: A decision point should the LLM functions pipeline should evaluated or tested with different LLM model or model parameters.\n",
    "  - *In view of robustness and reproducibility, systematic change of LLM models and LLM functions pipeline inputs should be considered.* \n",
    "- **Change model or model parameters**: If willing to change models or model parameters then do so.\n",
    "  - *Different models can have different adherence to prompt specs, evaluation speeds, and evaluation prices.*\n",
    "- **Make LLM example function**: If changes can be specified as training rules, make an example function for the LLM.\n",
    "- **End**: The end of the process.\n",
    "\n",
    "To summarise:\n",
    "- We work within an iterative process for refining the results of LLM function(s) pipeline.\n",
    "- If the overall results are not satisfactory, we loop back to the outlining workflow stage.\n",
    "- If additional LLM functions are made, we return to the pipeline creation stage.\n",
    "- If prompts or output descriptions are changed, we return the LLM function(s) creation stage. \n",
    "- Our (human) inability or unwillingness to program transformations has a few decision steps for delegation to LLMs.\n",
    "\n",
    "**Remark:** We leave as exercises to the reader to see how the workflows programmed below fit the flowchart above.\n",
    "\n",
    "**Remark:** The mapping of the workflow code below onto the flowchart can be made using LLMs. \n",
    "\n",
    "------\n",
    "\n",
    "## Plot data\n",
    "\n",
    "**Workflow:** Consider a workflow with the following steps:\n",
    "\n",
    "1. Request an LLM to produce in JSON format a dictionary of a certain numerical quantity during a certain year.\n",
    "2. The corresponding LLM function converts the JSON text into Raku data structure.\n",
    "3. Print or summarize obtained data in tabular form.\n",
    "4. A plot is made with the obtained data.\n",
    "\n",
    "Here is a general quantities finder LLM function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "297aed4d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:39.037306Z",
     "start_time": "2023-08-16T14:51:38.984492Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-> **@args, *%args { #`(Block|3658945632944) ... }"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my &qf3 = llm-function(\n",
    "        { \"What are the $^a of $^b in $^c? Give the result as name-number dictionary in JSON format.\" },\n",
    "        llm-evaluator => llm-configuration('openai', temperature => 0.2),\n",
    "        form => sub-parser('JSON'));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f642884a",
   "metadata": {},
   "source": [
    "### Countries GDP\n",
    "\n",
    "Consider finding and plotting the GDP of top 10 largest countries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "24ec3494",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:42.805776Z",
     "start_time": "2023-08-16T14:51:39.048736Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{Brazil => 2.8 trillion USD, Canada => 2.1 trillion USD, China => 25.5 trillion USD, France => 2.9 trillion USD, Germany => 4.2 trillion USD, India => 9.3 trillion USD, Italy => 2.3 trillion USD, Japan => 5.2 trillion USD, United Kingdom => 3.1 trillion USD, United States => 22.5 trillion USD}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my $gdp1 = &qf3('GDP', 'top 10 largest countries', '2022')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db844d0d",
   "metadata": {},
   "source": [
    "Here is a corresponding table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e019fe21",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:42.876938Z",
     "start_time": "2023-08-16T14:51:42.818892Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr>\n",
       "<th>Key</th>\n",
       "<th>Value</th>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Japan</td>\n",
       "<td>5.2 trillion USD</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>India</td>\n",
       "<td>9.3 trillion USD</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Brazil</td>\n",
       "<td>2.8 trillion USD</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>United States</td>\n",
       "<td>22.5 trillion USD</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Italy</td>\n",
       "<td>2.3 trillion USD</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>France</td>\n",
       "<td>2.9 trillion USD</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Canada</td>\n",
       "<td>2.1 trillion USD</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>China</td>\n",
       "<td>25.5 trillion USD</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Germany</td>\n",
       "<td>4.2 trillion USD</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>United Kingdom</td>\n",
       "<td>3.1 trillion USD</td>\n",
       "</tr>\n",
       "</table>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%html\n",
    "$gdp1 ==> to-pretty-table(field-names => <Key Value>) ==> to-html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d93e2374",
   "metadata": {},
   "source": [
    "Here is a plot attempt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2124fe1d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:42.928560Z",
     "start_time": "2023-08-16T14:51:42.890094Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "The second argument is expected to be a Positional with Numeric objects."
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text-list-plot($gdp1.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d197bd9",
   "metadata": {},
   "source": [
    "Here is another one based on the most frequent \"non-compliant\" output form:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "1a531a20",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:43.043556Z",
     "start_time": "2023-08-16T14:51:42.941547Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "+---+----------+-----------+----------+-----------+--------+       \n",
       "|                                                          |       \n",
       "+                                           *              +  25.00\n",
       "|                    *                                     |       \n",
       "+                                                          +  20.00\n",
       "|                                                          |       \n",
       "+                                                          +  15.00\n",
       "|                                                          |       \n",
       "|                                                          |       \n",
       "+        *                                                 +  10.00\n",
       "|                                                          |       \n",
       "+   *                                             *        +   5.00\n",
       "|              *           *    *     *                *   |       \n",
       "+                                                          +   0.00\n",
       "+---+----------+-----------+----------+-----------+--------+       \n",
       "    0.00       2.00        4.00       6.00        8.00             "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text-list-plot($gdp1.values.map({ sub-parser(Numeric).subparse($_).first }))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ccfe9184",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": "(function(element) { require(['d3'], function(d3) {\n\n// set the dimensions and margins of the graph\nvar margin = {\"top\":40,\"left\":40,\"bottom\":40,\"right\":40},\n    width = 600 - margin.left - margin.right,\n    height = 400 - margin.top - margin.bottom;\n\n// append the svg object to the body of the page\nvar svg = d3\n   .select(element.get(0))\n  .append(\"svg\")\n    .attr(\"width\", width + margin.left + margin.right)\n    .attr(\"height\", height + margin.top + margin.bottom)\n    .style(\"background\", \"white\")\n  .append(\"g\")\n    .attr(\"transform\",\n          \"translate(\" + margin.left + \",\" + margin.top + \")\")\n\n// Obtain title\nvar title = \"\"\n\nif ( title.length > 0 ) {\n    svg.append(\"text\")\n        .attr(\"x\", (width / 2))\n        .attr(\"y\", 0 - (margin.top / 2))\n        .attr(\"text-anchor\", \"middle\")\n        .style(\"font-size\", \"16px\")\n        //.style(\"text-decoration\", \"underline\")\n        .text(title);\n}\n\n// Obtain x-axis label\nvar xAxisLabel = \"\"\nvar xAxisLabelFontSize = 12\n\nif ( xAxisLabel.length > 0 ) {\n    svg.append(\"text\")\n        .attr(\"x\", (width / 2))\n        .attr(\"y\", height + margin.bottom - xAxisLabelFontSize/2)\n        .attr(\"text-anchor\", \"middle\")\n        .style(\"font-size\", xAxisLabelFontSize.toString() + \"px\")\n        .text(xAxisLabel);\n}\n\n// Obtain y-axis label\nvar yAxisLabel = \"\"\nvar yAxisLabelFontSize = 12\n\nif ( yAxisLabel.length > 0 ) {\n    svg.append(\"text\")\n        .attr(\"transform\", \"rotate(-90)\")\n        .attr(\"x\", - (height / 2))\n        .attr(\"y\", 0 - margin.left + yAxisLabelFontSize)\n        .attr(\"text-anchor\", \"middle\")\n        .style(\"font-size\", yAxisLabelFontSize.toString() + \"px\")\n        .text(yAxisLabel);\n}\n\n// Obtain data\nvar data = [{\"value\":5.2,\"variable\":1},{\"value\":9.3,\"variable\":2},{\"variable\":3,\"value\":2.8},{\"variable\":4,\"value\":22.5},{\"variable\":5,\"value\":2.3},{\"value\":2.9,\"variable\":6},{\"value\":2.1,\"variable\":7},{\"variable\":8,\"value\":25.5},{\"variable\":9,\"value\":4.2},{\"value\":3.1,\"variable\":10}]\n\nvar valueMin = Math.min.apply(Math, data.map(function(o) { return o.value; }))\nvar valueMax = Math.max.apply(Math, data.map(function(o) { return o.value; }))\n\n// X axis\nvar x = d3.scaleBand()\n  .range([ 0, width ])\n  .domain(data.map(function(d) { return d.variable; }))\n  .padding(0.2);\nsvg.append(\"g\")\n  .attr(\"transform\", \"translate(0,\" + height + \")\")\n  .call(d3.axisBottom(x))\n  .selectAll(\"text\")\n    .attr(\"transform\", \"translate(-10,0)rotate(-45)\")\n    .style(\"text-anchor\", \"end\");\n\n// Add Y axis\nvar y = d3.scaleLinear()\n  .domain([0, valueMax])\n  .range([ height, 0]);\nsvg.append(\"g\")\n  .call(d3.axisLeft(y));\n\n// Bars\nsvg.selectAll(\"mybar\")\n  .data(data)\n  .enter()\n  .append(\"rect\")\n    .attr(\"x\", function(d) { return x(d.variable); })\n    .attr(\"y\", function(d) { return y(d.value); })\n    .attr(\"width\", x.bandwidth())\n    .attr(\"height\", function(d) { return height - y(d.value); })\n    .attr(\"fill\", \"steelblue\")\n\n}) })(element);\n"
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%js\n",
    "js-d3-bar-chart($gdp1.values.map({ sub-parser(Numeric).subparse($_).first}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ede7245",
   "metadata": {},
   "source": [
    "Here we obtain the GDP for all countries and make the corresponding Pareto principle plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "cee51fff",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:54.628353Z",
     "start_time": "2023-08-16T14:51:43.056452Z"
    },
    "incorrectly_encoded_metadata": ", eval=FALSE"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[United States 20.49 trillion, China 13.6 trillion, Japan 4.97 trillion, Germany 3.68 trillion, United Kingdom 2.83 trillion, India 2.73 trillion, France 2.71 trillion, Brazil 2.14 trillion, Italy 1.99 trillion, Canada 1.69 trillion, Russia 1.66 trillion, Korea, South 1.61 trillion, Spain 1.35 trillion, Australia 1.34 trillion, Mexico 1.11 trillion, Indonesia 1.02 trillion, Netherlands 0.9 trillion, Turkey 0.82 trillion, Switzerland 0.77 trillion, Saudi Arabia 0.76 trillion, Sweden 0.58 trillion, Belgium 0.57 trillion, Poland 0.54 trillion, Argentina 0.53 trillion, Austria 0.48 trillion, Norway 0.48 trillion, Thailand 0.47 trillion, United Arab Emirates :]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my $gdp2 = &qf3('GDP', 'top 30 countries', '2018')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20cd542e",
   "metadata": {},
   "source": [
    "Here is a plot attempt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c947edcf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:54.679796Z",
     "start_time": "2023-08-16T14:51:54.642210Z"
    },
    "incorrectly_encoded_metadata": ", eval=FALSE"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "The first argument is expected to be a Positional with Numeric objects, Positional with Str objects, a Map, or Positional of Positionals."
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text-pareto-principle-plot($gdp2.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "500b8c40",
   "metadata": {},
   "source": [
    "Here is another one based on the most frequent \"non-compliant\" output form:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4d9065d8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:54.759106Z",
     "start_time": "2023-08-16T14:51:54.693217Z"
    },
    "incorrectly_encoded_metadata": ", eval=FALSE"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "    0.00      0.19     0.37      0.56      0.74      0.93   \n",
       "+---+---------+--------+---------+---------+---------+-----+      \n",
       "|                                                          |      \n",
       "+                                        * * * * * * * *   +  1.00\n",
       "|                            * * * * * *                   |      \n",
       "|                      * * *                               |      \n",
       "+                  * *                                     +  0.80\n",
       "|               * *                                        |      \n",
       "|           * *                                            |      \n",
       "+         *                                                +  0.60\n",
       "|       *                                                  |      \n",
       "|     *                                                    |      \n",
       "+                                                          +  0.40\n",
       "|   *                                                      |      \n",
       "|                                                          |      \n",
       "+---+---------+--------+---------+---------+---------+-----+      \n",
       "    0.00      5.00     10.00     15.00     20.00     25.00        "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text-pareto-principle-plot($gdp2.grep(* ~~ Numeric).List)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ed2c46c",
   "metadata": {},
   "source": [
    "### Gold medals\n",
    "\n",
    "Here we retrieve data for gold Olympic medal counts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d40cb1a0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:57.546232Z",
     "start_time": "2023-08-16T14:51:54.773407Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{Australia => 14, China => 38, France => 13, Germany => 16, Great Britain => 29, Italy => 8, Japan => 12, Russia => 24, South Korea => 9, United States => 48}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my $gmd = &qf3(\"counts of Olymipic gold medals\", \"countries\", \"the last decade\");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f347fdc2",
   "metadata": {},
   "source": [
    "Here is a corresponding table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "29503d1a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:57.606320Z",
     "start_time": "2023-08-16T14:51:57.560355Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "+---------------+-------+\n",
       "|      Key      | Value |\n",
       "+---------------+-------+\n",
       "| United States |   48  |\n",
       "|  South Korea  |   9   |\n",
       "|     China     |   38  |\n",
       "|    Germany    |   16  |\n",
       "|   Australia   |   14  |\n",
       "|     Russia    |   24  |\n",
       "|     Italy     |   8   |\n",
       "|     Japan     |   12  |\n",
       "|     France    |   13  |\n",
       "| Great Britain |   29  |\n",
       "+---------------+-------+"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "to-pretty-table($gmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2531a40d",
   "metadata": {},
   "source": [
    "Here is a plot attempt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "fb94731c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:57.663341Z",
     "start_time": "2023-08-16T14:51:57.619696Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "+---+----------+-----------+----------+-----------+--------+       \n",
       "+                                                          +  50.00\n",
       "|   *                                                      |       \n",
       "|                                                          |       \n",
       "+              *                                           +  40.00\n",
       "|                                                          |       \n",
       "+                                                          +  30.00\n",
       "|                                                      *   |       \n",
       "|                               *                          |       \n",
       "+                                                          +  20.00\n",
       "|                    *                                     |       \n",
       "|                          *                *     *        |       \n",
       "+        *                            *                    +  10.00\n",
       "|                                                          |       \n",
       "+---+----------+-----------+----------+-----------+--------+       \n",
       "    0.00       2.00        4.00       6.00        8.00             "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text-list-plot($gmd.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13c5377c",
   "metadata": {},
   "source": [
    "### Exercise questions\n",
    "\n",
    "- How does the code in this section maps on the flowchart in the section \"General structure of LLM-based workflows\"?\n",
    "- Come up with other argument values for the three slots of `&qf3` and execute the workflow. \n",
    "\n",
    "-------\n",
    "\n",
    "## Refining and adapting outputs\n",
    "\n",
    "**Workflow:** We want to transform text into a specific format that is both expected and ready for immediate processing.\n",
    "For example:\n",
    "\n",
    "- Remove certain pesky symbols and strings from LLM results\n",
    "- Put a Raku (or JSON) dataset into a tabular data format suitable for immediate rendering\n",
    "- Convert a dataset into a plotting language spec\n",
    "\n",
    "### Normalizing numerical outputs\n",
    "\n",
    "The following *LLM example* function \"normalizes\" outputs that have numerical values with certain number\n",
    "localization or currency units:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "3a7eb815",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:51:57.722474Z",
     "start_time": "2023-08-16T14:51:57.675708Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-> **@args, *%args { #`(Block|3659087034080) ... }"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my &num-norm = llm-example-function(['1,034' => '1_034', '13,003,553' => '13_003_553', '9,323,003,553' => '9_323_003_553',\n",
    "                                     '43 thousand USD' => '23E3', '3.9 thousand' => '3.9E3',\n",
    "                                     '23 million USD' => '23E6', '2.3 million' => '2.3E6',\n",
    "                                     '3.2343 trillion USD' => '3.2343E12', '0.3 trillion' => '0.3E12']);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b421534a",
   "metadata": {},
   "source": [
    "This LLM function can be useful to transform outputs of other LLM functions (before utilizing those outputs further.)\n",
    "\n",
    "Here is an example of normalizing the top 10 countries GDP query output above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "3939684f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:01.907238Z",
     "start_time": "2023-08-16T14:51:57.734598Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       " Japan\t5.2E12 India\t9.3E12 Brazil\t2.8E12 United States\t22.5E12 Italy\t2.3E12 France\t2.9E12 Canada\t2.1E12 China\t25.5E12 Germany\t4.2E12 United Kingdom\t3.1E12"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "&num-norm($gdp1.join(' '))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d887c6f1",
   "metadata": {},
   "source": [
    "### Dataset into tabular format\n",
    "\n",
    "Here is an LLM function that transforms the plain text data above into a GitHub Markdown table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "446e54ee",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:01.959845Z",
     "start_time": "2023-08-16T14:52:01.920456Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-> **@args, *%args { #`(Block|3658975069912) ... }"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my &fgt = llm-function({ \"Transform the plain-text table $_ into a GitHub table.\" })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6f1765d",
   "metadata": {},
   "source": [
    "Here is an example application:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "63fe08d8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:07.542177Z",
     "start_time": "2023-08-16T14:52:01.973298Z"
    },
    "incorrectly_encoded_metadata": ", results=asis"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "\n",
       "| Country        | GDP (in trillions) |\n",
       "| -------------- | ------------------ |\n",
       "| Japan          | 5.2                |\n",
       "| India          | 9.3                |\n",
       "| Brazil         | 2.8                |\n",
       "| United States  | 22.5               |\n",
       "| Italy          | 2.3                |\n",
       "| France         | 2.9                |\n",
       "| Canada         | 2.1                |\n",
       "| China          | 25.5               |\n",
       "| Germany        | 4.2                |\n",
       "| United Kingdom | 3.1                |"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "&fgt(to-pretty-table($gdp1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbda5344",
   "metadata": {},
   "source": [
    "Let us define a function that translates the dataset by converting to JSON format first,\n",
    "and then converting into a GitHub Markdown table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "0ff64830",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:07.596200Z",
     "start_time": "2023-08-16T14:52:07.557138Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-> **@args, *%args { #`(Block|3658975139160) ... }"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my &fjgt = llm-function({ \"Transform the JSON data $_ into a GitHub table.\" })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f29d0044",
   "metadata": {},
   "source": [
    "Here is an example application:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "4cb4bc97",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:10.479626Z",
     "start_time": "2023-08-16T14:52:07.609312Z"
    },
    "incorrectly_encoded_metadata": ", results=asis"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "\n",
       "| Country   | GDP (in Trillion USD) |\n",
       "|-----------|----------------------|\n",
       "| Japan     | 5.2                  |\n",
       "| India     | 9.3                  |\n",
       "| Brazil    | 2.8                  |\n",
       "| United States | 22.5             |\n",
       "| Italy     | 2.3                  |\n",
       "| France    | 2.9                  |\n",
       "| Canada    | 2.1                  |\n",
       "| China     | 25.5                 |\n",
       "| Germany   | 4.2                  |\n",
       "| United Kingdom | 3.1              |"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "&fjgt(to-json($gdp1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "580bb6c7",
   "metadata": {},
   "source": [
    "### Dataset into diagrams\n",
    "\n",
    "Here we define a reformatting function that translates JSON data into Mermaid diagrams:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "b2d5dab4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:10.534051Z",
     "start_time": "2023-08-16T14:52:10.494228Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-> **@args, *%args { #`(Block|3658975180768) ... }"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my &fjmmd = llm-function({ \"Transform the JSON data $^a into a Mermaid $^b spec.\" })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e769f05a",
   "metadata": {},
   "source": [
    "Here we convert the gold medals data into a pie chart:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "90e58c9a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:13.923519Z",
     "start_time": "2023-08-16T14:52:10.547569Z"
    },
    "incorrectly_encoded_metadata": ", output.prompt=NONE, output.lang=mermaid"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "\n",
       "pie \n",
       "    title United States Population\n",
       "    \"United States\" : 48\n",
       "    \"South Korea\" : 9\n",
       "    \"China\" : 38\n",
       "    \"Germany\" : 16\n",
       "    \"Australia\" : 14\n",
       "    \"Russia\" : 24\n",
       "    \"Italy\" : 8\n",
       "    \"Japan\" : 12\n",
       "    \"France\" : 13\n",
       "    \"Great Britain\" : 29"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "&fjmmd(to-json($gmd), 'pie chart')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7ca5b8e",
   "metadata": {},
   "source": [
    "Here is a more \"data creative\" example:\n",
    "\n",
    "1. First we get a dataset and cross-tabulate it\n",
    "2. Then we ask an LLM make the corresponding flow chart, or class-, or state diagram for it\n",
    "\n",
    "Here is a cross-tabulation of the Titanic dataset (over the sex and class variables):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "040de157",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:14.161735Z",
     "start_time": "2023-08-16T14:52:13.938642Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{female => {1st => 144, 2nd => 106, 3rd => 216}, male => {1st => 179, 2nd => 171, 3rd => 493}}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my %ct = cross-tabulate(get-titanic-dataset(), 'passengerSex', 'passengerClass')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fa7ef93",
   "metadata": {},
   "source": [
    "Here we convert the contingency matrix into a flow chart:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "56b7b642",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:17.483540Z",
     "start_time": "2023-08-16T14:52:14.174361Z"
    },
    "incorrectly_encoded_metadata": ", output.prompt=NONE, output.lang=mermaid"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "\n",
       "graph LR;\n",
       "male(Male)-->|3rd|493;\n",
       "male(Male)-->|2nd|171;\n",
       "male(Male)-->|1st|179;\n",
       "female(Female)-->|3rd|216;\n",
       "female(Female)-->|1st|144;\n",
       "female(Female)-->|2nd|106;"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "&fjmmd(to-json(%ct), 'flow chart')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93d86940",
   "metadata": {},
   "source": [
    "Here we convert the contingency matrix into a state diagram :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "4f55d2b6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:21.274469Z",
     "start_time": "2023-08-16T14:52:17.497594Z"
    },
    "incorrectly_encoded_metadata": ", output.prompt=NONE, output.lang=mermaid"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       " \n",
       "\n",
       "stateDiagram\n",
       "    participant male\n",
       "        state 3rd {\n",
       "            493\n",
       "        }\n",
       "        state 2nd {\n",
       "            171\n",
       "        }\n",
       "        state 1st {\n",
       "            179\n",
       "        }\n",
       "    end\n",
       "    participant female\n",
       "        state 3rd {\n",
       "            216\n",
       "        }\n",
       "        state 2nd {\n",
       "            106\n",
       "        }\n",
       "        state 1st {\n",
       "            144\n",
       "        }\n",
       "    end\n",
       "end"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "&fjmmd(to-json(%ct), 'state diagram')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebcb9e3d",
   "metadata": {},
   "source": [
    "### Exercise questions\n",
    "\n",
    "- To which parts of the flowchart above the workflow in this section corresponds to?\n",
    "- What is preferable: one LLM-function with complicated prompt and argument specs, \n",
    "  or several LLM-functions with simply structured prompts and arguments? \n",
    "\n",
    "------\n",
    "\n",
    "## Conversion to Raku objects\n",
    "\n",
    "**Workflow:** We want to retrieve different physical quantities and make corresponding Raku objects.\n",
    "(For further scientific computations with them.)\n",
    "\n",
    "The following LLM example function transforms different kinds of physical quantity specs into Raku code\n",
    "for the module [\"Physics::Units\"](https://raku.land/zef:librasteve/Physics::Unit), [SR1]:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "431f0457",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:21.329344Z",
     "start_time": "2023-08-16T14:52:21.288845Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-> **@args, *%args { #`(Block|3659087102680) ... }"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my &pu = llm-example-function(\n",
    "        ['11,042 m/s' => 'GetUnit(\"11_042 m/s\")',\n",
    "         '4,380,042 J' => 'GetUnit(\"4_380_042 J\")',\n",
    "         '304.342 m/s^2' => 'GetUnit(\"304.342 m/s^2\")'],\n",
    "        llm-evaluator => 'PaLM');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59a6fb65",
   "metadata": {},
   "source": [
    "Here is an example of speed query function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "ae672cf2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:21.382969Z",
     "start_time": "2023-08-16T14:52:21.342626Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-> **@args, *%args { #`(Block|3659087045272) ... }"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my &fs = llm-function({ \"What is the average speed of $^a in the units of $^b?\" }, llm-evaluator => 'PaLM');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c387342e",
   "metadata": {},
   "source": [
    "Here is a concrete query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "a10845d3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:22.237430Z",
     "start_time": "2023-08-16T14:52:21.395951Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10,900 m/s"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my $rs1 = &fs('rocket leaving Earth', 'meters per second');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b4194bf",
   "metadata": {},
   "source": [
    "Here we convert the LLM output into Raku code for making a unit object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "fd318371",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:23.140831Z",
     "start_time": "2023-08-16T14:52:22.252618Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GetUnit(\"10_900 m/s\")"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my $rs2 = &pu($rs1);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45c55f80",
   "metadata": {},
   "source": [
    "Here we evaluate the Raku code (into an object):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "43a211c7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:23.511855Z",
     "start_time": "2023-08-16T14:52:23.137720Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "  Unit.new( factor => 10900, offset => 0, defn => '10_900 m/s', type => Speed,\n",
       "  dims => [1,0,-1,0,0,0,0,0], dmix => (\"s\"=>-1,\"m\"=>1).MixHash, names => ['10_900 m/s'] );\n"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "use MONKEY-SEE-NO-EVAL;\n",
    "my  $uObj = EVAL($rs2);\n",
    "\n",
    "$uObj.raku;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2863b2d9",
   "metadata": {},
   "source": [
    "Of course, the steps above can be combined into one function.\n",
    "In general, though, care should be taken to handle or prevent situations in which function inputs and outputs\n",
    "do not agree with each other.\n",
    "\n",
    "### Exercise questions\n",
    "\n",
    "- Can you write a Raku function that combines the LLM-functions mentioned above?\n",
    "- What kind of computations involve the discussed unit objects?\n",
    "\n",
    "------\n",
    "\n",
    "## Chemical formulas\n",
    "\n",
    "**Workflow:** Assume that we want to:\n",
    "\n",
    "- Obtain a list of Stoichiometry equations according to some criteria\n",
    "- Evaluate the consistency of the equations\n",
    "- Find the molecular masses of the components for each equation\n",
    "- Tabulate the formulas and found component molecular masses\n",
    "\n",
    "Here we define LLM functions for retrieving chemical formulas with specified species:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "d3c31b85",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:23.565432Z",
     "start_time": "2023-08-16T14:52:23.525838Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-> **@args, *%args { #`(Block|3659087127264) ... }"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my &cfn = llm-function(\n",
    "        { \"Give $^a chemical stoichiometry formulas that includes $^b. Give the result as a JSON list.\" },\n",
    "        llm-evaluator => 'OpenAI', form => sub-parser('JSON'));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa037a06",
   "metadata": {},
   "source": [
    "Here is a query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "1638a6fc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:25.381731Z",
     "start_time": "2023-08-16T14:52:23.578911Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[S + O2 = SO2 2S + 3O2 = 2SO3 S + 3O2 = SO3]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my $chemRes1 = &cfn(3, 'sulfur');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db2e9300",
   "metadata": {},
   "source": [
    "Let us convince ourselves that we got a list of strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "0e53c8cb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:25.436595Z",
     "start_time": "2023-08-16T14:52:25.396471Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Vector(Atom((Str)), 3)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deduce-type($chemRes1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c89d9e9",
   "metadata": {},
   "source": [
    "Let us see do we have consistent reaction equations by checking that \n",
    "the molecular masses on Left Hand Sides (LHSs) and Right Hand Side (RHSs) are the same:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "035dc68e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:25.504771Z",
     "start_time": "2023-08-16T14:52:25.447907Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "+--------------------+-----------------+\n",
       "|     balancing      |     formula     |\n",
       "+--------------------+-----------------+\n",
       "|  64.058 => 64.058  |   S + O2 = SO2  |\n",
       "| 160.114 => 160.114 | 2S + 3O2 = 2SO3 |\n",
       "| 128.054 => 80.057  |  S + 3O2 = SO3  |\n",
       "+--------------------+-----------------+"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "to-pretty-table(transpose( %(formula => $chemRes1.Array, balancing => molecular-mass($chemRes1)>>.gist ) ))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5cde1c2",
   "metadata": {},
   "source": [
    "**Remark:** If the column \"balancing\" shows two different numbers separated by \"=>\" that \n",
    "means the LLM hallucinated an inconsistent chemical reaction equation.\n",
    "(Because the LLM does not know, or disregarded for some reason, the \n",
    "[law of conservation of mass](https://en.wikipedia.org/wiki/Conservation_of_mass).) \n",
    "\n",
    "Here we define a regex that parses chemical components:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "c3bd1f05",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:25.559171Z",
     "start_time": "2023-08-16T14:52:25.512478Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "&chem-component"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sub chem-component(Str $x) {\n",
    "  with Chemistry::Stoichiometry::Grammar.parse($x, rule => 'mult-molecule') {\n",
    "    return $_.Str.subst(/^ \\d+/, '') => molecular-mass($_.Str);\n",
    "  }\n",
    "  return Nil;\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e65fc75",
   "metadata": {},
   "source": [
    "Here for each formula we extract the chemical components and find the corresponding molecular masses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "f8677856",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:25.721966Z",
     "start_time": "2023-08-16T14:52:25.570886Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{O2 => 31.998, S => 32.06, SO2 => 64.058, formula => S + O2 = SO2} {O2 => 95.994, S => 64.12, SO3 => 160.114, formula => 2S + 3O2 = 2SO3} {O2 => 95.994, S => 32.06, SO3 => 80.057, formula => S + 3O2 = SO3}]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my @chemData = $chemRes1.map({ [formula => $_, |sub-parser(&chem-component).subparse($_).grep({ $_ ~~ Pair })].Hash });"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a22b87b",
   "metadata": {},
   "source": [
    "Here we find all unique column names (keys) in the obtained dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "bef28982",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:26.058719Z",
     "start_time": "2023-08-16T14:52:26.023231Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[O2 S SO2 SO3 formula]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my @colnames = @chemData>>.keys.flat.unique.sort"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b219a00",
   "metadata": {},
   "source": [
    "Here we tabulate the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "d6ea96b8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:26.118301Z",
     "start_time": "2023-08-16T14:52:26.071820Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "+-----------+-----------+-----------+------------+-----------------+\n",
       "| O2        | S         | SO2       | SO3        | formula         |\n",
       "+-----------+-----------+-----------+------------+-----------------+\n",
       "| 31.998000 | 32.060000 | 64.058000 |            | S + O2 = SO2    |\n",
       "| 95.994000 | 64.120000 |           | 160.114000 | 2S + 3O2 = 2SO3 |\n",
       "| 95.994000 | 32.060000 |           | 80.057000  | S + 3O2 = SO3   |\n",
       "+-----------+-----------+-----------+------------+-----------------+"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "to-pretty-table(@chemData, align => 'l', field-names => @colnames)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6fefc8b",
   "metadata": {},
   "source": [
    "### Alternative workflow and solution\n",
    "\n",
    "Assume that we only wanted to extract the chemical components together with their molecular masses\n",
    "from the LLM generated equations.\n",
    "\n",
    "Then we:\n",
    "- Use the function `chem-component` defined above as a sub-parser in the retrieval LLM-function\n",
    "- Pick `Pair` objects from the LLM function result \n",
    "\n",
    "Here is the LLM function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "57aab4e8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:26.172746Z",
     "start_time": "2023-08-16T14:52:26.131717Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-> **@args, *%args { #`(Block|3659263466160) ... }"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my &cfnp = llm-function(\n",
    "        { \"Give $^a chemical stoichiometry formulas that includes $^b.\" },\n",
    "        llm-evaluator => 'OpenAI', form => sub-parser(&chem-component));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71b7de20",
   "metadata": {},
   "source": [
    "Here is an invocation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "bf870233",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:28.830730Z",
     "start_time": "2023-08-16T14:52:26.186481Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\n",
       "\n",
       "1.  S => 32.06  +  H2 => 4.032  →  H2S => 34.076  \n",
       "2.  H2S => 68.152  +  O2 => 95.994  →  S => 64.12  +  H2O => 36.03  \n",
       "3.  H2S => 34.076  +  Cl2 => 212.70000000000002  →  HCl => 72.91600000000001  +  S2Cl2 => 135.02  \n",
       "4.  H2S => 68.152  +  O2 => 95.994  →  SO2 => 128.116  +  H2O => 36.03]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my $chemRes2 = &cfnp(4, 'sulfur and hydrogen');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d610a8e2",
   "metadata": {},
   "source": [
    "Here we filter result's elements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "b346af20",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:28.881265Z",
     "start_time": "2023-08-16T14:52:28.843739Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(S => 32.06 H2 => 4.032 H2S => 34.076 H2S => 68.152 O2 => 95.994 S => 64.12 H2O => 36.03 H2S => 34.076 Cl2 => 212.70000000000002 HCl => 72.91600000000001 S2Cl2 => 135.02 H2S => 68.152 O2 => 95.994 SO2 => 128.116 H2O => 36.03)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "$chemRes2.grep(* ~~ Pair)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3f313fd",
   "metadata": {},
   "source": [
    "### Exercise questions\n",
    "\n",
    "- What is a good approach to evaluate the ability of LLMs to respect the conservation of mass law?\n",
    "- Is it better for that evaluation to use predominantly Raku code or mostly LLM-functions?\n",
    "\n",
    "------\n",
    "\n",
    "## Making (embedded) Mermaid diagrams\n",
    "\n",
    "**Workflow:** We want to quickly initiate\n",
    "[Mermaid-JS](https://mermaid.js.org)\n",
    "code for specific types of diagrams.\n",
    "\n",
    "Here is an LLM function for generating a Mermaid JS spec:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "873c7f86",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:28.933553Z",
     "start_time": "2023-08-16T14:52:28.894391Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-> **@args, *%args { #`(Block|3659263610240) ... }"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my &fmmd = llm-function({ \"Generate the Mermaid-JS code of a $^a for $^b.\" })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67b2047d",
   "metadata": {},
   "source": [
    "Here we request to get the code of pie chart for the continent sizes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "8d30a049",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:30.579393Z",
     "start_time": "2023-08-16T14:52:28.947192Z"
    },
    "incorrectly_encoded_metadata": ", results=asis"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "\n",
       "pie title Relative continent sizes\n",
       "  \"Africa\" : 8.1\n",
       "  \"South America\" : 6.9\n",
       "  \"North America\" : 5.8\n",
       "  \"Europe\" : 4\n",
       "  \"Asia\" : 17.2\n",
       "  \"Oceania\" : 0.5\n",
       "  \"Antarctica\" : 0.2"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my $mmdRes = &fmmd(\"pie chart\", \"relative continent sizes\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7821898b",
   "metadata": {},
   "source": [
    "Here, \"just in case\", we normalize the numbers of the result and \"dump\" the code as Markdown code cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "b69389a2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:30.636581Z",
     "start_time": "2023-08-16T14:52:30.594399Z"
    },
    "incorrectly_encoded_metadata": ", output.prompt=NONE, output.lang=mermaid"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "\n",
       "pie title Relative continent sizes\n",
       "  \"Africa\" : 8.1\n",
       "  \"South America\" : 6.9\n",
       "  \"North America\" : 5.8\n",
       "  \"Europe\" : 4\n",
       "  \"Asia\" : 17.2\n",
       "  \"Oceania\" : 0.5\n",
       "  \"Antarctica\" : 0.2"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "$mmdRes.subst(:g, '%', '').subst(:g, ',', '').subst(\"{'`' x 3}mermaid\", '').subst(\"{'`' x 3}\", '')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6070edb9",
   "metadata": {},
   "source": [
    "Here is a flow chart request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "173bc3d0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:34.081838Z",
     "start_time": "2023-08-16T14:52:30.649999Z"
    },
    "incorrectly_encoded_metadata": ", results=asis"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "\n",
       "```\n",
       "graph TD\n",
       "A[Wake Up] --> B(Check Traffic & Weather)\n",
       "B --> |No traffic & Good weather| C[Dress & Eat]\n",
       "B --> |Traffic & Good weather| D[Leave Early]\n",
       "B --> |No traffic & Bad weather| E[Leave Early]\n",
       "C --> F[Leave for Work]\n",
       "D --> F\n",
       "E --> F\n",
       "```"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "&fmmd(\"flow chart\", \"going to work in the morning avoiding traffic jams and reacting to weather\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12ef26b4",
   "metadata": {},
   "source": [
    "### Exercise questions\n",
    "\n",
    "- What changes of the code in this section should be made in order to produce Plant-UML specs?\n",
    "\n",
    "------\n",
    "\n",
    "## Named entity recognition\n",
    "\n",
    "**Workflow:** We want to download text from the Web, extract the names of certain types of entities from it,\n",
    "and visualize relationships between them.\n",
    "\n",
    "For example, we might want to extract all album names and their release dates from\n",
    "a biographical web page of a certain music artist, and make a timeline plot."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "054a301d",
   "metadata": {},
   "source": [
    "```raku\n",
    "my &fner = llm-function({\"Extract $^a from the text: $^b . Give the result in a JSON format\"},                     \n",
    "                        llm-evaluator => 'PaLM', \n",
    "                        form => sub-parser('JSON'))\n",
    "```\n",
    "\n",
    "Here is a way to get a biography and discography text data of a music artist from Wikipedia:\n",
    "\n",
    "```raku, eval=FALSE\n",
    "my $url = 'https://en.wikipedia.org/wiki/Sinéad_O%27Connor';\n",
    "my $response = HTTP::Tiny.new.get: $url;            \n",
    "\n",
    "die \"Failed!\\n\" unless $response<success>;\n",
    "say \"response status: $response<status> $response<reason>\";\n",
    "\n",
    "my $text = $response<content>.decode;\n",
    "say \"text size: {$text.chars}\";\n",
    "```\n",
    "\n",
    "But now we have to convert the HTML code into plain text, *and* the text is too large\n",
    "to process all at once with LLMs. (Currently LLMs have ≈ 4096 ± 2048 input tokens limits.)\n",
    "\n",
    "**Remark:** A more completely worked out workflow would have included \n",
    "the breaking up of the text into suitably sized chunks, and combining the LLM processed results.\n",
    "\n",
    "Instead, we are going to ask an LLM to produce artist's bio and discography and then \n",
    "we going to pretend we got it from some repository or encyclopedia.\n",
    "\n",
    "Here we get the text:\n",
    "\n",
    "```raku\n",
    "my $text = llm-function(llm-evaluator => llm-configuration('PaLM', max-tokens=> 500))(\"What is Sinéad O'Connor's bio and discography?\")\n",
    "```\n",
    "\n",
    "Here we do Named Entity Recognition (NER) via the LLM function defined above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "a7535556",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:34.143947Z",
     "start_time": "2023-08-16T14:52:34.097028Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable '$text' is not declared.  Did you mean any of these: 'Text',\n",
       "'&next'?"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my $albRes = &fner('album names and years', $text);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e09cfe95",
   "metadata": {},
   "source": [
    "LLMs can produce NER data in several different structures. \n",
    "Using the function `deduce-type` from \n",
    "[\"Data::TypeSystem\"](https://raku.land/zef:antononcube/Data::TypeSystem), [AAp6],\n",
    "can help required post-processing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "ada9d1ce",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:34.215765Z",
     "start_time": "2023-08-16T14:52:34.156770Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable '$albRes' is not declared.  Perhaps you forgot a 'sub' if this\n",
       "was intended to be part of a signature?"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deduce-type($albRes);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "027303d9",
   "metadata": {},
   "source": [
    "Here are a few data type results based in multiple executions of `&fner` \n",
    "(more comprehensive study is given in the next section):\n",
    "\n",
    "```\n",
    "# Vector((Any), 24)\n",
    "# Tuple([Atom((Str)), Pair(Atom((Str)), Vector(Struct([name, year], [Str, Int]), 7)), Atom((Str))])\n",
    "```\n",
    "\n",
    "Based in our study of the result data type signatures, \n",
    "in this workflow we process result of `&fner` with this code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "e0cc7732",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:34.290768Z",
     "start_time": "2023-08-16T14:52:34.228323Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable '$albRes' is not declared.  Did you mean '$albRes2'?"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my $albRes2 = $albRes.grep({ $_ ~~ Pair }).rotor(2)>>.Hash; \n",
    "if not so $albRes2 { $albRes2 = $albRes.grep(* ~~ Pair).Hash<albums> }\n",
    "\n",
    "say $albRes2;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "676423aa",
   "metadata": {},
   "source": [
    "Here we tabulate the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "a373a943",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:34.368607Z",
     "start_time": "2023-08-16T14:52:34.302441Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Variable '$albRes2' is not declared.  Perhaps you forgot a 'sub' if\n",
       "this was intended to be part of a signature?"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "to-pretty-table($albRes2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9842963",
   "metadata": {},
   "source": [
    "Here we make a Mermaid-JS timeline plot (after we have figured out the structure of LLM's function output):\n",
    "\n",
    "```python, output.lang=mermaid, output.prompt=NONE\n",
    "my @timeline = ['timeline', \"title Sinéad O'Connor's discography\"];\n",
    "for |$albRes2 -> %record {\n",
    "    @timeline.append( \"{%record<year>} : {%record<name>}\");\n",
    "}\n",
    "@timeline.join(\"\\n\\t\");\n",
    "```\n",
    "\n",
    "### Exercise questions\n",
    "\n",
    "- How the LLM-functions pipeline above should be changed in order to produce timeline plots of different wars?\n",
    "- How the Raku code should be changed in order to produce timeline plots with Python? (Instead of Mermaid-JS.) \n",
    "\n",
    "------\n",
    "\n",
    "## Statistics of output data types\n",
    "\n",
    "**Workflow:** We want to see and evaluate the distribution of data types of LLM-function results:\n",
    "1. Make a pipeline of LLM-functions\n",
    "2. Create a list of random inputs \"expected\" by the pipeline\n",
    "   - Or use the same input multiple times.\n",
    "3. Deduce the data type of each output\n",
    "4. Compute descriptive statistics\n",
    "\n",
    "**Remark:** These kind of statistical workflows can be slow and expensive.\n",
    "(With the current line-up of LLM services.)\n",
    "\n",
    "Let us reuse the workflow from the previous section and enhance it with \n",
    "data type outputs finding. More precisely we:\n",
    "1. Generate random music artist names (using an LLM query)\n",
    "2. Retrieve short biography and discography for each music artist\n",
    "3. Extract album-and-release-date data for each artist (with NER-by-LLM)\n",
    "4. Deduce the type for each output, using several different type representations\n",
    "\n",
    "The data types are investigated with the functions `deduce-type` and `record-types` of \n",
    "[\"Data::TypeSystem\"](https://raku.land/zef:antononcube/Data::TypeSystem), [AAp6],\n",
    "and `tally` and `records-summary` of\n",
    "[\"Data::Summarizers\"](https://raku.land/zef:antononcube/Data::Summarizers), [AAp7].\n",
    "\n",
    "Here we define a data retrieval function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "7efab612",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:34.422643Z",
     "start_time": "2023-08-16T14:52:34.381510Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-> **@args, *%args { #`(Block|3659237037064) ... }"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my &fdb = llm-function({\"What is the short biography and discography of the artist $_?\"}, llm-evaluator => llm-configuration('PaLM', max-tokens=> 500));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed719d33",
   "metadata": {},
   "source": [
    "Here we define (again) the NER function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "00e302a5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:34.476806Z",
     "start_time": "2023-08-16T14:52:34.434547Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-> **@args, *%args { #`(Block|3659236989120) ... }"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my &fner = llm-function({\"Extract $^a from the text: $^b . Give the result in a JSON format\"},                     \n",
    "                        llm-evaluator => 'PaLM', \n",
    "                        form => sub-parser('JSON'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1559fb4d",
   "metadata": {},
   "source": [
    "Here we find 10 random music artists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "8de8b217",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:52:36.452705Z",
     "start_time": "2023-08-16T14:52:34.490634Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[```json {name => Taylor Swift} {name => Adele} {name => Ed Sheeran} {name => Justin Bieber} {name => Rihanna} {name => Katy Perry} {name => Lady Gaga} {name => Bruno Mars} {name => Beyoncé} {name => Jay-Z} ```]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my @artistNames = |llm-function(llm-evaluator=>'PaLM')(\"Give 10 random music artist names in a list in JSON format.\",\n",
    "                                  form => sub-parser('JSON'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67eb22ef",
   "metadata": {},
   "source": [
    "Here is a loop that generates the biographies and does NER over them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "ff65ac66",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:53:49.267656Z",
     "start_time": "2023-08-16T14:52:36.465634Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[```json release_dates => [22 March 1963 10 November 1963 10 July 1964 4 December 1964 23 August 1965 3 December 1965 5 August 1966 1 June 1967 22 November 1968 26 May 1969 26 September 1969 8 May 1970] album_names => [Please Please Me With the Beatles A Hard Day's Night Beatles for Sale Help! Rubber Soul Revolver Sgt. Pepper's Lonely Hearts Club Band The White Album Yellow Submarine Abbey Road Let It Be] ```] [```json albums => [{name => Taylor Swift, release_date => 2006} {name => Fearless, release_date => 2008} {name => Speak Now, release_date => 2010} {name => Red, release_date => 2012} {name => 1989, release_date => 2014} {name => Reputation, release_date => 2017} {name => Lover, release_date => 2019} {name => Folklore, release_date => 2020}] ```] [```json albums => [{name => 19, release_date => 2008} {name => 21, release_date => 2011} {name => 25, release_date => 2015} {name => 30, release_date => 2021}] ```] [```json albums => [{name => +, release_date => 2011} {name => x, release_date => 2014} {name => ÷, release_date => 2017} {name => =, release_date => 2021}] ```] [```json Under the Mistletoe => November 2011 Believe => June 2012 Changes => February 2020 Purpose => November 2015 My World => November 2009 My World 2.0 => March 2010 ```] [```json Unapologetic => 2012 Rated R => 2009 A Girl like Me => 2006 Anti => 2016 Talk That Talk => 2011 Music of the Sun => 2005 Loud => 2010 Good Girl Gone Bad => 2007 ```] [```json albums => [{name => Katy Hudson, release_date => 2001} {name => One of the Boys, release_date => 2008} {name => Teenage Dream, release_date => 2010} {name => Prism, release_date => 2013} {name => Witness, release_date => 2017} {name => Smile, release_date => 2020}] ```] [```json Born This Way => 2011 Chromatica => 2020 Cheek to Cheek => 2014 Artpop => 2013 The Fame => 2008 Joanne => 2016 ```] [```json album_names => [Doo-Wops & Hooligans Unorthodox Jukebox 24K Magic] release_dates => [2010 2012 2016] ```] [```json Dangerously in Love => 2003 B'Day => 2006 4 => 2010 Lemonade => 2016 Everything Is Love => 2018 Beyoncé => 2013 I Am... Sasha Fierce => 2008 ```] [```json\n",
       "{ albums : [ release_date => 1996 name => Reasonable Doubt release_date => 1997 name => In My Lifetime, Vol. 1 name => Vol. 2... Hard Knock Life release_date => 1998 name => The Blueprint release_date => 2001 release_date => 2002 name => The Blueprint 2: The Gift & the Curse release_date => 2003 name => The Black Album name => Kingdom Come release_date => 2006 release_date => 2007 name => American Gangster ,\n",
       "    { name The Blueprint 3 release_date : \" 2009] [```json albums => [{name => xx, release_date => 2009} {name => Coexist, release_date => 2012} {name => I See You, release_date => 2017} {name => xx, release_date => 2020}] ```]]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my @dbRes = do for @artistNames -> $a {\n",
    "    #say '=' x 6, \" $a \" , '=' x 6; \n",
    "    my $text = &fdb($a);\n",
    "    #say $text;\n",
    "    #say '-' x 12;\n",
    "    my $recs = &fner('album names and release dates', $text);    \n",
    "    #say $recs;\n",
    "    $recs\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3350c562",
   "metadata": {},
   "source": [
    "Here we call `deduce-type` on each LLM output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "f617d7a0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:53:49.415423Z",
     "start_time": "2023-08-16T14:53:49.279153Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuple([Atom((Str)), Pair(Atom((Str)), Vector(Atom((Str)), 12)), Pair(Atom((Str)), Vector(Atom((Str)), 12)), Atom((Str))])\n",
      "Tuple([Atom((Str)), Pair(Atom((Str)), Vector(Assoc(Atom((Str)), Atom((Str)), 2), 8)), Atom((Str))])\n",
      "Tuple([Atom((Str)), Pair(Atom((Str)), Vector(Assoc(Atom((Str)), Atom((Str)), 2), 4)), Atom((Str))])\n",
      "Tuple([Atom((Str)), Pair(Atom((Str)), Vector(Assoc(Atom((Str)), Atom((Str)), 2), 4)), Atom((Str))])\n",
      "Tuple([Atom((Str)), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Atom((Str))])\n",
      "Tuple([Atom((Str)), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Atom((Str))])\n",
      "Tuple([Atom((Str)), Pair(Atom((Str)), Vector(Assoc(Atom((Str)), Atom((Str)), 2), 6)), Atom((Str))])\n",
      "Tuple([Atom((Str)), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Atom((Str))])\n",
      "Tuple([Atom((Str)), Pair(Atom((Str)), Vector(Atom((Str)), 3)), Pair(Atom((Str)), Vector(Atom((Str)), 3)), Atom((Str))])\n",
      "Tuple([Atom((Str)), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Pair(Atom((Str)), Atom((Str))), Atom((Str))])\n",
      "Vector((Any), 25)\n",
      "Tuple([Atom((Str)), Pair(Atom((Str)), Vector(Assoc(Atom((Str)), Atom((Str)), 2), 4)), Atom((Str))])\n"
     ]
    }
   ],
   "source": [
    ".say for @dbRes.map({ deduce-type($_) })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f16c3c93",
   "metadata": {},
   "source": [
    "Here we redo the type deduction using the adverb `:tally`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "d20692bf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:53:49.558423Z",
     "start_time": "2023-08-16T14:53:49.425727Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuple([Atom((Str)) => 2, Pair(Atom((Str)), Vector(Atom((Str)), 12)) => 2], 4)\n",
      "Tuple([Atom((Str)) => 2, Pair(Atom((Str)), Vector(Assoc(Atom((Str)), Atom((Str)), 2), 8)) => 1], 3)\n",
      "Tuple([Atom((Str)) => 2, Pair(Atom((Str)), Vector(Assoc(Atom((Str)), Atom((Str)), 2), 4)) => 1], 3)\n",
      "Tuple([Atom((Str)) => 2, Pair(Atom((Str)), Vector(Assoc(Atom((Str)), Atom((Str)), 2), 4)) => 1], 3)\n",
      "Tuple([Atom((Str)) => 2, Pair(Atom((Str)), Atom((Str))) => 6], 8)\n",
      "Tuple([Atom((Str)) => 2, Pair(Atom((Str)), Atom((Str))) => 8], 10)\n",
      "Tuple([Atom((Str)) => 2, Pair(Atom((Str)), Vector(Assoc(Atom((Str)), Atom((Str)), 2), 6)) => 1], 3)\n",
      "Tuple([Atom((Str)) => 2, Pair(Atom((Str)), Atom((Str))) => 6], 8)\n",
      "Tuple([Atom((Str)) => 2, Pair(Atom((Str)), Vector(Atom((Str)), 3)) => 2], 4)\n",
      "Tuple([Atom((Str)) => 2, Pair(Atom((Str)), Atom((Str))) => 7], 9)\n",
      "Tuple([Atom((Int)) => 1, Atom((Str)) => 8, Pair(Atom((Str)), Atom((Str))) => 16], 25)\n",
      "Tuple([Atom((Str)) => 2, Pair(Atom((Str)), Vector(Assoc(Atom((Str)), Atom((Str)), 2), 4)) => 1], 3)\n"
     ]
    }
   ],
   "source": [
    ".say for @dbRes.map({ deduce-type($_):tally })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3aba3e8",
   "metadata": {},
   "source": [
    "We see that the LLM outputs produce lists of `Pair` objects \"surrounded\" by strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "66135d0e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:53:49.690722Z",
     "start_time": "2023-08-16T14:53:49.568098Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "((Str) (Pair) (Pair) (Str))\n",
      "((Str) (Pair) (Str))\n",
      "((Str) (Pair) (Str))\n",
      "((Str) (Pair) (Str))\n",
      "((Str) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Str))\n",
      "((Str) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Str))\n",
      "((Str) (Pair) (Str))\n",
      "((Str) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Str))\n",
      "((Str) (Pair) (Pair) (Str))\n",
      "((Str) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Str))\n",
      "((Str) (Str) (Str) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Pair) (Str) (Str) (Str) (Str) (Str) (Int))\n",
      "((Str) (Pair) (Str))\n"
     ]
    }
   ],
   "source": [
    ".say for @dbRes.map({record-types($_)})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "938d418d",
   "metadata": {},
   "source": [
    "Another tallying call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "55b3aad0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-08-16T14:53:49.824910Z",
     "start_time": "2023-08-16T14:53:49.699741Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{Pair => 2, Str => 2}\n",
      "{Pair => 1, Str => 2}\n",
      "{Pair => 1, Str => 2}\n",
      "{Pair => 1, Str => 2}\n",
      "{Pair => 6, Str => 2}\n",
      "{Pair => 8, Str => 2}\n",
      "{Pair => 1, Str => 2}\n",
      "{Pair => 6, Str => 2}\n",
      "{Pair => 2, Str => 2}\n",
      "{Pair => 7, Str => 2}\n",
      "{Int => 1, Pair => 16, Str => 8}\n",
      "{Pair => 1, Str => 2}\n"
     ]
    }
   ],
   "source": [
    ".say for @dbRes.map({record-types($_).map(*.^name).&tally})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fac89cd",
   "metadata": {},
   "source": [
    "The statistics show that most likely the output we get from the execution of the LLM-functions pipeline\n",
    "is a list of a few strings and 4-12 `Pair` objects. Hence, we might decide to use the transformation\n",
    "`.grep({ $_ ~~ Pair }).rotor(2)` (as in the previous section.)\n",
    "\n",
    "------\n",
    "\n",
    "## Other workflows\n",
    "\n",
    "In the future other workflows are going to be described:\n",
    "\n",
    "- Interactive building of grammars\n",
    "- Using LLM-based code writing assistants\n",
    "- Test suite generation via Gherkin specifications\n",
    "  - Here is a [teaser](https://github.com/antononcube/Raku-LLM-Functions/blob/main/docs/Convert-tests-into-Gherkin-specs_woven.md).\n",
    "- (Reliable) code generation from help pages\n",
    "\n",
    "Most likely all of the listed workflow would use chat objects and engineered prompts."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d99ef359",
   "metadata": {},
   "source": [
    "------\n",
    "\n",
    "## References\n",
    "\n",
    "### Articles\n",
    "\n",
    "[AA1] Anton Antonov,\n",
    "[\"Generating documents via templates and LLMs\"](https://rakuforprediction.wordpress.com/2023/07/11/generating-documents-via-templates-and-llms/),\n",
    "(2023),\n",
    "[RakuForPrediction at WordPress](https://rakuforprediction.wordpress.com).\n",
    "\n",
    "[AA2] Anton Antonov,\n",
    "[\"Connecting Mathematica and Raku\"](https://rakuforprediction.wordpress.com/2021/12/30/connecting-mathematica-and-raku/),\n",
    "(2021),\n",
    "[RakuForPrediction at WordPress](https://rakuforprediction.wordpress.com).\n",
    "\n",
    "[SW1] Stephen Wolfram,\n",
    "[\"The New World of LLM Functions: Integrating LLM Technology into the Wolfram Language\"](https://writings.stephenwolfram.com/2023/05/the-new-world-of-llm-functions-integrating-llm-technology-into-the-wolfram-language/),\n",
    "(2023),\n",
    "[Stephen Wolfram Writings](https://writings.stephenwolfram.com).\n",
    "\n",
    "### Repositories, sites\n",
    "\n",
    "[WRIr1] Wolfram Research, Inc.\n",
    "[Wolfram Prompt Repository](https://resources.wolframcloud.com/PromptRepository/).\n",
    "\n",
    "### Packages, paclets\n",
    "\n",
    "[AAp1] Anton Antonov,\n",
    "[LLM::Functions Raku package](https://github.com/antononcube/Raku-LLM-Functions),\n",
    "(2023),\n",
    "[GitHub/antononcube](https://github.com/antononcube).\n",
    "\n",
    "[AAp2] Anton Antonov,\n",
    "[WWW::OpenAI Raku package](https://github.com/antononcube/Raku-WWW-OpenAI),\n",
    "(2023),\n",
    "[GitHub/antononcube](https://github.com/antononcube).\n",
    "\n",
    "[AAp3] Anton Antonov,\n",
    "[WWW::PaLM Raku package](https://github.com/antononcube/Raku-WWW-PaLM),\n",
    "(2023),\n",
    "[GitHub/antononcube](https://github.com/antononcube).\n",
    "\n",
    "[AAp4] Anton Antonov,\n",
    "[Text::SubParsers Raku package](https://github.com/antononcube/Raku-Text-SubParsers),\n",
    "(2023),\n",
    "[GitHub/antononcube](https://github.com/antononcube).\n",
    "\n",
    "[AAp5] Anton Antonov,\n",
    "[Text::CodeProcessing Raku package](https://github.com/antononcube/Raku-Text-CodeProcessing),\n",
    "(2021),\n",
    "[GitHub/antononcube](https://github.com/antononcube).\n",
    "\n",
    "[AAp6] Anton Antonov,\n",
    "[Data::TypeSystem Raku package](https://github.com/antononcube/Raku-Data-TypeSystem),\n",
    "(2023),\n",
    "[GitHub/antononcube](https://github.com/antononcube).\n",
    "\n",
    "[AAp7] Anton Antonov,\n",
    "[Data::Summarizers Raku package](https://github.com/antononcube/Raku-Data-Summarizers),\n",
    "(2021-2023),\n",
    "[GitHub/antononcube](https://github.com/antononcube).\n",
    "\n",
    "[MSp1] Matthew Stuckwisch,\n",
    "[Intl::Token::Number Raku package](https://github.com/alabamenhu/IntlTokenNumber),\n",
    "(2021),\n",
    "[GitHub/alabamenhu](https://github.com/alabamenhu).\n",
    "\n",
    "[MSp2] Matthew Stuckwisch,\n",
    "[Polyglot::Regexen Raku package](https://github.com/alabamenhu/PolyglotRegexen),\n",
    "(2022),\n",
    "[GitHub/alabamenhu](https://github.com/alabamenhu).\n",
    "\n",
    "[SR1] Steve Roe,\n",
    "[Physics::Unit Raku package](https://github.com/librasteve/raku-Physics-Unit),\n",
    "(2020-2023),\n",
    "[GitHub/librasteve](https://github.com/librasteve).\n",
    "\n",
    "[WRIp1] Wolfram Research, Inc.,\n",
    "[LLMFunctions WL paclet](https://resources.wolframcloud.com/PacletRepository/resources/Wolfram/LLMFunctions/),\n",
    "(2023),\n",
    "[Wolfram Language Paclet Repository](https://resources.wolframcloud.com/PacletRepository/)."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "incorrectly_encoded_metadata,-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  },
  "kernelspec": {
   "display_name": "RakuChatbook",
   "language": "raku",
   "name": "raku"
  },
  "language_info": {
   "file_extension": ".raku",
   "mimetype": "text/plain",
   "name": "raku",
   "version": "6.d"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}