10_Scaling_up.ipynb

{"nbformat":4,"nbformat_minor":2,"metadata":{"colab":{"name":"10_Scaling_up.ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","source":["<center><a href=\"https://colab.research.google.com/github/AlisonJD/RTACourse/blob/main/10_Scaling_up.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a></center>"],"metadata":{"id":"view-in-github"}},{"cell_type":"markdown","source":["![tb_Logo Navbar.jpg](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEATgBOAAD/4QBWRXhpZgAATU0AKgAAAAgABAEaAAUAAAABAAAAPgEbAAUAAAABAAAARgEoAAMAAAABAAIAAAITAAMAAAABAAEAAAAAAAAAAABOAAAAAQAAAE4AAAAB/9sAhAACAgICAgICAgICAwMCAwMEAwMDAwQGBAQEBAQGCQUGBQUGBQkICQcHBwkIDgsJCQsOEA0MDRATERETGBcYHx8qAQICAgICAgICAgIDAwIDAwQDAwMDBAYEBAQEBAYJBQYFBQYFCQgJBwcHCQgOCwkJCw4QDQwNEBMRERMYFxgfHyr/wgARCAB4AZADASIAAhEBAxEB/8QAHgABAAICAgMBAAAAAAAAAAAAAAkKBwgFBgEDBAL/2gAIAQEAAAAAn8AAAAAAAAAPX5/YAAAAAAAI49S9jtldrPIAAAAAADjILMbSL7zdyAaT7WdsAD1+v6Brb2DOQH5+P7gYsiEi84C5f5DgYRZsez1UZvN8gAhq0xsxiAnJk1QEfkBNvk6bHFGdpd6G61oIEdVbywJJ/pHtH1rGnK68yCcPpnIZ+9QO2R+6Yza4BkI7JqjzWZNKc/4m3g0rxr+oIrfOgMVOh3GBMhOQCESGKTWfSp9PNhaHfaPA3drN9Oq1PsxT9nuwBEntXi7gbYFd3KklNWbaaQrmoU5D9U+kW+eHjrjH0Y+IWSZDwdApzXCMoVD55sLRcWutU6ql1Kt7tJvvWEuKw+RtWueEpuWcYpsqSU1VLk/bqlMuksUUsMFvkY7hqjHxwuA5XB0CnNcIyhUPnmwtHVaI15qW3So+oZd+MT2DYatK7NKnNYajXypJTVbunKe8+sicecCVvkIgtfOlcdZGA6fTXsqSBVEZ5sLR1WiNealt0r107e12Id2IaoZLM2C6+VvODDKklNVu6crX8dPJBHq/b5GNITG6sjv2AIL9CbPUAEvmLdJp+sMVyrSf016dGLff6jD1Lxfr3MRJ5CjkHeqvbaVYer5Y1kn09spiOXVKULYUAAA00gd3tnAAAAHrjkkK5UAAAQH8/NzzAAAA8eQAAAAAAAAAAAAAAAP/xAAaAQEAAwEBAQAAAAAAAAAAAAAAAwQFAgEG/9oACAECEAAAAAAAAAt28zg758ASRglv6WZmJNKtB4RS6uXFfp+82LEe12+dqp9erxakp1r0tObv2ner+Zfu3Z+b8dbNflZyd/G08nTe4m7nzZkujmxi/Dy7g3MC1NTmVbMsNeSMAJkIAAAAAf/EABgBAQADAQAAAAAAAAAAAAAAAAABAgME/9oACAEDEAAAAAAAAApTWSJAIkEZZ66oyvMkxjrbLQrWcIdN1cbTWL2zi8QvnZs569QwsV1w2y2yN+fSNYy1kZ2Ith0Ui9V6Ra0SAFVgAAAAA//EAD8QAAEEAgECAgcEBwYHAQAAAAUDBAYHAQIIAAkRExASFCA3OHYVFiF1MDU2QHN0dxcxNERQVxgZIiUyM0Fg/9oACAEBAAESAP8AWSBUYHbauSpNoxbbKao6r6b6K6aKJ767p7Yxtrt/qHNSwsE5CHrkev4sxGmCBLWM2BN4ZvjeLSsmNx4+OUorzPs4PlNKRshh9tj/AMt4hzKrg+qg0PCyoV8ptrprho6RfNW7xDzPIW01VT/02WyQfDowclBTb1WAtmq7VxIzxCUnzEjKq+YRJO1njjaQT6Hxf1tTR9qgvjw8W9NcRmEwi0VnUykLlIabGMzDYRC6rr6v0tNYpFmTJfGvq5efoLZ7hHHSnZo7ghkiaMHmC+WpTFb2TCbeh4qd1+cSLRh/jbKDn9G9JjBeE8kCDVp5nj5eWZ0A9W1bMDLBw528fBL3L+5d0vxtUHMbAKvV5A/R9pagqC5MVLyODECVbGlt3g7bTUmI9929ZD0cuHztFs38ca+YhI40opokjIRqiym2NNNPes2/6YpptutZNjBQq+unr6sOVncjidggPuJUcbJrDMvdF35eQ2xPJJ6+js4q3abf5aExp1NZpEoey9b208aYB0Mjh7QUwZDWSWqTNo3Sat0/dn0iWh0FmktbNk3C4MCRLpN/+b7Yn+zsc6rWVrzmuYDNnbRNq6kMbFHFm3V/8Yrvrm1pWHKQaQFm70w7cijXbjpee0vRZBrYbJcYYkR9Y8gE/R94f9V0B/NSbrtpfN3Avy077vcu48204u8jbYSMFT8IODhyWrntbUDaURm0styXgCUfiziOqgRzb3+6J8qhP6mDdcfPj5SH1/Gvcn9rVrVY77TsScBY8zzrnZPNr92GpY37UwqiKFJeR1/BMhbHcE5NWt7S0zNfusBW8cfZjl05euF3jxwqu6W32UWW9Hb1hX315YVnqql67EHu8kbr3+R3PKiYuyuSlSmkjzNERBeOb56qfuV8b4ZVlaQ8zpK/tgFFQwZ7mJSoXNInF5mFwt9jnxDI0wxMO5TxwhEtlMLN6Sv7ZAF3wR/1SN2wu/4PpYECwQwA2fOB+Or95KVvxqER0zY2pb2M07WZM+qZ53UbfE8YVxBdZFiRPGzpyjm7ubXHygiK8elcnXIy1H8VwIPuw8ciL9NmTjU4Ett9vDD2u7NgNsxtvLa5lTA8AWz6ntNx23GaPgJSx5k1Jqxscq3SeZb90vi2quij4y1LCm+umVdN9FdNFE99d099cba7dXNzvoeiZ4/riarHVZIybtnDrSheXlT8kDZwFWqR/dwHZaP36/eH/VdAfzUm67aXzdwL8tO9Xhzh4/UETXjcpkTorL0MYy5Ax3uu8byz/RiYATUI1328Pb4FYkHtGOtZbXsoHno648ddHnV583KW48zNCB2BrIMnVRiBbTqgOXNSckikiEVxqbw6CtEXj7No2PHahgMkseWYdfdwGho4eYGd0DjIVJDxbTSX+1PHKTRHq+OWFL8ckmqdiH1dj7pLz2keoHuA1XyGsNvWsXicrGmnLR29Qc9Wz3DeNNSknYFaSPpOea77JOWUR7qvGuQkkRxsfLo0kpvjX2/uRyIDLeHWZJGDDMqAIHwjhkQ4+fHykPr+NejkF3E6yo0q8izaEyo7MUMb41a2t3KeSVie1MY6YZwcEr464bmTZmRkXJiQF3pQu429dw+93tBwr2mXXBYiyP4DxLCPNVfe5V9vO05ZPbqvVjM4olG1diEmww6hfa2uWbw2JTYdP4WgPkAVgbaoVVFnsDrCuIMTcILkY5Fg4J2vyQ+Ye+v6iynrtbfKw1+qjHXd9+H1N/UJHqlbfO0fL3k5i+mPvHgEUFDnFddunlDcAfSbvUBIBIt4kElr64uXDxxfMEbIApain++yQ83wnv8AM0NeEYd4fqawmQvWwOTsrOgYuza6m1eF8a4HSEM8FqKSAETjB43GjTbLcyJfuRj5DhTZuLU4y1dIXDjzSrAZiPFMrroNUF3TlXRJuinsqqreNiLWzb9jWMtvvsmdPO3jTHa9qbMD4+bzd+28s1OyW5Tx7w/6roD+ak3VM22epKZ7z2LJ65kSQcqNHr1R27uRt2g0rAIrio6NMeJBu65BcMLt44M0TczGMSMRVW1b4kHFzkhK+NllDZSJdLrRR2qk2k4QKXGHxAo+GeJuhBFkg/Yue638zYn6FFddoH9vbm/IBnXPH5R7r/KWvQIlgKcDGMo+dhi+bvMpA6h5H81p/NLKARdwT3JlFVyJnhbwgviheQAOdzocG2jCQko0WddzvlOYhyDKgIAVUZkyrDD+Wv6+ryZ2nKxcIgIByYk5DfOrZnr2m+RWQvt+ZTBsFvL8z7Klsgt6loRYXFmyArtkzdFhxtMfx8+PlIfX8a9EthMOnwlQFN4sKPBt/wAdmVt9rKiJl7SQrsmUgphT1ttEbZ7cXJasvanwmOITUAl622HhAaREPXI0swcMiLffKa7X3O1zC/uxxhRkSiXg5lcjJF8b+9d3wXt76KP+jj18AqO/p/GfRyQ+Ye+v6iynrtbfKw1+qjHXd9+H1N/UJHrhhCxFgcoacjB1to5EqGt37hv1zvi4uV8U7fQJN9FNhgrU2zUxnOM421z4bYz44yCcqPQYZ4r/AO1di3W367mlT5r3ka+lTJt5YOcsEzyWe0RZf4WrT7xx/wDW0uFpc6bKxVnF+zCbZx5RYwyxFxnVeQorY87iECCa+JWQF2YlvmLR0VDo1HogBQwgECDGopkn3h/1XQH81JuuGlZjra5J1fDzbbRwB+0FSpNDGMYxjGMeGMf3YncLBWLDZPBJM01cgTo1wNeJSMI6jMhOxt94e2iiLoa467eEwcy3iZW+XyuyjwNl+A337rfzNifoUV12gf29ub8gGdc8flHuv8pa9J6bKqaJ6fjvvtjXXFZ1/H6rr6J19G2abcMCGoMUteuV0vcznkjdchcq5UxtLSLBvt2lawGDKznFtuWmmx84a3AtF+u7PWIwtVcLtdBppg+AN6BnLjj58fKQ+v417tk0pU9uD8j7KgAU9pjTy0nHL/gHVFYtw5asJSUGkizpXVIBI6nnUY9dR4EUcs9f816OPsN/s5o2poTlLy3QuLDEXuvvXd8F7e+ij/o49fAKjv6fxn0ckPmHvr+osp67W3ysNfqox13ffh9Tf1CR67efziU1/HNejmP8rd6/SL/0Rb9mI5+Vs+u5/U2J3x6xM2Db1zkFJaFsZ4TWX/ZXyaq4+u48oSQJYjpTru72Z55SrKfZuP8AoaoOJYUS7U9Tfe26j1nv23rioQKzq0367w/6roD+ak3XbaMNBPLiAJPN9dMEWJkejt6LYMNJDadlH2G2uzElKjJBvt2vmC7PikGcLYzhN9IzTpDrut/M2J+hRXXaB/b25vyAZ1zx+Ue6/wApa9Mv8Y0/jJ+m72DgVdFujHWudXLSaHm6uO1oaZP+LurFDfXLkVLCzNzp13SC7QZxZcsF99cLlZQIZNccfPj5SH1/Gve5Nz3792say2W9cQF/7Kx9EjruGyrzNjAJvu62/vdVbwgN2JP45mJKLu4mzMMVj+fftga6NVZZYdilso9fRUyzbp9cZ+bPHPFDVmLlNli4/JI/GRoIqMbroukEHTZXVRusnqqkpyQ+Ye+v6iynrtbfKw1+qjHXd9+H1N/UJHrt5/OJTX8c16OY/wArd6/SL/0Rb9mI5+Vs+pPGRUui8iiZ5DC4Y0Ndin6U+h5msp9LIOWzsmajhl2LVUvW4j98WQUseSaeUReMxzTyO3jVOKr4zRJy9beUfmG+8tf+jvD/AKroD+ak3UKlh6By2OTeML5QPASLYsxWpXnpx2tSLjyRieh4bKvI0wTCcwu4PWMYgEgg9LyxrJbANM1h2CYwa/Mkh4gUzVdlHzlJmza8fKz1pylK4rTx09qCBkUiCndb+ZsT9Ciuu0D+3tzfkAzrnj8o91/lLXpl/jGn8ZP09ymoH1ccizMtRabaxidJanWK/A/lsz41TQwImabhasJP5GpTfXl5xg2C/b+L2hnsHled5fPLlsx5KS4QDgyTnSr4t52zFbj58fKQ+v417tzzrWuq1k0l0U9Qhq3yzGY222322332ztttnxztppvvvqnprnbfbONddaY4jGJL7JIbK1XFAc+CqIgDHgcYFNQkdGtx4dtr6qTf9Byy7aU50l52fcfmLYtGyjlUg5iqXDzlGq9ww0omYYX9b1fWoZpMB1L1cMsAZuPmI+MjRxdpePCnlHKLqt+TAaiJOwRebSAmNd9v2sp1UXHxCIWNHVwkj0kJN5sy7lNJWldMLrAZV8PcH34w0+cvkuF3EbkZWnJqsJvOaufiYsMVKbPn/XJyKSGb8frbiMUG7v5EVjjxkOY/8BfLn/ZQp1H0FmYEI1cp50cID2ySunXdWq1GIXiBsVglomwnAf13GKHrJ1cdxV5WrbXfyzhlBu93aNWo9o1YMkNEGTZHRBuj13MaKti7R1No1XDHUg3DuD2xLHDXhXcYC5c5uupXbOuCUaNhiql+dtq8K2PEHVahXE4gW6u+7FwA4d8opK/0GsKLl6C+2/qeZwz7ezekirG0bZeMitkIa+uIF9dxDjDfNxXyOlla1y9OR5OJDx+7ztqcerlpSX2iStCCuwDIkHYNmKvL2GyaxON9qQuFiVCknKDm6LBg14Hct03TffelimNNVdM7Z9F+0FB+RVfPYDNkN9Mevh0LKW127eS1aFHaYeHqzOO4329lLRLhdyjmRNAYwpSTMdlNsY2cSTt8yGtOI0zisRHYl95yYiFWKq0xwm5Sxq4aokZyoCTQKKmIIkQde5zYnv2jIgNesl/FsKS+0iGsBriX2WY1CxIVu6Vx6uXDinONcOq7RsZJY0NTLGuNsv8A91tTn1x+p6wZNWsycnsSIFht7Vnl/wAlnnJ61NpagwXGxAY0wKjg3tOUm7cG5lfRhlnQezb7RqOb/uxsuxj4cqdJK+WPHtFnrneE8d5fc0oJ2TZmVwgIs+UIYaRaKRuFB24GLiEBwxHHjhH92uvtw8pJ1aM6niL+IlsSI6+L4VqbtLTNyVaP7qnIoeAT3xusKiENjMAiwWFw8QgKjAltq0Ysf3ZZFFdPKS6WiiWc4znT/wDQ/wD/xABHEAACAgAEAgQKBQkFCQAAAAABAgMEAAUREhMhEDFBYRQgIjJRYnKChIUGIzBSkxUkQFBxc4GSsmODoaLTM0JDU2B0lKPD/9oACAEBABM/AP1zbmSCNpH81FaQgansGFOqkHqII/WKnkbthdIkPfHEdffxWsOsLe3ESUb+IxJH4FZPv1/I/wAmOF4fBublor1xxD/JiSN4n2tzGqSAMv7CP1aDoX4Y1WNfWc8h347N8zFtq+hV6lHYMI3Gn58geFFubFGIRTrXuxLPGtiaXXa+1hvQLgpxbjj155d0nP0a6fY/R6mlqDL515PFPLNLEGkTqdY923EQZGV08l4po3AaORDyZGAP2lmZIt5Xr03ka6YgsRySNpzJCqSfFyWuLeYNX1Kcd1d44449RopdhuxmMPguY0+Lrw2ki1ZWRtOToWX7CeRY01PUNW0GEtxMzM3IAANzJ8eSfjZjIvpjpwb53/guM5C04rSwc40igjLuY93leWUOKH5tGO4lPKYe0Tgczxb0611/xfC9SQwqERR3ADxpCVSV6Nd51jYjmFYrocflCziJi8cD5jVSy0aluZVS+g6MvoT3q+aw2JTJHKksKsOKwPlx+crYs8p6FV4Iq6cdP+HNLwtzJ9p7lLHy+XxcrryXPyZPRgWq9edIQzRK+ziK/mktjNIHqT5nNYsQ2DMkMujcGEQ+f9h774+Yw+JmdtIZZdOyCInfK3qoCcXtcoyv2hxVad/ZKJj6Ioct8n0Na1ay3eOJtxM5kkkduZZ2bUkntJ6fu/k+Bmgb8dk8eHLkemLtqo0SaS8XzNX6K+Vo8XhNCpHXl2Nxhqu5MWECTeDX4VsxcRQTtfa41GK+VpJD4VQmatLw2Mw1XcmMygFafjVtN/kKz8vKxlVVbT8WBBI3EDOmmMxy9K8GypGZX1dZWx9HYBfuwd05LJDC3qO4bFmhUmgjX0utWy8n8qnFJ9TFJoGMU8baPDKAeaOAwxlNdbM8K2pRAkhRmTyN7AE4lylAia9r6TYU6qynmCCOjKaKWoYRaQTRoztInl7CGxmNAVa8aSvwkTfvby37Bj3KWPl8uPo3AL1uv2hZ3do4YW9R3D4uZfWngRfTIKdiWX+VTjL5eIFcczHKvJopF7UcBh0ZZQSzD4PYd4k1dpE56xnGa01qrwp3MabCjvqdRilEJp9kkqwjYhK7jucYfKYwnEmcIup42MohFzNJYericPciRIex5GUHGaw0xV4VRN7BzDYdlbo+iVdLywSLyKPZleKuWHaFclcZpl8M9RO9zQmnl/yYoTLPXniZ30ZHTHzGHot0J8gohkOnObMY1kZfQ8cTqcfRuLS6U9e9PvlD98WzGYWJLVmVvS8spZmP7T4x7TmMxtTBfZFZPHnkt+HCtVg4zRkLAU4hCdFmW8J44MwgWyiSbK5G9Q+KxYwSTZdUjrO8ZcKdjFNV1AOmPmc2Pw8fDLgkDwOxmMBrLbAIIZod25V7Tj6ZX54b93j/AFnHeOCGzKpk9MujHGWTeF5bZkQbjEsuiskmnPZIqsRgt9TJVsvwktFeySq7b1b0argjXgtYjKpOPWibR17xhuuKzUkMMqH2WUjBOr+F5OfA98nrSoiye9hztVEQbmZieoAYfzkpb+HVjP7uFVXDDRxllLWrTT+uVe58e5Sw508EnzKq9RbQGh3NDv3qvaRj6WWp/D8xFj6zwngwxzPpL17pdpbGQTPaoxzP5kc4lSKSEt2F12nCv9TmFDXRiE6hPECWhfsOIuaTVrSCWKRe51IIx8Vax8Q+PjoMBtm8QSCTbrz01068Xp0qZZUL+ZUjnsFQ4gTRFij1ZVxluZJYaOWxDsjBjYI+Kr7JoqE+qw5ejrzUz6FpvUxWA81Rq8kjsQscaDm7sQowb1zf+74orcPfi5JqKFyqxbwmmybkkgtoeZQ7SQDj5jD0ZvTiuQ6/eCzKwDehhzGK7HNMq399aywlX3JQuPorIbFkIPv0pAk+/ujV8W4mhnicdaukgDKe4+K3WYKpXLUH7Na7Hx/gZej5dD0fM5sfh4+GXEo3RzDKq8mYiNx2o5h0YdBHlQWsvlWdHQ9hIBU9x6PWkjDHC8oxfi/Nrqe0WUSt7eNfZo3f/jgHRzPnB8HkKH7yQl3GNNwRrUojMjeqgO5u4Y+5WpxLDGv7dq49yliQbo5q2VQPeMMg7UmaII3Qw1+rnXbvX0Oh8pD2MAcDkOLUlaF/8Vx/ZUbLiBfchZFx8Vax8Q+PjoMd5xGoXiyIv1k8npkmfV3Pax6CdfzTLJTQr/8AqhXDDV4sty5EkYIewSzSHf7A6ANHkyzMkdtrnt4cyLs9o4+Yw+LdrDwuEf2NmPbNF7jDGasL1RIIU1d4p/JmQKzKPL34ofnMWnpO3ylHewHT1aXHhWWyf4ysx8f4GXo+XQ9HzObH4ePhlx8ot9Hu9H90uFGrnLLmlW4n9ErdyYJ0Q1c4Hgm6T1YpHWT3cevOTSp/xULLhl5flXNw1eL+SAS9HuUsN1cZ6Msqj3tnSvU0Nm3JKhGnc2PSiyiv/VFj4q1j4h8fHQY97pb70d6VTjtDyiK0v+WXo7WkjL2/6ITj5jD4wOqnwcnjOPalLc+0adEI4NjX0mSPQt/HUYzGPSGCgsytOvHXRWkMeuyPTVvsEG5nlnqSRooHaST0ZqXhnWfK66VTJFy0lSXZvXZhDqro43KwPoIx8zmx+Hj4ZcfKLfR7vR/dLj79a5G0Mi8+0q2F1QmSnKYxMnc+m5DhW3pEKNWOsdndIyFz3thho/DvKFpr/wCOiN0e5Sxt3Ks1WQSLvHahI0Ydox9J7seXivY08vgWLJSKeP7jKdcZO/Gy/KILC7JLPhK+RLPtP1Sx66NzbECl5Z552EccSKObM7HQDEfmPmE5Nm4ydzTyMRj4q1j4h8fHQY97pA+rF5VWK9B7Yk0lPdJishmmy25BqIryRdbpoxWZF5lcHNIRd0/7Mnj7vV2a4sRmKXMrtjRJL0kZ5xx6AJCrY+Yw+L2m7Z+riI9OzXee4YPMkntOFGpJPIADC6x37S9Y43/IQ/z4rIEQekntZj2seZP2PGjqXMummbfItPjFI5a/3E1Dph8udIfxm0TEksU7rapwrAzM8DOhL7N3I4S1SVbFS3elnhlUPMCA6NixJFI4gn2bH1hZ154qywxcGKWAIjE2HTE1mo6QifLbFdNVhlZubuB0RMqvPPIuiopkKqCceGUP9fBPmOkaqynTtB6B25lk+yrM2nfC0WI/OioRnjW5R+7gRmxGNqRxRAKiKOwKBoOitNBFwBaWqIieO6efw2xcsVZInS/WMQTSCV3DHsbGW7DmteI9UNupyd3X78QZWxmuXSZVXHeZ7/BjxT1loZK7jQzF3A49r0HzI+ivPWiQWYbFh3j0nkRuQcYszV5hNLDOzug4Ej4hdEed0txSkAylV5KuPC6P+v01gPCssvICEsQFu46Oh5OuPoxpZkkTs4tPXjxv6Roy4z6m2TVo17XeS/wsUCkUUVSnZExp0mtGP6mPznc6GQ4e1SZYKtW7FNLKQkxJCKvir1G5ZXSJG744/wCvDeRVqo3+/PKeSju6z2DFhPqar+ipE3m+2fK/RqGXizWd7MCWRFE6SeeFcBtwGLLAzx1QxkeexsJUTTudWA6houJV5TWpysl2eP8AdIFj19dv0f0RQIXbTv0GDyv2o5G1SONX/wBjEq8gzc9OoYhXmzdReVjzdz2sxJP6PDmcsBijtymRImSxCmnDUhcfRl5Ld+yvbG1ieOOOD2gHxXGiRRL3nUszHmznmzEk/o8ihl1U7gdD6CNR/wBRf//EADURAAIBAgMEBgkEAwAAAAAAAAECAwARBBIhBRMxQRAzUWFysRQgIiMyNVJxkTBAc4FCUOH/2gAIAQIBAT8A/eYRLuXPBfOphEqhpV0PA21NOYyfdqQO8+okbyGyIWPdRBBIIsRoQf0d1Jk3mQ7v6raevHBLL1cZI7eVR7MY6yvYdi1FBFCuVF/s1tN7yRp2Lf8APTFG0jgKpbmQOysZhFCp6PCb31tc1hJJoncRxZ/qFSbyWZrp71jqorcy5xHu2z2va2tqeGWLV4yB2mkhlkGZI2YcLgdBhlCbwxnJa9+VQ/LW8D0uHnZcyxMV+1EEEgixoz4j0TIYfd5QM/dSQyyAskZYDS4FCGUoZBGcg50kbyHKiFj3VDDLFiIN4hW7jjW0/ji8JrDS4NLbyI5vqPtCo5I5BeNwR3dOMfPiZTyBy/jpgmeGQFDqdDWOnkgWMxkAkkG4rZpJlmJ4kXpPmJ/kasZiWw7gRqM7C5Y9lYeX0uBxKBfVTWzZLO8ROhFx9xU0JGKaIf5OLfZq2i4SKOBefktYUqMCC4ugDEjuuaXaMucFgu7v8I5CtpRKVSdRrfKe/sp/lo8C+dbN6h/GfIVLjncNGiqIiMoHO1ArgsIGCjOQP7Y1h8a8sqxzAFSdCBax5VtP44vCegEqbqSD2ioJZVjTO2Y2vrW/QAlja1MSzFjxJv0roy/cVtFHdI8ik2Y3sL1szrJfDSfMT/I1bT69PAPM1szqpPHUUhilST6WvTQh8RFOOAU/8rGS72dyD7I9kf1WFTeYEJ9Qcfkmlw0zSCPdsDexuNBW0WCwxx8yR+BT/LR4F862Z1D+M+Q6JlOJwamMXNgwH2rBwSNOjFCFU3JItwrafxxeE9EKbyRV5cTUsyRcdW7BUkrym7HTkOXqx7RmRQpVWtwJ41DiXgd3RVu3EGhOwn39hnuTblrU874hw7gAgW0qDFyYdSqKpBN9aRHkJCKWPGwrM+HwYMh94FsPvy6ExciQbgKuWxF+etLtKZVsVVj2mpZXmcvIbmji5DB6PlXJYC/PSoMXJh1KIqkE316IMXLBotin0mpNoTOVsAFBBsOdu2p8Q+IKlwBYW06EkMYbL8R50STqf04J3gLFLXIsbipsRLORvG0HADQf6T//xAAoEQACAgEEAQIGAwAAAAAAAAABAgARMQMQEiEzIIEiMDJBUXFAUGH/2gAIAQMBAT8A/mOeqi8sCC/v6CQMn5Vi6vv1lgMmHV/AhYk2TNIdE7kgDsxH7PIxwpAs1BQUd9TktXfUDA4MLAZO3IXV9xvL7icl/I24rzvl3eIWAyZyF1fcJAyYzAq1GaWDHDnBhBGRugpRuyhh3NNQxNzVwsPi9oich3iMODCpqjoGK3wAzSFktH8nWeodIV/s0ibKweX3mrkQaYFE5n1vGQAErNLB3YCzU4n7egzSIBNzVwIfF7TS+k/uauRCLWpypSsQUojmtS/1Cy1dzSFsTB5feauRsp4ObjsOJF9maWDsxoQKTAoHpOkpjIGABnEceMVQooRkDZlgDsygz9YvYoC3KHSW4AFFCcBy5X3GQNnZkDfuDTURVC3WxAOfmMobMChcD+k//9k=)\n","#Principles of Real-Time Analytics Course"],"metadata":{"id":"yqtq9rvgALAB"}},{"cell_type":"markdown","source":["# Scaling-up: How to Size Clusters Correctly\n","Using more than one machine adds a lot of complexity, so if you can stay using just one, you’ll save money and time. Moving from one to two machines multiples complexity by 10. \n","\n","If you can manage with PostgreSQL. MySQL or MongoDB without duplicating data, then do so. Remember the principles of maximum memory transfer; maybe you are scaling because you have a problem of memory bandwidth. Scaling vertically or adding more CPUs will not help here. You'll have to replicate on another machine to have more bandwidth.\n","\n","\n","There are two main reasons to use more than one machine:\n"," - You need computing power, either to run queries faster or more queries at the same time.\n"," - You need data redundancy to avoid data loss or have high availability.\n"],"metadata":{"id":"fJPoUbl9AVTC"}},{"cell_type":"markdown","source":["## Sharding vs. Replication\n","Quick definitions: \n","- Sharding is the way to split your data into several pieces. Those pieces go to different machines.You use sharding when you want to query more data in less time; more speed.\n","- Replication means copying your data to a different machine. You use replication when the speed that one machine provides is enough but you need to have more queries running at the same time. Replication gives redundancy (and also speed).\n","\n","We usually do not use sharding, if the dataset is so large or the query is too complex that you can’t query it with just one machine we try to use views. \n","\n","Wherever we can we use replication because disks are cheap. Also replicas can be used to distribute the load. We use replication most of the time. \n","\n","Replication is easier to set up, not that much more expensive from the storage point of view and, with the right load-balancing setup, it works as expected.\n"],"metadata":{"id":"OlaqwV6qAiem"}},{"cell_type":"markdown","source":["## Load-Balancing \n","When using sharding and/or replication you need to be very careful with the load-balancing. Normally databases do load-balancing themselves. For specific cases, you may want to balance the load how you wish, for example, to leverage machine cache.\n","\n","First and obviously if you don’t distribute the load according to the resources available for each machine you’ll have overloaded machines. But this is just the tip of the iceberg.\n","\n","In a database of maps you can ensure a geographic region is on a single machine because that way, when the data is being used by lots of users, the data you need is more likely to be in cache.\n","\n","If you query several tables or different parts of the tables you should try to balance queries taking into account using caches as much as possible. If you query, for example “February” in all your replicas, all of them will have that month in memory. \n","\n","There are more advanced load-balancing strategies based on usage statistics but those are usually related to a very specific use case. We keep track of every request made with stats about response time, query parameters and so on that we can analyze later.\n"],"metadata":{"id":"JS71teOwBDXB"}},{"cell_type":"markdown","source":["## Ingestion vs. Querying and Async Replication\n","When you are working on a real-time data system, you usually get several requests per second depending on the use case. Inserting data also takes resources so while inserting you could run out of IOPS, CPU and so on.\n","\n","If you want to keep query times low, you need a good load-balancing strategy (to send data to different shards so that the totality of the shards consume all the data in parallel) but also allocating specific resources to that task is important.\n","\n","In our usual setup we have a machine that gets less load when data is being inserted. That machine does the hard work of transfoming and inserting data and then the data is replicated to the others (which were busy consuming queries). We have a multimaster setup so all the machines in the cluster are ready to get data but we try to stick to the same one as much as possible (partially because it usually needs more memory).\n","\n","Replication is usually asynchronous in these setups, so if you want to have a consistent response for a specific client you’ll need to set-up the load balancer to send the same client to the same machine[s]. That goes against what we said in the previous section about load-balancing. You have to pick your fights. In our case, we have a system to make part of the insert system synchronous with a partition swapping algorithm."],"metadata":{"id":"Xju4q-PwBWX8"}},{"cell_type":"markdown","source":["## [Course Outline](https://colab.research.google.com/github/AlisonJD/RTACourse/blob/main/01_Getting_Started.ipynb)\n","\n","|Previous Notebook       |Next Notebook|\n","| :----------------- |:-------------|\n","|9. [Metrics](https://colab.research.google.com/github/AlisonJD/RTACourse/blob/main/09_Metrics.ipynb)|11. [Main Takeaways](https://colab.research.google.com/github/AlisonJD/RTACourse/blob/main/11_Main_Takeaways.ipynb)|"],"metadata":{"id":"Jmp7HIH9rImj"}}]}