Report of Tensorflow Dev Summit 2019
: Presentation Resurce in TensorFlow Dev Summit 2019 Extended Pangyo. This resource is written in Korean.
TensorFlow Dev Summit 2019 Extended Pangyo 발표자료.
| Time | Subject (YouTube vidoe link) | Speaker | Text Summary |
| 9:30 AM | Keynote |
Megan Kacholia, Engineering Director Rajat Monga, Engineering Director Kemal El Moujahid, Director, Product Management Alina Shinkarsky, Program Manager |
|
| 9:50 AM | TensorFlow 2.0 |
Karmel Allison, Engineering Manager Martin Wicke, Software Engineer |
link |
| 10:15 AM | TensorFlow Lite |
Pete Warden, Software Engineer Raziel Alvarez, Software Engineer |
link |
| 10:55 AM | Break | ||
| 12:00 PM | New features in TensorBoard | Gal Oshri, Product Manager | link |
| 12:10 PM | Building Models with tf.function & tf.autograph | Alexandre Passos, Software Engineer | link |
| 12:22 PM | Datasets and Models | Ryan Sepassi, Software Engineer | link |
| 12:30 PM | Swift for TensorFlow: Next-Generation Machine Learning Framework |
Chris Lattner, Distinguished Engineer Brennan Saeta, Software Engineer |
link |
| 1:00 PM | TensorFlow Community | Edd Wilder-James, Open Source Strategist | |
| 1:15 PM | Lunch | ||
| 2:35 PM | TensorFlow Extended (TFX): An End-to-End ML Platform Time stamp: start ~ to 19:51 |
Clemens Mewald, Product Manager | link |
| 2:55 PM | Data Preparation with TFX Data Validation Time stamp: 19:52 ~ end |
Clemens Mewald, Product Manager | link |
| 3:10 PM | TFX Model Validation and TensorFlow Serving | Christina Greer, Software Engineer | link |
| 3:25 PM | TensorFlow Hub | André Susano Pinto, Software Engineer | link |
| 3:40 PM | Break | ||
| 4:45 PM | TensorFlow Probability | Josh Dillon, Software Engineer | link |
| 4:55 PM | Reinforcement Learning in TensorFlow |
Sergio Guadarrama, Senior Software Engineer Eugene Brevdo, Software Engineer |
link |
| 5:10 PM | JavaScript: Writing models and deploying them in the browser and Node.js |
Yannick Assogba, Front End Software Engineer Nick Kreeger, Software Engineer |
link |
| 5:30 PM | UniRoma: In Codice Ratio | Elena Nieddu, UniRoma | |
| 9:50 AM | Lightning Talks |
Butterfly - Nathan Silberman TensorFlow.jl (Julia) - Jonathan Malmoud NetEase - Huijie Lin TF Lattice - Maya Gupta Alibaba - Wei Lin TF.text - Mark Omernick Uber Manifold - Lezhi Li TF.js at Creative Labs - Irene Alvarado |
link link |
| 11:15 AM | Breakout sessions |
2.0 and Porting Models Karmel Allison, Martin Wicke, Tomer Kaftan, Alex Passos, Anna Revinskaya TensorFlow at Scale Jiri Simsa, Reed Wanderman-Milne, Penporn Koanantakool, Yuefeng Zhou TensorFlow On-Device: Compressing, Quantizing, and Distributing YC Ling, Suharsh Sivakumar, Sara Sirajuddin, Tim Davis, Pete Warden |
|
| 12:00 PM | Lunch |
Contributors Luncheon for those interested in SIGs and more TensorFlow Extended (TFX) workshop TFX Workshop Meterials GitHub Repo |
|
| 1:15 PM | Research and the Future |
NERSC - Exascale Deep Learning for Climate Analytics Thorsten Kurth Federated Learning Krzysztof Ostrowski Mesh TensorFlow Noam Shazeer Sonnet 2.0 Tamara Norman, Malcolm Reynolds |
- skip
- TensorFlow 2.0 alpha realsed(since 2019/3/6)
- pip install -U --pre tensorflow
- What is changed:
- Usability:
- tf.keras as the high-level API
- Eager execution by default
- Clarity:
- Remove duplicate functionality
- Consistant intuitive syntax across APIs
- Compatibility throughout the TensorFlow ecosystem
- Flexibility:
- Full lower-level API
- Internal ops accessible in tf.raw_ops
- Inheritable interfaces for variables, checkpoints layers
- Usability:
- How do i upgrade from 1.0 to 2.0? # "Google is now under process of converting one of the largest codebases in the world"
- Provide migration guides and best practices
- Provide Escape to backwords compatibility module: tf.compat.v1(contains all of the 1.x API except tf.contrib)
- Provide conversion script: tf_upgrade_v2 # "Note that this automatic conversion it will fix it, so, it works, but it won't fix your style"
- Timeline
- Alpha
- Available now
- Next release candidate
- In spring
- Implementing some missing features
- Converting libraries
- Converting Google
- A lots of testing and optimization
- RC
- Release testing
- Integration testing
- Alpha
- Progress
- Go build
- pip installl -U --pre tensorflow
- Docs: tensorflow.org/r2.0
- High Level APIs
- Keras inside tensorflow: tf.keras
- Standardizing on the keras API for building layers and models
- Include all the power of estomators
- You cam move from prototype to distributed training, to production serving in one go
- In 2.0, 1.x keras model definition will run in Eager mode without any modification
- tf.keras.optimizer.*
- tf.keras.metrics.*
- tf.keras.losses.*
- tf.keras.layers.*
- Eager mode
- Easy to debug
- Dynamic control
- Unified RNN layers
- In 2.0, there is on version of the LSTM and one version of GRU layer and they select the write operation for available devices for runtime
- tf.keras provides an API that is easy to subclass and customize, so that user can innovate on top of the existing layers
- User can use feature colum to parse data and feed it directly in to downstream keras layers and this feature column work both with Keras and Estimators.
- User can mix ans match to create reusable data input pipelines
- Tensorboard
- Tensorboard integration with Keras as a simple as one line
- tf.keras.callbacks.TensorBoard(log_dir=log_dir) and Use the return value as a param of model.fit(...)
- Performance profilling is built-in
- Tensorboard integration with Keras as a simple as one line
- Going big: tf.distribute.Strategy
- Use Multi-GPU in simple code and Scaling efficiency is greater than 90% over multiple GPUs
- To SavedModel and beyond
- Easily export models for use with TF Serving, TF Lite and more
- Comming soon:
- Multi-node, Multi-worker synchronous training
- Distributing tf.keras with ParmeterServerStrategy
- Exposing canned Estimators from the Keras API
- Handling Very Large Models with variable partitioning
- ... and much more !
- Keras inside tensorflow: tf.keras
- Why Tensorflow Lite ? On-device ML allows building new types of products !
- Access to more data
- Fast and closely knit interactions
- Privacy preserving
- Challenges
- Reduced compute power
- Limited memory
- Battery constraints
- Simplifying ML on-device
- Tensorflow Lite makes these challenges much easier
- Use cases > 2B mobile devices (Have Tensorflow Lite deployed on them in production)
- Text - Classification, Prediction
- Speach - Recognition, Text to Speach, Speach to Text
- Image - Object detection, Object location, OCR, Gesture Recognition, Facial modeling, Segmentation, Clustering, Compression, Super Resolution
- Audio - Translation, Voice Synthesis
- Content - Voice generation, Text generation, Audio generation
- Use case 1 - Google assistant
- 1B + devices
- Wide ragne of devices: High/low end, arm, x86, battery powered, plugged in, many operating systems
- Key Speach On-Device Capabilites
- "Hey Google" Hotword with VoiceMatch
- Tiny memory and computation footprint running continuously
- Extremely latency sensitive
- On-device speech recognition
- High computation running in shorter bursts
- "Hey Google" Hotword with VoiceMatch
- Why Did We Migrate to TFLite?
- TensorFlow Lite meets or beats our existing libraries size and speed
- Tensorflow Lite lays the groundwork to accelerate our models on GPUs, Edge TPUs, etc.
- 1B + devices
- Use case 2 - NetEase youdao
- Company introduction
- Online Education Brand with the largest numbers of users in China (800 million Users, 22 million DAU)
- Youdao Application with TensorFlow Lite
- App introduction
- Youdao Dictionary - Most popular dictionary App in China
- Youdao Translator - Most populary translation App in China
- U-Dictionary - Most popular dictionary and language learning App in India
- Detail App features
- Conveniently look up words
- Camera recognize words from the images and do OCR and the translation on your devices
- Use Tensorflow Lite to accelerate the OCR and translation services
- Photo translation
- Scenarios:
- Use camera to take a photo from the many scenarios and it will do the whole image OCR ans translate the text into another languages
- Erase the text on the image and replace the origianl text to the translated text
- Replace the original text to the translated text
- Present to users with the translations
- Use TensorFlow Lite to accelerate the OCR and translation services
- Scenarios:
- Conveniently look up words
- App introduction
- Why NetEase youdao choose TensorFlow Lite ?
- OCR and Translation are very sensitive to the binary size and also computation resources and it responding time
- To accelerate abillites on device
- To provide the more efficient on device inferences
- Company introduction
- TensorFlow Lite themes
- Usability - Get your models up and running
- Model conversion
- Steps
- TensorFlow -> Savede Model -> TF Lite Converter -> TF Lite Model
- failures & solutions
- Limited ops => Tensorflow Select
- In the pipeline(future work): TensorFlow Select + Selective Registration
- Unsupported semantics (e.g. control-flow in RNNs) => Not yet
- In the pipeline(future work): Control flow support (e.g. loops conditions)
- Limited ops => Tensorflow Select
- Tensorflow Select
- Available now
- Enables hundresd more ops from TensorFlow on CPU
- Caveat: binary size increase(~6MB compressed)
- In the pipeline(future work)
- Selective registration
- Selective registration is already something user can take adavangate of TensorFlow Lite for our building ops. So, user only include in the binary the ops that user is really using. So, user don't end up increasing user's binary size unnecessarily
- Improved performance
- Trying to blur the of the TensorFlow ops and the Tensorflow Lite ops are. One of the key points of this is blur the performance gap that there might be between one and the other <--- blur ?? blow ??
- Selective registration
- Available now
- Control flow support
- In the pipeline(future work)
- Control flow are core to many ops(e.g RNNs) and graphs. Thus TensorFlow Lite team is adding support for:
- Loops
- Conditions
- Control flow are core to many ops(e.g RNNs) and graphs. Thus TensorFlow Lite team is adding support for:
- In the pipeline(future work)
- Converter 2.0
- Follow users feedback! TensorFlow Lite team is building a new converter that will answer:
- What went wrong ?
- Where it went wrong ?
- How can i fix it ?
- Follow users feedback! TensorFlow Lite team is building a new converter that will answer:
- Steps
- Model conversion
- Performance - Get your models executing as fast as possible
- Incrediable inference performance
- CPU: Pixel2 - Single threadded CPU
- Model: MobileNet v1
- Performance:
- CPU: 124 ms
- CPU(Quantization): 64 ms
- GPU(Flow OpenGL 16): 16 ms
- Edge TPU(Quatized Fixed point): 2 ms
- Benchmarking
- Model Benchmark tool
- Benchmarking and profiling
- Available
- Support for threading
- Per op profiling
- Support for Android NN API
- Available
- Benchmarking and profiling
- Model Benchmark tool
- Acceleration by delegate
- Edge TPU delegate
- High performance
- Small physical and power footprint
- GPU delegate
- Preview available!
- 2-7x faster than the floating point CPU implementation
- Adds ~250KB to binary size(Android/iOS)
- Simple to add GPU Support in source code(just need to add few lines)
- In the pipeline(future works)
- Expand coverage of operations
- Further optimize performance
- Evolve and finalize the APIs
- Preview available!
- Android Neural Network API(NNAPI) delete
- Enables hardware supported by the Android NN API
- CPU optimazations
- TensorFlow Lite team knows CPU deployments are important!
- In the pipeline
- Further optimizations on these architectures:
- Arm
- x86
- Further optimizations on these architectures:
- Edge TPU delegate
- Incrediable inference performance
- Optimization - Make your models even smaller and faster
- Quatization
- Available
- Post-training quatization (CPU) - recommaned to use
- Post-training quantization with float & fixed point
- Great for CPU deployments!
- Benefits:
- 4x reduction in model sizes
- models, which consist primarily of convolutional layers get 10~50% faster execution (CPU)
- Fully-connected & RNN-based models get up to 3x speed-up (CPU)
- Extremely simple to try
- User have to validate how the accuray is affected in user's particular model. But so far, TensorFlow Lite team have seen very good numbers
- Many of the 2B devices that above mentioned are running this way
- Post-training quatization (CPU) - recommaned to use
- In the pipeline(future work) - Even better performance on CPU, Plus enable many NPUs!
- Keras-based quantized training (CPU/NPU)
- Training with quantization Keras-based API
- Post-training quantization (CPU/NPU)
- Post-training quantization with fixed point math only
- Keras-based quantized training (CPU/NPU)
- Quantization Steps (post-training)
- Tensorflow(estimator or Keras) > Saved Model + Calibration Data > TF Lite Converter > TF Lite Model
- Quantization results: training vs post-training - Result is almost the same
- Top 1 accuracy
- Model | float baseline | Quantization during training | Quantization after training
- Mobilenet v1 | 70.95% | 69.97% | 69.54%
- Resnet v2 ___| 76.8% _| 76.7% _| 76.6%
- Mobilenet v1 | 77.9% _| 77.5% _| 77.7%
- Top 1 accuracy
- Available
- Other optimizations
- Available
- Model optimization toolkit
- In the pipeline(future work)
- Keras-based connection pruning
- Connection pruning
- What does is mean?
- Drop connections during training
- Dense tensors will now be sparse (filled with zeros)
- Benefits
- Smaller models - Sparse tensors can be compressed
- Faster models - Less operations to execute
- What does is mean?
- Coming soon
- Training with connection pruning in Keras-based API (compression benefits)
- In the pipeline(future work)
- Inference support for sparse models (speed-ups on CPU and selected NPUs)
- Simple to add pruning in source code(just need to add few lines)
- Connection pruning
- Keras-based connection pruning
- Pruning results
- Negligible accuracy loss at 50% sparsity
- Small accuracy loss at 75%
- Available
- Quatization
- Documentation - Resources to get the most out of TensorFlow Lite
- Docs
- Available
- New tensorflow.org/lite
- Revamped TensorFlow Lite documentation with new tutorials & guides
- Published TensorFlow Lite future roadmap
- New tensorflow.org/lite
- In the pipeplie(future work)
- More tutorials
- Available
- Model repository
- Available
- In depth sample application & tutorials for(5 models):
- Image classification
- Object detection
- Pose estimation
- Segmentation
- Smart reply
- In depth sample application & tutorials for(5 models):
- In the pipeplie(future work)
- Expand repository to many more models
- Available
- Docs
- TF Mobile Deprecated
- Provided 6+ monthes of notice
- Limiting developer support in favor of TensorFlow Lite
- Still available for training on Github
- What about Traing on device using TensorFlow Lite ?
- Training
- Building many of the pieces
- TF Select + Control Flow + Subgraphs + Variables
- Gradient calculation
- Forward Backward
- Initialization
- Storage during training
- TF Select + Control Flow + Subgraphs + Variables
- Nothing to announce now, but, TensorFlow Lite team working on it and thinking a lot about it
- Building many of the pieces
- Training
- What about Tensorflow 2.0 ?
- TensorFlow Lite support TensorFlow 2.0
- SavedModel -> Tensorflow Lite
- TensorFlow Lite support TensorFlow 2.0
- Before we go.. 2 last thing !
-
- Pretty Cool New Project by Pete Warden - MCU
- Tiny models on tiny computers !
- Microcontroller are everywhere
- Speech researchers were pioneers
- Models just tens of kilobytes
- Why the models had to be so small ?
- To save battery !
- Microcontrollers use far less power than CPUs
- Main CPU is turned off while MCU listens
- Only tens of kilobytes of RAM and Flash available
- No cloud connection
- Pete thought this approach would be widely useful
- Open up voice interfaces to more developers
- Work with other kinds of noisy sensor data
- Google releasing the first experimental support for embedded platforms in tensorflow light
- Pocket size demonstration board is showed up !
- A prototype of a development board built by Sparkfun link
- 32-bit ARM Cortex-M4F processor with Direct Memory Access
- 48MHz CPU clock, 96MHz with TurboSPOT™
- Extremely low-power usage: 6uA/MHz
- 1MB Flash
- 384KB SRAM
- Dedicated Bluetooth processor with BLE 5
- Available to run on a single coin battery for many days !
- Live Demo by Pete Warden
- Say "Yes"
- Yellow light blink
- Demo Video Link
- Why is this useful ?
- Running entirely on-device
- Tiny constraints
- It's using a 20 KB model
- Run using less than 100KB of RAM and 80KB of Flash
- All Open Source & Train your own model
- Try it for yourself!
- https://www.sparkfun.com/products/15170
- Designed to be portable, runs on many other platform
- Looking forward to collaborating !
-
- Teachable Machine - On-Device Transfer Learning on the Coral Dev Board
-
What is Coral ?
- Coral is a platform for creating products with on-device ML acceleration
- First products features Google's Edge TPU in SBC and USB accessory forms
-
Edge TPU
- A Google-designed AISC that lets you run inference on-device:
- Very fast inference speed(Object detection in less than 15ms)
- Enables greater data privacy
- No reliance on a network connection
- Runs inference with Tensorflow Lite
- Enables unique workloads and new applications Like on-device real-time transfer learning
- A Google-designed AISC that lets you run inference on-device:
-
Coral Products
- Coral Dev Board
- A single-board computer with a removable system-on-module (SOM) featuring the Edge TPU.
- Supported OS: Mendel Linux (derivative of Debian), Android
- Supported Framework: TensorFlow Lite
- Languages: Python (C++ coming soon)
- Datasheet
- Learn More
- Coral Accelerator
- A USB accessory featuring the Edge TPU that brings ML inferencing to existing systems.
- Supported OS: Debian Linux
- Compatible with Raspberry Pi boards
- Supported Framework: TensorFlow Lite
- Datasheet
- Learn More
- Edge Training Demo
- Three ways to do edge training:
- k-nearest neighbors
- Weight imprinting
- Last layer retraining
- Teachable Machine
- K-nearest neighbors approach is used
- Live Demo by June
- Teachable Machine showed up
- Edge TPU development board assembled with a camera and series of buttons
- Each botton corresponds with the class and lights up when the model identifes an objects from the camera
- The training process is done on edge device immediately
- Classify perfectly
- Demo Video Link
- Teachable Machine showed up
- Three ways to do edge training:
- Coral Dev Board
-
- Usability - Get your models up and running
- What's new with TensorBoard ?
- Summary
- TensorBoard in Colab and Jupyter Notebook
- Easier comparison of train and validation runs
- Keras conceptual graph visualization
- Hyperparameter tuning with the HParams dashboard
- Details
- "Tensor Board Team enables showing tensor board directly within colab and Jupyter Notebook"
- How ?
- Train with model.fit() and give it a tensorboard callback
- This logged the right data to visualize in Tensor Board
- Before this new features, User have to download the logs to local machine to visualize the data with Tensor Board
- How ?
- "Train and Validation showing up on the same charts to make it much easier to compare them in accuracy loss"
- More flexable Graph Dashboard
- Several APIs for using Tensor Board within notebooks that user change the height of the cell as well as list the active instances within user's collab notebook
- Hyperparameter Tuning
- Available in the TF 2.0 alpha
- How to use ?
- Several additional imports are needed
- Define which values of the hyperparameters user want to try
- Ex) num_units_list = [16, 32]
- dropout_rate_list = [0.1, 0.2]
- optimizer_list = ['adam']
- Define matrix
- ex) accuracy
- Wrap the existing training code
- Start Training !
- The progress of traing is showed within Tensor Board in Colab(Jupyter Notebook)
- Go to "HPARAMS" dashboard in TensorBoard Top Menu
- TABLE VIEW
- Each run is represented by a row
- Each columns for each of the hyper parameters
- Filtering and Sorting are available in the matrix
- PARALLEL COORDINATES VIEW
- SCATTER PLOT MATRIX VIEW
- TABLE VIEW
- Go to "HPARAMS" dashboard in TensorBoard Top Menu
- "Tensor Board Team enables showing tensor board directly within colab and Jupyter Notebook"
- Summary
- Why graphs ? If we have graphs,
- We can get full program optimization on some hardware like TPU
- We can take a model and deploy it on server, mobile devices and whatever thing you want
- So, Just keep doing with graphs ? No
- TensorFlow is fundamentally changing the programming model in v2.0
- Removing the model that first add a bunch of nodes to a graph and then rely on session that run to prune things out of the graph to figure out the precise thins you want to running the correct order.
- Replaceing it with a much simpler model based on this notion of a function
- It's "tf.function()"
- User will never have to use session or run anymore
- tf.function()
- A function is like an op
- Just like an operation execpt one that user get to define using composition of the other operations intensive flow
- Once have the funtion,
- User call it in inside another function
- User can take it's gradient
- User can run it on the GPU, TPU, CPU, distributed things
- Just like how user do with any other tensorflow operation
- Functions can be faster than eager code
- If user look at models there are large convolutions or big matrix multiplications large reductions, it's not actually any faster becuase the eager excution is pretty fast
- Whereas models get small and the operations get small, tf.function speed up tensorflow.
- ex) Tenfold speed up if user used tf.funtion with tiny lstm cells of ten units
- Functions are polymorphic
- Tensorflow graph is very much not polymorphic
- ex) A graph built for a float64 cannot use float32 or float16
- but tf.function is like python code tends to be very free into the types
- So, User just use the tf.function as normal tensorflow operation
- the autograph build up user's code to graph and the graph run blazingly fast
- And, It's not completely hidden, User can access to the graph that is generated
- If user need to manipulate the graphs, user still can do it
- Tensorflow graph is very much not polymorphic
- A function is like an op
- State in tf.function
- Never write tf.control_dependencies again
- Automatic control dependencies
- Never write tf.control_dependencies again
- Variables in tf.function
- Nerver use tf.global_variables_initializer again
- User don't need to manually initialize variables
- Non-ambiguous code is ok
- Inializers can be path-dependent
- Nerver use tf.global_variables_initializer again
- Control flow in tf.function
- Breaking up with tf.{cond, while_loop}
- python compiler called autograph that rewrite control flow expression into faster dynamic graph code
- How do we customize python ?
- Operator overloading !
- __add__, __sub__, etc
- We can't overload __if__
- Sadly, no __if__, __while__, etc
- Autograph:
- it overload __if__, __while__, etc
- Tensorflow compiler called autograph that take user's python code and rewrite it in a for that lets us call __if__, __while__, etc on tensors
- it overload __if__, __while__, etc
- Operator overloading !
- How do we customize python ?
- python compiler called autograph that rewrite control flow expression into faster dynamic graph code
- Breaking up with tf.{cond, while_loop}
- Summary
- session.run goes away
- tf.control_dependencies goes away
- tf.global_variables_initilizer goes away
- tf.cond, tf.while_loop goes away
- Just use tf.funtion like how we would use in a normal programming language
- It's new way to have a collection of public research datasets
- It's on GitHub and Pypy and ready to use with tensorflow
- Data and Models
- The ML partnership
- tf.data
- High-performance and flexible input pipelines
- Data is a little out of step
- Every single dataset is little bit different
- User is just interesed in having data in a format that sort of ready to move into TF data pipeline
- Google and user really want that data and models to be simpatico
- TensorFlow Datasets
- Let's play nice
- import tensorflow_datasets as tfds
- train_ds = tfds.load("<NAME_OF_DATASET>/<ENCODING_TYPE>", split="train", as_superviesed=True)
- train_ds = train_dataset.shuffle(1024).batch(128)
- for input, targets in train_ds.repeat(10): # using with loop
- ...
- Let's play nice
- class DatasetBuilder
- Each of tf.dataset is packaged together as a datasetBuilder
- First, it had a method called download and prepare
- Second, as_dataset. as_dataset take those pre-processed files on disk and produces a tf.data.dataset
- Third, Useful information. Metadata about the dataset which can be programmatically useful.
- DatasetInfo
- Documenting objects
- features (shape, class label)
- total_num_examples
- splits (train, test)
- etc
- DatasetInfo
- Each of tf.dataset is packaged together as a datasetBuilder
- TensorFlow Datasets support NumPy usage too
- tfds.as_numpy(...)
- It hide all the tensorflow from user
- return python generator over numpy array
- tfds.as_numpy(...)
- Installation
- pip install tensorflow-datasets
- tensorflow.org/datasets
- httsp://github.com/tensorflow/datasets
- 30+ datasets with more on the way
- There is more being added every day
- The community on github has actually gotten surprisingly active
- Make your data famous
- httsp://github.com/tensorflow/datasets
- Swift for TensorFlow
- TensorFlow with no boundaries!
- Swift
- Modern and Cross-platform programming language
- Designed to be easy to learn and use
- OpenSource (swift.org)
- Image classfication model
- look alot like Keras
- Code example
- Train a model
- Pick an optimizer and random input then pick a training loop
- Code example
- Demo: Basic Workflow
- Modifying machine learning models
- Environment
- Google collab hosted Jupyter notebooks and It comes with Swift for TensorFlow
- First try:
- Initiate a model
- Use stocastic gradient descent SDG optimizer
- Just train a model using swift for tensorflow
- Start training
- Found the loss is decreased
- Second try:
- Add dense layers
- Add skip connection
- Re-run training loop
- This is an example of what it's like to use Swift tensorflow to develop and iterate as you apply models to applications
- Environment
- Modifying machine learning models
- Swift for tensorflow was designed for researchers
- Researchers often ..
- need to do more than just change models
- change the way the archtecture fits together
- need to define entirely new abstractions or layers
- Writing custom layers
- Define DoubleBiasDenseLayer
- Code example
- Initiate
- Training using custom hand write training loop
- Simulate running on a super large data parallel GPU or TPU cluster on a single machine
- Experimenting with batch sizes
- Custom training loop
- Additional inner loop
- Run forward pass
- Take the gradients for each step and aggregate them in this grads variable
- Simulates running on four independent accelerateors four GPUS
- So plexable
- Custom training loop
- Experimenting with batch sizes
- Researchers often ..
- No boundary: Interoperability
- No wrappers, Just import it, then call it
- Demo: Interoperability
- Seamless Python Interoperability
- Import Tensoflow
- Import Python
- The Python object that allow you to import arbitray python libraries
- ex) matplotlib.pyplot and numpy from python
- Run the plt.plot and numpy and they just works
- Looks like the python code but it's actually pure Swift
- Just works seamlessly
- RL with OpenAI Gym
- Import Gym
- Define neural network
- Use cart pole v-0 environment
- Run
- Get gradient
- Record the mean reward
- Train
- It's all very simple straight forward Swift
- User can train using Swift TensorFlow model in an openAI gym environment using Python bridge. Totally seamless
- User can keep train of the parameters of the rewards and plot the mean rewards as the model train using Python Numpy. Totally seamless
- Seamless Python Interoperability
- No boundary: Differentiable programming
- Demo: Differentiable Programming
- Differential Programming
- annotate @differentiable then tensorflow will derive the derivative for the function
- Just plain old double is used becuased automatic differentiation is built right into the language
- annotate @differentiable then tensorflow will derive the derivative for the function
- Custom differential data types
- @differentiable helps catch errors
- Maginitude of point
- Swift doesn't include a square root function because I want a good excuse for you to see the interoperability with C
- We're actually goint to use a C's square root function that operates on doubles
- Error occured, Why ?
- Square root function is compiled by the C compiler and today the C compiler can't automaticcally compute deribatives for you
- This is great excuse to show you how to write custom gradients
- So, create own square root function with swift
- Demo Summary
- How to use
- customization
- custom gradients
- custom datatypes with the language integrated automatic differentiation built into swift for TensorFlow
- How to use
- Printing out values in the backward pass
- Differential Programming
- Demo: Differentiable Programming
- No boundary: Performance
- Speedy low level performance
- Thread-level scalability, no GIL
- Automatic graph extraction
- Example: AlphaGo Zero
- Combination of three technologies
- Deep learning
- "Old AI" techniques: Monte Carlo Search
- High performance TPU accelerators
- Demo: MiniGo in Colab
- Open-source go player inspired by deep minds alphago zero project
- It's available on github
- Monte Carlo tree search and the rest of mini go self play in pure Swift
- Combination of three technologies
- No boundary: AI education
- Jeremy Howard from Fast.AI
- Fast.AI always looking to push the boundaries of what's possible with deep learning especially pushing to make recent advances more accessible
- World's best document classifier
- Hundreds thousands have become deep learning practitioners through Fast.AI cources
- SOTA result with Fast.AI library
- Announcing that Fast.AI's next course will include a big Swift component
- Jeremy Howard from Fast.AI
- Most important part is that Swift for TensorFlow is really TensorFlow.
- Open Source
- colab
- Works great in colab
- v0.2 is released
- Available now: Intrepid researchers wanted!
- Not ready for production yet but
- Tensorflow swift team is very excited about
- Shaping future,
- Building out,
- Exploring new programming model.
- This is a great opportunity for this advanced researchers to get involved and help shape the future of tensorflow swift platform
- Tensorflow swift team is very excited about
- TensorFlow extended (TFX): An End-to-End ML Platform
- Figure 1: High-level component overview of a machine learning platform
- TFX powers our most important bets and products
- AlphaBets
- Major Products
- So far, we've made some of our libraries available
- Figure 1: High-level component overview of a machine learning platform
- ... and some of our most important partners
- Today, we share the horizontal layers that integrate libraries in one product
- Building Components out of Libraries
- What makes a Component
- Well defined config
- Context
- Metadata Store? That's new
- Task-Aware Pipelines
- Task-and Data-Aware Pipelines
- What's in the Metadata Store ?
- Type definitions of Artifacts and their Properties
- E.g. Models, Data, Evaluation Metrics
- Execution Records (Runs) of Components
- E.g. Runtime Configuration, Inputs + Outputs
- Lineage Tracking Across All Executions
- E.g. To recurse back to all inputs of a specific artifact
- Type definitions of Artifacts and their Properties
- List all training runs and attributes
- Visualize lineage of a specific model
- Visualize data a model was trained on
- Visualize sliced eval metrics associated with a model
- Launch TensorBoard for a specific model run
- Launch TensorBoard to compare multiple model runs
- Compare data statistics for multiple models
- Examples of Metadata-Powered Functionality **
- Use-cases enabled by lineage tracking
- Compare previous model runs
- Carry-over state from previous models
- Re-use previously computed outputs
- How do we orchestrate TFX?
- Component
- Driver and Publisher
- Execute
- TFX Config
- Bring your very own favorte orchestractor
- AirFlow Runtime
- Kubeflow Runtime
- Your own runtime
- TFX: Putting it all together
- Image
- Get started with TensorFlow Extended(TFX)
- An End-toEnd ML Platform
- https://github.com/tensorflow/tfx
- https://tensorflow.org/tfx
- TFX End-to-End Example
- Chicago Taxi Cab Dataset
- Features
- Categorical
- Bucket
- Vocab
- Dense Float
- Transfomrs
- Model (Wide+Deep)
- Output: High tip or not
- Features
- Chicago Taxi Cab Dataset
- Overview
- Data Ingestion
- Component: ExampleGen
- Inputs and Outputs
- Raw data(CSV, TFRecord) -> Example Gen -> Split TFRecord Data -> Training, Eval
- Data Analysis & Validation
- Why validation is important
- Data understanding is important for model understanding
- Treat data as you treat code
- Catching errors early is critical
- Why validation is important
- Inputs and Outputs
- Component: ExampleGen
- Data Analysis & Validation
- Component: StatisticsGen
- Inputs and Outputs
- ExampleGen -> data -> StatisticsGen -> Statistics
- Data
- Training
- Eval
- Serving Logs (for skew detection)
- Statistics
- Captures shape of data
- Visualization highlights unusual stats
- Overlay helps with comparison
- Whay are my tip predictions bad in the morming hours?
- Don't have much data at that time
- Data
- ExampleGen -> data -> StatisticsGen -> Statistics
- Inputs and Outputs
- Component: SchemaGen
- Inputs and Outputs
- StatisticsGen -> Statistics -> SchemaGen -> Schema
- Schema
- High-level description of the data
- Expected features
- Expected value domains
- Expected constraints
- and much more
- Codifies expectations of "good" data
- Initially inferred, then user-curated
- High-level description of the data
- What are expected values for payment types ?
- Schema
- StatisticsGen -> Statistics -> SchemaGen -> Schema
- Inputs and Outputs
- Component: Example Validator
- Inputs and Outputs
- StatisticsGen, SchemaGen -> Statistics, Schema -> Example Validator -> Anomalies Report
- Annomalies report
- Missing features
- Wrong feature valency
- Training/serving skew
- Data distribution drift
- ...
- Is this new taxi company name a typo or a new company
- Annomalies report
- StatisticsGen, SchemaGen -> Statistics, Schema -> Example Validator -> Anomalies Report
- Inputs and Outputs
- Component: StatisticsGen
- Data Transformation
- Component: Transform
- Using tf.Transform for feature transformations
- Inputs and Outputs
- ExampleGen, SchemaGen, Code -> Data, Schema -> Transform -> Transform Graph, Transformed Data -> Trainer
- Code
- User providing pre-processing function
- Schema for parsing
- Transform Graph
- Applied at training time
- Embedded in serving graph
- (Optional) Transformed Data
- For performance optimization
- Code
- ExampleGen, SchemaGen, Code -> Data, Schema -> Transform -> Transform Graph, Transformed Data -> Trainer
- Component: Transform
- Training
- Component: Trainer
- Inputs and Outputs
- Transform, SchemaGen, Code -> Transform Graph, Data, Schema -> Traniner -> model(s) -> Evaluator, Model Validator, Pusher
- User-provided training code(Tensorflow)
- Optionally, transformed data
- model(s)
- Highlight: SavedModel Format
- Train, Eval and Inference Graphs -> Tensorflow model analysis, TensorFlow Serving
- Highlight: SavedModel Format
- Transform, SchemaGen, Code -> Transform Graph, Data, Schema -> Traniner -> model(s) -> Evaluator, Model Validator, Pusher
- Inputs and Outputs
- Component: Trainer
- Model Analysis & Validation
- Why Model Evaluation is important
- Assess overall model quality overall
- Assess model quality on specific segments / slices
- Track performance over time
- Component: Evaluator
- Inputs and Outputs
- ExampleGen, Trainer > Data, Model > Evaluator > Evaluation Matrics
- Configuration and visualization
- Inputs and Outputs
- How well can i predict trips that result in tips > 20%?
- Data visualization
- Why Model Evaluation is important
- Model Validator
- Why Model Validation is important
- Avoid pushing models with degraded quality
- Avoid breaking downstream components (e.g. serving)
- Component: Model Validator
- Inputs and Outputs
- ExampleGen, Trainer > Data, Model(x2) > Model Validator > Validation Outcome
- Evaluation split of data
- Last Validated model
- New candidate model
- ExampleGen, Trainer > Data, Model(x2) > Model Validator > Validation Outcome
- Inputs and Outputs
- Pusher
- Component: Pusher
- Inputs and Outputs
- Model Validator > Validation Outcome > Pusher > Deployment Options
- Inputs and Outputs
- If model passed validation then, the pusher will copy the saved model input a file system destination
- Block push on validation outcome
- Push destinations suppored today
- File System
- TF Serving model server
- Component: Pusher
- Why Model Validation is important
- Model Deployment
- This is where we are ...
- We have a trained saved model
- Saved model is a universal serialization format for tensorflow models
- This is where we want to be ...
- API that we can query to get answers in real time
- Deployment Options
- TensorFlow JS
- TensorFlow Lite
- TensorFlow Hub
- TensorFlow Serving
- TensorFlow Serving
- Flexible
- Multi-tenancy
- Multiple models on a single server instance
- Optimize with GPU and TensorRT
- gRPC or REST API
- Multi-tenancy
- High-Performance
- Low-latency
- Request Batching
- Traffic Isolation
- Production-Ready
- Used for years at Google, millions of QPS
- Scale in minutes
- Dynamic version refresh
- Deploy a REST API for your model in minutes ...
- Using Docker (Recommended)
- Locally on your host
- Easily enable* hardware acceleration
- Basic Settings:
- Choose a port that you want to bind
- Provide the path to saved model that is generated by trainer (or Pushed by pusher)
- Provice model name
- Run the TensorFlow Serving binary
- If you're running on a host with a GPU an Nvidia docker
- Modify command line by a few tokens and then it will be running on acceleratied hardware
- Optimize your model for serving using TensorRT
- Now support optimizing model for serving using TensorRT
- Basic Settings:
- Flexible
- This is where we are ...
- Putting it all together again
- first, you have your data and we have tools to help you make sense of that and process it and prepare it for training.
- we then support training your model and after you train your model we provide tools that allow you to make sense of what you're seeing of what your models
- then we have the pusher that allows you to push to various deployment options and make your model available to serve users in the real world
- Get started with TEnsorFlow Extended (TFX)
- An End-to-End ML Platform
- https://github.com/tensorflow/tfx
- https://tensorflow.org/tfx
- Discover, share, and reuse machine learning
- TensorFlow Hub
- When to use ?
- Less Data
- If you have problems collecting that data to train your models from scratch then transfer learning is technique for you
- Less domain expertise
- Share without code dependencies
- Less Data
- Pre-trained building blocks
- Model in Repository
- TensorFlow Hub in 2.0
- What's comming ?
- Integrated with core
- New SavedModel features
- Easier to save and load parts of a model
- Let's see
- TF-2.0 and eager
- hub.load("model URL")
- Due to eager execution, once loaded we can call it right away
- SavedModel can share TF primitives such as tf.function and tf.variables
- Polymorphic functions
- Integration with Keras
- TF-2.0 and eager
- What's comming ?
- 2.0 Status
- Hub 0.3 version with hub.load() and hub.KerasLayer()
- Published a few modules in new format
- Next:Backporting existing modules
- tfhub.dev
- Cross lingual
- New universal sentence encoder
- Trained in English, French, German, Spanish, Italian, Chinese, Korean and Japanese
- New universal sentence encoder
- AutoAugment
- Image Augmentation
- More Modules & More Tools
- BERT
- Object detection
- BigGAN / Progressive GAN
- Text & Image features
- Video Action recognition
- TFJS
- AdaNet
- TF-Transform
- Cross lingual
- Try it today
- https://tfhub.dev
- Search for "tf2" modules
- Search for "xling" modules
- Try the Colab notebooks
- When to use ?
- So many losses! Wouldn't it be greate if
- Maximize Likelihood
- Summarize what you learned
- A Simple Example: (Deep) Regression
- Guess what! You can.
- Linear Regression
- Learn known unknowns
- Linear Regression (Heteroskedastic)
- Learn unknown unknowns
- Bayesian Weights
- Linear Regression
- Learn known and unknown unknowns
- Bayesian Weights
- Linearn Regression (Heteroskedastic)
- Still don't know ? No problem
- Bayesian Loss Function
- TensorFlow Probability
- What is TensorFlow Probability ?
- A toolbox for probabilistic modelling
- Why use TensorFlow Probability ?
- "If my back's to a wall, I predict 'cat.' but Don't make me!"
- Statisticians/data scientists: write and launch the same model.
- ML researchers/practitioners: make predictions with (un)certainty
- How do i use TensorFlow Probability ?
- Build model
- Distributions
- Bijectors
- Layers / Losses
- Edward2
- Do inference
- Markov chain Monte Carlo
- Variational Inference
- Optimizers
- Build model
- A Success Story: BHGE
- Automated Anomaly Detection: Known Unknowns + Unknown Unknowns
- A Success Story: You?
- An open source Python library using TF which makes it easy to combine deep learning with probabilistic models on modern hardware.
- Install
- pip install tensorflow-probability
- Learn
- Discuss
- Install
- Today's Demo: TFP Github Examples
- "Probabilistic_Layers_Regression.ipynb
- Recommended book:
- Probabilistic Programming & Bayesian Methods for Hackers
- Bayesian Methods for Hackers has been ported to TensorFlow Probability. With collaboration from the TensorFlow Probability team at Google, there is now an updated version of Bayesian Methods for Hackers that uses TensorFlow Probability (TFP).
- Amazon link
- Probabilistic Programming & Bayesian Methods for Hackers
- An open source Python library using TF which makes it easy to combine deep learning with probabilistic models on modern hardware.
- What is TensorFlow Probability ?
- Learnign to walk
- Initial policy
- After 1000 episodes
- After 1800 episodes
- Reinforcement Learning
- Have you tried implementing RL ?
- DQN, PPO, SAC ...
- Policy, Networks
- Environments
- Replay Buffers
- Training
- Bellman updates
- ...
- TF-Agents: Available today
- https://github.com/tensorflow/agents
- pip install tf-agents-nightly
- What is TF-Agents?
- A robust, scalable and easy to use Reinforcement Learning Library for TensorFlow
- Why use TF-Agents?
- Great for Learning RL: Colabs, examples, documentation
- Well suited for solving complex problems with RL
- Develop new RL algorithms quickly
- Well tested and easy to configure with gin-config
- TF-Agents is easy to use
- Build for TF 2.0:
- Develop and debug quickly with TF-Eager
- Use tf.keras to define your Networks
- Use tf.function to speed everything up
- Modular and extensible
- Compatible with TF 1.14:
- For those not ready to make the change
- Build for TF 2.0:
- Learning to walk in a nutshell (Code Example)
- Create an Actor and Critic Network
- Create a Soft-Actor-Critic Agent
- Get experience and train
- Available Environment Suites
- Gym
- Atari
- Mujoco
- PyBullet
- DM-Control
- Yours .. ?
- Available Agents(Fully tested, Model quality regression tests, Speed regression tests)
- DQN, DDQN, DQN-RNN
- DDPG, TD3
- PPO, PPO-RNN
- REINFORCE
- SAC
- Bebavioral Cloning
- More coming soon !
- Yours ?
- System Overview
- Environments
- Ex) Breakout
- Playing Breakout faster in TF
- You can run in parallel !
- Training
- Learnable Policy
- Observations > Policy > Actions
- Defining a Custom Network (Code Example)
- Define a Network using Keras layers(RNNs are supported too!)
- Policies (Code Example)
- QPolicy
- PPOPolicy
- ActorPolicy
- RandomPolicy
- GreedyPolicy
- Agent (Code Example)
- Create an agent
- Policy used to explore & collect training data
- Train from a batch of experience
- The deployment policy
- Train the Neural Network (Code Example)
- Create a Network
- Create the Agent
- Get experience and train teh Agent
- Learnable Policy
- Collection
- Collect Expeience (Code Example)
- Collect and Train (Code Example)
- Environments
- Roadmap
- Agents: C51, D4PG, IQN, Contextual Bandits
- Baselines: leaderboard, pre-trained models
- Replay buffers: Prioritized, episodic
- Distributed Collection
- Distributed Training: Multi-GPU, TPU
- Environments: Unity
- Visit us on GitHub
- https://github.com/tensorflow/agents
- pip install tf-agents-nightly
- Get started with DQN-Cartpole Colab
- Ready to solve important problems
- Contributions and PRs are welcome:
- Ex: new Environments, enw Agents
- https://github.com/tensorflow/agents
- TensorFlow.js
- No drivers
- Interactive
- Privacy
- With TensorFlow.js you can ...
- Use pre-trained off-the shelf model
- Covert existing Python models
- Train in the browser and Node.js
- Deploy on Platforms
- Pre-trained models
- Ready to use ML today
- Collection of pre-trained models
- Image, Audio and Text
- User friendly wrapper APIs
- NPM or hosted scripts
- More coming soon
- BodyPix Model(Pre-trained person segementation)
- Code Example
- Toxicity (Pre-trained toxic content classifier)
- Code Example
- Ready to use ML today
- Convert Existing Models
- Bring python models into JS
- Command line tool
- Saved Model TFHub Keras
- 170+ Ops
- TensorFlow 2.0 Support
- Python Model -> TensorFlow.js Converter Tool -> Javascript App
- Bring python models into JS
- Training
- Layers API (Write models from scratch in Javascript)
- High level Keras compatible API
- Low level ops API for fine control
- Code Example
- Import library
- CPU or GPU(cuda)
- Create model(conv, dense, etc)
- Train model from scratch or transfor learning
- Save the result of training
- After training, get prediction from model
- Import library
- Training Helpers (Data and Visualization)
- tf.data
- Lazily evaluated data pipelines
- JavaScript analog to Python tf.data
- Construct data processing pipelines using common transformation operators
- Work with data taht does not fit in memory
- Code Example
- Lazily evaluated data pipelines
- tfjs-vis
- In-browser visualization library
- Model training behaviour
- Model interanls
- Model evaluation
- Code Example
- In-browser visualization library
- tf.data
- TensorBoard Support for Node.js
- Model training behaviour
- Callback for model fit / model fitDataset
- Layers API (Write models from scratch in Javascript)
- Platforms
- JavaScript runs almost everywhere !
- Browser
- Server side (Node.js)
- Desktop apps
- Mobile
- Browser
- Creatability
- Exploring how creative tools can be made more accessible for everyone
- Uber Manifold
- ML Debugging
- Identiy Document Detection by Airbnb
- Creatability
- Server side (Node.js)
- Clinic.js
- ML powered profiling of server side applications
- Clinic.js
- Desktop
- Magenta Studio
- Make music with ML
- Available in Ableton Live
- Magenta Studio
- Mobile
- WeChat + TF
- 1B+ total users
- 1M+ Mini-Programs
- 200M+ DAU
- 1.5M+ developers
- Demo (WeChat Mini-Programs on iOS)
- Pose detection using TF.js on WeChat miniapp
- WeChat + TF
- JavaScript runs almost everywhere !
- TensorFlow.js 1.0
- Stable API
- Performance
- Mobilenet v1 Inference Time in Miliseconds
- Improved over 2018
- What's next ?
- Direct Tensorflow saved model execution in Node js
- More off-the-shelf TensorFlow.js models
- Tracking Browser Acceleration APIs
- Symbian wasm(??), WebGPU and WebML
- Expanding Platform Support
- ex) Electron, WeChat, Raspberry Pi and other hybrid mobile platforms that run JavaScript
- Thanks
- Resources
- Website: https://tensorflow.org/js
- Pre-trained Models: https://github.com/tensorflow/tfjs-models
- Code Examples: https://github.com/tensorflow/tfjs-examples
- Resources
- The Power of Ultrasound
- The Problem: Access
- 2/3 of the world has no access to medical imaging
- 5,480 children die daily from pediatric pneumonia
- 831 women die daily from preventable maternal-health complications
- Butterfly iQ
- Demotratizing Ultrasound
- 3-in-1 in your pocket:
- High-End Ultrasound
- Sonographer
- Radiologist
- 3-in-1 in your pocket:
- Turn-by-Turn Navigation for Ultrasound
- Heart Health: Ejection Fraction
- Healthy - Unhealthy
- What is true ? Annotator Variability
- Goal: Statistical Indistinguishability
- Train Model to Predict
- Connventional Encode - Decoder model
- Qualitative Results
- Quantitative Results
- Powered by TensorFlow
- Inference
- TensorFlow on-Device
- Custom Ops
- Looking at TensorFlow Lite
- Training
- GPU-based training
- Custom Sampling
- tf.data + Pandas Dataframes
- Active Learning
- Smart Annotation
- TensorFlow Serving
- Inference
- Life Saving Machine Learning
- What is Julia ?
- Dynamic yet performant programming language developed at MIT since 2009
- A modern rethinking of a language for scientific computing
- Version 1.0 (finally) released last year
- Code Example
- What is Jon ?
- N-th year MIT Phd student
- Research intersection of vision + text + curiousity by day
- Contribute to open-source ML projects by
daynight
- Ultra-fast JIT
- For-loop comparison
- Python - 13 secs
- C - 0.9 secs
- Julia - 0.95 secs
- For-loop comparison
- Powerful metaprogramming
- Muliple Dispatch
- Julia growing fast !
- Introducing TensorFlow.jl
- Eager by default, but graph mode available
- Partial Keara API (under development)
- Inspired by TensorFlow 2.0
- Graph mode available
- Although eager mode is preferred...
- Macros -> named tensors
- Macros -> implicit variables
- Fast preprocessing
- A rich ecosystem of packages
- Function - package
- Differential equations - DifferentialEquations.jl
- Canonical datasets - MLDatasets.jl
- Fourier transforms - FFT.jl
- Computer vision - Images.jl
- Any Python package - PyCall.jl
- Reach into Python if you need it
- Compatible with the TF ecosystem
- Tensor Board
- Share serialize dgraphs (in graph mode)
- the variable you can save graphs in Julia load them in Python and vise versa
- Custom operations
- Combile with DiffEq
- Roadmap
- Fuller Keara API
- Distributed training
- Expanded documentation and error handling
- You can help ...
- Get involved!
- Download Julia from https://julialang.org
- Then '] add TensorFlow#eager_mode' in the Julia REPL
- Github: https://github.com/malmaud/Tensorflow.jl
- Chat: https://slackinvite.julialang.org/
- Twitter: @malmaud
- Thanks to...
- Lyndon White
- Other contributors
- The super-friendly Julia community
- TensorFlow team