Chapter 4 - The BERT algorithm¶
2022 February 16
… but don’t forget about Ernie!
Tensorflow¶
We will walkthrough three tensorflow tutorials for this session:
word embeddings: https://www.tensorflow.org/text/guide/word_embeddings
word2vec: https://www.tensorflow.org/tutorials/text/word2vec
BERT: https://www.tensorflow.org/text/tutorials/classify_text_with_bert
Access the notebooks
It is strongly recommeneded that you download the notebooks (or setup your Colab environment) in advance of our meeting session.
At the top of each tutorial page, click the appropriate button to access the notebooks.
Run all code
Also be sure to run all code in advance.
The models will likely take 1-2 hours to fit and we will not have time to do so during the walkthrough.
Need help?
Contact muzzall {at} stanford {dot} edu
Setup and software library installation instructions¶
Install tensorflow¶
# !pip install tensorflow
Or, setup a virtual environment (you might find this more complicated, but it is worth it in the long run).
View the instructions: https://www.tensorflow.org/install/pip
A dependency of the preprocessing for BERT inputs¶
# !pip install -q -U tensorflow-text==2.7.3
AdamW optimizer¶
Use the AdamW optimizer from tensorflow/models: https://github.com/tensorflow/models
# !pip install -q -U tensorflow-text==2.7.3
Install pydot and graphviz¶
# install pydot
# !pip install pydot
graphviz installation instructions: https://graphviz.org/download/