DyNet for NLP research
This tutorial will quickly bring new NLP researchers to an engineering-level understanding of using a neural network library to quickly build and test machine learning models. It will cover installation and simple examples, with a focus on enabling good research practices.
We’ll be working with DyNet, the Dynamic Neural Network Toolkit. DyNet is a neural network library developed largely for research in NLP at Carnegie Mellon University (and by others.)
(Skip this section if you are just looking for an intro to DyNet.)
The maturation of neural network libraries has meant that many machine learning models, neural or not, are much easier to train to a first approximation. That is, you may be here to train a sequence-to-sequence model, but I’d argue that DyNet is just as useful for implementing a nice linear regression with an arbitrarily complex objective function as long as you’re okay with some variant on stochastic gradient descent as a training regimen.
Installing DyNet (as of 2017-12-22)
As a caveat in this section, (note the date in the section header), installation tips are subject to the vagaries of time. Always check the library documentation for the most up to date installation instructions. I’m going to assume we’re working with the Python bindings.
Here, I’m going to borrow directly from the library installation tutorial, and simply add some notes and extra steps that have proven useful. The following is 98% identical to library tutorial, and included here for completeness.
python3 -m venv dyenv source dyenv/bin/activate pip install cython
This is a step omitted by the official instructions.
It creates a new Python binary and corresponding add-on packages for your work with DyNet.
This is good practice for working with any new Python project.
With DyNet in particular, your dynet installation will be tied to one Python binary.
You don’t want to deal with conflicting dependencies on your global python install, so do this.
For the next steps, make sure you have
mercurial installed on your system.
git clone https://github.com/clab/dynet.git hg clone https://bitbucket.org/eigen/eigen -r 699b659
Eigen is a matrix algebra library. For the next steps, you’ll have to determine whether you have access to a GPU. If you do, awesome! You’ll be able to work on any sequence-to-sequence task. If you don’t, you won’t be able to work effectively on any task that involves generation of language. You will, however, be able to do insteresting work in tagging, for example.
cd dynet mkdir build cd build
Without GPU support:
cmake .. -DEIGEN3_INCLUDE_DIR=../../eigen -DPYTHON=`which python`
With GPU support:
cmake .. -DEIGEN3_INCLUDE_DIR=../../eigen -DPYTHON=`which python` -DBACKEND=cuda
which python provides the path to your current python binary.
This is the only python binary you’ll be able to use with DyNet, and you can’t move it to another disk location.
make -j 2 # Replace 3 with the number of available cores cd python python ../../setup.py build --build-dir=.. --skip-build install
Now determine whether it all worked. The following steps open an interactive Python shell and attempts to import DyNet. If no error is printed, you’re all set.
python import dynet as dy pc = dy.ParameterCollection()
A short feed-forward example
This section will show how DyNet can be used to implement simple ML algorithms quickly, for quick iteration.
Recurrent nets and encoder-decoder architectures
This section will demonstrate how to implement sequence-to-sequence models in DyNet.