anton/mlpractical
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Minor textual modifications

Browse Source
This commit is contained in:
Steve Renals 2015-11-01 20:15:09 +00:00
parent 232d43c363
commit dc115a1796
1 changed files with 30 additions and 33 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

63
04_Regularisation.ipynb
View File

@ -6,23 +6,20 @@
"source": [
"# Introduction\n",
"\n",
"This tutorial focuses on implementation of three reqularisaion techniques, two of them are norm based approaches which are added to optimised objective and the third technique, called *droput*, is a form of noise injection by random corruption of information carried by hidden units during training.\n",
"This tutorial focuses on implementation of three reqularisaion techniques: two of them add a regularisation term to the cost function based on the *L1* and *L2* norms; the third technique, called *Dropout*, is a form of noise injection by random corruption of information carried by the hidden units during training.\n",
"\n",
"\n",
"## Virtual environments\n",
"\n",
"Before you proceed onwards, remember to activate your virtual environment:\n",
" * If you were in last week's Tuesday or Wednesday group type `activate_mlp` or `source ~/mlpractical/venv/bin/activate`\n",
" * If you were in the Monday group:\n",
" + and if you have chosen the **comfy** way type: `workon mlpractical`\n",
" + and if you have chosen the **generic** way, `source` your virutal environment using `source` and specyfing the path to the activate script (you need to localise it yourself, there were not any general recommendations w.r.t dir structure and people have installed it in different places, usually somewhere in the home directories. If you cannot easily find it by yourself, use something like: `find . -iname activate` ):\n",
"Before you proceed onwards, remember to activate your virtual environment by typing `activate_mlp` or `source ~/mlpractical/venv/bin/activate` (or if you did the original install the \"comfy way\" type: `workon mlpractical`).\n",
"\n",
"\n",
"## Syncing the git repository\n",
"\n",
"Look <a href=\"https://github.com/CSTR-Edinburgh/mlpractical/blob/master/gitFAQ.md\">here</a> for more details. But in short, we recommend to create a separate branch for this lab, as follows:\n",
"\n",
"1. Enter the mlpractical directory `cd ~/mlpractical/repo-mlp`\n",
"2. List the branches and check which is currently active by typing: `git branch`\n",
"2. List the branches and check which are currently active by typing: `git branch`\n",
"3. If you have followed our recommendations, you should be in the `coursework1` branch, please commit your local changed to the repo index by typing:\n",
"```\n",
"git commit -am \"finished coursework\"\n",
@ -47,11 +44,11 @@
"source": [
"# Regularisation\n",
"\n",
"Regularisation add some *complexity term* to the cost function. It's purpose is to put some prior on the model's parameters. The most common prior is perhaps the one which assumes smoother solutions (the one which are not able to fit training data too well) are better as they are more likely to better generalise to unseen data. \n",
"Regularisation add a *complexity term* to the cost function. Its purpose is to put some prior on the model's parameters, which will penalise complexity. The most common prior is perhaps the one which assumes smoother solutions (the one which are not able to fit training data too well) are better as they are more likely to better generalise to unseen data. \n",
"\n",
"A way to incorporate such prior in the model is to add some term that penalise certain configurations of the parameters -- either from growing too large ($L_2$) or the one that prefers solution that could be modelled with less parameters ($L_1$), hence encouraging some parameters to become 0. One can, of course, combine many such priors when optimising the model, however, in the lab we shall use $L_1$ and/or $L_2$ priors.\n",
"A way to incorporate such a prior in the model is to add some term that penalise certain configurations of the parameters -- either from growing too large ($L_2$) or the one that prefers a solution that could be modelled with fewer parameters ($L_1$), hence encouraging some parameters to become 0. One can, of course, combine many such priors when optimising the model, however, in the lab we shall use $L_1$ and/or $L_2$ priors.\n",
"\n",
"They can be easily incorporated into the training objective by adding some additive terms, as follows:\n",
"$L_1$ and $L_2$ priors can be easily incorporated into the training objective through additive terms, as follows:\n",
"\n",
"(1) $\n",
" \\begin{align*}\n",
@ -60,7 +57,7 @@
"\\end{align*}\n",
"$\n",
"\n",
"where $ E^n_{\\text{train}} = - \\sum_{k=1}^K t^n_k \\ln y^n_k $, $\\beta_{L_1}$ and $\\beta_{L_2}$ some non-negative constants specified a priori (hyper-parameters) and $E^n_{L_1}$ and $E^n_{L_2}$ norm metric specifying certain properties of parameters:\n",
"where $ E^n_{\\text{train}} = - \\sum_{k=1}^K t^n_k \\ln y^n_k $ is the cross-entropy cost function, $\\beta_{L_1}$ and $\\beta_{L_2}$ are non-negative constants specified in advance (hyper-parameters) and $E^n_{L_1}$ and $E^n_{L_2}$ are norm metrics specifying certain properties of the parameters:\n",
"\n",
"(2) $\n",
" \\begin{align*}\n",
@ -136,28 +133,28 @@
"\n",
"## Dropout\n",
"\n",
"Dropout, for a given layer's output $\\mathbf{h}^i \\in \\mathbb{R}^{BxH^l}$ (where $B$ is batch size and $H^l$ is the $l$-th layer output dimensionality) implements the following transformation:\n",
"For a given layer's output $\\mathbf{h}^i \\in \\mathbb{R}^{BxH^l}$ (where $B$ is batch size and $H^l$ is the $l$-th layer output dimensionality), Dropout implements the following transformation:\n",
"\n",
"(10) $\\mathbf{\\hat h}^l = \\mathbf{d}^l\\circ\\mathbf{h}^l$\n",
"\n",
"where $\\circ$ denotes an elementwise product and $\\mathbf{d}^l \\in \\{0,1\\}^{BxH^i}$ is a matrix in which $d^l_{ij}$ element is sampled from the Bernoulli distribution:\n",
"where $\\circ$ denotes an elementwise product and $\\mathbf{d}^l \\in \\{0,1\\}^{BxH^i}$ is a matrix in which element $d^l_{ij}$ is sampled from the Bernoulli distribution:\n",
"\n",
"(11) $d^l_{ij} \\sim \\mbox{Bernoulli}(p^l_d)$\n",
"\n",
"with $0<p^l_d<1$ denoting the probability the given unit is kept unchanged (dropping probability is thus $1-p^l_d$). We ignore here edge scenarios where $p^l_d=1$ and there is no dropout applied (and the training would be exactly the same as in standard SGD) and $p^l_d=0$ where all units would have been dropped, hence the model would not learn anything.\n",
"with $0<p^l_d<1$ denoting the probability that the given unit is kept unchanged (the \"dropping probability\" is thus $1-p^l_d$). We ignore here the extreme scenarios in which $p^l_d=1$ and there is no dropout applied (hence the training would be exactly the same as in standard SGD) or in which $p^l_d=0$ whereby all units would be dropped, hence the model would not learn anything.\n",
"\n",
"The probability $p^l_d$ is a hyperparameter (like learning rate) meaning it needs to be provided before training and also very often tuned for the given task. As the notation suggest, it can be specified separately for each layer, including scenario where $l=0$ when some random dimensions in input features (pixels in the image for MNIST) are being also corrupted.\n",
"The probability $p^l_d$ is a hyperparameter (like learning rate) meaning it needs to be provided before training and also very often tuned for the given task. As the notation suggests, it can be specified separately for each layer, including the scenario where $l=0$ when some random dimensions in the input features (pixels in the image for MNIST) are being also corrupted.\n",
"\n",
"### Keeping the $l$-th layer output $\\mathbf{\\hat h}^l$ (input to the upper layer) appropiately scaled at test-time\n",
"\n",
"The other issue one needs to take into account is the mismatch that arises between training and test (runtime) stages when dropout is applied. It is due to the fact that droput is not applied at testing (run-time) stage hence the average input to the unit in the upper layer is going to be bigger compared to training stage (where some inputs were set to 0), in average $1/p^l_d$ times bigger. \n",
"The other issue one needs to take into account is the mismatch that arises between training and test (runtime) stages when dropout is applied. Since dropout is not applied at the testing (run-time) stage, the average input to the unit in the upper layer will be bigger compared to the training stage (where some inputs were set to 0), on average $1/p^l_d$ times bigger. \n",
"\n",
"So to account for this mismatch one could either:\n",
"To account for this mismatch one could either:\n",
"\n",
"1. When training is finished scale the final weight matrices $\\mathbf{W}^l, l=1,\\ldots,L$ by $p^{l-1}_d$ (remember, $p^{0}_d$ is the probability related to dropping input features)\n",
"1. When training is finished scale the final weight matrices $\\mathbf{W}^l, l=1,\\ldots,L$ by $p^{l-1}_d$ (remember, $p^{0}_d$ is the probability related to dropping input features), as mentioned in the lecture\n",
"2. Scale the activations in equation (10) during training, that is, for each mini-batch multiply $\\mathbf{\\hat h}^l$ by $1/p^l_d$ to compensate for dropped units and then at run-time use the model as usual, **without** scaling. Make sure the $1/p^l_d$ scaler is taken into account for both forward and backward passes.\n",
"\n",
"Our recommendation is option 2 as it will make some things easier from implementation perspective. "
"In this lab we recommend option 2 as it will make some things easier to implement. "
]
},
{
@ -181,15 +178,15 @@
"source": [
"# Exercise 1: Implement L2 based regularisation\n",
"\n",
"Implement L2 regularisation method (for weight matrices, optionally for biases). Test your solution on one hidden layer model similar to the one from coursework's Task 4 (800 hidden units) but limit training data to 1000 (random) data-points (keep validation and test sets the same). You may use data providers specified in the above cell. \n",
"Implement an L2 regularisation method (for the weight matrices, optionally for the biases). Test your solution on a one hidden layer model similar to the one used in Task 4 for coursework 1 (800 hidden units) -- but limit the training data to 1000 (random) data-points (keep the validation and test sets the same). You may use the data providers specified in the above cell. \n",
"\n",
"*Note (optional): We limit both the amount of data as well as the size of a mini-batch - it is due to the fact that those two parameters directly affect the number of updates we do to the model's parameters per epoch (i.e. for `batch_size=100` and `max_num_batches=10` one can only adjusts parameters `10` times per epoch versus `100` times in case those parameters are swapped -- `batch_size=10` and `max_num_batches=100`). Since SGD relies on making many small upates, this ratio (number of updates given data) is another hyper-parmater one need to consider before optimisation.*\n",
"*Note (optional): We limit both the amount of data as well as the size of a mini-batch - this is due to the fact that those two parameters directly affect the number of updates we do to the model's parameters per epoch (i.e. for `batch_size=100` and `max_num_batches=10` one can only adjust parameters `10` times per epoch versus `100` times in the case when `batch_size=10` and `max_num_batches=100`). Since SGD relies on making many small upates, this ratio (number of updates given data) is another hyper-parameter one should consider before optimisation.*\n",
"\n",
"To follow with this exercise first build and train not-regularised model as a basline. Then train regularised models starting with $\\beta_{L2}$ set to 0.0001 and do some grid search for better values. Observe how different $L_2$ penalties affect model ability to fit training and validation data.\n",
"First build and train an unregularised model as a basline. Then train regularised models starting with $\\beta_{L2}$ set to 0.0001 and do a search over different values of $\\beta_{L2}$. Observe how different $L_2$ penalties affect the model's ability to fit training and validation data.\n",
"\n",
"Implementation tips:\n",
"* Have a look at the constructor of mlp.optimiser.SGDOptimiser class, it has been modified to take more optimisation-related arguments.\n",
"* The best place to implement regularisation terms is `pgrads` method of mlp.layers.Layer (sub)-classes. See equations (6) and (9) why."
"* Have a look at the constructor of mlp.optimiser.SGDOptimiser class; it has been modified to take more optimisation-related arguments.\n",
"* The best place to implement regularisation terms is in the `pgrads` method of the mlp.layers.Layer class or its subclasses. See equations (6) and (9)."
]
},
{
@ -207,7 +204,7 @@
"source": [
"# Exercise 2: Implement L1 based regularisation\n",
"\n",
"Implement L1 regularisation penalty. Test your solution on one hidden layer model similar to the one from Exercise 1. Then train $L_1$ regularised model starting with $\\beta_{L1}$ set to 0.0001 and do some grid search for better values. Observe how different $L_1$ penalties affect the model ability to fit training and validation data."
"Implement the L1 regularisation penalty. Test your solution on a one hidden layer model similar to the one used in Exercise 1. Then train an $L_1$ regularised model starting with $\\beta_{L1}=0.0001$ and again search over different values of this parameter. Observe how different $L_1$ penalties affect the model's ability to fit training and validation data."
]
},
{
@ -225,9 +222,9 @@
"source": [
"# Exercise 3:\n",
" \n",
"Droput applied to input features (turning on/off some random pixels) may be also viewed as a form of data augmentation -- as we effectively create images that differ in some way from training one but also model is tasked to properly classify imperfect data-points.\n",
"Dropout applied to input features (turning some random pixels on or off) may be also viewed as a form of data augmentation -- as we effectively create images that differ in some way from the training set; but also the model is tasked to properly classify imperfect data-points.\n",
"\n",
"Your task in this exercise is to pick a random digit from MNIST dataset (use MNISTDataProvider) and corrupt it pixel-wise with different levels of probabilities $p_{d} \\in \\{0.9, 0.7, 0.5, 0.2, 0.1\\}$ (reminder, dropout probability is $1-p_d$) that is, for each pixel $x_{i,j}$ in image $\\mathbf{X} \\in \\mathbb{R}^{W\\times H}$:\n",
"Your task in this exercise is to pick a random digit from the MNIST dataset (use MNISTDataProvider) and corrupt it pixel-wise with different levels of probabilities $p_{d} \\in \\{0.9, 0.7, 0.5, 0.2, 0.1\\}$ (reminder, dropout probability is $1-p_d$) that is, for each pixel $x_{i,j}$ in image $\\mathbf{X} \\in \\mathbb{R}^{W\\times H}$:\n",
"\n",
"$\\begin{align}\n",
"d_{i,j} & \\sim\\ \\mbox{Bernoulli}(p_{d}) \\\\\n",
@ -250,14 +247,14 @@
"source": [
"# Exercise 4: Implement Dropout \n",
"\n",
"Implement dropout regularisation technique. Then for the same initial configuration as in Exercise 1. investigate effectivness of different dropout rates applied to input features and/or hidden layers. Start with $p_{inp}=0.5$ and $p_{hid}=0.5$ and do some search for better settings. Dropout usually slows training down (approximately two times) so train dropout models for around twice as many epochs as baseline model.\n",
"Implement the dropout regularisation technique. Then for the same initial configuration as used in Exercise 1. investigate the effectivness of different dropout rates applied to input features and/or hidden layers. Start with $p_{inp}=0.5$ and $p_{hid}=0.5$ and do a search for better settings of these parameters. Dropout usually slows training down (approximately by a factor of two) so train dropout models for around twice as many epochs as the baseline model.\n",
"\n",
"Implementation tips:\n",
"* Add a function `fprop_dropout` to `mlp.layers.MLP` class which (on top of `inputs` argument) takes also dropout-related argument(s) and perform dropout forward propagation through the model.\n",
"* One also would have to introduce required modificastions to `mlp.optimisers.SGDOptimiser.train_epoch()` function.\n",
"* Design and implemnt dropout scheduler in a similar way to how learning rates are handled (that is, allowing for some implementation dependent schedule which is kept independent of implementation in `mlp.optimisers.SGDOptimiser.train()`). \n",
" + For this exercise implement only fixed dropout scheduler - `DropoutFixed`, but implementation should allow to easily add other schedules in the future. \n",
" + Dropout scheduler of any type should return a tuple of two numbers $(p_{inp},\\; p_{hid})$, the first one is dropout factor for input features (data-points), and the latter dropout factor for hidden layers (assumed the same for all hidden layers)."
"* Also you need to introduce some modifications to the `mlp.optimisers.SGDOptimiser.train_epoch()` function.\n",
"* Design and implement a dropout scheduler in a similar way to how learning rates are handled (that is, allowing for a schedule which is kept independent of the implementation in `mlp.optimisers.SGDOptimiser.train()`). \n",
" + For this exercise implement only a fixed dropout scheduler - `DropoutFixed`, but your implementation should allow to easily add other schedules in the future. \n",
" + A dropout scheduler of any type should return a tuple of two numbers $(p_{inp},\\; p_{hid})$, the first one is dropout factor for input features (data-points), and the latter dropout factor for hidden layers (assumed the same for all hidden layers)."
]
},
{
@ -286,7 +283,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.9"
"version": "2.7.10"
}
},
"nbformat": 4,