anton/mlpractical
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #29 from pswietojanski/master

Browse Source 
solutions and coursework
This commit is contained in:
Pawel Swietojanski 2015-11-15 17:07:27 +00:00
commit 3c96e1957a
10 changed files with 2579 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1061
04_Regularisation_solution.ipynb Normal file
View File

File diff suppressed because one or more lines are too long

709
05_Transfer_functions_solution.ipynb Normal file
View File

@ -0,0 +1,709 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Introduction\n",
"\n",
"This tutorial focuses on implementation of alternatives to sigmoid transfer functions for hidden units. (*Transfer functions* are also called *activation functions* or *nonlinearities*.) First, we will work with hyperboilc tangent (tanh) and then unbounded (or partially unbounded) piecewise linear functions: Rectifying Linear Units (ReLU) and Maxout.\n",
"\n",
"\n",
"## Virtual environments\n",
"\n",
"Before you proceed onwards, remember to activate your virtual environment by typing `activate_mlp` or `source ~/mlpractical/venv/bin/activate` (or if you did the original install the \"comfy way\" type: `workon mlpractical`).\n",
"\n",
"\n",
"## Syncing the git repository\n",
"\n",
"Look <a href=\"https://github.com/CSTR-Edinburgh/mlpractical/blob/master/gitFAQ.md\">here</a> for more details. But in short, we recommend to create a separate branch for this lab, as follows:\n",
"\n",
"1. Enter the mlpractical directory `cd ~/mlpractical/repo-mlp`\n",
"2. List the branches and check which are currently active by typing: `git branch`\n",
"3. If you have followed our recommendations, you should be in the `lab4` branch, please commit your local changed to the repo index by typing:\n",
"```\n",
"git commit -am \"finished lab4\"\n",
"```\n",
"4. Now you can switch to `master` branch by typing: \n",
"```\n",
"git checkout master\n",
" ```\n",
"5. To update the repository (note, assuming master does not have any conflicts), if there are some, have a look <a href=\"https://github.com/CSTR-Edinburgh/mlpractical/blob/master/gitFAQ.md\">here</a>\n",
"```\n",
"git pull\n",
"```\n",
"6. And now, create the new branch & switch to it by typing:\n",
"```\n",
"git checkout -b lab5\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Overview of alternative transfer functions\n",
"\n",
"Now, we briefly summarise some other possible choices for hidden layer transfer functions.\n",
"\n",
"## Tanh\n",
"\n",
"Given a linear activation $a_{i}$ tanh implements the following operation:\n",
"\n",
"(1) $h_i(a_i) = \\mbox{tanh}(a_i) = \\frac{\\exp(a_i) - \\exp(-a_i)}{\\exp(a_i) + \\exp(-a_i)}$\n",
"\n",
"Hence, the derivative of $h_i$ with respect to $a_i$ is:\n",
"\n",
"(2) $\\begin{align}\n",
"\\frac{\\partial h_i}{\\partial a_i} &= 1 - h^2_i\n",
"\\end{align}\n",
"$\n",
"\n",
"\n",
"## ReLU\n",
"\n",
"Given a linear activation $a_{i}$ relu implements the following operation:\n",
"\n",
"(3) $h_i(a_i) = \\max(0, a_i)$\n",
"\n",
"Hence, the gradient is :\n",
"\n",
"(4) $\\begin{align}\n",
"\\frac{\\partial h_i}{\\partial a_i} &=\n",
"\\begin{cases}\n",
" 1 & \\quad \\text{if } a_i > 0 \\\\\n",
" 0 & \\quad \\text{if } a_i \\leq 0 \\\\\n",
"\\end{cases}\n",
"\\end{align}\n",
"$\n",
"\n",
"ReLU implements a form of data-driven sparsity, that is, on average the activations are sparse (many of them are 0) but the general sparsity pattern will depend on particular data-point. This is different from sparsity obtained in model's parameters one can obtain with $L1$ regularisation as the latter affect all data-points in the same way.\n",
"\n",
"## Maxout\n",
"\n",
"Maxout is an example of data-driven type of non-linearity in which the transfer function can be learned from data. That is, the model can build a non-linear transfer function from piecewise linear components. These linear components, depending on the number of linear regions used in the pooling operator (given by parameter $K$), can approximate arbitrary functions, such as ReLU, abs, etc.\n",
"\n",
"Given some subset (group, pool) of $K$ linear activations $a_{j}, a_{j+1}, \\ldots, a_{j+K}$ at the $l$-th layer, maxout implements the following operation:\n",
"\n",
"(5) $h_i(a_j, a_{j+1}, \\ldots, a_{j+K}) = \\max(a_j, a_{j+1}, \\ldots, a_{j+K})$\n",
"\n",
"Hence, the gradient of $h_i$ w.r.t to the pooling region $a_{j}, a_{j+1}, \\ldots, a_{j+K}$ is :\n",
"\n",
"(6) $\\begin{align}\n",
"\\frac{\\partial h_i}{\\partial (a_j, a_{j+1}, \\ldots, a_{j+K})} &=\n",
"\\begin{cases}\n",
" 1 & \\quad \\text{for the max activation} \\\\\n",
" 0 & \\quad \\text{otherwise} \\\\\n",
"\\end{cases}\n",
"\\end{align}\n",
"$\n",
"\n",
"Implementation tips are given in Exercise 3.\n",
"\n",
"# On weight initialisation\n",
"\n",
"Activation functions directly affect the \"network dynamics\", that is, the magnitudes of the statistics each layer is producing. For example, *slashing* non-linearities like sigmoid or tanh bring the linear activations to a certain bounded range. ReLU, on the contrary, has an unbounded positive side. This directly affects all statistics collected in forward and backward passes as well as the gradients w.r.t paramters - hence also the pace at which the model learns. That is why learning rate is usually required to be tuned for given the characterictics of the non-linearities used. \n",
"\n",
"Another important hyperparameter is the initial range used to initialise the weight matrices. We have largely ignored it so far (although if you did further experiments in coursework 1, you may have found setting it had an effect on training deeper networks with 4 or 5 hidden layers). However, for sigmoidal non-linearities (sigmoid, tanh) the initialisation range is an important hyperparameter and a considerable amount of research has been put into determining what is the best strategy for choosing it. In fact, one of the early triggers of the recent resurgence of deep learning was pre-training - techniques for initialising weights in an unsupervised manner so that one can effectively train deeper models in supervised fashion later. \n",
"\n",
"## Sigmoidal transfer functions\n",
"\n",
"Y. LeCun in [Efficient Backprop](http://link.springer.com/chapter/10.1007%2F3-540-49430-8_2) recommends the following setting of the initial range $r$ for sigmoidal units (assuming that the data has been normalised to zero mean, unit variance): \n",
"\n",
"(7) $ r = \\frac{1}{\\sqrt{N_{IN}}} $\n",
"\n",
"where $N_{IN}$ is the number of inputs to the given layer and the weights are then sampled from the (usually uniform) distribution $U(-r,r)$. The motivation is to keep the initial forward-pass signal in the linear region of the sigmoid non-linearity so that the gradients are large enough for training to proceed (note that the sigmoidal non-linearities saturate when activations are either very positive or very negative, leading to very small gradients and hence poor learning dynamics).\n",
"\n",
"The initialisation used in (7) however leads to different magnitudes of activations/gradients at different layers (due to multiplicative nature of the computations) and more recently, [Glorot et. al](http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf) proposed the so-called *normalised initialisation*, which ensures the variance of the forward signal (activations) is approximately the same in each layer. The same applies to the gradients obtained in backward pass. \n",
"\n",
"The $r$ in the *normalised initialisation* for $\\mbox{tanh}$ non-linearity is then:\n",
"\n",
"(8) $ r = \\frac{\\sqrt{6}}{\\sqrt{N_{IN}+N_{OUT}}} $\n",
"\n",
"For the sigmoid (logistic) non-linearity, to get similiar characteristics, one should scale $r$ in (8) by 4, that is:\n",
"\n",
"(9) $ r = \\frac{4\\sqrt{6}}{\\sqrt{N_{IN}+N_{OUT}}} $\n",
"\n",
"## Piece-wise linear transfer functions (ReLU, Maxout)\n",
"\n",
"For unbounded transfer functions initialisation is not as crucial as for sigmoidal ones. This is due to the fact that their gradients do not diminish (they are acutally more likely to explode) and they do not saturate (ReLU saturates at 0, but not on the positive slope, where gradient is 1 everywhere). (In practice ReLU is sometimes \"clipped\" with a maximum value, typically 20).\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 1: Implement the tanh transfer function\n",
"\n",
"Your implementation should follow the code conventions used to build other layer types (for example, Sigmoid and Softmax). Test your solution by training a one-hidden-layer model with 100 hidden units, similiar to the one used in Task 3a in the coursework. \n",
"\n",
"Tune the learning rate and compare the initial ranges in equations (7) and (8). Note that there might not be much difference for one-hidden-layer model, but you can easily notice a substantial gain from using (8) (or (9) for logistic sigmoid activation) for deeper models, for example, the 5 hidden-layer network from the first coursework.\n",
"\n",
"Implementation tip: Use numpy.tanh() to compute the non-linearity. Use the irange argument when creating the given layer type to provide the initial sampling range."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:root:Initialising data providers...\n"
]
}
],
"source": [
"import numpy\n",
"import logging\n",
"from mlp.dataset import MNISTDataProvider\n",
"\n",
"logger = logging.getLogger()\n",
"logger.setLevel(logging.INFO)\n",
"\n",
"# Note, you were asked to do run the experiments on all data and smaller models. \n",
"# Here I am running the exercises on 1000 training data-points only (similar to regularisation notebook)\n",
"logger.info('Initialising data providers...')\n",
"train_dp = MNISTDataProvider(dset='train', batch_size=10, max_num_batches=100, randomize=True)\n",
"valid_dp = MNISTDataProvider(dset='valid', batch_size=10000, max_num_batches=-10, randomize=False)\n",
"test_dp = MNISTDataProvider(dset='eval', batch_size=10000, max_num_batches=-10, randomize=False)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false,
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:root:Training started...\n",
"INFO:mlp.optimisers:Epoch 0: Training cost (ce) for initial model is 2.319. Accuracy is 10.50%\n",
"INFO:mlp.optimisers:Epoch 0: Validation cost (ce) for initial model is 2.315. Accuracy is 11.33%\n",
"INFO:mlp.optimisers:Epoch 1: Training cost (ce) is 1.048. Accuracy is 66.30%\n",
"INFO:mlp.optimisers:Epoch 1: Validation cost (ce) is 0.571. Accuracy is 82.72%\n",
"INFO:mlp.optimisers:Epoch 1: Took 2 seconds. Training speed 764 pps. Validation speed 12988 pps.\n",
"INFO:mlp.optimisers:Epoch 2: Training cost (ce) is 0.485. Accuracy is 84.40%\n",
"INFO:mlp.optimisers:Epoch 2: Validation cost (ce) is 0.455. Accuracy is 86.58%\n",
"INFO:mlp.optimisers:Epoch 2: Took 2 seconds. Training speed 720 pps. Validation speed 12988 pps.\n",
"INFO:mlp.optimisers:Epoch 3: Training cost (ce) is 0.362. Accuracy is 87.70%\n",
"INFO:mlp.optimisers:Epoch 3: Validation cost (ce) is 0.435. Accuracy is 86.90%\n",
"INFO:mlp.optimisers:Epoch 3: Took 2 seconds. Training speed 788 pps. Validation speed 12659 pps.\n",
"INFO:mlp.optimisers:Epoch 4: Training cost (ce) is 0.251. Accuracy is 92.10%\n",
"INFO:mlp.optimisers:Epoch 4: Validation cost (ce) is 0.417. Accuracy is 88.09%\n",
"INFO:mlp.optimisers:Epoch 4: Took 2 seconds. Training speed 788 pps. Validation speed 13159 pps.\n",
"INFO:mlp.optimisers:Epoch 5: Training cost (ce) is 0.175. Accuracy is 95.40%\n",
"INFO:mlp.optimisers:Epoch 5: Validation cost (ce) is 0.405. Accuracy is 88.16%\n",
"INFO:mlp.optimisers:Epoch 5: Took 2 seconds. Training speed 776 pps. Validation speed 12988 pps.\n",
"INFO:mlp.optimisers:Epoch 6: Training cost (ce) is 0.121. Accuracy is 96.40%\n",
"INFO:mlp.optimisers:Epoch 6: Validation cost (ce) is 0.458. Accuracy is 87.24%\n",
"INFO:mlp.optimisers:Epoch 6: Took 2 seconds. Training speed 690 pps. Validation speed 12659 pps.\n",
"INFO:mlp.optimisers:Epoch 7: Training cost (ce) is 0.091. Accuracy is 97.90%\n",
"INFO:mlp.optimisers:Epoch 7: Validation cost (ce) is 0.418. Accuracy is 88.37%\n",
"INFO:mlp.optimisers:Epoch 7: Took 2 seconds. Training speed 841 pps. Validation speed 12659 pps.\n",
"INFO:mlp.optimisers:Epoch 8: Training cost (ce) is 0.065. Accuracy is 98.70%\n",
"INFO:mlp.optimisers:Epoch 8: Validation cost (ce) is 0.400. Accuracy is 89.44%\n",
"INFO:mlp.optimisers:Epoch 8: Took 2 seconds. Training speed 794 pps. Validation speed 12501 pps.\n",
"INFO:mlp.optimisers:Epoch 9: Training cost (ce) is 0.043. Accuracy is 99.30%\n",
"INFO:mlp.optimisers:Epoch 9: Validation cost (ce) is 0.406. Accuracy is 89.35%\n",
"INFO:mlp.optimisers:Epoch 9: Took 2 seconds. Training speed 747 pps. Validation speed 12822 pps.\n",
"INFO:mlp.optimisers:Epoch 10: Training cost (ce) is 0.029. Accuracy is 99.50%\n",
"INFO:mlp.optimisers:Epoch 10: Validation cost (ce) is 0.410. Accuracy is 89.69%\n",
"INFO:mlp.optimisers:Epoch 10: Took 2 seconds. Training speed 953 pps. Validation speed 12822 pps.\n",
"INFO:mlp.optimisers:Epoch 11: Training cost (ce) is 0.023. Accuracy is 99.80%\n",
"INFO:mlp.optimisers:Epoch 11: Validation cost (ce) is 0.424. Accuracy is 89.41%\n",
"INFO:mlp.optimisers:Epoch 11: Took 2 seconds. Training speed 953 pps. Validation speed 13159 pps.\n",
"INFO:mlp.optimisers:Epoch 12: Training cost (ce) is 0.018. Accuracy is 99.80%\n",
"INFO:mlp.optimisers:Epoch 12: Validation cost (ce) is 0.429. Accuracy is 89.50%\n",
"INFO:mlp.optimisers:Epoch 12: Took 2 seconds. Training speed 870 pps. Validation speed 12988 pps.\n",
"INFO:mlp.optimisers:Epoch 13: Training cost (ce) is 0.015. Accuracy is 99.90%\n",
"INFO:mlp.optimisers:Epoch 13: Validation cost (ce) is 0.428. Accuracy is 89.58%\n",
"INFO:mlp.optimisers:Epoch 13: Took 2 seconds. Training speed 878 pps. Validation speed 12822 pps.\n",
"INFO:mlp.optimisers:Epoch 14: Training cost (ce) is 0.012. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 14: Validation cost (ce) is 0.436. Accuracy is 89.41%\n",
"INFO:mlp.optimisers:Epoch 14: Took 2 seconds. Training speed 894 pps. Validation speed 12501 pps.\n",
"INFO:mlp.optimisers:Epoch 15: Training cost (ce) is 0.010. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 15: Validation cost (ce) is 0.433. Accuracy is 89.64%\n",
"INFO:mlp.optimisers:Epoch 15: Took 2 seconds. Training speed 834 pps. Validation speed 12659 pps.\n",
"INFO:mlp.optimisers:Epoch 16: Training cost (ce) is 0.009. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 16: Validation cost (ce) is 0.439. Accuracy is 89.63%\n",
"INFO:mlp.optimisers:Epoch 16: Took 2 seconds. Training speed 820 pps. Validation speed 12988 pps.\n",
"INFO:mlp.optimisers:Epoch 17: Training cost (ce) is 0.008. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 17: Validation cost (ce) is 0.443. Accuracy is 89.78%\n",
"INFO:mlp.optimisers:Epoch 17: Took 2 seconds. Training speed 902 pps. Validation speed 12501 pps.\n",
"INFO:mlp.optimisers:Epoch 18: Training cost (ce) is 0.008. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 18: Validation cost (ce) is 0.446. Accuracy is 89.72%\n",
"INFO:mlp.optimisers:Epoch 18: Took 2 seconds. Training speed 870 pps. Validation speed 12659 pps.\n",
"INFO:mlp.optimisers:Epoch 19: Training cost (ce) is 0.007. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 19: Validation cost (ce) is 0.445. Accuracy is 89.83%\n",
"INFO:mlp.optimisers:Epoch 19: Took 2 seconds. Training speed 918 pps. Validation speed 12822 pps.\n",
"INFO:mlp.optimisers:Epoch 20: Training cost (ce) is 0.007. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 20: Validation cost (ce) is 0.451. Accuracy is 89.75%\n",
"INFO:mlp.optimisers:Epoch 20: Took 2 seconds. Training speed 834 pps. Validation speed 12659 pps.\n",
"INFO:mlp.optimisers:Epoch 21: Training cost (ce) is 0.006. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 21: Validation cost (ce) is 0.454. Accuracy is 89.80%\n",
"INFO:mlp.optimisers:Epoch 21: Took 2 seconds. Training speed 902 pps. Validation speed 12659 pps.\n",
"INFO:mlp.optimisers:Epoch 22: Training cost (ce) is 0.006. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 22: Validation cost (ce) is 0.456. Accuracy is 89.77%\n",
"INFO:mlp.optimisers:Epoch 22: Took 2 seconds. Training speed 863 pps. Validation speed 12501 pps.\n",
"INFO:mlp.optimisers:Epoch 23: Training cost (ce) is 0.005. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 23: Validation cost (ce) is 0.458. Accuracy is 89.84%\n",
"INFO:mlp.optimisers:Epoch 23: Took 2 seconds. Training speed 820 pps. Validation speed 12822 pps.\n",
"INFO:mlp.optimisers:Epoch 24: Training cost (ce) is 0.005. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 24: Validation cost (ce) is 0.460. Accuracy is 89.80%\n",
"INFO:mlp.optimisers:Epoch 24: Took 2 seconds. Training speed 856 pps. Validation speed 12988 pps.\n",
"INFO:mlp.optimisers:Epoch 25: Training cost (ce) is 0.005. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 25: Validation cost (ce) is 0.461. Accuracy is 89.86%\n",
"INFO:mlp.optimisers:Epoch 25: Took 2 seconds. Training speed 902 pps. Validation speed 12659 pps.\n",
"INFO:mlp.optimisers:Epoch 26: Training cost (ce) is 0.004. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 26: Validation cost (ce) is 0.467. Accuracy is 89.86%\n",
"INFO:mlp.optimisers:Epoch 26: Took 2 seconds. Training speed 910 pps. Validation speed 12659 pps.\n",
"INFO:mlp.optimisers:Epoch 27: Training cost (ce) is 0.004. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 27: Validation cost (ce) is 0.466. Accuracy is 89.81%\n",
"INFO:mlp.optimisers:Epoch 27: Took 2 seconds. Training speed 827 pps. Validation speed 12501 pps.\n",
"INFO:mlp.optimisers:Epoch 28: Training cost (ce) is 0.004. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 28: Validation cost (ce) is 0.468. Accuracy is 89.84%\n",
"INFO:mlp.optimisers:Epoch 28: Took 2 seconds. Training speed 894 pps. Validation speed 12501 pps.\n",
"INFO:mlp.optimisers:Epoch 29: Training cost (ce) is 0.004. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 29: Validation cost (ce) is 0.471. Accuracy is 89.83%\n",
"INFO:mlp.optimisers:Epoch 29: Took 2 seconds. Training speed 902 pps. Validation speed 12659 pps.\n",
"INFO:mlp.optimisers:Epoch 30: Training cost (ce) is 0.004. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 30: Validation cost (ce) is 0.473. Accuracy is 89.81%\n",
"INFO:mlp.optimisers:Epoch 30: Took 2 seconds. Training speed 918 pps. Validation speed 11495 pps.\n",
"INFO:root:Testing the model on test set:\n",
"INFO:root:MNIST test set accuracy is 89.33 %, cost (ce) is 0.480\n"
]
}
],
"source": [
"\n",
"from mlp.layers import MLP, Tanh, Softmax #import required layer types\n",
"from mlp.optimisers import SGDOptimiser #import the optimiser\n",
"from mlp.costs import CECost #import the cost we want to use for optimisation\n",
"from mlp.schedulers import LearningRateFixed\n",
"\n",
"rng = numpy.random.RandomState([2015,10,10])\n",
"\n",
"#some hyper-parameters\n",
"nhid = 100\n",
"learning_rate = 0.2\n",
"max_epochs = 30\n",
"cost = CECost()\n",
" \n",
"stats = []\n",
"for layer in xrange(1, 2):\n",
"\n",
" train_dp.reset()\n",
" valid_dp.reset()\n",
" test_dp.reset()\n",
" \n",
" #define the model\n",
" model = MLP(cost=cost)\n",
" model.add_layer(Tanh(idim=784, odim=nhid, irange=1./numpy.sqrt(784), rng=rng))\n",
" for i in xrange(1, layer):\n",
" logger.info(\"Stacking hidden layer (%s)\" % str(i+1))\n",
" model.add_layer(Tanh(idim=nhid, odim=nhid, irange=0.2, rng=rng))\n",
" model.add_layer(Softmax(idim=nhid, odim=10, rng=rng))\n",
"\n",
" # define the optimiser, here stochasitc gradient descent\n",
" # with fixed learning rate and max_epochs\n",
" lr_scheduler = LearningRateFixed(learning_rate=learning_rate, max_epochs=max_epochs)\n",
" optimiser = SGDOptimiser(lr_scheduler=lr_scheduler)\n",
"\n",
" logger.info('Training started...')\n",
" tr_stats, valid_stats = optimiser.train(model, train_dp, valid_dp)\n",
"\n",
" logger.info('Testing the model on test set:')\n",
" tst_cost, tst_accuracy = optimiser.validate(model, test_dp)\n",
" logger.info('MNIST test set accuracy is %.2f %%, cost (%s) is %.3f'%(tst_accuracy*100., cost.get_name(), tst_cost))\n",
" \n",
" stats.append((tr_stats, valid_stats, (tst_cost, tst_accuracy)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 2: Implement ReLU\n",
"\n",
"Again, your implementation should follow the conventions used to build Linear, Sigmoid and Softmax layers. As in exercise 1, test your solution by training a one-hidden-layer model with 100 hidden units, similiar to the one used in Task 3a in the coursework. Tune the learning rate (start with the initial one set to 0.1) with the initial weight range set to 0.05."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false,
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:root:Training started...\n",
"INFO:mlp.optimisers:Epoch 0: Training cost (ce) for initial model is 2.317. Accuracy is 15.20%\n",
"INFO:mlp.optimisers:Epoch 0: Validation cost (ce) for initial model is 2.317. Accuracy is 13.98%\n",
"INFO:mlp.optimisers:Epoch 1: Training cost (ce) is 1.452. Accuracy is 60.20%\n",
"INFO:mlp.optimisers:Epoch 1: Validation cost (ce) is 0.750. Accuracy is 81.69%\n",
"INFO:mlp.optimisers:Epoch 1: Took 2 seconds. Training speed 820 pps. Validation speed 13335 pps.\n",
"INFO:mlp.optimisers:Epoch 2: Training cost (ce) is 0.632. Accuracy is 82.40%\n",
"INFO:mlp.optimisers:Epoch 2: Validation cost (ce) is 0.503. Accuracy is 86.74%\n",
"INFO:mlp.optimisers:Epoch 2: Took 2 seconds. Training speed 788 pps. Validation speed 13335 pps.\n",
"INFO:mlp.optimisers:Epoch 3: Training cost (ce) is 0.446. Accuracy is 87.50%\n",
"INFO:mlp.optimisers:Epoch 3: Validation cost (ce) is 0.438. Accuracy is 87.24%\n",
"INFO:mlp.optimisers:Epoch 3: Took 2 seconds. Training speed 788 pps. Validation speed 13159 pps.\n",
"INFO:mlp.optimisers:Epoch 4: Training cost (ce) is 0.359. Accuracy is 90.00%\n",
"INFO:mlp.optimisers:Epoch 4: Validation cost (ce) is 0.444. Accuracy is 86.44%\n",
"INFO:mlp.optimisers:Epoch 4: Took 2 seconds. Training speed 710 pps. Validation speed 12822 pps.\n",
"INFO:mlp.optimisers:Epoch 5: Training cost (ce) is 0.304. Accuracy is 90.80%\n",
"INFO:mlp.optimisers:Epoch 5: Validation cost (ce) is 0.408. Accuracy is 87.90%\n",
"INFO:mlp.optimisers:Epoch 5: Took 2 seconds. Training speed 782 pps. Validation speed 13335 pps.\n",
"INFO:mlp.optimisers:Epoch 6: Training cost (ce) is 0.255. Accuracy is 93.80%\n",
"INFO:mlp.optimisers:Epoch 6: Validation cost (ce) is 0.390. Accuracy is 88.56%\n",
"INFO:mlp.optimisers:Epoch 6: Took 2 seconds. Training speed 782 pps. Validation speed 13515 pps.\n",
"INFO:mlp.optimisers:Epoch 7: Training cost (ce) is 0.225. Accuracy is 93.80%\n",
"INFO:mlp.optimisers:Epoch 7: Validation cost (ce) is 0.425. Accuracy is 87.46%\n",
"INFO:mlp.optimisers:Epoch 7: Took 2 seconds. Training speed 725 pps. Validation speed 13890 pps.\n",
"INFO:mlp.optimisers:Epoch 8: Training cost (ce) is 0.205. Accuracy is 95.00%\n",
"INFO:mlp.optimisers:Epoch 8: Validation cost (ce) is 0.399. Accuracy is 88.51%\n",
"INFO:mlp.optimisers:Epoch 8: Took 2 seconds. Training speed 834 pps. Validation speed 13335 pps.\n",
"INFO:mlp.optimisers:Epoch 9: Training cost (ce) is 0.163. Accuracy is 96.20%\n",
"INFO:mlp.optimisers:Epoch 9: Validation cost (ce) is 0.474. Accuracy is 85.74%\n",
"INFO:mlp.optimisers:Epoch 9: Took 2 seconds. Training speed 814 pps. Validation speed 13700 pps.\n",
"INFO:mlp.optimisers:Epoch 10: Training cost (ce) is 0.140. Accuracy is 96.40%\n",
"INFO:mlp.optimisers:Epoch 10: Validation cost (ce) is 0.418. Accuracy is 88.06%\n",
"INFO:mlp.optimisers:Epoch 10: Took 2 seconds. Training speed 788 pps. Validation speed 12988 pps.\n",
"INFO:mlp.optimisers:Epoch 11: Training cost (ce) is 0.120. Accuracy is 97.70%\n",
"INFO:mlp.optimisers:Epoch 11: Validation cost (ce) is 0.427. Accuracy is 87.93%\n",
"INFO:mlp.optimisers:Epoch 11: Took 2 seconds. Training speed 731 pps. Validation speed 13335 pps.\n",
"INFO:mlp.optimisers:Epoch 12: Training cost (ce) is 0.105. Accuracy is 98.10%\n",
"INFO:mlp.optimisers:Epoch 12: Validation cost (ce) is 0.449. Accuracy is 87.51%\n",
"INFO:mlp.optimisers:Epoch 12: Took 2 seconds. Training speed 725 pps. Validation speed 12988 pps.\n",
"INFO:mlp.optimisers:Epoch 13: Training cost (ce) is 0.088. Accuracy is 98.50%\n",
"INFO:mlp.optimisers:Epoch 13: Validation cost (ce) is 0.479. Accuracy is 87.14%\n",
"INFO:mlp.optimisers:Epoch 13: Took 2 seconds. Training speed 715 pps. Validation speed 12822 pps.\n",
"INFO:mlp.optimisers:Epoch 14: Training cost (ce) is 0.086. Accuracy is 98.30%\n",
"INFO:mlp.optimisers:Epoch 14: Validation cost (ce) is 0.455. Accuracy is 87.97%\n",
"INFO:mlp.optimisers:Epoch 14: Took 2 seconds. Training speed 681 pps. Validation speed 13515 pps.\n",
"INFO:mlp.optimisers:Epoch 15: Training cost (ce) is 0.070. Accuracy is 99.00%\n",
"INFO:mlp.optimisers:Epoch 15: Validation cost (ce) is 0.465. Accuracy is 87.76%\n",
"INFO:mlp.optimisers:Epoch 15: Took 2 seconds. Training speed 758 pps. Validation speed 12988 pps.\n",
"INFO:mlp.optimisers:Epoch 16: Training cost (ce) is 0.054. Accuracy is 99.50%\n",
"INFO:mlp.optimisers:Epoch 16: Validation cost (ce) is 0.467. Accuracy is 88.07%\n",
"INFO:mlp.optimisers:Epoch 16: Took 2 seconds. Training speed 776 pps. Validation speed 12501 pps.\n",
"INFO:mlp.optimisers:Epoch 17: Training cost (ce) is 0.052. Accuracy is 99.60%\n",
"INFO:mlp.optimisers:Epoch 17: Validation cost (ce) is 0.485. Accuracy is 87.69%\n",
"INFO:mlp.optimisers:Epoch 17: Took 2 seconds. Training speed 801 pps. Validation speed 13159 pps.\n",
"INFO:mlp.optimisers:Epoch 18: Training cost (ce) is 0.042. Accuracy is 99.70%\n",
"INFO:mlp.optimisers:Epoch 18: Validation cost (ce) is 0.500. Accuracy is 87.61%\n",
"INFO:mlp.optimisers:Epoch 18: Took 2 seconds. Training speed 686 pps. Validation speed 13335 pps.\n",
"INFO:mlp.optimisers:Epoch 19: Training cost (ce) is 0.035. Accuracy is 99.80%\n",
"INFO:mlp.optimisers:Epoch 19: Validation cost (ce) is 0.499. Accuracy is 87.76%\n",
"INFO:mlp.optimisers:Epoch 19: Took 2 seconds. Training speed 764 pps. Validation speed 12822 pps.\n",
"INFO:mlp.optimisers:Epoch 20: Training cost (ce) is 0.031. Accuracy is 99.80%\n",
"INFO:mlp.optimisers:Epoch 20: Validation cost (ce) is 0.506. Accuracy is 87.77%\n",
"INFO:mlp.optimisers:Epoch 20: Took 2 seconds. Training speed 801 pps. Validation speed 13159 pps.\n",
"INFO:mlp.optimisers:Epoch 21: Training cost (ce) is 0.027. Accuracy is 99.90%\n",
"INFO:mlp.optimisers:Epoch 21: Validation cost (ce) is 0.506. Accuracy is 87.61%\n",
"INFO:mlp.optimisers:Epoch 21: Took 2 seconds. Training speed 731 pps. Validation speed 13515 pps.\n",
"INFO:mlp.optimisers:Epoch 22: Training cost (ce) is 0.025. Accuracy is 99.80%\n",
"INFO:mlp.optimisers:Epoch 22: Validation cost (ce) is 0.516. Accuracy is 87.68%\n",
"INFO:mlp.optimisers:Epoch 22: Took 2 seconds. Training speed 758 pps. Validation speed 13335 pps.\n",
"INFO:mlp.optimisers:Epoch 23: Training cost (ce) is 0.022. Accuracy is 99.90%\n",
"INFO:mlp.optimisers:Epoch 23: Validation cost (ce) is 0.529. Accuracy is 87.33%\n",
"INFO:mlp.optimisers:Epoch 23: Took 2 seconds. Training speed 770 pps. Validation speed 13159 pps.\n",
"INFO:mlp.optimisers:Epoch 24: Training cost (ce) is 0.020. Accuracy is 99.90%\n",
"INFO:mlp.optimisers:Epoch 24: Validation cost (ce) is 0.526. Accuracy is 87.70%\n",
"INFO:mlp.optimisers:Epoch 24: Took 2 seconds. Training speed 715 pps. Validation speed 13700 pps.\n",
"INFO:mlp.optimisers:Epoch 25: Training cost (ce) is 0.018. Accuracy is 99.90%\n",
"INFO:mlp.optimisers:Epoch 25: Validation cost (ce) is 0.535. Accuracy is 87.55%\n",
"INFO:mlp.optimisers:Epoch 25: Took 2 seconds. Training speed 770 pps. Validation speed 13159 pps.\n",
"INFO:mlp.optimisers:Epoch 26: Training cost (ce) is 0.016. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 26: Validation cost (ce) is 0.540. Accuracy is 87.55%\n",
"INFO:mlp.optimisers:Epoch 26: Took 2 seconds. Training speed 741 pps. Validation speed 13515 pps.\n",
"INFO:mlp.optimisers:Epoch 27: Training cost (ce) is 0.015. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 27: Validation cost (ce) is 0.546. Accuracy is 87.57%\n",
"INFO:mlp.optimisers:Epoch 27: Took 2 seconds. Training speed 681 pps. Validation speed 13515 pps.\n",
"INFO:mlp.optimisers:Epoch 28: Training cost (ce) is 0.014. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 28: Validation cost (ce) is 0.546. Accuracy is 87.78%\n",
"INFO:mlp.optimisers:Epoch 28: Took 2 seconds. Training speed 753 pps. Validation speed 13700 pps.\n",
"INFO:mlp.optimisers:Epoch 29: Training cost (ce) is 0.012. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 29: Validation cost (ce) is 0.556. Accuracy is 87.56%\n",
"INFO:mlp.optimisers:Epoch 29: Took 2 seconds. Training speed 758 pps. Validation speed 13700 pps.\n",
"INFO:mlp.optimisers:Epoch 30: Training cost (ce) is 0.012. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 30: Validation cost (ce) is 0.558. Accuracy is 87.74%\n",
"INFO:mlp.optimisers:Epoch 30: Took 2 seconds. Training speed 747 pps. Validation speed 13515 pps.\n",
"INFO:root:Testing the model on test set:\n",
"INFO:root:MNIST test set accuracy is 87.19 %, cost (ce) is 0.554\n"
]
}
],
"source": [
"\n",
"from mlp.layers import MLP, Relu, Softmax \n",
"from mlp.optimisers import SGDOptimiser \n",
"from mlp.costs import CECost \n",
"from mlp.schedulers import LearningRateFixed\n",
"\n",
"rng = numpy.random.RandomState([2015,10,10])\n",
"\n",
"#some hyper-parameters\n",
"nhid = 100\n",
"learning_rate = 0.1\n",
"max_epochs = 30\n",
"cost = CECost()\n",
" \n",
"stats = []\n",
"for layer in xrange(1, 2):\n",
"\n",
" train_dp.reset()\n",
" valid_dp.reset()\n",
" test_dp.reset()\n",
" \n",
" #define the model\n",
" model = MLP(cost=cost)\n",
" model.add_layer(Relu(idim=784, odim=nhid, irange=0.05, rng=rng))\n",
" for i in xrange(1, layer):\n",
" logger.info(\"Stacking hidden layer (%s)\" % str(i+1))\n",
" model.add_layer(Relu(idim=nhid, odim=nhid, irange=0.2, rng=rng))\n",
" model.add_layer(Softmax(idim=nhid, odim=10, rng=rng))\n",
"\n",
" # define the optimiser, here stochasitc gradient descent\n",
" # with fixed learning rate and max_epochs\n",
" lr_scheduler = LearningRateFixed(learning_rate=learning_rate, max_epochs=max_epochs)\n",
" optimiser = SGDOptimiser(lr_scheduler=lr_scheduler)\n",
"\n",
" logger.info('Training started...')\n",
" tr_stats, valid_stats = optimiser.train(model, train_dp, valid_dp)\n",
"\n",
" logger.info('Testing the model on test set:')\n",
" tst_cost, tst_accuracy = optimiser.validate(model, test_dp)\n",
" logger.info('MNIST test set accuracy is %.2f %%, cost (%s) is %.3f'%(tst_accuracy*100., cost.get_name(), tst_cost))\n",
" \n",
" stats.append((tr_stats, valid_stats, (tst_cost, tst_accuracy)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 3: Implement Maxout\n",
"\n",
"As with the previous two exercises, your implementation should follow the conventions used to build the Linear, Sigmoid and Softmax layers. For now implement only non-overlapping pools (i.e. the pool in which all activations $a_{j}, a_{j+1}, \\ldots, a_{j+K}$ belong to only one pool). As before, test your solution by training a one-hidden-layer model with 100 hidden units, similiar to the one used in Task 3a in the coursework. Use the same optimisation hyper-parameters (learning rate, initial weights range) as you used for ReLU models. Tune the pool size $K$ (but keep the number of total parameters fixed).\n",
"\n",
"Note: The Max operator reduces dimensionality, hence for example, to get 100 hidden maxout units with pooling size set to $K=2$ the size of linear part needs to be set to $100K$ (assuming non-overlapping pools). This affects how you compute the total number of weights in the model.\n",
"\n",
"Implementation tips: To back-propagate through the maxout layer, one needs to keep track of which linear activation $a_{j}, a_{j+1}, \\ldots, a_{j+K}$ was the maximum in each pool. The convenient way to do so is by storing the indices of the maximum units in the fprop function and then in the backprop stage pass the gradient only through those (i.e. for example, one can build an auxiliary matrix where each element is either 1 (if unit was maximum, and passed forward through the max operator for a given data-point) or 0 otherwise. Then in the backward pass it suffices to upsample the maxout *igrads* signal to the linear layer dimension and element-wise multiply by the aforemenioned auxiliary matrix.\n",
"\n",
"*Optional:* Implement the generic pooling mechanism by introducing an additional *stride* hyper-parameter $0<S\\leq K$. It specifies how many units you move to build the next pool. For instance, for non-overlapping pooling with $S=K=3$ one would build the first two maxout units as: $h_1=\\max(a_1,a_2,a_3)$ and $h_2=\\max(a_4,a_5,a_6)$. However, after setting $S=1$ the pools should share some subset of linear activations: $h_1=\\max(a_1,a_2,a_3)$ and $h_2=\\max(a_2,a_3,a_4)$."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false,
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"ERROR: Line magic function `%autorelaod` not found.\n",
"INFO:root:Training started...\n",
"INFO:mlp.optimisers:Epoch 0: Training cost (ce) for initial model is 2.314. Accuracy is 9.30%\n",
"INFO:mlp.optimisers:Epoch 0: Validation cost (ce) for initial model is 2.323. Accuracy is 8.27%\n",
"INFO:mlp.optimisers:Epoch 1: Training cost (ce) is 1.206. Accuracy is 64.20%\n",
"INFO:mlp.optimisers:Epoch 1: Validation cost (ce) is 0.628. Accuracy is 79.70%\n",
"INFO:mlp.optimisers:Epoch 1: Took 9 seconds. Training speed 394 pps. Validation speed 1527 pps.\n",
"INFO:mlp.optimisers:Epoch 2: Training cost (ce) is 0.514. Accuracy is 85.80%\n",
"INFO:mlp.optimisers:Epoch 2: Validation cost (ce) is 0.429. Accuracy is 88.16%\n",
"INFO:mlp.optimisers:Epoch 2: Took 9 seconds. Training speed 361 pps. Validation speed 1532 pps.\n",
"INFO:mlp.optimisers:Epoch 3: Training cost (ce) is 0.355. Accuracy is 89.70%\n",
"INFO:mlp.optimisers:Epoch 3: Validation cost (ce) is 0.407. Accuracy is 87.77%\n",
"INFO:mlp.optimisers:Epoch 3: Took 10 seconds. Training speed 422 pps. Validation speed 1387 pps.\n",
"INFO:mlp.optimisers:Epoch 4: Training cost (ce) is 0.262. Accuracy is 92.30%\n",
"INFO:mlp.optimisers:Epoch 4: Validation cost (ce) is 0.387. Accuracy is 88.78%\n",
"INFO:mlp.optimisers:Epoch 4: Took 9 seconds. Training speed 441 pps. Validation speed 1488 pps.\n",
"INFO:mlp.optimisers:Epoch 5: Training cost (ce) is 0.194. Accuracy is 94.70%\n",
"INFO:mlp.optimisers:Epoch 5: Validation cost (ce) is 0.349. Accuracy is 89.86%\n",
"INFO:mlp.optimisers:Epoch 5: Took 9 seconds. Training speed 389 pps. Validation speed 1527 pps.\n",
"INFO:mlp.optimisers:Epoch 6: Training cost (ce) is 0.134. Accuracy is 97.50%\n",
"INFO:mlp.optimisers:Epoch 6: Validation cost (ce) is 0.347. Accuracy is 89.79%\n",
"INFO:mlp.optimisers:Epoch 6: Took 9 seconds. Training speed 426 pps. Validation speed 1497 pps.\n",
"INFO:mlp.optimisers:Epoch 7: Training cost (ce) is 0.094. Accuracy is 98.70%\n",
"INFO:mlp.optimisers:Epoch 7: Validation cost (ce) is 0.429. Accuracy is 87.88%\n",
"INFO:mlp.optimisers:Epoch 7: Took 9 seconds. Training speed 449 pps. Validation speed 1473 pps.\n",
"INFO:mlp.optimisers:Epoch 8: Training cost (ce) is 0.071. Accuracy is 99.10%\n",
"INFO:mlp.optimisers:Epoch 8: Validation cost (ce) is 0.345. Accuracy is 90.31%\n",
"INFO:mlp.optimisers:Epoch 8: Took 9 seconds. Training speed 455 pps. Validation speed 1508 pps.\n",
"INFO:mlp.optimisers:Epoch 9: Training cost (ce) is 0.053. Accuracy is 99.40%\n",
"INFO:mlp.optimisers:Epoch 9: Validation cost (ce) is 0.357. Accuracy is 90.00%\n",
"INFO:mlp.optimisers:Epoch 9: Took 9 seconds. Training speed 375 pps. Validation speed 1532 pps.\n",
"INFO:mlp.optimisers:Epoch 10: Training cost (ce) is 0.042. Accuracy is 99.50%\n",
"INFO:mlp.optimisers:Epoch 10: Validation cost (ce) is 0.356. Accuracy is 90.27%\n",
"INFO:mlp.optimisers:Epoch 10: Took 9 seconds. Training speed 421 pps. Validation speed 1525 pps.\n",
"INFO:mlp.optimisers:Epoch 11: Training cost (ce) is 0.031. Accuracy is 99.70%\n",
"INFO:mlp.optimisers:Epoch 11: Validation cost (ce) is 0.347. Accuracy is 90.57%\n",
"INFO:mlp.optimisers:Epoch 11: Took 9 seconds. Training speed 449 pps. Validation speed 1522 pps.\n",
"INFO:mlp.optimisers:Epoch 12: Training cost (ce) is 0.026. Accuracy is 99.70%\n",
"INFO:mlp.optimisers:Epoch 12: Validation cost (ce) is 0.353. Accuracy is 90.50%\n",
"INFO:mlp.optimisers:Epoch 12: Took 9 seconds. Training speed 449 pps. Validation speed 1504 pps.\n",
"INFO:mlp.optimisers:Epoch 13: Training cost (ce) is 0.021. Accuracy is 99.90%\n",
"INFO:mlp.optimisers:Epoch 13: Validation cost (ce) is 0.352. Accuracy is 90.51%\n",
"INFO:mlp.optimisers:Epoch 13: Took 9 seconds. Training speed 441 pps. Validation speed 1495 pps.\n",
"INFO:mlp.optimisers:Epoch 14: Training cost (ce) is 0.018. Accuracy is 99.90%\n",
"INFO:mlp.optimisers:Epoch 14: Validation cost (ce) is 0.355. Accuracy is 90.59%\n",
"INFO:mlp.optimisers:Epoch 14: Took 9 seconds. Training speed 410 pps. Validation speed 1456 pps.\n",
"INFO:mlp.optimisers:Epoch 15: Training cost (ce) is 0.015. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 15: Validation cost (ce) is 0.359. Accuracy is 90.66%\n",
"INFO:mlp.optimisers:Epoch 15: Took 9 seconds. Training speed 463 pps. Validation speed 1429 pps.\n",
"INFO:mlp.optimisers:Epoch 16: Training cost (ce) is 0.013. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 16: Validation cost (ce) is 0.363. Accuracy is 90.52%\n",
"INFO:mlp.optimisers:Epoch 16: Took 10 seconds. Training speed 365 pps. Validation speed 1403 pps.\n",
"INFO:mlp.optimisers:Epoch 17: Training cost (ce) is 0.012. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 17: Validation cost (ce) is 0.364. Accuracy is 90.71%\n",
"INFO:mlp.optimisers:Epoch 17: Took 10 seconds. Training speed 351 pps. Validation speed 1368 pps.\n",
"INFO:mlp.optimisers:Epoch 18: Training cost (ce) is 0.011. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 18: Validation cost (ce) is 0.364. Accuracy is 90.65%\n",
"INFO:mlp.optimisers:Epoch 18: Took 10 seconds. Training speed 348 pps. Validation speed 1439 pps.\n",
"INFO:mlp.optimisers:Epoch 19: Training cost (ce) is 0.010. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 19: Validation cost (ce) is 0.367. Accuracy is 90.62%\n",
"INFO:mlp.optimisers:Epoch 19: Took 11 seconds. Training speed 271 pps. Validation speed 1441 pps.\n",
"INFO:mlp.optimisers:Epoch 20: Training cost (ce) is 0.009. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 20: Validation cost (ce) is 0.366. Accuracy is 90.78%\n",
"INFO:mlp.optimisers:Epoch 20: Took 10 seconds. Training speed 309 pps. Validation speed 1387 pps.\n",
"INFO:mlp.optimisers:Epoch 21: Training cost (ce) is 0.008. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 21: Validation cost (ce) is 0.371. Accuracy is 90.66%\n",
"INFO:mlp.optimisers:Epoch 21: Took 10 seconds. Training speed 348 pps. Validation speed 1323 pps.\n",
"INFO:mlp.optimisers:Epoch 22: Training cost (ce) is 0.008. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 22: Validation cost (ce) is 0.370. Accuracy is 90.68%\n",
"INFO:mlp.optimisers:Epoch 22: Took 9 seconds. Training speed 435 pps. Validation speed 1488 pps.\n",
"INFO:mlp.optimisers:Epoch 23: Training cost (ce) is 0.007. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 23: Validation cost (ce) is 0.372. Accuracy is 90.70%\n",
"INFO:mlp.optimisers:Epoch 23: Took 9 seconds. Training speed 405 pps. Validation speed 1443 pps.\n",
"INFO:mlp.optimisers:Epoch 24: Training cost (ce) is 0.007. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 24: Validation cost (ce) is 0.373. Accuracy is 90.80%\n",
"INFO:mlp.optimisers:Epoch 24: Took 9 seconds. Training speed 389 pps. Validation speed 1482 pps.\n",
"INFO:mlp.optimisers:Epoch 25: Training cost (ce) is 0.006. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 25: Validation cost (ce) is 0.375. Accuracy is 90.71%\n",
"INFO:mlp.optimisers:Epoch 25: Took 9 seconds. Training speed 402 pps. Validation speed 1525 pps.\n",
"INFO:mlp.optimisers:Epoch 26: Training cost (ce) is 0.006. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 26: Validation cost (ce) is 0.380. Accuracy is 90.65%\n",
"INFO:mlp.optimisers:Epoch 26: Took 9 seconds. Training speed 405 pps. Validation speed 1522 pps.\n",
"INFO:mlp.optimisers:Epoch 27: Training cost (ce) is 0.006. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 27: Validation cost (ce) is 0.380. Accuracy is 90.75%\n",
"INFO:mlp.optimisers:Epoch 27: Took 9 seconds. Training speed 415 pps. Validation speed 1534 pps.\n",
"INFO:mlp.optimisers:Epoch 28: Training cost (ce) is 0.005. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 28: Validation cost (ce) is 0.381. Accuracy is 90.66%\n",
"INFO:mlp.optimisers:Epoch 28: Took 9 seconds. Training speed 410 pps. Validation speed 1493 pps.\n",
"INFO:mlp.optimisers:Epoch 29: Training cost (ce) is 0.005. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 29: Validation cost (ce) is 0.382. Accuracy is 90.67%\n",
"INFO:mlp.optimisers:Epoch 29: Took 9 seconds. Training speed 396 pps. Validation speed 1536 pps.\n",
"INFO:mlp.optimisers:Epoch 30: Training cost (ce) is 0.005. Accuracy is 100.00%\n",
"INFO:mlp.optimisers:Epoch 30: Validation cost (ce) is 0.384. Accuracy is 90.75%\n",
"INFO:mlp.optimisers:Epoch 30: Took 9 seconds. Training speed 463 pps. Validation speed 1532 pps.\n",
"INFO:root:Testing the model on test set:\n",
"INFO:root:MNIST test set accuracy is 90.02 %, cost (ce) is 0.391\n"
]
}
],
"source": [
"\n",
"from mlp.layers import MLP, Maxout, Softmax \n",
"from mlp.optimisers import SGDOptimiser\n",
"from mlp.costs import CECost \n",
"from mlp.schedulers import LearningRateFixed\n",
"\n",
"#some hyper-parameters\n",
"nhid = 100\n",
"learning_rate = 0.1\n",
"k = 2 #maxout pool size (stride is assumed k)\n",
"max_epochs = 30\n",
"cost = CECost()\n",
" \n",
"stats = []\n",
"for layer in xrange(1, 2):\n",
"\n",
" train_dp.reset()\n",
" valid_dp.reset()\n",
" test_dp.reset()\n",
" \n",
" #define the model\n",
" model = MLP(cost=cost)\n",
" model.add_layer(Maxout(idim=784, odim=nhid, k=k, irange=0.05, rng=rng))\n",
" for i in xrange(1, layer):\n",
" logger.info(\"Stacking hidden layer (%s)\" % str(i+1))\n",
" model.add_layer(Maxout(idim=nhid, odim=nhid, k=k, irange=0.2, rng=rng))\n",
" model.add_layer(Softmax(idim=nhid, odim=10, rng=rng))\n",
"\n",
" # define the optimiser, here stochasitc gradient descent\n",
" # with fixed learning rate and max_epochs\n",
" lr_scheduler = LearningRateFixed(learning_rate=learning_rate, max_epochs=max_epochs)\n",
" optimiser = SGDOptimiser(lr_scheduler=lr_scheduler)\n",
"\n",
" logger.info('Training started...')\n",
" tr_stats, valid_stats = optimiser.train(model, train_dp, valid_dp)\n",
"\n",
" logger.info('Testing the model on test set:')\n",
" tst_cost, tst_accuracy = optimiser.validate(model, test_dp)\n",
" logger.info('MNIST test set accuracy is %.2f %%, cost (%s) is %.3f'%(tst_accuracy*100., cost.get_name(), tst_cost))\n",
" \n",
" stats.append((tr_stats, valid_stats, (tst_cost, tst_accuracy)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercise 4: Train all the above models with dropout\n",
"\n",
"Try all of the above non-linearities with dropout training. Use the dropout hyper-parameters $\\{p_{inp}, p_{hid}\\}$ that worked best for sigmoid models from the previous lab.\n",
"\n",
"Note: the code for dropout you were asked to implement last week has not been given as a solution for this week - as a result you need to move/merge the required dropout parts from your previous *lab4* branch (or implement it if you haven't already done so). \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#This one is a simple merge of above experiments with last exercise in previous tutorial."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 0
}

37
06_MLP_Coursework2_Introduction.ipynb
View File

@ -8,6 +8,33 @@
"\n",
"This notebook contains some extended versions of hints and some code examples that are suppose to make it easier to proceed with certain tasks in Coursework #2.\n",
"\n",
"## Virtual environments\n",
"\n",
"Before you proceed onwards, remember to activate your virtual environment by typing `activate_mlp` or `source ~/mlpractical/venv/bin/activate` (or if you did the original install the \"comfy way\" type: `workon mlpractical`).\n",
"\n",
"## Syncing the git repository\n",
"\n",
"Look <a href=\"https://github.com/CSTR-Edinburgh/mlpractical/blob/master/gitFAQ.md\">here</a> for more details. But in short, we recommend to create a separate branch for the coursework, as follows:\n",
"\n",
"1. Enter the mlpractical directory `cd ~/mlpractical/repo-mlp`\n",
"2. List the branches and check which are currently active by typing: `git branch`\n",
"3. If you have followed our recommendations, you should be in the `lab5` branch, please commit your local changes to the repo index by typing:\n",
"```\n",
"git commit -am \"finished lab5\"\n",
"```\n",
"4. Now you can switch to `master` branch by typing: \n",
"```\n",
"git checkout master\n",
" ```\n",
"5. To update the repository (note, assuming master does not have any conflicts), if there are some, have a look <a href=\"https://github.com/CSTR-Edinburgh/mlpractical/blob/master/gitFAQ.md\">here</a>\n",
"```\n",
"git pull\n",
"```\n",
"6. And now, create the new branch & switch to it by typing:\n",
"```\n",
"git checkout -b coursework2\n",
"```\n",
"\n",
"# Store the intermediate results (check-pointing and pickling)\n",
"\n",
"Once you have finished a task it is a good idea to check-point your current notebook's status (logs, plots and whatever else has been stored in the notebook). By doing this, you can always revert to this state later when necessary. You can do this by going to menus `File->Save and Checkpoint` and `File->Revert to Checkpoint`.\n",
@ -59,13 +86,15 @@
"* `numpy.amax` - the same as with sum\n",
"* `numpy.transpose` - can specify which axes you want to get transposed in a tensor\n",
"* `numpy.argmax` - gives you the argument (index) of the maximum value in a tensor\n",
"* `numpy.flatten` - collapses the n-dimensional tensor into vector\n",
"* `numpy.flatten` - collapses the n-dimensional tensor into vector (copy)\n",
"* `numpy.ravel` - collapses the n-dimensional tensor into vector (creates a view)\n",
"* `numpy.reshape` - allows to reshape a tensor into another (valid from data perspective) tensor (matrix, vector) with a different shape (but the same number of total elements)\n",
"* `numpy.rot90(m, k)` - rotate matrix `m` by 90 degrees `k` times (counter-clockwise)\n",
"* `numpy.newaxis` - add an axis with dimension 1 (handy for keeping tensor shapes compatible with expected broadcasting)\n",
"* `numpy.rollaxis` - roll an axis in a tensor\n",
"* `slice` - allows to specify a range (can be used when indexing numpy arrays)\n",
"* `ellipsis` - allows to pick an arbitrary number of dimensions (inferred)\n",
"* `max_and_argmax` - `(mlp.layers)` - an auxiliary function we have provided to get both max and argmax of a tensor across an arbitrary axes, possibly in the format preserving tensor's original shape (this is not trivial to do using numpy *out-of-the-shelf* functionality).\n",
"\n",
"The below cells contain some simple examples showing the basics of tensor manipulation in numpy (go through them if you haven't used numpy in this context before)."
]
@ -230,7 +259,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"You can also check the backprop implementation in the layer. Notice, it **does not** necessarily check whether your layer implementation is correct but rather if the gradient computation is correct, given the forward pass computation. If you get the forward pass wrong, and somehow got the gradients right w.r.t what the forward pass is computing, the below check will not capture it (obviously). "
"You can also check the backprop implementation in the layer. Notice, it **does not** necessarily check whether your layer implementation is correct but rather if the gradient computation is correct, given the forward pass computation. If you get the forward pass wrong, and somehow got the gradients right w.r.t what the forward pass is computing, the below check will not capture it (obviously). Contrary to normal scenraio where 32 floating point precision is sufficient, when checking gradients please make sure 64bit precision is used (or tune the tolerance)."
]
},
{
@ -270,7 +299,7 @@
"\n",
"## Using Cython for the crucial bottleneck pieces\n",
"\n",
"Cython will compile them to C and the code should be comparable in terms of efficiency to numpy using similar operations in numpy. Of course, one can only rely on numpy. Slicing numpy across many dimensions gets much more complicated than working than working with vectors and matrices and we do understand that this can be confusing. Hence, we allow the basic implementation (with any penalty or preference from our side) to be based on embedded loops (which is perhaps much easier to comprehend and debug).\n",
"Cython will compile them to C and the code should be comparable in terms of efficiency to numpy using similar operations in numpy. Of course, one can only rely on numpy. Slicing numpy across many dimensions gets much more complicated than working with vectors and matrices and we do understand that this can be confusing. Hence, we allow the basic implementation (with any penalty or preference from our side) to be based on embedded loops (which is perhaps much easier to comprehend and debug).\n",
"\n",
"Below we give some example cython code for the matrix-matrix dot function from the second tutorial so that you can see the basic differences and compare the obtained speeds. They give you all the necessary patterns needed to implement naive (reasonably efficient) convolution. If you use native python, rather than Cython, then naive looping will be *very* slow.\n",
"\n",
@ -278,7 +307,7 @@
" * [Cython, language basics](http://docs.cython.org/src/userguide/language_basics.html#language-basics)\n",
" * [Cython, basic tutorial](http://docs.cython.org/src/tutorial/cython_tutorial.html)\n",
" * [Cython in ipython notebooks](http://docs.cython.org/src/quickstart/build.html)\n",
" * [A tutorial on how to optimise the cython code](http://docs.cython.org/src/tutorial/numpy.html) (includes a working example which is actually simple convolution code)\n",
" * [A tutorial on how to optimise the cython code](http://docs.cython.org/src/tutorial/numpy.html) (includes a working example which is actually simple convolution code, do not use it `as is`)\n",
" \n",
"\n",
"Before you proceed, check that you have installed `cython` (it should be installed with scipy). If the below imports do not work, then - staying in the activated virtual environment - type:\n",

362
07_MLP_Coursework2.ipynb Normal file
View File

@ -0,0 +1,362 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Please don't edit this cell!**\n",
"\n",
"# Marks and Feedback\n",
"\n",
"**Total Marks:** XX/100\n",
"\n",
"**Overall comments:**\n",
"\n",
"\n",
"## Part 1. Investigations into Neural Networks (35 marks)\n",
"\n",
"* **Task 1**: *Experiments with learning rate schedules* - XX/5\n",
" * learning rate schedulers implemented\n",
" * experiments carried out\n",
" * further comments\n",
"\n",
"\n",
"* **Task 2**: *Experiments with regularisation* - XX/5\n",
" * L1 experiments\n",
" * L2 experiments\n",
" * dropout experiments\n",
" * annealed dropout implmented\n",
" * further experiments carried out\n",
" * further comments\n",
" \n",
"\n",
"* **Task 3**: *Experiments with pretraining* - XX/15\n",
" * autoencoder pretraining implemented\n",
" * denoising autoencoder pretraining implemented\n",
" * CE layer-by-layer pretraining implemented\n",
" * experiments\n",
" * further comments\n",
"\n",
"\n",
"* **Task 4**: *Experiments with data augmentation* - XX/5\n",
" * training data augmneted using noise, rotation, ...\n",
" * any further augmnetations\n",
" * experiments \n",
" * further comments\n",
"\n",
"\n",
"* **Task 5**: *State of the art* - XX/5\n",
" * motivation for systems constructed\n",
" * experiments\n",
" * accuracy of best system\n",
" * further comments\n",
"\n",
"\n",
"\n",
"## Part 2. Convolutional Neural Networks (55 marks)\n",
"\n",
"* **Task 6**: *Implement convolutional layer* - XX/20\n",
" * linear conv layer\n",
" * sigmoid conv layer\n",
" * relu conv layer\n",
" * any checks for correctness\n",
" * loop-based or vectorised implementations\n",
" * timing comparisons\n",
"\n",
"\n",
"* **Task 7**: *Implement maxpooling layer* - XX/10\n",
" * implementation of non-overlapping pooling\n",
" * generic implementation\n",
" * any checks for correctness\n",
"\n",
"\n",
"* **Task 8**: *Experiments with convolutional networks* - XX/25\n",
" * 1 conv layer (1 fmap)\n",
" * 1 conv layer (5 fmaps)\n",
" * 2 conv layers\n",
" * further experiments\n",
"\n",
"\n",
"\n",
"## Presentation (10 marks)\n",
"\n",
"* ** Marks:** XX/10\n",
" * Concise description of each system constructed\n",
" * Experiment design and motivations for different systems\n",
" * Presentation of results - graphs, tables, diagrams\n",
" * Conclusions\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Coursework #2\n",
"\n",
"## Introduction\n",
"\n",
"\n",
"## Previous Tutorials\n",
"\n",
"Before starting this coursework make sure that you have completed the following labs:\n",
"\n",
"* [04_Regularisation.ipynb](https://github.com/CSTR-Edinburgh/mlpractical/blob/master/04_Regularisation.ipynb) - regularising the model\n",
"* [05_Transfer_functions.ipynb](https://github.com/CSTR-Edinburgh/mlpractical/blob/master/05_Transfer_functions.ipynb) - building and training different activation functions\n",
"* [06_MLP_Coursework2_Introduction.ipynb](https://github.com/CSTR-Edinburgh/mlpractical/blob/master/06_MLP_Coursework2_Introduction.ipynb) - Notes on numpy and tensors\n",
"\n",
"\n",
"## Submission\n",
"**Submission Deadline: Thursday 14 January 2016, 16:00** \n",
"\n",
"Submit the coursework as an ipython notebook file, using the `submit` command in the terminal on a DICE machine. If your file is `06_MLP_Coursework1.ipynb` then you would enter:\n",
"\n",
"`submit mlp 2 06_MLP_Coursework1.ipynb` \n",
"\n",
"where `mlp 2` indicates this is the second coursework of MLP.\n",
"\n",
"After submitting, you should receive an email of acknowledgment from the system confirming that your submission has been received successfully. Keep the email as evidence of your coursework submission.\n",
"\n",
"**Please make sure you submit a single `ipynb` file (and nothing else)!**\n",
"\n",
"**Submission Deadline: Thursday 14 January 2016, 16:00** \n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Getting Started\n",
"Please enter your student number and the date in the next code cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#MLP Coursework 2\n",
"#Student number: <ENTER STUDENT NUMBER>\n",
"#Date: <ENTER DATE>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Part 1. Investigations into Neural Networks (35 marks)\n",
"\n",
"In this part you are may choose exactly what you implement. However, you are expected to express your motivations, observations, and findings in a clear and cohesive way. Try to make it clear why you decided to do certain things. Use graphs and/or tables of results to show trends and other characteristics you think are important. \n",
"\n",
"For example, in Task 1 you could experiment with different schedulers in order to compare their convergence properties. In Task 2 you could look into (and visualise) what happens to weights when applying L1 and/or L2 regularisation when training. For instance, you could create sorted histograms of weight magnitudes in in each layer, etc..\n",
"\n",
"**Before submission, please collapse all the log entries into smaller boxes (by clicking on the bar on the left hand side)**\n",
"\n",
"### Task 1 - Experiments with learning rate schedules (5 marks)\n",
"\n",
"Investigate the effect of learning rate schedules on training and accuracy. Implement at least one additional learning rate scheduler mentioned in the lectures. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"#load the corresponding code here, and also attach scripts that run the experiments ()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Task 2 - Experiments with regularisers (5 marks)\n",
"\n",
"Investigate the effect of different regularisation approaches (L1, L2, dropout). Implement the annealing dropout scheduler (mentioned in lecture 5). Do some further investigations and experiments with model structures (and regularisers) of your choice. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Task 3 - Experiments with pretraining (15 marks)\n",
"\n",
"Implement pretraining of multi-layer networks with autoencoders, denoising autoencoders, and using layer-by-layer cross-entropy training. \n",
"\n",
"Implementation tip: You could add the corresponding methods to `optimiser`, namely, `pretrain()` and `pretrain_epoch()`, for autoencoders. Simiilarly, `pretrain_discriminative()` and `pretrain_epoch_discriminative()` for cross-entropy layer-by-layer pretraining. Of course, you can modify any other necessary pieces, but include all the modified fragments below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Task 4 - Experiments with data augmentation (5 marks)\n",
"\n",
"Using the standard MNIST training data, generate some augmented training examples (for example, using noise or rotation). Perform experiments on using this expanded training data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Task 5 - State of the art (5 marks)\n",
"\n",
"Using any techniques you have learnt so far (combining any number of them), build and train the best model you can (no other constraints)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"# Part 2. Convolutional Neural Networks (55 marks)\n",
"\n",
"In this part of the coursework, you are required to implement deep convolutional networks. This includes code for forward prop, back prop, and weight updates for convolutional and max-pooling layers, and should support the stacking of convolutional + pooling layers. You should implement all the parts relating to the convolutional layer in the mlp/conv.py module; if you decide to implement some routines in cython, keep them in mlp/conv.pyx). Attach both files in this notebook.\n",
"\n",
"Implementation tips: Look at [lecture 7](http://www.inf.ed.ac.uk/teaching/courses/mlp/2015/mlp07-cnn.pdf) and [lecture 8](http://www.inf.ed.ac.uk/teaching/courses/mlp/2015/mlp08-cnn2.pdf), and the introductory tutorial, [06_MLP_Coursework2_Introduction.ipynb](https://github.com/CSTR-Edinburgh/mlpractical/blob/master/06_MLP_Coursework2_Introduction.ipynb)\n",
"\n",
"### Task 6 - Implement convolutional layer (20 marks)\n",
"\n",
"Implement linear convolutional layer, and then extend to sigmoid and ReLU transfer functions (do it in a similar way to fully-connected layers). Include all relevant code. It is recommended that you first implement in the naive way with nested loops (python and/or cython); optionally you may then implement in a vectorised way in numpy. Include logs for each way you implement the convolutional layer, as timings for different implementations are of interest. Include all relevant code."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Task 7 - Implement max-pooling layer (10 marks)\n",
"\n",
"Implement a max-pooling layer. Non-overlapping pooling (which was assumed in the lecture presentation) is required. You may also implement a more generic solution with striding as well. Include all relevant code."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Task 8 - Experiments with convolutional networks (25 marks)\n",
"\n",
"Construct convolutional networks with a softmax output layer and a single fully connected hidden layer. Your first experiments should use one convolutional+pooling layer. As a default use convolutional kernels of dimension 5x5 (stride 1) and pooling regions of 2x2 (stride 2, hence non-overlapping).\n",
"\n",
"* Implement and test a convolutional network with 1 feature map\n",
"* Implement and test a convolutional network with 5 feature maps\n",
"\n",
"Explore convolutional networks with two convolutional layers, by implementing, training, and evaluating a network with two convolutional+maxpooling layers with 5 feature maps in the first convolutional layer, and 10 feature maps in the second convolutional layer.\n",
"\n",
"Carry out further experiments to optimise the convolutional network architecture (you could explore kernel sizes and strides, number of feature maps, sizes and strides of pooling operator, etc. - it is up to you)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"**This is the end of coursework 2.**\n",
"\n",
"Please remember to save your notebook, and submit your notebook following the instructions at the top. Please make sure that you have executed all the code cells when you submit the notebook.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 0
}

126
mlp/conv.py Normal file
View File

@ -0,0 +1,126 @@
# Machine Learning Practical (INFR11119),
# Pawel Swietojanski, University of Edinburgh
import numpy
import logging
from mlp.layers import Layer
logger = logging.getLogger(__name__)
"""
You have been given some very initial skeleton below. Feel free to build on top of it and/or
modify it according to your needs. Just notice, you can factor out the convolution code out of
the layer code, and just pass (possibly) different conv implementations for each of the stages
in the model where you are expected to apply the convolutional operator. This will allow you to
keep the layer implementation independent of conv operator implementation, and you can easily
swap it layer, for example, for more efficient implementation if you came up with one, etc.
"""
def my1_conv2d(image, kernels, strides=(1, 1)):
"""
Implements a 2d valid convolution of kernels with the image
Note: filer means the same as kernel and convolution (correlation) of those with the input space
produces feature maps (sometimes refereed to also as receptive fields). Also note, that
feature maps are synonyms here to channels, and as such num_inp_channels == num_inp_feat_maps
:param image: 4D tensor of sizes (batch_size, num_input_channels, img_shape_x, img_shape_y)
:param filters: 4D tensor of filters of size (num_inp_feat_maps, num_out_feat_maps, kernel_shape_x, kernel_shape_y)
:param strides: a tuple (stride_x, stride_y), specifying the shift of the kernels in x and y dimensions
:return: 4D tensor of size (batch_size, num_out_feature_maps, feature_map_shape_x, feature_map_shape_y)
"""
raise NotImplementedError('Write me!')
class ConvLinear(Layer):
def __init__(self,
num_inp_feat_maps,
num_out_feat_maps,
image_shape=(28, 28),
kernel_shape=(5, 5),
stride=(1, 1),
irange=0.2,
rng=None,
conv_fwd=my1_conv2d,
conv_bck=my1_conv2d,
conv_grad=my1_conv2d):
"""
:param num_inp_feat_maps: int, a number of input feature maps (channels)
:param num_out_feat_maps: int, a number of output feature maps (channels)
:param image_shape: tuple, a shape of the image
:param kernel_shape: tuple, a shape of the kernel
:param stride: tuple, shift of kernels in both dimensions
:param irange: float, initial range of the parameters
:param rng: RandomState object, random number generator
:param conv_fwd: handle to a convolution function used in fwd-prop
:param conv_bck: handle to a convolution function used in backward-prop
:param conv_grad: handle to a convolution function used in pgrads
:return:
"""
super(ConvLinear, self).__init__(rng=rng)
raise NotImplementedError()
def fprop(self, inputs):
raise NotImplementedError()
def bprop(self, h, igrads):
raise NotImplementedError()
def bprop_cost(self, h, igrads, cost):
raise NotImplementedError('ConvLinear.bprop_cost method not implemented')
def pgrads(self, inputs, deltas, l1_weight=0, l2_weight=0):
raise NotImplementedError()
def get_params(self):
raise NotImplementedError()
def set_params(self, params):
raise NotImplementedError()
def get_name(self):
return 'convlinear'
#you can derive here particular non-linear implementations:
#class ConvSigmoid(ConvLinear):
#...
class ConvMaxPool2D(Layer):
def __init__(self,
num_feat_maps,
conv_shape,
pool_shape=(2, 2),
pool_stride=(2, 2)):
"""
:param conv_shape: tuple, a shape of the lower convolutional feature maps output
:param pool_shape: tuple, a shape of pooling operator
:param pool_stride: tuple, a strides for pooling operator
:return:
"""
super(ConvMaxPool2D, self).__init__(rng=None)
raise NotImplementedError()
def fprop(self, inputs):
raise NotImplementedError()
def bprop(self, h, igrads):
raise NotImplementedError()
def get_params(self):
return []
def pgrads(self, inputs, deltas, **kwargs):
return []
def set_params(self, params):
pass
def get_name(self):
return 'convmaxpool2d'

7
mlp/dataset.py
View File

@ -83,7 +83,8 @@ class MNISTDataProvider(DataProvider):
max_num_batches=-1,
max_num_examples=-1,
randomize=True,
rng=None):
rng=None,
conv_reshape=False):
super(MNISTDataProvider, self).\
__init__(batch_size, randomize, rng)
@ -119,6 +120,7 @@ class MNISTDataProvider(DataProvider):
self.x = x
self.t = t
self.num_classes = 10
self.conv_reshape = conv_reshape
self._rand_idx = None
if self.randomize:
@ -162,6 +164,9 @@ class MNISTDataProvider(DataProvider):
self._curr_idx += self.batch_size
if self.conv_reshape:
rval_x = rval_x.reshape(self.batch_size, 1, 28, 28)
return rval_x, self.__to_one_of_k(rval_t)
def num_examples(self):

201
mlp/layers.py
View File

@ -10,6 +10,51 @@ from mlp.costs import Cost
logger = logging.getLogger(__name__)
def max_and_argmax(x, axes=None, keepdims_max=False, keepdims_argmax=False):
"""
Return both max and argmax for the given multi-dimensional array, possibly
preserve the original shapes
:param x: input tensor
:param axes: tuple of ints denoting axes across which
one should perform reduction
:param keepdims_max: boolean, if true, shape of x is preserved in result
:param keepdims_argmax:, boolean, if true, shape of x is preserved in result
:return: max (number) and argmax (indices) of max element along certain axes
in multi-dimensional tensor
"""
if axes is None:
rval_argmax = numpy.argmax(x)
if keepdims_argmax:
rval_argmax = numpy.unravel_index(rval_argmax, x.shape)
else:
if isinstance(axes, int):
axes = (axes,)
axes = tuple(axes)
keep_axes = numpy.array([i for i in range(x.ndim) if i not in axes])
transposed_x = numpy.transpose(x, numpy.concatenate((keep_axes, axes)))
reshaped_x = transposed_x.reshape(transposed_x.shape[:len(keep_axes)] + (-1,))
rval_argmax = numpy.asarray(numpy.argmax(reshaped_x, axis=-1), dtype=numpy.int64)
# rval_max_arg keeps the arg index referencing to the axis along which reduction was performed (axis=-1)
# when keepdims_argmax is True we need to map it back to the original shape of tensor x
# print 'rval maxaarg', rval_argmax.ndim, rval_argmax.shape, rval_argmax
if keepdims_argmax:
dim = tuple([x.shape[a] for a in axes])
rval_argmax = numpy.array([idx + numpy.unravel_index(val, dim)
for idx, val in numpy.ndenumerate(rval_argmax)])
# convert to numpy indexing convention (row indices first, then columns)
rval_argmax = zip(*rval_argmax)
if keepdims_max is False and keepdims_argmax is True:
# this could potentially save O(N) steps by not traversing array once more
# to get max value, haven't benchmark it though
rval_max = x[rval_argmax]
else:
rval_max = numpy.asarray(numpy.amax(x, axis=axes, keepdims=keepdims_max))
return rval_max, rval_argmax
class MLP(object):
"""
This is a container for an arbitrary sequence of other transforms
@ -18,7 +63,7 @@ class MLP(object):
through the model (for a mini-batch), which is required to compute
the gradients for the parameters
"""
def __init__(self, cost):
def __init__(self, cost, rng=None):
assert isinstance(cost, Cost), (
"Cost needs to be of type mlp.costs.Cost, got %s" % type(cost)
@ -31,6 +76,11 @@ class MLP(object):
# for a given minibatch and each layer
self.cost = cost
if rng is None:
self.rng = numpy.random.RandomState([2015,11,11])
else:
self.rng = rng
def fprop(self, x):
"""
@ -46,6 +96,32 @@ class MLP(object):
self.activations[i+1] = self.layers[i].fprop(self.activations[i])
return self.activations[-1]
def fprop_dropout(self, x, dp_scheduler):
"""
:param inputs: mini-batch of data-points x
:param dp_scheduler: dropout scheduler
:return: y (top layer activation) which is an estimate of y given x
"""
if len(self.activations) != len(self.layers) + 1:
self.activations = [None]*(len(self.layers) + 1)
p_inp, p_hid = dp_scheduler.get_rate()
d_inp = 1
p_inp_scaler, p_hid_scaler = 1.0/p_inp, 1.0/p_hid
if p_inp < 1:
d_inp = self.rng.binomial(1, p_inp, size=x.shape)
self.activations[0] = p_inp_scaler*d_inp*x
for i in xrange(0, len(self.layers)):
d_hid = 1
if p_hid < 1 and i > 0:
d_hid = self.rng.binomial(1, p_hid, size=self.activations[i].shape)
self.activations[i+1] = self.layers[i].fprop(p_hid_scaler*d_hid*self.activations[i])
return self.activations[-1]
def bprop(self, cost_grad):
"""
:param cost_grad: matrix -- grad of the cost w.r.t y
@ -189,6 +265,11 @@ class Linear(Layer):
:param inputs: matrix of features (x) or the output of the previous layer h^{i-1}
:return: h^i, matrix of transformed by layer features
"""
#input comes from 4D convolutional tensor, reshape to expected shape
if inputs.ndim == 4:
inputs = inputs.reshape(inputs.shape[0], -1)
a = numpy.dot(inputs, self.W) + self.b
# here f() is an identity function, so just return a linear transformation
return a
@ -258,8 +339,24 @@ class Linear(Layer):
since W and b are only layer's parameters
"""
grad_W = numpy.dot(inputs.T, deltas)
grad_b = numpy.sum(deltas, axis=0)
#input comes from 4D convolutional tensor, reshape to expected shape
if inputs.ndim == 4:
inputs = inputs.reshape(inputs.shape[0], -1)
#you could basically use different scalers for biases
#and weights, but it is not implemented here like this
l2_W_penalty, l2_b_penalty = 0, 0
if l2_weight > 0:
l2_W_penalty = l2_weight*self.W
l2_b_penalty = l2_weight*self.b
l1_W_penalty, l1_b_penalty = 0, 0
if l1_weight > 0:
l1_W_penalty = l1_weight*numpy.sign(self.W)
l1_b_penalty = l1_weight*numpy.sign(self.b)
grad_W = numpy.dot(inputs.T, deltas) + l2_W_penalty + l1_W_penalty
grad_b = numpy.sum(deltas, axis=0) + l2_b_penalty + l1_b_penalty
return [grad_W, grad_b]
@ -323,12 +420,12 @@ class Softmax(Linear):
odim,
rng=rng,
irange=irange)
def fprop(self, inputs):
# compute the linear outputs
a = super(Softmax, self).fprop(inputs)
# apply numerical stabilisation by subtracting max
# apply numerical stabilisation by subtracting max
# from each row (not required for the coursework)
# then compute exponent
assert a.ndim in [1, 2], (
@ -355,3 +452,97 @@ class Softmax(Linear):
def get_name(self):
return 'softmax'
class Relu(Linear):
def __init__(self, idim, odim,
rng=None,
irange=0.1):
super(Relu, self).__init__(idim, odim, rng, irange)
def fprop(self, inputs):
#get the linear activations
a = super(Relu, self).fprop(inputs)
h = numpy.clip(a, 0, 20.0)
#h = numpy.maximum(a, 0)
return h
def bprop(self, h, igrads):
deltas = (h > 0)*igrads + (h <= 0)*igrads
___, ograds = super(Relu, self).bprop(h=None, igrads=deltas)
return deltas, ograds
def cost_bprop(self, h, igrads, cost):
raise NotImplementedError('Relu.bprop_cost method not implemented '
'for the %s cost' % cost.get_name())
def get_name(self):
return 'relu'
class Tanh(Linear):
def __init__(self, idim, odim,
rng=None,
irange=0.1):
super(Tanh, self).__init__(idim, odim, rng, irange)
def fprop(self, inputs):
#get the linear activations
a = super(Tanh, self).fprop(inputs)
numpy.clip(a, -30.0, 30.0, out=a)
h = numpy.tanh(a)
return h
def bprop(self, h, igrads):
deltas = (1.0 - h**2) * igrads
___, ograds = super(Tanh, self).bprop(h=None, igrads=deltas)
return deltas, ograds
def cost_bprop(self, h, igrads, cost):
raise NotImplementedError('Tanh.bprop_cost method not implemented '
'for the %s cost' % cost.get_name())
def get_name(self):
return 'tanh'
class Maxout(Linear):
def __init__(self, idim, odim, k,
rng=None,
irange=0.05):
super(Maxout, self).__init__(idim, odim*k, rng, irange)
self.max_odim = odim
self.k = k
def fprop(self, inputs):
#get the linear activations
a = super(Maxout, self).fprop(inputs)
ar = a.reshape(a.shape[0], self.max_odim, self.k)
h, h_argmax = max_and_argmax(ar, axes=2, keepdims_max=True, keepdims_argmax=True)
self.h_argmax = h_argmax
return h[:, :, 0] #get rid of the last reduced dimensison (of size 1)
def bprop(self, h, igrads):
#convert into the shape where upsampling is easier
igrads_up = igrads.reshape(igrads.shape[0], self.max_odim, 1)
#upsample to the linear dimension (but reshaped to (batch_size, maxed_num (1), pool_size)
igrads_up = numpy.tile(igrads_up, (1, 1, self.k))
#generate mask matrix and set to 1 maxed elements
mask = numpy.zeros_like(igrads_up)
mask[self.h_argmax] = 1.0
#do bprop through max operator and then reshape into 2D
deltas = (igrads_up * mask).reshape(igrads_up.shape[0], -1)
#and then do bprop thorough linear part
___, ograds = super(Maxout, self).bprop(h=None, igrads=deltas)
return deltas, ograds
def cost_bprop(self, h, igrads, cost):
raise NotImplementedError('Maxout.bprop_cost method not implemented '
'for the %s cost' % cost.get_name())
def get_name(self):
return 'maxout'

6
mlp/optimisers.py
View File

@ -112,8 +112,12 @@ class SGDOptimiser(Optimiser):
acc_list, nll_list = [], []
for x, t in train_iterator:
# get the prediction
y = model.fprop(x)
if self.dp_scheduler is not None:
y = model.fprop_dropout(x, self.dp_scheduler)
else:
y = model.fprop(x)
# compute the cost and grad of the cost w.r.t y
cost = model.cost.cost(y, t)

15
mlp/schedulers.py
View File

@ -153,3 +153,18 @@ class LearningRateNewBob(LearningRateScheduler):
self.epoch += 1
return self.rate
class DropoutFixed(LearningRateList):
def __init__(self, p_inp_keep, p_hid_keep):
assert 0 < p_inp_keep <= 1 and 0 < p_hid_keep <= 1, (
"Dropout 'keep' probabilites are suppose to be in (0, 1] range"
)
super(DropoutFixed, self).__init__([(p_inp_keep, p_hid_keep)], max_epochs=999)
def get_rate(self):
return self.lr_list[0]
def get_next_rate(self, current_error=None):
return self.get_rate()

66
mlp/utils.py Normal file
View File

@ -0,0 +1,66 @@
# Machine Learning Practical (INFR11119),
# Pawel Swietojanski, University of Edinburgh
import numpy
from mlp.layers import Layer
def numerical_gradient(f, x, eps=1e-4, **kwargs):
"""
Implements the following numerical gradient rule
df(x)/dx = (f(x+eps)-f(x-eps))/(2eps)
"""
xc = x.copy()
g = numpy.zeros_like(xc)
xf = xc.ravel()
gf = g.ravel()
for i in xrange(xf.shape[0]):
xx = xf[i]
xf[i] = xx + eps
fp_eps, ___ = f(xc, **kwargs)
xf[i] = xx - eps
fm_eps, ___ = f(xc, **kwargs)
xf[i] = xx
gf[i] = (fp_eps - fm_eps)/(2*eps)
return g
def verify_gradient(f, x, eps=1e-4, tol=1e-6, **kwargs):
"""
Compares the numerical and analytical gradients.
"""
fval, fgrad = f(x=x, **kwargs)
ngrad = numerical_gradient(f=f, x=x, eps=eps, tol=tol, **kwargs)
fgradnorm = numpy.sqrt(numpy.sum(fgrad**2))
ngradnorm = numpy.sqrt(numpy.sum(ngrad**2))
diffnorm = numpy.sqrt(numpy.sum((fgrad-ngrad)**2))
if fgradnorm > 0 or ngradnorm > 0:
norm = numpy.maximum(fgradnorm, ngradnorm)
if not (diffnorm < tol or diffnorm/norm < tol):
raise Exception("Numerical and analytical gradients "
"are different: %s != %s!" % (ngrad, fgrad))
else:
if not (diffnorm < tol):
raise Exception("Numerical and analytical gradients "
"are different: %s != %s!" % (ngrad, fgrad))
return True
def verify_layer_gradient(layer, x, eps=1e-4, tol=1e-6):
assert isinstance(layer, Layer), (
"Expected to get the instance of Layer class, got"
" %s " % type(layer)
)
def grad_layer_wrapper(x, **kwargs):
h = layer.fprop(x)
deltas, ograds = layer.bprop(h=h, igrads=numpy.ones_like(h))
return numpy.sum(h), ograds
return verify_gradient(f=grad_layer_wrapper, x=x, eps=eps, tol=tol, layer=layer)