solutions
This commit is contained in:
pswietojanski
parent
c14f7a44a1
commit
4404649ba1
622
00_Introduction_solution.ipynb
Normal file
622
00_Introduction_solution.ipynb
Normal file
File diff suppressed because one or more lines are too long
@ -4,45 +4,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Introduction\n",
|
||||
"\n",
|
||||
"This tutorial is about linear transforms - a basic building block of neural networks, including deep learning models.\n",
|
||||
"\n",
|
||||
"# Virtual environments and syncing repositories\n",
|
||||
"\n",
|
||||
"Before you proceed onwards, remember to activate you virtual environments so you can use the software you installed last week as well as run the notebooks in interactive mode, not through the github.com website.\n",
|
||||
"\n",
|
||||
"## Virtual environments\n",
|
||||
"\n",
|
||||
"To activate the virtual environment:\n",
|
||||
" * If you were in last week's Tuesday or Wednesday group type `activate_mlp` or `source ~/mlpractical/venv/bin/activate`\n",
|
||||
" * If you were in the Monday group:\n",
|
||||
" + and if you have chosen the **comfy** way type: `workon mlpractical`\n",
|
||||
" + and if you have chosen the **generic** way, `source` your virutal environment using `source` and specyfing the path to the activate script (you need to localise it yourself, there were not any general recommendations w.r.t dir structure and people have installed it in different places, usually somewhere in the home directories. If you cannot easily find it by yourself, use something like: `find . -iname activate` ):\n",
|
||||
"\n",
|
||||
"## On Synchronising repositories\n",
|
||||
"\n",
|
||||
"Enter the git mlp repository you set up last week (i.e. `~/mlpractical/repo-mlp`) and once you sync the repository (in one of the two below ways, or look at our short Git FAQ <a href=\"https://github.com/CSTR-Edinburgh/mlpractical/blob/master/gitFAQ.md\">here</a>), start the notebook session by typing:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"ipython notebook\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"### Default way\n",
|
||||
"\n",
|
||||
"To avoid potential conflicts between the changes you have made since last week and our additions, we recommend `stash` your changes and `pull` the new code from the mlpractical repository by typing:\n",
|
||||
"\n",
|
||||
"1. `git stash save \"Lab1 work\"`\n",
|
||||
"2. `git pull`\n",
|
||||
"\n",
|
||||
"Then, if you need to, you can always (temporaily) restore a desired state of the repository (look <a href=\"https://github.com/CSTR-Edinburgh/mlpractical/blob/master/gitFAQ.md\">here</a>).\n",
|
||||
"\n",
|
||||
"**Otherwise** you may also create a branch for each lab separately (again, look <a href=\"https://github.com/CSTR-Edinburgh/mlpractical/blob/master/gitFAQ.md\">here</a> and git tutorials we linked there), this will allow you to keep `master` branch clean, and pull changes into it every week from the central repository. At the same time branching gives you much more flexibility with changes you introduce to the code as potential conflicts will not occur until you try to make an explicit merge.\n",
|
||||
"\n",
|
||||
"### For advanced github users\n",
|
||||
"\n",
|
||||
"It is OK if you want to keep your changes and merge the new code with whatever you already have, but you need to know what you are doing and how to resolve conflicts.\n",
|
||||
" "
|
||||
"-"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
890
01_Linear_Models_solution.ipynb
Normal file
890
01_Linear_Models_solution.ipynb
Normal file
@ -0,0 +1,890 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Introduction\n",
|
||||
"\n",
|
||||
"This tutorial is about linear transforms - a basic building block of neural networks, including deep learning models.\n",
|
||||
"\n",
|
||||
"# Virtual environments and syncing repositories\n",
|
||||
"\n",
|
||||
"Before you proceed onwards, remember to activate you virtual environments so you can use the software you installed last week as well as run the notebooks in interactive mode, not through the github.com website.\n",
|
||||
"\n",
|
||||
"## Virtual environments\n",
|
||||
"\n",
|
||||
"To activate the virtual environment:\n",
|
||||
" * If you were in last week's Tuesday or Wednesday group type `activate_mlp` or `source ~/mlpractical/venv/bin/activate`\n",
|
||||
" * If you were in the Monday group:\n",
|
||||
" + and if you have chosen the **comfy** way type: `workon mlpractical`\n",
|
||||
" + and if you have chosen the **generic** way, `source` your virutal environment using `source` and specyfing the path to the activate script (you need to localise it yourself, there were not any general recommendations w.r.t dir structure and people have installed it in different places, usually somewhere in the home directories. If you cannot easily find it by yourself, use something like: `find . -iname activate` ):\n",
|
||||
"\n",
|
||||
"## On Synchronising repositories\n",
|
||||
"\n",
|
||||
"Enter the git mlp repository you set up last week (i.e. `~/mlpractical/repo-mlp`) and once you sync the repository (in one of the two below ways, or look at our short Git FAQ <a href=\"https://github.com/CSTR-Edinburgh/mlpractical/blob/master/gitFAQ.md\">here</a>), start the notebook session by typing:\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"ipython notebook\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"### Default way\n",
|
||||
"\n",
|
||||
"To avoid potential conflicts between the changes you have made since last week and our additions, we recommend `stash` your changes and `pull` the new code from the mlpractical repository by typing:\n",
|
||||
"\n",
|
||||
"1. `git stash save \"Lab1 work\"`\n",
|
||||
"2. `git pull`\n",
|
||||
"\n",
|
||||
"Then, if you need to, you can always (temporaily) restore a desired state of the repository (look <a href=\"https://github.com/CSTR-Edinburgh/mlpractical/blob/master/gitFAQ.md\">here</a>).\n",
|
||||
"\n",
|
||||
"**Otherwise** you may also create a branch for each lab separately (again, look <a href=\"https://github.com/CSTR-Edinburgh/mlpractical/blob/master/gitFAQ.md\">here</a> and git tutorials we linked there), this will allow you to keep `master` branch clean, and pull changes into it every week from the central repository. At the same time branching gives you much more flexibility with changes you introduce to the code as potential conflicts will not occur until you try to make an explicit merge.\n",
|
||||
"\n",
|
||||
"### For advanced github users\n",
|
||||
"\n",
|
||||
"It is OK if you want to keep your changes and merge the new code with whatever you already have, but you need to know what you are doing and how to resolve conflicts.\n",
|
||||
" \n",
|
||||
" "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Single Layer Models\n",
|
||||
"\n",
|
||||
"***\n",
|
||||
"### Note on storing matrices in computer memory\n",
|
||||
"\n",
|
||||
"Consider you want to store the following array in memory: $\\left[ \\begin{array}{ccc}\n",
|
||||
"1 & 2 & 3 \\\\\n",
|
||||
"4 & 5 & 6 \\\\\n",
|
||||
"7 & 8 & 9 \\end{array} \\right]$ \n",
|
||||
"\n",
|
||||
"In computer memory the above matrix would be organised as a vector in either (assume you allocate the memory at once for the whole matrix):\n",
|
||||
"\n",
|
||||
"* Row-wise layout where the order would look like: $\\left [ \\begin{array}{ccccccccc}\n",
|
||||
"1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 \\end{array} \\right ]$\n",
|
||||
"* Column-wise layout where the order would look like: $\\left [ \\begin{array}{ccccccccc}\n",
|
||||
"1 & 4 & 7 & 2 & 5 & 8 & 3 & 6 & 9 \\end{array} \\right ]$\n",
|
||||
"\n",
|
||||
"Although `numpy` can easily handle both formats (possibly with some computational overhead), in our code we will stick with modern (and default) `c`-like approach and use row-wise format (contrary to Fortran that used column-wise approach). \n",
|
||||
"\n",
|
||||
"This means, that in this tutorial:\n",
|
||||
"* vectors are kept row-wise $\\mathbf{x} = (x_1, x_1, \\ldots, x_D) $ (rather than $\\mathbf{x} = (x_1, x_1, \\ldots, x_D)^T$)\n",
|
||||
"* similarly, in case of matrices we will stick to: $\\left[ \\begin{array}{cccc}\n",
|
||||
"x_{11} & x_{12} & \\ldots & x_{1D} \\\\\n",
|
||||
"x_{21} & x_{22} & \\ldots & x_{2D} \\\\\n",
|
||||
"x_{31} & x_{32} & \\ldots & x_{3D} \\\\ \\end{array} \\right]$ and each row (i.e. $\\left[ \\begin{array}{cccc} x_{11} & x_{12} & \\ldots & x_{1D} \\end{array} \\right]$) represents a single data-point (like one MNIST image or one window of observations)\n",
|
||||
"\n",
|
||||
"In lecture slides you will find the equations following the conventional mathematical column-wise approach, but you can easily map them one way or the other using using matrix transpose.\n",
|
||||
"\n",
|
||||
"***\n",
|
||||
"\n",
|
||||
"## Linear and Affine Transforms\n",
|
||||
"\n",
|
||||
"The basis of all linear models is so called affine transform, that is a transform that implements some linear transformation and translation of input features. The transforms we are going to use are parameterised by:\n",
|
||||
"\n",
|
||||
" * Weight matrix $\\mathbf{W} \\in \\mathbb{R}^{D\\times K}$: where element $w_{ik}$ is the weight from input $x_i$ to output $y_k$\n",
|
||||
" * Bias vector $\\mathbf{b}\\in R^{K}$ : where element $b_{k}$ is the bias for output $k$\n",
|
||||
"\n",
|
||||
"Note, the bias is simply some additve term, and can be easily incorporated into an additional row in weight matrix and an additinal input in the inputs which is set to $1.0$ (as in the below picture taken from the lecture slides). However, here (and in the code) we will keep them separate.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"For instance, for the above example of 5-dimensional input vector by $\\mathbf{x} = (x_1, x_2, x_3, x_4, x_5)$, weight matrix $\\mathbf{W}=\\left[ \\begin{array}{ccc}\n",
|
||||
"w_{11} & w_{12} & w_{13} \\\\\n",
|
||||
"w_{21} & w_{22} & w_{23} \\\\\n",
|
||||
"w_{31} & w_{32} & w_{33} \\\\\n",
|
||||
"w_{41} & w_{42} & w_{43} \\\\\n",
|
||||
"w_{51} & w_{52} & w_{53} \\\\ \\end{array} \\right]$, bias vector $\\mathbf{b} = (b_1, b_2, b_3)$ and outputs $\\mathbf{y} = (y_1, y_2, y_3)$, one can write the transformation as follows:\n",
|
||||
"\n",
|
||||
"(for the $i$-th output)\n",
|
||||
"\n",
|
||||
"(1) $\n",
|
||||
"\\begin{equation}\n",
|
||||
" y_i = b_i + \\sum_j x_jw_{ji}\n",
|
||||
"\\end{equation}\n",
|
||||
"$\n",
|
||||
"\n",
|
||||
"or the equivalent vector form (where $\\mathbf w_i$ is the $i$-th column of $\\mathbf W$, but note, when we **slice** the $i$th column we will get a **vector** $\\mathbf w_i = (w_{1i}, w_{2i}, w_{3i}, w_{4i}, w_{5i})$, hence the transpose for $\\mathbf w_i$ in the below equation):\n",
|
||||
"\n",
|
||||
"(2) $\n",
|
||||
"\\begin{equation}\n",
|
||||
" y_i = b_i + \\mathbf x \\mathbf w_i^T\n",
|
||||
"\\end{equation}\n",
|
||||
"$\n",
|
||||
"\n",
|
||||
"The same operation can be also written in matrix form, to compute all the outputs $\\mathbf{y}$ at the same time:\n",
|
||||
"\n",
|
||||
"(3) $\n",
|
||||
"\\begin{equation}\n",
|
||||
" \\mathbf y=\\mathbf x\\mathbf W + \\mathbf b\n",
|
||||
"\\end{equation}\n",
|
||||
"$\n",
|
||||
"\n",
|
||||
"This is equivalent to slides 12/13 in lecture 1, except we are using row vectors.\n",
|
||||
"\n",
|
||||
"When $\\mathbf{x}$ is a mini-batch (contains $B$ data-points of dimension $D$ each), i.e. $\\left[ \\begin{array}{cccc}\n",
|
||||
"x_{11} & x_{12} & \\ldots & x_{1D} \\\\\n",
|
||||
"x_{21} & x_{22} & \\ldots & x_{2D} \\\\\n",
|
||||
"\\cdots \\\\\n",
|
||||
"x_{B1} & x_{B2} & \\ldots & x_{BD} \\\\ \\end{array} \\right]$ equation (3) effectively becomes to be\n",
|
||||
"\n",
|
||||
"(4) $\n",
|
||||
"\\begin{equation}\n",
|
||||
" \\mathbf Y=\\mathbf X\\mathbf W + \\mathbf b\n",
|
||||
"\\end{equation}\n",
|
||||
"$\n",
|
||||
"\n",
|
||||
"where both $\\mathbf{X}\\in\\mathbb{R}^{B\\times D}$ and $\\mathbf{Y}\\in\\mathbb{R}^{B\\times K}$ are matrices, and $\\mathbf{b}$ needs to be <a href=\"http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html\">broadcasted</a> $B$ times (numpy will do this by default). However, we will not make an explicit distinction between a special case for $B=1$ and $B>1$ and simply use equation (3) instead, although $\\mathbf{x}$ and hence $\\mathbf{y}$ could be matrices. From an implementation point of view, it does not matter.\n",
|
||||
"\n",
|
||||
"The desired functionality for matrix multiplication in numpy is provided by <a href=\"http://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html\">numpy.dot</a> function. If you haven't use it so far, get familiar with it as we will use it extensively."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### A general note on random number generators\n",
|
||||
"\n",
|
||||
"It is generally a good practice (for machine learning applications **not** for cryptography!) to seed a pseudo-random number generator once at the beginning of the experiment, and use it later through the code where necesarry. This makes it easier to reproduce results since random initialisations can be replicated. As such, within this course we are going use a single random generator object, similar to the below:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy\n",
|
||||
"import sys\n",
|
||||
"\n",
|
||||
"#initialise the random generator to be used later\n",
|
||||
"seed=[2015, 10, 1]\n",
|
||||
"random_generator = numpy.random.RandomState(seed)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 1 \n",
|
||||
"\n",
|
||||
"Using numpy.dot, implement **forward** propagation through the linear transform defined by equations (3) and (4) for $B=1$ and $B>1$. As data ($\\mathbf{x}$) use `MNISTDataProvider` from previous laboratories. For case when $B=1$ write a function to compute the 1st output ($y_1$) using equations (1) and (2). Check if the output is the same as the corresponding one obtained with numpy. \n",
|
||||
"\n",
|
||||
"Tip: To generate random data you can use `random_generator.uniform(-0.1, 0.1, (D, 10))` from the preceeding cell."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from mlp.dataset import MNISTDataProvider\n",
|
||||
"\n",
|
||||
"mnist_dp = MNISTDataProvider(dset='valid', batch_size=3, max_num_batches=1, randomize=False)\n",
|
||||
"\n",
|
||||
"irange = 0.1\n",
|
||||
"W = random_generator.uniform(-irange, irange, (784,10)) \n",
|
||||
"b = numpy.zeros((10,))\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"y1e1 0.55861474982\n",
|
||||
"y1e2 0.55861474982\n",
|
||||
"ye3 [[ 0.55861475 0.79450077 0.17439693 0.00265688 0.66272539 -0.09985686\n",
|
||||
" 0.56468591 0.58105588 -0.18613727 0.08151257]\n",
|
||||
" [-0.43965864 0.59573972 -0.22691119 0.26767124 -0.31343979 0.07224664\n",
|
||||
" -0.19616183 0.0851733 -0.24088286 -0.19305162]\n",
|
||||
" [-0.20176359 0.42394166 -1.03984446 0.15492101 0.15694745 -0.53741022\n",
|
||||
" 0.05887668 -0.21124527 -0.07870156 -0.00506471]]\n",
|
||||
"ye4 [[ 0.55861475 0.79450077 0.17439693 0.00265688 0.66272539 -0.09985686\n",
|
||||
" 0.56468591 0.58105588 -0.18613727 0.08151257]\n",
|
||||
" [-0.43965864 0.59573972 -0.22691119 0.26767124 -0.31343979 0.07224664\n",
|
||||
" -0.19616183 0.0851733 -0.24088286 -0.19305162]\n",
|
||||
" [-0.20176359 0.42394166 -1.03984446 0.15492101 0.15694745 -0.53741022\n",
|
||||
" 0.05887668 -0.21124527 -0.07870156 -0.00506471]]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"\n",
|
||||
"mnist_dp.reset()\n",
|
||||
"\n",
|
||||
"#implement following functions, then run the cell\n",
|
||||
"def y1_equation_1(x, W, b):\n",
|
||||
" y1=0\n",
|
||||
" for j in xrange(0, x.shape[0]):\n",
|
||||
" y1 += x[j]*W[j,0]\n",
|
||||
" return y1 + b[0]\n",
|
||||
" \n",
|
||||
"def y1_equation_2(x, W, b):\n",
|
||||
" return numpy.dot(x, W[:,0].T) + b[0]\n",
|
||||
"\n",
|
||||
"def y_equation_3(x, W, b):\n",
|
||||
" return numpy.dot(x,W) + b\n",
|
||||
"\n",
|
||||
"def y_equation_4(x, W, b):\n",
|
||||
" return numpy.dot(x,W) + b\n",
|
||||
"\n",
|
||||
"for x, t in mnist_dp:\n",
|
||||
" y1e1 = y1_equation_1(x[0], W, b)\n",
|
||||
" y1e2 = y1_equation_2(x[0], W, b)\n",
|
||||
" ye3 = y_equation_3(x, W, b)\n",
|
||||
" ye4 = y_equation_4(x, W, b)\n",
|
||||
"\n",
|
||||
"print 'y1e1', y1e1\n",
|
||||
"print 'y1e2', y1e2\n",
|
||||
"print 'ye3', ye3\n",
|
||||
"print 'ye4', ye4\n",
|
||||
" "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"## Exercise 2\n",
|
||||
"\n",
|
||||
"Modify (if necessary) examples from Exercise 1 to perform **backward** propagation, that is, given $\\mathbf{y}$ (obtained in previous step) and weight matrix $\\mathbf{W}$, project $\\mathbf{y}$ onto the input space $\\mathbf{x}$ (ignore or set to zero the biases towards $\\mathbf{x}$ in backward pass). Mathematically, we are interested in the following transformation: $\\mathbf{z}=\\mathbf{y}\\mathbf{W}^T$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[[-0.00683757 -0.13638553 0.00203203 ..., 0.02690207 -0.07364245\n",
|
||||
" 0.04403087]\n",
|
||||
" [-0.00447621 -0.06409652 0.01211384 ..., 0.0402248 -0.04490571\n",
|
||||
" -0.05013801]\n",
|
||||
" [ 0.03981022 -0.13705957 0.05882239 ..., 0.04491902 -0.08644539\n",
|
||||
" -0.07106441]]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"y = y_equation_3(x, W, b)\n",
|
||||
"z = numpy.dot(y, W.T)\n",
|
||||
"\n",
|
||||
"print z\n",
|
||||
"assert z.shape == x.shape"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"***\n",
|
||||
"## Exercise 3 (optional)\n",
|
||||
"\n",
|
||||
"In case you do not fully understand how matrix-vector and/or matrix-matrix products work, consider implementing `my_dot_mat_mat` function (you have been given `my_dot_vec_mat` code to look at as an example) which takes as the input the following arguments:\n",
|
||||
"\n",
|
||||
"* D-dimensional input vector $\\mathbf{x} = (x_1, x_2, \\ldots, x_D) $.\n",
|
||||
"* Weight matrix $\\mathbf{W}\\in\\mathbb{R}^{D\\times K}$:\n",
|
||||
"\n",
|
||||
"and returns:\n",
|
||||
"\n",
|
||||
"* K-dimensional output vector $\\mathbf{y} = (y_1, \\ldots, y_K) $\n",
|
||||
"\n",
|
||||
"Your job is to write a variant that works in a mini-batch mode where both $\\mathbf{x}\\in\\mathbb{R}^{B\\times D}$ and $\\mathbf{y}\\in\\mathbb{R}^{B\\times K}$ are matrices in which each rows contain one of $B$ data-points from mini-batch (rather than $\\mathbf{x}\\in\\mathbb{R}^{D}$ and $\\mathbf{y}\\in\\mathbb{R}^{K}$)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def my_dot_vec_mat(x, W):\n",
|
||||
" J = x.shape[0]\n",
|
||||
" K = W.shape[1]\n",
|
||||
" assert (J == W.shape[0]), (\n",
|
||||
" \"Number of columns of x expected to \"\n",
|
||||
" \" to be equal to the number of rows in \"\n",
|
||||
" \"W, bot got shapes %s, %s\" % (x.shape, W.shape)\n",
|
||||
" )\n",
|
||||
" y = numpy.zeros((K,))\n",
|
||||
" for k in xrange(0, K):\n",
|
||||
" for j in xrange(0, J):\n",
|
||||
" y[k] += x[j] * W[j,k]\n",
|
||||
" \n",
|
||||
" return y"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Well done!\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"irange = 0.1 #+-range from which we draw the random numbers\n",
|
||||
"\n",
|
||||
"x = random_generator.uniform(-irange, irange, (5,)) \n",
|
||||
"W = random_generator.uniform(-irange, irange, (5,3)) \n",
|
||||
"\n",
|
||||
"y_my = my_dot_vec_mat(x, W)\n",
|
||||
"y_np = numpy.dot(x, W)\n",
|
||||
"\n",
|
||||
"same = numpy.allclose(y_my, y_np)\n",
|
||||
"\n",
|
||||
"if same:\n",
|
||||
" print 'Well done!'\n",
|
||||
"else:\n",
|
||||
" print 'Matrices are different:'\n",
|
||||
" print 'y_my is: ', y_my\n",
|
||||
" print 'y_np is: ', y_np"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def my_dot_mat_mat(x, W):\n",
|
||||
" I = x.shape[0]\n",
|
||||
" J = x.shape[1]\n",
|
||||
" K = W.shape[1]\n",
|
||||
" assert (J == W.shape[0]), (\n",
|
||||
" \"Number of columns in of x expected to \"\n",
|
||||
" \" to be the same as rows in W, got\"\n",
|
||||
" )\n",
|
||||
" #allocate the output container\n",
|
||||
" y = numpy.zeros((I, K))\n",
|
||||
" \n",
|
||||
" #implement here matrix-matrix inner product here\n",
|
||||
" for i in xrange(0, I):\n",
|
||||
" for k in xrange(0, K):\n",
|
||||
" for j in xrange(0, J):\n",
|
||||
" y[i, k] += x[i, j] * W[j,k]\n",
|
||||
" \n",
|
||||
" return y"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Test whether you get comparable numbers to what numpy is producing:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Well done!\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"irange = 0.1 #+-range from which we draw the random numbers\n",
|
||||
"\n",
|
||||
"x = random_generator.uniform(-irange, irange, (2,5)) \n",
|
||||
"W = random_generator.uniform(-irange, irange, (5,3)) \n",
|
||||
"\n",
|
||||
"y_my = my_dot_mat_mat(x, W)\n",
|
||||
"y_np = numpy.dot(x, W)\n",
|
||||
"\n",
|
||||
"same = numpy.allclose(y_my, y_np)\n",
|
||||
"\n",
|
||||
"if same:\n",
|
||||
" print 'Well done!'\n",
|
||||
"else:\n",
|
||||
" print 'Matrices are different:'\n",
|
||||
" print 'y_my is: ', y_my\n",
|
||||
" print 'y_np is: ', y_np"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now we benchmark each approach (we do it in separate cells, as timeit currently can measure whole cell execuiton only)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"#generate bit bigger matrices, to better evaluate timings\n",
|
||||
"x = random_generator.uniform(-irange, irange, (10, 1000))\n",
|
||||
"W = random_generator.uniform(-irange, irange, (1000, 100))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"my_dot timings:\n",
|
||||
"10 loops, best of 3: 726 ms per loop\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print 'my_dot timings:'\n",
|
||||
"%timeit -n10 my_dot_mat_mat(x, W)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"numpy.dot timings:\n",
|
||||
"10 loops, best of 3: 1.17 ms per loop\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"print 'numpy.dot timings:'\n",
|
||||
"%timeit -n10 numpy.dot(x, W)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Optional section ends here**\n",
|
||||
"***"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Iterative learning of linear models\n",
|
||||
"\n",
|
||||
"We will learn the model with stochastic gradient descent on N data-points using mean square error (MSE) loss function, which is defined as follows:\n",
|
||||
"\n",
|
||||
"(5) $\n",
|
||||
"E = \\frac{1}{2} \\sum_{n=1}^N ||\\mathbf{y}^n - \\mathbf{t}^n||^2 = \\sum_{n=1}^N E^n \\\\\n",
|
||||
" E^n = \\frac{1}{2} ||\\mathbf{y}^n - \\mathbf{t}^n||^2\n",
|
||||
"$\n",
|
||||
"\n",
|
||||
"(6) $ E^n = \\frac{1}{2} \\sum_{k=1}^K (y_k^n - t_k^n)^2 $\n",
|
||||
" \n",
|
||||
"Hence, the gradient w.r.t (with respect to) the $r$ output y of the model is defined as, so called delta function, $\\delta_r$: \n",
|
||||
"\n",
|
||||
"(8) $\\frac{\\partial{E^n}}{\\partial{y_{r}}} = (y^n_r - t^n_r) = \\delta^n_r \\quad ; \\quad\n",
|
||||
" \\delta^n_r = y^n_r - t^n_r \\\\\n",
|
||||
" \\frac{\\partial{E}}{\\partial{y_{r}}} = \\sum_{n=1}^N \\frac{\\partial{E^n}}{\\partial{y_{r}}} = \\sum_{n=1}^N \\delta^n_r\n",
|
||||
"$\n",
|
||||
"\n",
|
||||
"Similarly, using the above $\\delta^n_r$ one can express the gradient of the weight $w_{sr}$ (from the s-th input to the r-th output) for linear model and MSE cost as follows:\n",
|
||||
"\n",
|
||||
"(9) $\n",
|
||||
" \\frac{\\partial{E^n}}{\\partial{w_{sr}}} = (y^n_r - t^n_r)x_s^n = \\delta^n_r x_s^n \\quad\\\\\n",
|
||||
" \\frac{\\partial{E}}{\\partial{w_{sr}}} = \\sum_{n=1}^N \\frac{\\partial{E^n}}{\\partial{w_{rs}}} = \\sum_{n=1}^N \\delta^n_r x_s^n\n",
|
||||
"$\n",
|
||||
"\n",
|
||||
"and the gradient for bias parameter at the $r$-th output is:\n",
|
||||
"\n",
|
||||
"(10) $\n",
|
||||
" \\frac{\\partial{E}}{\\partial{b_{r}}} = \\sum_{n=1}^N \\frac{\\partial{E^n}}{\\partial{b_{r}}} = \\sum_{n=1}^N \\delta^n_r\n",
|
||||
"$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"\n",
|
||||
"\n",
|
||||
" \n",
|
||||
" * Input vector $\\mathbf{x} = (x_1, x_2, \\ldots, x_D) $\n",
|
||||
" * Output scalar $y_1$\n",
|
||||
" * Weight matrix $\\mathbf{W}$: $w_{ik}$ is the weight from input $x_i$ to output $y_k$. Note, here this is really a vector since a single scalar output, y_1.\n",
|
||||
" * Scalar bias $b$ for the only output in our model \n",
|
||||
" * Scalar target $t$ for the only output in out model\n",
|
||||
" \n",
|
||||
"First, ensure you can make use of data provider (note, for training data has been normalised to zero mean and unit variance, hence different effective range than one can find in file):"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Observations: [[-0.12 -0.13 -0.13 -0.13 -0.13 -0.13 -0.13 -0.13 -0.13 -0.13]\n",
|
||||
" [-0.11 -0.1 0.09 -0.06 -0.09 -0. 0.28 -0.12 -0.12 -0.08]\n",
|
||||
" [-0.13 0.05 -0.13 -0.01 -0.11 -0.13 -0.13 -0.13 -0.13 -0.13]\n",
|
||||
" [ 0.2 0.12 0.25 0.16 0.03 -0. 0.15 0.08 -0.08 -0.11]\n",
|
||||
" [-0.13 -0.12 -0.13 -0.13 -0.13 -0.13 -0.13 -0.13 -0.13 -0.13]\n",
|
||||
" [-0.1 0.51 1.52 0.14 -0.02 0.77 0.11 0.79 -0.02 0.08]\n",
|
||||
" [ 0.24 0.15 -0.01 0.08 -0.1 0.45 -0.12 -0.1 -0.13 0.48]\n",
|
||||
" [ 0.13 -0.06 -0.07 -0.11 -0.11 -0.11 -0.13 -0.11 -0.02 -0.12]\n",
|
||||
" [-0.06 0.28 -0.13 0.06 0.09 0.09 0.01 -0.07 0.14 -0.11]\n",
|
||||
" [-0.13 -0.13 -0.1 -0.06 -0.13 -0.13 -0.13 -0.13 -0.13 -0.13]]\n",
|
||||
"To predict: [[-0.12]\n",
|
||||
" [-0.12]\n",
|
||||
" [-0.13]\n",
|
||||
" [-0.1 ]\n",
|
||||
" [-0.13]\n",
|
||||
" [-0.08]\n",
|
||||
" [ 0.24]\n",
|
||||
" [-0.13]\n",
|
||||
" [-0.02]\n",
|
||||
" [-0.13]]\n",
|
||||
"Observations: [[-0.09 -0.13 -0.13 -0.03 -0.05 -0.11 -0.13 -0.13 -0.13 -0.13]\n",
|
||||
" [-0.03 0.32 0.28 0.09 -0.04 0.19 0.31 -0.13 0.37 0.34]\n",
|
||||
" [ 0.12 0.13 0.06 -0.1 -0.1 0.94 0.24 0.12 0.28 -0.04]\n",
|
||||
" [ 0.26 0.17 -0.04 -0.13 -0.12 -0.09 -0.12 -0.13 -0.1 -0.13]\n",
|
||||
" [-0.1 -0.1 -0.01 -0.03 -0.07 0.05 -0.03 -0.12 -0.05 -0.13]\n",
|
||||
" [-0.13 -0.13 -0.13 -0.13 -0.13 -0.13 0.1 -0.13 -0.13 -0.13]\n",
|
||||
" [-0.01 -0.1 -0.13 -0.13 -0.12 -0.13 -0.13 -0.13 -0.13 -0.11]\n",
|
||||
" [-0.11 -0.06 -0.11 0.02 -0.03 -0.02 -0.05 -0.11 -0.13 -0.13]\n",
|
||||
" [-0.01 0.25 -0.08 0.04 -0.1 -0.12 0.06 -0.1 0.08 -0.06]\n",
|
||||
" [-0.09 -0.09 -0.09 -0.13 -0.11 -0.12 -0. -0.02 0.19 -0.11]]\n",
|
||||
"To predict: [[-0.13]\n",
|
||||
" [-0.11]\n",
|
||||
" [-0.09]\n",
|
||||
" [-0.08]\n",
|
||||
" [ 0.19]\n",
|
||||
" [-0.13]\n",
|
||||
" [-0.13]\n",
|
||||
" [-0.03]\n",
|
||||
" [-0.13]\n",
|
||||
" [-0.11]]\n",
|
||||
"Observations: [[-0.08 -0.11 -0.11 0.32 0.05 -0.11 -0.13 0.07 0.08 0.63]\n",
|
||||
" [-0.07 -0.1 -0.09 -0.08 0.26 -0.05 -0.1 -0. 0.36 -0.12]\n",
|
||||
" [-0.03 -0.1 0.19 -0.02 0.35 0.38 -0.1 0.44 -0.02 0.21]\n",
|
||||
" [-0.12 -0. -0.02 0.19 -0.11 -0.11 -0.13 -0.11 -0.02 -0.13]\n",
|
||||
" [ 0.09 0.1 -0.03 -0.05 0. -0.12 -0.12 -0.13 -0.13 -0.13]\n",
|
||||
" [ 0.21 0.05 -0.12 -0.05 -0.08 -0.1 -0.13 -0.13 -0.13 -0.13]\n",
|
||||
" [-0.04 -0.11 0.19 0.16 -0.01 -0.07 -0. -0.06 -0.03 0.16]\n",
|
||||
" [ 0.09 0.05 0.51 0.34 0.16 0.51 0.56 0.21 -0.06 -0. ]\n",
|
||||
" [-0.13 -0.13 -0.13 -0.13 -0.13 -0.13 -0.13 -0.13 -0.09 0.49]\n",
|
||||
" [-0.06 -0.11 -0.13 0.06 -0.01 -0.12 0.54 0.2 -0.1 -0.11]]\n",
|
||||
"To predict: [[ 0.1 ]\n",
|
||||
" [ 0.09]\n",
|
||||
" [ 0.16]\n",
|
||||
" [-0.13]\n",
|
||||
" [-0.13]\n",
|
||||
" [ 0.04]\n",
|
||||
" [-0.1 ]\n",
|
||||
" [ 0.05]\n",
|
||||
" [-0.1 ]\n",
|
||||
" [-0.11]]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from mlp.dataset import MetOfficeDataProvider\n",
|
||||
"\n",
|
||||
"modp = MetOfficeDataProvider(10, batch_size=10, max_num_batches=3, randomize=True)\n",
|
||||
"\n",
|
||||
"%precision 2\n",
|
||||
"for x, t in modp:\n",
|
||||
" print 'Observations: ', x\n",
|
||||
" print 'To predict: ', t"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 4\n",
|
||||
"\n",
|
||||
"The below code implements a very simple variant of stochastic gradient descent for the weather regression example. Your task is to implement 5 functions in the next cell and then run two next cells that 1) build sgd functions and 2) run the actual training."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\n",
|
||||
"#When implementing those, take into account the mini-batch case, for which one is\n",
|
||||
"#expected to sum the errors for each example\n",
|
||||
"\n",
|
||||
"def fprop(x, W, b):\n",
|
||||
" #code implementing eq. (3)\n",
|
||||
" return numpy.dot(x, W) + b\n",
|
||||
"\n",
|
||||
"def cost(y, t):\n",
|
||||
" #Mean Square Error cost, equation (5)\n",
|
||||
" return numpy.mean(0.5*numpy.sum((y - t)**2, axis=1))\n",
|
||||
"\n",
|
||||
"def cost_grad(y, t):\n",
|
||||
" #Gradient of the cost w.r.t y equation (8)\n",
|
||||
" return y - t\n",
|
||||
"\n",
|
||||
"def cost_wrt_W(cost_grad, x):\n",
|
||||
" #Gradient of the cost w.r.t W, equation (9)\n",
|
||||
" return numpy.dot(x.T, cost_grad)\n",
|
||||
" \n",
|
||||
"def cost_wrt_b(cost_grad):\n",
|
||||
" #Gradient of the cost w.r.t to b, equation (10)\n",
|
||||
" return numpy.sum(cost_grad, axis = 0)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\n",
|
||||
"def sgd_epoch(data_provider, W, b, learning_rate):\n",
|
||||
" mse_stats = []\n",
|
||||
" \n",
|
||||
" #get the minibatch of data\n",
|
||||
" for x, t in data_provider:\n",
|
||||
"\n",
|
||||
" #1. get the estimate of y\n",
|
||||
" y = fprop(x, W, b)\n",
|
||||
"\n",
|
||||
" #2. compute the loss function\n",
|
||||
" tmp = cost(y, t)\n",
|
||||
" mse_stats.append(tmp)\n",
|
||||
" \n",
|
||||
" #3. compute the grad of the cost w.r.t the output layer activation y\n",
|
||||
" #i.e. how the cost changes when output y changes\n",
|
||||
" cost_grad_deltas = cost_grad(y, t)\n",
|
||||
"\n",
|
||||
" #4. compute the gradients w.r.t model's parameters\n",
|
||||
" grad_W = cost_wrt_W(cost_grad_deltas, x)\n",
|
||||
" grad_b = cost_wrt_b(cost_grad_deltas)\n",
|
||||
"\n",
|
||||
" #4. Update the model, we update with the mean gradient\n",
|
||||
" # over the minibatch, rather than sum of particular gradients\n",
|
||||
" # in a minibatch, to do so we scale the learning rate by batch_size\n",
|
||||
" batch_size = x.shape[0]\n",
|
||||
" effect_learn_rate = learning_rate / batch_size\n",
|
||||
"\n",
|
||||
" W = W - effect_learn_rate * grad_W\n",
|
||||
" b = b - effect_learn_rate * grad_b\n",
|
||||
" \n",
|
||||
" return W, b, numpy.mean(mse_stats)\n",
|
||||
"\n",
|
||||
"def sgd(data_provider, W, b, learning_rate=0.1, max_epochs=10):\n",
|
||||
" \n",
|
||||
" for epoch in xrange(0, max_epochs):\n",
|
||||
" #reset the data provider\n",
|
||||
" data_provider.reset()\n",
|
||||
" \n",
|
||||
" #train for one epoch\n",
|
||||
" W, b, mean_cost = \\\n",
|
||||
" sgd_epoch(data_provider, W, b, learning_rate)\n",
|
||||
" \n",
|
||||
" print \"MSE training cost after %d-th epoch is %f\" % (epoch + 1, mean_cost)\n",
|
||||
" \n",
|
||||
" return W, b\n",
|
||||
" \n",
|
||||
" "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"MSE training cost after 1-th epoch is 0.017213\n",
|
||||
"MSE training cost after 2-th epoch is 0.016103\n",
|
||||
"MSE training cost after 3-th epoch is 0.015705\n",
|
||||
"MSE training cost after 4-th epoch is 0.015437\n",
|
||||
"MSE training cost after 5-th epoch is 0.015255\n",
|
||||
"MSE training cost after 6-th epoch is 0.015128\n",
|
||||
"MSE training cost after 7-th epoch is 0.015041\n",
|
||||
"MSE training cost after 8-th epoch is 0.014981\n",
|
||||
"MSE training cost after 9-th epoch is 0.014936\n",
|
||||
"MSE training cost after 10-th epoch is 0.014903\n",
|
||||
"MSE training cost after 11-th epoch is 0.014879\n",
|
||||
"MSE training cost after 12-th epoch is 0.014862\n",
|
||||
"MSE training cost after 13-th epoch is 0.014849\n",
|
||||
"MSE training cost after 14-th epoch is 0.014839\n",
|
||||
"MSE training cost after 15-th epoch is 0.014830\n",
|
||||
"MSE training cost after 16-th epoch is 0.014825\n",
|
||||
"MSE training cost after 17-th epoch is 0.014820\n",
|
||||
"MSE training cost after 18-th epoch is 0.014813\n",
|
||||
"MSE training cost after 19-th epoch is 0.014813\n",
|
||||
"MSE training cost after 20-th epoch is 0.014810\n",
|
||||
"MSE training cost after 21-th epoch is 0.014808\n",
|
||||
"MSE training cost after 22-th epoch is 0.014805\n",
|
||||
"MSE training cost after 23-th epoch is 0.014806\n",
|
||||
"MSE training cost after 24-th epoch is 0.014804\n",
|
||||
"MSE training cost after 25-th epoch is 0.014796\n",
|
||||
"MSE training cost after 26-th epoch is 0.014798\n",
|
||||
"MSE training cost after 27-th epoch is 0.014801\n",
|
||||
"MSE training cost after 28-th epoch is 0.014802\n",
|
||||
"MSE training cost after 29-th epoch is 0.014801\n",
|
||||
"MSE training cost after 30-th epoch is 0.014799\n",
|
||||
"MSE training cost after 31-th epoch is 0.014799\n",
|
||||
"MSE training cost after 32-th epoch is 0.014793\n",
|
||||
"MSE training cost after 33-th epoch is 0.014800\n",
|
||||
"MSE training cost after 34-th epoch is 0.014796\n",
|
||||
"MSE training cost after 35-th epoch is 0.014799\n",
|
||||
"MSE training cost after 36-th epoch is 0.014800\n",
|
||||
"MSE training cost after 37-th epoch is 0.014798\n",
|
||||
"MSE training cost after 38-th epoch is 0.014799\n",
|
||||
"MSE training cost after 39-th epoch is 0.014799\n",
|
||||
"MSE training cost after 40-th epoch is 0.014794\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(array([[ 0.01],\n",
|
||||
" [ 0.03],\n",
|
||||
" [ 0.03],\n",
|
||||
" [ 0.04],\n",
|
||||
" [ 0.06],\n",
|
||||
" [ 0.07],\n",
|
||||
" [ 0.26]]), array([-0.]))"
|
||||
]
|
||||
},
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"\n",
|
||||
"#some hyper-parameters\n",
|
||||
"window_size = 7\n",
|
||||
"irange = 0.1\n",
|
||||
"learning_rate = 0.001\n",
|
||||
"max_epochs=40\n",
|
||||
"\n",
|
||||
"# note, while developing you can set max_num_batches to some positive number to limit\n",
|
||||
"# the number of training data-points (you will get feedback faster)\n",
|
||||
"mdp = MetOfficeDataProvider(window_size, batch_size=10, max_num_batches=-100, randomize=True)\n",
|
||||
"\n",
|
||||
"#initialise the parameters\n",
|
||||
"W = random_generator.uniform(-irange, irange, (window_size, 1))\n",
|
||||
"b = random_generator.uniform(-irange, irange, (1, ))\n",
|
||||
"\n",
|
||||
"#train the model\n",
|
||||
"sgd(mdp, W, b, learning_rate=learning_rate, max_epochs=max_epochs)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"source": [
|
||||
"## Exercise 5\n",
|
||||
"\n",
|
||||
"Modify the above regression problem so the model makes binary classification whether the the weather is going to be one of those \\{rainy, sunny} (look at slide 12 of the 2nd lecture)\n",
|
||||
"\n",
|
||||
"Tip: You need to introduce the following changes:\n",
|
||||
"1. Modify `MetOfficeDataProvider` (for example, inherit from MetOfficeDataProvider to create a new class MetOfficeDataProviderBin) and modify `next()` function so it returns as `targets` either 0 (sunny - if the the amount of rain [before mean/variance normalisation] is equal to 0 or 1 (rainy -- otherwise).\n",
|
||||
"2. Modify the functions from previous exercise so the fprop implements `sigmoid` on top of affine transform.\n",
|
||||
"3. Modify cost function to binary cross-entropy\n",
|
||||
"4. Make sure you compute the gradients correctly (as you have changed both the output and the cost)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"#sorry, this one will be added later..."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 2",
|
||||
"language": "python",
|
||||
"name": "python2"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 2
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython2",
|
||||
"version": "2.7.9"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
@ -60,6 +60,13 @@ class DataProvider(object):
|
||||
"""
|
||||
raise NotImplementedError()
|
||||
|
||||
def num_examples(self):
|
||||
"""
|
||||
Returns a number of data-points in dataset
|
||||
"""
|
||||
return NotImplementedError()
|
||||
|
||||
|
||||
|
||||
class MNISTDataProvider(DataProvider):
|
||||
"""
|
||||
@ -142,6 +149,9 @@ class MNISTDataProvider(DataProvider):
|
||||
|
||||
return rval_x, self.__to_one_of_k(rval_t)
|
||||
|
||||
def num_examples(self):
|
||||
return self.x.shape[0]
|
||||
|
||||
def __to_one_of_k(self, y):
|
||||
rval = numpy.zeros((y.shape[0], self.num_classes), dtype=numpy.float32)
|
||||
for i in xrange(y.shape[0]):
|
||||
|
||||
Loading…
Reference in New Issue
Block a user