anton/mlpractical
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge remote branch 'upstream/master'

Browse Source 
Conflicts:
	00_Introduction.ipynb
This commit is contained in:
pswietojanski 2015-10-01 13:28:51 +01:00
commit c71d6973f6
2 changed files with 681 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

400
00_Introduction.ipynb Normal file
View File

@ -0,0 +1,400 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Introduction\n",
"\n",
"This notebook shows how to set-up a working python envirnoment for the Machine Learning Practical course.\n",
"\n",
"\n",
"# Setting up the software\n",
"\n",
"Within this course we are going to work with python (using some auxiliary libraries like numpy and scipy). Depending on the infrastracture and working environment (e.g. DICE), root permission may not be not available so the packages cannot be installed in default locations. A convenient python configuration, which allows us to install and update third party libraries easily using package manager, are so called virtual environments. They can be also used to work (and test) the code with different versions of software.\n",
"\n",
"## Instructions for Windows\n",
"\n",
"The fastest way to get working setup on Windows is to install Anaconda (http://www.continuum.io) package. It's a python environment with precompiled versions of the most popular scientific python libraries. It also works on MacOS, but numpy is not linked without a fee to a numerical library, hence for MacOS we recommend the following procedure.\n",
"\n",
"## Instructions for MacOS\n",
"\n",
" * Install macports following instructions at https://www.macports.org/install.php\n",
" * Install the relevant python packages in macports\n",
"\n",
" ```\n",
" sudo port install py27-scipy +openblas\n",
" sudo port install py27-ipython +notebook\n",
" sudo port install py27-notebook\n",
" sudo port install py27-matplotlib\n",
" sudo port select --set python python27\n",
" sudo port select --set ipython2 py27-ipython\n",
" sudo port select --set ipython py27-ipython\n",
" ```\n",
"\n",
"Make sure that your `$PATH` has `/opt/local/bin` before `/usr/bin` so you pick up the version of python you just installed.\n",
"\n",
"## Instructions for DICE:\n",
"\n",
"### Directory structure and getting things organised\n",
"\n",
"To get things somehow standarized between people, and make life of everyone easier, we propse to organise your DICE setup in the following directory structure:\n",
"\n",
" * `~/mlpractical/` -- for a general course repository\n",
" * `~/mlpractical/repos-3rd` -- for stuff you download, build and install (numpy, OpenBlas, virtualenv)\n",
" * `~/mlpractical/repo-mlp` -- this is the actual course repository you clone from our website (do not create a dir for it yet!)\n",
" * `~/mlpractical/venv` -- this is where virutal envirnoment will make its dir (do not create a dir for it yet!)\n",
"\n",
"Create now repos-3rd directory (option -p in the below command will automatically create (non-existing) **p**arent directories (mlpractical):\n",
"\n",
" * `mkdir -p ~/mlpractical/repos-3rd`\n",
"\n",
"And now, let us set an MLP_WDIR environmental variable (MLP Working DIRectory) that will keep an absolute path of working dir pointing to `~/mlpractial`, **add the below line** to your `~/.bashrc` file (if it does not exists, create one using a text editor! e.g. by running `gedit ~/.bashrc`):\n",
"\n",
"```\n",
"export MLP_WDIR=~/mlpractical\n",
"```\n",
"\n",
"Now re-source `~/.bashrc` by typing (so the env variables get updated!): `source ~/.bashrc`\n",
"\n",
"Enter the `repos-3rd` directory by typing: `cd ~/mlpractical/repos-3rd` (or ```cd $MLP_WDIR/repos-3rd``` if you want)\n",
"\n",
"### Configuring virtual environment\n",
"\n",
"Make sure you are in `repos-3rd` directory and that MLP_WDIR variable has been exported (you may type export in the terminal and examine the list of availabe variables in the current session), then type:\n",
"\n",
" * `git clone https://github.com/pypa/virtualenv`\n",
" * Enter the cloned repository and type ```./virtualenv.py --python /usr/bin/python2.7 --no-site-packages $MLP_WDIR/venv```\n",
" * Activate the environment by typing `source ~/mlpractical/venv/bin/activate` (to leave the virtual environment one may type `decativate`)\n",
" * Environments need to be activated every time ones start the new session so we will now create a handy alias to it in `~/.bashrc` script, by typing the below command (note, MLP_WDIR export needs to preceed this command):\n",
" \n",
" ```alias activate_mlp=\"source $MLP_WDIR/venv/bin/activate\"```\n",
" \n",
"Then every time you open new session and want to activate the right virtual environment, simply type `activate_mlp` instead `source ~/mlpractical/venv/bin/activate`. Note, you need to re-soure the .bashrc in order alias to be visible in the current session.\n",
"\n",
"### Installing remaining packages\n",
"\n",
"Then, before you follow next, install/upgrade the following packages:\n",
"\n",
"```\n",
"pip install --upgrade pip\n",
"pip install setuptools\n",
"pip install setuptools --upgrade\n",
"pip install ipython\n",
"pip install notebook\n",
"```\n",
"\n",
"### Installing numpy\n",
"\n",
"Note, having virtual environment properly installed one may then run `pip install numpy` to use pip to install numpy, though this will most likely lead to the suboptimal configuration where numpy is linked to ATLAS numerical library, which on DICE is compiled in multi-threaded mode. This means whenever numpy use BLAS accelerated computations (using ATLAS), it will use **all** the available cores at the given machine. This happens because ATLAS can be compiled to either run computations in single *or* multi threaded modes. However, contrary to some other backends, the latter does not allow to use an arbitrary number of threads (specified by the user prior to computation). This is highly suboptimal, as the potential speed-up resulting from paralleism depends on many factors like the communication overhead between threads, the size of the problem, etc. Using all cores for our exercises is not-necessary.\n",
"\n",
"For which reason, we are going to compile our own version of BLAS package, called *OpenBlas*. It allows to specify the number of threads manually by setting an environmental variable OMP_NUM_THREADS=N, where N is a desired number of parallel threads (please use 1 by default). You can set an environment variable in the current shell by running\n",
"\n",
"```\n",
"export OMP_NUM_THREADS=1\n",
"```\n",
"\n",
"(note the lack of spaces around the equals sign and use of `export` to define an environment variable which will be available in sub-shells rather than just a variable local to the current shell).\n",
"\n",
"#### OpenBlas\n",
"\n",
"Enter again repos-3rd directory and copy into terminal the following commands (one at the time):\n",
"\n",
"```\n",
"cd ~/mlpractical/repos-3rd\n",
"OBDir=$MLP_WDIR/repos-3rd/OpenBLAS\n",
"git clone git://github.com/xianyi/OpenBLAS\n",
"cd OpenBLAS\n",
"make\n",
"make PREFIX=$OBDir install\n",
"```\n",
"\n",
"Once OpenBLAS is finished compiling we need to ensure the compiled shared library files in the `lib` subdirectory are available to the shared library loader. This can be done by appending the absolute path to the `lib` subdirectory to the `LD_LIBRARY_PATH` environment variable. To ensure this changes persist we will change the bash start up file `~/.bashrc` by opening it in a text editor (e.g. by running `gedit ~/.bashrc`) and adding the following line\n",
"\n",
"```\n",
"export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MLP_WDIR/repos-3rd/OpenBLAS/lib\n",
"```\n",
"\n",
"Note, we again are using MLP_WDIR here, so the above line needs to be placed after you set MLP_WDIR.\n",
"\n",
"After you have edited `.bashrc` run\n",
"\n",
"```\n",
"source ~/.bashrc\n",
"activate_mlp # This is the alias you set up in the bashrc\n",
"#source ~/mlpractical/venv/bin/activate\n",
"```\n",
"\n",
"to rerun the bash start up script make sure the new environment variable is available in the current shell and then reactivate the virtual environment.\n",
"\n",
"#### Numpy\n",
"\n",
"To install `numpy` linked against the OpenBLAS libraries we just compiled, first run the following commands (one at a time)\n",
"\n",
"```\n",
"cd ~/mlpractical/repos-3rd/\n",
"wget http://downloads.sourceforge.net/project/numpy/NumPy/1.9.2/numpy-1.9.2.zip\n",
"unzip numpy-1.9.2.zip\n",
"cd numpy-1.9.2\n",
"echo \"[openblas]\" >> site.cfg\n",
"echo \"library_dirs = $OBDir/lib\" >> site.cfg\n",
"echo \"include_dirs = $OBDir/include\" >> site.cfg\n",
"python setup.py build --fcompiler=gnu95\n",
"```\n",
"\n",
"Assuming the virtual environment is activated, the below command will install numpy in a desired space (`~/mlpractical/venv/...`):\n",
"\n",
"```\n",
"python setup.py install\n",
"```\n",
"\n",
"Now use pip to install remaining packages: `scipy`, `matplotlib`, `argparse`, and `nose` by executing:\n",
"\n",
"```\n",
"pip install scipy matplotlib argparse nose\n",
"```\n",
"\n",
"### Getting the mlpractical repository\n",
"\n",
"Clone the course repository from the github, by navigating to `~/mlpractical` directory and typing:\n",
"\n",
"```\n",
"cd $MLP_WDIR\n",
"git clone https://github.com/CSTR-Edinburgh/mlpractical.git repo-mlp\n",
"```\n",
"\n",
"When download is ready, enter the repo-mlp directory and start the actual interactive notebook session by typing:\n",
"\n",
"```\n",
"cd repo-mlp\n",
"ipython notebook\n",
"```\n",
"\n",
"This should start a ipython server which opens a new browser window listing files in `repo-mlp` directory, including `00_Introduction.ipynb.`. Open it and run (from the browser interface) the following examples and exercies."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"scrolled": true
},
"outputs": [],
"source": [
"%clear\n",
"import numpy\n",
"# show_config() prints the configuration of numpy numerical backend \n",
"# you should be able to see linkage to OpenBlas or some other library\n",
"# in case those are empty, it means something went wrong and \n",
"# numpy will use a default (slow) pythonic implementation for algebra\n",
"numpy.show_config()\n",
"#numpy.test()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Also, below we check whether and how much speedup one may expect by using different number of cores:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import os\n",
"import multiprocessing\n",
"import timeit\n",
"\n",
"num_cores = multiprocessing.cpu_count()\n",
"N = 1000\n",
"x = numpy.random.random((N,N))\n",
"\n",
"for i in xrange(0, num_cores):\n",
" # first, set the number of threads OpenBLAS\n",
" # should use, the below line is equivalent\n",
" # to typing export OMP_NUM_THREADS=i+1 in bash shell\n",
" print 'Running matrix-matrix product on %i core(s)' % i\n",
" os.environ['OMP_NUM_THREADS'] = str(i+1)\n",
" %%timeit numpy.dot(x,x.T)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Test whether you can plot and display the figures using pyplot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Remove the below line if not running this code in an ipython notebook\n",
"# It's a special command allowing the notebook to display plots inline\n",
"%matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"\n",
"x = numpy.linspace(0.0, 2*numpy.pi, 100)\n",
"y1 = numpy.sin(x)\n",
"y2 = numpy.cos(x)\n",
"\n",
"plt.plot(x, y1, lw=2, label=r'$\\sin(x)$')\n",
"plt.plot(x, y2, lw=2, label=r'$\\cos(x)$')\n",
"plt.xlabel('x')\n",
"plt.ylabel('y')\n",
"plt.legend()\n",
"plt.xlim(0.0, 2*numpy.pi)\n",
"plt.grid()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Exercises\n",
"\n",
"Today exercises are meant to get you familiar with ipython notebooks (if you haven't used them so far), data organisation and how to access it. Next week onwars, we will follow with the material covered in lectures.\n",
"\n",
"## Data providers\n",
"\n",
"Open (in the browser) `mlp.dataset` module (go to `Home` tab and navigate to mlp package, then click on the link `dataset.py`). Have a look thourgh the code and comments, then follow to exercises.\n",
"\n",
"<b>General note:</b> you can load the mlp code into your favourite python IDE but it is totally OK if you work (modify & save) the code directly in the browser by opening/modyfing the necessary modules in the tabs.\n",
"\n",
"### Exercise 1 \n",
"\n",
"Using MNISTDataProvider, write a code that iterates over the first 5 minibatches of size 100 data-points. Print MNIST digits in 10x10 images grid plot. Images are returned from the provider as tuples of numpy arrays `(features, targets)`. The `features` matrix has shape BxD while the `targets` vector is of size B, where B is the size of a mini-batch and D is dimensionality of the features. By deafult, each data-point (image) is stored in a 784 dimensional vector of pixel intensities normalised to [0,1] range from an inital integer values [0-255]. However, the original spatial domain is two dimensional, so before plotting you need to convert it into 2D matrix (MNIST images have the same number of pixels for height and width).\n",
"\n",
"Tip: Useful functions for this exercise are: imshow, subplot, gridspec"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import matplotlib.gridspec as gridspec\n",
"import matplotlib.cm as cm\n",
"from mlp.dataset import MNISTDataProvider\n",
"\n",
"def show_mnist_image(img):\n",
" fig = plt.figure()\n",
" gs = gridspec.GridSpec(1, 1)\n",
" ax1 = fig.add_subplot(gs[0,0])\n",
" ax1.imshow(img, cmap=cm.Greys_r)\n",
" plt.show()\n",
"\n",
"def show_mnist_images(batch):\n",
" raise NotImplementedError('Write me!')\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# An example for a single MNIST image\n",
"mnist_dp = MNISTDataProvider(dset='valid', batch_size=1, max_num_examples=2, randomize=False)\n",
"\n",
"for batch in mnist_dp:\n",
" features, targets = batch\n",
" show_mnist_image(features.reshape(28, 28))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"#implement here Exercise 1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exercise 2\n",
"\n",
"`MNISTDataProvider` as `targets` currently returns a vector of integers, each element in this vector represents an id of the category `features` data-point represent. Later in the course we are going to need 1-of-K representation of targets, for instance, given the minibatch of size 3 and the corresponding targets vector $[2, 2, 0]$ (and assuming there are only 3 different classes to discriminate between), one needs to convert it into matrix $\\left[ \\begin{array}{ccc}\n",
"0 & 0 & 1 \\\\\n",
"0 & 0 & 1 \\\\\n",
"1 & 0 & 0 \\end{array} \\right]$. \n",
"\n",
"Implement `__to_one_of_k` method of `MNISTDataProvider` class. Then modify (uncomment) an appropriate line in its `next` method, so the raw targets get converted to `1 of K` coding. Test the code in the cell below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Exercise 3\n",
"\n",
"Write your own data provider `MetOfficeDataProvider` that wraps the weather data for south Scotland (could be obtained from: http://www.metoffice.gov.uk/hadobs/hadukp/data/daily/HadSSP_daily_qc.txt). The file was also downloaded and stored in `data` directory for your convenience. The provider should return a tuple `(x,t)` of the estimates over an arbitrary time windows (i.e. last N-1 days) for `x` and the N-th day as the one which model should be able to predict, `t`. For now, skip missing data-points (denoted by -99.9) and simply use the next correct value. Make sure the provider works for arbitrary `batch_size` settings, including the case where single mini-batch is equal to all datapoints in the dataset. Test the dataset in the cell below.\n",
"\n",
"Tip: To follow with this exercise, copy MNISTDataProvider in dataset.py, rename it to `MetOfficeDataProvider` and reimplement necesarry parts (including the arguments you pass to the constructor)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.5"
}
},
"nbformat": 4,
"nbformat_minor": 0
}

281
01_Linear_Models.ipynb Normal file
View File

@ -0,0 +1,281 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Introduction\n",
"\n",
"This tutorial is about linear transforms - a basic building block of many, including deep learning, models.\n",
"\n",
"# Short recap and syncing repositories\n",
"\n",
"Before you proceed onwards, remember to activate you virtual environments so you can use the software you installed last week as well as run the notebooks in interactive mode, no through github.com website.\n",
"\n",
"## Virtual environments\n",
"\n",
"To activate virtual environment:\n",
" * If you were on Tuesday/Wednesday group type `activate_mlp` or `source ~/mlpractical/venv/bin/activate`\n",
" * If you were on Monday group:\n",
" + and if you have chosen **comfy** way type: workon mlpractival\n",
" + and if you have chosen **generic** way, `source` your virutal environment using `source` and specyfing the path to the activate script (you need to localise it yourself, there were not any general recommendations w.r.t dir structure and people have installed it in different places, usually somewhere in the home directories. If you cannot easily find it by yourself, use something like: `find . -iname activate` ):\n",
"\n",
"## On Synchronising repositories\n",
"\n",
"I started writing this, but do not think giving students a choice is a good way to progess, the most painless way to follow would be to ask them to stash their changes (with some meaningful message) and work on the clean updated repository. This way one can always (temporarily) recover the work once needed but everyone starts smoothly the next lab. We do not want to help anyone how to resolve the conflicts...\n",
"\n",
"Enter your git mlp repository you set up last week (i.e. ~/mlpractical/repo-mlp) and depending on how you want to proceed you either can:\n",
" 1. Overridde some changes you have made (both in the notebooks and/or in the code if you happen to modify parts that were updated by us) with the code we have provided for this lab\n",
" 2. Try to merge your code with ours (for example, if you want to use `MetOfficeDataProvider` you have written)\n",
" \n",
"Our recommendation is, you should at least keep the progress in the notebooks (so you can peek some details when needed)\n",
" \n",
"```\n",
"git pull\n",
"```\n",
"\n",
"## Default Synchronising Strategy\n",
"\n",
"Need to think/discuss this."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Linear and Affine Transforms\n",
"\n",
"Depending on the required level of details, one may need to. The basis of all linear models is so called affine transform, that is the transform that implements some (linear) rotation of some input points and shift (translation) them. Denote by $\\vec x$ some input vector, then the affine transform is defined as follows:\n",
"\n",
"![Making Predictions](res/singleLayerNetWts-1.png)\n",
"\n",
"$\n",
"\\begin{equation}\n",
" \\mathbf y=\\mathbf W \\mathbf x + \\mathbf b\n",
"\\end{equation}\n",
"$\n",
"\n",
"<b>Note:</b> the bias term can be incorporated as an additional column in the weight matrix, though in this tutorials we will use a separate variable to for this purpose.\n",
"\n",
"An $i$th element of vecotr $\\mathbf y$ is hence computed as:\n",
"\n",
"$\n",
"\\begin{equation}\n",
" y_i=\\mathbf w_i \\mathbf x + b_i\n",
"\\end{equation}\n",
"$\n",
"\n",
"where $\\mathbf w_i$ is the $i$th row of $\\mathbf W$\n",
"\n",
"$\n",
"\\begin{equation}\n",
" y_i=\\sum_j w_{ji}x_j + b_i\n",
"\\end{equation}\n",
"$\n",
"\n",
"???\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ 0.06875593 -0.69616488 0.08823301 0.34533413 -0.22129962]\n"
]
}
],
"source": [
"import numpy\n",
"x=numpy.random.uniform(-1,1,(4,)); \n",
"W=numpy.random.uniform(-1,1,(5,4)); \n",
"y=numpy.dot(W,x);\n",
"print y"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[ 0.63711 0.11566944 0.74416104]\n",
" [-0.01335825 0.46206922 -0.1109265 ]\n",
" [-0.37523063 -0.06755371 0.04352121]\n",
" [ 0.25885831 -0.53660826 -0.40905639]]\n"
]
}
],
"source": [
"def my_dot(x, W, b):\n",
" y = numpy.zeros_like((x.shape[0], W.shape[1]))\n",
" raise NotImplementedError('Write me!')\n",
" return y"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ 0 1 2 3 4 5 6 7 8 9 10]\n"
]
}
],
"source": [
"\n",
"for itr in xrange(0,100):\n",
" my_dot(W,x)\n",
" \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Iterative learning of linear models\n",
"\n",
"We will learn the model with (batch for now) gradient descent.\n",
"\n",
"\n",
"## Running example\n",
"\n",
"![Making Predictions](res/singleLayerNetPredict.png)\n",
" \n",
"\n",
" * Input vector $\\mathbf{x} = (x_1, x_1, \\ldots, x_d)^T $\n",
" * Output vector $\\mathbf{y} = (y_1, \\ldots, y_K)^T $\n",
" * Weight matrix $\\mathbf{W}$: $w_{ki}$ is the weight from input $x_i$ to output $y_k$\n",
" * Bias $w_{k0}$ is the bias for output $k$\n",
" * Targets vector $\\mathbf{t} = (t_1, \\ldots, t_K)^T $\n",
"\n",
"\n",
"$\n",
" y_k = \\sum_{i=1}^d w_{ki} x_i + w_{k0}\n",
"$\n",
"\n",
"If we define $x_0=1$ we can simplify the above to\n",
"\n",
"$\n",
" y_k = \\sum_{i=0}^d w_{ki} x_i \\quad ; \\quad \\mathbf{y} = \\mathbf{Wx}\n",
"$\n",
"\n",
"$\n",
"E = \\frac{1}{2} \\sum_{n=1}^N ||\\mathbf{y}^n - \\mathbf{t}^n||^2 = \\sum_{n=1}^N E^n \\\\\n",
" E^n = \\frac{1}{2} ||\\mathbf{y}^n - \\mathbf{t}^n||^2\n",
"$\n",
"\n",
" $ E^n = \\frac{1}{2} \\sum_{k=1}^K (y_k^n - t_k^n)^2 $\n",
" set $\\mathbf{W}$ to minimise $E$ given the training set\n",
" \n",
"$\n",
" E^n = \\frac{1}{2} \\sum_{k=1}^K (y^n_k - t^n_k)^2 \n",
" = \\frac{1}{2} \\sum_{k=1}^K \\left( \\sum_{i=0}^d w_{ki} x^n_i - t^n_k \\right)^2 \\\\\n",
" \\pderiv{E^n}{w_{rs}} = (y^n_r - t^n_r)x_s^n = \\delta^n_r x_s^n \\quad ; \\quad\n",
" \\delta^n_r = y^n_r - t^n_r \\\\\n",
" \\pderiv{E}{w_{rs}} = \\sum_{n=1}^N \\pderiv{E^n}{w_{rs}} = \\sum_{n=1}^N \\delta^n_r x_s^n\n",
"$\n",
"\n",
"\n",
"\\begin{algorithmic}[1]\n",
" \\Procedure{gradientDescentTraining}{$\\mvec{X}, \\mvec{T},\n",
" \\mvec{W}$}\n",
" \\State initialize $\\mvec{W}$ to small random numbers\n",
"% \\State randomize order of training examples in $\\mvec{X}\n",
" \\While{not converged}\n",
" \\State for all $k,i$: $\\Delta w_{ki} \\gets 0$\n",
" \\For{$n \\gets 1,N$}\n",
" \\For{$k \\gets 1,K$}\n",
" \\State $y_k^n \\gets \\sum_{i=0}^d w_{ki} x_{ki}^n$\n",
" \\State $\\delta_k^n \\gets y_k^n - t_k^n$\n",
" \\For{$i \\gets 1,d$}\n",
" \\State $\\Delta w_{ki} \\gets \\Delta w_{ki} + \\delta_k^n \\cdot x_i^n$\n",
" \\EndFor\n",
" \\EndFor\n",
" \\EndFor\n",
" \\State for all $k,i$: $w_{ki} \\gets w_{ki} - \\eta \\cdot \\Delta w_{ki}$\n",
" \\EndWhile\n",
" \\EndProcedure\n",
"\\end{algorithmic}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Excercises"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Fun Stuff\n",
"\n",
"So what on can do with linear transform, and what are the properties of those?\n",
"\n",
"Exercise, show, the LT is invertible, basically, solve the equation:\n",
"\n",
"y=Wx+b, given y (transformed image), find such x that is the same as the original one."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 0
}