anton/mlpractical
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Updating instructions for accessing data.

Browse Source
This commit is contained in:
Matt Graham 2017-01-27 15:26:33 +00:00
parent 773c05eabe
commit c6a4905fd3
2 changed files with 134 additions and 54 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

95
notebooks/09a_Object_recognition_with_CIFAR-10_and_CIFAR-100.ipynb
View File

@ -1,21 +1,5 @@
{ {
"cells": [ "cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import os\n",
"import tensorflow as tf\n",
"import numpy as np\n",
"from mlp.data_providers import CIFAR10DataProvider, CIFAR100DataProvider\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -132,7 +116,7 @@
"\n", "\n",
"Both CIFAR-10 and CIFAR-100 have standard splits into 50000 training examples and 10000 test examples. For CIFAR-100 as there is an optional Kaggle competition (see below) scored on predictions on the test set, we have used a non-standard assignation of examples to test and training set and only provided the inputs (and not target labels) for the 10000 examples chosen for the test set. \n", "Both CIFAR-10 and CIFAR-100 have standard splits into 50000 training examples and 10000 test examples. For CIFAR-100 as there is an optional Kaggle competition (see below) scored on predictions on the test set, we have used a non-standard assignation of examples to test and training set and only provided the inputs (and not target labels) for the 10000 examples chosen for the test set. \n",
"\n", "\n",
"For CIFAR-10 the 10000 test set examples have labels provided: to avoid any accidental over-fitting to the test set **you should only use these for the final evaluation of your model(s)**. If you repeatedly evaluate models on the test set during model development it is easy to end up indirectly fitting to the test labels - for those who have not already read it see this [excellent cautionary note from the MLPR notes by Iain Murray](http://www.inf.ed.ac.uk/teaching/courses/mlpr/2016/notes/w2a_train_test_val.html#fnref2). \n", "For CIFAR-10 the 10000 test set examples have labels provided: to avoid any accidental over-fitting to the test set **you should only use these for the final evaluation of your model(s)**. If you repeatedly evaluate models on the test set during model development it is easy to end up indirectly fitting to the test labels - for those who have not already read it see this [excellent cautionary note from the MLPR notes by Iain Murray](http://www.inf.ed.ac.uk/teaching/courses/mlpr/2016/notes/w2a_train_test_val.html#warning-dont-fool-yourself-or-make-a-fool-of-yourself). \n",
"\n", "\n",
"For both CIFAR-10 and CIFAR-100, the remaining 50000 non-test examples have been split in to a 40000 example training dataset and a 10000 example validation dataset, each with target labels provided. If you wish to use a more complex cross-fold validation scheme you may want to combine these two portions of the dataset and define your own functions for separating out a validation set.\n", "For both CIFAR-10 and CIFAR-100, the remaining 50000 non-test examples have been split in to a 40000 example training dataset and a 10000 example validation dataset, each with target labels provided. If you wish to use a more complex cross-fold validation scheme you may want to combine these two portions of the dataset and define your own functions for separating out a validation set.\n",
"\n", "\n",
@ -140,14 +124,18 @@
"\n", "\n",
"The CIFAR-100 data provider also takes an optional `use_coarse_targets` argument in its constructor. By default this is set to `False` and the targets returned by the data provider correspond to 1-of-K encoded binary vectors for the 100 fine-grained object classes. If `use_coarse_targets=True` then instead the data provider will return 1-of-K encoded binary vector targets for the 20 coarse-grained superclasses associated with each input instead.\n", "The CIFAR-100 data provider also takes an optional `use_coarse_targets` argument in its constructor. By default this is set to `False` and the targets returned by the data provider correspond to 1-of-K encoded binary vectors for the 100 fine-grained object classes. If `use_coarse_targets=True` then instead the data provider will return 1-of-K encoded binary vector targets for the 20 coarse-grained superclasses associated with each input instead.\n",
"\n", "\n",
"Both data provider classes provide a `label_map` attribute which is a list of strings which are the class labels corresponding to the integer targets (i.e. prior to conversion to a 1-of-K encoded binary vector).\n", "Both data provider classes provide a `label_map` attribute which is a list of strings which are the class labels corresponding to the integer targets (i.e. prior to conversion to a 1-of-K encoded binary vector)."
"\n", ]
"Below example code is given for creating instances of the CIFAR-10 and CIFAR-100 data provider objects and using them to train simple two-layer feedforward network models with rectified linear activations in TensorFlow. You may wish to use this code as a starting point for your own experiments.\n", },
"\n", {
"\n", "cell_type": "markdown",
"metadata": {},
"source": [
"### Accessing the CIFAR-10 and CIFAR-100 data\n", "### Accessing the CIFAR-10 and CIFAR-100 data\n",
"\n", "\n",
"Before using the data provider objects you will need to copy the associated data files in to your local `mlp/data` directory (or wherever your `MLP_DATA_DIR` environment variable points to if different). The data is available as six compressed NumPy `.npz` files\n", "Before using the data provider objects you will need to make sure the data files are accessible to the `mlp` package by existing under the directory specified by the `MLP_DATA_DIR` path.\n",
"\n",
"The data is available as compressed NumPy `.npz` files\n",
"\n", "\n",
" cifar-10-train.npz 235MB\n", " cifar-10-train.npz 235MB\n",
" cifar-10-valid.npz 59MB\n", " cifar-10-valid.npz 59MB\n",
@ -157,12 +145,40 @@
" cifar-100-valid.npz 59MB\n", " cifar-100-valid.npz 59MB\n",
" cifar-100-test-inputs.npz 59MB\n", " cifar-100-test-inputs.npz 59MB\n",
"\n", "\n",
"in the AFS directory `/afs/inf.ed.ac.uk/group/teaching/mlp/data`. Assuming your local `mlpractical` repository is in your home directory you should be able to copy the required files by running\n", "\n",
"in the AFS directory `/afs/inf.ed.ac.uk/group/teaching/mlp/data`.\n",
"\n",
"If you are working on DICE one option is to redefine your `MLP_DATA_DIR` to directly point to the shared AFS data directory by editing the `env_vars.sh` start up file for your environment. This will avoid using up your DICE quota by storing the data files in your homespace but may involve slower initial loading of the data on initialising the data providers if many people are trying access the same files at once. The environment variable can be redefined by running\n",
"\n",
"```\n",
"gedit ~/miniconda2/envs/mlp/etc/conda/activate.d/env_vars.sh\n",
"```\n",
"\n",
"in a terminal window (assuming you installed `miniconda2` to your home directory), and changing the line\n",
"\n",
"```\n",
"export MLP_DATA_DIR=$HOME/mlpractical/data\n",
"```\n",
"\n",
"to\n",
"\n",
"```\n",
"export MLP_DATA_DIR=\"/afs/inf.ed.ac.uk/group/teaching/mlp/data\"\n",
"```\n",
"\n",
"and then saving and closing the editor. You will need reload the `mlp` environment using `source activate mlp` and restart the Jupyter notebook server in the reloaded environment for the new environment variable definition to be available.\n",
"\n",
"For those working on DICE who have sufficient quota remaining or those using there own machine, an alternative option is to copy the data files in to your local `mlp/data` directory (or wherever your `MLP_DATA_DIR` environment variable currently points to if different). \n",
"\n",
"\n",
"Assuming your local `mlpractical` repository is in your home directory you should be able to copy the required files on DICE by running\n",
"\n", "\n",
"```\n", "```\n",
"cp /afs/inf.ed.ac.uk/group/teaching/mlp/data/cifar*.npz ~/mlpractical/data\n", "cp /afs/inf.ed.ac.uk/group/teaching/mlp/data/cifar*.npz ~/mlpractical/data\n",
"```\n", "```\n",
"\n", "\n",
"On a non-DICE machine, you will need to either [set up local access to AFS](http://computing.help.inf.ed.ac.uk/informatics-filesystem), use a remote file transfer client like `scp` or you can alternatively download the files using the iFile web interface [here](https://ifile.inf.ed.ac.uk/?path=%2Fafs%2Finf.ed.ac.uk%2Fgroup%2Fteaching%2Fmlp%2Fdata&goChange=Go) (requires DICE credentials).\n",
"\n",
"As some of the files are quite large you may wish to copy only those you are using currently (e.g. only the files for one of the two tasks) to your local filespace to avoid filling up your quota. The `cifar-100-test-inputs.npz` file will only be needed by those intending to enter the associated optional Kaggle competition." "As some of the files are quite large you may wish to copy only those you are using currently (e.g. only the files for one of the two tasks) to your local filespace to avoid filling up your quota. The `cifar-100-test-inputs.npz` file will only be needed by those intending to enter the associated optional Kaggle competition."
] ]
}, },
@ -170,7 +186,32 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Example two-layer classifier model on CIFAR-10" "## Example two-layer classifier models\n",
"\n",
"Below example code is given for creating instances of the CIFAR-10 and CIFAR-100 data provider objects and using them to train simple two-layer feedforward network models with rectified linear activations in TensorFlow. You may wish to use this code as a starting point for your own experiments."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import os\n",
"import tensorflow as tf\n",
"import numpy as np\n",
"from mlp.data_providers import CIFAR10DataProvider, CIFAR100DataProvider\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### CIFAR-10"
] ]
}, },
{ {
@ -276,7 +317,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Example two-layer classifier model on CIFAR-100" "### CIFAR-100"
] ]
}, },
{ {
@ -368,7 +409,7 @@
"source": [ "source": [
"## Predicting test data classes and creating a Kaggle submission file\n", "## Predicting test data classes and creating a Kaggle submission file\n",
"\n", "\n",
"An optional [Kaggle in Class](https://inclass.kaggle.com/c/mlp2016-7-cifar-100) competition (see email for invite link, you will need to sign-up with a `ed.ac.uk` email address to be able to enter) is being run on the CIFAR-100 (fine-grained) classification task. The scores for the competition are calculated by calculating the proportion of classes correctly predicted on the test set inputs (for which no class labels are provided). Half of the 10000 test inputs are used to calculate a public leaderboard score which will be visible while the competition is in progress and the other half are used to compute the private leaderboard score which will only be unveiled at the end of the competition. Each entrant can make up to two submissions of predictions each day during the competition.\n", "An optional [Kaggle in Class](https://inclass.kaggle.com/c/ml2016-7-cifar-100) competition (see email for invite link, you will need to sign-up with a `ed.ac.uk` email address to be able to enter) is being run on the CIFAR-100 (fine-grained) classification task. The scores for the competition are calculated by calculating the proportion of classes correctly predicted on the test set inputs (for which no class labels are provided). Half of the 10000 test inputs are used to calculate a public leaderboard score which will be visible while the competition is in progress and the other half are used to compute the private leaderboard score which will only be unveiled at the end of the competition. Each entrant can make up to two submissions of predictions each day during the competition.\n",
"\n", "\n",
"The code and helper function below illustrate how to use the predicted outputs of the TensorFlow network model we just trained to create a submission file which can be uploaded to Kaggle. The required format of the submission file is a `.csv` (Comma Separated Variable) file with two columns: the first is the integer index of the test input in the array in the provided data file (i.e. first row 0, second row 1 and so on) and the second column the corresponding predicted class label as an integer between 0 and 99 inclusive. The predictions must be preceded by a header line as in the following example\n", "The code and helper function below illustrate how to use the predicted outputs of the TensorFlow network model we just trained to create a submission file which can be uploaded to Kaggle. The required format of the submission file is a `.csv` (Comma Separated Variable) file with two columns: the first is the integer index of the test input in the array in the provided data file (i.e. first row 0, second row 1 and so on) and the second column the corresponding predicted class label as an integer between 0 and 99 inclusive. The predictions must be preceded by a header line as in the following example\n",
"\n", "\n",

93
notebooks/09b_Music_genre_classification_with_the_Million_Song_Dataset.ipynb
View File

@ -1,21 +1,5 @@
{ {
"cells": [ "cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import os\n",
"import tensorflow as tf\n",
"import numpy as np\n",
"from mlp.data_providers import MSD10GenreDataProvider, MSD25GenreDataProvider\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
]
},
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
@ -64,7 +48,7 @@
"\n", "\n",
"For each of these 10 classes, 5000 labelled examples have been collected for training / validation (i.e. 50000 example in total) and a further 1000 example per class for testing, with the exception of the `Blues` class for which only 991 testing examples are provided due to there being insufficient labelled tracks of the minimum required length (i.e. a total of 9991 test examples). \n", "For each of these 10 classes, 5000 labelled examples have been collected for training / validation (i.e. 50000 example in total) and a further 1000 example per class for testing, with the exception of the `Blues` class for which only 991 testing examples are provided due to there being insufficient labelled tracks of the minimum required length (i.e. a total of 9991 test examples). \n",
"\n", "\n",
"The 9991 test set examples have labels provided: however to avoid any accidental over-fitting to the test set **you should only use these for the final evaluation of your model(s)**. If you repeatedly evaluate models on the test set during model development it is easy to end up indirectly fitting to the test labels - for those who have not already read it see this [excellent cautionary note int the MLPR notes by Iain Murray](http://www.inf.ed.ac.uk/teaching/courses/mlpr/2016/notes/w2a_train_test_val.html#fnref2). \n", "The 9991 test set examples have labels provided: however to avoid any accidental over-fitting to the test set **you should only use these for the final evaluation of your model(s)**. If you repeatedly evaluate models on the test set during model development it is easy to end up indirectly fitting to the test labels - for those who have not already read it see this [excellent cautionary note int the MLPR notes by Iain Murray](http://www.inf.ed.ac.uk/teaching/courses/mlpr/2016/notes/w2a_train_test_val.html#warning-dont-fool-yourself-or-make-a-fool-of-yourself). \n",
"\n", "\n",
"\n", "\n",
"The 25-genre classification tasks uses the [*MSD Allmusic Style Dataset*](http://www.ifs.tuwien.ac.at/mir/msd/MASD.html) labels derived from the [AllMusic.com](http://www.allmusic.com/) database by [Alexander Schindler, Rudolf Mayer and Andreas Rauber of Vienna University of Technology](http://www.ifs.tuwien.ac.at/~schindler/pubs/ISMIR2012.pdf). The 25 genre / style labels used are:\n", "The 25-genre classification tasks uses the [*MSD Allmusic Style Dataset*](http://www.ifs.tuwien.ac.at/mir/msd/MASD.html) labels derived from the [AllMusic.com](http://www.allmusic.com/) database by [Alexander Schindler, Rudolf Mayer and Andreas Rauber of Vienna University of Technology](http://www.ifs.tuwien.ac.at/~schindler/pubs/ISMIR2012.pdf). The 25 genre / style labels used are:\n",
@ -103,13 +87,16 @@
"\n", "\n",
"Data provider classes for both fixed length input data for the 10 and 25 genre classification tasks in the `mlp.data_providers` module as `MSD10GenreDataProvider` and `MSD25GenreDataProvider`. Both have similar behaviour to the `MNISTDataProvider` used extensively last semester. A `which_set` argument can be used to specify whether to return a data provided for the training dataset (`which_set='train'`) or validation dataset (`which_set='valid'`). Both data provider classes provide a `label_map` attribute which is a list of strings which are the class labels corresponding to the integer targets (i.e. prior to conversion to a 1-of-K encoded binary vector).\n", "Data provider classes for both fixed length input data for the 10 and 25 genre classification tasks in the `mlp.data_providers` module as `MSD10GenreDataProvider` and `MSD25GenreDataProvider`. Both have similar behaviour to the `MNISTDataProvider` used extensively last semester. A `which_set` argument can be used to specify whether to return a data provided for the training dataset (`which_set='train'`) or validation dataset (`which_set='valid'`). Both data provider classes provide a `label_map` attribute which is a list of strings which are the class labels corresponding to the integer targets (i.e. prior to conversion to a 1-of-K encoded binary vector).\n",
"\n", "\n",
"The corresponding variable length input data are included as data files `msd-10-genre-train_var-length.npz`, `msd-10-genre-valid-var-length.npz`.\n", "The test dataset files for the 10 genre classification task are provided as two separate NumPy data files `msd-10-genre-test-inputs.npz` and `msd-10-genre-test-targets.npz`. These can be loaded using [`np.load`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.load.html) function. The inputs are stored as a $10000\\times3000$ array under the key `inputs` in the file `msd-10-genre-test-inputs.npz` and the targets in a 10000 element array of integer labels under the key `targets` in `msd-10-genre-test-targets.npz`. A corresponding `msd-25-genre-test-inputs.npz` file is provided for the 25 genre task inputs."
"\n", ]
"Below example code is given for creating instances of the 10-genre and 25-genre fixed-length input data provider objects and using them to train simple two-layer feedforward network models with rectified linear activations in TensorFlow.\n", },
"\n", {
"cell_type": "markdown",
"metadata": {},
"source": [
"### Accessing the Million Song Dataset data\n", "### Accessing the Million Song Dataset data\n",
"\n", "\n",
"Before using the data provider objects you will need to copy the associated data files in to your local `mlp/data` directory (or wherever your `MLP_DATA_DIR` environment variable points to if different). \n", "Before using the data provider objects you will need to make sure the data files are accessible to the `mlp` package by existing under the directory specified by the `MLP_DATA_DIR` path.\n",
"\n", "\n",
"The fixed length input data and associated targets is available as compressed NumPy `.npz` files\n", "The fixed length input data and associated targets is available as compressed NumPy `.npz` files\n",
"\n", "\n",
@ -121,7 +108,32 @@
" msd-25-genre-valid.npz 53MB\n", " msd-25-genre-valid.npz 53MB\n",
" msd-25-genre-test-inputs.npz 53MB\n", " msd-25-genre-test-inputs.npz 53MB\n",
"\n", "\n",
"in the AFS directory `/afs/inf.ed.ac.uk/group/teaching/mlp/data`. Assuming your local `mlpractical` repository is in your home directory you should be able to copy the required files by running\n", "in the AFS directory `/afs/inf.ed.ac.uk/group/teaching/mlp/data`.\n",
"\n",
"If you are working on DICE one option is to redefine your `MLP_DATA_DIR` to directly point to the shared AFS data directory by editing the `env_vars.sh` start up file for your environment. This will avoid using up your DICE quota by storing the data files in your homespace but may involve slower initial loading of the data on initialising the data providers if many people are trying access the same files at once. The environment variable can be redefined by running\n",
"\n",
"```\n",
"gedit ~/miniconda2/envs/mlp/etc/conda/activate.d/env_vars.sh\n",
"```\n",
"\n",
"in a terminal window (assuming you installed `miniconda2` to your home directory), and changing the line\n",
"\n",
"```\n",
"export MLP_DATA_DIR=$HOME/mlpractical/data\n",
"```\n",
"\n",
"to\n",
"\n",
"```\n",
"export MLP_DATA_DIR=\"/afs/inf.ed.ac.uk/group/teaching/mlp/data\"\n",
"```\n",
"\n",
"and then saving and closing the editor. You will need reload the `mlp` environment using `source activate mlp` and restart the Jupyter notebook server in the reloaded environment for the new environment variable definition to be available.\n",
"\n",
"For those working on DICE who have sufficient quota remaining or those using there own machine, an alternative option is to copy the data files in to your local `mlp/data` directory (or wherever your `MLP_DATA_DIR` environment variable currently points to if different). \n",
"\n",
"\n",
"Assuming your local `mlpractical` repository is in your home directory you should be able to copy the required files on DICE by running\n",
"\n", "\n",
"```\n", "```\n",
"cp /afs/inf.ed.ac.uk/group/teaching/mlp/data/msd-*-train.npz ~/mlpractical/data\n", "cp /afs/inf.ed.ac.uk/group/teaching/mlp/data/msd-*-train.npz ~/mlpractical/data\n",
@ -129,9 +141,11 @@
"cp /afs/inf.ed.ac.uk/group/teaching/mlp/data/msd-*-test-*.npz ~/mlpractical/data\n", "cp /afs/inf.ed.ac.uk/group/teaching/mlp/data/msd-*-test-*.npz ~/mlpractical/data\n",
"```\n", "```\n",
"\n", "\n",
"On a non-DICE machine, you will need to either [set up local access to AFS](http://computing.help.inf.ed.ac.uk/informatics-filesystem), use a remote file transfer client like `scp` or you can alternatively download the files using the iFile web interface [here](https://ifile.inf.ed.ac.uk/?path=%2Fafs%2Finf.ed.ac.uk%2Fgroup%2Fteaching%2Fmlp%2Fdata&goChange=Go) (requires DICE credentials).\n",
"\n",
"As some of the files are quite large you may wish to copy only those you are using (e.g. only the files for one of the two tasks) to your local filespace to avoid filling up your quota. The `msd-25-genre-test-inputs.npz` files will only be needed by those intending to enter the associated optional Kaggle competition.\n", "As some of the files are quite large you may wish to copy only those you are using (e.g. only the files for one of the two tasks) to your local filespace to avoid filling up your quota. The `msd-25-genre-test-inputs.npz` files will only be needed by those intending to enter the associated optional Kaggle competition.\n",
"\n", "\n",
"In addition to the fixed length input files there are also corresponding files with the variable length input data\n", "In addition to the fixed length input files there are also corresponding files with the variable length input data in the same AFS directory\n",
"\n", "\n",
" msd-10-genre-train-var-length.npz 1.6GB\n", " msd-10-genre-train-var-length.npz 1.6GB\n",
" msd-10-genre-valid-var-length.npz 403MB\n", " msd-10-genre-valid-var-length.npz 403MB\n",
@ -141,14 +155,39 @@
" msd-25-genre-valid-var-length.npz 367MB\n", " msd-25-genre-valid-var-length.npz 367MB\n",
" msd-25-genre-test-inputs-var-length.npz 363MB\n", " msd-25-genre-test-inputs-var-length.npz 363MB\n",
" \n", " \n",
"As you can see some of these files, particularly the training sets, are very large so you will likely need to be careful when copying to your filespace to make sure you have sufficient quota available." "As you can see some of these files, particularly the training sets, are very large so you will need to be careful if copying to your filespace to make sure you have sufficient quota available."
] ]
}, },
{ {
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Example two-layer classifier model on MSD 10-genre task" "## Example two-layer classifier models\n",
"\n",
"Below example code is given for creating instances of the 10-genre and 25-genre fixed-length input data provider objects and using them to train simple two-layer feedforward network models with rectified linear activations in TensorFlow."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import os\n",
"import tensorflow as tf\n",
"import numpy as np\n",
"from mlp.data_providers import MSD10GenreDataProvider, MSD25GenreDataProvider\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### MSD 10 genre task"
] ]
}, },
{ {
@ -254,7 +293,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"## Example two-layer classifier model on MSD 25-genre task" "### MSD 25-genre task"
] ]
}, },
{ {