\n",
+ "\n",
+ "The dashboard above is showing the `Files` tab, a list of files in the directory the notebook server was launched from. We can navigate in to a sub-directory by clicking on a directory name and back up to the parent directory by clicking the `..` link. An important point to note is that the top-most level that you will be able to navigate to is the directory you run the server from. This is a security feature and generally you should try to limit the access the server has by launching it in the highest level directory which gives you access to all the files you need to work with.\n",
+ "\n",
+ "As well as allowing you to launch existing notebooks, the `Files` tab of the dashboard also allows new notebooks to be created using the `New` drop-down on the right. It can also perform basic file-management tasks such as renaming and deleting files (select a file by checking the box alongside it to bring up a context menu toolbar).\n",
+ "\n",
+ "In addition to opening notebook files, we can also edit text files such as `.py` source files, directly in the browser by opening them from the dashboard. The in-built text-editor is less-featured than a full IDE but is useful for quick edits of source files and previewing data files.\n",
+ "\n",
+ "The `Running` tab of the dashboard gives a list of the currently running notebook instances. This can be useful to keep track of which notebooks are still running and to shutdown (or reopen) old notebook processes when the corresponding tab has been closed.\n",
+ "\n",
+ "### The notebook interface\n",
+ "\n",
+ "The top of your notebook window should appear something like this:\n",
+ "\n",
+ "
\n",
+ "\n",
+ "The name of the current notebook is displayed at the top of the page and can be edited by clicking on the text of the name. Displayed alongside this is an indication of the last manual *checkpoint* of the notebook file. On-going changes are auto-saved at regular intervals; the check-point mechanism is mainly meant as a way to recover an earlier version of a notebook after making unwanted changes. Note the default system only currently supports storing a single previous checkpoint despite the `Revert to checkpoint` dropdown under the `File` menu perhaps suggesting otherwise.\n",
+ "\n",
+ "As well as having options to save and revert to checkpoints, the `File` menu also allows new notebooks to be created in same directory as the current notebook, a copy of the current notebook to be made and the ability to export the current notebook to various formats.\n",
+ "\n",
+ "The `Edit` menu contains standard clipboard functions as well as options for reorganising notebook *cells*. Cells are the basic units of notebooks, and can contain formatted text like the one you are reading at the moment or runnable code as we will see below. The `Edit` and `Insert` drop down menus offer various options for moving cells around the notebook, merging and splitting cells and inserting new ones, while the `Cell` menu allow running of code cells and changing cell types.\n",
+ "\n",
+ "The `Kernel` menu offers some useful commands for managing the Python process (kernel) running in the notebook. In particular it provides options for interrupting a busy kernel (useful for example if you realise you have set a slow code cell running with incorrect parameters) and to restart the current kernel. This will cause all variables currently defined in the workspace to be lost but may be necessary to get the kernel back to a consistent state after polluting the namespace with lots of global variables or when trying to run code from an updated module and `reload` is failing to work. \n",
+ "\n",
+ "To the far right of the menu toolbar is a kernel status indicator. When a dark filled circle is shown this means the kernel is currently busy and any further code cell run commands will be queued to happen after the currently running cell has completed. An open status circle indicates the kernel is currently idle.\n",
+ "\n",
+ "The final row of the top notebook interface is the notebook toolbar which contains shortcut buttons to some common commands such as clipboard actions and cell / kernel management. If you are interested in learning more about the notebook user interface you may wish to run through the `User Interface Tour` under the `Help` menu drop down.\n",
+ "\n",
+ "### Markdown cells: easy text formatting\n",
+ "\n",
+ "This entire introduction has been written in what is termed a *Markdown* cell of a notebook. [Markdown](https://en.wikipedia.org/wiki/Markdown) is a lightweight markup language intended to be readable in plain-text. As you may wish to use Markdown cells to keep your own formatted notes in notebooks, a small sampling of the formatting syntax available is below (escaped mark-up on top and corresponding rendered output below that); there are many much more extensive syntax guides - for example [this cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet).\n",
+ "\n",
+ "---\n",
+ "\n",
+ "```\n",
+ "## Level 2 heading\n",
+ "### Level 3 heading\n",
+ "\n",
+ "*Italicised* and **bold** text.\n",
+ "\n",
+ " * bulleted\n",
+ " * lists\n",
+ " \n",
+ "and\n",
+ "\n",
+ " 1. enumerated\n",
+ " 2. lists\n",
+ "\n",
+ "Inline maths $y = mx + c$ using [MathJax](https://www.mathjax.org/) as well as display style\n",
+ "\n",
+ "$$ ax^2 + bx + c = 0 \\qquad \\Rightarrow \\qquad x = \\frac{-b \\pm \\sqrt{b^2 - 4ac}}{2a} $$\n",
+ "```\n",
+ "---\n",
+ "\n",
+ "## Level 2 heading\n",
+ "### Level 3 heading\n",
+ "\n",
+ "*Italicised* and **bold** text.\n",
+ "\n",
+ " * bulleted\n",
+ " * lists\n",
+ " \n",
+ "and\n",
+ "\n",
+ " 1. enumerated\n",
+ " 2. lists\n",
+ "\n",
+ "Inline maths $y = mx + c$ using [MathJax]() as well as display maths\n",
+ "\n",
+ "$$ ax^2 + bx + c = 0 \\qquad \\Rightarrow \\qquad x = \\frac{-b \\pm \\sqrt{b^2 - 4ac}}{2a} $$\n",
+ "\n",
+ "---\n",
+ "\n",
+ "We can also directly use HTML tags in Markdown cells to embed rich content such as images and videos.\n",
+ "\n",
+ "---\n",
+ "```\n",
+ "We will therefore use branches here in a slightly non-standard way. The code for each week's lab and for each of the assignments will be maintained in a separate branch. This will allow us to stage the release of the notebooks and code for each lab and assignment while allowing you to commit the changes you make to the code each week without having to merge those changes when new code is released. Similarly this structure will allow us to release updated notebooks from previous labs with proposed solutions without overwriting your own work.
+ +To list the branches present in the local repository, run: + +``` +git branch +``` + +This will display a list of branches with a `*` next to the current branch. To switch to a different existing branch in the local repository run + +``` +git checkout branch-name +``` + +This will change the code in the working directory to the current state of the checked out branch. Any files added to the staging area and committed will then create a new commit on this branch. + +You should make sure you are on the first lab branch now by running: + +``` +git checkout mlp2017-8/lab1 +``` + +## Installing the `mlp` Python package + +In your local repository we noted above the presence of a `mlp` subdirectory. This contains the custom Python package implementing the NumPy based neural network framework we will be using in this course. + +In order to make the modules in this package available in your environment we need install it. A [setuptools](https://setuptools.readthedocs.io/en/latest/) `setup.py` script is provided in the root of the `mlpractical` directory for this purpose. + +The standard way to install a Python package using a `setup.py` script is to run `python setup.py install`. This creates a copy of the package in the `site-packages` directory of the currently active Python environment. + +As we will be updating the code in the `mlp` package during the course of the labs this would require you to re-run `python setup.py install` every time a change is made to the package. Instead therefore you should install the package in development mode by running: + +``` +python setup.py develop +``` + +Instead of copying the package, this will instead create a symbolic link to the copy in the local repository. This means any changes made will be immediately available without the need to reinstall the package. + +--- + +**Aside on importing/reloading Python modules:** + +Note that after the first time a Python module is loaded into an interpreter instance, using for example: + +``` +import mlp +``` + +Running the `import` statement any further times will have no effect even if the underlying module code has been changed. To reload an already imported module we instead need to use the [`reload`](https://docs.python.org/2.7/library/functions.html#reload) function, e.g. + +``` +reload(mlp) +``` + +**Note: To be clear as this has caused some confusion in previous labs the above `import ...` / `reload(...)` statements should NOT be run directly in a bash terminal. They are examples Python statements - you could run them in a terminal by first loading a Python interpreter using:** + +``` +python +``` + +**however you do not need to do so now. This is meant as information to help you later when importing modules as there was some confusion last year about the difference between `import` and `reload`.** + +--- + +## Adding a data directory variable to the environment + +We observed previously the presence of a `data` subdirectory in the local repository. This directory holds the data files that will be used in the course. To enable the data loaders in the `mlp` package to locate these data files we need to set a `MLP_DATA_DIR` environment variable pointing to this directory. + +Assuming you used the recommended Miniconda install location and cloned the `mlpractical` repository to your home directory, this variable can be automatically defined when activating the environment by running the following commands (on non-Windows systems): + +``` +cd ~/miniconda3/envs/mlp +mkdir -p ./etc/conda/activate.d +mkdir -p ./etc/conda/deactivate.d +echo -e '#!/bin/sh\n' >> ./etc/conda/activate.d/env_vars.sh +echo "export MLP_DATA_DIR=$HOME/mlpractical/data" >> ./etc/conda/activate.d/env_vars.sh +echo -e '#!/bin/sh\n' >> ./etc/conda/deactivate.d/env_vars.sh +echo 'unset MLP_DATA_DIR' >> ./etc/conda/deactivate.d/env_vars.sh +export MLP_DATA_DIR=$HOME/mlpractical/data +``` + +And on Windows systems (replacing the `[]` placeholders with the relevant paths): + +``` +cd [path-to-conda-root]\envs\mlp +mkdir .\etc\conda\activate.d +mkdir .\etc\conda\deactivate.d +@echo "set MLP_DATA_DIR=[path-to-local-repository]\data" >> .\etc\conda\activate.d\env_vars.bat +@echo "set MLP_DATA_DIR=" >> .\etc\conda\deactivate.d\env_vars.bat +set MLP_DATA_DIR=[path-to-local-repository]\data +``` + +## Loading the first lab notebook + +Your environment is now all set up so you can move on to the introductory exercises in the first lab notebook. + +One of the dependencies you installed in your environment earlier was Jupyter. Jupyter notebooks allow combining formatted text with runnable code cells and visualisation of the code output in an intuitive web application interface. Although originally specific to Python (under the previous moniker IPython notebooks) the notebook interface has now been abstracted making them available to a wide range of languages. + +There will be a Jupyter notebook available for each lab and assignment in this course, with a combination of explanatory sections for you to read through which will complement the material covered in lectures, as well as series of practical coding exercises to be written and run in the notebook interface. The first lab notebook will cover some of the basics of the notebook interface. + +To open a notebook, you first need to launch a Jupyter notebook server instance. From within the `mlpractical` directory containing your local copy of the repository (and with the `mlp` environment activated) run: + +``` +jupyter notebook +``` + +This will start a notebook server instance in the current terminal (with a series of status messages being streamed to the terminal output) and launch a browser window which will load the notebook application interface. + +By default the notebook interface will show a list of the files in the directory the notebook server was launched from when first loaded. If you click on the `notebooks` directory in this file list, a list of files in this directory should then be displayed. Click the `01_Introduction.ipynb` entry to load the first notebook. + +# Minimal set-up instructions for DICE + +Below are instructions for setting up the environment without additional explanation. These are intentionally terse and if you do not understand what a particular command is doing you might be better following the more detailed instructions above which explain each step. + +--- + +Start a new bash terminal. Download the latest 64-bit Python 2.7 Miniconda install script: + +``` +wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh +``` + +Run the install script: + +``` +bash Miniconda3-latest-Linux-x86_64.sh +``` + +Review the software license agreement and choose whether to accept. Assuming you accept, you be asked to choose an install location for Miniconda. The default is to install in the root of your home directory `~/miniconda3`. We will assume below you have used this default. **If you use a different path you will need to adjust the paths in the commands below to suit.** + +You will then be asked whether to prepend the Miniconda binaries directory to the `PATH` system environment variable definition in `.bashrc`. You should respond `no` here as we will set up the addition to `PATH` manually in the next step. + +Append the Miniconda binaries directory to `PATH` in manually in `~/.benv`: +``` +echo "export PATH=\""\$PATH":$HOME/miniconda3/bin\"" >> ~/.benv +``` + +`source` the updated `~/.benv`: + +``` +source ~/.benv +``` + +Create a new `mlp` Conda environment: + +``` +conda create -n mlp python=3 +``` + +Activate our created environment: + +``` +source activate mlp +``` + +Install the dependencies for the course into the new environment: + +``` +conda install numpy scipy matplotlib jupyter +``` + +Clear the package tarballs Conda just downloaded: + +``` +conda clean -t +``` + +Clone the course repository to your home directory: + +``` +git clone https://github.com/CSTR-Edinburgh/mlpractical.git ~/mlpractical +``` + +Make sure we are on the first lab branch + +``` +cd ~/mlpractical +git checkout mlp2017-8/lab1 +``` + +Install the `mlp` package in the environment in develop mode + +``` +python ~/mlpractical/setup.py develop +``` + +Add an `MLP_DATA_DIR` variable to the environment + +``` +cd ~/miniconda3/envs/mlp +mkdir -p ./etc/conda/activate.d +mkdir -p ./etc/conda/deactivate.d +echo -e '#!/bin/sh\n' >> ./etc/conda/activate.d/env_vars.sh +echo "export MLP_DATA_DIR=$HOME/mlpractical/data" >> ./etc/conda/activate.d/env_vars.sh +echo -e '#!/bin/sh\n' >> ./etc/conda/deactivate.d/env_vars.sh +echo 'unset MLP_DATA_DIR' >> ./etc/conda/deactivate.d/env_vars.sh +export MLP_DATA_DIR=$HOME/mlpractical/data +``` + +Environment is now set up. Load the notebook server from `mlpractical` directory + +``` +cd ~/mlpractical +jupyter notebook +``` + +and then open the first lab notebook from the `notebooks` directory. + + +--- + +[1] The `echo` command causes the following text to be streamed to an output (standard terminal output by default). Here we use the append redirection operator `>>` to redirect the `echo` output to a file `~/.benv`, with it being appended to the end of the current file. The text actually added is `export PATH="$PATH:[your-home-directory]/miniconda/bin"` with the `\"` being used to escape the quote characters. The `export` command defines system-wide environment variables (more rigorously those inherited by child shells) with `PATH` being the environment variable defining where `bash` searches for executables as a colon-seperated list of directories. Here we add the Miniconda binary directory to the end of the current `PATH` definition. [↩](#a1) diff --git a/notes/getting-started-in-a-lab.md b/notes/getting-started-in-a-lab.md new file mode 100644 index 0000000..7f53851 --- /dev/null +++ b/notes/getting-started-in-a-lab.md @@ -0,0 +1,55 @@ +# Getting started in a lab on DICE computers + +Once your [environment is set up](environment-set-up.md), at the beginning of each lab you should be able follow the steps below to get the lab notebook for that session running. + +Open a terminal window (`Applications > Terminal`). + +We first need to activate our `mlp` Conda environment: + +``` +source activate mlp +``` + +We now need to fetch any new code for the lab from the Github repository and create a new branch for this lab's work. First change in to the `mlpractical` repoistory directory (if you cloned the repository to a different directory than the default you will need to adjust the command below accordingly): + +``` +cd ~/mlpractical +``` + +If you have not yet commited the changes you made to the current branch in the previous lab you should do so now. You can check if you have changes not yet commited by running `git status`. If there are files with changes to be commited (they will appear in red) you should first add them to the staging area using + +``` +git add path/to/file1 path/to/file2 +``` + +then commit them with a descriptive commit message using + +``` +git commit -m "Description of changes e.g. Exercises for first lab notebook." +``` + +We are now ready to fetch any updated code from the remote repository on Github. This can be done by running + +``` +git fetch origin +``` + +This should display a message indicate a new branch has been found and fetched, named `origin/mlp2017-8/lab[n]` where `[n]` is the relevant lab number e.g. `origin/mlp2017-8/lab2` for the second lab. + +We now need to create and checkout a new local branch from the remote branch fetched above. This can be done by running + +``` +git checkout -b lab[n] origin/mlp2017-8/lab[n] +``` + +where again `lab[n]` corresponds to the relevant lab number fetched above e.g. `lab2`. This command creates a new local branch named `lab[n]` from the fetched branch on the remote repository `origin/mlp2017-8/lab[n]`. + +Inside the `notebooks` directory there should new be a new notebook for today's lab. The notebook for the previous lab will now also have proposed solutions filled in. + +To get started with the new notebook from the `~/mlpractical` directory start up a Jupyter notebook server + +``` +jupyter notebook +``` + +then open the new notebook from the dashboard. diff --git a/notes/quota-issue.md b/notes/quota-issue.md new file mode 100644 index 0000000..db09687 --- /dev/null +++ b/notes/quota-issue.md @@ -0,0 +1,29 @@ +# Exceeded quota problems on DICE + +Apologies to those who may have issues with having insufficient quota space on DICE in the labs on Monday (25th September). + +This was caused by the [dynamic AFS quota system](http://computing.help.inf.ed.ac.uk/dynamic-afs-quotas) which only initially allocates users a subset of their maximum quota and then checks hourly to increase this quota as needed. Unfortunately the amount of disk space needed to store the temporary files used in installing the course dependencies exceeded the current dynamic quota for some people. This meant when running the `conda install ...` command it exited with a quota exceeded error. + +Those who experienced that issue should now have sufficient quota space available. From any DICE computer, If you run in a terminal + +``` +source activate mlp +conda remove -y numpy scipy matplotlib jupyter +conda install -y numpy scipy matplotlib jupyter +conda clean -t -y +``` + +this should clean out the old partially installed packages and reinstall them from scratch which should now run to completion without a quota exceeded error. + +Your homespace can be accessed from any Informatics computer running DICE (e.g. any of the computers in the [Forrest Hill labs](http://web.inf.ed.ac.uk/infweb/student-services/ito/students/year2/student-support/facilities/computer-labs) which are open-access outside of booked lab sessions or for those who know how to use SSH you can [log in remotely](http://computing.help.inf.ed.ac.uk/external-login)). You can therefore finish your environment set up prior to the next lab if you want though it is also fine to wait till the beginning of the next lab (it will take around 5 minutes to complete the installation). + +At this point assuming you ran through the rest of the instructions to clone the Git repository to your homespace and install the `mlp` package (i.e. the instructions from [here](https://github.com/CSTR-Edinburgh/mlpractical/blob/mlp2016-7/lab1/environment-set-up.md#getting-the-course-code-and-a-short-introduction-to-git) on-wards), you should have a fully working environment. + +Once your environment is set up in all future labs you will only need to activate it to get started. So at the beginning of each subsequent lab we will ask you to do something like the following + +``` +source activate mlp # Activate the mlp environment +cd ~/mlpractical # Change the current directory to mlpractical repository +git checkout mlp2017-8/lab[...] # Checkout the branch for this week's lab +jupyter notebook # Launch the notebook server +``` diff --git a/notes/running-notebooks-remotely.md b/notes/running-notebooks-remotely.md new file mode 100644 index 0000000..54f433b --- /dev/null +++ b/notes/running-notebooks-remotely.md @@ -0,0 +1,84 @@ +# Running Jupyter notebooks over SSH + +Below is a guide for how to start a Jupyter notebook server remotely on one of the shared-use `student.compute` servers and to connect to it on a local machine by port-forwarding over SSH. It is assumed you already have a SSH client set up on the machine you are connecting from and that you are familiar with how to use SSH. These instructions have been written for use with a SSH client running within a terminal session - although it may be possible to replicate the relevant commands within a GUI based SSH client, you will need to figure out how to do this yourself. They were written and tested on Ubuntu 14.04 and no attempt has been made to test them on other operating systems. + +## Securing your notebook server + +Before running a Jupyter notebook server instance on one of the shared compute servers you **must** make sure you have secured your server by configuring it to use a password and to communicate that password between the browser client and server by secure HTTP. This can be done on by running the `secure-notebook-server.sh` bash script provided in the `scripts` directory of the `mlpractical` repository. You can either do this when logged on to DICE in one of the labs or after connecting to DICE remotely over SSH as described below. + +To run the script, in a DICE terminal enter the `mlpractical` repository directory and run +``` +bash scripts/secure-notebook-server.sh +``` +As this script creates a self-signed certificate to set up the secure HTTP encrypted communication between the browser and server, you will be shown a security warning when you load up the URL the notebooks are being served on. + +If you want to manually secure the notebook server yourself or to create a certificate which will stop the security warnings appearing you can refer to the [relevant official Jupyter documentation page](http://jupyter-notebook.readthedocs.io/en/latest/public_server.html). + +## Connecting to a remote `student.compute` server over SSH + +To start an SSH session, open a terminal window and run + +``` +ssh [dice-username]@student.ssh.inf.ed.ac.uk +``` + +If this is this is the first time you have logged on to the SSH gateway server from this computer you will be asked to confirm you wish to connect and a ECDSA key fingerprint printed. You can check this against the reference values on the [school help pages](http://computing.help.inf.ed.ac.uk/external-login). + +You will then be asked to enter your password. This is the same password you usually use to log on to DICE. + +Assuming you enter the correct password, you will at this point be logged in to the SSH *gateway server*. As the message printed when you log in points out this is intended only for accessing the Informatics network externally and you should **not** attempt to work on this server. You should log in to one of the `student.compute` shared-use servers by running + +``` +ssh student.compute +``` + +You should now be logged on to one of the shared-use compute servers. The name of the server you are logged on to will appear at the bash prompt e.g. + +``` +ashbury:~$ +``` + +You will need to know the name of the remote server you are using later on. + +## Starting a notebook server on the remote computer + +You should now activate your `mlp` Conda environment by running + +``` +source activate mlp +``` + +Now move in to the `mlpractical` local repository directory e.g. by running + +``` +cd ~/mlpractical +``` + +if you chose the default of putting the repository in your home directory. + +We will now launch a notebook server on the remote compute-server. There are two key differences in the command we use to do this compared to how we usually start up a server on a local machine. First as the server will be running remotely you should set the `--no-browser` option as this will prevent the remote server attempting open a browser to connect to the notebook server. + +Secondly we will prefix the command with `nice`. `nice` is a shell command which alters the scheduling priority of the process it is used to start. Its important to use `nice` when running on the shared `student.compute` servers to make sure they remain usable by all of the students who need to run jobs on them. You can set a priority level between 10 (highest priority) and 19 (lowest priority) using the `-n` argument. Running the command below will start up a notebook server at the lowest priority level. + +``` +nice -n 19 jupyter notebook --no-browser +``` + +Once the notebook server starts running you should take note of the port it is being served on as indicated in the `The Jupyter Notebook is running at: https://localhost:[port]/` message. + +## Forwarding a connection to the notebook server over SSH + +Now that the notebook server is running on the remote server you need to connect to it on your local machine. We will do this by forwarding the port the notebook server is being run on over SSH to you local machine. As all external connections from outside the `inf.ed.ac.uk` domain have to go via the SSH gateway server we need to go via this gateway server. + +In a **new terminal window / tab** run the command below with the `[...]` placeholders substituted with the appropriate values to securely forward the specified port on the remote server to your local machine and bind it to a local port. You should choose `[remote-port]` to be the port the notebook server is running on on the remote server, `[local-port]` to be a currently unused port on your local machine and `[remote-server-name]` to be the host name of the remote server the notebook server is being run on. + +``` +ssh -N -o ProxyCommand="ssh -q [dice-username]@student.ssh.inf.ed.ac.uk nc [remote-server-name] 22" \ + -L [local-port]:localhost:[remote-port] [dice-username]@[remote-server-name] +``` + +You will be asked to enter your (DICE) password twice, once to log on to the gateway server and a second time to log on to the remote compute server. + +Assuming you enter your password both times correctly, the remote port will now be getting forwarded to the specified local port on your computer. If you now open up a browser on your computer and go to `https://localhost:[local-port]` you should (potentially after seeing a security warning about the self-signed certicate) now asked to enter the notebook server password you specified earlier. Once you enter this password you should be able to access the notebook dashboard and open and edit notebooks as you usually do in labratories. + +When you are finished working you should both close down the notebook server by entering `Ctrl+C` twice in the terminal window the SSH session you used to start up the notebook server is running and halt the port forwarding command by entering `Ctrl+C` in the terminal it is running in. diff --git a/scripts/secure-notebook-server.sh b/scripts/secure-notebook-server.sh new file mode 100644 index 0000000..95f0547 --- /dev/null +++ b/scripts/secure-notebook-server.sh @@ -0,0 +1,73 @@ +#!/bin/bash +# Configure Jupyter notebook server to use password authentication +# Make sure Conda environment is active as will assume it is later +[ -z "$CONDA_PREFIX" ] && echo "Need to have Conda environment activated." && exit 1 +if [ "$#" -gt 2 ]; then + echo "Usage: bash secure-notebook-server.sh [jupyter-path] [open-ssl-config-path]" + exit 1 +fi +# If specified read Jupyter directory from passed argument +JUPYTER_DIR=${1:-"$HOME/.jupyter"} +# If specified read OpenSSL config file path from passed argument +# This is needed due to bug in how Conda handles config path +export OPENSSL_CONF=${2:-"$CONDA_PREFIX/ssl/openssl.cnf"} +SEPARATOR="=================================================================\n" +# Create default config file if one does not already exist +if [ ! -f "$JUPYTER_DIR/jupyter_notebook_config.py" ]; then + echo "No existing notebook configuration file found, creating new one ..." + printf $SEPARATOR + jupyter notebook --generate-config + printf $SEPARATOR + echo "... notebook configuration file created." +fi +# Get user to enter notebook server password +echo "Getting notebook server password hash. Enter password when prompted ..." +printf $SEPARATOR +HASH=$(python -c "from notebook.auth import passwd; print(passwd());") +printf $SEPARATOR +echo "... got password hash." +# Generate self-signed OpenSSL certificate and key file +echo "Creating certificate file ..." +printf $SEPARATOR +CERT_INFO="/C=UK/ST=Scotland/L=Edinburgh/O=University of Edinburgh/OU=School of Informatics/CN=$USER/emailAddress=$USER@sms.ed.ac.uk" +openssl req \ + -x509 -nodes -days 365 \ + -subj "/C=UK/ST=Scotland/L=Edinburgh/O=University of Edinburgh/OU=School of Informatics/CN=$USER/emailAddress=$USER@sms.ed.ac.uk" \ + -newkey rsa:1024 -keyout "$JUPYTER_DIR/key.key" \ + -out "$JUPYTER_DIR/cert.pem" +printf $SEPARATOR +echo "... certificate created." +# Setting permissions on key file +chmod 600 "$JUPYTER_DIR/key.key" +# Add password hash and certificate + key file paths to config file +echo "Setting up configuration file..." +printf $SEPARATOR +echo " adding password hash" +SRC_PSW="^#\?c\.NotebookApp\.password[ ]*=[ ]*u['"'"'"]\(sha1:[a-fA-F0-9]\+\)\?['"'"'"]" +DST_PSW="c.NotebookApp.password = u'$HASH'" +grep -q "c.NotebookApp.password" $JUPYTER_DIR/jupyter_notebook_config.py +if [ ! $? -eq 0 ]; then + echo DST_PSW >> $JUPYTER_DIR/jupyter_notebook_config.py +else + sed -i "s/$SRC_PSW/$DST_PSW/" $JUPYTER_DIR/jupyter_notebook_config.py +fi +echo " adding certificate file path" +SRC_CRT="^#\?c\.NotebookApp\.certfile[ ]*=[ ]*u['"'"'"]\([^'"'"'"]+\)\?['"'"'"]" +DST_CRT="c.NotebookApp.certfile = u'$JUPYTER_DIR/cert.pem'" +grep -q "c.NotebookApp.certfile" $JUPYTER_DIR/jupyter_notebook_config.py +if [ ! $? -eq 0 ]; then + echo DST_CRT >> $JUPYTER_DIR/jupyter_notebook_config.py +else + sed -i "s|$SRC_CRT|$DST_CRT|" $JUPYTER_DIR/jupyter_notebook_config.py +fi +echo " adding key file path" +SRC_KEY="^#\?c\.NotebookApp\.keyfile[ ]*=[ ]*u['"'"'"]\([^'"'"'"]+\)\?['"'"'"]" +DST_KEY="c.NotebookApp.keyfile = u'$JUPYTER_DIR/key.key'" +grep -q "c.NotebookApp.keyfile" $JUPYTER_DIR/jupyter_notebook_config.py +if [ ! $? -eq 0 ]; then + echo DST_KEY >> $JUPYTER_DIR/jupyter_notebook_config.py +else + sed -i "s|$SRC_KEY|$DST_KEY|" $JUPYTER_DIR/jupyter_notebook_config.py +fi +printf $SEPARATOR +echo "... finished setting up configuration file." diff --git a/setup.py b/setup.py new file mode 100644 index 0000000..46266b1 --- /dev/null +++ b/setup.py @@ -0,0 +1,13 @@ +""" Setup script for mlp package. """ + +from setuptools import setup + +setup( + name = "mlp", + author = "Pawel Swietojanski, Steve Renals, Matt Graham and Antreas Antoniou", + description = ("Neural network framework for University of Edinburgh " + "School of Informatics Machine Learning Practical course."), + url = "https://github.com/CSTR-Edinburgh/mlpractical", + packages=['mlp'] +) +