anton/mlpractical
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Applying softmax stability fix to cost gradient.

Browse Source
This commit is contained in:
Matt Graham 2016-10-20 13:23:19 +01:00
parent b4b00e0c65
commit a021ce585b
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
mlp/errors.py
View File

@ -170,7 +170,9 @@ class CrossEntropySoftmaxError(object):
Returns: Returns:
Gradient of error function with respect to outputs. Gradient of error function with respect to outputs.
""" """
probs = np.exp(outputs) # subtract max inside exponential to improve numerical stability -
# when we divide through by sum this term cancels
probs = np.exp(outputs - outputs.max(-1)[:, None])
probs /= probs.sum(-1)[:, None] probs /= probs.sum(-1)[:, None]
return (probs - targets) / outputs.shape[0] return (probs - targets) / outputs.shape[0]