I’ve taken an interest in neural networks and am currently trying to build one from ground up. Synthesizing some of the things I’ve learned so far because that’s how my brain absorbs.

Before the neural network can do anything, we have to build neurons and we have to feed them input. Input is a vector that maps input “words” to a count. That means there’s an explicit vocabulary, and for the input we count how many times a word comes up. That also means words that are not in the input will be ignored. I understand this technique is called bag of words. For the vocabulary [hello world pancakes dinner] and the input hello world we’d get an input vector of [1 1 0 0] whereas hello pancakes would give us [1 0 1 0].

A neuron computes a weighted sum of word votes. It gets the input, it has weights for each input word and with that calculates a predicted value. Each neuron ideally answers a question about one thing that can be represented on a numerical scale that has a natural ordering, e.g. ratings terrible->bad->ok->good->great work well. A neuron is basically a function that takes an input vector and a set of weights, one for each input vector slot. Given an input vector it produces an output vector that represents something.

In my experiment I’ve build a simple prototype that takes input text and then several neurons determine what output category these words best fit into.

The vocabulary lists the known words:

vocab: [turn on off the light play music]

And during training input vectors are provided with the expected output vector. In this example I used a vector with 3 elements, [turn_on, turn_off, play_music].

training data: "turn on the light": [1 0 0]
training data: "turn the light on": [1 0 0]
training data: "turn off the light": [0 1 0]
training data: "turn the light off": [0 1 0]
training data: "play music": [0 0 1]
training data: "play the music": [0 0 1]

The training process iterates over all the input, and for each input runs the prediction function and then compares it to the actual value. It then calculates with a small epsilon the gradient towards the expected value, updates the weights based on a small step and the gradient and then repeats. I used 500 iterations for this, and these are the produced weights:

------------- weights -------------
output      bias      turn      on        off       the       light     play      music
turn_on     0.105203  0.15785   0.57892   -0.42108  0.00017   0.15785   -0.05265  -0.05265
turn_off    0.105203  0.15785   -0.42108  0.57892   0.00017   0.15785   -0.05265  -0.05265
play_music  0.263121  -0.10529  -0.05265  -0.05265  0.00010   -0.10529  0.36841   0.36841

There’s probably a lot of statistical insight here but I didn’t pay enough attention in statistics to put that into words. What stands out is that words that are unique to a neuron seem to have more distinct values. Common words like the are very low because they don’t contribute to the decision which makes me think I should exclude them from the vocabulary.

With the weights in place I can now feed input vectors into the prediction function and its output tells me how much the input matches the neuron’s training data:

"turn on the light"   turn_on=1.00   turn_off=-0.00  play_music=-0.00
"play some music"     turn_on=-0.00  turn_off=-0.00  play_music=1.00
"turn off"            turn_on=-0.16  turn_off=0.84   play_music=0.11
"release the hounds"  turn_on=0.11   turn_off=0.11   play_music=0.26

Pretty cool. Reminds me of a Bayes classifier. But this is not a full neural network yet because I haven’t grokked the layers yet, but I’m quite happy with this so far. More on this probably soon.