Perceptron

Jérémie Decock (www.jdhp.org)

Attention:

Veuillez noter que ce document est en cours de rédaction. Dans son état actuel, il n'est pas destiné à être lu par d'autres personnes que ses auteurs.

Définition des macros LaTeX...

$$ \newcommand{\activthres}{\theta} \newcommand{\activfunc}{f} \newcommand{\pot}{p} \newcommand{\learnrate}{\eta} \newcommand{\it}{t} \newcommand{\sigin}{s_i} \newcommand{\sigout}{s_j} \newcommand{\sigoutdes}{d_j} \newcommand{\wij}{w_{ij}} $$

Auteur: F. Rosenblatt

Reference: F. Rosenblatt 1958 The Perceptron: a Probabilistic Model for Information Storage and Organization in the Brain Psychological Review, 65, 386-408

Le modéle est constitué des éléments suivants:

  • des unités sensitives (S-units): réagissent à un stimuli extérieur (lumière, son, touché, ...)
    • retournent 0 ou 1:
      • 1 si le signal d'entrée dépasse un seuil $\activthres$
      • 0 sinon
  • des unités d'associations (A-units)
    • retournent 0 ou 1:
      • 1 si la somme des signaux d'entrée dépasse un seuil $\activthres$
      • 0 sinon
  • des unités de réponse (R-units): sortie du réseau
    • retournent 1, -1 ou une valeur indéterminée:
      • 1 si la somme des signaux d'entrée est positive
      • -1 si elle est négative
      • une valeur indéterminée si elle est égale à 0
  • une matrice d'intéractions

Evaluation de la fonction: $$ \pot = \sum \sigin \wij $$

$$ \sigout = \activfunc(\pot - \activthres) $$

Fonction de transfert: signe et heaviside

Out[1]:
[-5, 5, -2, 2]
Out[2]:
[-5, 5, -2, 2]

Règle du Perceptron (mise à jour des poid $\wij$):

$$ \wij(\it + 1) = \wij(\it) + \learnrate (\sigoutdes - \sigout) \sigin $$
  • $\learnrate$: pas d'apprentissage, $\learnrate \in [0, 1]$. Géneralement, on lui donne une valeur proche de 1 au début de l'apprentissage et on diminue sa valeur à chaque itération.

Poids de depart des synapses du réseau Nombre de neurones associatifs (A-units) Nombre d'unités sensitives Motif à apprendre

In [3]:
%matplotlib inline

import numpy as np
import matplotlib
import matplotlib.pyplot as plt

import matplotlib.patches as patches
import matplotlib.lines as mlines
import matplotlib.patches as mpatches

import itertools
In [5]:
# Poids de depart des synapses du réseau
initial_weights = np.array([0., 0., 0., 0., 2.])

# Pas d'apprentissage eta=1
learning_rate = 1.
In [6]:
class Log:
    def __init__(self):
        self.input_signal = []
        self.output_signal = []
        self.desired_output_signal = []
        self.error = []
        self.weights = []
        self.iteration = []
        self.current_iteration = 0
        
    def log(self, input_signal, output_signal, desired_output_signal, error, weights):
        self.input_signal.append(input_signal)
        self.output_signal.append(output_signal)
        self.desired_output_signal.append(desired_output_signal)
        self.error.append(error)
        self.weights.append(weights)
        self.iteration.append(self.current_iteration)
        
log = Log()
In [7]:
def sign_function(x):
    y = 1. if x >= 0. else -1.
    return y

def heaviside_function(x):
    y = 1. if x >= 0. else 0.
    return y
In [8]:
def activation_function(p):
    return heaviside_function(p)

def evaluate_network(weights, input_signal):      # TODO: find a better name
    p = np.sum(input_signal * weights)
    output_signal = activation_function(p)
    return output_signal

def update_weights(weights, input_signal, desired_output_signal):
    output_signal = evaluate_network(weights, input_signal)
    error = desired_output_signal - output_signal
    weights = weights + learning_rate * error * input_signal
    log.log(input_signal, output_signal, desired_output_signal, error, weights)
    return weights

def learn_examples(example_list, label_list, weights, num_iterations):
    for it in range(num_iterations):
        log.current_iteration = it
        for input_signal, desired_output_signal in zip(example_list, label_list):
            weights = update_weights(weights, np.array(input_signal + (-1,)), desired_output_signal)
    return weights

Rappel: $\sigin \in \{0, 1\}$

In [9]:
example_list = tuple(reversed(tuple(itertools.product((0., 1.), repeat=4))))

# Motif à apprendre: (1 0 0 1)
label_list = [1. if x == (1., 0., 0., 1.) else 0. for x in example_list]

print(example_list)
print(label_list)
((1.0, 1.0, 1.0, 1.0), (1.0, 1.0, 1.0, 0.0), (1.0, 1.0, 0.0, 1.0), (1.0, 1.0, 0.0, 0.0), (1.0, 0.0, 1.0, 1.0), (1.0, 0.0, 1.0, 0.0), (1.0, 0.0, 0.0, 1.0), (1.0, 0.0, 0.0, 0.0), (0.0, 1.0, 1.0, 1.0), (0.0, 1.0, 1.0, 0.0), (0.0, 1.0, 0.0, 1.0), (0.0, 1.0, 0.0, 0.0), (0.0, 0.0, 1.0, 1.0), (0.0, 0.0, 1.0, 0.0), (0.0, 0.0, 0.0, 1.0), (0.0, 0.0, 0.0, 0.0))
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
In [10]:
weights = learn_examples(example_list, label_list, initial_weights, 5)
weights
Out[10]:
array([ 1., -1., -1.,  2.,  3.])
In [11]:
for input_signal, output_signal, desired_output_signal, error, weights, iteration in zip(log.input_signal, log.output_signal, log.desired_output_signal, log.error, log.weights, log.iteration):
    print(iteration, input_signal, output_signal, desired_output_signal, error, weights)
0 [ 1.  1.  1.  1. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  0.  2.]
0 [ 1.  1.  1.  0. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  0.  2.]
0 [ 1.  1.  0.  1. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  0.  2.]
0 [ 1.  1.  0.  0. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  0.  2.]
0 [ 1.  0.  1.  1. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  0.  2.]
0 [ 1.  0.  1.  0. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  0.  2.]
0 [ 1.  0.  0.  1. -1.] 0.0 1.0 1.0 [ 1.  0.  0.  1.  1.]
0 [ 1.  0.  0.  0. -1.] 1.0 0.0 -1.0 [ 0.  0.  0.  1.  2.]
0 [ 0.  1.  1.  1. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
0 [ 0.  1.  1.  0. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
0 [ 0.  1.  0.  1. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
0 [ 0.  1.  0.  0. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
0 [ 0.  0.  1.  1. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
0 [ 0.  0.  1.  0. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
0 [ 0.  0.  0.  1. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
0 [ 0.  0.  0.  0. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
1 [ 1.  1.  1.  1. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
1 [ 1.  1.  1.  0. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
1 [ 1.  1.  0.  1. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
1 [ 1.  1.  0.  0. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
1 [ 1.  0.  1.  1. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
1 [ 1.  0.  1.  0. -1.] 0.0 0.0 0.0 [ 0.  0.  0.  1.  2.]
1 [ 1.  0.  0.  1. -1.] 0.0 1.0 1.0 [ 1.  0.  0.  2.  1.]
1 [ 1.  0.  0.  0. -1.] 1.0 0.0 -1.0 [ 0.  0.  0.  2.  2.]
1 [ 0.  1.  1.  1. -1.] 1.0 0.0 -1.0 [ 0. -1. -1.  1.  3.]
1 [ 0.  1.  1.  0. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
1 [ 0.  1.  0.  1. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
1 [ 0.  1.  0.  0. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
1 [ 0.  0.  1.  1. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
1 [ 0.  0.  1.  0. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
1 [ 0.  0.  0.  1. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
1 [ 0.  0.  0.  0. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
2 [ 1.  1.  1.  1. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
2 [ 1.  1.  1.  0. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
2 [ 1.  1.  0.  1. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
2 [ 1.  1.  0.  0. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
2 [ 1.  0.  1.  1. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
2 [ 1.  0.  1.  0. -1.] 0.0 0.0 0.0 [ 0. -1. -1.  1.  3.]
2 [ 1.  0.  0.  1. -1.] 0.0 1.0 1.0 [ 1. -1. -1.  2.  2.]
2 [ 1.  0.  0.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  2.]
2 [ 0.  1.  1.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  2.]
2 [ 0.  1.  1.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  2.]
2 [ 0.  1.  0.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  2.]
2 [ 0.  1.  0.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  2.]
2 [ 0.  0.  1.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  2.]
2 [ 0.  0.  1.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  2.]
2 [ 0.  0.  0.  1. -1.] 1.0 0.0 -1.0 [ 1. -1. -1.  1.  3.]
2 [ 0.  0.  0.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  1.  3.]
3 [ 1.  1.  1.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  1.  3.]
3 [ 1.  1.  1.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  1.  3.]
3 [ 1.  1.  0.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  1.  3.]
3 [ 1.  1.  0.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  1.  3.]
3 [ 1.  0.  1.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  1.  3.]
3 [ 1.  0.  1.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  1.  3.]
3 [ 1.  0.  0.  1. -1.] 0.0 1.0 1.0 [ 2. -1. -1.  2.  2.]
3 [ 1.  0.  0.  0. -1.] 1.0 0.0 -1.0 [ 1. -1. -1.  2.  3.]
3 [ 0.  1.  1.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
3 [ 0.  1.  1.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
3 [ 0.  1.  0.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
3 [ 0.  1.  0.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
3 [ 0.  0.  1.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
3 [ 0.  0.  1.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
3 [ 0.  0.  0.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
3 [ 0.  0.  0.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 1.  1.  1.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 1.  1.  1.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 1.  1.  0.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 1.  1.  0.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 1.  0.  1.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 1.  0.  1.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 1.  0.  0.  1. -1.] 1.0 1.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 1.  0.  0.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 0.  1.  1.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 0.  1.  1.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 0.  1.  0.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 0.  1.  0.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 0.  0.  1.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 0.  0.  1.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 0.  0.  0.  1. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
4 [ 0.  0.  0.  0. -1.] 0.0 0.0 0.0 [ 1. -1. -1.  2.  3.]
In [12]:
plt.plot(log.error)
Out[12]:
[<matplotlib.lines.Line2D at 0x11b98b8d0>]
In [13]:
import pandas as pd

df = pd.DataFrame(np.array([log.iteration, log.error]).T, columns=["Iteration", "Error"])
abs_err_per_it = abs(df).groupby(["Iteration"]).sum()
abs_err_per_it.plot(title="Sum of absolute errors per iteration")
Out[13]:
<matplotlib.axes._subplots.AxesSubplot at 0x11e1916a0>