Neural Spatial Curves - AI Tinkerers Ottawa

# Neural Spatial Curves

Press P for speaker notes

<a href="./README.md">Text Version</text>

<a href="https://ranger.mauve.moe">By Mauve Signweaver</a>

---

## TODO:

- Space Filling Curves
- Auto-Encoders
- Neural Spatial Curves
- Improvements

---

### Space Filling Curves

- Maps 1 Dimension to N Dimensions
- Variable fidelity in mapping
- Fractals

---

### Hilbert Curves

`hilbertCurve(n) -> (X,Y)`

![A curve that evenly squiggles accross the 2d plane in a sort of continuous snake pattern](hilbert-curve.jpg)

---

### MATH

![Equation for hilbert curves ripped from stackechange](hilbert-equation.png)

---

### Can a Neural Network do it instead?

- Smallest model possible
- Incremental improvements
- Interpret inner representation
- Build intuition

---

## Auto Encoders

- Train to output same as input
- `Encoder` -> `Latent Space` -> `Decoder`
- Hourglass Shape
- Compress data to latent space
- Encoder compresses points
- Decoder is the spatial curve

---

## Structure

![X and Y go into an encoder which goes into a latent space which goes into a decoder which goes back out into X and Y](autoencoder.svg)

---

## Initial Model

```python
X = np.random.rand(1000, 2)

input_layer = Input(shape=(2,))
hidden_layer_1 = Dense(64, activation='relu')(input_layer)
# Bottleneck layer with one neuron
hidden_layer_2 = Dense(1, activation='linear')(hidden_layer_1)
hidden_layer_3 = Dense(64, activation='relu')(hidden_layer_2)
output_layer = Dense(2)(hidden_layer_3) # Output layer

autoencoder = Model(inputs=input_layer, outputs=output_layer)

# Compile the model
autoencoder.compile(optimizer='adam', loss='mean_squared_error')

# Train the model
autoencoder.fit(X, X, epochs=500, batch_size=32)
```

---

## Mapping Internal Space

![Lots of predictions where it is apparent the model only knows points along an S curve](initial-predictions.svg)

---

## All ReLU Activations

Fix: Linear activation as last layer

![All the points go to zero](all-relu.svg)

---

## More Hidden Layers

![Internal curve now looks more like a four pointed star](more-layers-1.svg)

= **More complexity**

---

## Batch Size

~~32~~ **256**

![Line with an s curve with an extra squiggle on the bottom left and right](larger-batch.svg)

---

## Batch Normalization

![The curve is looking more complex and seems to be closer to what an order 2 hilbert curve](batch-normalized.svg)

Normalized Inputs = Stabilized Training

---

## More neurons

~~2 -> 64 -> 32 -> 1 -> 32 -> 64 -> 2~~

`2 -> 128 -> 64 -> 1 -> 64 -> 128 -> 2`

![Similar shape as batch normalized but more squiggly](more-neurons-1.svg)

---

## Dropout Layers

![Drastically simplified horseshoe shape](./dropout-layers.svg)

Reducing complexity helps improve generalization.

We kinda want the complexity here 😅

---

## More Layers

~~128 -> 64~~

`256 -> 128 -> 64`

![Curve is more uneven and wiggly](more-layers-2.svg)

---

## More Neurons!!

~~256 -> 128 -> 64~~

`512 -> 256 -> 128`

![Seems like a hilbert curve with an extra order of precision](./more-neurons-2.svg)

---

## Mean Absolute Error Loss Function

![Curve has fewer straight lines and is more organic looking](./mean-absolute-error.svg)

- Better for nonlinear relationships
- Robust to large variation in data

---

## Dynamic Learning Rate

```python
ReduceLROnPlateau(
monitor='loss', # Monitor loss
# Epochs without improvement before learning rate is reduced
patience=3,
# Factor by which the learning rate will be reduced
factor=0.5,
# Minimum learning rate to prevent it from becoming too low
min_lr=1e-6
)
```

---

## Reduce LR On Plateau Result

![Similar level of complexity, maybe a bit less wiggly](./dynamic-learning-rate.svg)

---

## Way more epocs

![Slightly more complex shape than before](./more-epochs.svg)

---

## Conclusions

- Least Complex model "good enough"
- Size / Depth makes a huge difference
- Normalization can help with training
- Linear activation at the end

---

- Come chat [on matrix](https://matrix.to/#/#userless-agents:mauve.moe)
- `contact@mauve.moe`