LAWSON DONG / FIELD NOTES
01 / RESEARCHSELF-DIRECTED EXPLORATIONS

Working questions,
ongoing experiments.

Exploratory work on representations and learning dynamics, alongside notes on coding and AI engineering.

01

COGNITIVE SCIENCE × AI

Representation
Analysis

Exploring the structure of neural representations and their relationship to human cognition.

A SMALL PRIMER / 01

What is a representation?

In a deep neural network, a representation is an internal vector learned for an input at a hidden layer. The network maps raw data, such as pixels or words, into features that can carry more useful structure for a task. A layer’s activation, an embedding, and a hidden state are all examples.

h(0)=x,h(ℓ)=σ ⁣(W(ℓ)h(ℓ−1)+b(ℓ))h^{(0)}=x,\qquad h^{(\ell)}=\sigma\!\left(W^{(\ell)}h^{(\ell-1)}+b^{(\ell)}\right)
A SMALL PRIMER / 02

And its space?

For a chosen layer, mapping many inputs through the network gives a collection of representation vectors. Their distances, directions, and similarities form the geometry we call a representation space. Each input becomes a point in that space.

Hℓ={h(ℓ)(x;θ)∣x∈X}⊆Rdℓ\mathcal H_\ell=\{h^{(\ell)}(x;\theta)\mid x\in\mathcal X\}\subseteq\mathbb R^{d_\ell}fθ:X→Hℓf_\theta:\mathcal X\to\mathcal H_\ell
01.A / MODEL–BRAIN CORRESPONDENCE

Representational Similarity Analysis across ResNet-18, ResNet-50, and ResNet-152, compared with early visual cortex (EVC) and inferotemporal cortex (IT). Do stages of artificial visual processing correspond to stages of biological vision?

View on GitHub
01.B / SEMANTIC COMPOSITION

Using hybrid concepts in CLIP, such as a dog body with a cat head, to ask how component concepts combine and transform inside a learned representation space.

View on GitHub
01.C / REPRESENTATION-VECTOR GEOMETRIC DYNAMICS

Tracking the same cat and dog images through ResNet, ConvNeXt, ViT, and Swin to study how class separation, adjacent-layer geometry, and local label mixing change as representations propagate through a frozen network.

View on GitHub The measuring tools
02

PHYSICS × DEEP LEARNING

Learning
Mechanics

FIXED LOSS LANDSCAPEGRADIENT DESCENT / SGD
min Lstart θ₀
0.40
converges · 0/26 steps

Illustrative 2D quadratic loss · same contours and start point for every rate.

What if neural-network training can be studied as a discrete dynamical system?

02.A / TRAJECTORIES

Exploring optimizer trajectories under SGD, momentum SGD, and Adam, with discrete velocity and acceleration in parameter space.

02.B / TRAINING DYNAMICS

Investigating learning-rate schedules, overfitting, a proposed “learning Reynolds number,” and experiments with GPT-2 Small, Medium, and Large. These are exploratory questions, not established results.

View research