A translation book between software and machine learning

Types

int, float, bool,

• Fixed size vector.
  • Code: [float; 32]
  • Math: $\alpha \in \mathbb{R_d}$, where $d=32$

Type conversion.

• softmax converts real numbers to probabilities
• logprobs are $log(P)$. why? because when we are doing sequence modelling, we multiply probabilities - P(x1) * P(x2). If the probabilities are small, the precision is vanishing. Whereas for logprobs, multiplication is defined as addition.

Anthropomorphisms

qualitative terms (this problem is hard) vs. quantitative terms (this objective is costly)

• tasks = objectives
• failure = error
• easy = cheap
• hard = costly
• implicit unit of measurement is energy, as in Kj
• intelligence = prediction
• ??? = differentiable (some things are not differentiable)