andrejmuhic.github.io

Most independent subset selection

Linear Algebra

Numerical Linear Algebra

Do you want to find the most independent subset of columns of a matrix? This can be useful for example for determining independent subset of exogenous variables, or catching…

Attention dropout

attention

dropout

In the self attention head we apply dropout on softmax weights. This means that the representation of the first word vector in batch can be \(0\) with dropout probability \(p…

Kalman filter exposed

State space

Kalman filter

The canonical example of using Kalman filter would be kinematic motion of a car, where state consists of position, velocity and potentially acceleration and other variables.…

Strict typed named dimension checking

Engineering

Coding

We will look at Jax Typing and how to do named dimension checking with a code example. This is really useful as the code is much easier to read and on top of that input and…

Formula that generates all primes

Algebra

Number Theory

Fun fact, one possible formula that generates all primes, \(2\) many times is: \[2+\left( (2 \times (n-1)!) \bmod n \right)\] Prime numbers are numbers that have only \(1\) …

GPT and more from scratch with comments

NLP

transformers

deep learning

code

analysis

All original credits to companion notebook to the Zero To Hero video on GPT by Andrej Karpathy. Code was modified and heavily commented.

Welcome To My Blog

news

This is the first post in my Quarto blog. Welcome! I will mainly post about the topics I like playing with. Sometimes I will use this blog to motivate the cleaner exposition…