This project was submitted by Raymond Tana. It was one of the top submissions in our AI Alignment course (Mar 2024). Participants worked on these projects for 4 weeks.
The main points one should take away from this project are as follows:
An understanding of the theoretical connections between compression and induction.
A sense that large language models likely do not (yet) qualify as general-compression algorithms; thus are “far” from performing universal induction.
A series of jumping-off points for further research in this area, including the measuring of a model’s compression capabilities via latent representation sizes in a manner inspired by Shannon information theory.
Read the full piece here.