Some investigations I made with AMD cloud credits: training language models from scratch, exploring low-bit and byte-level architectures, and running ablation studies.