Breaking Down Decoders From the Bottom Up