Wavelets Wall background

~90% of AI’s energy is devoted to one core, expensive operation: matrix multiplication.This partly explains why so much research and engineering on LLM inference targets reducing the IO and complexity of MatMuls: O(n³) FLOPs to process only O(n²) data. When n = 30K, the gap is enormous...A question I find fascinating is whether there exist alternative bilinear operators f(W, X) for “mixing” LLM tokens, that are as “expressive” as MatMul for human-related tasks, but require ≪ n³ FLOPs (and ideall

clip-19189429-thrown-stone-plops-into-water-producing-waves