Rixven

The Lab

The research log, unfiltered.

Notes from building Sponor, published as they're written. Some of this will be wrong. We'll say so here when it is.

  1. An illustration of an interlocking chain with one link broken and highlighted in ember orange, illustrating the KV cache as a chain of stored lookups. loading=lazy fetchpriority=auto />
    Exploring

    Notes on removing attention

    The KV cache is not a bug in the transformer. It is the transformer, presenting its bill.
  2. A row of evenly spaced dots with a gap between two of them highlighted in ember orange, illustrating the spectral gap the bound is derived from. loading=lazy fetchpriority=auto />
    In progress

    The Effective Context Bound

    If a model's context is a resonance field, its maximum useful length is not a hyperparameter. It falls out of the spectral gap.