Describe Images with AI

Executive Mansion  
Washington, Nov 21, 1864  

To Mrs. Bixby, Boston, Mass,  

Dear Madam,  

I have been shown in the files of the War Department a statement of the Adjutant General of Massachusetts that you are the mother of five sons who have died gloriously on the field of battle. I feel how weak and fruitless must be any word of mine w…Executive Mansion  
Washington, Nov 21, 1864  

To Mrs. Bixby, Boston, Mass,  

Dear Madam,  

I have been shown in the files of the War Department a statement of the Adjutant General of Massachusetts that you are the mother of five sons who have died gloriously on the field of battle. I feel how weak and fruitless must be any word of mine which should attempt to beguile you from the grief of a loss so overwhelming. But I cannot refrain from tendering you the consolation that may be found in the thanks of the republic they died to save. I pray that our Heavenly Father may assuage the anguish of your bereavement, and leave you only the cherished memory of the loved and lost, and the solemn pride that must be yours to have laid so costly a sacrifice upon the altar of freedom.  

Yours very sincerely and respectfully,  

A. Lincoln  

The Celebrated "Bixby Letter."  
Facsimile of the Original Manuscript on Exhibition at Huber's Museum in New York City.  
A Letter of Condolence Written by Abraham Lincoln to Mrs. Bixby, of Boston, Mass., November 21, 1864

2.3 Attention with relative position bias

Given queries $Q$, keys $K$ and values $V$, each head computes

$$\text{Attn}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + B\right)V,$$

where $B_{ij} = b_{\text{clip}(i-j,-k,k)}$ is a learned bias shared across layers. The training objective adds a weight penalty:

$$\mathcal{L} = -…2.3 Attention with relative position bias

Given queries $Q$, keys $K$ and values $V$, each head computes

$$\text{Attn}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + B\right)V,$$

where $B_{ij} = b_{\text{clip}(i-j,-k,k)}$ is a learned bias shared across layers. The training objective adds a weight penalty:

$$\mathcal{L} = -\sum_{t=1}^{T}\log p_\theta(x_t \mid x_{<t}) + \lambda\|\theta\|_2^2.$$

Table 2. Perplexity on the validation set (lower is better).

<table>
  <tr>
    <th>Model</th>
    <th>Params</th>
    <th>$k$</th>
    <th>PPL</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>124M</td>
    <td>—</td>
    <td>21.4</td>
  </tr>
  <tr>
    <td>+ relative bias</td>
    <td>124M</td>
    <td>16</td>
    <td>19.8</td>
  </tr>
  <tr>
    <td>+ relative bias</td>
    <td>124M</td>
    <td>64</td>
    <td>19.1</td>
  </tr>
</table>

The gain saturates beyond $k = 64$, consistent with the effective context length of $\approx 2^9$ tokens.