Temperature

I was recently studying the working of RNNs (Recurrent Neural Networks) and I tried to experiment with generating a character based language model by writing my own implementation of RNN layers after studying. If anyone is interested in seeing the code please do check this out. So I used this implementation to create a Language Model.

After training for the corpus of "The Time Machine" book I was trying to get my predictions for some prompts. And the result I got was very interesting. So I am using Greedy sampling here and what I saw was that for a given prompt let's say for 15-20 chars and the tokens to be generated for let's say next 5 chars the greedy sampling works really good. For eg :

Prompt length:  19

Prompt: he called me and sa
=================================

Generated Text (Greedy): he called me and sa[w a m]

But when I want to generate a large token string for let's say 50 something gets strange in the output :

Prompt: he called me and sa
=================================
Generated Text (Greedy): he called me and sa[w a minute of the solent at the 
sound the solent a]

Here the result becomes repititive and the model will keep generasting this boring text for eternity. Then I came across a technique called Temperature Sampling.

The Math

Let's suppose my model generates some logits and I get some raw scores :

$[1, 2, 0, 5, \dots]$

When applying softmax we simply do this :

p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}

But for temprature we add an extra term $T$ :

p_i(T) = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}} \space\space,\space T > 0

Effect of temperature

For the logits $[2,1,0]$ :

Low temperature: T = 0.5

The highest-scoring token becomes much more likely.
```
z/T = [4,2,0]
p(0.5) ≈ [0.867,0.117,0.016]
```
Normal temperature: T = 1

This is the model’s original softmax distribution.
```
p(1) ≈ [0.665,0.245,0.090]
```
High temperature: T = 2

The distribution becomes more uniform, giving unlikely tokens a greater chance.
```
z/T = [1,0.5,0]
p(2) ≈ [0.506,0.307,0.186]
```

Consider two tokens $i$ and $j$ . Their probability ratio is:

\frac{p_i(T)}{p_j(T)} = \frac{e^{z_i/T}}{e^{z_j/T}} = e^{(z_i-z_j)/T}

Taking log

\log\left(\frac{p_i(T)}{p_j(T)}\right)= \frac{z_i-z_j}{T}

\begin{aligned} T < 1 &\quad \text{enlarges log-probability differences},\\ T > 1 &\quad \text{reduces log-probability differences},\\ T = 1 &\quad \text{leaves log-probability differences unchanged}. \end{aligned}

So more temperature means more uncertainity in tokens and more creative output.

Same prompt in greedy sampling and temperature sampling :

Prompt length:  19


Prompt: he called me and sa
=================================
Generated Text (Temperature): he called me and s[avage animal in the
same bar, i and tolects had imi]


Prompt: he called me and sa
=================================
Generated Text (Greedy): he called me and s[aw it
first place, i saw the little people were sli]

There is a relation between entropy and temprature. As entropy measures uncertainity for the next token in language models. Given context $x_{< t}$ , the model predicts a probability distribution over vocabulary tokens:

p(wᵢ ∣ x_{< t})

H_t= -\sum_{i=1}^{V} p(w_i \mid x_{<t}) \log p(w_i \mid x_{<t})

Low entropy: one token has most probability --> the model is confident.
High entropy: probability is spread across many tokens --> the model is uncertain.

Therefore:

Low temperature --> low entropy --> predictable output.
High temperature --> high entropy --> varied output.

Sampling example in Pytorch :

next_logits = logits[0, -1]
probabilities = torch.softmax(next_logits / temperature, dim=-1)
next_token = torch.multinomial(probabilities, 1)