Translation Model - 2

⠀⠀⠀⠀⠀⠀⠀⠀⢀⣴⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⢀⣠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⡄⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⣠⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣆⠀⠀⠀⠀⠀
⠀⠀⠀⣼⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣇⠀⠀⠀⠀
⠀⠀⣴⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣆⠀⠀⠀
⠀⣸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡄⠀⠀
⠀⣿⣿⣿⣿⣿⣿⣿⣷⢿⣿⣿⣿⣿⣿⣿⣿⣿⢻⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡀⠀
⢀⣿⠘⣿⡿⣿⣿⣿⣿⡎⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⢿⣿⣿⣿⣿⣿⣿⡇⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣧⠀
⠈⣿⠀⢿⣿⢹⣻⣿⣯⣿⡄⢻⣿⣿⣿⣿⣿⣿⡿⣿⣿⣿⣿⢸⣿⣿⣿⣿⣿⣿⣿⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡞⣿⣿⣿⡄
⠀⣿⡄⠘⣿⡇⢿⣿⡿⡜⢿⣀⣹⣯⠿⠟⠛⠋⠁⠉⠉⠉⠈⠀⢿⣿⣿⣿⣿⣿⣿⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⢻⣿⣿⣧
⠀⢿⣇⠀⠹⠿⠌⠋⠋⠉⠉⠉⠁⠀⣀⣠⡤⠶⢶⣶⣶⣶⣿⣿⣿⣿⣿⣿⣿⣿⣿⡆⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡆⣿⣿⣿
⠀⠀⢻⡤⠖⣒⣢⣄⠀⠀⠀⠀⠀⠀⠀⡟⠀⠀⠈⢿⠓⣻⡿⠋⠀⣿⣿⣿⣿⣿⣿⡇⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⢻⣿⡟
⠀⠀⠀⢹⣌⣠⠶⣾⣷⡄⠀⠀⠀⠀⠀⠀⠀⢀⢀⣀⣽⠋⠀⠀⠀⢹⣿⣿⣿⣿⣿⡇⣿⣿⣿⣿⣿⣿⣿⣿⣟⣿⣿⣿⣿⣿⠿⠋⠀
⠀⠀⠀⢸⣿⣿⡄⠘⠭⡇⠀⠀⠀⠀⠀⠀⠀⠈⠻⠛⠀⠀⠀⠀⠀⠘⣿⣿⣿⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣷⢿⣿⣿⣿⠟⠀⠀⠀
⠀⠀⠀⢸⣿⣿⡍⠳⢾⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣧⢸⣿⣿⣿⣿⣿⣿⣿⣿⠸⢛⠟⠁⠀⠀⠀⠀
⠀⠀⠀⢸⣿⣿⣇⢀⠊⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⣿⣿⣿⣿⣿⢸⣿⣿⣿⣿⣿⣿⣿⢿⠔⠁⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠨⣿⣿⣿⡌⣷⣄⠠⠄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⣿⣿⢸⣿⣿⣿⣿⣿⠿⠋⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⣿⣿⣇⠹⡌⠉⠁⠀⠀⠀⠀⣀⡤⢤⣴⡶⠀⠀⠀⠀⠀⠀⠘⣿⣿⣿⣿⠋⣾⢹⣿⣿⠋⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⣿⣿⣯⠀⠹⣄⠀⠀⠰⣯⣵⢟⣋⣽⡋⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⠀⣿⠸⠟⢹⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⢸⣿⣿⠀⠀⠙⢦⡀⠀⠈⠿⠿⠿⠛⠁⠀⠀⠀⠀⠀⠀⢀⣴⣿⣿⣿⡿⠀⠉⠀⠀⠘⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⢸⣿⣿⠀⠀⠀⢸⣿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⠴⠋⠀⣿⣿⣿⡇⠀⠀⣀⣠⣤⣧⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⣿⣿⡀⠀⠀⠀⢻⣿⣿⣄⠀⠀⠀⠀⠀⣀⣴⠛⠁⠀⠀⠀⣿⣿⣿⣷⣶⠿⠛⠋⠉⠘⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⢸⣿⡇⠀⠀⠀⠸⣿⣿⣿⣦⣀⣠⣴⡟⠋⢹⠀⠀⣀⣠⡴⣿⣿⣿⠇⠀⠀⠀⠀⠀⠀⢸⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀

This blog is continuation of the translation model blog which I added few weeks back. You can check it out here.

To give some context in the previous blog I explained the machine translation state pre attention, how it worked and what are the problems that might occur when training models for translation on corpus.

In this blog I would be focusing on attention based translation. I will be using GRU with attention mechanism and will try to explain what is happening under the hood.

Context in RNN models

Code Ref : RNN Enc-Dec

So the translator defined in Encoder Decoder Architecture is as follows :

Encoder : 2 Layer GRU
Decoder : 2 Layer GRU

Encoder

So each layer has it's own hidden layer and the encoder outputs two things :

encoder_outputs, encoder_hidden = encoder(source)

encoder_outputs: hidden state for every source token
encoder_hidden: final hidden state for each GRU layer

If we consider a Pytorch library GRU implementation we get the outputs as follows :

encoder_outputs : For each token passed through the encoder for example
```
  "I" → "love" → "cats"
  
  t_1     t_2      t_3
```
At each token position $t$ :

Layer 1: $h_t^1$ = GRU₁(token_embeddingₜ, $h_{t-1}^1$ ) Layer 2: $h_t^2$ = GRU₂( $h_t^1$ , $h_{t-1}^2$ )

Pytorch will return the last layer hidden states at every token position :

encoder_outputs = $[h_1^2, h_2^2, h_3^2]$
encoder_hidden : Final hidden state for each GRU layer after all the tokens are processed in the encoder. so for the above example :

encoder_hidden : $[h_3^1, h_3^2]$

Now we have these two outputs and they together form what we call EncoderState. So we will construct the DecoderState from this.

Decoder

Now we have everything from encoder as EncoderState now we pass those inputs to the decoder after extracting the DecoderState from it.

_outputs, hidden_state = EncoderState
context = hidden_state[-1]
hidden_state = hidden_state
DecoderState(context, hidden_state)

So our context is just the last layer's hidden state for the last token processed and our hidden state initialisation for decoder is initialised as encoder's last token hidden state for all the layers.

The input to the decoder is :

While training :

Input to the network is target_token_embedding + fixed_context. The context remains same for the whole decoding passes.
While eval/inference :

Input to the network is start/prev token embedding + fixed_context

  Decoder input       Decoder prediction
  <BOS>          →    "je"
  "je"           →    "suis"
  "suis"         →    "ici"
  "ici"          →    <EOS>

Attention based GRU

⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠿⠛⠛⠛⢿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡟⠁⠄⠄⠄⠄⠈⠄⠈⠙⢿⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣟⠄⠄⣠⣶⣷⣦⣠⡄⠄⠄⠈⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣿⡻⠟⠄⢰⣿⣿⣿⣿⣿⡿⠄⠄⠄⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣽⢉⠁⠄⠚⠙⠻⢿⢿⣿⡀⠄⠄⠄⣿⣿⣿⣿⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⣃⠆⠄⠄⠄⠄⢀⡆⠄⠄⠄⠄⠠⠳⠃⡏⠈⢙⣿⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⣿⠯⠄⠓⣤⣦⡶⢿⠇⠄⠄⠄⠄⠄⡄⡜⠄⠄⠌⡫⣿⣿⣿
⣿⣿⣿⣿⣿⣿⣿⡣⠁⠄⠄⠋⣤⣄⡀⠄⠄⠄⠄⠄⠄⢣⢍⠄⠄⠄⠆⣿⣿⣿
⣿⣿⣿⠛⠛⠉⠙⢂⣠⡖⡀⠄⠓⠄⠄⠄⠄⠄⠄⢐⣾⣷⡀⠄⠄⠘⠄⣿⣿⣿
⣿⣿⣿⠄⠄⠄⠄⣿⣿⣿⣿⡝⡄⠄⠄⠄⠄⠄⠄⠄⠙⢿⣧⡀⠄⠄⢀⢿⣿⣿
⣿⣿⣿⡀⠄⠄⠄⢿⣿⣿⣿⣇⠁⣀⡀⠄⠄⠄⠄⠄⠄⠰⣿⡂⠄⠁⠄⠈⣿

V.
Party’s not over.
We’ve still got attention to jack into this thing.

Adding attention to GRU is simple. We create an Additive Attention Block for our translation use case.

There are other types of attention too such as Dot Product Attention which is very fast to calculate and it has few scoring params and is Efficient on GPUs becuase of batch matrix multiplication while is highly parallelisable. It is used in large models, self attention because these cases have queries and keys sharing an embedding dimension.

Rule for using attentions :

Use scaled dot-product attention when:

query and key dimensions match,
speed matters,
there are many queries and keys,
or you are building a Transformer.

Use additive attention when:

encoder and decoder dimensions differ,
you want a learned nonlinear scoring function,
or you are reproducing classic RNN-based translation (which is what I am doing here)

scaled dot-product attention is the default because learned linear projections can already transform queries and keys before the dot product while retaining flexibility while remaining much more efficient.

Additive Attention

Additive Attention is describes as follows in mathematical representation. Everything is related to queries, keys and values here.

e_{t,i} = \mathbf{v}_a^\top \tanh\left( \mathbf{W}_q \mathbf{q}_t + \mathbf{W}_k \mathbf{k}_i \right)

Attention weights:

\alpha_{t,i}= \frac{\exp(e_{t,i})} {\sum_{j=1}^{n}\exp(e_{t,j})}

Context vector:

\mathbf{c}_t = \sum_{i=1}^{n} \alpha_{t,i}\mathbf{v}_i

Combined form:

\mathbf{c}_t = \sum_{i=1}^{n} \operatorname{softmax}_i \left( \mathbf{v}_a^\top \tanh\left( \mathbf{W}_q\mathbf{q}_t + \mathbf{W}_k\mathbf{k}_i \right) \right) \mathbf{v}_i

You can refer to the Additve Attention Implementation

How is attention integrated with GRU ?

Attention is nothing but a small feed forward neural network. As you can see in the implementation we first calculate the features and then calculate the scores from features. Our attention weights are the softmax of each of these scores. And to get the context we simply multiply the attention weights with the values thus giving us the context which we need according to the query.

Before explaining the attention in GRU let's see what the encoder returns in this case :

Encoder :

Similar to RNN encoder we get

encoder_outputs, encoder_hidden = encoder(source)

Decoder :

When initialising the DecoderState we add one more tensor of encoder_valid_lens which contains number of real source tokens excluding the padding tokens (this is related to how we pre process the corpus for training).

DecoderState(encoder_outputs, encoder_hidden, encoder_valid_lens)

We pass this decoder state to the decoder and we handle the logic there.

Coming back to how is attention added to GRU :

We take the token_embedding and decoder_state as input.
After converting the embeddings tensor to time stamp based input we calculate the attention query using the last layer of GRU decoder (in the very first decoder token generation we use the last layer's hidden layer from the encoder).
```
query = torch.unsqueeze(hidden_state[-1], dim=1)
```

To generate the context we pass the query and each token's encoder_output as key and value (yes it acts as both key and value).

context = self.attention(
    query, 
    enc_outputs, 
    enc_outputs, 
    enc_valid_lens
)

It basically means : 
Use encoder states to calculate relevance, 
then retrieve a weighted combination of those 
same states.

Now we generate the input to GRU by concatenating context with token embedding.
Hidden states of GRU as initialised as encoder's last hidden state for the first pass and then we update for next timestamps.
GRU output is stored for producing vocabulary logits and then we move towards the next iteration.
Then we convert the output logits in to vocab logits.

So in step 1 :

Query : encoder's last layer's hidden state
Key : encoder's last layer's hidden state for each token processed
Value : encoder's last layer's hidden state for each token processed

After step 1 :

Query : decoder's last layer's hidden state
Key : encoder's last layer's hidden state for each token processed
Value : encoder's last layer's hidden state for each token processed

Translation Model

Now coming to the translation model so I am using the same teacher forcing technique to train. Same Encoder Decoder architecture but with attention added.

One of the shortcoming of the RNN models was that It can't handle long sentences because fixed context gets noisy in long token chains.

Now let's see whether adding attention actually helped.

The model was trained for 15 epochs on the English–French translation dataset. The maximum sequence length was set to 20 tokens.

For inference I tested both:

Greedy decoding
Beam search

For quantitative evaluation I used BLEU-4 on the validation set.

BLEU Score

The attention-based model achieved:

Validation BLEU-4: 38.60

BLEU measures the overlap between the generated translation and the reference translation using token n-grams. You can see my article on BLEU score

A higher BLEU score does not necessarily mean that every translation is semantically correct but it gives us a useful aggregate measure over the validation dataset.

The more interesting part is looking at actual translations.

Here are some examples produced by the model using beam search:

English	Generated French
I am tired.	Je suis fatigué.
Where is the station?	Où se trouve la gare ?
I went to the market.	Je suis allé au marché.
My friend called me yesterday.	Mon ami m'a appelé hier.
I bought some fruit this morning.	J'ai acheté des fruits ce matin.
The weather is nice today.	Il fait beau aujourd'hui.
I saw him yesterday.	Je l'ai vu hier.
We decided to meet at the station tomorrow.	Nous avons décidé de nous rejoindre à la gare demain.

For short and medium-sized sentences, the model generally preserves both the sentence structure and its meaning quite well.

Greedy Decoding vs Beam Search

Greedy decoding selects the highest-probability token at every decoding step:

y_t = \arg\max_y P(y \mid y_{<t}, x)

The problem is that the best token right now may not lead to the best complete sentence.

Beam search instead keeps multiple candidate sequences alive.

For example:

Input

I want to walk in the park.

Greedy

Je veux faire dans le parc.

Beam Search

Je veux marcher dans le parc.

Here greedy decoding makes a locally plausible choice with faire, while beam search preserves another candidate and eventually produces the much better sequence marcher dans le parc.

Another example:

Input

I went to the market this morning.

Greedy

Je suis au marché, ce matin.

Beam Search

Je suis allé au marché ce matin.

However beam search does not automatically fix the model. For most of the test examples both decoding methods generated exactly the same output. This means that many remaining errors come from the probability distribution learned by the model itself rather than from the search strategy.

Where it failed ?

The model is much better than the fixed-context version, but attention does not make translation perfect.

Consider:

Input

We will meet tomorrow.

Generated

Nous partirons demain.

The generated sentence means approximately:

We will leave tomorrow.

The model correctly captures the future tense and tomorrow, but chooses the wrong verb.

Another example:

Input

I bought some fruit and came home.

Generated

J'ai acheté des fruits et chez moi.

The first clause is translated correctly, but the model fails to generate the verb required for "came home".

A better translation would be:

J'ai acheté des fruits et je suis rentré chez moi.

And for a longer sentence:

Input

The weather is nice today, so I want to walk in the park.

Generated

Il fait beau, donc je veux rester dans le parc.

This is grammatically reasonable French, but walk has become stay and today has disappeared.

This is an important distinction: the decoder can produce fluent French while still changing the meaning of the source sentence.

Conclusion

Results are much better than non attention RNN with fixed context but still attention didn't fix all of our problems. One thing that can be experimented with is bidirectional encoder that should improve the translation results. And a much more larger corpus can be used for training as the current corpus I am using is very small.

Effect of a Bidirectional Encoder

Implementation can be viewed Here

Replacing the unidirectional GRU encoder with a bidirectional GRU improved the validation BLEU-4 score from:

Unidirectional encoder: 38.60
Bidirectional encoder: 41.09

This is an absolute improvement of 2.49 BLEU points.

The qualitative outputs also changed in an interesting way. Longer sentences preserved their meaning more reliably, although some local lexical and grammatical errors remained.

Input

My friend called me yesterday and asked me to come.

Unidirectional

mon ami m'a appelé hier et je devais venir.

Bidirectional

mon ami m'a appelé hier et m'a demandé de venir.

Overall the improvement is there but it still didn't fix all of our problems obviously. One more thing which has to be tried is increasing the corpus so that model can learn more.