Vocabulary adaptation of the Bielik v3 PL models combines FOCUS-based embedding initialisation with continued pretraining on 4B tokens that updates only the input embedding layer, the language modelling head and four boundary transformer layers, followed by 16B tokens with all parameters unfrozen, to mitigate catastrophic forgetting.

Section 7, p. 15

To mitigate catastrophic forgetting during vocabulary adaptation, we combined FOCUS-based embedding initialization with a two-stage continued pretraining pipeline (4B tokens with partial freezing, followed by 16B tokens of full adaptation)

Section 4.1.1, p. 4

Continued pretraining is performed on 4B tokens, while most of the model parameters remain frozen. Only the following components are updated: • The input embedding layer, • The language modeling head (lm_head), • Four boundary transformer layers (two lowest and two highest layers).

Section 4.1.2, p. 4

After initial stabilization, all model parameters are unfrozen. The model then undergoes continued pretraining on an additional 16B tokens.