arXiv Artificial Intelligence

SFT-as-Context Mitigates Forgetting in Supervised Fine-Tuning

1 hour ago 09.10.2026 07:00 👁 1 views
SFT-as-Context Mitigates Forgetting in Supervised Fine-Tuning

Quick summary

arXiv:2610.11132v1 Announce Type: cross Abstract: Supervised fine-tuning (SFT) equips large language models (LLMs) with specialized capabilities, but often comes at the cost of forgetting the general capabilities of their parent models (i.e., the pretrained models before fine-tuning). This trade-off is especially limiting for queries that require both specialized and general capabilities. We introduce SFT-as-context, a training-free method in which the parent model uses the SFT model's response as context to answer the query. This allows the parent model to acquire fine-tuned capabilities from

Key takeaways

  • arXiv:2610.11132v1 Announce Type: cross Abstract: Supervised fine-tuning (SFT) equips large language models (LLMs) with specialized capabilities, but often comes at the cost of forgetting the general capabilities of their parent models (i.e., the pretrained models before fine-tuning).
  • This trade-off is especially limiting for queries that require both specialized and general capabilities.
  • We introduce SFT-as-context, a training-free method in which the parent model uses the SFT model's response as context to answer the query.

Why it matters

This model development creates a new option for users and a new testing obligation for developers. A fixed evaluation set comparing quality, cost and failure behavior is more useful than launch claims.

Kaynak sitede devamını oku: arXiv Artificial Intelligence ↗