Nov 1, 2019 · We introduce $k$NN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a $k$-nearest neighbors ($k$NN) model. The nearest neighbors …

本文提出 kNN-LM s，把上文的语义编码特征向量的k最近邻和一般的语言模型结合从而显著提高语言模型的效果。 本方法在 WIKITEXT-103 上取得15.79的ppl（将最好效果提高2.9个点），这表明了学习文本间相似度比预测下一个token更为简单。 语言模型（Language Model, LM）指的是利用链式法则给出一个句子的概率，主要要解决两个问题：（1）得到上文表示；（2）用上文表示预测下一个token。 这两个问题一般使用一个autoregressive模型解决。 使用AR模型去进行语言建 …

This is a HuggingFace's 🤗 transformers + Lightning ⚡️ implementation of K-Nearest Neighbors Augmented Language Models, designed to be easy to read & understand, useful in research, and for experimenting with new kNN-based model ideas.

We introduce kNN-LM, an approach that extends a pre-trained LM by linearly interpolating its next word distribution with a k-nearest neighbors (kNN) model. The nearest neighbors are computed

kNN-LM Explicitly memorizing the training data helps generalization. LMs can scaleto larger text collections without the added cost of training, by simply adding the data to the datastore. A single LM can adaptto multiple domains without the in-domain training, by adding domain-specific data to the datastore. 35

If your hardware constraints make this too slow, you can run it without using full precision keys by adding two flags: --no-load-keys and --knn-sim-func "do_not_recomp_l2". This uses the quantized versions of keys stored within the FAISS index.

Mar 2, 2020 · We introduce kNN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a k-nearest neighbors (kNN) model. The...

6 days ago · One such approach, the kNN-LM, interpolates any existing LM`s predictions with the output of a k-nearest neighbors model and requires no additional training. In this paper, we explore the importance of lexical and semantic matching in the context of …

We implement k-nearest-neighbor language model (kNN-LM) (Khandelwal et al., ICLR'2020), k-nearest-neighbor machine translation (kNN-MT) (Khandelwal et al., ICLR'2021) and this is also an official implementation of the RetoMaton model described in: Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval (ICML'2022).

6 days ago · We extensively study one such model, the k-nearest neighbor LM (kNN-LM), showing that the gains marginally transfer. The main challenge is to achieve coverage of the verbalizer tokens that define the different end-task class labels.