接受的答案清楚地说明了如何保存令牌生成器。以下是关于拟合或保存后 （通常）得分问题的评论。假设一个列表texts是由两个列表的Train_text和Test_text ，其中集令牌Test_text是集令牌的一个子集Train_text （乐观的假设）。然后fit_on_texts(Train_text)给出了不同的结果texts_to_sequences(Test_text)与第一通话相比fit_on_texts(texts) ，然后text_to_sequences(Test_text)

具体示例：

from keras.preprocessing.text import Tokenizer

docs = ["A heart that",
         "full up like",
         "a landfill",
        "no surprises",
        "and no alarms"
         "a job that slowly"
         "Bruises that",
         "You look so",
         "tired happy",
         "no alarms",
        "and no surprises"]
docs_train = docs[:7]
docs_test = docs[7:]
# EXPERIMENT 1: FIT  TOKENIZER ONLY ON TRAIN
T_1 = Tokenizer()
T_1.fit_on_texts(docs_train)  # only train set
encoded_train_1 = T_1.texts_to_sequences(docs_train)
encoded_test_1 = T_1.texts_to_sequences(docs_test)
print("result for test 1:\n%s" %(encoded_test_1,))

# EXPERIMENT 2: FIT TOKENIZER ON BOTH TRAIN + TEST
T_2 = Tokenizer()
T_2.fit_on_texts(docs)  # both train and test set
encoded_train_2 = T_2.texts_to_sequences(docs_train)
encoded_test_2 = T_2.texts_to_sequences(docs_test)
print("result for test 2:\n%s" %(encoded_test_2,))

结果：

result for test 1:
[[3], [10, 3, 9]]
result for test 2:
[[1, 19], [5, 1, 4]]

当然，如果不满足上述乐观假设，并且Test_text中的标记集与Train_test的标记集不相交，则测试1会得到一个空括号[].的列表[].

最常见的方法是使用pickle或joblib 。在这里，您有一个有关如何使用pickle保存Tokenizer的示例：

import pickle

# saving
with open('tokenizer.pickle', 'wb') as handle:
    pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)

# loading
with open('tokenizer.pickle', 'rb') as handle:
    tokenizer = pickle.load(handle)

Tokenizer类具有将日期保存为JSON格式的功能：

tokenizer_json = tokenizer.to_json()
with io.open('tokenizer.json', 'w', encoding='utf-8') as f:
    f.write(json.dumps(tokenizer_json, ensure_ascii=False))

可以使用keras_preprocessing.text tokenizer_from_json函数加载数据：

with open('tokenizer.json') as f:
    data = json.load(f)
    tokenizer = tokenizer_from_json(data)