如何为xgboost实施增量培训?
machine-learning
python
xgboost
18
0

问题是由于火车数据大小,我的火车数据无法放入RAM。因此,我需要一种方法,该方法首先在整个火车数据集上构建一棵树,然后计算残差来构建另一棵树,依此类推(就像梯度增强树一样)。显然,如果我在某个循环中调用model = xgb.train(param, batch_dtrain, 2) -这将无济于事,因为在这种情况下,它只会为每个批次重建整个模型。

参考资料:
Stack Overflow
收藏
评论
共 5 个回答
高赞 时间 活跃

我创建了jupyter笔记本的要点,以演示可以逐步训练xgboost模型。我使用波士顿数据集来训练模型。我做了3个实验-一枪学习,迭代一枪学习,迭代增量学习。在增量训练中,我将波士顿数据分批传递给模型,大小为50。

要点是,您必须多次遍历数据才能使模型收敛到一次射击(所有数据)学习所获得的精度。

这是用于使用xgboost进行迭代增量学习的相应代码。

batch_size = 50
iterations = 25
model = None
for i in range(iterations):
    for start in range(0, len(x_tr), batch_size):
        model = xgb.train({
            'learning_rate': 0.007,
            'update':'refresh',
            'process_type': 'update',
            'refresh_leaf': True,
            #'reg_lambda': 3,  # L2
            'reg_alpha': 3,  # L1
            'silent': False,
        }, dtrain=xgb.DMatrix(x_tr[start:start+batch_size], y_tr[start:start+batch_size]), xgb_model=model)

        y_pr = model.predict(xgb.DMatrix(x_te))
        #print('    MSE itr@{}: {}'.format(int(start/batch_size), sklearn.metrics.mean_squared_error(y_te, y_pr)))
    print('MSE itr@{}: {}'.format(i, sklearn.metrics.mean_squared_error(y_te, y_pr)))

y_pr = model.predict(xgb.DMatrix(x_te))
print('MSE at the end: {}'.format(sklearn.metrics.mean_squared_error(y_te, y_pr)))

XGBoost版本:0.6

收藏
评论

看起来除了再次调用xgb.train(....)之外,您不需要其他任何东西,但可以提供前一批的模型结果:

# python
params = {} # your params here
ith_batch = 0
n_batches = 100
model = None
while ith_batch < n_batches:
    d_train = getBatchData(ith_batch)
    model = xgb.train(params, d_train, xgb_model=model)
    ith_batch += 1

这是基于https://xgboost.readthedocs.io/en/latest/python/python_api.html 在此处输入图片说明

收藏
评论

如果您的问题与数据集大小有关,而您实际上并不需要增量学习(例如,您没有在使用Streaming应用程序),则应查看Spark或Flink。

这两个框架可以利用磁盘内存在具有较小RAM的超大型数据集上进行训练。这两个框架都在内部处理内存问题。尽管Flink首先解决了问题,但Spark赶上了最新版本。

看一眼:

收藏
评论

现在有一个(版本0.6?)可能有用的process_update参数。这是一个实验:

import pandas as pd
import xgboost as xgb
from sklearn.model_selection import ShuffleSplit
from sklearn.datasets import load_boston
from sklearn.metrics import mean_squared_error as mse

boston = load_boston()
features = boston.feature_names
X = boston.data
y = boston.target

X=pd.DataFrame(X,columns=features)
y = pd.Series(y,index=X.index)

# split data into training and testing sets
rs = ShuffleSplit(test_size=0.3, n_splits=1, random_state=0)
for train_idx,test_idx in rs.split(X):  # this looks silly
    pass

train_split = round(len(train_idx) / 2)
train1_idx = train_idx[:train_split]
train2_idx = train_idx[train_split:]
X_train = X.loc[train_idx]
X_train_1 = X.loc[train1_idx]
X_train_2 = X.loc[train2_idx]
X_test = X.loc[test_idx]
y_train = y.loc[train_idx]
y_train_1 = y.loc[train1_idx]
y_train_2 = y.loc[train2_idx]
y_test = y.loc[test_idx]

xg_train_0 = xgb.DMatrix(X_train, label=y_train)
xg_train_1 = xgb.DMatrix(X_train_1, label=y_train_1)
xg_train_2 = xgb.DMatrix(X_train_2, label=y_train_2)
xg_test = xgb.DMatrix(X_test, label=y_test)

params = {'objective': 'reg:linear', 'verbose': False}
model_0 = xgb.train(params, xg_train_0, 30)
model_1 = xgb.train(params, xg_train_1, 30)
model_1.save_model('model_1.model')
model_2_v1 = xgb.train(params, xg_train_2, 30)
model_2_v2 = xgb.train(params, xg_train_2, 30, xgb_model=model_1)

params.update({'process_type': 'update',
               'updater'     : 'refresh',
               'refresh_leaf': True})
model_2_v2_update = xgb.train(params, xg_train_2, 30, xgb_model=model_1)

print('full train\t',mse(model_0.predict(xg_test), y_test)) # benchmark
print('model 1 \t',mse(model_1.predict(xg_test), y_test))  
print('model 2 \t',mse(model_2_v1.predict(xg_test), y_test))  # "before"
print('model 1+2\t',mse(model_2_v2.predict(xg_test), y_test))  # "after"
print('model 1+update2\t',mse(model_2_v2_update.predict(xg_test), y_test))  # "after"

输出:

full train   17.8364309709
model 1      24.2542132108
model 2      25.6967017352
model 1+2    22.8846455135
model 1+update2  14.2816257268
收藏
评论

免责声明:我也是xgboost的新手,但我想我已经明白了。

训练第一批后,尝试保存模型。然后,在连续运行中,向xgb.train方法提供已保存模型的文件路径。

这是我进行的一个小实验,目的是让自己确信它是可行的:

首先,将波士顿数据集分为训练和测试集。然后将训练集分成两半。将模型与前半部分拟合,并得到一个分数作为基准。然后在后半部分安装两个模型;一个模型将具有附加参数xgb_model 。如果传入多余的参数没有影响,那么我们希望他们的得分是相似的。但是,幸运的是,新模型的性能似乎比第一个模型好得多。

import xgboost as xgb
from sklearn.cross_validation import train_test_split as ttsplit
from sklearn.datasets import load_boston
from sklearn.metrics import mean_squared_error as mse

X = load_boston()['data']
y = load_boston()['target']

# split data into training and testing sets
# then split training set in half
X_train, X_test, y_train, y_test = ttsplit(X, y, test_size=0.1, random_state=0)
X_train_1, X_train_2, y_train_1, y_train_2 = ttsplit(X_train, 
                                                     y_train, 
                                                     test_size=0.5,
                                                     random_state=0)

xg_train_1 = xgb.DMatrix(X_train_1, label=y_train_1)
xg_train_2 = xgb.DMatrix(X_train_2, label=y_train_2)
xg_test = xgb.DMatrix(X_test, label=y_test)

params = {'objective': 'reg:linear', 'verbose': False}
model_1 = xgb.train(params, xg_train_1, 30)
model_1.save_model('model_1.model')

# ================= train two versions of the model =====================#
model_2_v1 = xgb.train(params, xg_train_2, 30)
model_2_v2 = xgb.train(params, xg_train_2, 30, xgb_model='model_1.model')

print(mse(model_1.predict(xg_test), y_test))     # benchmark
print(mse(model_2_v1.predict(xg_test), y_test))  # "before"
print(mse(model_2_v2.predict(xg_test), y_test))  # "after"

# 23.0475232194
# 39.6776876084
# 27.2053239482

让我知道是否有任何不清楚的地方!

参考: https : //github.com/dmlc/xgboost/blob/master/python-package/xgboost/training.py

收藏
评论
新手导航
  • 社区规范
  • 提出问题
  • 进行投票
  • 个人资料
  • 优化问题
  • 回答问题

关于我们

常见问题

内容许可

联系我们

@2020 AskGo
京ICP备20001863号