Out of bag ошибка

From Wikipedia, the free encyclopedia

Out-of-bag (OOB) error, also called out-of-bag estimate, is a method of measuring the prediction error of random forests, boosted decision trees, and other machine learning models utilizing bootstrap aggregating (bagging). Bagging uses subsampling with replacement to create training samples for the model to learn from. OOB error is the mean prediction error on each training sample x_i, using only the trees that did not have x_i in their bootstrap sample.^[1]

Bootstrap aggregating allows one to define an out-of-bag estimate of the prediction performance improvement by evaluating predictions on those observations that were not used in the building of the next base learner.

Out-of-bag dataset[edit]

When bootstrap aggregating is performed, two independent sets are created. One set, the bootstrap sample, is the data chosen to be «in-the-bag» by sampling with replacement. The out-of-bag set is all data not chosen in the sampling process.

When this process is repeated, such as when building a random forest, many bootstrap samples and OOB sets are created. The OOB sets can be aggregated into one dataset, but each sample is only considered out-of-bag for the trees that do not include it in their bootstrap sample. The picture below shows that for each bag sampled, the data is separated into two groups.

Visualizing the bagging process. Sampling 4 patients from the original set with replacement and showing the out-of-bag sets. Only patients in the bootstrap sample would be used to train the model for that bag.

This example shows how bagging could be used in the context of diagnosing disease. A set of patients are the original dataset, but each model is trained only by the patients in its bag. The patients in each out-of-bag set can be used to test their respective models. The test would consider whether the model can accurately determine if the patient has the disease.

Calculating out-of-bag error[edit]

Since each out-of-bag set is not used to train the model, it is a good test for the performance of the model. The specific calculation of OOB error depends on the implementation of the model, but a general calculation is as follows.

Find all models (or trees, in the case of a random forest) that are not trained by the OOB instance.
Take the majority vote of these models’ result for the OOB instance, compared to the true value of the OOB instance.
Compile the OOB error for all instances in the OOB dataset.

An illustration of OOB error

The bagging process can be customized to fit the needs of a model. To ensure an accurate model, the bootstrap training sample size should be close to that of the original set.^[2] Also, the number of iterations (trees) of the model (forest) should be considered to find the true OOB error. The OOB error will stabilize over many iterations so starting with a high number of iterations is a good idea.^[3]

Shown in the example to the right, the OOB error can be found using the method above once the forest is set up.

Comparison to cross-validation[edit]

Out-of-bag error and cross-validation (CV) are different methods of measuring the error estimate of a machine learning model. Over many iterations, the two methods should produce a very similar error estimate. That is, once the OOB error stabilizes, it will converge to the cross-validation (specifically leave-one-out cross-validation) error.^[3] The advantage of the OOB method is that it requires less computation and allows one to test the model as it is being trained.

Accuracy and Consistency[edit]

Out-of-bag error is used frequently for error estimation within random forests but with the conclusion of a study done by Silke Janitza and Roman Hornung, out-of-bag error has shown to overestimate in settings that include an equal number of observations from all response classes (balanced samples), small sample sizes, a large number of predictor variables, small correlation between predictors, and weak effects.^[4]

References[edit]

^ James, Gareth; Witten, Daniela; Hastie, Trevor; Tibshirani, Robert (2013). An Introduction to Statistical Learning. Springer. pp. 316–321.
^ Ong, Desmond (2014). A primer to bootstrapping; and an overview of doBootstrap (PDF). pp. 2–4.
^ ^a ^b Hastie, Trevor; Tibshirani, Robert; Friedman, Jerome (2008). The Elements of Statistical Learning (PDF). Springer. pp. 592–593.
^ Janitza, Silke; Hornung, Roman (2018-08-06). «On the overestimation of random forest’s out-of-bag error». PLOS ONE. 13 (8): e0201904. doi:10.1371/journal.pone.0201904. ISSN 1932-6203. PMC 6078316. PMID 30080866.

Источник

From Wikipedia, the free encyclopedia

Out-of-bag dataset[edit]

Calculating out-of-bag error[edit]

Find all models (or trees, in the case of a random forest) that are not trained by the OOB instance.
Take the majority vote of these models’ result for the OOB instance, compared to the true value of the OOB instance.
Compile the OOB error for all instances in the OOB dataset.

An illustration of OOB error

Shown in the example to the right, the OOB error can be found using the method above once the forest is set up.

Comparison to cross-validation[edit]

Accuracy and Consistency[edit]

References[edit]

^ James, Gareth; Witten, Daniela; Hastie, Trevor; Tibshirani, Robert (2013). An Introduction to Statistical Learning. Springer. pp. 316–321.
^ Ong, Desmond (2014). A primer to bootstrapping; and an overview of doBootstrap (PDF). pp. 2–4.
^ ^a ^b Hastie, Trevor; Tibshirani, Robert; Friedman, Jerome (2008). The Elements of Statistical Learning (PDF). Springer. pp. 592–593.
^ Janitza, Silke; Hornung, Roman (2018-08-06). «On the overestimation of random forest’s out-of-bag error». PLOS ONE. 13 (8): e0201904. doi:10.1371/journal.pone.0201904. ISSN 1932-6203. PMC 6078316. PMID 30080866.

Источник

Note

Click here
to download the full example code or to run this example in your browser via Binder

The RandomForestClassifier is trained using bootstrap aggregation, where
each new tree is fit from a bootstrap sample of the training observations
(z_i = (x_i, y_i)). The out-of-bag (OOB) error is the average error for
each (z_i) calculated using predictions from the trees that do not
contain (z_i) in their respective bootstrap sample. This allows the
RandomForestClassifier to be fit and validated whilst being trained [1].

The example below demonstrates how the OOB error can be measured at the
addition of each new tree during training. The resulting plot allows a
practitioner to approximate a suitable value of n_estimators at which the
error stabilizes.

plot ensemble oob

# Author: Kian Ho <hui.kian.ho@gmail.com>
#         Gilles Louppe <g.louppe@gmail.com>
#         Andreas Mueller <amueller@ais.uni-bonn.de>
#
# License: BSD 3 Clause

import matplotlib.pyplot as plt

from collections import OrderedDict
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier

RANDOM_STATE = 123

# Generate a binary classification dataset.
X, y = make_classification(
    n_samples=500,
    n_features=25,
    n_clusters_per_class=1,
    n_informative=15,
    random_state=RANDOM_STATE,
)

# NOTE: Setting the `warm_start` construction parameter to `True` disables
# support for parallelized ensembles but is necessary for tracking the OOB
# error trajectory during training.
ensemble_clfs = [
    (
        "RandomForestClassifier, max_features='sqrt'",
        RandomForestClassifier(
            warm_start=True,
            oob_score=True,
            max_features="sqrt",
            random_state=RANDOM_STATE,
        ),
    ),
    (
        "RandomForestClassifier, max_features='log2'",
        RandomForestClassifier(
            warm_start=True,
            max_features="log2",
            oob_score=True,
            random_state=RANDOM_STATE,
        ),
    ),
    (
        "RandomForestClassifier, max_features=None",
        RandomForestClassifier(
            warm_start=True,
            max_features=None,
            oob_score=True,
            random_state=RANDOM_STATE,
        ),
    ),
]

# Map a classifier name to a list of (<n_estimators>, <error rate>) pairs.
error_rate = OrderedDict((label, []) for label, _ in ensemble_clfs)

# Range of `n_estimators` values to explore.
min_estimators = 15
max_estimators = 150

for label, clf in ensemble_clfs:
    for i in range(min_estimators, max_estimators + 1, 5):
        clf.set_params(n_estimators=i)
        clf.fit(X, y)

        # Record the OOB error for each `n_estimators=i` setting.
        oob_error = 1 - clf.oob_score_
        error_rate[label].append((i, oob_error))

# Generate the "OOB error rate" vs. "n_estimators" plot.
for label, clf_err in error_rate.items():
    xs, ys = zip(*clf_err)
    plt.plot(xs, ys, label=label)

plt.xlim(min_estimators, max_estimators)
plt.xlabel("n_estimators")
plt.ylabel("OOB error rate")
plt.legend(loc="upper right")
plt.show()

Total running time of the script: ( 0 minutes 3.257 seconds)

Gallery generated by Sphinx-Gallery

Источник

Out-of-bag datasetEdit

Calculating out-of-bag errorEdit

Find all models (or trees, in the case of a random forest) that are not trained by the OOB instance.
Take the majority vote of these models’ result for the OOB instance, compared to the true value of the OOB instance.
Compile the OOB error for all instances in the OOB dataset.

An illustration of OOB error

Shown in the example to the right, the OOB error can be found using the method above once the forest is set up.

Comparison to cross-validationEdit

Accuracy and ConsistencyEdit

ReferencesEdit

^ James, Gareth; Witten, Daniela; Hastie, Trevor; Tibshirani, Robert (2013). An Introduction to Statistical Learning. Springer. pp. 316–321.
^ Ong, Desmond (2014). A primer to bootstrapping; and an overview of doBootstrap (PDF). pp. 2–4.
^ ^a ^b Hastie, Trevor; Tibshirani, Robert; Friedman, Jerome (2008). The Elements of Statistical Learning (PDF). Springer. pp. 592–593.
^ Janitza, Silke; Hornung, Roman (2018-08-06). «On the overestimation of random forest’s out-of-bag error». PLOS ONE. 13 (8): e0201904. doi:10.1371/journal.pone.0201904. ISSN 1932-6203. PMC 6078316. PMID 30080866.

Источник

Что из ошибки мешка в случайных лесах?
Является ли это оптимальным параметром для нахождения правильного количества деревьев в случайном лесу?

2 ответов

я попытаюсь объяснить:

предположим, что наш набор данных обучения представлен T и предположим, что набор данных имеет M функций (или атрибутов или переменных).

T = {(X1,y1), (X2,y2), ... (Xn, yn)}

Xi is input vector {xi1, xi2, ... xiM}

yi is the label (or output or class).

резюме РФ:

алгоритм случайных лесов-это классификатор, основанный в основном на двух методах —

мешков
метод случайных подпространств.

Предположим, мы решим иметь S количество деревьев в нашем лесу, то мы сначала создаем S наборы "same size as original" создано из случайной повторной выборки данных в T с заменой (n раз для каждого набора данных). Это приведет к {T1, T2, ... TS} наборы данных. Каждый из них называется набором данных bootstrap. Из-за «with-replacement» каждый набор данных Ti может иметь повторяющиеся записи данных, и Ti может отсутствовать несколько записей данных из исходных наборов данных. Это называется Bootstrapping. (en.wikipedia.org/wiki/Bootstrapping_(статистика))

Bagging-это процесс взятия бутстрапов , а затем агрегирования моделей, изученных на каждом бутстрапе.

теперь RF создает S деревья и использует m (=sqrt(M) or =floor(lnM+1)) случайные компоненты из M варианты для создания любого дерева. Это называется методом случайного подпространства.

для каждого Ti bootstrap dataset вы создаете дерево Ki. Если вы хотите классифицировать некоторые входные данные D = {x1, x2, ..., xM} вы дайте ему пройти через каждое дерево и произвести S выходы (по одному для каждого дерева), которые могут быть обозначены Y = {y1, y2, ..., ys}. Окончательный прогноз-большинство голосов на этом наборе.

ошибка выхода из мешка:

после создания классификаторов (S деревьев) для каждого (Xi,yi) в оригинальном наборе обучения, т. е. T, выделить все Tk, который не включает в себя (Xi,yi). Это подмножество, обратите внимание, представляет собой набор наборов данных boostrap, который не содержит конкретной записи из исходного набора данных. Этот набор называется out-of-bag образцы. Есть n такие подмножества (по одному для каждой записи данных в исходном наборе данных T). Классификатор OOB-это агрегация голосов только за Tk такой, что не содержит (xi,yi).

Out-of-bag оценка ошибки обобщения-это частота ошибок классификатора out-of-bag на обучающем наборе (сравните его с известным yi ‘ s).

почему это важно?
Исследование оценок погрешностей для мешочных классификаторов в Breiman [1996b], дает эмпирические доказательства того, что оценка «вне мешка» столь же точна, как и использование тестового набора того же размера, что и учебный набор. Таким образом с помощью оценки ошибки из мешка устраняет необходимость в наборе отложенных тестов.

(спасибо @Rudolf за исправления. Его комментарии приводятся ниже.)

в оригинальной реализации алгоритма случайного леса Бреймана каждое дерево обучается примерно на 2/3 от общих данных обучения. По мере построения леса каждое дерево может быть протестировано (аналогично исключению перекрестной проверки) на образцах, не используемых при построении этого дерева. Это оценка ошибки out of bag — внутренняя оценка ошибки случайного леса по мере его построения.

Источник

Out of bag ошибка

Out-of-bag dataset[edit]

Calculating out-of-bag error[edit]

Comparison to cross-validation[edit]

Accuracy and Consistency[edit]

See also[edit]

References[edit]

Out-of-bag dataset[edit]

Calculating out-of-bag error[edit]

Comparison to cross-validation[edit]

Accuracy and Consistency[edit]

See also[edit]

References[edit]

Out-of-bag datasetEdit

Calculating out-of-bag errorEdit

Comparison to cross-validationEdit

Accuracy and ConsistencyEdit

See alsoEdit

ReferencesEdit

2 ответов

А вот еще интересные материалы: