Basic damage detection in Wikipedia¶

This notebook demonstrates the basic contruction of a vandalism classification system using the revscoring library that we have developed specifically for classification models of MediaWiki stuff.

The basic process that we'll follow is this:

  1. Gather example of human judgement applied to Wikipedia edits. In this case, we'll take advantage of reverts.
  2. Split the data into a training and testing set
  3. Training the machine learning model
  4. Testing the machine learning model

And then we'll have some fun applying the model to some edits using RCStream. The following diagram gives a good sense for the whole process of training and evaluating a model.

Part 1: Getting labeled observations¶

Regretfully, running SQL queries isn't something we can do directly from the notebook yet. So, we'll use Quarry to generate a nice random sample of edits. 20,000 observations should do just fine. Here's the query I want to run:

USE enwiki_p;
SELECT rev_id 
FROM revision 
WHERE rev_timestamp BETWEEN "20150201" AND "20160201" 
ORDER BY RAND() 
LIMIT 20000;

See http://quarry.wmflabs.org/query/7530. By clicking around the UI, I can see that this URL will download my tab-separated file: http://quarry.wmflabs.org/run/65415/output/0/tsv?download=true

In [1]:
# Magical ipython notebook stuff puts the result of this command into a variable
revids_f = !wget http://quarry.wmflabs.org/run/65415/output/0/tsv?download=true -qO- 

revids = [int(line) for line in revids_f[1:]]
len(revids)
Out[1]:
20000

OK. Now that we have a set of revisions, we need to label them. In this case, we're going to label them as reverted/not. We want to exclude a few different types of reverts -- e.g. when a user reverts themself or when an edit is reverted back to by someone else. For this, we'll use the mwreverts and mwapi libraries.

In [2]:
import sys, traceback
import mwreverts.api
import mwapi

# We'll use the mwreverts API check.  In order to do that, we need an API session
session = mwapi.Session("https://en.wikipedia.org", 
                        user_agent="Revert detection demo <ahalfaker@wikimedia.org>")

# For each revision, find out if it was "reverted" and label it so.
rev_reverteds = []
for rev_id in revids[:20]:  # TODO: Limiting to the first 20!!!!
    try:
        _, reverted, reverted_to = mwreverts.api.check(
            session, rev_id, radius=5,  # most reverts within 5 edits
            window=48*60*60,  # 2 days
            rvprop={'user', 'ids'})  # Some properties we'll make use of
    except RuntimeError as e:
        sys.stderr.write(str(e))
        continue
    
    if reverted is not None:
        reverted_doc = [r for r in reverted.reverteds
                        if r['revid'] == rev_id][0]

        # self-reverts
        self_revert = \
            reverted_doc['user'] == reverted.reverting['user']
        
        # revisions that are reverted back to by others
        reverted_back_to = \
            reverted_to is not None and \
            reverted_doc['user'] != \
            reverted_to.reverting['user']
        
        # If we are reverted, not by self or reverted back to by someone else, 
        # then, let's assume it was damaging.
        damaging_reverted = !(self_revert or reverted_back_to)
    else:
        damaging_reverted = False

    rev_reverteds.append((rev_id, damaging_reverted))
    sys.stderr.write("r" if damaging_reverted else ".")

...............r....

Eeek! This takes too long. You get the idea. So, I uploaded dataset that has already been labeled here @ ../datasets/demo/enwiki.rev_reverted.20k_2015.tsv.bz2

In [3]:
rev_reverteds_f = !bzcat ../datasets/demo/enwiki.rev_reverted.20k_2015.tsv.bz2
rev_reverteds = [line.strip().split("\t") for line in rev_reverteds_f[1:]]
rev_reverteds = [(int(rev_id), reverted == "True") for rev_id, reverted in rev_reverteds]
len(rev_reverteds)
Out[3]:
19868

OK. It looks like we got an error when trying to extract the reverted status of ~132 edits, which is an acceptable loss. Now just to make sure we haven't gone crazy, let's check some of the reverted edits:

  • https://en.wikipedia.org/wiki/?diff=695071713 (section blanking)
  • https://en.wikipedia.org/wiki/?diff=667375206 (unexplained addition of nonsense)
  • https://en.wikipedia.org/wiki/?diff=670204366 (vandalism "I don't know")
  • https://en.wikipedia.org/wiki/?diff=680329354 (adds non-existent category)
  • https://en.wikipedia.org/wiki/?diff=668682186 (test edit -- removes punctuation)
  • https://en.wikipedia.org/wiki/?diff=666882037 (adds spamlink)
  • https://en.wikipedia.org/wiki/?diff=663302354 (adds nonsense special char)
  • https://en.wikipedia.org/wiki/?diff=675803278 (unconstructive link changes)
  • https://en.wikipedia.org/wiki/?diff=680203994 (vandalism -- "Pepe meme")
  • https://en.wikipedia.org/wiki/?diff=656734057 ("JELENAS BOOTY UNDSO")

OK. Looks like we are doing pretty good. :)

Part 2: Split the data into a training and testing set¶

Before we move on with training, it's important that we hold back some of the data for testing later. If we train on the same data we'll test with, we risk overfitting and not noticing!

In this section, we'll both split the training and testing set and gather prective features for each of the labeled observations.

In [4]:
train_set = rev_reverteds[:15000]
test_set = rev_reverteds[15000:]
print("training:", len(train_set))
print("testing:", len(test_set))

training: 15000
testing: 4868

OK. In order to train the machine learning model, we'll need to give it a source of signal. This is where "features" come into play. A feature represents a simple numerical statistic that we can extract from our observations that we think will be predictive of our outcome. Luckily, revscoring provides a whole suite of features that work well for damage detection. In this case, we'll be looking at features of the edit diff.

In [5]:
from revscoring.features import wikitext, revision_oriented, temporal
from revscoring.languages import english

features = [
    # Catches long key mashes like kkkkkkkkkkkk
    wikitext.revision.diff.longest_repeated_char_added,
    # Measures the size of the change in added words
    wikitext.revision.diff.words_added,
    # Measures the size of the change in removed words
    wikitext.revision.diff.words_removed,
    # Measures the proportional change in "badwords"
    english.badwords.revision.diff.match_prop_delta_sum,
    # Measures the proportional change in "informals"
    english.informals.revision.diff.match_prop_delta_sum,
    # Measures the proportional change meaningful words
    english.stopwords.revision.diff.non_stopword_prop_delta_sum,
    # Is the user anonymous
    revision_oriented.revision.user.is_anon,
    # Is the user a bot or a sysop
    revision_oriented.revision.user.in_group({'bot', 'sysop'}),
    # How long ago did the user register?
    temporal.revision.user.seconds_since_registration
]

Now, we'll need to turn to revscorings feature extractor to help us get us feature values for each revision.

In [9]:
from revscoring.extractors import api
api_extractor = api.Extractor(session)

print("https://en.wikipedia.org/wiki/?diff={0}".format(695071713))
print(list(api_extractor.extract(695071713, features)))
print("https://en.wikipedia.org/wiki/?diff={0}".format(667375206))
print(list(api_extractor.extract(667375206, features)))

https://en.wikipedia.org/wiki/?diff=695071713
[1, 0, 10974, -1.0, -2.5476190476190474, -1477.9699604325438, True, False, 313948852]
https://en.wikipedia.org/wiki/?diff=667375206
[1, 1, 1, 0.0, 0.0, 0.33333333333333337, False, False, 9844289]
In [10]:
# Now for the whole set!
training_features_reverted = []
for rev_id, reverted in train_set[:20]:
    try:
        feature_values = list(api_extractor.extract(rev_id, features))
    except RuntimeError as e:
        sys.stderr.write(str(e))
        continue
    
    sys.stderr.write(".")
    training_features_reverted.append((rev_id, feature_values, reverted))
    

....................

Eeek! Again this takes too long, so again, I uploaded a dataset with features already extracted @ ../datasets/demo/enwiki.features_reverted.training.20k_2015.tsv.bz2

In [11]:
from revscoring.utilities.util import read_observations
training_features_reverted_f = !bzcat ../datasets/demo/enwiki.features_reverted.training.20k_2015.tsv.bz2 | cut -f2-
training_features_reverted = list(read_observations(training_features_reverted_f, features, lambda v: v=="True"))
len(training_features_reverted)
Out[11]:
14979

Part 3: Training the model¶

Now that we have a set of features extracted for our training set, it's time to train a model. revscoring provides a set of different classifier algorithms. From past experience, I know a gradient boosting classifier works well, so we'll use that.

In [12]:
from revscoring.scorer_models import GradientBoosting
is_reverted = GradientBoosting(features, version="live demo!", 
                               learning_rate=0.01, max_features="log2", 
                               n_estimators=700, max_depth=5,
                               balanced_sample_weight=True, scale=True, center=True)

is_reverted.train(training_features_reverted)
Out[12]:
{'seconds_elapsed': 13.163022756576538}

We now have a trained model that we can play around with. Let's try a few edits from our test set.

In [13]:
reverted_obs = [rev_id for rev_id, reverted in test_set if reverted]
non_reverted_obs = [rev_id for rev_id, reverted in test_set if not reverted]

for rev_id in reverted_obs[:10]:
    feature_values = list(api_extractor.extract(rev_id, features))
    score = is_reverted.score(feature_values)
    print(True, "https://en.wikipedia.org/wiki/?diff=" + str(rev_id), 
          score['prediction'], round(score['probability'][True], 2))

for rev_id in non_reverted_obs[:10]:
    feature_values = list(api_extractor.extract(rev_id, features))
    score = is_reverted.score(feature_values)
    print(False, "https://en.wikipedia.org/wiki/?diff=" + str(rev_id), 
          score['prediction'], round(score['probability'][True], 2))

True https://en.wikipedia.org/wiki/?diff=699665317 True 0.82
True https://en.wikipedia.org/wiki/?diff=683832871 True 0.81
True https://en.wikipedia.org/wiki/?diff=653913156 True 0.72
True https://en.wikipedia.org/wiki/?diff=654545786 True 0.78
True https://en.wikipedia.org/wiki/?diff=670608733 True 0.77
True https://en.wikipedia.org/wiki/?diff=689399141 True 0.7
True https://en.wikipedia.org/wiki/?diff=662365029 True 0.92
True https://en.wikipedia.org/wiki/?diff=656782076 True 0.86
True https://en.wikipedia.org/wiki/?diff=698954388 True 0.86
True https://en.wikipedia.org/wiki/?diff=645603577 True 0.66
False https://en.wikipedia.org/wiki/?diff=687073859 False 0.38
False https://en.wikipedia.org/wiki/?diff=665341163 False 0.16
False https://en.wikipedia.org/wiki/?diff=654524549 False 0.08
False https://en.wikipedia.org/wiki/?diff=682425664 False 0.07
False https://en.wikipedia.org/wiki/?diff=674780271 False 0.24
False https://en.wikipedia.org/wiki/?diff=684793059 False 0.08
False https://en.wikipedia.org/wiki/?diff=655583788 True 0.7
False https://en.wikipedia.org/wiki/?diff=700003789 False 0.23
False https://en.wikipedia.org/wiki/?diff=659306547 False 0.08
False https://en.wikipedia.org/wiki/?diff=662149200 False 0.17

Part 4: Testing the model¶

So, the above analysis can help give us a sense for whether the model is working or not, but it's hard to standardize between models. So, we can apply some metrics that are specially crafted for machine learning models.

But first, I'll need to load the pre-generated feature values.

In [14]:
testing_features_reverted_f = !bzcat ../datasets/demo/enwiki.features_reverted.testing.20k_2015.tsv.bz2 | cut -f2-
testing_features_reverted = list(read_observations(testing_features_reverted_f, features, lambda v: v=="True"))
len(testing_features_reverted)
Out[14]:
4862
  • Accuracy -- The proportion of correct predictions
  • Precision -- The proportion of correct positive predictions
  • Recall -- The proportion of positive examples predicted as positive
  • Filter rate at 90% recall -- The proportion of observations that can be ignored while still catching 90% of "reverted" edits.

We'll use revscoring statistics to measure these against the test set.

In [16]:
from revscoring.scorer_models.test_statistics import (accuracy, precision, recall, 
                                                      filter_rate_at_recall)

is_reverted.test(testing_features_reverted, 
                 test_statistics=[accuracy(), precision(), recall(), 
                                  filter_rate_at_recall(0.90)])

print(is_reverted.format_info())

ScikitLearnClassifier
 - type: GradientBoosting
 - params: max_depth=5, random_state=null, loss="deviance", center=true, balanced_sample_weight=true, scale=true, min_samples_split=2, init=null, n_estimators=700, max_features="log2", balanced_sample=false, min_weight_fraction_leaf=0.0, verbose=0, subsample=1.0, presort="auto", warm_start=false, learning_rate=0.01, min_samples_leaf=1, max_leaf_nodes=null
 - version: live demo!
 - trained: 2016-04-06T11:46:35.377272

Accuracy: 0.812
Precision: 0.201
Recall: 0.82
Filter rate @ 0.9 recall: threshold=0.231, filter_rate=0.631, recall=0.902

Bonus round! Let's listen to Wikipedia's vandalism!¶

So we don't have the most powerful damage detection classifier, but then again, we're only including 9 features. Usually we run with ~60 features and get to much higher levels of fitness. but this model is still useful and it should help us detect the most aggregious vandalism in Wikipedia. In order to listen to Wikipedia, we'll need to connect to RCStream -- the same live feed that powers listen to Wikipedia.

In [18]:
import socketIO_client

class WikiNamespace(socketIO_client.BaseNamespace):
    def on_change(self, change):
        if change['type'] not in ('new', 'edit'):
            return
        
        rev_id = change['revision']['new']
        feature_values = list(api_extractor.extract(rev_id, features))
        score = is_reverted.score(feature_values)
        if score['prediction']:
            print("!!!Please review", "https://en.wikipedia.org/wiki/?diff=" + str(rev_id), 
                  round(score['probability'][True], 2), flush=True)
        else:
            print("Good edit", "https://en.wikipedia.org/wiki/?diff=" + str(rev_id),
                  round(score['probability'][True], 2), flush=True)

    def on_connect(self):
        self.emit('subscribe', 'en.wikipedia.org')


socketIO = socketIO_client.SocketIO('stream.wikimedia.org', 80)
socketIO.define(WikiNamespace, '/rc')

socketIO.wait(120)

WARNING:socketIO_client:stream.wikimedia.org:80/socket.io/1: [packet error] unhandled namespace path ()

Good edit https://en.wikipedia.org/wiki/?diff=713932732 0.12
Good edit https://en.wikipedia.org/wiki/?diff=713932733 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932734 0.08
Good edit https://en.wikipedia.org/wiki/?diff=713932735 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932736 0.09
Good edit https://en.wikipedia.org/wiki/?diff=713932737 0.07
!!!Please review https://en.wikipedia.org/wiki/?diff=713932738 0.5
Good edit https://en.wikipedia.org/wiki/?diff=713932739 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932740 0.17
Good edit https://en.wikipedia.org/wiki/?diff=713932741 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932742 0.34
!!!Please review https://en.wikipedia.org/wiki/?diff=713932743 0.75
Good edit https://en.wikipedia.org/wiki/?diff=713932744 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932745 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932746 0.06
Good edit https://en.wikipedia.org/wiki/?diff=713932747 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932748 0.25
Good edit https://en.wikipedia.org/wiki/?diff=713932749 0.35
!!!Please review https://en.wikipedia.org/wiki/?diff=713932751 0.55
Good edit https://en.wikipedia.org/wiki/?diff=713932753 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932754 0.11
Good edit https://en.wikipedia.org/wiki/?diff=713932755 0.1
Good edit https://en.wikipedia.org/wiki/?diff=713932752 0.1
!!!Please review https://en.wikipedia.org/wiki/?diff=713932757 0.75
Good edit https://en.wikipedia.org/wiki/?diff=713932756 0.33
Good edit https://en.wikipedia.org/wiki/?diff=713932758 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932760 0.37
Good edit https://en.wikipedia.org/wiki/?diff=713932761 0.07
Good edit https://en.wikipedia.org/wiki/?diff=713932750 0.24
Good edit https://en.wikipedia.org/wiki/?diff=713932759 0.21
Good edit https://en.wikipedia.org/wiki/?diff=713932762 0.13
Good edit https://en.wikipedia.org/wiki/?diff=713932763 0.13
Good edit https://en.wikipedia.org/wiki/?diff=713932765 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932766 0.28
!!!Please review https://en.wikipedia.org/wiki/?diff=713932764 0.51
Good edit https://en.wikipedia.org/wiki/?diff=713932768 0.04
Good edit https://en.wikipedia.org/wiki/?diff=713932767 0.17
!!!Please review https://en.wikipedia.org/wiki/?diff=713932769 0.7
Good edit https://en.wikipedia.org/wiki/?diff=713932770 0.16
Good edit https://en.wikipedia.org/wiki/?diff=713932771 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932772 0.1
Good edit https://en.wikipedia.org/wiki/?diff=713932773 0.11
Good edit https://en.wikipedia.org/wiki/?diff=713932774 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932775 0.3
Good edit https://en.wikipedia.org/wiki/?diff=713932776 0.27
Good edit https://en.wikipedia.org/wiki/?diff=713932778 0.18
!!!Please review https://en.wikipedia.org/wiki/?diff=713932779 0.76
Good edit https://en.wikipedia.org/wiki/?diff=713932780 0.32
!!!Please review https://en.wikipedia.org/wiki/?diff=713932777 0.54
Good edit https://en.wikipedia.org/wiki/?diff=713932781 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932784 0.13
Good edit https://en.wikipedia.org/wiki/?diff=713932782 0.08
Good edit https://en.wikipedia.org/wiki/?diff=713932785 0.16
Good edit https://en.wikipedia.org/wiki/?diff=713932783 0.04

WARNING:socketIO_client:stream.wikimedia.org:80/socket.io/1: [packet error] unhandled namespace path ()

Good edit https://en.wikipedia.org/wiki/?diff=713932786 0.15
!!!Please review https://en.wikipedia.org/wiki/?diff=713932787 0.52
!!!Please review https://en.wikipedia.org/wiki/?diff=713932788 0.81
Good edit https://en.wikipedia.org/wiki/?diff=713932789 0.04
!!!Please review https://en.wikipedia.org/wiki/?diff=713932790 0.75
Good edit https://en.wikipedia.org/wiki/?diff=713932791 0.16
Good edit https://en.wikipedia.org/wiki/?diff=713932794 0.1
Good edit https://en.wikipedia.org/wiki/?diff=713932793 0.35
Good edit https://en.wikipedia.org/wiki/?diff=713932792 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932795 0.16
!!!Please review https://en.wikipedia.org/wiki/?diff=713932797 0.69
!!!Please review https://en.wikipedia.org/wiki/?diff=713932798 0.75
Good edit https://en.wikipedia.org/wiki/?diff=713932796 0.25
Good edit https://en.wikipedia.org/wiki/?diff=713932799 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932800 0.12
Good edit https://en.wikipedia.org/wiki/?diff=713932802 0.12
Good edit https://en.wikipedia.org/wiki/?diff=713932803 0.17
Good edit https://en.wikipedia.org/wiki/?diff=713932806 0.02
Good edit https://en.wikipedia.org/wiki/?diff=713932804 0.39
Good edit https://en.wikipedia.org/wiki/?diff=713932807 0.22
!!!Please review https://en.wikipedia.org/wiki/?diff=713932808 0.65
Good edit https://en.wikipedia.org/wiki/?diff=713932809 0.21
Good edit https://en.wikipedia.org/wiki/?diff=713932801 0.07
Good edit https://en.wikipedia.org/wiki/?diff=713932810 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932811 0.05
Good edit https://en.wikipedia.org/wiki/?diff=713932812 0.08
Good edit https://en.wikipedia.org/wiki/?diff=713932814 0.1
Good edit https://en.wikipedia.org/wiki/?diff=713932813 0.07
Good edit https://en.wikipedia.org/wiki/?diff=713932815 0.38
Good edit https://en.wikipedia.org/wiki/?diff=713932816 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932817 0.11
Good edit https://en.wikipedia.org/wiki/?diff=713932818 0.07
Good edit https://en.wikipedia.org/wiki/?diff=713932819 0.15
!!!Please review https://en.wikipedia.org/wiki/?diff=713932820 0.55
Good edit https://en.wikipedia.org/wiki/?diff=713932821 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932822 0.13
Good edit https://en.wikipedia.org/wiki/?diff=713932824 0.14
!!!Please review https://en.wikipedia.org/wiki/?diff=713932823 0.66
Good edit https://en.wikipedia.org/wiki/?diff=713932825 0.06
Good edit https://en.wikipedia.org/wiki/?diff=713932826 0.22
!!!Please review https://en.wikipedia.org/wiki/?diff=713932827 0.56
Good edit https://en.wikipedia.org/wiki/?diff=713932828 0.18
Good edit https://en.wikipedia.org/wiki/?diff=713932829 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932830 0.24
Good edit https://en.wikipedia.org/wiki/?diff=713932833 0.15
!!!Please review https://en.wikipedia.org/wiki/?diff=713932831 0.66
Good edit https://en.wikipedia.org/wiki/?diff=713932834 0.12
Good edit https://en.wikipedia.org/wiki/?diff=713932832 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932835 0.08
Good edit https://en.wikipedia.org/wiki/?diff=713932836 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932837 0.04
Good edit https://en.wikipedia.org/wiki/?diff=713932839 0.42
Good edit https://en.wikipedia.org/wiki/?diff=713932840 0.06
Good edit https://en.wikipedia.org/wiki/?diff=713932841 0.11
Good edit https://en.wikipedia.org/wiki/?diff=713932838 0.1
!!!Please review https://en.wikipedia.org/wiki/?diff=713932844 0.51
Good edit https://en.wikipedia.org/wiki/?diff=713932845 0.11
Good edit https://en.wikipedia.org/wiki/?diff=713932842 0.15

WARNING:socketIO_client:stream.wikimedia.org:80/socket.io/1: [packet error] unhandled namespace path ()

Good edit https://en.wikipedia.org/wiki/?diff=713932843 0.12
Good edit https://en.wikipedia.org/wiki/?diff=713932846 0.34
!!!Please review https://en.wikipedia.org/wiki/?diff=713932847 0.73
Good edit https://en.wikipedia.org/wiki/?diff=713932848 0.16
Good edit https://en.wikipedia.org/wiki/?diff=713932849 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932850 0.07
Good edit https://en.wikipedia.org/wiki/?diff=713932851 0.15
!!!Please review https://en.wikipedia.org/wiki/?diff=713932852 0.75
Good edit https://en.wikipedia.org/wiki/?diff=713932854 0.13
!!!Please review https://en.wikipedia.org/wiki/?diff=713932853 0.86
Good edit https://en.wikipedia.org/wiki/?diff=713932855 0.16
Good edit https://en.wikipedia.org/wiki/?diff=713932856 0.22
Good edit https://en.wikipedia.org/wiki/?diff=713932857 0.24
!!!Please review https://en.wikipedia.org/wiki/?diff=713932859 0.63
Good edit https://en.wikipedia.org/wiki/?diff=713932858 0.36
Good edit https://en.wikipedia.org/wiki/?diff=713932860 0.08
Good edit https://en.wikipedia.org/wiki/?diff=713932861 0.16
Good edit https://en.wikipedia.org/wiki/?diff=713932862 0.16
Good edit https://en.wikipedia.org/wiki/?diff=713932863 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932864 0.26
Good edit https://en.wikipedia.org/wiki/?diff=713932865 0.22
Good edit https://en.wikipedia.org/wiki/?diff=713932866 0.2
Good edit https://en.wikipedia.org/wiki/?diff=713932867 0.1
Good edit https://en.wikipedia.org/wiki/?diff=713932868 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932869 0.06
Good edit https://en.wikipedia.org/wiki/?diff=713932870 0.46
Good edit https://en.wikipedia.org/wiki/?diff=713932872 0.15
!!!Please review https://en.wikipedia.org/wiki/?diff=713932873 0.8
Good edit https://en.wikipedia.org/wiki/?diff=713932875 0.04
Good edit https://en.wikipedia.org/wiki/?diff=713932874 0.08
Good edit https://en.wikipedia.org/wiki/?diff=713932871 0.05
Good edit https://en.wikipedia.org/wiki/?diff=713932877 0.11
Good edit https://en.wikipedia.org/wiki/?diff=713932876 0.18
Good edit https://en.wikipedia.org/wiki/?diff=713932878 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932879 0.07
Good edit https://en.wikipedia.org/wiki/?diff=713932881 0.48
Good edit https://en.wikipedia.org/wiki/?diff=713932880 0.07
Good edit https://en.wikipedia.org/wiki/?diff=713932882 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932883 0.49
Good edit https://en.wikipedia.org/wiki/?diff=713932884 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932885 0.03
Good edit https://en.wikipedia.org/wiki/?diff=713932886 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932887 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932888 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932889 0.18
!!!Please review https://en.wikipedia.org/wiki/?diff=713932890 0.53
Good edit https://en.wikipedia.org/wiki/?diff=713932891 0.26
Good edit https://en.wikipedia.org/wiki/?diff=713932892 0.25
Good edit https://en.wikipedia.org/wiki/?diff=713932893 0.03

WARNING:socketIO_client:stream.wikimedia.org:80/socket.io/1: [packet error] unhandled namespace path ()

Good edit https://en.wikipedia.org/wiki/?diff=713932894 0.23
Good edit https://en.wikipedia.org/wiki/?diff=713932895 0.19
Good edit https://en.wikipedia.org/wiki/?diff=713932896 0.15
Good edit https://en.wikipedia.org/wiki/?diff=713932897 0.16
Good edit https://en.wikipedia.org/wiki/?diff=713932898 0.34
Good edit https://en.wikipedia.org/wiki/?diff=713932900 0.19
Good edit https://en.wikipedia.org/wiki/?diff=713932901 0.29
Good edit https://en.wikipedia.org/wiki/?diff=713932903 0.12
Good edit https://en.wikipedia.org/wiki/?diff=713932902 0.14
Good edit https://en.wikipedia.org/wiki/?diff=713932904 0.24
Good edit https://en.wikipedia.org/wiki/?diff=713932905 0.11
Good edit https://en.wikipedia.org/wiki/?diff=713932899 0.3
Good edit https://en.wikipedia.org/wiki/?diff=713932906 0.12
In [ ]: