This notebook demonstrates the basic contruction of a vandalism classification system using the revscoring library that we have developed specifically for classification models of MediaWiki stuff.
The basic process that we'll follow is this:
And then we'll have some fun applying the model to some edits using RCStream. The following diagram gives a good sense for the whole process of training and evaluating a model.
Regretfully, running SQL queries isn't something we can do directly from the notebook yet. So, we'll use Quarry to generate a nice random sample of edits. 20,000 observations should do just fine. Here's the query I want to run:
USE enwiki_p;
SELECT rev_id
FROM revision
WHERE rev_timestamp BETWEEN "20150201" AND "20160201"
ORDER BY RAND()
LIMIT 20000;
See http://quarry.wmflabs.org/query/7530. By clicking around the UI, I can see that this URL will download my tab-separated file: http://quarry.wmflabs.org/run/65415/output/0/tsv?download=true
# Magical ipython notebook stuff puts the result of this command into a variable
revids_f = !wget http://quarry.wmflabs.org/run/65415/output/0/tsv?download=true -qO-
revids = [int(line) for line in revids_f[1:]]
len(revids)
20000
OK. Now that we have a set of revisions, we need to label them. In this case, we're going to label them as reverted/not. We want to exclude a few different types of reverts -- e.g. when a user reverts themself or when an edit is reverted back to by someone else. For this, we'll use the mwreverts and mwapi libraries.
import sys, traceback
import mwreverts.api
import mwapi
# We'll use the mwreverts API check. In order to do that, we need an API session
session = mwapi.Session("https://en.wikipedia.org",
user_agent="Revert detection demo <ahalfaker@wikimedia.org>")
# For each revision, find out if it was "reverted" and label it so.
rev_reverteds = []
for rev_id in revids[:20]: # TODO: Limiting to the first 20!!!!
try:
_, reverted, reverted_to = mwreverts.api.check(
session, rev_id, radius=5, # most reverts within 5 edits
window=48*60*60, # 2 days
rvprop={'user', 'ids'}) # Some properties we'll make use of
except RuntimeError as e:
sys.stderr.write(str(e))
continue
if reverted is not None:
reverted_doc = [r for r in reverted.reverteds
if r['revid'] == rev_id][0]
# self-reverts
self_revert = \
reverted_doc['user'] == reverted.reverting['user']
# revisions that are reverted back to by others
reverted_back_to = \
reverted_to is not None and \
reverted_doc['user'] != \
reverted_to.reverting['user']
# If we are reverted, not by self or reverted back to by someone else,
# then, let's assume it was damaging.
damaging_reverted = !(self_revert or reverted_back_to)
else:
damaging_reverted = False
rev_reverteds.append((rev_id, damaging_reverted))
sys.stderr.write("r" if damaging_reverted else ".")
...............r....
Eeek! This takes too long. You get the idea. So, I uploaded dataset that has already been labeled here @ ../datasets/demo/enwiki.rev_reverted.20k_2015.tsv.bz2
rev_reverteds_f = !bzcat ../datasets/demo/enwiki.rev_reverted.20k_2015.tsv.bz2
rev_reverteds = [line.strip().split("\t") for line in rev_reverteds_f[1:]]
rev_reverteds = [(int(rev_id), reverted == "True") for rev_id, reverted in rev_reverteds]
len(rev_reverteds)
19868
OK. It looks like we got an error when trying to extract the reverted status of ~132 edits, which is an acceptable loss. Now just to make sure we haven't gone crazy, let's check some of the reverted edits:
OK. Looks like we are doing pretty good. :)
Before we move on with training, it's important that we hold back some of the data for testing later. If we train on the same data we'll test with, we risk overfitting and not noticing!
In this section, we'll both split the training and testing set and gather prective features for each of the labeled observations.
train_set = rev_reverteds[:15000]
test_set = rev_reverteds[15000:]
print("training:", len(train_set))
print("testing:", len(test_set))
training: 15000 testing: 4868
OK. In order to train the machine learning model, we'll need to give it a source of signal. This is where "features" come into play. A feature represents a simple numerical statistic that we can extract from our observations that we think will be predictive of our outcome. Luckily, revscoring
provides a whole suite of features that work well for damage detection. In this case, we'll be looking at features of the edit diff.
from revscoring.features import wikitext, revision_oriented, temporal
from revscoring.languages import english
features = [
# Catches long key mashes like kkkkkkkkkkkk
wikitext.revision.diff.longest_repeated_char_added,
# Measures the size of the change in added words
wikitext.revision.diff.words_added,
# Measures the size of the change in removed words
wikitext.revision.diff.words_removed,
# Measures the proportional change in "badwords"
english.badwords.revision.diff.match_prop_delta_sum,
# Measures the proportional change in "informals"
english.informals.revision.diff.match_prop_delta_sum,
# Measures the proportional change meaningful words
english.stopwords.revision.diff.non_stopword_prop_delta_sum,
# Is the user anonymous
revision_oriented.revision.user.is_anon,
# Is the user a bot or a sysop
revision_oriented.revision.user.in_group({'bot', 'sysop'}),
# How long ago did the user register?
temporal.revision.user.seconds_since_registration
]
Now, we'll need to turn to revscoring
s feature extractor to help us get us feature values for each revision.
from revscoring.extractors import api
api_extractor = api.Extractor(session)
print("https://en.wikipedia.org/wiki/?diff={0}".format(695071713))
print(list(api_extractor.extract(695071713, features)))
print("https://en.wikipedia.org/wiki/?diff={0}".format(667375206))
print(list(api_extractor.extract(667375206, features)))
https://en.wikipedia.org/wiki/?diff=695071713 [1, 0, 10974, -1.0, -2.5476190476190474, -1477.9699604325438, True, False, 313948852] https://en.wikipedia.org/wiki/?diff=667375206 [1, 1, 1, 0.0, 0.0, 0.33333333333333337, False, False, 9844289]
# Now for the whole set!
training_features_reverted = []
for rev_id, reverted in train_set[:20]:
try:
feature_values = list(api_extractor.extract(rev_id, features))
except RuntimeError as e:
sys.stderr.write(str(e))
continue
sys.stderr.write(".")
training_features_reverted.append((rev_id, feature_values, reverted))
....................
Eeek! Again this takes too long, so again, I uploaded a dataset with features already extracted @ ../datasets/demo/enwiki.features_reverted.training.20k_2015.tsv.bz2
from revscoring.utilities.util import read_observations
training_features_reverted_f = !bzcat ../datasets/demo/enwiki.features_reverted.training.20k_2015.tsv.bz2 | cut -f2-
training_features_reverted = list(read_observations(training_features_reverted_f, features, lambda v: v=="True"))
len(training_features_reverted)
14979
Now that we have a set of features extracted for our training set, it's time to train a model. revscoring
provides a set of different classifier algorithms. From past experience, I know a gradient boosting classifier works well, so we'll use that.
from revscoring.scorer_models import GradientBoosting
is_reverted = GradientBoosting(features, version="live demo!",
learning_rate=0.01, max_features="log2",
n_estimators=700, max_depth=5,
balanced_sample_weight=True, scale=True, center=True)
is_reverted.train(training_features_reverted)
{'seconds_elapsed': 13.163022756576538}
We now have a trained model that we can play around with. Let's try a few edits from our test set.
reverted_obs = [rev_id for rev_id, reverted in test_set if reverted]
non_reverted_obs = [rev_id for rev_id, reverted in test_set if not reverted]
for rev_id in reverted_obs[:10]:
feature_values = list(api_extractor.extract(rev_id, features))
score = is_reverted.score(feature_values)
print(True, "https://en.wikipedia.org/wiki/?diff=" + str(rev_id),
score['prediction'], round(score['probability'][True], 2))
for rev_id in non_reverted_obs[:10]:
feature_values = list(api_extractor.extract(rev_id, features))
score = is_reverted.score(feature_values)
print(False, "https://en.wikipedia.org/wiki/?diff=" + str(rev_id),
score['prediction'], round(score['probability'][True], 2))
True https://en.wikipedia.org/wiki/?diff=699665317 True 0.82 True https://en.wikipedia.org/wiki/?diff=683832871 True 0.81 True https://en.wikipedia.org/wiki/?diff=653913156 True 0.72 True https://en.wikipedia.org/wiki/?diff=654545786 True 0.78 True https://en.wikipedia.org/wiki/?diff=670608733 True 0.77 True https://en.wikipedia.org/wiki/?diff=689399141 True 0.7 True https://en.wikipedia.org/wiki/?diff=662365029 True 0.92 True https://en.wikipedia.org/wiki/?diff=656782076 True 0.86 True https://en.wikipedia.org/wiki/?diff=698954388 True 0.86 True https://en.wikipedia.org/wiki/?diff=645603577 True 0.66 False https://en.wikipedia.org/wiki/?diff=687073859 False 0.38 False https://en.wikipedia.org/wiki/?diff=665341163 False 0.16 False https://en.wikipedia.org/wiki/?diff=654524549 False 0.08 False https://en.wikipedia.org/wiki/?diff=682425664 False 0.07 False https://en.wikipedia.org/wiki/?diff=674780271 False 0.24 False https://en.wikipedia.org/wiki/?diff=684793059 False 0.08 False https://en.wikipedia.org/wiki/?diff=655583788 True 0.7 False https://en.wikipedia.org/wiki/?diff=700003789 False 0.23 False https://en.wikipedia.org/wiki/?diff=659306547 False 0.08 False https://en.wikipedia.org/wiki/?diff=662149200 False 0.17
So, the above analysis can help give us a sense for whether the model is working or not, but it's hard to standardize between models. So, we can apply some metrics that are specially crafted for machine learning models.
But first, I'll need to load the pre-generated feature values.
testing_features_reverted_f = !bzcat ../datasets/demo/enwiki.features_reverted.testing.20k_2015.tsv.bz2 | cut -f2-
testing_features_reverted = list(read_observations(testing_features_reverted_f, features, lambda v: v=="True"))
len(testing_features_reverted)
4862
We'll use revscoring
statistics to measure these against the test set.
from revscoring.scorer_models.test_statistics import (accuracy, precision, recall,
filter_rate_at_recall)
is_reverted.test(testing_features_reverted,
test_statistics=[accuracy(), precision(), recall(),
filter_rate_at_recall(0.90)])
print(is_reverted.format_info())
ScikitLearnClassifier - type: GradientBoosting - params: max_depth=5, random_state=null, loss="deviance", center=true, balanced_sample_weight=true, scale=true, min_samples_split=2, init=null, n_estimators=700, max_features="log2", balanced_sample=false, min_weight_fraction_leaf=0.0, verbose=0, subsample=1.0, presort="auto", warm_start=false, learning_rate=0.01, min_samples_leaf=1, max_leaf_nodes=null - version: live demo! - trained: 2016-04-06T11:46:35.377272 Accuracy: 0.812 Precision: 0.201 Recall: 0.82 Filter rate @ 0.9 recall: threshold=0.231, filter_rate=0.631, recall=0.902
So we don't have the most powerful damage detection classifier, but then again, we're only including 9 features. Usually we run with ~60 features and get to much higher levels of fitness. but this model is still useful and it should help us detect the most aggregious vandalism in Wikipedia. In order to listen to Wikipedia, we'll need to connect to RCStream -- the same live feed that powers listen to Wikipedia.
import socketIO_client
class WikiNamespace(socketIO_client.BaseNamespace):
def on_change(self, change):
if change['type'] not in ('new', 'edit'):
return
rev_id = change['revision']['new']
feature_values = list(api_extractor.extract(rev_id, features))
score = is_reverted.score(feature_values)
if score['prediction']:
print("!!!Please review", "https://en.wikipedia.org/wiki/?diff=" + str(rev_id),
round(score['probability'][True], 2), flush=True)
else:
print("Good edit", "https://en.wikipedia.org/wiki/?diff=" + str(rev_id),
round(score['probability'][True], 2), flush=True)
def on_connect(self):
self.emit('subscribe', 'en.wikipedia.org')
socketIO = socketIO_client.SocketIO('stream.wikimedia.org', 80)
socketIO.define(WikiNamespace, '/rc')
socketIO.wait(120)
WARNING:socketIO_client:stream.wikimedia.org:80/socket.io/1: [packet error] unhandled namespace path ()
Good edit https://en.wikipedia.org/wiki/?diff=713932732 0.12 Good edit https://en.wikipedia.org/wiki/?diff=713932733 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932734 0.08 Good edit https://en.wikipedia.org/wiki/?diff=713932735 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932736 0.09 Good edit https://en.wikipedia.org/wiki/?diff=713932737 0.07 !!!Please review https://en.wikipedia.org/wiki/?diff=713932738 0.5 Good edit https://en.wikipedia.org/wiki/?diff=713932739 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932740 0.17 Good edit https://en.wikipedia.org/wiki/?diff=713932741 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932742 0.34 !!!Please review https://en.wikipedia.org/wiki/?diff=713932743 0.75 Good edit https://en.wikipedia.org/wiki/?diff=713932744 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932745 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932746 0.06 Good edit https://en.wikipedia.org/wiki/?diff=713932747 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932748 0.25 Good edit https://en.wikipedia.org/wiki/?diff=713932749 0.35 !!!Please review https://en.wikipedia.org/wiki/?diff=713932751 0.55 Good edit https://en.wikipedia.org/wiki/?diff=713932753 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932754 0.11 Good edit https://en.wikipedia.org/wiki/?diff=713932755 0.1 Good edit https://en.wikipedia.org/wiki/?diff=713932752 0.1 !!!Please review https://en.wikipedia.org/wiki/?diff=713932757 0.75 Good edit https://en.wikipedia.org/wiki/?diff=713932756 0.33 Good edit https://en.wikipedia.org/wiki/?diff=713932758 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932760 0.37 Good edit https://en.wikipedia.org/wiki/?diff=713932761 0.07 Good edit https://en.wikipedia.org/wiki/?diff=713932750 0.24 Good edit https://en.wikipedia.org/wiki/?diff=713932759 0.21 Good edit https://en.wikipedia.org/wiki/?diff=713932762 0.13 Good edit https://en.wikipedia.org/wiki/?diff=713932763 0.13 Good edit https://en.wikipedia.org/wiki/?diff=713932765 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932766 0.28 !!!Please review https://en.wikipedia.org/wiki/?diff=713932764 0.51 Good edit https://en.wikipedia.org/wiki/?diff=713932768 0.04 Good edit https://en.wikipedia.org/wiki/?diff=713932767 0.17 !!!Please review https://en.wikipedia.org/wiki/?diff=713932769 0.7 Good edit https://en.wikipedia.org/wiki/?diff=713932770 0.16 Good edit https://en.wikipedia.org/wiki/?diff=713932771 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932772 0.1 Good edit https://en.wikipedia.org/wiki/?diff=713932773 0.11 Good edit https://en.wikipedia.org/wiki/?diff=713932774 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932775 0.3 Good edit https://en.wikipedia.org/wiki/?diff=713932776 0.27 Good edit https://en.wikipedia.org/wiki/?diff=713932778 0.18 !!!Please review https://en.wikipedia.org/wiki/?diff=713932779 0.76 Good edit https://en.wikipedia.org/wiki/?diff=713932780 0.32 !!!Please review https://en.wikipedia.org/wiki/?diff=713932777 0.54 Good edit https://en.wikipedia.org/wiki/?diff=713932781 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932784 0.13 Good edit https://en.wikipedia.org/wiki/?diff=713932782 0.08 Good edit https://en.wikipedia.org/wiki/?diff=713932785 0.16 Good edit https://en.wikipedia.org/wiki/?diff=713932783 0.04
WARNING:socketIO_client:stream.wikimedia.org:80/socket.io/1: [packet error] unhandled namespace path ()
Good edit https://en.wikipedia.org/wiki/?diff=713932786 0.15 !!!Please review https://en.wikipedia.org/wiki/?diff=713932787 0.52 !!!Please review https://en.wikipedia.org/wiki/?diff=713932788 0.81 Good edit https://en.wikipedia.org/wiki/?diff=713932789 0.04 !!!Please review https://en.wikipedia.org/wiki/?diff=713932790 0.75 Good edit https://en.wikipedia.org/wiki/?diff=713932791 0.16 Good edit https://en.wikipedia.org/wiki/?diff=713932794 0.1 Good edit https://en.wikipedia.org/wiki/?diff=713932793 0.35 Good edit https://en.wikipedia.org/wiki/?diff=713932792 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932795 0.16 !!!Please review https://en.wikipedia.org/wiki/?diff=713932797 0.69 !!!Please review https://en.wikipedia.org/wiki/?diff=713932798 0.75 Good edit https://en.wikipedia.org/wiki/?diff=713932796 0.25 Good edit https://en.wikipedia.org/wiki/?diff=713932799 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932800 0.12 Good edit https://en.wikipedia.org/wiki/?diff=713932802 0.12 Good edit https://en.wikipedia.org/wiki/?diff=713932803 0.17 Good edit https://en.wikipedia.org/wiki/?diff=713932806 0.02 Good edit https://en.wikipedia.org/wiki/?diff=713932804 0.39 Good edit https://en.wikipedia.org/wiki/?diff=713932807 0.22 !!!Please review https://en.wikipedia.org/wiki/?diff=713932808 0.65 Good edit https://en.wikipedia.org/wiki/?diff=713932809 0.21 Good edit https://en.wikipedia.org/wiki/?diff=713932801 0.07 Good edit https://en.wikipedia.org/wiki/?diff=713932810 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932811 0.05 Good edit https://en.wikipedia.org/wiki/?diff=713932812 0.08 Good edit https://en.wikipedia.org/wiki/?diff=713932814 0.1 Good edit https://en.wikipedia.org/wiki/?diff=713932813 0.07 Good edit https://en.wikipedia.org/wiki/?diff=713932815 0.38 Good edit https://en.wikipedia.org/wiki/?diff=713932816 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932817 0.11 Good edit https://en.wikipedia.org/wiki/?diff=713932818 0.07 Good edit https://en.wikipedia.org/wiki/?diff=713932819 0.15 !!!Please review https://en.wikipedia.org/wiki/?diff=713932820 0.55 Good edit https://en.wikipedia.org/wiki/?diff=713932821 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932822 0.13 Good edit https://en.wikipedia.org/wiki/?diff=713932824 0.14 !!!Please review https://en.wikipedia.org/wiki/?diff=713932823 0.66 Good edit https://en.wikipedia.org/wiki/?diff=713932825 0.06 Good edit https://en.wikipedia.org/wiki/?diff=713932826 0.22 !!!Please review https://en.wikipedia.org/wiki/?diff=713932827 0.56 Good edit https://en.wikipedia.org/wiki/?diff=713932828 0.18 Good edit https://en.wikipedia.org/wiki/?diff=713932829 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932830 0.24 Good edit https://en.wikipedia.org/wiki/?diff=713932833 0.15 !!!Please review https://en.wikipedia.org/wiki/?diff=713932831 0.66 Good edit https://en.wikipedia.org/wiki/?diff=713932834 0.12 Good edit https://en.wikipedia.org/wiki/?diff=713932832 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932835 0.08 Good edit https://en.wikipedia.org/wiki/?diff=713932836 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932837 0.04 Good edit https://en.wikipedia.org/wiki/?diff=713932839 0.42 Good edit https://en.wikipedia.org/wiki/?diff=713932840 0.06 Good edit https://en.wikipedia.org/wiki/?diff=713932841 0.11 Good edit https://en.wikipedia.org/wiki/?diff=713932838 0.1 !!!Please review https://en.wikipedia.org/wiki/?diff=713932844 0.51 Good edit https://en.wikipedia.org/wiki/?diff=713932845 0.11 Good edit https://en.wikipedia.org/wiki/?diff=713932842 0.15
WARNING:socketIO_client:stream.wikimedia.org:80/socket.io/1: [packet error] unhandled namespace path ()
Good edit https://en.wikipedia.org/wiki/?diff=713932843 0.12 Good edit https://en.wikipedia.org/wiki/?diff=713932846 0.34 !!!Please review https://en.wikipedia.org/wiki/?diff=713932847 0.73 Good edit https://en.wikipedia.org/wiki/?diff=713932848 0.16 Good edit https://en.wikipedia.org/wiki/?diff=713932849 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932850 0.07 Good edit https://en.wikipedia.org/wiki/?diff=713932851 0.15 !!!Please review https://en.wikipedia.org/wiki/?diff=713932852 0.75 Good edit https://en.wikipedia.org/wiki/?diff=713932854 0.13 !!!Please review https://en.wikipedia.org/wiki/?diff=713932853 0.86 Good edit https://en.wikipedia.org/wiki/?diff=713932855 0.16 Good edit https://en.wikipedia.org/wiki/?diff=713932856 0.22 Good edit https://en.wikipedia.org/wiki/?diff=713932857 0.24 !!!Please review https://en.wikipedia.org/wiki/?diff=713932859 0.63 Good edit https://en.wikipedia.org/wiki/?diff=713932858 0.36 Good edit https://en.wikipedia.org/wiki/?diff=713932860 0.08 Good edit https://en.wikipedia.org/wiki/?diff=713932861 0.16 Good edit https://en.wikipedia.org/wiki/?diff=713932862 0.16 Good edit https://en.wikipedia.org/wiki/?diff=713932863 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932864 0.26 Good edit https://en.wikipedia.org/wiki/?diff=713932865 0.22 Good edit https://en.wikipedia.org/wiki/?diff=713932866 0.2 Good edit https://en.wikipedia.org/wiki/?diff=713932867 0.1 Good edit https://en.wikipedia.org/wiki/?diff=713932868 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932869 0.06 Good edit https://en.wikipedia.org/wiki/?diff=713932870 0.46 Good edit https://en.wikipedia.org/wiki/?diff=713932872 0.15 !!!Please review https://en.wikipedia.org/wiki/?diff=713932873 0.8 Good edit https://en.wikipedia.org/wiki/?diff=713932875 0.04 Good edit https://en.wikipedia.org/wiki/?diff=713932874 0.08 Good edit https://en.wikipedia.org/wiki/?diff=713932871 0.05 Good edit https://en.wikipedia.org/wiki/?diff=713932877 0.11 Good edit https://en.wikipedia.org/wiki/?diff=713932876 0.18 Good edit https://en.wikipedia.org/wiki/?diff=713932878 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932879 0.07 Good edit https://en.wikipedia.org/wiki/?diff=713932881 0.48 Good edit https://en.wikipedia.org/wiki/?diff=713932880 0.07 Good edit https://en.wikipedia.org/wiki/?diff=713932882 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932883 0.49 Good edit https://en.wikipedia.org/wiki/?diff=713932884 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932885 0.03 Good edit https://en.wikipedia.org/wiki/?diff=713932886 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932887 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932888 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932889 0.18 !!!Please review https://en.wikipedia.org/wiki/?diff=713932890 0.53 Good edit https://en.wikipedia.org/wiki/?diff=713932891 0.26 Good edit https://en.wikipedia.org/wiki/?diff=713932892 0.25 Good edit https://en.wikipedia.org/wiki/?diff=713932893 0.03
WARNING:socketIO_client:stream.wikimedia.org:80/socket.io/1: [packet error] unhandled namespace path ()
Good edit https://en.wikipedia.org/wiki/?diff=713932894 0.23 Good edit https://en.wikipedia.org/wiki/?diff=713932895 0.19 Good edit https://en.wikipedia.org/wiki/?diff=713932896 0.15 Good edit https://en.wikipedia.org/wiki/?diff=713932897 0.16 Good edit https://en.wikipedia.org/wiki/?diff=713932898 0.34 Good edit https://en.wikipedia.org/wiki/?diff=713932900 0.19 Good edit https://en.wikipedia.org/wiki/?diff=713932901 0.29 Good edit https://en.wikipedia.org/wiki/?diff=713932903 0.12 Good edit https://en.wikipedia.org/wiki/?diff=713932902 0.14 Good edit https://en.wikipedia.org/wiki/?diff=713932904 0.24 Good edit https://en.wikipedia.org/wiki/?diff=713932905 0.11 Good edit https://en.wikipedia.org/wiki/?diff=713932899 0.3 Good edit https://en.wikipedia.org/wiki/?diff=713932906 0.12