Journal Names¶
Questions in the context of fuzzy matching.
- How many journal names appear more than once?
- What is the average length of the duplicated names vs the unique names?
Input file is a single larger JSON, mapping names to issns.
{
"Acta Orientalia.": [
"0001-6438"
],
"Acta Orientalia (København)": [
"0001-6438"
],
..
In [4]:
import json
import pandas as pd
In [5]:
with open("../data/name_to_issn.json") as f:
mapping = json.load(f)
We have about 3M keys.
In [7]:
len(mapping)
Out[7]:
In [21]:
df = pd.DataFrame(((k, len(v)) for k, v in mapping.items()), columns=["name", "issn_count"])
In [25]:
len(df)
Out[25]:
In [26]:
df.head()
Out[26]:
In [31]:
unique_name = df[df.issn_count == 1]
In [32]:
repeated_names = df[df.issn_count > 1]
In [34]:
len(repeated_names)
Out[34]:
In [33]:
len(repeated_names) / len(df)
Out[33]:
About 6% (or 194241) names are repeated.
In [35]:
repeated_names.describe()
Out[35]:
Which name is shared by over 8000 ISSN?
In [40]:
repeated_names.iloc[repeated_names.issn_count.argmax()] # Annual report.
Out[40]:
It is the "Annual report."
In [42]:
mapping["Annual report."][:10]
Out[42]:
On average a repeated name will point to 3 ISSN. About 24k names point to more than 3 ISSN.
In [45]:
len(repeated_names[repeated_names.issn_count > 3])
Out[45]:
In [49]:
repeated_names[repeated_names.issn_count > 3].sample(n=10)
Out[49]:
In [50]:
mapping["Philosophica."]
Out[50]:
In [61]:
repeated_names[repeated_names.issn_count > 3].issn_count.hist(bins=20)
Out[61]:
In [60]:
repeated_names[(repeated_names.issn_count > 3) & (repeated_names.issn_count < 50)].issn_count.hist(bins=10)
Out[60]:
In [62]:
repeated_names[(repeated_names.issn_count > 3) & (repeated_names.issn_count < 20)].issn_count.hist(bins=10)
Out[62]:
In [64]:
repeated_names[(repeated_names.issn_count > 3) & (repeated_names.issn_count < 8)].issn_count.hist()
Out[64]:
In [70]:
repeated_names[repeated_names.issn_count > 1000].issn_count.hist(bins=10)
Out[70]:
In [71]:
repeated_names[repeated_names.issn_count > 1000]
Out[71]:
In [72]:
repeated_names[repeated_names.issn_count > 500]
Out[72]:
In [75]:
repeated_names[repeated_names.issn_count > 200]
Out[75]:
In [76]:
repeated_names[repeated_names.issn_count > 100]
Out[76]:
In [82]:
repeated_names
Out[82]:
If a name matches a repeated name exactly or fuzzy matches to a repeated name and there is not other information available, the match status must be ambigious.
In [ ]: