blob: 7c6468dc31d8a84ba32cfd70ea03aa02426b2470 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
|
# fuzzycat (wip)
Fuzzy matching publications for [fatcat](https://fatcat.wiki).
* [fuzzycat](https://pypi.org/project/fuzzycat/)
## Motivation
Most of the results on sites like [Google
Scholar](https://scholar.google.com/scholar?q=fuzzy+matching) group
publications into clusters. Each cluster represents one publication, abstracted
from its concrete representation as a link to a PDF.
We call the abstract publication *work* and the concrete instance a *release*.
The goal is to group releases under works and to implement a versions feature.
This repository contains both generic code for matching as well as fatcat
specific code using the fatcat openapi client.
## Datasets
* release and container metadata from: [https://archive.org/details/fatcat_bulk_exports_2020-08-05](https://archive.org/details/fatcat_bulk_exports_2020-08-05).
* issn journal level data, via [issnlister](https://github.com/miku/issnlister)
* abbreviation lists
## Matching approaches
![](static/approach.png)
## Performance data point
Candidate generation via elasticsearch, 40 parallel queries, sustained speed at
about 17857 queries per hour, that is around 5 queries/s.
```
$ time cat ~/data/researchgate/x04 | \
parallel -j40 --pipe -N 1 ./fatcatx_rg_unmatched.py - \
> ~/data/researchgate/x04_results.ndj
...
real 3409m16.442s
user 29177m5.516s
sys 4927m3.277s
```
|