fatcat/python/fatcat_tools/workers, branch bnewbold-rust-gen-v5[no description]https://git.bnewbold.net/fatcat/atom?h=bnewbold-rust-gen-v52020-04-17T23:29:40Zmore changelog ES fixes2020-04-17T23:29:40ZBryan Newboldbnewbold@robocracy.org2020-04-17T23:29:40Zurn:sha1:9e9f7f1da115458d87f8bcdc011b2843e2b31d3bES changelog worker: fixes for ident; fetch update from API if needed2020-04-17T22:32:20ZBryan Newboldbnewbold@robocracy.org2020-04-17T22:32:18Zurn:sha1:026e352f5d99652f088b6bcdc28d43106b8f52d2
The API fetch update may be needed for old changelog entries in the
kafka feed.
Merge branch 'martin-changelog-to-es' into 'master'2020-04-17T18:13:14Zbnewboldbnewbold@archive.org2020-04-17T18:13:14Zurn:sha1:963faf6cf6e7e5c6685ffe89e080134c7590957f
derive changelog worker from release worker
See merge request webgroup/fatcat!43derive changelog worker from release worker2020-04-17T12:43:31ZMartin Czyganmartin.czygan@gmail.com2020-04-17T12:30:57Zurn:sha1:89db8df9eef40b92454ed9bd64830ebe5b726b9a
Early versions of changelog entries may not have all the fields
required for the current transform.
changelog: limit types2020-04-16T18:54:20ZMartin Czyganmartin.czygan@gmail.com2020-04-16T18:54:20Zurn:sha1:e063d2f470951f1735ccca8c6ea4b37029a6fede
No partial docs (e.g. abstract), too generic components and entries, not
HTML blogs.
changelog: extend release_types considered documents2020-04-15T23:22:57ZMartin Czyganmartin.czygan@gmail.com2020-04-15T23:17:45Zurn:sha1:0071b77eb7fc20be4af1bbf9b6c0bfcb4e26816a
according to release_rev.release_type, we have 29 values:
fatcat_prod=# select release_type, count(release_type) from release_rev group by release_type;
release_type | count
-------------------+-----------
abstract | 2264
article | 6371076
article-journal | 101083841
article-newspaper | 17062
book | 1676941
chapter | 13914854
component | 58990
dataset | 6860325
editorial | 133573
entry | 1628487
graphic | 1809471
interview | 19898
legal_case | 3581
legislation | 1626
letter | 275119
paper-conference | 6074669
peer_review | 30581
post | 245807
post-weblog | 135
report | 1010699
retraction | 1292
review-book | 96219
software | 316
song | 24027
speech | 4263
standard | 312364
stub | 1036813
thesis | 414397
| 0
(29 rows)
ingest: more DOI patterns to treat as OA2020-03-29T02:57:41ZBryan Newboldbnewbold@robocracy.org2020-03-29T02:57:35Zurn:sha1:4b75a81cbd0faeefa6a0f04b97ecc6832924ee69
These are journal/publisher patterns which we suspect to actually be OA
based on the large quantity of papers that crawl successfully. The
better long-term solution will be to flag containers in some way as OA
(or "should crawl"), but this is a good short-term solution.
ingest: always try some lancet journals2020-03-20T04:18:18ZBryan Newboldbnewbold@robocracy.org2020-03-20T04:18:18Zurn:sha1:b694e74bf72b498301e31459dedfcc1f56400c21entity worker: ingest more releases2020-02-22T23:07:33ZBryan Newboldbnewbold@robocracy.org2020-02-22T22:57:44Zurn:sha1:dc4116f8ebd225eb4af1cecfc75f5c1291589694
If release is a dataset or image, don't do a pdf ingest request.
If release is a datacite DOI, and release_type is a "document", crawl
regardless of is_oa detection. This is mostly to crawl repositories
(institutional or subject).
always crawl researchgate DOIs2020-02-19T07:08:58ZBryan Newboldbnewbold@robocracy.org2020-02-19T07:08:56Zurn:sha1:9cc369e19d7ba82d07be3d6b24c2526339135a0a
Now that ingest is fixed