aboutsummaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'martin-changelog-to-es' into 'master'bnewbold2020-04-172-2/+23
|\ | | | | | | | | derive changelog worker from release worker See merge request webgroup/fatcat!43
| * derive changelog worker from release workerMartin Czygan2020-04-172-2/+23
| | | | | | | | | | Early versions of changelog entries may not have all the fields required for the current transform.
* | changelog: limit typesMartin Czygan2020-04-161-5/+1
| | | | | | | | | | No partial docs (e.g. abstract), too generic components and entries, not HTML blogs.
* | changelog: extend release_types considered documentsMartin Czygan2020-04-161-10/+19
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | according to release_rev.release_type, we have 29 values: fatcat_prod=# select release_type, count(release_type) from release_rev group by release_type; release_type | count -------------------+----------- abstract | 2264 article | 6371076 article-journal | 101083841 article-newspaper | 17062 book | 1676941 chapter | 13914854 component | 58990 dataset | 6860325 editorial | 133573 entry | 1628487 graphic | 1809471 interview | 19898 legal_case | 3581 legislation | 1626 letter | 275119 paper-conference | 6074669 peer_review | 30581 post | 245807 post-weblog | 135 report | 1010699 retraction | 1292 review-book | 96219 software | 316 song | 24027 speech | 4263 standard | 312364 stub | 1036813 thesis | 414397 | 0 (29 rows)
* Merge branch 'bnewbold-pubmed-get_text' into 'master'bnewbold2020-04-014-39/+47
|\ | | | | | | | | beautifulsoup XML parsing: .string vs. .get_text() See merge request webgroup/fatcat!40
| * pubmed: use untranslated title if translated not availableBryan Newbold2020-04-011-0/+6
| | | | | | | | | | | | | | The primary motivation for this change is that fatcat *requires* a non-empty title for each release entity. Pubmed/Medline occasionally indexes just a VenacularTitle with no ArticleTitle for foreign publications, and currently those records don't end up in fatcat at all.
| * importers: replace newlines in get_text() stringsBryan Newbold2020-04-014-23/+25
| |
| * importers: more string/get_text swapsBryan Newbold2020-03-283-27/+27
| | | | | | | | See previous pubmed commit for details.
| * pubmed: bunch of .get_text() instead of .stringBryan Newbold2020-03-281-12/+12
| | | | | | | | | | | | | | | | | | | | | | Yikes! Apparently when a tag has child tags, .string will return None instead of all the strings. .get_text() returns all of it: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text https://www.crummy.com/software/BeautifulSoup/bs4/doc/#string I've things like identifiers as .string, when we expect only a single string inside.
* | crossref: switch from index-date to update-dateBryan Newbold2020-03-301-1/+1
| | | | | | | | | | | | This goes against what the API docs recommend, but we are currently far behind on updates and need to catch up. Other than what the docs say, this seems to be consistent with the behavior we want.
* | crossref: longer comment about crossref API date fieldsBryan Newbold2020-03-301-2/+22
|/
* ingest: more DOI patterns to treat as OABryan Newbold2020-03-281-0/+26
| | | | | | | These are journal/publisher patterns which we suspect to actually be OA based on the large quantity of papers that crawl successfully. The better long-term solution will be to flag containers in some way as OA (or "should crawl"), but this is a good short-term solution.
* Merge pull request #53 from EdwardBetts/spellingbnewbold2020-03-274-9/+9
|\ | | | | Correct spelling mistakes
| * Correct spelling mistakesEdward Betts2020-03-274-9/+9
| |
* | Merge branch 'bnewbold-citeproc-fixes' into 'master'bnewbold2020-03-261-6/+12
|\ \ | | | | | | | | | | | | improve citeproc/CSL web interface See merge request webgroup/fatcat!36
| * | improve citeproc/CSL web interfaceBryan Newbold2020-03-251-6/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This tries to show the citeproc (bibtext, MLA, CSL-JSON) options for more releases, and not show the links when they would break. The primary motivation here is to work around two exceptions being thrown in prod every day (according to sentry): KeyError: 'role' ValueError: CLS requries some surname (family name) I'm guessing these are mostly coming from crawlers following the citeproc links on release landing pages.
* | | datacite: nameIdentifier corner caseBryan Newbold2020-03-261-1/+2
|/ / | | | | | | | | | | | | | | | | Works around a bug in production: AttributeError: 'NoneType' object has no attribute 'replace' (datacite.py:724) NOTE: there are no tests for this code path
* | jalc: avoid meaningless pages valuesBryan Newbold2020-03-231-4/+8
| |
* | datacite: add year sanity restrictionsbnewbold2020-03-231-0/+7
| | | | | | | | | | | | | | | | | | Example of entities with bogus years: https://fatcat.wiki/release/search?q=doi_registrar%3Adatacite+year%3A%3E2100 We can do a clean-up task, but first need to prevent creation of new bad metadata.
* | pubmed: handle multiple ReferenceListBryan Newbold2020-03-201-1/+4
| | | | | | | | | | | | | | This resolves a situation noticed in prod where we were only importing/updating a single reference per article. Includes a regression test.
* | pubmed: update many more metadata fieldsBryan Newbold2020-03-191-0/+22
| | | | | | | | | | | | | | In particular, with daily updates in most cases the DOI will be registered first, then the entity updated with PMID when that is available. Often the pubmed metadata will be more complete, with abstracts etc, and we'll want those improvements.
* | crossref: skip stub OUP titleBryan Newbold2020-03-191-0/+8
| | | | | | | | | | | | It seems like OUP pre-registers DOIs with this place-holder title, then updates the Crossref metdata when the paper is actually published. We should wait until the real title is available before creating an entity.
* | ingest: always try some lancet journalsBryan Newbold2020-03-191-0/+3
| |
* | Merge branch 'martin-kafka-bs4-import' into 'master'Martin Czygan2020-03-105-22/+318
|\ \ | |/ |/| | | | | pubmed and arxiv harvest preparations See merge request webgroup/fatcat!28
| * common: use smaller batch size since XML parsing may be slowMartin Czygan2020-03-101-1/+1
| | | | | | | | | | | | | | | | Address kafka tradeoff between long and short time-outs. Shorter time-outs would facilitate > consumer group re-balances and other consumer group state changes [...] in a reasonable human time-frame.
| * pubmed: log to stderrMartin Czygan2020-03-101-1/+1
| |
| * pubmed: move mapping generation out of fetch_dateMartin Czygan2020-03-101-7/+8
| | | | | | | | | | * fetch_date will fail on missing mapping * adjust tests (test will require access to pubmed ftp)
| * harvest: fix imports from HarvestPubmedWorker cleanupMartin Czygan2020-03-101-2/+2
| |
| * pubmed: citations is a bit more preciseMartin Czygan2020-03-091-1/+1
| | | | | | | | | | > Each day, NLM produces update files that include new, revised and deleted citations. -- ftp://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/README.txt
| * pubmed: we sync from FTPMartin Czygan2020-03-091-1/+1
| |
| * oaipmh: HarvestPubmedWorker obsoleted by PubmedFTPWorkerMartin Czygan2020-03-091-34/+0
| |
| * more pubmed adjustmentsMartin Czygan2020-02-222-70/+118
| | | | | | | | | | * regenerate map in continuous mode * add tests
| * pubmed ftp: fix urlMartin Czygan2020-02-191-4/+6
| |
| * pubmed ftp harvest and KafkaBs4XmlPusherMartin Czygan2020-02-194-1/+280
| | | | | | | | | | | | | | * add PubmedFTPWorker * utils are currently stored alongside pubmed (e.g. ftpretr, xmlstream) but may live elsewhere, as they are more generic * add KafkaBs4XmlPusher
* | Merge branch 'bnewbold-elastic-v03b'Bryan Newbold2020-02-262-46/+198
|\ \
| * | improve is_oa flag accuracyBryan Newbold2020-02-261-8/+4
| | | | | | | | | | | | | | | | | | Particularly, the ezb=green match seems mostly incorrect. Note that pmcid being assigned could still be in an embargo window?
| * | ES container last tweaksBryan Newbold2020-02-261-0/+3
| | |
| * | ES release: last minor tweaksBryan Newbold2020-02-261-2/+2
| | |
| * | ES files: don't remove archive.org domains/hostsBryan Newbold2020-02-071-5/+0
| | |
| * | ES releases: host/domain fixesBryan Newbold2020-01-311-2/+2
| | |
| * | fix release es transform missing 'issue'Bryan Newbold2020-01-301-0/+1
| | |
| * | add upper-case work-around from kibana map joinBryan Newbold2020-01-301-0/+1
| | |
| * | tweak file ES archive.org domain trackingBryan Newbold2020-01-301-0/+6
| | |
| * | implement host+domain parsing for file ES transformBryan Newbold2020-01-301-9/+5
| | |
| * | fix ES file schema plural field namesBryan Newbold2020-01-291-4/+3
| | |
| * | elastic schema fixesBryan Newbold2020-01-291-0/+5
| | |
| * | add country to v03b release schemaBryan Newbold2020-01-291-0/+2
| | |
| * | actually implement changelog transformBryan Newbold2020-01-291-17/+45
| | |
| * | fix some transform bugs, add some testsBryan Newbold2020-01-291-6/+8
| | |
| * | ES release schema updatesBryan Newbold2020-01-291-5/+76
| | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-27 09:21:19 +0000