aboutsummaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools
Commit message (Collapse)AuthorAgeFilesLines
...
* initial implementation of file_meta importerBryan Newbold2020-08-212-0/+71
|
* entity updater: handle doi=None case betterBryan Newbold2020-08-141-1/+1
|
* entity updater: es['publisher_type'] not always setBryan Newbold2020-08-141-1/+1
| | | | This is a small bugfix for a production issue.
* Merge branch 'bnewbold-ingest-improvements' into 'master'Martin Czygan2020-08-132-33/+114
|\ | | | | | | | | ingest behavior changes; some datacite metadata tweaks See merge request webgroup/fatcat!78
| * entity update: change big5 ingest behaviorBryan Newbold2020-08-111-9/+15
| | | | | | | | | | | | | | | | | | In addition to changing the OA default, this was the main intended behavior change in this group of commits: want to ingest fewer attempts that we *expect* to fail, but default to ingest/crawl attempt if we are uncertain. This is because there is a long tail of journals that register DOIs and are defacto OA (fulltext is available), but we don't have metadata indicating them as such.
| * entity update: default to ingest non-OA worksBryan Newbold2020-08-111-9/+10
| |
| * entity update: skip ingest of figshare+zenodo 'group' DOIsBryan Newbold2020-08-111-0/+15
| |
| * datacite import: figshare-specific hacksBryan Newbold2020-08-111-3/+3
| |
| * datacite import: refactor release_type detection into static methodBryan Newbold2020-08-111-14/+51
| |
| * datacite import: refactor publisher-specific hacks into static methodBryan Newbold2020-08-111-15/+29
| | | | | | | | Also tweak title/publisher detection to use DOI prefixes
| * update crawl blocklist for SPNv2 requests which mostly failBryan Newbold2020-08-101-2/+10
| |
* | harvest: datacite API yields HTTP 200 with broken JSONMartin Czygan2020-08-101-1/+8
|/ | | | As a first step: log response body for debugging.
* release ES transform tweaksBryan Newbold2020-08-071-3/+5
| | | | | | | | pass-through publisher_type from container extra metadata (ES field already existed; this is from newer chocula metadata) count arxiv and PMCID papers which haven't been crawled (by IA) as "dark", not "bright"
* chocula import update tweaksBryan Newbold2020-08-041-10/+14
|
* more update keys and cases for chocula importerBryan Newbold2020-08-041-5/+11
|
* fix key name mismatch in chocula importerBryan Newbold2020-08-041-1/+1
| | | | chocula 'export-fatcat' uses 'ident', not 'fatcat_ident'
* basic toml transform helperBryan Newbold2020-07-302-4/+20
|
* Merge branch 'bnewbold-more-lint-fixes' into 'master'Martin Czygan2020-07-246-25/+18
|\ | | | | | | | | more lint fixes See merge request webgroup/fatcat!69
| * fix issnl typo in pubmedBryan Newbold2020-07-231-1/+1
| | | | | | | | | | | | | | | | | | | | Oh no! This bug may actually have had significant negative impact on metadata in fatcat, in terms of missing container_id associations with pubmed entities. There are about 500k release entities with a PMID but no container_id. Of those, 89k have at least a container_name. Unclear how many would have matched to ISSN-L and thus to a container.
| * remove isascii() work around definition in importers/datacite.pyBryan Newbold2020-07-231-7/+1
| | | | | | | | We are python3.7 now, so this isn't needed.
| * simple lint (flake8) fixes over python codebaseBryan Newbold2020-07-235-17/+16
| | | | | | | | | | | | These should not have any behavior changes, though a number of exception catches are now more general, and there may be long-tail exceptions getting thrown in these statements.
* | simplify in_kbart check statementBryan Newbold2020-07-231-1/+1
| | | | | | | | Thanks @martin
* | make in_kbart transform inclusive of last yearBryan Newbold2020-07-231-0/+9
|/ | | | | | | | | | | | | | | | | Frequently when looking at preservation coverage of journals, the current year shows as "un-preserved" when in fact there is robust KBART (keepers, eg CLOCKSS/Portico) coverage. This is partially because we don't update containers with KBART year spans very frequently (which is on us), and partially because KBART reports are often a bit out of day (eg, doesn't show coverage for the current year. For that matter, they probably take a few months to update the previous year as well, but that is a larger time span to fudge over. This patch means we will count Portico/LOCKSS/etc coverage for "last year" to count as coverage of publications dated "this year". Note that for this to be effective/correct, it is assumed that we will update containers with coverage year spans at least once a year, and that we will re-index all releases at least once a year.
* Merge branch 'martin-datacite-duplicated-author-gh-59' into 'master'bnewbold2020-07-111-6/+60
|\ | | | | | | | | datacite: address duplicated contributor issue See merge request webgroup/fatcat!65
| * datacite: resolve formatting issues in testsMartin Czygan2020-07-1033-133/+51
| |\
| * | datacite: there should be no index gapsMartin Czygan2020-07-101-2/+8
| | |
| * | datacite: document contributor typesMartin Czygan2020-07-101-0/+25
| | |
| * | wip: contrib, GH59Martin Czygan2020-07-101-16/+22
| | |
| * | datacite: address duplicated contributor issueMartin Czygan2020-07-071-0/+16
| | | | | | | | | | | | | | | | | | | | | Use string comparison. * https://fatcat.wiki/release/spjysmrnsrgyzgq6ise5o44rlu/contribs * https://api.datacite.org/dois/10.25940/roper-31098406
* | | Merge branch 'martin-datacite-bugfix-sentry-44035' into 'master'bnewbold2020-07-111-0/+4
|\ \ \ | |_|/ |/| | | | | | | | datacite: mitigate sentry #44035 See merge request webgroup/fatcat!66
| * | datacite: mitigate sentry #44035Martin Czygan2020-07-101-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | According to sentry, running `c.get('nameIdentifiers', []) or []` on a c with value: ``` {'affiliation': [], 'familyName': 'Guidon', 'givenName': 'Manuel', 'nameIdentifiers': {'nameIdentifier': 'https://orcid.org/0000-0003-3543-6683', 'nameIdentifierScheme': 'ORCID', 'schemeUri': 'https://orcid.org'}, 'nameType': 'Personal'} ``` results in a string, which I cannot reproduce. The document in question at: https://api.datacite.org/dois/10.26275/kuw1-fdls seems fine, too.
* | | Merge branch 'martin-arxiv-fix-http-503' into 'master'bnewbold2020-07-101-1/+1
|\ \ \ | |/ / |/| | | | | | | | arxiv: address 503, "Retry after specified interval" error See merge request webgroup/fatcat!64
| * | arxiv: do retry five times of HTTP 503Martin Czygan2020-07-101-1/+1
| | |
* | | datacite: fix attribute errorMartin Czygan2020-07-071-1/+1
|/ / | | | | | | refs: #44035
* | lint (flake8) tool python filesBryan Newbold2020-07-0133-130/+46
| |
* | reviewer: fix bugs in common code found by mypyBryan Newbold2020-07-011-2/+3
|/
* add new license mappingsBryan Newbold2020-06-302-0/+27
|
* datacite: improve license mappingMartin Czygan2020-06-301-9/+15
| | | | via "missed potential license", refs #58
* datacite: hard cast possible date value to stringMartin Czygan2020-06-291-1/+1
|
* disallow a specific unicode character from DOIsBryan Newbold2020-06-261-0/+6
|
* ES schema: add best_url to file schemaBryan Newbold2020-06-041-0/+12
| | | | | | | | | This will increase index size (URLs are often long in our corpus, and we have many file entities), but seems worth it. Initially added `ia_url` as a second field, guaranteed to always be an *.archive.org URL, but `best_url` defaults to that anyways so didn't seem worthwhile.
* harvest: fail on HTTP 400Martin Czygan2020-05-291-4/+0
| | | | | | | | | In the past harvest of datacite resulted in occasional HTTP 400. Meanwhile, various API bugs have been fixed (most recently: https://github.com/datacite/lupo/pull/537, https://github.com/datacite/datacite/issues/1038). Downside of ignoring this error was that state lives in kafka, which has limited support for deletion of arbitrary messages from a topic.
* Merge branch 'bnewbold-ingest-stage' into 'master'Martin Czygan2020-05-281-0/+5
|\ | | | | | | | | verify release_stage in ingest importer See merge request webgroup/fatcat!52
| * ingest importer: check that stage is consistent with releaseBryan Newbold2020-05-261-0/+5
| |
* | rename HarvestState.next() to HarvestState.next_span()Bryan Newbold2020-05-264-5/+5
|/ | | | | | | | | "span" short for "timespan" to harvest; there may be a better name to use. Motivation for this is to work around a pylint erorr that .next() was not callable. This might be a bug with pylint, but .next() is also a very generic name.
* HACK: skip pylint errors on lines that seem to be fineBryan Newbold2020-05-223-3/+3
| | | | | It seems to be an inadvertantly ugraded version of pylint saying that these lines are not-callable.
* Merge remote-tracking branch 'github/master'Bryan Newbold2020-05-221-2/+2
|\
| * Indentity is not the same this as equality in PythonChristian Clauss2020-05-141-2/+2
| |
* | importers: clarify handling of ApiExceptionBryan Newbold2020-05-223-4/+10
| | | | | | | | | | | | | | | | One of these (in ingest importer pipeline) is an actual bug, the others are just changing the syntax to be more explicit/conservative. The ingest importer bug seems to have resulted in some bad file match imports; scale of impact is unknown.
* | ingest importer: don't use glutton matchesBryan Newbold2020-05-221-3/+3
| | | | | | | | | | | | | | Until reviewing I didn't realize we were even doing this currently. Hopefluly has not impacted too many imports, as almost all ingests use an external identifer, so only those with identifers not in fatcat for whatever reason.
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-15 14:55:26 +0000