summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/importers/datacite.py
Commit message (Collapse)AuthorAgeFilesLines
* datacite: add comment about potential date parsing bugBryan Newbold2021-11-031-0/+1
|
* datacite importer: dateparser.date.DateDataParser()Bryan Newbold2021-11-031-1/+1
| | | | Perhaps this was a change when upgrading 'dateparser'?
* more involved type wrangling and fixes for importersBryan Newbold2021-11-031-2/+3
|
* typing: relatively simple type check fixesBryan Newbold2021-11-031-8/+10
| | | | | | | These mostly add new variable names so that existing variables aren't overwritten with a new type; delay coercing '{}' or '[]' to 'None' until the last minute; adding is-not-None checks to conditional clauses; and similar small changes.
* typing: initial annotations on importersBryan Newbold2021-11-031-30/+59
| | | | | This commit just adds the type annotations, doesn't do fixes to code to make type checking pass.
* fmt (black): fatcat_tools/Bryan Newbold2021-11-021-380/+444
|
* python: isort everythingBryan Newbold2021-11-021-1/+1
|
* lint: simple, safe inline lint fixesBryan Newbold2021-11-021-2/+2
| | | | '==' vs 'is'; 'not a in b' vs 'a not in b'; etc
* datacite: skip empty abstractsMartin Czygan2021-10-011-1/+4
| | | | | Do not add abstracts where `clean` results in the empty string - this violates a constraint: `either abstract_sha1 or content is required`
* datacite: more careful title string access; fixes sentry #88350Martin Czygan2021-06-111-1/+1
| | | | | Caused by a partial "title entry without title" coming *first* (e.g. just holding, e.g. a language, like: {'lang': 'da'}
* datacite: a missing surname should be None, not the empty stringMartin Czygan2021-04-021-2/+1
| | | | refs sentry #77700
* crossref+datacite: remove confusing early update bailBryan Newbold2020-11-201-2/+0
| | | | | Easy to miss that we skip updates *twice*, and with this early bailout were not updating counts correctly.
* refactor: white/black -> allow/blockBryan Newbold2020-11-051-4/+4
|
* address spammy datacite titlesMartin Czygan2020-09-231-0/+19
| | | | | | | | | seemingly from zenodo: * https://fatcat.wiki/release/rzcpjwukobd4pj36ipla22cnoi * https://doi.org/10.5281/zenodo.4041777 About 3400 records with "FULL MOVIE" in title, currently.
* datacite: handle case of empty-string versionBryan Newbold2020-09-101-1/+1
| | | | | Includes a tiny tweak to the datacite import sample file to test this code path.
* datacite import: figshare-specific hacksBryan Newbold2020-08-111-3/+3
|
* datacite import: refactor release_type detection into static methodBryan Newbold2020-08-111-14/+51
|
* datacite import: refactor publisher-specific hacks into static methodBryan Newbold2020-08-111-15/+29
| | | | Also tweak title/publisher detection to use DOI prefixes
* remove isascii() work around definition in importers/datacite.pyBryan Newbold2020-07-231-7/+1
| | | | We are python3.7 now, so this isn't needed.
* simple lint (flake8) fixes over python codebaseBryan Newbold2020-07-231-7/+7
| | | | | | These should not have any behavior changes, though a number of exception catches are now more general, and there may be long-tail exceptions getting thrown in these statements.
* Merge branch 'martin-datacite-duplicated-author-gh-59' into 'master'bnewbold2020-07-111-6/+60
|\ | | | | | | | | datacite: address duplicated contributor issue See merge request webgroup/fatcat!65
| * datacite: resolve formatting issues in testsMartin Czygan2020-07-101-2/+1
| |\
| * | datacite: there should be no index gapsMartin Czygan2020-07-101-2/+8
| | |
| * | datacite: document contributor typesMartin Czygan2020-07-101-0/+25
| | |
| * | wip: contrib, GH59Martin Czygan2020-07-101-16/+22
| | |
| * | datacite: address duplicated contributor issueMartin Czygan2020-07-071-0/+16
| | | | | | | | | | | | | | | | | | | | | Use string comparison. * https://fatcat.wiki/release/spjysmrnsrgyzgq6ise5o44rlu/contribs * https://api.datacite.org/dois/10.25940/roper-31098406
* | | datacite: mitigate sentry #44035Martin Czygan2020-07-101-0/+4
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | According to sentry, running `c.get('nameIdentifiers', []) or []` on a c with value: ``` {'affiliation': [], 'familyName': 'Guidon', 'givenName': 'Manuel', 'nameIdentifiers': {'nameIdentifier': 'https://orcid.org/0000-0003-3543-6683', 'nameIdentifierScheme': 'ORCID', 'schemeUri': 'https://orcid.org'}, 'nameType': 'Personal'} ``` results in a string, which I cannot reproduce. The document in question at: https://api.datacite.org/dois/10.26275/kuw1-fdls seems fine, too.
* | datacite: fix attribute errorMartin Czygan2020-07-071-1/+1
| | | | | | | | refs: #44035
* | lint (flake8) tool python filesBryan Newbold2020-07-011-2/+0
|/
* add new license mappingsBryan Newbold2020-06-301-0/+14
|
* datacite: improve license mappingMartin Czygan2020-06-301-9/+15
| | | | via "missed potential license", refs #58
* datacite: hard cast possible date value to stringMartin Czygan2020-06-291-1/+1
|
* datacite: fix type errorMartin Czygan2020-04-221-1/+3
| | | | | | | Up to now, we expected the description to be a string or list. Add handling for int as well. First appeared: Apr 22 19:58:39.
* datacite: fix a raw name constraint violationMartin Czygan2020-04-201-0/+8
| | | | | | | It was possible that contribs got added which had no raw name. One example would be a name consisting of whitespace only. This fix adds a final check for this case.
* Merge pull request #53 from EdwardBetts/spellingbnewbold2020-03-271-4/+4
|\ | | | | Correct spelling mistakes
| * Correct spelling mistakesEdward Betts2020-03-271-4/+4
| |
* | datacite: nameIdentifier corner caseBryan Newbold2020-03-261-1/+2
| | | | | | | | | | | | | | | | | | Works around a bug in production: AttributeError: 'NoneType' object has no attribute 'replace' (datacite.py:724) NOTE: there are no tests for this code path
* | datacite: add year sanity restrictionsbnewbold2020-03-231-0/+7
|/ | | | | | | | | Example of entities with bogus years: https://fatcat.wiki/release/search?q=doi_registrar%3Adatacite+year%3A%3E2100 We can do a clean-up task, but first need to prevent creation of new bad metadata.
* datacite: prevent noneMartin Czygan2020-01-311-1/+1
|
* datacite: name shall not be NoneMartin Czygan2020-01-311-1/+1
|
* datacite: add exception for https://www.micropublication.org/Martin Czygan2020-01-311-0/+5
|
* datacite: do not skip records w/o dateMartin Czygan2020-01-311-2/+1
|
* datacite: improve docstringMartin Czygan2020-01-311-4/+4
|
* datacite: improve date handling and minor tweakMartin Czygan2020-01-301-19/+42
| | | | | | | | | | | | | Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too.
* datacite: skip records without a doiMartin Czygan2020-01-131-0/+4
|
* datacite: add entry to license slug mapMartin Czygan2020-01-091-0/+1
|
* datacite: ignore known unknown values in resourceType*Martin Czygan2020-01-091-2/+2
|
* datacite: abstracts may be strings or list of stringsMartin Czygan2020-01-091-2/+15
|
* datacite: improve license_slug handlingMartin Czygan2020-01-091-60/+101
|
* datacite: add 'Unknown' to blacklistMartin Czygan2020-01-091-1/+5
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-18 06:00:28 +0000