summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/importers
Commit message (Collapse)AuthorAgeFilesLines
* more python normalizers, and move from importer commonBryan Newbold2020-11-191-154/+4
| | | | | | | | | | | | Moved several normalizer helpers out of fatcat_tools.importers.common to fatcat_tools.normal. Copied language name and country name parser helpers from chocula repository (built on existing pycountry helper library). Have not gone through and refactored other importers to point to these helpers yet; that should be a separate PR when this branch is merged. Current changes are backwards compatible via re-imports.
* initial implementation of DOAJ importerBryan Newbold2020-11-192-0/+290
| | | | Several things to finish implementing and polish.
* html ingest: actual xhtml mimetypeBryan Newbold2020-11-161-2/+2
|
* html ingest: remaining implementationBryan Newbold2020-11-061-22/+19
|
* ingest: progress on HTML ingestBryan Newbold2020-11-051-14/+30
|
* ingest: initial 'web' worker implementationBryan Newbold2020-11-052-67/+259
|
* refactor: white/black -> allow/blockBryan Newbold2020-11-051-4/+4
|
* ingest: whitelist -> allowlistBryan Newbold2020-11-051-3/+3
|
* ingest: basic checks for ingest_typeBryan Newbold2020-11-051-3/+29
|
* chocula importer: small tweaks to update behaviorBryan Newbold2020-10-081-8/+6
|
* address spammy datacite titlesMartin Czygan2020-09-231-0/+19
| | | | | | | | | seemingly from zenodo: * https://fatcat.wiki/release/rzcpjwukobd4pj36ipla22cnoi * https://doi.org/10.5281/zenodo.4041777 About 3400 records with "FULL MOVIE" in title, currently.
* datacite: handle case of empty-string versionBryan Newbold2020-09-101-1/+1
| | | | | Includes a tiny tweak to the datacite import sample file to test this code path.
* remove spurious print statementBryan Newbold2020-09-031-1/+0
|
* generic file entity clean-ups as part of file_meta importerBryan Newbold2020-09-022-0/+50
|
* fix comment typo (thanks martin)Bryan Newbold2020-08-271-1/+1
|
* fixes and test coverage for file_meta importerBryan Newbold2020-08-211-5/+10
|
* initial implementation of file_meta importerBryan Newbold2020-08-212-0/+71
|
* datacite import: figshare-specific hacksBryan Newbold2020-08-111-3/+3
|
* datacite import: refactor release_type detection into static methodBryan Newbold2020-08-111-14/+51
|
* datacite import: refactor publisher-specific hacks into static methodBryan Newbold2020-08-111-15/+29
| | | | Also tweak title/publisher detection to use DOI prefixes
* chocula import update tweaksBryan Newbold2020-08-041-10/+14
|
* more update keys and cases for chocula importerBryan Newbold2020-08-041-5/+11
|
* fix key name mismatch in chocula importerBryan Newbold2020-08-041-1/+1
| | | | chocula 'export-fatcat' uses 'ident', not 'fatcat_ident'
* fix issnl typo in pubmedBryan Newbold2020-07-231-1/+1
| | | | | | | | | | Oh no! This bug may actually have had significant negative impact on metadata in fatcat, in terms of missing container_id associations with pubmed entities. There are about 500k release entities with a PMID but no container_id. Of those, 89k have at least a container_name. Unclear how many would have matched to ISSN-L and thus to a container.
* remove isascii() work around definition in importers/datacite.pyBryan Newbold2020-07-231-7/+1
| | | | We are python3.7 now, so this isn't needed.
* simple lint (flake8) fixes over python codebaseBryan Newbold2020-07-235-17/+16
| | | | | | These should not have any behavior changes, though a number of exception catches are now more general, and there may be long-tail exceptions getting thrown in these statements.
* Merge branch 'martin-datacite-duplicated-author-gh-59' into 'master'bnewbold2020-07-111-6/+60
|\ | | | | | | | | datacite: address duplicated contributor issue See merge request webgroup/fatcat!65
| * datacite: resolve formatting issues in testsMartin Czygan2020-07-1016-72/+28
| |\
| * | datacite: there should be no index gapsMartin Czygan2020-07-101-2/+8
| | |
| * | datacite: document contributor typesMartin Czygan2020-07-101-0/+25
| | |
| * | wip: contrib, GH59Martin Czygan2020-07-101-16/+22
| | |
| * | datacite: address duplicated contributor issueMartin Czygan2020-07-071-0/+16
| | | | | | | | | | | | | | | | | | | | | Use string comparison. * https://fatcat.wiki/release/spjysmrnsrgyzgq6ise5o44rlu/contribs * https://api.datacite.org/dois/10.25940/roper-31098406
* | | datacite: mitigate sentry #44035Martin Czygan2020-07-101-0/+4
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | According to sentry, running `c.get('nameIdentifiers', []) or []` on a c with value: ``` {'affiliation': [], 'familyName': 'Guidon', 'givenName': 'Manuel', 'nameIdentifiers': {'nameIdentifier': 'https://orcid.org/0000-0003-3543-6683', 'nameIdentifierScheme': 'ORCID', 'schemeUri': 'https://orcid.org'}, 'nameType': 'Personal'} ``` results in a string, which I cannot reproduce. The document in question at: https://api.datacite.org/dois/10.26275/kuw1-fdls seems fine, too.
* | datacite: fix attribute errorMartin Czygan2020-07-071-1/+1
| | | | | | | | refs: #44035
* | lint (flake8) tool python filesBryan Newbold2020-07-0116-72/+27
|/
* add new license mappingsBryan Newbold2020-06-302-0/+27
|
* datacite: improve license mappingMartin Czygan2020-06-301-9/+15
| | | | via "missed potential license", refs #58
* datacite: hard cast possible date value to stringMartin Czygan2020-06-291-1/+1
|
* ingest importer: check that stage is consistent with releaseBryan Newbold2020-05-261-0/+5
|
* Merge remote-tracking branch 'github/master'Bryan Newbold2020-05-221-2/+2
|\
| * Indentity is not the same this as equality in PythonChristian Clauss2020-05-141-2/+2
| |
* | importers: clarify handling of ApiExceptionBryan Newbold2020-05-223-4/+10
| | | | | | | | | | | | | | | | One of these (in ingest importer pipeline) is an actual bug, the others are just changing the syntax to be more explicit/conservative. The ingest importer bug seems to have resulted in some bad file match imports; scale of impact is unknown.
* | ingest importer: don't use glutton matchesBryan Newbold2020-05-221-3/+3
| | | | | | | | | | | | | | Until reviewing I didn't realize we were even doing this currently. Hopefluly has not impacted too many imports, as almost all ingests use an external identifer, so only those with identifers not in fatcat for whatever reason.
* | datacite: fix type errorMartin Czygan2020-04-221-1/+3
| | | | | | | | | | | | | | Up to now, we expected the description to be a string or list. Add handling for int as well. First appeared: Apr 22 19:58:39.
* | datacite: fix a raw name constraint violationMartin Czygan2020-04-201-0/+8
|/ | | | | | | It was possible that contribs got added which had no raw name. One example would be a name consisting of whitespace only. This fix adds a final check for this case.
* consistently use raw string prefix for regexBryan Newbold2020-04-171-1/+1
|
* pubmed: use untranslated title if translated not availableBryan Newbold2020-04-011-0/+6
| | | | | | | The primary motivation for this change is that fatcat *requires* a non-empty title for each release entity. Pubmed/Medline occasionally indexes just a VenacularTitle with no ArticleTitle for foreign publications, and currently those records don't end up in fatcat at all.
* importers: replace newlines in get_text() stringsBryan Newbold2020-04-014-23/+25
|
* importers: more string/get_text swapsBryan Newbold2020-03-283-27/+27
| | | | See previous pubmed commit for details.
* pubmed: bunch of .get_text() instead of .stringBryan Newbold2020-03-281-12/+12
| | | | | | | | | | | Yikes! Apparently when a tag has child tags, .string will return None instead of all the strings. .get_text() returns all of it: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text https://www.crummy.com/software/BeautifulSoup/bs4/doc/#string I've things like identifiers as .string, when we expect only a single string inside.
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-24 01:10:15 +0000