aboutsummaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools
Commit message (Collapse)AuthorAgeFilesLines
* add fuzzy match filtering to DOAJ importerBryan Newbold2020-12-161-2/+9
| | | | | | | | | | | In this default configuration, any entities with a fuzzy match (even "ambiguous") will be skipped at import time, to prevent creating duplicates. This is conservative towards not creating new/duplicate entities. In the future, as we get more confidence in fuzzy match/verification, we can start to ignore AMBIGUOUS, handle EXACT as same release, and merge STRONG (and WEAK?) matches under the same work entity.
* add fuzzy matching helper to importer base classBryan Newbold2020-12-161-2/+62
| | | | Using fuzzycat. Add basic test coverage.
* html ingest: small fixes to try_update() code pathBryan Newbold2020-12-151-5/+5
| | | | | Don't currently have test coverage for most try_update() code; run the inserts manually in testing.
* HACK: squash intermitent failure of detect_text_lang() testBryan Newbold2020-12-111-1/+2
| | | | | This is an open bug; it is important that tests pass on master branch however.
* langdetect: more text for 'zh' test caseBryan Newbold2020-11-201-1/+1
| | | | | | This is an attempt to fix spurious test failures, in which this text block was getting detected as 'kr' on occasion. Apparently there is non-determinism in the langdetect package.
* crossref+datacite: remove confusing early update bailBryan Newbold2020-11-202-4/+0
| | | | | Easy to miss that we skip updates *twice*, and with this early bailout were not updating counts correctly.
* doaj: fix update code path (getattr not __dict__)Bryan Newbold2020-11-201-4/+3
| | | | Also add missing code coverage for update path (disabled by default).
* DOAJ: handle empty identifier 'id' caseBryan Newbold2020-11-201-0/+2
|
* clean DOI: ban all non-ASCII charactersBryan Newbold2020-11-191-1/+4
| | | | | | | I believe this is safe and matches the regex filter in rust (fatcatd). Keep hitting one-off DOIs that were passing through python check, so being more strict from here forward.
* normal: handle langdetect of 'zh-cn' (not len=2)Bryan Newbold2020-11-191-0/+3
|
* tweak DOAJ importer class args and default for do_updatesBryan Newbold2020-11-191-2/+2
|
* if a release has DOAJ article id, count as OABryan Newbold2020-11-191-0/+3
|
* implement remainder of DOAJ article importerBryan Newbold2020-11-191-57/+125
|
* handle more non-ASCII DOI casesBryan Newbold2020-11-191-1/+3
|
* more python normalizers, and move from importer commonBryan Newbold2020-11-192-154/+326
| | | | | | | | | | | | Moved several normalizer helpers out of fatcat_tools.importers.common to fatcat_tools.normal. Copied language name and country name parser helpers from chocula repository (built on existing pycountry helper library). Have not gone through and refactored other importers to point to these helpers yet; that should be a separate PR when this branch is merged. Current changes are backwards compatible via re-imports.
* initial implementation of DOAJ importerBryan Newbold2020-11-192-0/+290
| | | | Several things to finish implementing and polish.
* html ingest: actual xhtml mimetypeBryan Newbold2020-11-161-2/+2
|
* ingest tool: support for setting ingest typeBryan Newbold2020-11-061-6/+6
|
* html ingest: remaining implementationBryan Newbold2020-11-061-22/+19
|
* ingest: progress on HTML ingestBryan Newbold2020-11-051-14/+30
|
* ingest: initial 'web' worker implementationBryan Newbold2020-11-052-67/+259
|
* refactor: white/black -> allow/blockBryan Newbold2020-11-051-4/+4
|
* ingest: whitelist -> allowlistBryan Newbold2020-11-051-3/+3
|
* ingest: basic checks for ingest_typeBryan Newbold2020-11-051-3/+29
|
* normalizer: filter out a specific non-ASCII character in DOIBryan Newbold2020-11-041-1/+3
|
* entity updates: don't ingest JSTOR DOI prefixesBryan Newbold2020-10-231-0/+2
|
* entity updater: new work update feed (ident and changelog metadata only)Bryan Newbold2020-10-161-2/+24
|
* chocula importer: small tweaks to update behaviorBryan Newbold2020-10-081-8/+6
|
* elastic transform: more preservation keepersBryan Newbold2020-10-081-1/+2
|
* address spammy datacite titlesMartin Czygan2020-09-231-0/+19
| | | | | | | | | seemingly from zenodo: * https://fatcat.wiki/release/rzcpjwukobd4pj36ipla22cnoi * https://doi.org/10.5281/zenodo.4041777 About 3400 records with "FULL MOVIE" in title, currently.
* ingest: default to crawl protocols.io DOIsBryan Newbold2020-09-101-0/+2
|
* datacite: handle case of empty-string versionBryan Newbold2020-09-101-1/+1
| | | | | Includes a tiny tweak to the datacite import sample file to test this code path.
* remove spurious print statementBryan Newbold2020-09-031-1/+0
|
* generic file entity clean-ups as part of file_meta importerBryan Newbold2020-09-022-0/+50
|
* fix comment typo (thanks martin)Bryan Newbold2020-08-271-1/+1
|
* fixes and test coverage for file_meta importerBryan Newbold2020-08-211-5/+10
|
* initial implementation of file_meta importerBryan Newbold2020-08-212-0/+71
|
* entity updater: handle doi=None case betterBryan Newbold2020-08-141-1/+1
|
* entity updater: es['publisher_type'] not always setBryan Newbold2020-08-141-1/+1
| | | | This is a small bugfix for a production issue.
* Merge branch 'bnewbold-ingest-improvements' into 'master'Martin Czygan2020-08-132-33/+114
|\ | | | | | | | | ingest behavior changes; some datacite metadata tweaks See merge request webgroup/fatcat!78
| * entity update: change big5 ingest behaviorBryan Newbold2020-08-111-9/+15
| | | | | | | | | | | | | | | | | | In addition to changing the OA default, this was the main intended behavior change in this group of commits: want to ingest fewer attempts that we *expect* to fail, but default to ingest/crawl attempt if we are uncertain. This is because there is a long tail of journals that register DOIs and are defacto OA (fulltext is available), but we don't have metadata indicating them as such.
| * entity update: default to ingest non-OA worksBryan Newbold2020-08-111-9/+10
| |
| * entity update: skip ingest of figshare+zenodo 'group' DOIsBryan Newbold2020-08-111-0/+15
| |
| * datacite import: figshare-specific hacksBryan Newbold2020-08-111-3/+3
| |
| * datacite import: refactor release_type detection into static methodBryan Newbold2020-08-111-14/+51
| |
| * datacite import: refactor publisher-specific hacks into static methodBryan Newbold2020-08-111-15/+29
| | | | | | | | Also tweak title/publisher detection to use DOI prefixes
| * update crawl blocklist for SPNv2 requests which mostly failBryan Newbold2020-08-101-2/+10
| |
* | harvest: datacite API yields HTTP 200 with broken JSONMartin Czygan2020-08-101-1/+8
|/ | | | As a first step: log response body for debugging.
* release ES transform tweaksBryan Newbold2020-08-071-3/+5
| | | | | | | | pass-through publisher_type from container extra metadata (ES field already existed; this is from newer chocula metadata) count arxiv and PMCID papers which haven't been crawled (by IA) as "dark", not "bright"
* chocula import update tweaksBryan Newbold2020-08-041-10/+14
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-30 23:40:45 +0000