summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* update fuzzy helper to pass 'reason' through to import codeBryan Newbold2020-12-172-5/+5
| | | | | The motivation for this change is to enable passing the 'reason' through to edit extra metadata, in cases where we merge or cluster releases.
* pipenv: bump fuzzycat to 0.1.9Bryan Newbold2020-12-172-5/+5
|
* add fuzzy match filtering to DOAJ importerBryan Newbold2020-12-162-4/+23
| | | | | | | | | | | In this default configuration, any entities with a fuzzy match (even "ambiguous") will be skipped at import time, to prevent creating duplicates. This is conservative towards not creating new/duplicate entities. In the future, as we get more confidence in fuzzy match/verification, we can start to ignore AMBIGUOUS, handle EXACT as same release, and merge STRONG (and WEAK?) matches under the same work entity.
* add fuzzy matching helper to importer base classBryan Newbold2020-12-163-2/+147
| | | | Using fuzzycat. Add basic test coverage.
* pipenv: add fuzzycat dependencyBryan Newbold2020-12-162-261/+374
|
* html ingest: small fixes to try_update() code pathBryan Newbold2020-12-151-5/+5
| | | | | Don't currently have test coverage for most try_update() code; run the inserts manually in testing.
* notes on partial-progress DOAJ release metadata importBryan Newbold2020-12-141-0/+105
|
* bulk import notes on ORCIDBryan Newbold2020-12-141-0/+55
|
* Revert "gitlab CI: explicitly use xenial tag of image"Bryan Newbold2020-12-111-1/+1
| | | | This reverts commit dbfc6e9bacaab4960e814192d66eefea87ef8930.
* Revert "docker xenial base image: include python3.8"Bryan Newbold2020-12-111-6/+1
| | | | This reverts commit 91628426678a635f26cf992dbd5caedb4a3ae24b.
* gitlab CI: explicitly use xenial tag of imageBryan Newbold2020-12-111-1/+1
|
* docker xenial base image: include python3.8Bryan Newbold2020-12-111-1/+6
|
* HACK: squash intermitent failure of detect_text_lang() testBryan Newbold2020-12-111-1/+2
| | | | | This is an open bug; it is important that tests pass on master branch however.
* guide: small updates to container extra schema notes (from dblp work)Bryan Newbold2020-12-111-2/+7
|
* bulk edits: note ORCID updateBryan Newbold2020-12-111-1/+5
|
* docker: how to push to dockerhubBryan Newbold2020-12-111-0/+4
|
* Merge branch 'bnewbold-doaj-metadata' into 'master'Martin Czygan2020-11-2437-1549/+2845
|\ | | | | | | | | DOAJ article metadata import See merge request webgroup/fatcat!89
| * cargo: update sentry to fix memory initialization issueBryan Newbold2020-11-202-274/+332
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Older sentry had an unsafe memory initialization error, which wasn't caught by older compilers. Rust 1.48 catches the problem at runtime and raises a panic. This meant that new builds (eg, on QA machine after update) were panic-ing. Newest versions of sentry have modern dependencies, which breaks our crufty old 'iron' dependency tree. Work-around is to only partially update (v0.12 to v0.15). This is a fairly frustrating situation. I'm hopeful that when we update to a different web framework and openapi generator 5.0 (not yet released), many of these dependency issues will be resolved, but i'm not certain. I did notice that if we entirely remove Sentry, which has not really been used much (only a small handful of issues reported over several years), we might be able to resolve openssl dependency issues.
| * DOAJ: remove accidentally commited 'skip' of a testBryan Newbold2020-11-201-1/+0
| |
| * langdetect: more text for 'zh' test caseBryan Newbold2020-11-201-1/+1
| | | | | | | | | | | | This is an attempt to fix spurious test failures, in which this text block was getting detected as 'kr' on occasion. Apparently there is non-determinism in the langdetect package.
| * DOAJ: update importer README with example invocationBryan Newbold2020-11-201-0/+7
| |
| * crossref+datacite: remove confusing early update bailBryan Newbold2020-11-202-4/+0
| | | | | | | | | | Easy to miss that we skip updates *twice*, and with this early bailout were not updating counts correctly.
| * doaj: fix update code path (getattr not __dict__)Bryan Newbold2020-11-203-15/+70
| | | | | | | | Also add missing code coverage for update path (disabled by default).
| * DOAJ: handle empty identifier 'id' caseBryan Newbold2020-11-201-0/+2
| |
| * clean DOI: ban all non-ASCII charactersBryan Newbold2020-11-191-1/+4
| | | | | | | | | | | | | | I believe this is safe and matches the regex filter in rust (fatcatd). Keep hitting one-off DOIs that were passing through python check, so being more strict from here forward.
| * normal: handle langdetect of 'zh-cn' (not len=2)Bryan Newbold2020-11-191-0/+3
| |
| * update fatcatd rust code for 'oai' external identifierBryan Newbold2020-11-194-11/+189
| |
| * codegen rust schema crateBryan Newbold2020-11-196-3/+20
| |
| * codegen python openapi clientBryan Newbold2020-11-192-4/+36
| |
| * schema: also add 'oai' identifer (OAI-PMH) for releasesBryan Newbold2020-11-192-2/+9
| |
| * tweak DOAJ importer class args and default for do_updatesBryan Newbold2020-11-191-2/+2
| |
| * show DOAJ (and dblp) identifiers in release viewBryan Newbold2020-11-191-1/+7
| |
| * if a release has DOAJ article id, count as OABryan Newbold2020-11-191-0/+3
| |
| * implement remainder of DOAJ article importerBryan Newbold2020-11-193-68/+168
| |
| * handle more non-ASCII DOI casesBryan Newbold2020-11-191-1/+3
| |
| * more python normalizers, and move from importer commonBryan Newbold2020-11-192-154/+326
| | | | | | | | | | | | | | | | | | | | | | | | Moved several normalizer helpers out of fatcat_tools.importers.common to fatcat_tools.normal. Copied language name and country name parser helpers from chocula repository (built on existing pycountry helper library). Have not gone through and refactored other importers to point to these helpers yet; that should be a separate PR when this branch is merged. Current changes are backwards compatible via re-imports.
| * initial implementation of DOAJ importerBryan Newbold2020-11-194-0/+387
| | | | | | | | Several things to finish implementing and polish.
| * python API client: resolve warning about '\d' in stringBryan Newbold2020-11-191-2/+2
| |
| * rustfmtBryan Newbold2020-11-195-87/+138
| |
| * rust: fatcatd changes for DOAJ+dblp identifiersBryan Newbold2020-11-196-949/+1062
| |
| * codegen rust crate for v0.3.3Bryan Newbold2020-11-198-227/+244
| |
| * codegen python client library for v0.3.3Bryan Newbold2020-11-197-16/+80
| |
| * schema: DOAJ+dblp ext_ids; bump to v0.3.3Bryan Newbold2020-11-192-1/+25
|/
* ingest and proposal updatesBryan Newbold2020-11-192-0/+45
|
* Merge branch 'bnewbold-xml-html-ingest' into 'master'Martin Czygan2020-11-1910-66/+409
|\ | | | | | | | | HTML webcapture ingest (and XML file ingest) See merge request webgroup/fatcat!88
| * html ingest: actual xhtml mimetypeBryan Newbold2020-11-161-2/+2
| |
| * ingest tool: support for setting ingest typeBryan Newbold2020-11-062-6/+10
| |
| * html ingest: remaining implementationBryan Newbold2020-11-061-22/+19
| |
| * ingest: fix XML ingest test fileBryan Newbold2020-11-051-1/+1
| |
| * ingest: progress on HTML ingestBryan Newbold2020-11-053-16/+74
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-09 21:09:01 +0000