summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* wikidata QID normalize helperBryan Newbold2020-12-171-2/+24
|
* initial implementation of dblp release importer (in progress)Bryan Newbold2020-12-173-0/+474
|
* add 'lxml' mode for large XML file import, and multi-tagsBryan Newbold2020-12-173-19/+31
|
* rust: fix malformed ext id error typeBryan Newbold2020-12-171-2/+2
| | | | This bug was due to copy/paste of SHA-1 check
* rust: rename and improve dblp key (id) syntax checkBryan Newbold2020-12-172-9/+17
|
* fix sloppy is_preserved ES transfom test failureBryan Newbold2020-12-171-1/+1
|
* DOAJ import notesBryan Newbold2020-12-172-2/+23
|
* add dblp as an ingest source and identifierBryan Newbold2020-12-171-1/+2
|
* ingest: allow doaj ingest responsesBryan Newbold2020-12-171-1/+2
|
* bug fix: is_preserved should always be boolBryan Newbold2020-12-171-2/+2
|
* Merge branch 'bnewbold-doaj-fuzzy' into 'master'bnewbold2020-12-187-267/+544
|\ | | | | | | | | DOAJ import fuzzy match filter See merge request webgroup/fatcat!92
| * update fuzzy helper to pass 'reason' through to import codeBryan Newbold2020-12-172-5/+5
| | | | | | | | | | The motivation for this change is to enable passing the 'reason' through to edit extra metadata, in cases where we merge or cluster releases.
| * pipenv: bump fuzzycat to 0.1.9Bryan Newbold2020-12-172-5/+5
| |
| * add fuzzy match filtering to DOAJ importerBryan Newbold2020-12-162-4/+23
| | | | | | | | | | | | | | | | | | | | | | In this default configuration, any entities with a fuzzy match (even "ambiguous") will be skipped at import time, to prevent creating duplicates. This is conservative towards not creating new/duplicate entities. In the future, as we get more confidence in fuzzy match/verification, we can start to ignore AMBIGUOUS, handle EXACT as same release, and merge STRONG (and WEAK?) matches under the same work entity.
| * add fuzzy matching helper to importer base classBryan Newbold2020-12-163-2/+147
| | | | | | | | Using fuzzycat. Add basic test coverage.
| * pipenv: add fuzzycat dependencyBryan Newbold2020-12-162-261/+374
| |
* | Merge pull request #65 from ibnesayeed/patch-1bnewbold2020-12-171-1/+1
|\ \ | | | | | | Improve status counting efficiency
| * | Improve status counting efficiencySawood Alam2020-12-171-1/+1
| | | | | | | | | When the input is large with a small number of unique items to be counted then counting as we go would be linear and more efficient approach than sorting and unique counting.
* | | Merge branch 'bnewbold-es-transform-html' into 'master'Martin Czygan2020-12-175-146/+296
|\ \ \ | |_|/ |/| | | | | | | | Elasticsearch release transform updates: handle webcaptures better, and refactoring See merge request webgroup/fatcat!91
| * | entity update worker: treat fileset and webcapture updates like file updatesBryan Newbold2020-12-161-3/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | When webcapture or fileset entities are updated, then the release entities associated with them also need to be updated (and work entities, recursively). A TODO is to handle the case where a release_id is *removed* as well as *added*, and reprocess the releases in that case as well.
| * | fix indentationBryan Newbold2020-12-161-2/+2
| | |
| * | have release elasticsearch transform count webcaptures and filesets towards ↵Bryan Newbold2020-12-161-26/+57
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | preservation These are simple/partial changes to have webcaptures and filesets show up in 'preservation', 'in_ia', and 'in_web' ES schema flags. A longer-term TODO is to update the ES schema to have more granular analytic flags. Also includes a small generalization refactor for URL object parsing into preservation status, shared across file+fileset+webcapture entity types (all have similar URL objects with url+rel fields).
| * | improve release elasticsearch transform test coverageBryan Newbold2020-12-163-11/+86
| | |
| * | small release_to_elasticsearch refactorsBryan Newbold2020-12-161-7/+12
| | | | | | | | | | | | | | | | | | | | | These should have almost no change in behavior, but improve code quality. The one behavior change is counting ftp URLs as "in_web"
| * | refactor release_to_elasticsearch transformBryan Newbold2020-12-161-131/+148
|/ / | | | | | | | | | | | | | | | | | | | | | | This method was huge an monolithic. This commit splits out the content and container specific sections into helper functions to make it more managable. This involved refactoring to make many flags ("is_*" and "in_*") part of the output dict through the entire code path, allowing simple update() calls on the dict. Noting that in the future should refactor to use a type-annotated class for the elasticsearch output object. Perhaps something auto-generated from the ES schema itself (JSON files).
* | html ingest: small fixes to try_update() code pathBryan Newbold2020-12-151-5/+5
| | | | | | | | | | Don't currently have test coverage for most try_update() code; run the inserts manually in testing.
* | notes on partial-progress DOAJ release metadata importBryan Newbold2020-12-141-0/+105
| |
* | bulk import notes on ORCIDBryan Newbold2020-12-141-0/+55
| |
* | Revert "gitlab CI: explicitly use xenial tag of image"Bryan Newbold2020-12-111-1/+1
| | | | | | | | This reverts commit dbfc6e9bacaab4960e814192d66eefea87ef8930.
* | Revert "docker xenial base image: include python3.8"Bryan Newbold2020-12-111-6/+1
| | | | | | | | This reverts commit 91628426678a635f26cf992dbd5caedb4a3ae24b.
* | gitlab CI: explicitly use xenial tag of imageBryan Newbold2020-12-111-1/+1
| |
* | docker xenial base image: include python3.8Bryan Newbold2020-12-111-1/+6
| |
* | HACK: squash intermitent failure of detect_text_lang() testBryan Newbold2020-12-111-1/+2
| | | | | | | | | | This is an open bug; it is important that tests pass on master branch however.
* | guide: small updates to container extra schema notes (from dblp work)Bryan Newbold2020-12-111-2/+7
| |
* | bulk edits: note ORCID updateBryan Newbold2020-12-111-1/+5
| |
* | docker: how to push to dockerhubBryan Newbold2020-12-111-0/+4
| |
* | Merge branch 'bnewbold-doaj-metadata' into 'master'Martin Czygan2020-11-2437-1549/+2845
|\ \ | | | | | | | | | | | | DOAJ article metadata import See merge request webgroup/fatcat!89
| * | cargo: update sentry to fix memory initialization issueBryan Newbold2020-11-202-274/+332
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Older sentry had an unsafe memory initialization error, which wasn't caught by older compilers. Rust 1.48 catches the problem at runtime and raises a panic. This meant that new builds (eg, on QA machine after update) were panic-ing. Newest versions of sentry have modern dependencies, which breaks our crufty old 'iron' dependency tree. Work-around is to only partially update (v0.12 to v0.15). This is a fairly frustrating situation. I'm hopeful that when we update to a different web framework and openapi generator 5.0 (not yet released), many of these dependency issues will be resolved, but i'm not certain. I did notice that if we entirely remove Sentry, which has not really been used much (only a small handful of issues reported over several years), we might be able to resolve openssl dependency issues.
| * | DOAJ: remove accidentally commited 'skip' of a testBryan Newbold2020-11-201-1/+0
| | |
| * | langdetect: more text for 'zh' test caseBryan Newbold2020-11-201-1/+1
| | | | | | | | | | | | | | | | | | This is an attempt to fix spurious test failures, in which this text block was getting detected as 'kr' on occasion. Apparently there is non-determinism in the langdetect package.
| * | DOAJ: update importer README with example invocationBryan Newbold2020-11-201-0/+7
| | |
| * | crossref+datacite: remove confusing early update bailBryan Newbold2020-11-202-4/+0
| | | | | | | | | | | | | | | Easy to miss that we skip updates *twice*, and with this early bailout were not updating counts correctly.
| * | doaj: fix update code path (getattr not __dict__)Bryan Newbold2020-11-203-15/+70
| | | | | | | | | | | | Also add missing code coverage for update path (disabled by default).
| * | DOAJ: handle empty identifier 'id' caseBryan Newbold2020-11-201-0/+2
| | |
| * | clean DOI: ban all non-ASCII charactersBryan Newbold2020-11-191-1/+4
| | | | | | | | | | | | | | | | | | | | | I believe this is safe and matches the regex filter in rust (fatcatd). Keep hitting one-off DOIs that were passing through python check, so being more strict from here forward.
| * | normal: handle langdetect of 'zh-cn' (not len=2)Bryan Newbold2020-11-191-0/+3
| | |
| * | update fatcatd rust code for 'oai' external identifierBryan Newbold2020-11-194-11/+189
| | |
| * | codegen rust schema crateBryan Newbold2020-11-196-3/+20
| | |
| * | codegen python openapi clientBryan Newbold2020-11-192-4/+36
| | |
| * | schema: also add 'oai' identifer (OAI-PMH) for releasesBryan Newbold2020-11-192-2/+9
| | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-23 19:15:19 +0000