aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #65 from ibnesayeed/patch-1bnewbold2020-12-171-1/+1
|\ | | | | Improve status counting efficiency
| * Improve status counting efficiencySawood Alam2020-12-171-1/+1
| | | | | | When the input is large with a small number of unique items to be counted then counting as we go would be linear and more efficient approach than sorting and unique counting.
* | Merge branch 'bnewbold-es-transform-html' into 'master'Martin Czygan2020-12-175-146/+296
|\ \ | | | | | | | | | | | | Elasticsearch release transform updates: handle webcaptures better, and refactoring See merge request webgroup/fatcat!91
| * | entity update worker: treat fileset and webcapture updates like file updatesBryan Newbold2020-12-161-3/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | When webcapture or fileset entities are updated, then the release entities associated with them also need to be updated (and work entities, recursively). A TODO is to handle the case where a release_id is *removed* as well as *added*, and reprocess the releases in that case as well.
| * | fix indentationBryan Newbold2020-12-161-2/+2
| | |
| * | have release elasticsearch transform count webcaptures and filesets towards ↵Bryan Newbold2020-12-161-26/+57
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | preservation These are simple/partial changes to have webcaptures and filesets show up in 'preservation', 'in_ia', and 'in_web' ES schema flags. A longer-term TODO is to update the ES schema to have more granular analytic flags. Also includes a small generalization refactor for URL object parsing into preservation status, shared across file+fileset+webcapture entity types (all have similar URL objects with url+rel fields).
| * | improve release elasticsearch transform test coverageBryan Newbold2020-12-163-11/+86
| | |
| * | small release_to_elasticsearch refactorsBryan Newbold2020-12-161-7/+12
| | | | | | | | | | | | | | | | | | | | | These should have almost no change in behavior, but improve code quality. The one behavior change is counting ftp URLs as "in_web"
| * | refactor release_to_elasticsearch transformBryan Newbold2020-12-161-131/+148
|/ / | | | | | | | | | | | | | | | | | | | | | | This method was huge an monolithic. This commit splits out the content and container specific sections into helper functions to make it more managable. This involved refactoring to make many flags ("is_*" and "in_*") part of the output dict through the entire code path, allowing simple update() calls on the dict. Noting that in the future should refactor to use a type-annotated class for the elasticsearch output object. Perhaps something auto-generated from the ES schema itself (JSON files).
* | html ingest: small fixes to try_update() code pathBryan Newbold2020-12-151-5/+5
| | | | | | | | | | Don't currently have test coverage for most try_update() code; run the inserts manually in testing.
* | notes on partial-progress DOAJ release metadata importBryan Newbold2020-12-141-0/+105
| |
* | bulk import notes on ORCIDBryan Newbold2020-12-141-0/+55
| |
* | Revert "gitlab CI: explicitly use xenial tag of image"Bryan Newbold2020-12-111-1/+1
| | | | | | | | This reverts commit dbfc6e9bacaab4960e814192d66eefea87ef8930.
* | Revert "docker xenial base image: include python3.8"Bryan Newbold2020-12-111-6/+1
| | | | | | | | This reverts commit 91628426678a635f26cf992dbd5caedb4a3ae24b.
* | gitlab CI: explicitly use xenial tag of imageBryan Newbold2020-12-111-1/+1
| |
* | docker xenial base image: include python3.8Bryan Newbold2020-12-111-1/+6
| |
* | HACK: squash intermitent failure of detect_text_lang() testBryan Newbold2020-12-111-1/+2
| | | | | | | | | | This is an open bug; it is important that tests pass on master branch however.
* | guide: small updates to container extra schema notes (from dblp work)Bryan Newbold2020-12-111-2/+7
| |
* | bulk edits: note ORCID updateBryan Newbold2020-12-111-1/+5
| |
* | docker: how to push to dockerhubBryan Newbold2020-12-111-0/+4
| |
* | Merge branch 'bnewbold-doaj-metadata' into 'master'Martin Czygan2020-11-2437-1549/+2845
|\ \ | | | | | | | | | | | | DOAJ article metadata import See merge request webgroup/fatcat!89
| * | cargo: update sentry to fix memory initialization issueBryan Newbold2020-11-202-274/+332
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Older sentry had an unsafe memory initialization error, which wasn't caught by older compilers. Rust 1.48 catches the problem at runtime and raises a panic. This meant that new builds (eg, on QA machine after update) were panic-ing. Newest versions of sentry have modern dependencies, which breaks our crufty old 'iron' dependency tree. Work-around is to only partially update (v0.12 to v0.15). This is a fairly frustrating situation. I'm hopeful that when we update to a different web framework and openapi generator 5.0 (not yet released), many of these dependency issues will be resolved, but i'm not certain. I did notice that if we entirely remove Sentry, which has not really been used much (only a small handful of issues reported over several years), we might be able to resolve openssl dependency issues.
| * | DOAJ: remove accidentally commited 'skip' of a testBryan Newbold2020-11-201-1/+0
| | |
| * | langdetect: more text for 'zh' test caseBryan Newbold2020-11-201-1/+1
| | | | | | | | | | | | | | | | | | This is an attempt to fix spurious test failures, in which this text block was getting detected as 'kr' on occasion. Apparently there is non-determinism in the langdetect package.
| * | DOAJ: update importer README with example invocationBryan Newbold2020-11-201-0/+7
| | |
| * | crossref+datacite: remove confusing early update bailBryan Newbold2020-11-202-4/+0
| | | | | | | | | | | | | | | Easy to miss that we skip updates *twice*, and with this early bailout were not updating counts correctly.
| * | doaj: fix update code path (getattr not __dict__)Bryan Newbold2020-11-203-15/+70
| | | | | | | | | | | | Also add missing code coverage for update path (disabled by default).
| * | DOAJ: handle empty identifier 'id' caseBryan Newbold2020-11-201-0/+2
| | |
| * | clean DOI: ban all non-ASCII charactersBryan Newbold2020-11-191-1/+4
| | | | | | | | | | | | | | | | | | | | | I believe this is safe and matches the regex filter in rust (fatcatd). Keep hitting one-off DOIs that were passing through python check, so being more strict from here forward.
| * | normal: handle langdetect of 'zh-cn' (not len=2)Bryan Newbold2020-11-191-0/+3
| | |
| * | update fatcatd rust code for 'oai' external identifierBryan Newbold2020-11-194-11/+189
| | |
| * | codegen rust schema crateBryan Newbold2020-11-196-3/+20
| | |
| * | codegen python openapi clientBryan Newbold2020-11-192-4/+36
| | |
| * | schema: also add 'oai' identifer (OAI-PMH) for releasesBryan Newbold2020-11-192-2/+9
| | |
| * | tweak DOAJ importer class args and default for do_updatesBryan Newbold2020-11-191-2/+2
| | |
| * | show DOAJ (and dblp) identifiers in release viewBryan Newbold2020-11-191-1/+7
| | |
| * | if a release has DOAJ article id, count as OABryan Newbold2020-11-191-0/+3
| | |
| * | implement remainder of DOAJ article importerBryan Newbold2020-11-193-68/+168
| | |
| * | handle more non-ASCII DOI casesBryan Newbold2020-11-191-1/+3
| | |
| * | more python normalizers, and move from importer commonBryan Newbold2020-11-192-154/+326
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Moved several normalizer helpers out of fatcat_tools.importers.common to fatcat_tools.normal. Copied language name and country name parser helpers from chocula repository (built on existing pycountry helper library). Have not gone through and refactored other importers to point to these helpers yet; that should be a separate PR when this branch is merged. Current changes are backwards compatible via re-imports.
| * | initial implementation of DOAJ importerBryan Newbold2020-11-194-0/+387
| | | | | | | | | | | | Several things to finish implementing and polish.
| * | python API client: resolve warning about '\d' in stringBryan Newbold2020-11-191-2/+2
| | |
| * | rustfmtBryan Newbold2020-11-195-87/+138
| | |
| * | rust: fatcatd changes for DOAJ+dblp identifiersBryan Newbold2020-11-196-949/+1062
| | |
| * | codegen rust crate for v0.3.3Bryan Newbold2020-11-198-227/+244
| | |
| * | codegen python client library for v0.3.3Bryan Newbold2020-11-197-16/+80
| | |
| * | schema: DOAJ+dblp ext_ids; bump to v0.3.3Bryan Newbold2020-11-192-1/+25
|/ /
* | ingest and proposal updatesBryan Newbold2020-11-192-0/+45
| |
* | Merge branch 'bnewbold-xml-html-ingest' into 'master'Martin Czygan2020-11-1910-66/+409
|\ \ | |/ |/| | | | | HTML webcapture ingest (and XML file ingest) See merge request webgroup/fatcat!88
| * html ingest: actual xhtml mimetypeBryan Newbold2020-11-161-2/+2
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-24 17:46:26 +0000