summaryrefslogtreecommitdiffstats
path: root/python
Commit message (Collapse)AuthorAgeFilesLines
* mergers: remove entity mergers from __init__ (to work around warning)Bryan Newbold2021-11-231-2/+0
|
* initial file merger, with testsBryan Newbold2021-11-232-0/+388
|
* mergers: fmt, lint, refactorsBryan Newbold2021-11-233-96/+200
| | | | | These old merger code is from an old branch and needed significant cleanup
* remove top-level fatcat_merge.py; going to call module __main__ going forwardBryan Newbold2021-11-231-112/+0
|
* first iteration of mergersBryan Newbold2021-11-234-0/+355
|
* Merge branch 'bnewbold-content-scope'Bryan Newbold2021-11-225-1/+8
|\
| * bump python client to 0.5.0Bryan Newbold2021-11-171-1/+1
| |
| * content_scope: include in file ES schema and transformBryan Newbold2021-11-171-0/+1
| |
| * minimal python test coverage of content_scope fieldsBryan Newbold2021-11-173-0/+6
| |
| * python code: update python_openapi_client in lockfileBryan Newbold2021-11-171-1/+1
| |
* | typo: don't expand containers for release revs (TOML)Bryan Newbold2021-11-191-1/+1
| |
* | web editgroup diff: don't enrich in TOML diff; fix overlapping breakBryan Newbold2021-11-192-5/+8
| |
* | web generic entity helpers: make enrichment optionalBryan Newbold2021-11-191-18/+49
| |
* | polish editgroup diff viewBryan Newbold2021-11-184-92/+83
| | | | | | | | Still not as great as it could be, but useful in this state.
* | initial implementation of editgroup 'diff' for reviewBryan Newbold2021-11-174-6/+183
| |
* | web: fix API URL link for review pages of entitiesBryan Newbold2021-11-171-2/+2
|/
* web: handle ES non-int error codes betterBryan Newbold2021-11-121-9/+12
|
* Merge branch 'bnewbold-import-refactors' into 'master'bnewbold2021-11-1126-1599/+828
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | import refactors and deprecations Some of these are from old stale branches (the datacite subject metadata patch), but most are from yesterday and today. Sort of a hodge-podge, but the general theme is getting around to deferred cleanups and refactors specific to importer code before making some behavioral changes. The Datacite-specific stuff could use review here. Remove unused/deprecated/dead code: - cdl_dash_dat and wayback_static importers, which were for specific early example entities and have been superseded by other importers - "extid map" sqlite3 feature from several importers, was only used for initial bulk imports (and maybe should not have been used) Refactors: - moved a number of large datastructures out of importer code and into a dedicated static file (`biblio_lookup_tables.py`). Didn't move all, just the ones that were either generic or very large (making it hard to read code) - shuffled around relative imports and some function names ("clean_str" vs. "clean") Some actual behavioral changes: - remove some Datacite-specific license slugs - stop trying to fix double-slashes in DOIs, that was causing more harm than help (some DOIs do actually have double-slashes!) - remove some excess metadata from datacite 'extra' fields
| * update datacite tests for license slug changesBryan Newbold2021-11-102-8/+7
| | | | | | | | | | Use datacite-specific wrapper function, and remove a couple non-OA/TDM-limited licenses.
| * improve lookup_license_slug helper and lookup tableBryan Newbold2021-11-102-56/+62
| |
| * refactor importer metadata tables into separate file; move some helpers aroundBryan Newbold2021-11-1010-702/+682
| | | | | | | | | | | | | | - MAX_ABSTRACT_LENGTH set in a single place (importer common) - merge datacite license slug table in to common table, removing some TDM-specific licenses (which do not apply in the context of preserving the full work)
| * importers: refactor imports of clean() and other normalization helpersBryan Newbold2021-11-1012-95/+104
| |
| * remove cdl_dash_dat and wayback_static importersBryan Newbold2021-11-104-596/+0
| | | | | | | | | | | | | | | | Cleaning out dead code. These importers were used to create demonstration fileset and webcapture entities early in development. They have been replaced by the fileset and webcapture ingest importers.
| * datacite import: store less subject metadataBryan Newbold2021-11-101-1/+7
| | | | | | | | | | | | | | | | Many of these 'subject' objects have the equivalent of several lines of text, with complex URLs that don't compress well. I think it is fine we have included these thus far instead of parsing more deeply, but going forward I don't think this nested 'extra' metadata is worth the database space.
| * importers: use clean_doi() in many more (all?) importersBryan Newbold2021-11-096-12/+29
| |
| * clean_doi: stop mutating double-slash DOIs, except for 10.1037 prefixBryan Newbold2021-11-091-1/+2
| |
| * remove deprecated extid sqlite3 lookup table feature from importersBryan Newbold2021-11-0910-203/+10
| | | | | | | | | | | | | | | | This was used during initial bulk imports, but is no longer used and could create serious metadata problems if used accidentially. In retrospect, it also made metadata provenance less transparent, and may have done more harm than good overall.
* | Merge branch 'bnewbold-cleanups-nov2021' into 'master'bnewbold2021-11-115-1/+749
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fatcat metadata cleanups/fixups, November 2021 Three cleanups implemented in this branch: - update non-lowercase DOIs on releases (couple hundred thousand entities) - fix incorrectly imported file/release pairs, on the file entity side (~250k entities) - expand truncated wayback URL timestamps in file entities (up to 10 million entities) Instead of proposals, there are documents for each cleanup in `notes/cleanups/`. Have done spot testing of tens of thousands of entities each in QA, and confident about running in production. Plan is to run updates in the order above. DOI and bugfix updates will go fairly fast; the wayback timestamp updates will go slower, and result in large re-indexing load both in fatcat and scholar, because both release and work entities will get triggered for update when file entities are updated.
| * | file/release bugfix: handle files with multiple editsBryan Newbold2021-11-091-6/+6
| | |
| * | cleanups: add more state=active checksBryan Newbold2021-11-092-0/+8
| | |
| * | update link source filters in file/release bugfixBryan Newbold2021-11-091-2/+8
| | |
| * | initial file/release bugfix cleanup worker and notesBryan Newbold2021-11-091-0/+231
| | |
| * | updates to lowercase DOI cleanupBryan Newbold2021-11-091-7/+15
| | |
| * | lowercase DOI lint and check entity statusBryan Newbold2021-11-091-4/+5
| | |
| * | more iteration on short wayback timestamp cleanupBryan Newbold2021-11-091-1/+1
| | |
| * | lint: minor import tweakBryan Newbold2021-11-091-1/+1
| | |
| * | cleanups: tweaks to wayback CDX cleanup scriptsBryan Newbold2021-11-091-5/+13
| | |
| * | cleanups: initial lowercase DOI cleanup scriptBryan Newbold2021-11-091-0/+145
| | |
| * | wayback short ts: another regression test, and some small fmt/tweaksBryan Newbold2021-11-091-3/+38
| | |
| * | wayback cleanup: actually update entityBryan Newbold2021-11-091-2/+4
| | |
| * | imports: generic file cleanup removes exact duplicate URLsBryan Newbold2021-11-091-0/+9
| | |
| * | wayback short ts: add regression test for dupe URLsBryan Newbold2021-11-091-0/+44
| | |
| * | short wayback ts: initial cleanup script implementationBryan Newbold2021-11-091-0/+251
| |/
* / pubmed: allow updates if PMCID does not exist yetBryan Newbold2021-11-101-1/+6
|/ | | | | | | | | | | The intent of this change is to start updating Pubmed metadata records when a PMCID has been assigned, but that ext_id hasn't been recorded in fatcat yet. It is likely that this change will result in some additional duplicate PMCIDs in the catalog. But the principle is that the PMID is the primary pubmed identifier, and all records with a PMID should have the PMCID that pubmed indicates, even if there exists another incorrect record.
* python tests: verify array sort orderBryan Newbold2021-11-054-20/+18
| | | | | | | In a couple cases (eg, filesets), had made tests agnostic to sort order, because the sort order was not stable. In other cases, simply small cleanups and comment improvements.
* enable type annotation checking with flake8 by default ('make lint')Bryan Newbold2021-11-031-4/+2
|
* cleanups: create a separate JsonLinePusher for cleanup workers (distinct ↵Bryan Newbold2021-11-033-4/+20
| | | | base class)
* facat_import.py: work around corner case in run_cdl_dash_dat()Bryan Newbold2021-11-031-1/+1
|
* datacite importer: remove unused 'year_only' variableBryan Newbold2021-11-031-2/+3
|
* web: work around remaining type annotation issuesBryan Newbold2021-11-032-11/+15
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-15 11:02:30 +0000