aboutsummaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'bnewbold-container-web' into 'master'bnewbold2022-03-103-2/+185
|\ | | | | | | | | container web interface improvements See merge request webgroup/fatcat!140
| * move container_status ES query code from fatcat_web to fatcat_toolsBryan Newbold2022-02-093-2/+185
| | | | | | | | | | | | The main motivation is to never have fatcat_tools import from fatcat_web, only vica-versa. Some code in fatcat_tools needs container stats, so starting with that code path (plus some generic helpers).
* | entity updates: don't try to ingest arxiv DOIs (for now)Bryan Newbold2022-02-281-0/+2
| |
* | datacite importer: skip container_id for some repository sourcesBryan Newbold2022-02-091-0/+34
|/
* doaj importer: TODO note to skip some larger publishersBryan Newbold2022-02-091-0/+4
|
* container ES transform: include old extra.issne/p fieldsBryan Newbold2022-02-031-1/+4
| | | | | These were removed prematurely. Not all containers have been updated to use these fields yet.
* Merge branch 'bnewbold-file-es' into 'master'bnewbold2022-01-213-4/+38
|\ | | | | | | | | File entity elasticsearch index worker See merge request webgroup/fatcat!136
| * entity worker: expand creators in release entitiesBryan Newbold2021-12-151-1/+1
| |
| * small default config typo fixes for elasticsearch workersBryan Newbold2021-12-151-2/+2
| |
| * file elasticsearch index workerBryan Newbold2021-12-152-1/+35
| |
* | crossref importer: skip affiliations lacking 'name'Bryan Newbold2021-12-151-0/+3
|/ | | | Relatedly, we should start handling ROR affiliations in contribs soon.
* mergers: fix typo in env var nameBryan Newbold2021-12-073-3/+3
|
* ES container schema: add 'sim_pubid' and `ia_sim_collection` fieldsBryan Newbold2021-12-031-0/+2
|
* ES transform: remove prototype microfilm linksBryan Newbold2021-12-031-20/+0
| | | | This ended up being a feature in scholar.archive.org, not fatcat.
* chocula importer: handle not-upper-case ISSNsBryan Newbold2021-11-301-2/+6
|
* chocula importer: handle broken ISSNs in extra metadataBryan Newbold2021-11-301-2/+7
|
* chocula importer: tweak counting, conditions for doing updatesBryan Newbold2021-11-301-15/+7
|
* chocula importer: move issne/issnp 'extra' to top-level fields if doing updatesBryan Newbold2021-11-301-0/+6
|
* chocula: don't do name cleanups in importerBryan Newbold2021-11-301-8/+2
| | | | This kind of cleanup should be done in 'chocula' instead.
* container merger: fix bug with filtering by release countBryan Newbold2021-11-301-13/+15
| | | | | Also apply the "human edit" and "release count" checks only to the dupe (to-be-redirected) idents.
* release merger: same editgroup_id fixes as for file and container mergersBryan Newbold2021-11-241-1/+5
|
* container merger: fixes from QA testingBryan Newbold2021-11-241-8/+13
|
* mergers: don't try to accept empty editgroups in dry-run-modeBryan Newbold2021-11-241-2/+4
|
* ES release transform: handle redirected containers betterBryan Newbold2021-11-241-1/+1
| | | | | Despite the inline comment, we were not actually grabbing the "redirected" ident correctly, meaning some counts would not be accurate.
* container merger: defer allocation of editgroup_id; and dummy code pathBryan Newbold2021-11-241-1/+5
|
* initial implementation of container mergerBryan Newbold2021-11-241-0/+237
|
* file merger: allocate editgroup id later in 'merge' processBryan Newbold2021-11-241-1/+5
| | | | | The motivation is to avoid creating empty editgroups in dry-run mode, and when all entities are "skipped"
* Merge branch 'bnewbold-mergers' into 'master'bnewbold2021-11-254-0/+640
|\ | | | | | | | | entity mergers framework See merge request webgroup/fatcat!133
| * mergers common: remove inaccurate commentBryan Newbold2021-11-241-2/+0
| | | | | | | | Caught in review, thanks miku
| * file merger: add content_scope to list of merged fieldsBryan Newbold2021-11-241-1/+1
| |
| * release merger: some progress, but also disable (not complete)Bryan Newbold2021-11-231-12/+72
| |
| * file merges: fixes from testing in QABryan Newbold2021-11-231-14/+23
| |
| * mergers: small tweaksBryan Newbold2021-11-232-3/+3
| |
| * mergers: remove entity mergers from __init__ (to work around warning)Bryan Newbold2021-11-231-2/+0
| |
| * initial file merger, with testsBryan Newbold2021-11-231-0/+228
| |
| * mergers: fmt, lint, refactorsBryan Newbold2021-11-233-96/+200
| | | | | | | | | | These old merger code is from an old branch and needed significant cleanup
| * first iteration of mergersBryan Newbold2021-11-233-0/+243
| |
* | codespell fixes in python code (comments)Bryan Newbold2021-11-242-3/+3
|/
* content_scope: include in file ES schema and transformBryan Newbold2021-11-171-0/+1
|
* Merge branch 'bnewbold-import-refactors' into 'master'bnewbold2021-11-1118-1462/+811
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | import refactors and deprecations Some of these are from old stale branches (the datacite subject metadata patch), but most are from yesterday and today. Sort of a hodge-podge, but the general theme is getting around to deferred cleanups and refactors specific to importer code before making some behavioral changes. The Datacite-specific stuff could use review here. Remove unused/deprecated/dead code: - cdl_dash_dat and wayback_static importers, which were for specific early example entities and have been superseded by other importers - "extid map" sqlite3 feature from several importers, was only used for initial bulk imports (and maybe should not have been used) Refactors: - moved a number of large datastructures out of importer code and into a dedicated static file (`biblio_lookup_tables.py`). Didn't move all, just the ones that were either generic or very large (making it hard to read code) - shuffled around relative imports and some function names ("clean_str" vs. "clean") Some actual behavioral changes: - remove some Datacite-specific license slugs - stop trying to fix double-slashes in DOIs, that was causing more harm than help (some DOIs do actually have double-slashes!) - remove some excess metadata from datacite 'extra' fields
| * improve lookup_license_slug helper and lookup tableBryan Newbold2021-11-102-56/+62
| |
| * refactor importer metadata tables into separate file; move some helpers aroundBryan Newbold2021-11-1010-702/+682
| | | | | | | | | | | | | | - MAX_ABSTRACT_LENGTH set in a single place (importer common) - merge datacite license slug table in to common table, removing some TDM-specific licenses (which do not apply in the context of preserving the full work)
| * importers: refactor imports of clean() and other normalization helpersBryan Newbold2021-11-1012-95/+104
| |
| * remove cdl_dash_dat and wayback_static importersBryan Newbold2021-11-103-510/+0
| | | | | | | | | | | | | | | | Cleaning out dead code. These importers were used to create demonstration fileset and webcapture entities early in development. They have been replaced by the fileset and webcapture ingest importers.
| * datacite import: store less subject metadataBryan Newbold2021-11-101-1/+7
| | | | | | | | | | | | | | | | Many of these 'subject' objects have the equivalent of several lines of text, with complex URLs that don't compress well. I think it is fine we have included these thus far instead of parsing more deeply, but going forward I don't think this nested 'extra' metadata is worth the database space.
| * importers: use clean_doi() in many more (all?) importersBryan Newbold2021-11-096-12/+29
| |
| * clean_doi: stop mutating double-slash DOIs, except for 10.1037 prefixBryan Newbold2021-11-091-1/+2
| |
| * remove deprecated extid sqlite3 lookup table feature from importersBryan Newbold2021-11-093-160/+0
| | | | | | | | | | | | | | | | This was used during initial bulk imports, but is no longer used and could create serious metadata problems if used accidentially. In retrospect, it also made metadata provenance less transparent, and may have done more harm than good overall.
* | Merge branch 'bnewbold-cleanups-nov2021' into 'master'bnewbold2021-11-114-0/+748
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fatcat metadata cleanups/fixups, November 2021 Three cleanups implemented in this branch: - update non-lowercase DOIs on releases (couple hundred thousand entities) - fix incorrectly imported file/release pairs, on the file entity side (~250k entities) - expand truncated wayback URL timestamps in file entities (up to 10 million entities) Instead of proposals, there are documents for each cleanup in `notes/cleanups/`. Have done spot testing of tens of thousands of entities each in QA, and confident about running in production. Plan is to run updates in the order above. DOI and bugfix updates will go fairly fast; the wayback timestamp updates will go slower, and result in large re-indexing load both in fatcat and scholar, because both release and work entities will get triggered for update when file entities are updated.
| * | file/release bugfix: handle files with multiple editsBryan Newbold2021-11-091-6/+6
| | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-29 14:00:12 +0000