| Commit message (Collapse) | Author | Age | Files | Lines | ||
|---|---|---|---|---|---|---|
| ... | ||||||
| | * | rust: bump crate version and lockfile | Bryan Newbold | 2021-11-17 | 2 | -3/+3 | |
| | | | ||||||
| | * | rust: implement content_scope | Bryan Newbold | 2021-11-17 | 5 | -0/+22 | |
| | | | ||||||
| | * | SQL implementation of content_scope | Bryan Newbold | 2021-11-17 | 2 | -0/+36 | |
| | | | ||||||
| | * | codegen rust code for content_scope | Bryan Newbold | 2021-11-17 | 3 | -4/+19 | |
| | | | ||||||
| | * | schema: add content_scope fields, and bump to 0.4.1 | Bryan Newbold | 2021-11-17 | 1 | -1/+10 | |
| | | | ||||||
| | * | proposal: content_scope field | Bryan Newbold | 2021-11-17 | 1 | -0/+84 | |
| | | | ||||||
| * | | typo: don't expand containers for release revs (TOML) | Bryan Newbold | 2021-11-19 | 1 | -1/+1 | |
| | | | ||||||
| * | | web editgroup diff: don't enrich in TOML diff; fix overlapping break | Bryan Newbold | 2021-11-19 | 2 | -5/+8 | |
| | | | ||||||
| * | | web generic entity helpers: make enrichment optional | Bryan Newbold | 2021-11-19 | 1 | -18/+49 | |
| | | | ||||||
| * | | polish editgroup diff view | Bryan Newbold | 2021-11-18 | 4 | -92/+83 | |
| | | | | | | | | | Still not as great as it could be, but useful in this state. | |||||
| * | | initial implementation of editgroup 'diff' for review | Bryan Newbold | 2021-11-17 | 4 | -6/+183 | |
| | | | ||||||
| * | | web: fix API URL link for review pages of entities | Bryan Newbold | 2021-11-17 | 1 | -2/+2 | |
| |/ | ||||||
| * | updated notes on possible cleanups | Bryan Newbold | 2021-11-17 | 1 | -4/+27 | |
| | | ||||||
| * | ISSN-L dupes check: output all matches | Bryan Newbold | 2021-11-17 | 1 | -1/+1 | |
| | | ||||||
| * | document cleanups run this week | Bryan Newbold | 2021-11-12 | 5 | -0/+244 | |
| | | ||||||
| * | web: handle ES non-int error codes better | Bryan Newbold | 2021-11-12 | 1 | -9/+12 | |
| | | ||||||
| * | Merge branch 'bnewbold-import-refactors' into 'master' | bnewbold | 2021-11-11 | 27 | -1599/+874 | |
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | import refactors and deprecations Some of these are from old stale branches (the datacite subject metadata patch), but most are from yesterday and today. Sort of a hodge-podge, but the general theme is getting around to deferred cleanups and refactors specific to importer code before making some behavioral changes. The Datacite-specific stuff could use review here. Remove unused/deprecated/dead code: - cdl_dash_dat and wayback_static importers, which were for specific early example entities and have been superseded by other importers - "extid map" sqlite3 feature from several importers, was only used for initial bulk imports (and maybe should not have been used) Refactors: - moved a number of large datastructures out of importer code and into a dedicated static file (`biblio_lookup_tables.py`). Didn't move all, just the ones that were either generic or very large (making it hard to read code) - shuffled around relative imports and some function names ("clean_str" vs. "clean") Some actual behavioral changes: - remove some Datacite-specific license slugs - stop trying to fix double-slashes in DOIs, that was causing more harm than help (some DOIs do actually have double-slashes!) - remove some excess metadata from datacite 'extra' fields | |||||
| | * | update datacite tests for license slug changes | Bryan Newbold | 2021-11-10 | 2 | -8/+7 | |
| | | | | | | | | | | | Use datacite-specific wrapper function, and remove a couple non-OA/TDM-limited licenses. | |||||
| | * | improve lookup_license_slug helper and lookup table | Bryan Newbold | 2021-11-10 | 2 | -56/+62 | |
| | | | ||||||
| | * | refactor importer metadata tables into separate file; move some helpers around | Bryan Newbold | 2021-11-10 | 10 | -702/+682 | |
| | | | | | | | | | | | | | | | - MAX_ABSTRACT_LENGTH set in a single place (importer common) - merge datacite license slug table in to common table, removing some TDM-specific licenses (which do not apply in the context of preserving the full work) | |||||
| | * | importers: refactor imports of clean() and other normalization helpers | Bryan Newbold | 2021-11-10 | 12 | -95/+104 | |
| | | | ||||||
| | * | remove cdl_dash_dat and wayback_static importers | Bryan Newbold | 2021-11-10 | 4 | -596/+0 | |
| | | | | | | | | | | | | | | | | | Cleaning out dead code. These importers were used to create demonstration fileset and webcapture entities early in development. They have been replaced by the fileset and webcapture ingest importers. | |||||
| | * | datacite import: store less subject metadata | Bryan Newbold | 2021-11-10 | 1 | -1/+7 | |
| | | | | | | | | | | | | | | | | | Many of these 'subject' objects have the equivalent of several lines of text, with complex URLs that don't compress well. I think it is fine we have included these thus far instead of parsing more deeply, but going forward I don't think this nested 'extra' metadata is worth the database space. | |||||
| | * | add notes about 'double slash in DOI' issue | Bryan Newbold | 2021-11-09 | 1 | -0/+46 | |
| | | | ||||||
| | * | importers: use clean_doi() in many more (all?) importers | Bryan Newbold | 2021-11-09 | 6 | -12/+29 | |
| | | | ||||||
| | * | clean_doi: stop mutating double-slash DOIs, except for 10.1037 prefix | Bryan Newbold | 2021-11-09 | 1 | -1/+2 | |
| | | | ||||||
| | * | remove deprecated extid sqlite3 lookup table feature from importers | Bryan Newbold | 2021-11-09 | 10 | -203/+10 | |
| | | | | | | | | | | | | | | | | | This was used during initial bulk imports, but is no longer used and could create serious metadata problems if used accidentially. In retrospect, it also made metadata provenance less transparent, and may have done more harm than good overall. | |||||
| * | | Merge branch 'bnewbold-cleanups-nov2021' into 'master' | bnewbold | 2021-11-11 | 9 | -1/+1504 | |
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fatcat metadata cleanups/fixups, November 2021 Three cleanups implemented in this branch: - update non-lowercase DOIs on releases (couple hundred thousand entities) - fix incorrectly imported file/release pairs, on the file entity side (~250k entities) - expand truncated wayback URL timestamps in file entities (up to 10 million entities) Instead of proposals, there are documents for each cleanup in `notes/cleanups/`. Have done spot testing of tens of thousands of entities each in QA, and confident about running in production. Plan is to run updates in the order above. DOI and bugfix updates will go fairly fast; the wayback timestamp updates will go slower, and result in large re-indexing load both in fatcat and scholar, because both release and work entities will get triggered for update when file entities are updated. | |||||
| | * | | wayback ts cleanup: one more filter tweak | Bryan Newbold | 2021-11-09 | 1 | -1/+2 | |
| | | | | ||||||
| | * | | update cleanups notes | Bryan Newbold | 2021-11-09 | 2 | -0/+72 | |
| | | | | ||||||
| | * | | file/release bugfix: handle files with multiple edits | Bryan Newbold | 2021-11-09 | 1 | -6/+6 | |
| | | | | ||||||
| | * | | cleanups: add more state=active checks | Bryan Newbold | 2021-11-09 | 2 | -0/+8 | |
| | | | | ||||||
| | * | | update link source filters in file/release bugfix | Bryan Newbold | 2021-11-09 | 1 | -2/+8 | |
| | | | | ||||||
| | * | | initial file/release bugfix cleanup worker and notes | Bryan Newbold | 2021-11-09 | 2 | -0/+375 | |
| | | | | ||||||
| | * | | updates to lowercase DOI cleanup | Bryan Newbold | 2021-11-09 | 2 | -7/+86 | |
| | | | | ||||||
| | * | | lowercase DOI lint and check entity status | Bryan Newbold | 2021-11-09 | 1 | -4/+5 | |
| | | | | ||||||
| | * | | more iteration on short wayback timestamp cleanup | Bryan Newbold | 2021-11-09 | 3 | -4/+129 | |
| | | | | ||||||
| | * | | lint: minor import tweak | Bryan Newbold | 2021-11-09 | 1 | -1/+1 | |
| | | | | ||||||
| | * | | cleanups: tweaks to wayback CDX cleanup scripts | Bryan Newbold | 2021-11-09 | 2 | -6/+21 | |
| | | | | ||||||
| | * | | cleanups: initial lowercase DOI cleanup script | Bryan Newbold | 2021-11-09 | 1 | -0/+145 | |
| | | | | ||||||
| | * | | wayback short ts: another regression test, and some small fmt/tweaks | Bryan Newbold | 2021-11-09 | 1 | -3/+38 | |
| | | | | ||||||
| | * | | wayback cleanup: actually update entity | Bryan Newbold | 2021-11-09 | 1 | -2/+4 | |
| | | | | ||||||
| | * | | imports: generic file cleanup removes exact duplicate URLs | Bryan Newbold | 2021-11-09 | 1 | -0/+9 | |
| | | | | ||||||
| | * | | wayback short ts: add regression test for dupe URLs | Bryan Newbold | 2021-11-09 | 1 | -0/+44 | |
| | | | | ||||||
| | * | | short wayback ts: initial cleanup script implementation | Bryan Newbold | 2021-11-09 | 1 | -0/+251 | |
| | | | | ||||||
| | * | | wayback timestamps: updates to handle 4-digit case | Bryan Newbold | 2021-11-09 | 2 | -11/+108 | |
| | | | | ||||||
| | * | | start work on wayback short-timestamp cleanup | Bryan Newbold | 2021-11-09 | 2 | -0/+238 | |
| | |/ | ||||||
| * | | update crawlability docs | Bryan Newbold | 2021-11-10 | 1 | -1/+9 | |
| | | | ||||||
| * | | sitemap generation improvements | Bryan Newbold | 2021-11-10 | 2 | -1/+2 | |
| | | | ||||||
| * | | start notes/proposal about 'crawlability' improvements | Bryan Newbold | 2021-11-10 | 1 | -0/+68 | |
| | | | ||||||
 |
index : fatcat | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |