| Commit message (Collapse) | Author | Age | Files | Lines | |
|---|---|---|---|---|---|
| * | fileset ingest: handle missing/partial file-level metadata | Bryan Newbold | 2022-04-05 | 1 | -3/+3 |
| | | |||||
| * | ingest importer: improved extra/edit_extra code flow | Bryan Newbold | 2022-04-05 | 1 | -20/+13 |
| | | |||||
| * | fileset ingest: remove a TODO | Bryan Newbold | 2022-04-04 | 1 | -1/+0 |
| | | |||||
| * | filesets: typo bugfix, and test 'mimetype' on entity, not extra | Bryan Newbold | 2022-04-04 | 1 | -1/+1 |
| | | |||||
| * | fileset ingest: fix mimetype handling | Bryan Newbold | 2022-03-31 | 1 | -4/+5 |
| | | |||||
| * | bugfix: logic flow in fileset release checking | Bryan Newbold | 2022-03-23 | 1 | -3/+6 |
| | | |||||
| * | single-file variant of fileset importer for dataset attempts | Bryan Newbold | 2022-03-23 | 2 | -0/+202 |
| | | |||||
| * | fix typo in fileset comparison helper | Bryan Newbold | 2022-03-23 | 1 | -1/+1 |
| | | |||||
| * | ingest fileset fixes, and some test coverage | Bryan Newbold | 2022-03-23 | 2 | -13/+30 |
| | | |||||
| * | dataset ingest: JSON object fixes | Bryan Newbold | 2022-03-22 | 1 | -5/+5 |
| | | |||||
| * | Merge branch 'bnewbold-container-web' into 'master' | bnewbold | 2022-03-10 | 3 | -2/+185 |
| |\ | | | | | | | | | container web interface improvements See merge request webgroup/fatcat!140 | ||||
| | * | move container_status ES query code from fatcat_web to fatcat_tools | Bryan Newbold | 2022-02-09 | 3 | -2/+185 |
| | | | | | | | | | | | | | The main motivation is to never have fatcat_tools import from fatcat_web, only vica-versa. Some code in fatcat_tools needs container stats, so starting with that code path (plus some generic helpers). | ||||
| * | | entity updates: don't try to ingest arxiv DOIs (for now) | Bryan Newbold | 2022-02-28 | 1 | -0/+2 |
| | | | |||||
| * | | datacite importer: skip container_id for some repository sources | Bryan Newbold | 2022-02-09 | 1 | -0/+34 |
| |/ | |||||
| * | doaj importer: TODO note to skip some larger publishers | Bryan Newbold | 2022-02-09 | 1 | -0/+4 |
| | | |||||
| * | container ES transform: include old extra.issne/p fields | Bryan Newbold | 2022-02-03 | 1 | -1/+4 |
| | | | | | | These were removed prematurely. Not all containers have been updated to use these fields yet. | ||||
| * | Merge branch 'bnewbold-file-es' into 'master' | bnewbold | 2022-01-21 | 3 | -4/+38 |
| |\ | | | | | | | | | File entity elasticsearch index worker See merge request webgroup/fatcat!136 | ||||
| | * | entity worker: expand creators in release entities | Bryan Newbold | 2021-12-15 | 1 | -1/+1 |
| | | | |||||
| | * | small default config typo fixes for elasticsearch workers | Bryan Newbold | 2021-12-15 | 1 | -2/+2 |
| | | | |||||
| | * | file elasticsearch index worker | Bryan Newbold | 2021-12-15 | 2 | -1/+35 |
| | | | |||||
| * | | crossref importer: skip affiliations lacking 'name' | Bryan Newbold | 2021-12-15 | 1 | -0/+3 |
| |/ | | | | Relatedly, we should start handling ROR affiliations in contribs soon. | ||||
| * | mergers: fix typo in env var name | Bryan Newbold | 2021-12-07 | 3 | -3/+3 |
| | | |||||
| * | ES container schema: add 'sim_pubid' and `ia_sim_collection` fields | Bryan Newbold | 2021-12-03 | 1 | -0/+2 |
| | | |||||
| * | ES transform: remove prototype microfilm links | Bryan Newbold | 2021-12-03 | 1 | -20/+0 |
| | | | | | This ended up being a feature in scholar.archive.org, not fatcat. | ||||
| * | chocula importer: handle not-upper-case ISSNs | Bryan Newbold | 2021-11-30 | 1 | -2/+6 |
| | | |||||
| * | chocula importer: handle broken ISSNs in extra metadata | Bryan Newbold | 2021-11-30 | 1 | -2/+7 |
| | | |||||
| * | chocula importer: tweak counting, conditions for doing updates | Bryan Newbold | 2021-11-30 | 1 | -15/+7 |
| | | |||||
| * | chocula importer: move issne/issnp 'extra' to top-level fields if doing updates | Bryan Newbold | 2021-11-30 | 1 | -0/+6 |
| | | |||||
| * | chocula: don't do name cleanups in importer | Bryan Newbold | 2021-11-30 | 1 | -8/+2 |
| | | | | | This kind of cleanup should be done in 'chocula' instead. | ||||
| * | container merger: fix bug with filtering by release count | Bryan Newbold | 2021-11-30 | 1 | -13/+15 |
| | | | | | | Also apply the "human edit" and "release count" checks only to the dupe (to-be-redirected) idents. | ||||
| * | release merger: same editgroup_id fixes as for file and container mergers | Bryan Newbold | 2021-11-24 | 1 | -1/+5 |
| | | |||||
| * | container merger: fixes from QA testing | Bryan Newbold | 2021-11-24 | 1 | -8/+13 |
| | | |||||
| * | mergers: don't try to accept empty editgroups in dry-run-mode | Bryan Newbold | 2021-11-24 | 1 | -2/+4 |
| | | |||||
| * | ES release transform: handle redirected containers better | Bryan Newbold | 2021-11-24 | 1 | -1/+1 |
| | | | | | | Despite the inline comment, we were not actually grabbing the "redirected" ident correctly, meaning some counts would not be accurate. | ||||
| * | container merger: defer allocation of editgroup_id; and dummy code path | Bryan Newbold | 2021-11-24 | 1 | -1/+5 |
| | | |||||
| * | initial implementation of container merger | Bryan Newbold | 2021-11-24 | 1 | -0/+237 |
| | | |||||
| * | file merger: allocate editgroup id later in 'merge' process | Bryan Newbold | 2021-11-24 | 1 | -1/+5 |
| | | | | | | The motivation is to avoid creating empty editgroups in dry-run mode, and when all entities are "skipped" | ||||
| * | Merge branch 'bnewbold-mergers' into 'master' | bnewbold | 2021-11-25 | 4 | -0/+640 |
| |\ | | | | | | | | | entity mergers framework See merge request webgroup/fatcat!133 | ||||
| | * | mergers common: remove inaccurate comment | Bryan Newbold | 2021-11-24 | 1 | -2/+0 |
| | | | | | | | | | Caught in review, thanks miku | ||||
| | * | file merger: add content_scope to list of merged fields | Bryan Newbold | 2021-11-24 | 1 | -1/+1 |
| | | | |||||
| | * | release merger: some progress, but also disable (not complete) | Bryan Newbold | 2021-11-23 | 1 | -12/+72 |
| | | | |||||
| | * | file merges: fixes from testing in QA | Bryan Newbold | 2021-11-23 | 1 | -14/+23 |
| | | | |||||
| | * | mergers: small tweaks | Bryan Newbold | 2021-11-23 | 2 | -3/+3 |
| | | | |||||
| | * | mergers: remove entity mergers from __init__ (to work around warning) | Bryan Newbold | 2021-11-23 | 1 | -2/+0 |
| | | | |||||
| | * | initial file merger, with tests | Bryan Newbold | 2021-11-23 | 1 | -0/+228 |
| | | | |||||
| | * | mergers: fmt, lint, refactors | Bryan Newbold | 2021-11-23 | 3 | -96/+200 |
| | | | | | | | | | | | These old merger code is from an old branch and needed significant cleanup | ||||
| | * | first iteration of mergers | Bryan Newbold | 2021-11-23 | 3 | -0/+243 |
| | | | |||||
| * | | codespell fixes in python code (comments) | Bryan Newbold | 2021-11-24 | 2 | -3/+3 |
| |/ | |||||
| * | content_scope: include in file ES schema and transform | Bryan Newbold | 2021-11-17 | 1 | -0/+1 |
| | | |||||
| * | Merge branch 'bnewbold-import-refactors' into 'master' | bnewbold | 2021-11-11 | 18 | -1462/+811 |
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | import refactors and deprecations Some of these are from old stale branches (the datacite subject metadata patch), but most are from yesterday and today. Sort of a hodge-podge, but the general theme is getting around to deferred cleanups and refactors specific to importer code before making some behavioral changes. The Datacite-specific stuff could use review here. Remove unused/deprecated/dead code: - cdl_dash_dat and wayback_static importers, which were for specific early example entities and have been superseded by other importers - "extid map" sqlite3 feature from several importers, was only used for initial bulk imports (and maybe should not have been used) Refactors: - moved a number of large datastructures out of importer code and into a dedicated static file (`biblio_lookup_tables.py`). Didn't move all, just the ones that were either generic or very large (making it hard to read code) - shuffled around relative imports and some function names ("clean_str" vs. "clean") Some actual behavioral changes: - remove some Datacite-specific license slugs - stop trying to fix double-slashes in DOIs, that was causing more harm than help (some DOIs do actually have double-slashes!) - remove some excess metadata from datacite 'extra' fields | ||||
 |
index : fatcat | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |