| Commit message (Collapse) | Author | Age | Files | Lines | ||
|---|---|---|---|---|---|---|
| ... | ||||||
| | * | importers: use clean_doi() in many more (all?) importers | Bryan Newbold | 2021-11-09 | 6 | -12/+29 | |
| | | | ||||||
| | * | clean_doi: stop mutating double-slash DOIs, except for 10.1037 prefix | Bryan Newbold | 2021-11-09 | 1 | -1/+2 | |
| | | | ||||||
| | * | remove deprecated extid sqlite3 lookup table feature from importers | Bryan Newbold | 2021-11-09 | 10 | -203/+10 | |
| | | | | | | | | | | | | | | | | | This was used during initial bulk imports, but is no longer used and could create serious metadata problems if used accidentially. In retrospect, it also made metadata provenance less transparent, and may have done more harm than good overall. | |||||
| * | | Merge branch 'bnewbold-cleanups-nov2021' into 'master' | bnewbold | 2021-11-11 | 5 | -1/+749 | |
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fatcat metadata cleanups/fixups, November 2021 Three cleanups implemented in this branch: - update non-lowercase DOIs on releases (couple hundred thousand entities) - fix incorrectly imported file/release pairs, on the file entity side (~250k entities) - expand truncated wayback URL timestamps in file entities (up to 10 million entities) Instead of proposals, there are documents for each cleanup in `notes/cleanups/`. Have done spot testing of tens of thousands of entities each in QA, and confident about running in production. Plan is to run updates in the order above. DOI and bugfix updates will go fairly fast; the wayback timestamp updates will go slower, and result in large re-indexing load both in fatcat and scholar, because both release and work entities will get triggered for update when file entities are updated. | |||||
| | * | | file/release bugfix: handle files with multiple edits | Bryan Newbold | 2021-11-09 | 1 | -6/+6 | |
| | | | | ||||||
| | * | | cleanups: add more state=active checks | Bryan Newbold | 2021-11-09 | 2 | -0/+8 | |
| | | | | ||||||
| | * | | update link source filters in file/release bugfix | Bryan Newbold | 2021-11-09 | 1 | -2/+8 | |
| | | | | ||||||
| | * | | initial file/release bugfix cleanup worker and notes | Bryan Newbold | 2021-11-09 | 1 | -0/+231 | |
| | | | | ||||||
| | * | | updates to lowercase DOI cleanup | Bryan Newbold | 2021-11-09 | 1 | -7/+15 | |
| | | | | ||||||
| | * | | lowercase DOI lint and check entity status | Bryan Newbold | 2021-11-09 | 1 | -4/+5 | |
| | | | | ||||||
| | * | | more iteration on short wayback timestamp cleanup | Bryan Newbold | 2021-11-09 | 1 | -1/+1 | |
| | | | | ||||||
| | * | | lint: minor import tweak | Bryan Newbold | 2021-11-09 | 1 | -1/+1 | |
| | | | | ||||||
| | * | | cleanups: tweaks to wayback CDX cleanup scripts | Bryan Newbold | 2021-11-09 | 1 | -5/+13 | |
| | | | | ||||||
| | * | | cleanups: initial lowercase DOI cleanup script | Bryan Newbold | 2021-11-09 | 1 | -0/+145 | |
| | | | | ||||||
| | * | | wayback short ts: another regression test, and some small fmt/tweaks | Bryan Newbold | 2021-11-09 | 1 | -3/+38 | |
| | | | | ||||||
| | * | | wayback cleanup: actually update entity | Bryan Newbold | 2021-11-09 | 1 | -2/+4 | |
| | | | | ||||||
| | * | | imports: generic file cleanup removes exact duplicate URLs | Bryan Newbold | 2021-11-09 | 1 | -0/+9 | |
| | | | | ||||||
| | * | | wayback short ts: add regression test for dupe URLs | Bryan Newbold | 2021-11-09 | 1 | -0/+44 | |
| | | | | ||||||
| | * | | short wayback ts: initial cleanup script implementation | Bryan Newbold | 2021-11-09 | 1 | -0/+251 | |
| | |/ | ||||||
| * / | pubmed: allow updates if PMCID does not exist yet | Bryan Newbold | 2021-11-10 | 1 | -1/+6 | |
| |/ | | | | | | | | | | | The intent of this change is to start updating Pubmed metadata records when a PMCID has been assigned, but that ext_id hasn't been recorded in fatcat yet. It is likely that this change will result in some additional duplicate PMCIDs in the catalog. But the principle is that the PMID is the primary pubmed identifier, and all records with a PMID should have the PMCID that pubmed indicates, even if there exists another incorrect record. | |||||
| * | python tests: verify array sort order | Bryan Newbold | 2021-11-05 | 4 | -20/+18 | |
| | | | | | | | | In a couple cases (eg, filesets), had made tests agnostic to sort order, because the sort order was not stable. In other cases, simply small cleanups and comment improvements. | |||||
| * | enable type annotation checking with flake8 by default ('make lint') | Bryan Newbold | 2021-11-03 | 1 | -4/+2 | |
| | | ||||||
| * | cleanups: create a separate JsonLinePusher for cleanup workers (distinct ↵ | Bryan Newbold | 2021-11-03 | 3 | -4/+20 | |
| | | | | | base class) | |||||
| * | facat_import.py: work around corner case in run_cdl_dash_dat() | Bryan Newbold | 2021-11-03 | 1 | -1/+1 | |
| | | ||||||
| * | datacite importer: remove unused 'year_only' variable | Bryan Newbold | 2021-11-03 | 1 | -2/+3 | |
| | | ||||||
| * | web: work around remaining type annotation issues | Bryan Newbold | 2021-11-03 | 2 | -11/+15 | |
| | | ||||||
| * | ignore type errors in cors.py (third party code) | Bryan Newbold | 2021-11-03 | 1 | -2/+2 | |
| | | ||||||
| * | web: fix bytes/text warning logging | Bryan Newbold | 2021-11-03 | 1 | -3/+3 | |
| | | | | | Minor issue. Caught by type checking | |||||
| * | lint: remove unused import | Bryan Newbold | 2021-11-03 | 1 | -1/+0 | |
| | | | | | This was caught by new linting, I think not caused by any recent changes | |||||
| * | web config: add helper for coercing env vars into booleans | Bryan Newbold | 2021-11-03 | 1 | -3/+32 | |
| | | ||||||
| * | web: add type annotations | Bryan Newbold | 2021-11-03 | 12 | -297/+347 | |
| | | | | | | This commit does not include type fixes, only annotations. A small number of tuples were also converted to lists. | |||||
| * | introduce 'AnyResponse' type for Flask views | Bryan Newbold | 2021-11-03 | 2 | -0/+15 | |
| | | ||||||
| * | pubmed harvester: remove unused variables | Bryan Newbold | 2021-11-03 | 1 | -2/+2 | |
| | | ||||||
| * | pubmed harvester: explicit assertions to mark unreachable code paths | Bryan Newbold | 2021-11-03 | 1 | -0/+2 | |
| | | ||||||
| * | typing: add assertions to fatcat_tool code to make type assumptions explicit | Bryan Newbold | 2021-11-03 | 3 | -0/+3 | |
| | | ||||||
| * | typing: add annotations to remaining fatcat_tools code | Bryan Newbold | 2021-11-03 | 9 | -122/+186 | |
| | | | | | | Again, these are just annotations, no changes made to get type checks to pass | |||||
| * | datacite: add comment about potential date parsing bug | Bryan Newbold | 2021-11-03 | 1 | -0/+1 | |
| | | ||||||
| * | datacite importer: dateparser.date.DateDataParser() | Bryan Newbold | 2021-11-03 | 1 | -1/+1 | |
| | | | | | Perhaps this was a change when upgrading 'dateparser'? | |||||
| * | more involved type wrangling and fixes for importers | Bryan Newbold | 2021-11-03 | 3 | -12/+14 | |
| | | ||||||
| * | typing: relatively simple type check fixes | Bryan Newbold | 2021-11-03 | 14 | -87/+82 | |
| | | | | | | | | These mostly add new variable names so that existing variables aren't overwritten with a new type; delay coercing '{}' or '[]' to 'None' until the last minute; adding is-not-None checks to conditional clauses; and similar small changes. | |||||
| * | typing: initial annotations on importers | Bryan Newbold | 2021-11-03 | 22 | -274/+443 | |
| | | | | | | This commit just adds the type annotations, doesn't do fixes to code to make type checking pass. | |||||
| * | typing: first batch of python bulk type annotations | Bryan Newbold | 2021-11-03 | 21 | -139/+200 | |
| | | | | | | | While these changes are more delicate than simple lint changes, this specific batch of edits and annotations was *relatively* simple, and resulted in few code changes other than function signature additions. | |||||
| * | importers: remove unused __main__ routine | Bryan Newbold | 2021-11-03 | 4 | -19/+0 | |
| | | | | | | | These perhaps were used in initial develoment or testing? fatcat_import.py is the correct way to do these imports, even for testing/development. | |||||
| * | lint: resolve existing mypy type errors | Bryan Newbold | 2021-11-02 | 12 | -81/+125 | |
| | | | | | | | | | | Adds annotations and re-workes dataflow to satisfy existing mypy issues, without adding any additional type annotations to, eg, function signatures. There will probably be many more type errors when annotations are all added. | |||||
| * | web: annotate 'app' as 'Any', and document why | Bryan Newbold | 2021-11-02 | 1 | -1/+5 | |
| | | ||||||
| * | flake8 config: update ignore list after big fmt | Bryan Newbold | 2021-11-02 | 1 | -3/+4 | |
| | | ||||||
| * | re-fix some lint issues after big 'fmt' | Bryan Newbold | 2021-11-02 | 2 | -4/+5 | |
| | | ||||||
| * | fmt (black): fatcat_tools/ | Bryan Newbold | 2021-11-02 | 43 | -3194/+4020 | |
| | | ||||||
| * | fmt (black): fatcat_web/ | Bryan Newbold | 2021-11-02 | 13 | -1297/+1992 | |
| | | ||||||
| * | fmt (black): *.py | Bryan Newbold | 2021-11-02 | 11 | -715/+1110 | |
| | | ||||||
 |
index : fatcat | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |