aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* update datacite tests for license slug changesBryan Newbold2021-11-102-8/+7
| | | | | Use datacite-specific wrapper function, and remove a couple non-OA/TDM-limited licenses.
* improve lookup_license_slug helper and lookup tableBryan Newbold2021-11-102-56/+62
|
* refactor importer metadata tables into separate file; move some helpers aroundBryan Newbold2021-11-1010-702/+682
| | | | | | | - MAX_ABSTRACT_LENGTH set in a single place (importer common) - merge datacite license slug table in to common table, removing some TDM-specific licenses (which do not apply in the context of preserving the full work)
* importers: refactor imports of clean() and other normalization helpersBryan Newbold2021-11-1012-95/+104
|
* remove cdl_dash_dat and wayback_static importersBryan Newbold2021-11-104-596/+0
| | | | | | | | Cleaning out dead code. These importers were used to create demonstration fileset and webcapture entities early in development. They have been replaced by the fileset and webcapture ingest importers.
* datacite import: store less subject metadataBryan Newbold2021-11-101-1/+7
| | | | | | | | Many of these 'subject' objects have the equivalent of several lines of text, with complex URLs that don't compress well. I think it is fine we have included these thus far instead of parsing more deeply, but going forward I don't think this nested 'extra' metadata is worth the database space.
* add notes about 'double slash in DOI' issueBryan Newbold2021-11-091-0/+46
|
* importers: use clean_doi() in many more (all?) importersBryan Newbold2021-11-096-12/+29
|
* clean_doi: stop mutating double-slash DOIs, except for 10.1037 prefixBryan Newbold2021-11-091-1/+2
|
* remove deprecated extid sqlite3 lookup table feature from importersBryan Newbold2021-11-0910-203/+10
| | | | | | | | This was used during initial bulk imports, but is no longer used and could create serious metadata problems if used accidentially. In retrospect, it also made metadata provenance less transparent, and may have done more harm than good overall.
* update CHANGELOG for recent developmentBryan Newbold2021-11-051-0/+26
|
* python tests: verify array sort orderBryan Newbold2021-11-054-20/+18
| | | | | | | In a couple cases (eg, filesets), had made tests agnostic to sort order, because the sort order was not stable. In other cases, simply small cleanups and comment improvements.
* api: add SQL 'ORDER BY' to many reads to stabilize API array orderingBryan Newbold2021-11-051-3/+14
| | | | | | | | | | | | | | The hope is to make things like file entity URLs, fileset manifests, and other arrays in the JSON API "stable", meaning that if you create an entity with a list of a given order, a read back (in any environment, including prod/QA, bulk dumps, etc) will return the array with the same sort order. This was informally happening most of the time, but occasionally not (!) Assumption is that these sorts will have little or no performance impact, as the common case is less than a dozen elements, and the hard cases are a few thousand at most, and there is already a sorted index.
* enable type annotation checking with flake8 by default ('make lint')Bryan Newbold2021-11-031-4/+2
|
* cleanups: create a separate JsonLinePusher for cleanup workers (distinct ↵Bryan Newbold2021-11-033-4/+20
| | | | base class)
* facat_import.py: work around corner case in run_cdl_dash_dat()Bryan Newbold2021-11-031-1/+1
|
* datacite importer: remove unused 'year_only' variableBryan Newbold2021-11-031-2/+3
|
* web: work around remaining type annotation issuesBryan Newbold2021-11-032-11/+15
|
* ignore type errors in cors.py (third party code)Bryan Newbold2021-11-031-2/+2
|
* web: fix bytes/text warning loggingBryan Newbold2021-11-031-3/+3
| | | | Minor issue. Caught by type checking
* lint: remove unused importBryan Newbold2021-11-031-1/+0
| | | | This was caught by new linting, I think not caused by any recent changes
* web config: add helper for coercing env vars into booleansBryan Newbold2021-11-031-3/+32
|
* web: add type annotationsBryan Newbold2021-11-0312-297/+347
| | | | | This commit does not include type fixes, only annotations. A small number of tuples were also converted to lists.
* introduce 'AnyResponse' type for Flask viewsBryan Newbold2021-11-032-0/+15
|
* pubmed harvester: remove unused variablesBryan Newbold2021-11-031-2/+2
|
* pubmed harvester: explicit assertions to mark unreachable code pathsBryan Newbold2021-11-031-0/+2
|
* typing: add assertions to fatcat_tool code to make type assumptions explicitBryan Newbold2021-11-033-0/+3
|
* typing: add annotations to remaining fatcat_tools codeBryan Newbold2021-11-039-122/+186
| | | | | Again, these are just annotations, no changes made to get type checks to pass
* datacite: add comment about potential date parsing bugBryan Newbold2021-11-031-0/+1
|
* datacite importer: dateparser.date.DateDataParser()Bryan Newbold2021-11-031-1/+1
| | | | Perhaps this was a change when upgrading 'dateparser'?
* more involved type wrangling and fixes for importersBryan Newbold2021-11-033-12/+14
|
* typing: relatively simple type check fixesBryan Newbold2021-11-0314-87/+82
| | | | | | | These mostly add new variable names so that existing variables aren't overwritten with a new type; delay coercing '{}' or '[]' to 'None' until the last minute; adding is-not-None checks to conditional clauses; and similar small changes.
* typing: initial annotations on importersBryan Newbold2021-11-0322-274/+443
| | | | | This commit just adds the type annotations, doesn't do fixes to code to make type checking pass.
* typing: first batch of python bulk type annotationsBryan Newbold2021-11-0321-139/+200
| | | | | | While these changes are more delicate than simple lint changes, this specific batch of edits and annotations was *relatively* simple, and resulted in few code changes other than function signature additions.
* importers: remove unused __main__ routineBryan Newbold2021-11-034-19/+0
| | | | | | These perhaps were used in initial develoment or testing? fatcat_import.py is the correct way to do these imports, even for testing/development.
* lint: resolve existing mypy type errorsBryan Newbold2021-11-0212-81/+125
| | | | | | | | | Adds annotations and re-workes dataflow to satisfy existing mypy issues, without adding any additional type annotations to, eg, function signatures. There will probably be many more type errors when annotations are all added.
* web: annotate 'app' as 'Any', and document whyBryan Newbold2021-11-021-1/+5
|
* flake8 config: update ignore list after big fmtBryan Newbold2021-11-021-3/+4
|
* re-fix some lint issues after big 'fmt'Bryan Newbold2021-11-022-4/+5
|
* fmt (black): fatcat_tools/Bryan Newbold2021-11-0243-3194/+4020
|
* fmt (black): fatcat_web/Bryan Newbold2021-11-0213-1297/+1992
|
* fmt (black): *.pyBryan Newbold2021-11-0211-715/+1110
|
* fmt (black): tests/Bryan Newbold2021-11-0255-1430/+1852
|
* python: isort everythingBryan Newbold2021-11-0285-184/+278
|
* revert '-vv' for pytest in Makefile (too verbose)Bryan Newbold2021-11-021-1/+1
|
* arabesque import 'hit' field is 1/0, not true/falseBryan Newbold2021-11-021-2/+2
|
* lint: simple, safe inline lint fixesBryan Newbold2021-11-0239-221/+220
| | | | '==' vs 'is'; 'not a in b' vs 'a not in b'; etc
* lint/fmt: remove all 'import *'Bryan Newbold2021-11-0212-62/+207
|
* entity transforms: add basic type annotationsBryan Newbold2021-11-021-7/+19
|
* ftfy 'fix_entities' argument has been renamedBryan Newbold2021-11-021-4/+4
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-25 22:12:24 +0000