aboutsummaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'bnewbold-ingest-tweaks' into 'master'bnewbold2021-10-023-39/+106
|\ | | | | | | | | ingest importer behavior tweaks See merge request webgroup/fatcat!120
| * kafka import: optional 'force-flush' mode for some importersBryan Newbold2021-10-011-0/+13
| | | | | | | | Behavior and motivation described in the kafka json import comment.
| * new SPN web (html) importerBryan Newbold2021-10-012-27/+81
| |
| * ingest importer behavior tweaksBryan Newbold2021-10-011-8/+8
| | | | | | | | | | - change order of 'want()' checks, so that result counts are clearer - don't require GROBID success for file imports with SPN
| * importer common: more verbose logging (with counts)Bryan Newbold2021-10-011-4/+4
| |
* | datacite: skip empty abstractsMartin Czygan2021-10-011-1/+4
|/ | | | | Do not add abstracts where `clean` results in the empty string - this violates a constraint: `either abstract_sha1 or content is required`
* pubmed: workaround a networking issueMartin Czygan2021-09-091-24/+21
| | | | | | use an http proxy (https://github.com/miku/ftpup) to fetch files from FTP, keep some retry logic; also, hardcoding the proxy path as this should be a temporary workaround
* pubmed: add option to ftp download with lftpMartin Czygan2021-09-081-2/+31
| | | | | lftp is a classic command line ftp client, and we hope that its retry capabilities are enough of a workaround for the current networking issue
* pubmed harvester: add basic retry logicMartin Czygan2021-08-201-8/+21
| | | | | | | | Related to a previous issue with seemingly random EOFError from FTP connections, this patch wrap "ftpretr" helper function with a basic retry. Refs: fatcat-workers/issues/92151, fatcat-workers/issues/91102
* refs: default to *not* consolidating worksBryan Newbold2021-08-061-1/+1
| | | | | | | We don't handle counts for consolidated refs yet, so just don't consolidate. This should fix, eg, "Showing 1-18 of 19" type UX confusion, with the trade-off that some works will be duplicated in inbound ref tables.
* refs: lint fixesBryan Newbold2021-07-271-0/+1
|
* refs: support for wikipedia outbound refs, and display in tablesBryan Newbold2021-07-271-2/+2
|
* refs: generalize web endpoints; JSON content negotiation; openlibrary ↵Bryan Newbold2021-07-232-22/+57
| | | | inbound view; etc
* refs: small refactors/tweaksBryan Newbold2021-07-231-11/+17
|
* remove unused imports (lint)Bryan Newbold2021-07-232-3/+2
|
* pylint: skip pydantic import check (dynamic/extensions)Bryan Newbold2021-07-231-8/+2
|
* refs: refactor web paths; enrich refs as generic; remove old refs linkBryan Newbold2021-07-231-50/+35
|
* refs fetch: add some hacks; sort hitsBryan Newbold2021-07-231-6/+16
|
* fixes for newer ref indexBryan Newbold2021-07-231-1/+1
|
* references: refactor to point to access_options transform; comment out CSL ↵Bryan Newbold2021-07-231-57/+8
| | | | fields
* partial access options transform for releasesBryan Newbold2021-07-231-0/+58
|
* initial inbound/outbound reference query helpersBryan Newbold2021-07-231-0/+450
|
* pubmed: update docsMartin Czygan2021-07-171-2/+3
|
* pubmed: do not fail when accessing missing fileMartin Czygan2021-07-171-2/+8
| | | | | | | after a sync gap (e.g. 06/07 2021) harvester wanted to fetch a file, that was not on the server (any more) - do not fail in this case we'll need to backfill missing records via full data dump
* pubmed: reconnect on errorMartin Czygan2021-07-161-4/+30
| | | | | | | | | ftp retrieval would run but fail with EOFError on /pubmed/updatefiles/pubmed21n1328_stats.html - not able to find the root cause; using a fresh client, the exact same file would work just fine. So when we retry, we reconnect on failure. Refs: sentry #91102.
* more consistent and defensive lower-casing of DOIsBryan Newbold2021-06-233-3/+8
| | | | | | | After noticing more upper/lower ambiguity in production. In particular, we have some old ingest requests in sandcrawler DB, which get re-submitted/re-tried, which have capitalized DOIs in the link source id field.
* datacite: more careful title string access; fixes sentry #88350Martin Czygan2021-06-111-1/+1
| | | | | Caused by a partial "title entry without title" coming *first* (e.g. just holding, e.g. a language, like: {'lang': 'da'}
* clean_doi() should lower-case returned DOIBryan Newbold2021-06-071-1/+4
| | | | | | | | | | Code in a number of places (including Pubmed importer) assumed that this was already lower-casing DOIs, resulting in some broken metadata getting created. See also: https://github.com/internetarchive/fatcat/issues/83 This is just the first step of mitigation.
* ingest: swap ingest and file checks, to result in clearer stats/counts of ↵Bryan Newbold2021-06-031-2/+2
| | | | skipping
* ingest: don't accept mag and s2 URLsBryan Newbold2021-06-031-4/+4
|
* changelog worker: fix file/fileset typo, caught by lintBryan Newbold2021-05-251-1/+1
| | | | | This would have been resulting in some releases not getting re-indexed into search.
* small python lint fixes (no behavior change)Bryan Newbold2021-05-253-4/+2
|
* ingest: add per-container ingest type overridesBryan Newbold2021-05-211-1/+17
|
* arabesque importer: ensure full 14-digit timestampsBryan Newbold2021-05-211-1/+3
|
* transforms: fix 'display_ame' typoBryan Newbold2021-04-191-2/+2
|
* prefer contrib.creator.display_name over contrib.raw_nameBryan Newbold2021-04-122-4/+7
| | | | | | | | These will be getting updates from ORCID and are usually more complete and more correct for display, attribution, and search purposes. Might need to tweak fuzzycat code to handle multiple names at the verification stage.
* es worker: ensure kafka messages get clearedBryan Newbold2021-04-121-0/+2
|
* es indexing: more 'wip' fixesBryan Newbold2021-04-121-1/+5
|
* ES indexing: skip 'wip' entities with a warningBryan Newbold2021-04-121-11/+16
|
* container ES index worker: support for querying statusBryan Newbold2021-04-061-5/+32
|
* ES schema updates: doc_index_ts as a str, not datetimeBryan Newbold2021-04-061-4/+4
| | | | | The schema is a timestamp, but python needs to serialize as JSON, and doesn't do datetime automatically.
* container search schema: preservation stats, new fieldsBryan Newbold2021-04-061-2/+18
| | | | Includes transform code updates and partial test coverage.
* release ES: add discipline fieldBryan Newbold2021-04-061-0/+2
|
* ES schemas: add doc_index_ts to all mappingsBryan Newbold2021-04-061-0/+4
|
* indexing: don't use document namesBryan Newbold2021-04-061-14/+4
|
* datacite: a missing surname should be None, not the empty stringMartin Czygan2021-04-021-2/+1
| | | | refs sentry #77700
* elasticsearch: simple new dblp and doaj fieldsBryan Newbold2021-01-201-0/+4
|
* web ingest: terminal URL mismatch as skip, not assertBryan Newbold2020-12-301-1/+3
|
* dblp release import: skip arxiv_id releasesBryan Newbold2020-12-241-0/+9
|
* normalizer: test for un-versioned arxiv_idBryan Newbold2020-12-241-0/+4
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-25 02:29:44 +0000