Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | ingest: add 'extid' and 'query' modes; filters; refactor | Bryan Newbold | 2020-02-04 | 1 | -38/+147 |
| | | | | | | This is a large refactor of the ingest script. It adds a number of filtering options (for all modes), and new modes for free-form queries or limiting to specific external identifiers. | ||||
* | Merge branch 'martin-datacite-strptime-none' into 'master' | Martin Czygan | 2020-01-31 | 1 | -1/+1 |
|\ | | | | | | | | | datacite: prevent none See merge request webgroup/fatcat!25 | ||||
| * | datacite: prevent none | Martin Czygan | 2020-01-31 | 1 | -1/+1 |
|/ | |||||
* | Merge branch 'martin-datacite-micropub' into 'master' | Martin Czygan | 2020-01-31 | 4 | -25/+162 |
|\ | | | | | | | | | datacite: improve date handling and minor tweak See merge request webgroup/fatcat!24 | ||||
| * | datacite: name shall not be None | Martin Czygan | 2020-01-31 | 1 | -1/+1 |
| | | |||||
| * | datacite: add exception for https://www.micropublication.org/ | Martin Czygan | 2020-01-31 | 2 | -1/+7 |
| | | |||||
| * | datacite: do not skip records w/o date | Martin Czygan | 2020-01-31 | 1 | -2/+1 |
| | | |||||
| * | datacite: improve docstring | Martin Czygan | 2020-01-31 | 1 | -4/+4 |
| | | |||||
| * | datacite: improve date handling and minor tweak | Martin Czygan | 2020-01-30 | 4 | -21/+153 |
|/ | | | | | | | | | | | | | Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too. | ||||
* | fix KafkaError worker reporting for partition errors | Bryan Newbold | 2020-01-29 | 3 | -3/+3 |
| | |||||
* | additional DOI prefix filters | Bryan Newbold | 2020-01-28 | 1 | -0/+8 |
| | | | | From martin, thanks. | ||||
* | increase kafka-pixy timeout to 25 seconds | Bryan Newbold | 2020-01-28 | 1 | -1/+1 |
| | |||||
* | apply ingest request filtering in entity worker | Bryan Newbold | 2020-01-28 | 1 | -3/+34 |
| | | | | | | | `ingest_oa_only` behavior, and other filters, now handled in the entity update worker, instead of in the transform function. Also add a DOI prefix blocklist feature. | ||||
* | remove 'oa_only' feature from ingest transform | Bryan Newbold | 2020-01-28 | 2 | -15/+1 |
| | | | | Refactoring to move this filter elsewhere | ||||
* | more TODO/proposal cleanup | Bryan Newbold | 2020-01-22 | 4 | -10/+34 |
| | |||||
* | more details on potential _edit table disk savings | Bryan Newbold | 2020-01-22 | 1 | -3/+23 |
| | |||||
* | proposal of ideas for reducing database size | Bryan Newbold | 2020-01-21 | 1 | -0/+154 |
| | |||||
* | cleanup some of old TODO list into proposals | Bryan Newbold | 2020-01-21 | 4 | -44/+269 |
| | |||||
* | refactor fatcat_import kafka group names | Bryan Newbold | 2020-01-21 | 1 | -13/+54 |
| | | | | | | | | | | | | | My current understanding is that consumer group names should be one-to-one with topic names. I previously though offsets were stored on a {topic, group} key, but they seem to be mixed and having too many workers in the same group is bad. In particular, we don't want cross-talk or load between QA and prod. All these topics are caught up in prod, so deploying this change and restarting workers should be safe. This commit does not update the elasticsearch or entity updates workers. | ||||
* | fix trivial typo in file importer | Bryan Newbold | 2020-01-20 | 1 | -1/+1 |
| | |||||
* | stats: remove internal PG table sizes from old dumps | Bryan Newbold | 2020-01-19 | 2 | -292/+0 |
| | | | | For ease of reading and comparison | ||||
* | update stats and table sizes | Bryan Newbold | 2020-01-19 | 4 | -0/+96 |
| | |||||
* | Merge branch 'martin-openapi-typo-exmaple' into 'master' | bnewbold | 2020-01-19 | 1 | -1/+1 |
|\ | | | | | | | | | fix a typo in openapi definition See merge request webgroup/fatcat!20 | ||||
| * | fix a typo in openapi definition | Martin Czygan | 2020-01-18 | 1 | -1/+1 |
| | | |||||
* | | Merge branch 'martin-guide-typos-sentance' into 'master' | bnewbold | 2020-01-19 | 1 | -2/+2 |
|\ \ | | | | | | | | | | | | | fix two typos in editing guide See merge request webgroup/fatcat!21 | ||||
| * | | fix two typos in editing guide | Martin Czygan | 2020-01-18 | 1 | -2/+2 |
| |/ | |||||
* | | basic notes in bulk edit changelog | Bryan Newbold | 2020-01-19 | 1 | -0/+7 |
| | | |||||
* | | bulk edit notes for datacite (QA) | Bryan Newbold | 2020-01-19 | 1 | -0/+152 |
| | | |||||
* | | pubmed update notes | Bryan Newbold | 2020-01-19 | 1 | -1/+46 |
| | | |||||
* | | normal: DOI corner-case from pubmed import | Bryan Newbold | 2020-01-19 | 1 | -0/+9 |
| | | |||||
* | | Merge branch 'martin-openapi-typos' into 'master' | Martin Czygan | 2020-01-19 | 1 | -2/+2 |
|\ \ | |/ |/| | | | | | fix typos in openapi spec descriptions See merge request webgroup/fatcat!22 | ||||
| * | fix typos in openapi spec descriptions | Martin Czygan | 2020-01-19 | 1 | -2/+2 |
|/ | |||||
* | Merge branch 'martin-fix-container-search' into 'master' | Martin Czygan | 2020-01-18 | 2 | -6/+6 |
|\ | | | | | | | | | pagination: use correct endpoint in navigation See merge request webgroup/fatcat!19 | ||||
| * | pagination: use correct endpoint in navigation | Martin Czygan | 2020-01-18 | 2 | -6/+6 |
| | | |||||
* | | fix trivial one-character typo in fatcat_import.py | Bryan Newbold | 2020-01-17 | 1 | -1/+1 |
| | | | | | | | | Should have run tests before pushing! | ||||
* | | yet more schema update ideas | Bryan Newbold | 2020-01-17 | 1 | -2/+4 |
| | | |||||
* | | actually control pubmed updates with a flag | Bryan Newbold | 2020-01-17 | 1 | -0/+4 |
| | | |||||
* | | Merge branch 'bnewbold-abstracts-corner-case' into 'master' | bnewbold | 2020-01-17 | 1 | -1/+4 |
|\ \ | | | | | | | | | | | | | fatcatd: fix corner-case in abstracts insertion See merge request webgroup/fatcat!17 | ||||
| * | | fatcatd: fix corner-case in abstracts insertion | Bryan Newbold | 2020-01-15 | 1 | -1/+4 |
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | Both the release_abstract relation table and the abstracts table inserts were being gated by a check on new abstracts table rows. I *think* the chance of this having caused problems is low. The most likely would have been updates to exiting entities that somehow removed the abstracts content, but not the sha1 keys. This is not the default behavior of the API: either the entire abstract (content and hash) is returned, or the abstracts are hidden entirely (via `hide` flag). Still, best to be careful! | ||||
* | | Merge branch 'martin-clean-doi' into 'master' | Martin Czygan | 2020-01-17 | 2 | -3/+6 |
|\ \ | |/ |/| | | | | | do not normalize "en dash" in DOI See merge request webgroup/fatcat!18 | ||||
| * | do not normalize "en dash" in DOI | Martin Czygan | 2020-01-17 | 2 | -3/+6 |
|/ | | | | | | | | | Technically, [...] DOI names may incorporate any printable characters from the Universal Character Set (UCS-2), of ISO/IEC 10646, which is the character set defined by Unicode (https://www.doi.org/doi_handbook/2_Numbering.html#2.5.1). For mostly QA reasons, we currently treat a DOI with an "en dash" as invalid. | ||||
* | ingest: improve tests, support old ingest results | Bryan Newbold | 2020-01-15 | 4 | -4/+30 |
| | |||||
* | update ingest worker for schema tweaks | Bryan Newbold | 2020-01-15 | 1 | -8/+15 |
| | | | | | | Should be backwards compatible with old ingest results. Fixed a bug with glutton ident detection. | ||||
* | ingest: allow more sources to auto-import | Bryan Newbold | 2020-01-15 | 1 | -1/+2 |
| | |||||
* | sql table size script: shorter output | Bryan Newbold | 2020-01-15 | 1 | -0/+1 |
| | | | | This skips postgres-internal tables in size output | ||||
* | Merge branch 'martin-datacite-fix-isascii' into 'master' | Martin Czygan | 2020-01-13 | 1 | -0/+4 |
|\ | | | | | | | | | datacite: skip records without a doi See merge request webgroup/fatcat!16 | ||||
| * | datacite: skip records without a doi | Martin Czygan | 2020-01-13 | 1 | -0/+4 |
|/ | |||||
* | add missing sentry/raven tags | Bryan Newbold | 2020-01-10 | 3 | -2/+19 |
| | | | | | | Good to have exceptions tracked and stored even for commands run from the command line. But in particular the importer runs as a kafka worker and should be tracking excpetions. | ||||
* | datacite: add entry to license slug map | Martin Czygan | 2020-01-09 | 2 | -0/+2 |
| | |||||
* | datacite: ignore known unknown values in resourceType* | Martin Czygan | 2020-01-09 | 4 | -3/+97 |
| |