summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* refactor fatcat_import kafka group namesBryan Newbold2020-01-211-13/+54
| | | | | | | | | | | | | My current understanding is that consumer group names should be one-to-one with topic names. I previously though offsets were stored on a {topic, group} key, but they seem to be mixed and having too many workers in the same group is bad. In particular, we don't want cross-talk or load between QA and prod. All these topics are caught up in prod, so deploying this change and restarting workers should be safe. This commit does not update the elasticsearch or entity updates workers.
* fix trivial typo in file importerBryan Newbold2020-01-201-1/+1
|
* stats: remove internal PG table sizes from old dumpsBryan Newbold2020-01-192-292/+0
| | | | For ease of reading and comparison
* update stats and table sizesBryan Newbold2020-01-194-0/+96
|
* Merge branch 'martin-openapi-typo-exmaple' into 'master'bnewbold2020-01-191-1/+1
|\ | | | | | | | | fix a typo in openapi definition See merge request webgroup/fatcat!20
| * fix a typo in openapi definitionMartin Czygan2020-01-181-1/+1
| |
* | Merge branch 'martin-guide-typos-sentance' into 'master'bnewbold2020-01-191-2/+2
|\ \ | | | | | | | | | | | | fix two typos in editing guide See merge request webgroup/fatcat!21
| * | fix two typos in editing guideMartin Czygan2020-01-181-2/+2
| |/
* | basic notes in bulk edit changelogBryan Newbold2020-01-191-0/+7
| |
* | bulk edit notes for datacite (QA)Bryan Newbold2020-01-191-0/+152
| |
* | pubmed update notesBryan Newbold2020-01-191-1/+46
| |
* | normal: DOI corner-case from pubmed importBryan Newbold2020-01-191-0/+9
| |
* | Merge branch 'martin-openapi-typos' into 'master'Martin Czygan2020-01-191-2/+2
|\ \ | |/ |/| | | | | fix typos in openapi spec descriptions See merge request webgroup/fatcat!22
| * fix typos in openapi spec descriptionsMartin Czygan2020-01-191-2/+2
|/
* Merge branch 'martin-fix-container-search' into 'master'Martin Czygan2020-01-182-6/+6
|\ | | | | | | | | pagination: use correct endpoint in navigation See merge request webgroup/fatcat!19
| * pagination: use correct endpoint in navigationMartin Czygan2020-01-182-6/+6
| |
* | fix trivial one-character typo in fatcat_import.pyBryan Newbold2020-01-171-1/+1
| | | | | | | | Should have run tests before pushing!
* | yet more schema update ideasBryan Newbold2020-01-171-2/+4
| |
* | actually control pubmed updates with a flagBryan Newbold2020-01-171-0/+4
| |
* | Merge branch 'bnewbold-abstracts-corner-case' into 'master'bnewbold2020-01-171-1/+4
|\ \ | | | | | | | | | | | | fatcatd: fix corner-case in abstracts insertion See merge request webgroup/fatcat!17
| * | fatcatd: fix corner-case in abstracts insertionBryan Newbold2020-01-151-1/+4
| |/ | | | | | | | | | | | | | | | | | | | | | | | | Both the release_abstract relation table and the abstracts table inserts were being gated by a check on new abstracts table rows. I *think* the chance of this having caused problems is low. The most likely would have been updates to exiting entities that somehow removed the abstracts content, but not the sha1 keys. This is not the default behavior of the API: either the entire abstract (content and hash) is returned, or the abstracts are hidden entirely (via `hide` flag). Still, best to be careful!
* | Merge branch 'martin-clean-doi' into 'master'Martin Czygan2020-01-172-3/+6
|\ \ | |/ |/| | | | | do not normalize "en dash" in DOI See merge request webgroup/fatcat!18
| * do not normalize "en dash" in DOIMartin Czygan2020-01-172-3/+6
|/ | | | | | | | | Technically, [...] DOI names may incorporate any printable characters from the Universal Character Set (UCS-2), of ISO/IEC 10646, which is the character set defined by Unicode (https://www.doi.org/doi_handbook/2_Numbering.html#2.5.1). For mostly QA reasons, we currently treat a DOI with an "en dash" as invalid.
* ingest: improve tests, support old ingest resultsBryan Newbold2020-01-154-4/+30
|
* update ingest worker for schema tweaksBryan Newbold2020-01-151-8/+15
| | | | | | Should be backwards compatible with old ingest results. Fixed a bug with glutton ident detection.
* ingest: allow more sources to auto-importBryan Newbold2020-01-151-1/+2
|
* sql table size script: shorter outputBryan Newbold2020-01-151-0/+1
| | | | This skips postgres-internal tables in size output
* Merge branch 'martin-datacite-fix-isascii' into 'master'Martin Czygan2020-01-131-0/+4
|\ | | | | | | | | datacite: skip records without a doi See merge request webgroup/fatcat!16
| * datacite: skip records without a doiMartin Czygan2020-01-131-0/+4
|/
* add missing sentry/raven tagsBryan Newbold2020-01-103-2/+19
| | | | | | Good to have exceptions tracked and stored even for commands run from the command line. But in particular the importer runs as a kafka worker and should be tracking excpetions.
* datacite: add entry to license slug mapMartin Czygan2020-01-092-0/+2
|
* datacite: ignore known unknown values in resourceType*Martin Czygan2020-01-094-3/+97
|
* datacite: abstracts may be strings or list of stringsMartin Czygan2020-01-096-3/+202
|
* datacite: improve license_slug handlingMartin Czygan2020-01-094-62/+134
|
* datacite: add 'Unknown' to blacklistMartin Czygan2020-01-092-8/+6
|
* datacite: get rid of schemaVersionMartin Czygan2020-01-0918-35/+14
|
* update depsMartin Czygan2020-01-081-86/+69
|
* Merge branch 'martin-datacite-import'Martin Czygan2020-01-0864-5/+5736
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pipfile.lock is broken. * martin-datacite-import: (68 commits) datacite: pass in doi into factored out method datacite: reformat test cases and use jq . --sort-keys datacite: factor out contributor handling datacite: catch type mismatch in language detection datacite: adjust tests for release_month datacite: name extra.month, extra.release_month datacite: mark additional files as stub datacite: CCDC are entries, mostly datacite: use more specific release_type, if possible datacite: ignore certain names datacite: over 3% records have the same title: stub datacite: fill a few more release_type gaps datacite: adding datacite-specific extra metadata datacite: apply pylint suggestions datacite: fix typos datacite: set release_stage to published by default datacite: month field should be top-level datacite: include month in extra datacite: indicate mismatched file in test datacite: clean abstracts, use unknown value tokens ...
| * datacite: pass in doi into factored out methodMartin Czygan2020-01-081-2/+3
| |
| * datacite: reformat test cases and use jq . --sort-keysMartin Czygan2020-01-0854-2299/+2301
| |
| * datacite: factor out contributor handlingMartin Czygan2020-01-086-82/+210
| | | | | | | | | | | | | | Use values from: * attributes.creators[] * attributes.contributors[]
| * datacite: catch type mismatch in language detectionMartin Czygan2020-01-081-3/+2
| |
| * datacite: adjust tests for release_monthMartin Czygan2020-01-0812-12/+12
| |
| * datacite: name extra.month, extra.release_monthMartin Czygan2020-01-081-1/+3
| |
| * datacite: mark additional files as stubMartin Czygan2020-01-084-1/+77
| |
| * datacite: CCDC are entries, mostlyMartin Czygan2020-01-082-1/+5
| |
| * datacite: use more specific release_type, if possibleMartin Czygan2020-01-081-0/+6
| |
| * datacite: ignore certain namesMartin Czygan2020-01-081-0/+6
| |
| * datacite: over 3% records have the same title: stubMartin Czygan2020-01-081-0/+7
| | | | | | | | | | | | | | | | | | | | The GBIF (https://www.gbif.org/) deposits most records under the titles: * 599243 GBIF Occurrence Download * 41176 Occurrence Download Mark them as "stub" for the moment (https://guide.fatcat.wiki/entity_release.html#release_type-vocabulary).
| * datacite: fill a few more release_type gapsMartin Czygan2020-01-081-17/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * citeproc: http://docs.citationstyles.org/en/stable/specification.html#appendix-iii-types * resourceTypeGeneral: https://schema.datacite.org/meta/kernel-4.0/doc/DataCite-MetadataKernel_v4.0.pdf#page=32 * resourceType: uncontrolled, over 170000 distinct values, frequent: null, Dataset, JournalArticle, PGRFA Material, Journal Article, Dataset/UNITE Species Hypothesis, ... General frequency: * "attributes.types": 18210075, * "attributes.types.ris": 18058890, * "attributes.types.bibtex": 18058888, * "attributes.types.citeproc": 18058890, * "attributes.types.schemaOrg": 18058929, * "attributes.types.resourceType": 12737988, * "attributes.types.resourceTypeGeneral": 16576139,
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-15 13:40:25 +0000