summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* sql_dumps: stop doing redundant release dumpsBryan Newbold2020-04-011-1/+3
|
* Merge branch 'bnewbold-crossref-deposit' into 'master'bnewbold2020-04-012-4/+24
|\ | | | | | | | | change crossref harvest date field See merge request webgroup/fatcat!41
| * crossref: switch from index-date to update-dateBryan Newbold2020-03-302-2/+2
| | | | | | | | | | | | This goes against what the API docs recommend, but we are currently far behind on updates and need to catch up. Other than what the docs say, this seems to be consistent with the behavior we want.
| * crossref: longer comment about crossref API date fieldsBryan Newbold2020-03-301-2/+22
|/
* web: small corrections to subtitle/original_nameBryan Newbold2020-03-282-2/+4
|
* Merge branch 'bnewbold-translation-view'Bryan Newbold2020-03-282-0/+7
|\
| * web: use more idiomatic type checkBryan Newbold2020-03-281-1/+1
| | | | | | | | Thanks to Martin for suggestion
| * web: display container original titlesBryan Newbold2020-03-261-0/+3
| |
| * web: copy extra.subtitle to subtitle for release viewBryan Newbold2020-03-261-0/+4
| | | | | | | | | | The release view will display subtitles, but it needs to be in the correct "location".
* | ingest: more DOI patterns to treat as OABryan Newbold2020-03-281-0/+26
| | | | | | | | | | | | | | These are journal/publisher patterns which we suspect to actually be OA based on the large quantity of papers that crawl successfully. The better long-term solution will be to flag containers in some way as OA (or "should crawl"), but this is a good short-term solution.
* | update CONTRIBUTORSBryan Newbold2020-03-271-2/+15
| | | | | | | | | | | | | | So far only updating "what was contributed" for past work, not recent or (potentially) ongoing contributions. Thank you everybody!
* | Merge pull request #53 from EdwardBetts/spellingbnewbold2020-03-2713-20/+20
|\ \ | | | | | | Correct spelling mistakes
| * | Correct spelling mistakesEdward Betts2020-03-2713-20/+20
| | |
* | | Merge branch 'bnewbold-400-bad-revisions' into 'master'Martin Czygan2020-03-262-2/+14
|\ \ \ | | | | | | | | | | | | | | | | catch ApiValueError in some generic API calls See merge request webgroup/fatcat!35
| * | | catch ApiValueError in some generic API callsBryan Newbold2020-03-252-2/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The motivation for this change is to handle bogus revision IDs in URLs, which were causing 500 errors not 400 errors. Eg: https://qa.fatcat.wiki/file/rev/5d5d5162-b676-4f0a-968f-e19dadeaf96e%2B2019-11-27%2B13:49:51%2B0%2B6 I have no idea where these URLs are actually coming from, but they should be 4xx not 5xx. Investigating made me realize there is a whole category of ApiValueError exceptions we were not catching and should have been.
* | | | Merge branch 'bnewbold-citeproc-fixes' into 'master'bnewbold2020-03-265-22/+72
|\ \ \ \ | |_|_|/ |/| | | | | | | | | | | improve citeproc/CSL web interface See merge request webgroup/fatcat!36
| * | | improve citeproc/CSL web interfaceBryan Newbold2020-03-255-22/+72
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This tries to show the citeproc (bibtext, MLA, CSL-JSON) options for more releases, and not show the links when they would break. The primary motivation here is to work around two exceptions being thrown in prod every day (according to sentry): KeyError: 'role' ValueError: CLS requries some surname (family name) I'm guessing these are mostly coming from crawlers following the citeproc links on release landing pages.
* | | | datacite: nameIdentifier corner caseBryan Newbold2020-03-261-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Works around a bug in production: AttributeError: 'NoneType' object has no attribute 'replace' (datacite.py:724) NOTE: there are no tests for this code path
* | | | api spec: fix a typoMartin Czygan2020-03-261-1/+1
| |/ / |/| |
* | | Merge branch 'martin-pubmed-bulk-edit-notes' into 'master'Martin Czygan2020-03-241-2/+22
|\ \ \ | |/ / |/| | | | | | | | notes: pubmed backfill (03/2020) See merge request webgroup/fatcat!34
| * | notes: pubmed backfill (03/2020)Martin Czygan2020-03-241-2/+22
|/ /
* | cleanup unused code in fatcat_harvest.pyBryan Newbold2020-03-231-7/+0
| |
* | jalc: avoid meaningless pages valuesBryan Newbold2020-03-231-4/+8
| |
* | Merge branch 'bnewbold-datacite-year-limits' into 'master'Martin Czygan2020-03-231-0/+7
|\ \ | | | | | | | | | | | | datacite: add year sanity restrictions See merge request webgroup/fatcat!33
| * | datacite: add year sanity restrictionsbnewbold2020-03-231-0/+7
|/ / | | | | | | | | | | | | | | | | Example of entities with bogus years: https://fatcat.wiki/release/search?q=doi_registrar%3Adatacite+year%3A%3E2100 We can do a clean-up task, but first need to prevent creation of new bad metadata.
* | notes on arxiv+pubmed backfillBryan Newbold2020-03-201-0/+37
| |
* | pubmed: handle multiple ReferenceListBryan Newbold2020-03-203-1/+222
| | | | | | | | | | | | | | This resolves a situation noticed in prod where we were only importing/updating a single reference per article. Includes a regression test.
* | pubmed: update many more metadata fieldsBryan Newbold2020-03-191-0/+22
| | | | | | | | | | | | | | In particular, with daily updates in most cases the DOI will be registered first, then the entity updated with PMID when that is available. Often the pubmed metadata will be more complete, with abstracts etc, and we'll want those improvements.
* | crossref: skip stub OUP titleBryan Newbold2020-03-191-0/+8
| | | | | | | | | | | | It seems like OUP pre-registers DOIs with this place-holder title, then updates the Crossref metdata when the paper is actually published. We should wait until the real title is available before creating an entity.
* | ingest: always try some lancet journalsBryan Newbold2020-03-191-0/+3
| |
* | Merge branch 'martin-lookup-by-identifier-issn-link' into 'master'bnewbold2020-03-181-4/+3
|\ \ | | | | | | | | | | | | container lookup: link to issn portal search See merge request webgroup/fatcat!32
| * | container lookup: link to issn portal searchMartin Czygan2020-03-181-4/+3
|/ / | | | | | | | | | | | | | | | | Example: https://fatcat.wiki/container/lookup?issnl=2007-1248 - the linked https://portal.issn.org/2007-1248 yields a "page not found", while search yields results: https://portal.issn.org/api/search?search[]=MUST=allissnbis=2007-1248
* | Merge branch 'bnewbold-update-stats' into 'master'Martin Czygan2020-03-181-3/+3
|\ \ | | | | | | | | | | | | update front-page stats See merge request webgroup/fatcat!31
| * | update front-page statsBryan Newbold2020-03-171-3/+3
|/ /
* | bulk exports README different from SQL READMEBryan Newbold2020-03-171-1/+1
| |
* | Merge branch 'martin-kafka-bs4-import' into 'master'Martin Czygan2020-03-1010-43/+428
|\ \ | | | | | | | | | | | | pubmed and arxiv harvest preparations See merge request webgroup/fatcat!28
| * | common: use smaller batch size since XML parsing may be slowMartin Czygan2020-03-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | Address kafka tradeoff between long and short time-outs. Shorter time-outs would facilitate > consumer group re-balances and other consumer group state changes [...] in a reasonable human time-frame.
| * | pubmed: log to stderrMartin Czygan2020-03-101-1/+1
| | |
| * | pubmed: move mapping generation out of fetch_dateMartin Czygan2020-03-102-7/+10
| | | | | | | | | | | | | | | * fetch_date will fail on missing mapping * adjust tests (test will require access to pubmed ftp)
| * | harvest: fix imports from HarvestPubmedWorker cleanupMartin Czygan2020-03-102-4/+4
| | |
| * | pubmed: citations is a bit more preciseMartin Czygan2020-03-091-1/+1
| | | | | | | | | | | | | | | > Each day, NLM produces update files that include new, revised and deleted citations. -- ftp://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/README.txt
| * | pubmed: we sync from FTPMartin Czygan2020-03-091-1/+1
| | |
| * | oaipmh: HarvestPubmedWorker obsoleted by PubmedFTPWorkerMartin Czygan2020-03-091-34/+0
| | |
| * | fatcat_import: address potential hanging, if stdin is emptyMartin Czygan2020-03-091-0/+2
| | |
| * | more pubmed adjustmentsMartin Czygan2020-02-226-71/+197
| | | | | | | | | | | | | | | * regenerate map in continuous mode * add tests
| * | pubmed ftp: fix urlMartin Czygan2020-02-191-4/+6
| | |
| * | pubmed ftp harvest and KafkaBs4XmlPusherMartin Czygan2020-02-196-21/+307
| | | | | | | | | | | | | | | | | | | | | * add PubmedFTPWorker * utils are currently stored alongside pubmed (e.g. ftpretr, xmlstream) but may live elsewhere, as they are more generic * add KafkaBs4XmlPusher
* | | add --force-crawl flag to ingest toolBryan Newbold2020-03-021-0/+5
| |/ |/|
* | pipenv: lock authlib to less than v0.13; rebuild lock fileBryan Newbold2020-02-282-112/+109
| |
* | ES README: really need to limit to 1k esbulk batchesBryan Newbold2020-02-261-3/+3
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-27 19:06:06 +0000