Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | notes on arxiv+pubmed backfill | Bryan Newbold | 2020-03-20 | 1 | -0/+37 |
| | |||||
* | pubmed: handle multiple ReferenceList | Bryan Newbold | 2020-03-20 | 3 | -1/+222 |
| | | | | | | | This resolves a situation noticed in prod where we were only importing/updating a single reference per article. Includes a regression test. | ||||
* | pubmed: update many more metadata fields | Bryan Newbold | 2020-03-19 | 1 | -0/+22 |
| | | | | | | | In particular, with daily updates in most cases the DOI will be registered first, then the entity updated with PMID when that is available. Often the pubmed metadata will be more complete, with abstracts etc, and we'll want those improvements. | ||||
* | crossref: skip stub OUP title | Bryan Newbold | 2020-03-19 | 1 | -0/+8 |
| | | | | | | It seems like OUP pre-registers DOIs with this place-holder title, then updates the Crossref metdata when the paper is actually published. We should wait until the real title is available before creating an entity. | ||||
* | ingest: always try some lancet journals | Bryan Newbold | 2020-03-19 | 1 | -0/+3 |
| | |||||
* | Merge branch 'martin-lookup-by-identifier-issn-link' into 'master' | bnewbold | 2020-03-18 | 1 | -4/+3 |
|\ | | | | | | | | | container lookup: link to issn portal search See merge request webgroup/fatcat!32 | ||||
| * | container lookup: link to issn portal search | Martin Czygan | 2020-03-18 | 1 | -4/+3 |
|/ | | | | | | | | | Example: https://fatcat.wiki/container/lookup?issnl=2007-1248 - the linked https://portal.issn.org/2007-1248 yields a "page not found", while search yields results: https://portal.issn.org/api/search?search[]=MUST=allissnbis=2007-1248 | ||||
* | Merge branch 'bnewbold-update-stats' into 'master' | Martin Czygan | 2020-03-18 | 1 | -3/+3 |
|\ | | | | | | | | | update front-page stats See merge request webgroup/fatcat!31 | ||||
| * | update front-page stats | Bryan Newbold | 2020-03-17 | 1 | -3/+3 |
|/ | |||||
* | bulk exports README different from SQL README | Bryan Newbold | 2020-03-17 | 1 | -1/+1 |
| | |||||
* | Merge branch 'martin-kafka-bs4-import' into 'master' | Martin Czygan | 2020-03-10 | 10 | -43/+428 |
|\ | | | | | | | | | pubmed and arxiv harvest preparations See merge request webgroup/fatcat!28 | ||||
| * | common: use smaller batch size since XML parsing may be slow | Martin Czygan | 2020-03-10 | 1 | -1/+1 |
| | | | | | | | | | | | | | | | | Address kafka tradeoff between long and short time-outs. Shorter time-outs would facilitate > consumer group re-balances and other consumer group state changes [...] in a reasonable human time-frame. | ||||
| * | pubmed: log to stderr | Martin Czygan | 2020-03-10 | 1 | -1/+1 |
| | | |||||
| * | pubmed: move mapping generation out of fetch_date | Martin Czygan | 2020-03-10 | 2 | -7/+10 |
| | | | | | | | | | | * fetch_date will fail on missing mapping * adjust tests (test will require access to pubmed ftp) | ||||
| * | harvest: fix imports from HarvestPubmedWorker cleanup | Martin Czygan | 2020-03-10 | 2 | -4/+4 |
| | | |||||
| * | pubmed: citations is a bit more precise | Martin Czygan | 2020-03-09 | 1 | -1/+1 |
| | | | | | | | | | | > Each day, NLM produces update files that include new, revised and deleted citations. -- ftp://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/README.txt | ||||
| * | pubmed: we sync from FTP | Martin Czygan | 2020-03-09 | 1 | -1/+1 |
| | | |||||
| * | oaipmh: HarvestPubmedWorker obsoleted by PubmedFTPWorker | Martin Czygan | 2020-03-09 | 1 | -34/+0 |
| | | |||||
| * | fatcat_import: address potential hanging, if stdin is empty | Martin Czygan | 2020-03-09 | 1 | -0/+2 |
| | | |||||
| * | more pubmed adjustments | Martin Czygan | 2020-02-22 | 6 | -71/+197 |
| | | | | | | | | | | * regenerate map in continuous mode * add tests | ||||
| * | pubmed ftp: fix url | Martin Czygan | 2020-02-19 | 1 | -4/+6 |
| | | |||||
| * | pubmed ftp harvest and KafkaBs4XmlPusher | Martin Czygan | 2020-02-19 | 6 | -21/+307 |
| | | | | | | | | | | | | | | * add PubmedFTPWorker * utils are currently stored alongside pubmed (e.g. ftpretr, xmlstream) but may live elsewhere, as they are more generic * add KafkaBs4XmlPusher | ||||
* | | add --force-crawl flag to ingest tool | Bryan Newbold | 2020-03-02 | 1 | -0/+5 |
| | | |||||
* | | pipenv: lock authlib to less than v0.13; rebuild lock file | Bryan Newbold | 2020-02-28 | 2 | -112/+109 |
| | | |||||
* | | ES README: really need to limit to 1k esbulk batches | Bryan Newbold | 2020-02-26 | 1 | -3/+3 |
| | | |||||
* | | Merge branch 'bnewbold-elastic-v03b' | Bryan Newbold | 2020-02-26 | 16 | -257/+674 |
|\ \ | |||||
| * | | improve is_oa flag accuracy | Bryan Newbold | 2020-02-26 | 2 | -10/+6 |
| | | | | | | | | | | | | | | | | | | Particularly, the ezb=green match seems mostly incorrect. Note that pmcid being assigned could still be in an embargo window? | ||||
| * | | update ES transform README | Bryan Newbold | 2020-02-26 | 1 | -2/+3 |
| | | | | | | | | | | | | | | | - smaller batch sizes to prevent esbulk errors - file transform/index | ||||
| * | | fix fatcat_transform state filters | Bryan Newbold | 2020-02-26 | 1 | -4/+4 |
| | | | |||||
| * | | bulk ES transform: skip non-active entities | Bryan Newbold | 2020-02-26 | 1 | -0/+8 |
| | | | |||||
| * | | ES container last tweaks | Bryan Newbold | 2020-02-26 | 2 | -3/+7 |
| | | | |||||
| * | | ES release: last minor tweaks | Bryan Newbold | 2020-02-26 | 2 | -5/+7 |
| | | | |||||
| * | | ES updates: fix tests to accept archive.org in host/domain | Bryan Newbold | 2020-02-14 | 1 | -2/+3 |
| | | | |||||
| * | | release schema: do doc_value on DOIs | Bryan Newbold | 2020-02-13 | 1 | -1/+1 |
| | | | | | | | | | | | | | | | | | | Because DOIs are pseudo-structured (prefix, and often structure within the publisher-controlled area), I suspect we will in fact be wanting to do analytics over these strings. | ||||
| * | | ES files: don't remove archive.org domains/hosts | Bryan Newbold | 2020-02-07 | 1 | -5/+0 |
| | | | |||||
| * | | ES release: actually do want doc_values for work_id | Bryan Newbold | 2020-02-05 | 1 | -1/+1 |
| | | | | | | | | | | | | Eg, for fast "unique count" | ||||
| * | | fix axiv/arxiv typo in release schema | Bryan Newbold | 2020-02-04 | 1 | -1/+1 |
| | | | |||||
| * | | ES release schema: fix typo | Bryan Newbold | 2020-01-31 | 1 | -1/+1 |
| | | | |||||
| * | | ES releases: host/domain fixes | Bryan Newbold | 2020-01-31 | 2 | -2/+5 |
| | | | |||||
| * | | pipenv: lock zipp version to work around python3.6 requirement | Bryan Newbold | 2020-01-30 | 2 | -7/+20 |
| | | | |||||
| * | | fix release es transform missing 'issue' | Bryan Newbold | 2020-01-30 | 1 | -0/+1 |
| | | | |||||
| * | | fix json typos in changelog schema | Bryan Newbold | 2020-01-30 | 1 | -2/+2 |
| | | | |||||
| * | | add upper-case work-around from kibana map join | Bryan Newbold | 2020-01-30 | 2 | -0/+2 |
| | | | |||||
| * | | JSON typo in release mapping | Bryan Newbold | 2020-01-30 | 1 | -1/+0 |
| | | | |||||
| * | | ES schemas: make keywords case-insensitive by default | Bryan Newbold | 2020-01-30 | 4 | -66/+115 |
| | | | | | | | | | | | | But not applying asciifolding; don't see any need to do so? | ||||
| * | | tweak file ES archive.org domain tracking | Bryan Newbold | 2020-01-30 | 2 | -0/+7 |
| | | | |||||
| * | | implement host+domain parsing for file ES transform | Bryan Newbold | 2020-01-30 | 2 | -13/+8 |
| | | | |||||
| * | | pipenv: add tldextract (url parser) and update deps | Bryan Newbold | 2020-01-30 | 2 | -136/+159 |
| | | | |||||
| * | | fix ES file schema plural field names | Bryan Newbold | 2020-01-29 | 2 | -5/+4 |
| | | | |||||
| * | | new biblio-only general search | Bryan Newbold | 2020-01-29 | 1 | -2/+2 |
| | | | | | | | | | | | | The other fields are now "copy_to" the merged biblio field. |