summaryrefslogtreecommitdiffstats
path: root/notes/bulk_edits
diff options
context:
space:
mode:
authorMartin Czygan <martin.czygan@gmail.com>2020-03-24 19:19:32 +0100
committerMartin Czygan <martin.czygan@gmail.com>2020-03-24 19:19:32 +0100
commit06e0ffd981bf59c96a0f0c620871b6961c0055df (patch)
treeaa617719499bb1a2cb534a4d351ee6b42d4a1911 /notes/bulk_edits
parent40fd4863c3c68261119aaa5b8f53d6b9dfc7a4cd (diff)
downloadfatcat-06e0ffd981bf59c96a0f0c620871b6961c0055df.tar.gz
fatcat-06e0ffd981bf59c96a0f0c620871b6961c0055df.zip
notes: pubmed backfill (03/2020)
Diffstat (limited to 'notes/bulk_edits')
-rw-r--r--notes/bulk_edits/2020-03-19_arxiv_pubmed.md24
1 files changed, 22 insertions, 2 deletions
diff --git a/notes/bulk_edits/2020-03-19_arxiv_pubmed.md b/notes/bulk_edits/2020-03-19_arxiv_pubmed.md
index 25220ad3..b2fd29d5 100644
--- a/notes/bulk_edits/2020-03-19_arxiv_pubmed.md
+++ b/notes/bulk_edits/2020-03-19_arxiv_pubmed.md
@@ -1,6 +1,6 @@
On 2020-03-20, automated daily harvesting and importing of arxiv and pubmed
-medata started. In the case of pubmed, updates are enabled, so that recently
+metadata started. In the case of pubmed, updates are enabled, so that recently
created DOI releases get updated with PMID and extra metdata.
We also want to do last backfills of metadata since the last import up through
@@ -34,4 +34,24 @@ Ran fairly quickly only some ~80-90k entities to process.
## PubMed
-TODO: martin will import daily update files from the 2020 baseline through XYZ date.
+First, mirror update files from FTP, e.g. via lftp:
+
+ mkdir -p /srv/fatcat/datasets/pubmed_updates
+ lftp -e 'mirror -c /pubmed/updatefiles /srv/fatcat/datasets/pubmed_updates; bye' ftp://ftp.ncbi.nlm.nih.gov
+
+Inspect completed dates from kafka:
+
+ kafkacat -b $KAFKA_BROKER -t fatcat-prod.ftp-pubmed-state -C
+
+Show dates and corresponding files:
+
+ find /srv/fatcat/datasets/pubmed_updates -name "*html" | xargs cat | grep "Created" | sort
+
+For this bulk import, we used files pubmed20n1016.xml.gz (2019-12-16) up to pubmed20n1110.xml.gz (2020-03-06).
+
+To import the corresponding files, run:
+
+ printf "%s\n" /srv/fatcat/datasets/pubmed_updates/pubmed20n{1016..1110}.xml.gz | shuf | \
+ parallel -j16 'gunzip -c {} | ./fatcat_import.py pubmed --do-updates - /srv/fatcat/datasets/ISSN-to-ISSN-L.txt'
+
+Import took 254 min, there were 1715427 PubmedArticle docs in these update files.