aboutsummaryrefslogtreecommitdiffstats
path: root/python
diff options
context:
space:
mode:
authorBryan Newbold <bnewbold@robocracy.org>2019-05-29 18:34:11 -0700
committerBryan Newbold <bnewbold@robocracy.org>2019-05-29 18:34:11 -0700
commiteaf5039116fbde796d9905c4986fc6a308a36ff4 (patch)
treeff73d47b15232d85a96d0c32e19691415298ea8f /python
parent57c05973ed8e9648bcd95dcf0a43f64d4fe697d1 (diff)
downloadfatcat-eaf5039116fbde796d9905c4986fc6a308a36ff4.tar.gz
fatcat-eaf5039116fbde796d9905c4986fc6a308a36ff4.zip
fix 'fd' XML pattern for bulk imports
Diffstat (limited to 'python')
-rw-r--r--python/README_import.md5
1 files changed, 2 insertions, 3 deletions
diff --git a/python/README_import.md b/python/README_import.md
index 027ce7ad..43100db1 100644
--- a/python/README_import.md
+++ b/python/README_import.md
@@ -79,7 +79,7 @@ Single file:
Bulk (one file per process):
- fd .xml /srv/fatcat/datasets/arxiv_raw_oai_snapshot_2019-05-22/ | parallel -j15 ./fatcat_import.py arxiv {}
+ fd '.xml$' /srv/fatcat/datasets/arxiv_raw_oai_snapshot_2019-05-22/ | parallel -j15 ./fatcat_import.py arxiv {}
## PubMed
@@ -93,8 +93,7 @@ Run single:
Bulk:
- # very memory intensive to parse these big XML files, so need to limit parallelism
- fd .xml /srv/fatcat/datasets/pubmed_medline_baseline_2019 | time parallel -j3 ./fatcat_import.py pubmed {} /srv/fatcat/datasets/ISSN-to-ISSN-L.txt
+ fd '.xml$' /srv/fatcat/datasets/pubmed_medline_baseline_2019 | time parallel -j16 ./fatcat_import.py pubmed {} /srv/fatcat/datasets/ISSN-to-ISSN-L.txt
## Matched