# V4 Have not release v3, but many change to `skate` so we may continue with v4. # Unstructured ``` { "biblio": { "unstructured": "J. Häger, W. Krieger, T. Rüegg, and H. Walther, J. Chem. Phys. 72, 4286 (1980).JCPSA60021-9606" }, "index": 8, "key": "_r4", "ref_source": "crossref", "release_year": 1983, "release_ident": "tebzylkszzbyjggye5ssmebdcy", "work_ident": "aaaofyp6uzcdnbe7hvfahylyha" } ``` We should be able to match: "J. Chem. Phys.", also maybe "72, 4286 (1980)", w/ id, title we should be able to match to: https://fatcat.wiki/release/d2k7en7tzzdzzddwfo5xlqx4ce If nothing else defined, and unstructured contains a URL, we may extract that. ``` { "biblio": { "unstructured": "Friedrich-Ebert-Stiftung (FES) 2008 FES in Nepal. FES http://www.fesnepal.org/about/fes_in_nepal.htm (accessed February 15, 2009)" }, "index": 19, "key": "CIT0020", "ref_source": "crossref", "release_year": 2011, "release_ident": "xqgaanhpf5gotdxxxytgyxw2ty", "work_ident": "aaaq35j3angwzdpzcvzdil3v4y" } ``` Also, these may say: "accessed at ..." # URL * url cleanup in place # Partial Data Mapping * how to map partial docs onto a key # OL beyond ISBN Example: ``` { "biblio": { "container_name": "The Debt: What America Owes to Blacks", "contrib_raw_names": [ "R Robinson" ], "unstructured": "Randall Robinson, The Debt: What America Owes to Blacks. New York: Dutton Books, 2000, pp. 219–220.", "year": 2000 }, "index": 22, "key": "8_CR23", "ref_source": "crossref", "release_year": 2009, "release_ident": "2igycuiobvhxrcmmrzz6anufuq", "work_ident": "aaacj23jqbdxvajwj5kc6jpejq" } ``` * https://openlibrary.org/works/OL488811W/The_debt?edition=debtwhatamerica000robi However, there is no explicit "subtitle" fields, and in this case, the subtitle is buried in "text": ``` { "key": "/works/OL488811W", "text": [ "/works/OL488811W", "The debt", "The Debt", "The Debt ", "what America owes to Blacks", "What America Owes to Blacks", "OL46591M", "OL7771042M", "OL7590904M", "OL3382710M", "Randall Robinson.", "2004556979", "99045728", "0452282101", "0525945245", ``` Subtitle in editions. ``` { "biblio": { "container_name": "BLACK AFRICA: The Economic and Cultural Basis for a Federated State", "unstructured": "For details on African Renaissance see Cheikh Anta Diop, BLACK AFRICA: The Economic and Cultural Basis for a Federated State, New Expanded Edition. Trenton, NJ: Africa World Press, 1987.", "year": 1987 }, "index": 28, "key": "8_CR29", "ref_source": "crossref", "release_year": 2009, "release_ident": "2igycuiobvhxrcmmrzz6anufuq", "work_ident": "aaacj23jqbdxvajwj5kc6jpejq" } ``` ## OL Loop Some do not have an explicit "works" key, but still link to an edition. * https://openlibrary.org/books/OL10000230M/Parliamentary_Debates_House_Of_Lords_2003-2004?edition= > An edition of Parliamentary Debates, House Of Lords 2003-2004 Example edition: ``` { "publishers": [ "Du Temps" ], "languages": [ { "key": "/languages/fre" } ], "last_modified": { "type": "/type/datetime", "value": "2010-04-24T18:46:01.556464" }, "weight": "5 ounces", "title": "Les Fleurs bleues de Raymond Queneau", "identifiers": { "goodreads": [ "487215" ] }, "isbn_13": [ "9782842741013" ], "covers": [ 3140044 ], "physical_format": "Paperback", "isbn_10": [ "2842741013" ], "publish_date": "January 1, 2000", "key": "/books/OL12622734M", "authors": [ { "key": "/authors/OL3964945A" } ], "latest_revision": 5, "works": [ { "key": "/works/OL10000008W" } ], "type": { "key": "/type/edition" }, "physical_dimensions": "8.4 x 5.7 x 0.3 inches", "revision": 5 } ``` Example Work: ``` { "title": "Les Fleurs bleues de Raymond Queneau", "created": { "type": "/type/datetime", "value": "2009-12-11T01:57:19.964652" }, "covers": [ 3140044 ], "last_modified": { "type": "/type/datetime", "value": "2010-04-28T06:54:19.472104" }, "latest_revision": 3, "key": "/works/OL10000008W", "authors": [ { "type": "/type/author_role", "author": { "key": "/authors/OL3964945A" } } ], "type": { "key": "/type/work" }, "revision": 3 } ``` ---- ## Unmatched If we exclude any id and title, we'll roughly have the following fields: ``` container_name|contrib_raw_names|year 64064559 unstructured 61711602 container_name|contrib_raw_names|volume|year 49701699 container_name|contrib_raw_names|unstructured|volume|year 36401044 container_name|contrib_raw_names|unstructured|year 26663422 contrib_raw_names|unstructured 16731608 container_name|contrib_raw_names|doi|unstructured|year 14207167 container_name|contrib_raw_names|doi|year 13159340 ``` Some examples: ``` { "biblio": { "container_name": "Intern. J. Comput. Math.", "contrib_raw_names": [ "D. Levin" ], "volume": "B3", "year": 1973 }, "index": 19, "key": "PhysRevB.48.6913Cc15R1", "ref_source": "crossref", "release_year": 1993, "release_ident": "i6s6e64n55hh5oned32mdwrs2i", "work_ident": "aaaeuvgitzfafczctw3bseauri" } ``` This refers to: * https://www.tandfonline.com/doi/abs/10.1080/00207167308803075 * 1972, and not 1973, 1993 * https://fatcat.wiki/release/3cstmufhszalvpnppwxjohnnsa It would help to go from "container name" to "issn", e.g. here: 0020-7160 * https://fatcat.wiki/release/search?q=levin+container_id%3A%22y4k3i2fvabgarkvywismzvy23a%22+year%3A1972 ``` $ grep -i "Intern.*J.*Comput.*Math.*" jabbrev.json {"name": "COMPEL-THE INTERNATIONAL JOURNAL FOR COMPUTATION AND MATHEMATICS IN ELECTRICAL AND ELECTRONIC ENGINEERING", "abbrev": "COMPEL"} {"name": "INTERNATIONAL JOURNAL OF APPLIED MATHEMATICS AND COMPUTER SCIENCE", "abbrev": "INT J APPL MATH COMP"} {"name": "INTERNATIONAL JOURNAL OF COMPUTER MATHEMATICS", "abbrev": "INT J COMPUT MATH"} ``` Lookup name in issn: ``` $ zstdcat tmp/data.ndj.zst | grep -i "INTERNATIONAL JOURNAL OF COMPUTER MATHEMATICS" | jq . "@graph": [ { "@id": "http://id.loc.gov/vocabulary/countries/enk", "label": "England" }, { "@id": "organization/ISSNCenter#_1", "@type": "http://schema.org/Organization" }, { "@id": "resource/ISSN-L/0020-7160", "identifiedBy": "resource/ISSN/0020-7160#ISSN-L" }, { "@id": "resource/ISSN/0020-7160", "@type": [ "http://id.loc.gov/ontologies/bibframe/Instance", "http://id.loc.gov/ontologies/bibframe/Work", "http://schema.org/Periodical" ], "format": "vocabularies/medium#Print", "http://purl.org/ontology/bibo/issn": "0020-7160", "identifiedBy": [ "resource/ISSN/0020-7160#ISSN-L", "resource/ISSN/0020-7160#ISSN", "resource/ISSN/0020-7160#KeyTitle" ], ``` We would need: * rough abbrev name -> full name (jabbrev) -> issn (issnlister) -> container id (fatcat) Example, title match with OL: ``` { "biblio": { "container_name": "Private schooling in less economically developed countries", "contrib_raw_names": [ "Caddell M." ], "year": 2008 }, "index": 7, "key": "CIT0008", "ref_source": "crossref", "release_year": 2011, "release_ident": "xqgaanhpf5gotdxxxytgyxw2ty", "work_ident": "aaaq35j3angwzdpzcvzdil3v4y" } ``` A matching OL edition record: ``` { "publishers": [ "Symposium Books" ], "languages": [ { "key": "/languages/eng" } ], "number_of_pages": 214, "subtitle": "Asian and African Perspectives (Oxford Studies in Comparative Education)", "weight": "12.6 ounces", "title": "Private Schooling in Less Economically Developed Countries", "isbn_10": [ "1873927851" ], "type": { "key": "/type/edition" }, "identifiers": { "goodreads": [ "1078335" ] }, "isbn_13": [ "9781873927854" ], "covers": [ 3020365 ], "physical_format": "Paperback", "key": "/books/OL12102259M", "publish_date": "April 1, 2007", "contributions": [ "Prachi Srivastava (Editor)", "Geoffrey Walford (Editor)" ], "subjects": [ "Organization & management of education", "ASIA", "Africa", "Reference / General" ], "physical_dimensions": "9.1 x 6.1 x 0.7 inches", "works": [ { "key": "/works/OL24081822W" } ], "lccn": [ "2007408632" ], "lc_classifications": [ "LC57.5 .P75 2007" ], ``` ---- # Completeness ``` { "biblio": { "container_name": "La vida y época de Prebisch", "year": 2010 }, "index": 5, "key": "key20191115064515_B6", "ref_source": "crossref", "release_year": 2019, "release_ident": "oc6nhkoah5gcnjfsjpct4ij3ea", "work_ident": "aaachbf2kbdnxekwdujbmnlw4a" } ``` * https://fatcat.wiki/release/oc6nhkoah5gcnjfsjpct4ij3ea/references * https://www.iberoamericana.se/articles/10.16993/iberoamericana.467/galley/445/download/ In the PDF, we find a DOI as well, but it seems to be not extracted. In fact; the ref data comes from crossref. Grobid gets the DOI: ``` La vida y época de Prebisch. 1901-1986. Madrid: Marcial Pons EJDosman 10.18356/40a5d411-es ``` Other issues: * year vs release_year ``` { "biblio": { "container_name": "The Methodology of Scientific Research Programmes", "year": 1980 }, "index": 13, "key": "key20191115064515_B14", "ref_source": "crossref", "release_year": 2019, "release_ident": "oc6nhkoah5gcnjfsjpct4ij3ea", "work_ident": "aaachbf2kbdnxekwdujbmnlw4a" } ``` ## Conservative Verification * closeby, but different years, although it seems it would actually be a match ``` different year /works/OL13199655W lvtfhk63kjbthacu2aam3jgudu 1000000delinquents 1000000 delinquents 1,000,000 Delinquents different year /works/OL13199655W wpp46slm6nca7b3nwdwtjbegla 1000000delinquents 1000000 delinquents 1,000,000 Delinquents different year /works/OL13199655W gvzsp7pz75d6roxrrarduprmie 1000000delinquents 1000000 delinquents 1,000,000 Delinquents different year /works/OL13199655W ilga2kj4nnaqdh4rmogbsgbgbe 1000000delinquents 1000000 delinquents 1,000,000 Delinquents different year /works/OL13199655W 5ujpef3vjzhkvmse6ovey2q2zi 1000000delinquents 1000000 delinquents 1,000,000 Delinquents ``` ## Journal name augmentation In ~160M unmatched refs (release format) we could resolve 14M container names, via `skate-resolve-journal-name`. ``` $ zstdcat date-2021-05-06.tsv.zst | skate-resolve-journal-name -B -A /magna/data/jabbrev.json | cut -f 2 | pv -l | LC_ALL=C grep -cF resolved_container_name 2021/06/01 13:02:20 found 27178 abbreviation mappings 160M 0:14:49 [ 180k/s] [ <=> ] 14090677 ``` ## Discrepancy * https://fatcat.wiki/release/cgmnjwrhlvccxnxyewd4buuhzm/references UnmatchedRefs contains entry 11: ``` { "biblio": { "container_name": "Med J Aust", "contrib_raw_names": [ "Dracup K" ], "volume": "166", "year": 1997 }, "index": 9, "key": "bibr11-010740830802800102", "ref_source": "crossref", "release_year": 2008, "release_ident": "cgmnjwrhlvccxnxyewd4buuhzm", "work_ident": "aabzzlohgza2pfaol7cgqlvpke" } ``` In frontend, we only have a DOI; https://fatcat.wiki/release/lookup?doi=10.1016/s0147-9563(97)90082-0 ## More OL matching > ran open library and fatcat fuzzy matching (via container name) on all docs, > that did not have an id-based match; found 139M link candidates, of which > verification 11M strong or exact matches, of which around 3M had some IA > identifier (about 200K unique; but looking at a few of them, it seems these were somewhat > restricted items, e.g. "print-disabled" Most referenced items were: ``` 13010 ia:discoverygrounde00glas 9351 ia:selfefficacyexer0000band 8341 ia:basicsofqualitat0000stra 7562 ia:researchdesignqu00cres 7027 ia:basicsqualitativ00stra 6397 ia:qualitativedataa00mile 5958 ia:briefhistoryneol00harv 5779 ia:constructinggrou00char 5291 ia:reassemblingsoci00lato 4908 ia:econometricanaly0000gree_f5q0 4762 ia:powerknowledges00fouc 4733 ia:stressappraisalc00rich 4673 ia:locationculture00bhab_220 4405 ia:threeworldswelfa00espi 4330 ia:bodiesthatmatter00butl_662 4306 ia:gendertroublefem0000butl_d7d5 4249 ia:contentanalysisi00krip 4059 ia:practiceofeveryd01cert 4013 ia:intermolecularsu00isra 3925 ia:culturesorganiza00hofs 3621 ia:fractalgeometryo00beno 3564 ia:modernityatlarge00appa 3483 ia:modelselectionmu00burn_141 3454 ia:economicinstitut00will 3300 ia:economictheoryof00down_0 3289 ia:seeinglikestateh00scot_250 3204 ia:densityfunctiona00parr 3158 ia:experienceeducat00dewe_0 3127 ia:infraredramanspe00naka 3114 ia:qualitativeinqui00cres_711 3088 ia:sciencehumanbeha00bfsk 3050 ia:structuralequati0000byrn_g1v4 3050 ia:numericalrecipes0000unse_j9c5 3003 ia:conductionheatso00cars 2955 ia:fifthdisciplineasen00seng 2793 ia:homosacersoverei00agam_937 2746 ia:naturalisticinqu00linc 2737 ia:postmoderncondit00lyot_037 2673 ia:mathematicsdiffu00cran 2576 ia:principlespracti0000klin 2530 ia:hydrodynamichydr00chan 2499 ia:strategicmanagem00free 2493 ia:viscoelasticprop00ferr 2454 ia:wehaveneverbeenm00lato_404 2428 ia:foucaultreader00fouc 2386 ia:languagesymbolic0000bour_1991 2381 ia:weaponsofweakeve0000scot 2380 ia:greattransforma000pola 2370 ia:crossingqualityc00amer_984 ``` ## Glitch in GS? The fractal geometry of nature is "cited by 47428" (https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22fractal+geometry+of+nature%22&btnG=); on page one of references, there is "Gaussian processes in machine learning" (2003), via: https://www.researchgate.net/profile/Olivier_Bousquet/publication/238718428_Advanced_Lectures_on_Machine_Learning_ML_Summer_Schools_2003_Canberra_Australia_February_2-14_2003_Tubingen_Germany_August_4-16_2003_Revised_Lectures/links/02e7e52c5870850311000000/Advanced-Lectures-on-Machine-Learning-ML-Summer-Schools-2003-Canberra-Australia-February-2-14-2003-Tuebingen-Germany-August-4-16-2003-Revised-Lectures.pdf#page=70 - the paper itself does not contain a reference -- in the whole document. ## OL fuzzy different Reasons, why pairs were marked as *different*: ``` $ zstdcat -T UnmatchedOpenLibraryMatchTable/date-2021-05-06.tsv.zst | grep ^different | cut -f2 | LC_ALL=C sort -S50% | uniq -c | sort -nr 47324670 year 46016349 contribintersectionempty 582618 pagecount 460 titlefilename 25 numdiff ``` The `year` may refer to different editions: * https://fatcat.wiki/release/kngofkvoo5cinj4wqerrey4tpi/references * https://openlibrary.org/works/OL16286792W/One_hundred_and_seventeen_days?edition=onehundredsevent00firs > 117 Days: An Account of Confinement and Interrogation under the South African > 90-Day Detention Law.2006 | vs This edition was published in 1965 by Penguin > Books ## Data mismatch * FE: https://fatcat.wiki/release/niivpohpabhajdsf35x7hr4efm/references, [8]: 2011 refs (2017 only) ``` { "container_name": "19 & 20: Notes for a New Social Protagonism", "container": { "container_type": "", "ident": "", "issnl": "", "name": "", "publisher": "", "revision": "", "state": "", "wikidata_qid": "" }, "contribs": [ { "raw_name": "Colective Situaciones" } ], "ext_ids": {}, "ident": "niivpohpabhajdsf35x7hr4efm", "release_year": "2017", "work_id": "7eghl5lcivfmha6d4uavrrkpce", "extra": { "crossref": {}, "datacite": {}, "skate": { "status": "ref", "ref": { "index": 7, "key": "\nkey\n\t\t\t\t20171225032503_CIT0007" }, "rg": {}, "resolved_container_name": "" }, "ol": {} } } { "container_name": "A Dictionary of Marxist Thought (2nd ed.)", "container": { "container_type": "", "ident": "", "issnl": "", "name": "", "publisher": "", "revision": "", "state": "", "wikidata_qid": "" }, "ext_ids": {}, "ident": "niivpohpabhajdsf35x7hr4efm", "release_year": "2017", "title": "Price of production and the transformation problem", "work_id": "7eghl5lcivfmha6d4uavrrkpce", "extra": { "crossref": {}, "datacite": {}, "skate": { "status": "ref", "ref": { "index": 12, "key": "\nkey\n\t\t\t\t20171225032503_CIT0012" }, "rg": {}, "resolved_container_name": "" }, "ol": {} } } { "container_name": "A Grammar of the Multitude: For an Analysis of Contemporary Forms of Life", "container": { "container_type": "", "ident": "", "issnl": "", "name": "", "publisher": "", "revision": "", "state": "", "wikidata_qid": "" }, "ext_ids": {}, "ident": "niivpohpabhajdsf35x7hr4efm", "release_year": "2017", "work_id": "7eghl5lcivfmha6d4uavrrkpce", "extra": { "crossref": {}, "datacite": {}, "skate": { "status": "ref", "ref": { "index": 45, "key": "\nkey\n\t\t\t\t20171225032503_CIT0044" }, "rg": {}, "resolved_container_name": "" }, "ol": {} } } { "container_name": "An Introduction to the Three Volumes of Karl Marx's Capital", "container": { "container_type": "", "ident": "", "issnl": "", "name": "", "publisher": "", "revision": "", "state": "", "wikidata_qid": "" }, "ext_ids": {}, "ident": "niivpohpabhajdsf35x7hr4efm", "release_year": "2017", "work_id": "7eghl5lcivfmha6d4uavrrkpce", "extra": { "crossref": {}, "datacite": {}, "skate": { "status": "ref", "ref": { "index": 21, "key": "\nkey\n\t\t\t\t20171225032503_CIT0020" }, "rg": {}, "resolved_container_name": "" }, "ol": {} } } ``` ## Grobid misses (ISBN) * PDF: https://web.archive.org/web/20031204233716/http://grace.wharton.upenn.edu:80/~sok/sokpapers/1999-0/indiana-transparency/flbc-transparency.pdf * seems grobid does not recognize ISBN? ``` Electronic data interchange in logistics MargaretAEmmelhainz The Logistics Handbook James F. Robeson and William C. Copacino
New York, NY
 The Free Press WordNet: An Electronic Lexical Database Christiane Fellbaum The MIT Press Cambridge, MA ``` ## Duplicates in combined dataset When we merge matches with refs, we find duplicates, e.g.: ``` { "_id": "4kg2dejsgzaf3cszs2lt5hz4by_9", "indexed_ts": "2021-06-15T15:30:42Z", "source_release_ident": "4kg2dejsgzaf3cszs2lt5hz4by", "source_work_ident": "2222jduvonfg3p2no5gvvf2sj4", "source_year": "2011", "ref_index": 9, "ref_key": "ref9", "target_release_ident": "itntzdjbczfmhhaynqvqcwp6wm", "target_work_ident": "csff4o7yjzbz3mfszl4zvfkcua", "match_provenance": "crossref", "match_status": "exact", "match_reason": "doi" } { "_id": "4kg2dejsgzaf3cszs2lt5hz4by_9", "indexed_ts": "2021-06-15T15:30:42Z", "source_release_ident": "4kg2dejsgzaf3cszs2lt5hz4by", "source_work_ident": "2222jduvonfg3p2no5gvvf2sj4", "source_year": "2011", "ref_index": 9, "ref_key": "b9", "match_status": "unmatched", "match_reason": "unknown", "target_unstructured": "Danaceau JP, Deering CE, Day JE, Smeal SJ, Johnson-Davis KL, et al. (2007) Persistence of tolerance to methamphetamine-induced monoamine deficits. Eur J Pharmacol 559: 46-54." } { "_id": "4kg2dejsgzaf3cszs2lt5hz4by_9", "indexed_ts": "2021-06-15T15:30:42Z", "source_release_ident": "4kg2dejsgzaf3cszs2lt5hz4by", "source_work_ident": "2222jduvonfg3p2no5gvvf2sj4", "source_year": "2011", "ref_index": 9, "ref_key": "ref10", "match_status": "unmatched", "match_reason": "unknown" } ``` Here, the ref index is 9, but ref keys are different, which might come from a different grobid run. I feel, we should not depend on a value that we have little control over. As a mititgation, we'll run a final deduplication step; but that won't catch all duplicates, e.g. when the indices are different, but the reference is actually the same. Would need to "uniq" tool for the whole ref blob or something like that. ---- ## QA: duplicates There seem to be many self-links in the dataset: * sample: 25668733, duplicate rows: 1913155; about 8% (although only 145030 uniq; many repetitions) ``` $ LC_ALL=C awk '$1 == $2' bref_tabs.tsv # .... 56fbxcue6rdxlmxqto7vibg2xi 56fbxcue6rdxlmxqto7vibg2xi exact doi crossref o2juqzskxzdtpbait5gxg3yf4q o2juqzskxzdtpbait5gxg3yf4q exact doi crossref 6mwdlhvbljgtdntz5qifywhsn4 6mwdlhvbljgtdntz5qifywhsn4 exact doi crossref t7vluqxmgbe4pipf4nkfcayedq t7vluqxmgbe4pipf4nkfcayedq exact doi crossref iofm6brptvczlnrys5vxw34x3i iofm6brptvczlnrys5vxw34x3i exact doi crossref soa44abzivcnfnsx4ymxvbyg44 soa44abzivcnfnsx4ymxvbyg44 exact doi crossref 7fs4c3u2ofcmxie344o5e4wuxi 7fs4c3u2ofcmxie344o5e4wuxi exact doi crossref igyewr6er5epfozhk7dyfqa5tu igyewr6er5epfozhk7dyfqa5tu exact doi crossref ``` ---- # Numbers * old crossref dump; no partial matches yet * 740,248,530 unique edges ---- # Stats ``` 553414112 exact doi 75738037 strong jaccardauthors 66257136 exact pmid 19646986 strong slugtitleauthormatch 17202451 strong tokenizedauthors 3730080 exact arxiv 2798816 exact titleauthormatch 482811 strong versioneddoi 303336 strong pmiddoipair 279212 exact isbn 240405 exact workid 52678 strong customieeearxiv 43797 strong dataciterelatedid 29027 strong arxivversion 27150 exact pmcid 1652 strong figshareversion 832 strong titleartifact 10 strong custombsiundated 2 strong custombsisubdoc ``` * total unique edges: 740248530 * matches by id: 623707690 * matches though title/author (fuzzy) matching: 116540840 * scholarly resources: 727853720 * linked open library titles: 12394810 * URLs extracted from corpus: 25405592 * sample ratio IA/URL from corpus (N=100000): ---- # Timing Completed 24/25 jobs in 42h with a few failures from disk space issues. * 2562m34.844s ---- # Stats * BrefCombined ``` 1,175,653,287 exact doi 540,957,187 unmatched unknown 62,621,153 strong jaccardauthors 56,950,207 exact pmid 11,942,348 strong slugtitleauthormatch 10,278,122 strong tokenizedauthors 3,419,958 exact arxiv 2,479,491 exact titleauthormatch 592,522 exact isbn 462,252 strong versioneddoi 91,076 strong pmiddoipair 77,030 strong customieeearxiv 57,290 strong dataciterelatedid 33,837 strong arxivversion 18,448 exact pmcid 1,749 exact workid 1,241 strong figshareversion 557 strong titleartifact 10 strong custombsiundated 2 strong custombsisubdoc ``` Of the 540M unmatched, we have: ``` Unstr CSL 404285013 true false 136672174 false true ``` ---- ## Web links A single copy in archive, and no-live web copy: * [http://web.archive.org/web/20180509015159/http://0-hera.ugr.es.adrastea.ugr.es/tesisugr/21007196.pdf](http://web.archive.org/web/20180509015159/http://0-hera.ugr.es.adrastea.ugr.es/tesisugr/21007196.pdf) * [http://web.archive.org/web/20130108031412/http://www.birdlife.org/datazone/userfiles/file/IBAs/AfricaCntryPDFs/Djibouti.pdf](http://web.archive.org/web/20130108031412/http://www.birdlife.org/datazone/userfiles/file/IBAs/AfricaCntryPDFs/Djibouti.pdf) ---- # Rerun 2021-07-28 Only a single thread; ssd tmp; * 1340min, then low disk space * another 428min for the remaining two tasks * total:1768min, about 30h Performance headroom: run all mappers in one go; run tasks in parallel Indexing time, about 35h. So whole derivation about 3 days (30h+ for "BrefCombined" and 35h to index). ``` 2021/08/04 09:56:57 [worker-5] @148044000 2021/08/04 09:56:57 2077597833 docs in 127016.87s at 16356.865 docs/s with 14 workers 2021/08/04 09:56:57 applied setting: {"index": {"refresh_interval": "1s"}} with status 200 OK 2021/08/04 09:56:57 applied setting: {"index": {"number_of_replicas": "0"}} with status 200 OK 2021/08/04 09:56:58 index flushed: 200 OK real 2116m57.951s user 2193m20.229s sys 268m34.241s ```