diff options
| -rw-r--r-- | notes/2022_01_10_refcat_update.md | 15 |
1 files changed, 15 insertions, 0 deletions
diff --git a/notes/2022_01_10_refcat_update.md b/notes/2022_01_10_refcat_update.md new file mode 100644 index 0000000..795a9d4 --- /dev/null +++ b/notes/2022_01_10_refcat_update.md @@ -0,0 +1,15 @@ +# Refcat update + +* new refs export, about 10% more (2.7B) +* new fatcat export + +New wikipedia extraction: + +``` +martin@ia601101:/magna/data/wikipedia_citations_2020-07-14 $ LC_ALL=C grep ID_list minimal_dataset.json | grep -c DOI +1442189 + +$ jq -rc '.refs[] | select(.ID_list != null) | {"URL": .URL, "Title": .title, "ID_list": .ID_list}' enwiki-20211201-pages-articles.citations.json | pv -l > minimal.json +$ grep -c DOI minimal.json +1932578 +``` |