fatcat - [no description]

	Commit message (Collapse)	Author	Age	Files	Lines
...
\| * \|	implement host+domain parsing for file ES transform	Bryan Newbold	2020-01-30	2	-13/+8
\| \| \|
\| * \|	pipenv: add tldextract (url parser) and update deps	Bryan Newbold	2020-01-30	2	-136/+159
\| \| \|
\| * \|	fix ES file schema plural field names	Bryan Newbold	2020-01-29	2	-5/+4
\| \| \|
\| * \|	new biblio-only general search	Bryan Newbold	2020-01-29	1	-2/+2
\| \| \| \| \| \| \| \| \| \| \| \|	The other fields are now "copy_to" the merged biblio field.
\| * \|	elastic schema fixes	Bryan Newbold	2020-01-29	3	-7/+12
\| \| \|
\| * \|	add country to v03b release schema	Bryan Newbold	2020-01-29	2	-0/+3
\| \| \|
\| * \|	update ES docs and proposal	Bryan Newbold	2020-01-29	2	-4/+6
\| \| \|
\| * \|	actually implement changelog transform	Bryan Newbold	2020-01-29	3	-19/+78
\| \| \|
\| * \|	fix some transform bugs, add some tests	Bryan Newbold	2020-01-29	6	-13/+48
\| \| \|
\| * \|	ES release schema updates	Bryan Newbold	2020-01-29	2	-28/+122
\| \| \|
\| * \|	container ES schema changes	Bryan Newbold	2020-01-29	2	-29/+38
\| \| \|
\| * \|	first implementation of ES file schema	Bryan Newbold	2020-01-29	4	-3/+115
\| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Includes a trivial test and transform, but not any workers or doc updates.
* \| \|	Merge branch 'bnewbold-more-ingest' into 'master'	bnewbold	2020-02-25	1	-1/+37
\|\ \ \ \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	entity worker: ingest more Datacite releases; filter some out See merge request webgroup/fatcat!29
\| * \| \|	entity worker: ingest more releases	Bryan Newbold	2020-02-22	1	-1/+37
\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	If release is a dataset or image, don't do a pdf ingest request. If release is a datacite DOI, and release_type is a "document", crawl regardless of is_oa detection. This is mostly to crawl repositories (institutional or subject).
* \| \| \|	add some more domain/rel URL mappings	Bryan Newbold	2020-02-22	1	-0/+9
\|/ / /
* \| \|	table size snapshots	Bryan Newbold	2020-02-19	2	-0/+47
\| \| \|
* \| \|	Merge branch 'bnewbold-shadow-import'	Bryan Newbold	2020-02-19	7	-2/+289
\|\ \ \
\| * \| \|	remove arabesque short wayback URL hack	Bryan Newbold	2020-02-14	1	-6/+0
\| \| \| \|
\| * \| \|	clarify shadow filter code	Bryan Newbold	2020-02-13	1	-1/+1
\| \| \| \|
\| * \| \|	filter out shadows-only files in webface	Bryan Newbold	2020-02-13	2	-2/+7
\| \| \| \|
\| * \| \|	improve shadow import file url cleanup path	Bryan Newbold	2020-02-13	1	-2/+12
\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Should probably be refactored out in to shared cleanup code.
\| * \| \|	shadow import fixes from QA testing	Bryan Newbold	2020-02-13	2	-1/+7
\| \| \| \|
\| * \| \|	shadow import: more filtering of file_meta fields	Bryan Newbold	2020-02-13	3	-18/+28
\| \| \| \|
\| * \| \|	basic shadow importer	Bryan Newbold	2020-02-13	5	-0/+262
\| \| \| \|
* \| \| \|	more SQL db size reduction notes	Bryan Newbold	2020-02-19	1	-0/+16
\| \| \| \|
* \| \| \|	always crawl researchgate DOIs	Bryan Newbold	2020-02-18	1	-0/+2
\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Now that ingest is fixed
* \| \| \|	ingest import: fix edit_extra path	Bryan Newbold	2020-02-18	1	-1/+1
\| \| \| \|
* \| \| \|	ingest importer: edit_extra is a top-level key	Bryan Newbold	2020-02-18	1	-1/+1
\| \| \| \|
* \| \| \|	ingest import: allow short version of corpus names	Bryan Newbold	2020-02-18	1	-0/+3
\| \| \| \|
* \| \| \|	ingest importer: pass through link rel	Bryan Newbold	2020-02-18	1	-1/+6
\| \|_\|/ \|/\| \|
* \| \|	fatcat_ingest: as 'fatcat-ingest', not 'fatcat-ingest-container'	Bryan Newbold	2020-02-14	1	-1/+1
\| \| \| \| \| \| \| \| \| \| \| \|	This tool is more generic now.
* \| \|	Merge branch 'martin-harvest-print' into 'master'	bnewbold	2020-02-14	3	-17/+22
\|\ \ \ \| \|/ / \|/\| \| \| \| \| \| \| \|	harvest: log state on startup and use stderr for diagnostics See merge request webgroup/fatcat!27
\| * \|	harvest: log state on startup and use stderr for diagnostics	Martin Czygan	2020-02-14	3	-17/+22
\|/ /
* \|	add acceptlist override for biorxiv/medrxiv	Bryan Newbold	2020-02-10	1	-2/+12
\| \|
* \|	check ingest_request_source existance for SPN as well as ingest	Bryan Newbold	2020-02-06	1	-0/+3
\| \|
* \|	additional trusted link sources	Bryan Newbold	2020-02-06	1	-0/+3
\| \|
* \|	add mag and s2 as trusted link sources	Bryan Newbold	2020-02-06	1	-1/+1
\| \|
* \|	ingest worker: handle missing ingest_request_source	Bryan Newbold	2020-02-06	1	-0/+3
\| \| \| \| \| \| \| \| \| \|	Seeing a bunch of these due to re-ingests not including this field because of an earlier persist bug.
* \|	switch '!= None' to 'is not None'	Bryan Newbold	2020-02-04	1	-3/+3
\| \| \| \| \| \| \| \|	As reminded in code review, thanks Martin.
* \|	allow-non-oa is a top-level flag, not sub-command	Bryan Newbold	2020-02-04	1	-3/+0
\| \|
* \|	default to PMC ingest URLs over DOI	Bryan Newbold	2020-02-04	1	-4/+4
\| \| \| \| \| \| \| \| \| \| \| \| \| \|	For cases where there might be both PMC and DOI urls, do the europmc.org PMC ones over DOI option. May want to turn this into a config or command-line option in the future.
* \|	ingest: add 'extid' and 'query' modes; filters; refactor	Bryan Newbold	2020-02-04	1	-38/+147
\| \| \| \| \| \| \| \| \| \| \| \|	This is a large refactor of the ingest script. It adds a number of filtering options (for all modes), and new modes for free-form queries or limiting to specific external identifiers.
* \|	Merge branch 'martin-datacite-strptime-none' into 'master'	Martin Czygan	2020-01-31	1	-1/+1
\|\ \ \| \| \| \| \| \| \| \| \| \| \| \|	datacite: prevent none See merge request webgroup/fatcat!25
\| * \|	datacite: prevent none	Martin Czygan	2020-01-31	1	-1/+1
\|/ /
* \|	Merge branch 'martin-datacite-micropub' into 'master'	Martin Czygan	2020-01-31	4	-25/+162
\|\ \ \| \|/ \|/\| \| \| \| \|	datacite: improve date handling and minor tweak See merge request webgroup/fatcat!24
\| *	datacite: name shall not be None	Martin Czygan	2020-01-31	1	-1/+1
\| \|
\| *	datacite: add exception for https://www.micropublication.org/	Martin Czygan	2020-01-31	2	-1/+7
\| \|
\| *	datacite: do not skip records w/o date	Martin Czygan	2020-01-31	1	-2/+1
\| \|
\| *	datacite: improve docstring	Martin Czygan	2020-01-31	1	-4/+4
\| \|
\| *	datacite: improve date handling and minor tweak	Martin Czygan	2020-01-30	4	-21/+153
\|/ \| \| \| \| \| \| \| \| \| \| \| \|	Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too.