aboutsummaryrefslogtreecommitdiffstats
path: root/python/tests/files
Commit message (Collapse)AuthorAgeFilesLines
* pubmed: handle multiple ReferenceListBryan Newbold2020-03-201-0/+206
| | | | | | | This resolves a situation noticed in prod where we were only importing/updating a single reference per article. Includes a regression test.
* Merge branch 'martin-kafka-bs4-import' into 'master'Martin Czygan2020-03-102-0/+0
|\ | | | | | | | | pubmed and arxiv harvest preparations See merge request webgroup/fatcat!28
| * more pubmed adjustmentsMartin Czygan2020-02-222-0/+0
| | | | | | | | | | * regenerate map in continuous mode * add tests
* | Merge branch 'bnewbold-elastic-v03b'Bryan Newbold2020-02-263-0/+3
|\ \
| * | fix some transform bugs, add some testsBryan Newbold2020-01-293-0/+3
| | |
* | | shadow import: more filtering of file_meta fieldsBryan Newbold2020-02-131-12/+10
| | |
* | | basic shadow importerBryan Newbold2020-02-131-0/+12
| |/ |/|
* | datacite: add exception for https://www.micropublication.org/Martin Czygan2020-01-311-1/+2
| |
* | datacite: improve date handling and minor tweakMartin Czygan2020-01-302-0/+110
|/ | | | | | | | | | | | | Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too.
* do not normalize "en dash" in DOIMartin Czygan2020-01-171-1/+1
| | | | | | | | | Technically, [...] DOI names may incorporate any printable characters from the Universal Character Set (UCS-2), of ISO/IEC 10646, which is the character set defined by Unicode (https://www.doi.org/doi_handbook/2_Numbering.html#2.5.1). For mostly QA reasons, we currently treat a DOI with an "en dash" as invalid.
* ingest: improve tests, support old ingest resultsBryan Newbold2020-01-152-1/+2
|
* datacite: ignore known unknown values in resourceType*Martin Czygan2020-01-092-0/+94
|
* datacite: abstracts may be strings or list of stringsMartin Czygan2020-01-094-0/+186
|
* datacite: improve license_slug handlingMartin Czygan2020-01-092-1/+3
|
* datacite: add 'Unknown' to blacklistMartin Czygan2020-01-091-7/+1
|
* datacite: get rid of schemaVersionMartin Czygan2020-01-0917-32/+14
|
* datacite: reformat test cases and use jq . --sort-keysMartin Czygan2020-01-0854-2299/+2301
|
* datacite: factor out contributor handlingMartin Czygan2020-01-084-0/+105
| | | | | | | Use values from: * attributes.creators[] * attributes.contributors[]
* datacite: adjust tests for release_monthMartin Czygan2020-01-0812-12/+12
|
* datacite: mark additional files as stubMartin Czygan2020-01-082-0/+72
|
* datacite: CCDC are entries, mostlyMartin Czygan2020-01-081-1/+1
|
* datacite: adding datacite-specific extra metadataMartin Czygan2020-01-0730-1468/+1570
| | | | | | | | | | | | | * attributes.metadataVersion * attributes.schemaVersion * attributes.version (source dependent values, follows suggestions in https://schema.datacite.org/meta/kernel-4.3/doc/DataCite-MetadataKernel_v4.3.pdf#page=26, but values vary) Furthermore: * attributes.types.resourceTypeGeneral * attributes.types.resourceType
* datacite: month field should be top-levelMartin Czygan2020-01-0611-14/+14
|
* datacite: include month in extraMartin Czygan2020-01-0611-11/+13
| | | | | > include release_month as a top-level extra field [...] to auto-populate the schema field from that
* datacite: clean abstracts, use unknown value tokensMartin Czygan2020-01-063-3/+3
| | | | | | | | Datacite defines placeholders for unknown values: * https://support.datacite.org/docs/schema-values-unknown-information-v43 Clean abstracts.
* datacite: always include "datacite" key in extraMartin Czygan2020-01-0414-26/+26
| | | | | | > always include extra values for the respective DOI registrars (datacite, crossref, jalc), even if they are empty ({}), to be used as a flag so we know which DOI registrar supplied the metadata.
* datacite: remove --lang-detect flagMartin Czygan2020-01-035-10/+15
| | | | Estimated time for a single call is in the order of 50ms.
* datacite: add another test caseMartin Czygan2020-01-022-0/+70
|
* datacite: open case for editing after creationMartin Czygan2020-01-021-0/+2
|
* datacite: add helper script to create new test caseMartin Czygan2020-01-021-0/+14
|
* datacite: address raw_name index form commentMartin Czygan2020-01-0219-111/+111
| | | | | | | | | > The convention for display_name and raw_name is to be how the name would normally be printed, not in index form (surname comma given_name). So we might need to un-encode names like "Tricart, Pierre". Use an additional `index_form_to_display_name` function to convert index from to display form, heuristically.
* datacite: add conversion fixturesMartin Czygan2020-01-0249-0/+3924
| | | | | | | | | | | | | The `test_datacite_conversions` function will compare an input (datacite) document to an expected output (release entity as JSON). This way, it should not be too hard to add more cases by adding: input, output - and by increasing the counter in the range loop within the test. To view input and result side by side with vim, change into the test directory and run: tests/files/datacite $ ./caseview.sh 18
* improve datacite field mapping and importMartin Czygan2019-12-282-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current version succeeded to import a random sample of 100000 records (0.5%) from datacite. The --debug (write JSON to stdout) and --insert-log-file (log batch before committing to db) flags are temporary added to help debugging. Add few unit tests. Some edge cases: a) Existing keys without value requires a slightly awkward: ``` titles = attributes.get('titles', []) or [] ``` b) There can be 0, 1, or more (first one wins) titles. c) Date handling is probably not ideal. Datacite has a potentiall fine grained list of dates. The test case (tests/files/datacite_sample.jsonl) refers to https://ssl.fao.org/glis/doi/10.18730/8DYM9, which has date (main descriptor) 1986. The datacite record contains: 2017 (publicationYear, probably the year of record creation with reference system), 1978-06-03 (collected, e.g. experimental sample), 1986 ("Accepted"). The online version of the resource knows even one more date (2019-06-05 10:14:43 by WIEWS update).
* datacite: add simple test and fixture for datacite api interactionMartin Czygan2019-12-271-0/+1
|
* add regression test for medlinedate -> year parsingBryan Newbold2019-12-231-0/+95
|
* add basic test for crossref harvest API callBryan Newbold2019-12-061-0/+1
|
* ingest file result importerBryan Newbold2019-11-151-0/+1
|
* release elasticsearch results: stage not statusBryan Newbold2019-06-131-1/+1
|
* JALC bulk file importerBryan Newbold2019-05-211-0/+100
|
* basic JALC XML DOI metadata parserBryan Newbold2019-05-211-0/+176
|
* basic JSTOR XML parserBryan Newbold2019-05-211-0/+58
|
* basic arxivraw XML parserBryan Newbold2019-05-211-0/+31
|
* basic pubmed parserBryan Newbold2019-05-211-0/+36822
|
* fix releases/release_ids in math_universe.json test fileBryan Newbold2019-05-201-1/+1
|
* importer code updatesBryan Newbold2019-05-131-1/+1
|
* update example release JSON to new schema (ext_id, release_stage)Bryan Newbold2019-05-132-11/+11
|
* arabesque import testsBryan Newbold2019-04-182-0/+10
|
* many web test improvementsBryan Newbold2019-04-042-0/+2
|
* more integration of transform refactorBryan Newbold2019-03-111-0/+10
|
* crossref import tweaks/fixesBryan Newbold2019-01-291-0/+1
| | | | | - refs: article-title not title; save unstructured; authors not author - save 'language' field (already an ISO code)
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-09 17:45:47 +0000