aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* apply first round of feedback on matchingHEADmasterMartin Czygan2021-12-2113-9/+73
|
* matching: track_total_hits, use FalseMartin Czygan2021-12-161-4/+4
| | | | integer, despite supported according to the docs, yielded a 400 parse-error
* matching: we do not need exact match countsMartin Czygan2021-12-161-4/+4
| | | | up to 100 or even will be ok; see also: https://www.elastic.co/guide/en/elasticsearch/reference/7.16/search-your-data.html#track-total-hits
* matching: add hdl, remove mag idMartin Czygan2021-12-161-1/+1
|
* matching: cleanup and documentationMartin Czygan2021-12-071-47/+29
|
* matching: update docsMartin Czygan2021-12-071-9/+8
|
* v0.1.23Martin Czygan2021-12-061-1/+1
|
* matching: cleanup test filesMartin Czygan2021-12-0624-202/+1
|
* complete FuzzyReleaseMatcher refactoringMartin Czygan2021-12-0614-362/+644
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We keep the name, since the api - "matcher.match(release)" - is the same; simplified queries; at most one query is performed against elasticsearch; parallel release retrieval from the API; optional support for release year windows; Test cases are expressed in yaml and will be auto-loaded from the specified directory; test work against the current search endpoint, which means the actual output may change on index updates; for the moment, we think this setup is relatively simple and not too unstable. about: title contrib, partial name input: > { "contribs": [ { "raw_name": "Adams" } ], "title": "digital libraries", "ext_ids": {} } release_year_padding: 1 expected: - 7rmvqtrb2jdyhcxxodihzzcugy - a2u6ougtsjcbvczou6sazsulcm - dy45vilej5diros6zmax46nm4e - exuwhhayird4fdjmmsiqpponlq - gqrj7jikezgcfpjfazhpf4e7c4 - mkmqt3453relbpuyktnmsg6hjq - t2g5sl3dgzchtnq7dynxyzje44 - t4tvenhrvzamraxrvvxivxmvga - wd3oeoi3bffknfbg2ymleqc4ja - y63a6dhrfnb7bltlxfynydbojy
* complete migration from away from match_release_fuzzyMartin Czygan2021-11-164-247/+7
| | | | | Instead, use `FuzzyReleaseMatcher.match`, which has approximately the same behavior.
* update todoMartin Czygan2021-11-161-1/+2
|
* Merge branch 'martin-matcher-class' into 'master'Martin Czygan2021-11-1624-112/+1410
|\ | | | | | | | | turn "match_release_fuzzy" into a class See merge request webgroup/fuzzycat!10
| * use elasticsearch <7.14 search argsMartin Czygan2021-11-161-11/+47
| |
| * setup: add missing pyyaml dependencyMartin Czygan2021-11-161-0/+1
| |
| * setup: add thefuzz dependencyMartin Czygan2021-11-161-1/+2
| |
| * turn "match_release_fuzzy" into a classMartin Czygan2021-11-1623-111/+1371
|/ | | | | | | | Goal of this refactoring was to make the matching process a bit more configurable by using a class and a cascade of queries. For a limited test set: `FuzzyReleaseMatcher.match` is works the same as `match_release_fuzzy`.
* Merge branch 'bnewbold-grobid-tei-xml' into 'master'Martin Czygan2021-11-044-277/+56
|\ | | | | | | | | use grobid_tei_xml for grobid unstructured lookups See merge request webgroup/fuzzycat!9
| * use grobid_tei_xml for grobid unstructured lookupsBryan Newbold2021-10-284-277/+56
|/
* Merge branch 'bnewbold-tweaks' into 'master'Martin Czygan2021-10-283-3/+5
|\ | | | | | | | | tweaks to deps and packaging; add files,contribs in live match release lookups See merge request webgroup/fuzzycat!8
| * bump fatcat-openapi-client version to 0.4.0Bryan Newbold2021-10-271-1/+1
| | | | | | | | | | | | There isn't any new feature required in the new version of the client library, but feels like we should aggressively update everywhere when possible.
| * matching: include contribs,files in release entityBryan Newbold2021-10-271-1/+1
| | | | | | | | | | | | | | | | | | | | This makes several downstream applications simpler, like showing PDF links without an additional fatcat API fetch. The 'contrib' entities may be required as part of bibliographic matching (checking the creator names as well as the release-local versions of the name) In theory we could add webcaptures,filesets as well, but those are still rare, and occasionally result in very large sub-documents.
| * packaging: include py.typed for mypy to detectBryan Newbold2021-10-272-0/+1
| |
| * deps: pin elasticsearch to less than 7.14Bryan Newbold2021-10-271-1/+2
|/ | | | | This is to avoid 'elasticsearch.exceptions.UnsupportedProductError' errors in newer versions of the elasticsearch client libraries.
* start larger refactoring: remove clusterMartin Czygan2021-09-249-723/+188
| | | | | | | | | | | | | | | | | | background: verifying hundreds of millions of documents turned out to be a bit slow; anecdata: running clustering and verification over 1.8B inputs tooks over 50h; cf. the Go port (skate) required about 2-4h for those operations. Also: with Go we do not need the extra GNU parallel wrapping. In any case, we aim for fuzzycat refactoring to provide: * better, more configurable verification and small scale matching * removal of batch clustering code (and improve refcat docs) * a place for a bit more generic, similarity based utils The most important piece in fuzzycat is a CSV file containing hand picked test examples for verification - and the code that is able to fulfill that test suite. We want to make this part more robust.
* setup: narrow dependency versionsMartin Czygan2021-09-211-4/+4
|
* Merge branch 'wip-martin-review-cleanup' into 'master'Martin Czygan2021-09-2113-20/+272
|\ | | | | | | | | review notes and some cleanup See merge request webgroup/fuzzycat!7
| * tests: temporarily disable testsMartin Czygan2021-09-211-12/+12
| | | | | | | | | | We want to first move to elasticsearch dsl and will reactivate and extends after refactoring.
| * matching: run an additional es query for fuzzy matchingMartin Czygan2021-09-212-3/+93
| |
| * reorganize notesMartin Czygan2021-09-216-2/+153
| |
| * style: apply formattingMartin Czygan2021-09-217-11/+22
|/
* matching: actually return the specified number of resultsMartin Czygan2021-09-151-2/+2
|
* add todoMartin Czygan2021-09-141-0/+28
|
* remove pipenv related filesMartin Czygan2021-09-135-979/+25
| | | | | | | | | fuzzycat is mostly a library; the command line tool will switch to a bundled executable (e.g. via shiv) soon; removed pipenv in order to lower confusion which setup to use; also pipenv unfortunately at time cat take a bit of time to complete operations
* v0.1.22Martin Czygan2021-09-131-1/+1
|
* cluster: adjust tests to jellyfish nysiis implementationMartin Czygan2021-09-131-7/+7
|
* update READMEMartin Czygan2021-09-131-4/+7
|
* remove dependency on fuzzy; use jellyfishMartin Czygan2021-09-134-304/+286
|
* cleanup makefileMartin Czygan2021-09-131-2/+0
|
* update mentions of cgraph to refcatBryan Newbold2021-09-102-2/+2
|
* Merge branch 'master' of git.archive.org:webgroup/fuzzycatMartin Czygan2021-07-098-224/+318
|\ | | | | | | | | | | | | | | * 'master' of git.archive.org:webgroup/fuzzycat: simplify README for general audience; move some content to notes sandcrawler slugify: lower-case greek ambiguity (OCR) DOI clean/normalize helper; and use in verification etc verify: page count parsing and comparison improvements
| * Merge branch 'bnewbold-readme' into 'master'Martin Czygan2021-07-072-210/+245
| |\ | | | | | | | | | | | | simplify README for general audience; move some content to notes See merge request webgroup/fuzzycat!6
| | * simplify README for general audience; move some content to notesBryan Newbold2021-07-012-210/+245
| | |
| * | Merge branch 'bnewbold-verify-improvements' into 'master'Martin Czygan2021-07-026-14/+73
| |\ \ | | |/ | |/| | | | | | | verify improvements See merge request webgroup/fuzzycat!4
| | * sandcrawler slugify: lower-case greek ambiguity (OCR)Bryan Newbold2021-07-011-2/+13
| | |
| | * DOI clean/normalize helper; and use in verification etcBryan Newbold2021-07-015-6/+35
| | |
| | * verify: page count parsing and comparison improvementsBryan Newbold2021-07-013-6/+25
| |/
* | add a few (open) tests casesMartin Czygan2021-07-096-0/+176
| |
* | notes on matching metricsMartin Czygan2021-07-081-0/+16
| |
* | cleanup notesMartin Czygan2021-07-082-13/+0
|/
* add test caseMartin Czygan2021-06-214-0/+1339
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-21 08:48:49 +0000