| Commit message (Collapse) | Author | Age | Files | Lines | ||
|---|---|---|---|---|---|---|
| ... | ||||||
| * | | gitlab CI: add libmagic1 dep (was a recommends) | Bryan Newbold | 2021-01-05 | 1 | -1/+1 | |
| | | | ||||||
| * | | gitlab CI: add build deps | Bryan Newbold | 2021-01-05 | 1 | -1/+1 | |
| | | | ||||||
| * | | gitlab CI: install fewer packages (no recommends) | Bryan Newbold | 2021-01-05 | 1 | -2/+2 | |
| | | | ||||||
| * | | python makefile: don't duplicate 'lint' commands during 'test' | Bryan Newbold | 2021-01-05 | 1 | -2/+0 | |
| | | | ||||||
| * | | update to python3.8 | Bryan Newbold | 2021-01-05 | 4 | -403/+417 | |
| | | | ||||||
| * | | pdf: yet more bad SHA1 (commiting lines from prod) | Bryan Newbold | 2021-01-05 | 1 | -0/+20 | |
| | | | ||||||
| * | | ia CDX: handle bad CDX rows | Bryan Newbold | 2021-01-05 | 1 | -2/+4 | |
| | | | ||||||
| * | | late-2020 OA DOI crawl ingest notes | Bryan Newbold | 2021-01-04 | 1 | -3/+46 | |
| | | | ||||||
| * | | DOAJ crawl ingest stats | Bryan Newbold | 2020-12-31 | 1 | -0/+295 | |
| | | | ||||||
| * | | progress notes on OA DOI ingest (still running) | Bryan Newbold | 2020-12-28 | 1 | -11/+102 | |
| | | | ||||||
| * | | update HTML ingest proposal | Bryan Newbold | 2020-12-23 | 1 | -1/+3 | |
| | | | ||||||
| * | | spn: more status codes | Bryan Newbold | 2020-12-21 | 1 | -1/+2 | |
| | | | ||||||
| * | | HTML ingest deployment notes | Bryan Newbold | 2020-12-16 | 1 | -1/+71 | |
| | | | ||||||
| * | | persist: html_meta is ON CONFLICT DO UPDATE | Bryan Newbold | 2020-12-15 | 1 | -1/+1 | |
| | | | ||||||
| * | | persist: don't expect HTML TEI-XML in result object | Bryan Newbold | 2020-12-15 | 1 | -1/+1 | |
| | | | ||||||
| * | | unpaywall crawl/ingest update (from Oct 2020) | Bryan Newbold | 2020-12-08 | 1 | -0/+134 | |
| | | | ||||||
| * | | commit sept 2020 scielo ingest notes | Bryan Newbold | 2020-12-08 | 1 | -0/+21 | |
| | | | ||||||
| * | | handle more wayback error conditions | Bryan Newbold | 2020-11-20 | 1 | -0/+6 | |
| | | | ||||||
| * | | kafka docs for rolling back a consumer group | Bryan Newbold | 2020-11-20 | 1 | -0/+9 | |
| | | | ||||||
| * | | html: more conservative parsing of element attr | Bryan Newbold | 2020-11-20 | 1 | -2/+4 | |
| | | | ||||||
| * | | xml: catch parse error | Bryan Newbold | 2020-11-19 | 1 | -3/+8 | |
| | | | ||||||
| * | | SQL: more ingest monitoring | Bryan Newbold | 2020-11-16 | 3 | -1/+660 | |
| | | | ||||||
| * | | spn 'forbidden' status code | Bryan Newbold | 2020-11-12 | 1 | -1/+1 | |
| | | | ||||||
| * | | html biblio: handle 'content not in attrs' case | Bryan Newbold | 2020-11-12 | 1 | -2/+2 | |
| | | | ||||||
| * | | add implementation notes about HTML ingest | Bryan Newbold | 2020-11-10 | 1 | -0/+248 | |
| | | | ||||||
| * | | fuzzy matching notes | Bryan Newbold | 2020-11-10 | 1 | -0/+148 | |
| | | | ||||||
| * | | blacklist -> denylist | Bryan Newbold | 2020-11-10 | 2 | -9/+9 | |
| | | | ||||||
| * | | pipenv: updates (mostly for trafilatura 0.6.0) | Bryan Newbold | 2020-11-10 | 1 | -25/+32 | |
| | | | ||||||
| * | | DOAJ and HTML ingest tweaks from QA run | Bryan Newbold | 2020-11-10 | 2 | -3/+3 | |
| | | | ||||||
| * | | html: handle more traf error cases | Bryan Newbold | 2020-11-08 | 1 | -2/+2 | |
| | | | ||||||
| * | | html: more adblock | Bryan Newbold | 2020-11-08 | 1 | -1/+3 | |
| | | | ||||||
| * | | ingest: small html_bibli typo | Bryan Newbold | 2020-11-08 | 1 | -1/+1 | |
| | | | ||||||
| * | | html: most small platform tweaks | Bryan Newbold | 2020-11-08 | 1 | -5/+4 | |
| | | | ||||||
| * | | move fuzzy URL match method to misc | Bryan Newbold | 2020-11-08 | 3 | -19/+20 | |
| | | | ||||||
| * | | move some PDF URL extraction into declarative format | Bryan Newbold | 2020-11-08 | 5 | -143/+177 | |
| | | | ||||||
| * | | basic DOAJ ingest request conversion script | Bryan Newbold | 2020-11-08 | 1 | -0/+139 | |
| | | | ||||||
| * | | ingest: default to html_biblio for PDF URL extraction | Bryan Newbold | 2020-11-08 | 1 | -24/+17 | |
| | | | ||||||
| * | | ingest: shorted scope+platform keys; use html_biblio extraction for PDFs | Bryan Newbold | 2020-11-08 | 1 | -15/+35 | |
| | | | ||||||
| * | | html: more robust ingest; better platform and scope detection | Bryan Newbold | 2020-11-08 | 1 | -32/+96 | |
| | | | ||||||
| * | | html: more extraction patterns; bugfix; skip more crossmark | Bryan Newbold | 2020-11-08 | 1 | -1/+24 | |
| | | | ||||||
| * | | ingest html: return better status based on sniffed scope | Bryan Newbold | 2020-11-08 | 1 | -9/+31 | |
| | | | ||||||
| * | | ingest tool: more ingest control args | Bryan Newbold | 2020-11-08 | 1 | -1/+10 | |
| | | | ||||||
| * | | spn2-internal-server-error is a problem with remote server, not SPN2 | Bryan Newbold | 2020-11-08 | 1 | -0/+2 | |
| | | | ||||||
| * | | ingest: better non-full URL fixup | Bryan Newbold | 2020-11-08 | 1 | -4/+3 | |
| | | | ||||||
| * | | html: small ingest improvements | Bryan Newbold | 2020-11-08 | 2 | -0/+19 | |
| | | | ||||||
| * | | html: start improving scope detection | Bryan Newbold | 2020-11-08 | 2 | -5/+49 | |
| | | | ||||||
| * | | ingest: retain html_biblio through hops; all ingest types | Bryan Newbold | 2020-11-08 | 1 | -1/+13 | |
| | | | ||||||
| * | | ingest tool: flag for HTML quick mode (CDX-only) | Bryan Newbold | 2020-11-08 | 2 | -1/+6 | |
| | | | ||||||
| * | | html: try to detect and mark XHTML (vs. HTML or XML) | Bryan Newbold | 2020-11-08 | 2 | -4/+6 | |
| | | | ||||||
| * | | gen_file_metadata: allow empty/null bodies (if flag set) | Bryan Newbold | 2020-11-08 | 2 | -3/+5 | |
| | | | | | | | | | This is for HTML sub-resources, which can validly be empty (I think) | |||||
 |
index : sandcrawler | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |