[no description] https://git.bnewbold.net/fatcat/atom?h=v0.3.2 2020-04-01T22:03:19+00:00 2020-04-01T22:03:19+00:00 bnewbold bnewbold@archive.org 2020-04-01T22:03:19+00:00 urn:sha1:32f195cec41459045f3d3453dad7a97b38d4e288 beautifulsoup XML parsing: .string vs. .get_text() See merge request webgroup/fatcat!40 2020-04-01T19:02:45+00:00 Bryan Newbold bnewbold@robocracy.org 2020-04-01T19:02:43+00:00 urn:sha1:938d2c5366d80618b839c83baadc9b5c62d10dce The primary motivation for this change is that fatcat *requires* a non-empty title for each release entity. Pubmed/Medline occasionally indexes just a VenacularTitle with no ArticleTitle for foreign publications, and currently those records don't end up in fatcat at all. 2020-04-01T19:02:20+00:00 Bryan Newbold bnewbold@robocracy.org 2020-04-01T19:02:20+00:00 urn:sha1:f77a553350238c8ccc9c3bc0edcf47fb9dd067b3 2020-03-31T04:23:11+00:00 Bryan Newbold bnewbold@robocracy.org 2020-03-31T03:56:04+00:00 urn:sha1:851c40143d44a73a92ff2c9556b3a63f29668c2d This goes against what the API docs recommend, but we are currently far behind on updates and need to catch up. Other than what the docs say, this seems to be consistent with the behavior we want. 2020-03-31T03:55:44+00:00 Bryan Newbold bnewbold@robocracy.org 2020-03-31T03:55:44+00:00 urn:sha1:98933a068ec3d918deb0e7dff30aed517ca515d9 2020-03-29T03:12:58+00:00 Bryan Newbold bnewbold@robocracy.org 2020-03-29T03:12:54+00:00 urn:sha1:6681500eeffe39b7d029a0e0d6b2ed83729f555f See previous pubmed commit for details. 2020-03-29T03:01:48+00:00 Bryan Newbold bnewbold@robocracy.org 2020-03-29T03:01:46+00:00 urn:sha1:d6af7b7544ddb3b5e7b1f4a0fd76bd9cd5ed9125 Yikes! Apparently when a tag has child tags, .string will return None instead of all the strings. .get_text() returns all of it: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text https://www.crummy.com/software/BeautifulSoup/bs4/doc/#string I've things like identifiers as .string, when we expect only a single string inside. 2020-03-29T02:57:41+00:00 Bryan Newbold bnewbold@robocracy.org 2020-03-29T02:57:35+00:00 urn:sha1:4b75a81cbd0faeefa6a0f04b97ecc6832924ee69 These are journal/publisher patterns which we suspect to actually be OA based on the large quantity of papers that crawl successfully. The better long-term solution will be to flag containers in some way as OA (or "should crawl"), but this is a good short-term solution. 2020-03-27T23:50:08+00:00 bnewbold bnewbold@archive.org 2020-03-27T23:50:08+00:00 urn:sha1:98abe2e751187aa7c2e751b355ffb56d9b1f8c6a Correct spelling mistakes 2020-03-27T21:25:54+00:00 Edward Betts edward@4angle.com 2020-03-27T21:25:54+00:00 urn:sha1:94710b2803780ab16fb30b79010f8e27cf115512