From 97a3a8ea4122ed97d29176c570389e1db6a91b44 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sat, 18 May 2019 16:23:07 -0700 Subject: move finished posts into sub-directories --- posts/2016/elm-everything-broken.md | 427 ++++++++++++++++++++++++++++++++++++ posts/2016/juliacon.md | 339 ++++++++++++++++++++++++++++ posts/2016/new-server-2016.md | 81 +++++++ posts/elm-everything-broken.md | 427 ------------------------------------ posts/juliacon.md | 339 ---------------------------- posts/new-server-2016.md | 81 ------- 6 files changed, 847 insertions(+), 847 deletions(-) create mode 100644 posts/2016/elm-everything-broken.md create mode 100644 posts/2016/juliacon.md create mode 100644 posts/2016/new-server-2016.md delete mode 100644 posts/elm-everything-broken.md delete mode 100644 posts/juliacon.md delete mode 100644 posts/new-server-2016.md diff --git a/posts/2016/elm-everything-broken.md b/posts/2016/elm-everything-broken.md new file mode 100644 index 0000000..a9978d1 --- /dev/null +++ b/posts/2016/elm-everything-broken.md @@ -0,0 +1,427 @@ +Title: Everything Is Broken (Installing Elm Edition) +Author: bnewbold +Date: 2016-05-06 + +*This post is a narrative rant (in the same vein of Dan Luu's ["Everything is +Broken"][danluu] post) about my problems one afternoon getting a Fancy New +Programming Language to work on my laptop. [Skip down](#nowwhat) for my +thoughts on solutions.* + +[danluu]: https://danluu.com/everything-is-broken/ + +
+ +
+ +photo of E.C. giving an elm talk + +
+ +A few folks here at the [Recurse Center][rc] had nice things to say about the +[Elm programming language][rc], and after watching Evan Czaplicki's ["Let's Be +Mainstream" talk][0] I got pretty excited also. "Let's try Elm!" and the +adventure begins. + +[rc]: https://recurse.com/ +[elm]: https://elm-lang.org +[0]: https://www.youtube.com/watch?v=oYk8CKH7OhE + +First I go in the front door with the [install page][install], which redirects +me to a page which instructs me to use npm. + +[install]: http://elm-lang.org/install + +Hrm, I've had painful experiences with npm in the past, and anyways, isn't +the Elm compiler implemented in Haskell? Why are they using a package +management tool for Javascript libraries for installation? No thanks, I'll +[build from source][build], following these directions from the `README.md`: + +[build]: https://github.com/elm-lang/elm-platform + + # If you are on LINUX, you need to install a dependency of elm-repl. + # Uncomment the following line and run it. + # sudo apt-get install libtinfo-dev + + # if you are on windows, or some other place without curl, just download this file manually + curl https://raw.githubusercontent.com/elm-lang/elm-platform/master/installers/BuildFromSource.hs > BuildFromSource.hs + + runhaskell BuildFromSource.hs 0.16 + + + +Weird that these build-from-source directions involve curl-to-file magics. Why +not just clone the repo and use a Makefile, which is roughly the same number of +commands to enter? I clone repo and run the build script from there instead. +I'm on Debian stable ("jessie"), so installing GHC (the most popular way to use +Haskell) is easy. I seem to have a recent-ish version of cabal: + +
+bnewbold@eschaton$ cabal --version
+cabal-install version 1.20.0.3
+using version 1.20.0.2 of the Cabal library 
+
+ +... so I think i'm ready to go ahead run the `BuildFromSource.hs` script: + +
+bnewbold@eschaton$ runhaskell ~/src/elm-platform/installers/BuildFromSource.hs 0.16
+
+GHCi runtime linker: fatal error: I found a duplicate definition for symbol
+_hs_bytestring_long_long_uint_hex
+whilst processing object file
+/home/bnewbold/.cabal/lib/x86_64-linux-ghc-7.6.3/bytestring-0.10.6.0/HSbytestring-0.10.6.0.o
+This could be caused by:
+* Loading two different object files which export the same symbol
+* Specifying the same object file twice on the GHCi command line
+* An incorrect `package.conf' entry, causing some object to be
+    loaded twice.
+GHCi cannot safely continue in this situation.  Exiting now.  Sorry.
+
+ +Uh-oh! Not sure what that means. A search returns [a FAQ entry][faq-entry] +about "Duplicate Directions" which talks about linker flags, but *I* didn't +compile anything by hand so it doesn't seem like *I* have done anything wrong. +I'll try updating `cabal`'s package index: + +[faq-entry]: https://wiki.haskell.org/GHC/FAQ#Duplicate_Definitions + +
+bnewbold@eschaton$ cabal update
+Downloading the latest package list from hackage.haskell.org
+cabal: Failed to download
+http://hackage.haskell.org/packages/archive/00-index.tar.gz : ErrorMisc "Error
+HTTP code: 502"
+
+ +No go! At first I assumed I was doing something wrong, or maybe the WiFi was +flakey, but it turns out that the Hackage package repository [is +down](https://status.haskell.org/): + +> hackage.haskell.org down +> +> Full Service Disruption +> +> [Investigating] The hackage server is in an out of memory condition. We're investigating. + +
+As far as I know, the only Debian packages out of the huge stable archive which +don't receive security updates are web browsers, mediawiki, and all V8/node.js +packages and libraries. +
+ +Well, it wasn't even clear if updating my package index would fix the problem, +it was just a wild guess. I'll give up and try npm, which these days has been +[sort of packaged][deb-node] in Debian. I'll just uninstall npm when I'm done +installing Elm. I install with `apt` (which pulls in a huge list of `nodejs` +packages), but then can finally run: + +[deb-node]: https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#libv8 + +
+bnewbold@eschaton$ npm install elm
+|
+> elm@0.16.0 install /home/bnewbold/bin/node_modules/elm
+> node install.js
+
+sh: 1: node: not found
+npm WARN This failure might be due to the use of legacy binary "node"
+npm WARN For further explanations, please read
+/usr/share/doc/nodejs/README.Debian
+
+npm ERR! elm@0.16.0 install: `node install.js`
+npm ERR! Exit status 127
+npm ERR! 
+npm ERR! Failed at the elm@0.16.0 install script.
+npm ERR! This is most likely a problem with the elm package,
+npm ERR! not with npm itself.
+npm ERR! Tell the author that this fails on your system:
+npm ERR!     node install.js
+npm ERR! You can get their info via:
+npm ERR!     npm owner ls elm
+npm ERR! There is likely additional logging output above.
+
+[...]
+
+
+ +
+Later I learned that I could have instead installed the nodejs-legacy +package, which just installs a symlink /usr/bin/node which points to +/usr/bin/nodejs. +
+ +Looks like Debian wants to refer to node.js as "`nodejs`" while npm/Elm expects +it to be "`node`". Ok, I create this shim named `node` in my `~/bin` directory +(which is on my `$PATH`), and make it executable: + +
+#!/bin/sh
+nodejs $*
+
+ +Now I can do things like: + +
+bnewbold@eschaton$ node --version
+v0.10.29
+
+ +Great! Now let's install Elm with npm! + +
+bnewbold@eschaton$ npm install elm
+
+> elm@0.16.0 install /home/bnewbold/bin/node_modules/elm
+> node install.js
+
+Downloading Elm Reactor assets from https://dl.bintray.com/elmlang/elm-platform/0.16.0/elm-reactor-assets.tar.gz
+Error communicating with URL https://dl.bintray.com/elmlang/elm-platform/0.16.0/linux-x64.tar.gz Error: CERT_UNTRUSTED
+npm WARN This failure might be due to the use of legacy binary "node"
+npm WARN For further explanations, please read
+/usr/share/doc/nodejs/README.Debian
+
+npm ERR! elm@0.16.0 install: `node install.js`
+npm ERR! Exit status 1
+npm ERR! 
+npm ERR! Failed at the elm@0.16.0 install script.
+npm ERR! This is most likely a problem with the elm package,
+npm ERR! not with npm itself.
+npm ERR! Tell the author that this fails on your system:
+npm ERR!     node install.js
+npm ERR! You can get their info via:
+npm ERR!     npm owner ls elm
+npm ERR! There is likely additional logging output above.
+
+npm ERR! System Linux 3.16.0-4-amd64
+npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "elm"
+npm ERR! cwd /home/bnewbold/bin
+npm ERR! node -v v0.10.29
+npm ERR! npm -v 1.4.21
+npm ERR! code ELIFECYCLE
+npm ERR! 
+npm ERR! Additional logging details can be found in:
+npm ERR!     /home/bnewbold/bin/npm-debug.log
+npm ERR! not ok code 0
+
+ +Whoops. There is [a github issue][elm-issue] that has some tips about this: the +recommended solutions online are basically "upgrade node.js and npm" or +"disable SSL security". That doesn't sound great, but i'm feeling very +impatient at this point so I tried disabling SSL checks with `npm config set ca +null`, but even then I get the certificate error. Enough of that approach: + +[elm-issue]: https://github.com/elm-lang/elm-platform/issues/100 + +
+sudo apt remove npm nodejs
+rm ~/bin/node
+
+ +Poking around a bit more, I find the promisingly-titled "`fubar.sh`" script in +the `elm-platform` repo cloned earlier, which seems to nuke the local user's +Haskell (`cabal`) stuff (but doesn't muck with system-wide package, eg `apt` +managed GHC libraries). After running `fubar.sh` the `BuildFromSource.hs` +script seems to start working (success!), but Hackage is still down (now, an +hour after I started this process, approaching 5 hours total downtime), so I +can't install dependencies. + +I search for a hackage mirror and find that [FPComplete runs +one](https://www.fpcomplete.com/blog/2015/03/hackage-mirror). In addition to +their configuration instructions I had to set `remote-repo-cache` in my +`~/.cabal/config`, like so: + + remote-repo: hackage.fpcomplete.com:http://hackage.fpcomplete.com/ + remote-repo-cache: /home/bnewbold/.cabal/packages-fpcomplete + +After that I `cabal update` and run the `BuildFromSource.hs` script again. It +fails to build the dependency `websockets-snap`, but I try installing that +regularly (`cabal install websockets-snap`) and that works, and then all the +dependencies with BuildFromSource.hs build. Yay! + +I still get an error with compiling Elm itself though: + + src/Elm/Package.hs:60:25: + Not in scope: `<$>' + Perhaps you meant `' (imported from System.FilePath) + +It [sounds like](https://github.com/elm-lang/elm-platform/issues/30) I am using +an out of date version of GHC (the Haskell compiler): I have 7.6.3 (the version +that comes with Debian), and Elm wants 7.10. Fair enough: unlike a package +manager, I can see how new features in the compiler would be helpful, and when +I go back and check this was outlined in the README. Fortunately there is a new +version of GHC in the `jessie-backports` repo, so: + +
+sudo apt install -t jessie-backports ghc
+
+ +And now: + +
+bnewbold@eschaton$ cabal --version
+cabal-install version 1.22.6.0
+using version 1.22.5.0 of the Cabal library 
+bnewbold@eschaton$ ghc --version
+The Glorious Glasgow Haskell Compilation System, version 7.10.3
+
+ +After that, I'm miraculously able to compile. Horray! I add +`~/bin/Elm-Platform/0.16/.cabal-sandbox/bin` to my `$PATH` and I can run: + +
+bnewbold@eschaton$ elm-repl 
+---- elm repl 0.16.0 -----------------------------------------------------------
+:help for help, :exit to exit, more at 
+--------------------------------------------------------------------------------
+> 
+
+ +Success! + +To be clear, once I got this far I had few or no problems with Elm. The +language seems reasonably clean and well documented for it's maturity, and it +was only this install procress that felt horribly broken. + +Throughout this debugging process I had problems with both my external [Lenovo +Thinkpad USB keyboard][keyboard] glitching (workaround: unplug and re-plug) and +with WiFi at the Recurse Center disconnecting (workaround: reset wifi card). +Others here using GNU/Linux have the same wifi problems, but we don't have a +fix yet... something to do with wireless regulatory zones. + +It's not all pain though; installing the `elm.vim` syntax highlighting plugin +was very easy, just a single `git clone` into `~/.vim/bundle/`. + +[keyboard]: https://www.amazon.com/Lenovo-ThinkPad-Compact-Keyboard-TrackPoint/dp/B00F3U4TQS + + + +So Now What? +---------------- + +There was a great long-form blog post by Sam Boyer a few months ago ("[So You +Want To Write a Package Manager][5]") which describes the jumble of *system*, +*language*, and *project* dependency manager problems I experienced above. Sam +basically concludes that Rust's project/language dependency manager (Cargo) +both encapsulates best practices for dependable and reproducible builds, while +still allowing rapid evolution of a package "ecosystem". Cargo was designed by +[Yehuda Katz and Carl Lerche][6], and along with the +[crates.io](https://crates.io) archive it does sound very nice. My personal +feeling are usually that system-wide package managers (like Debian's `apt`) are +underappreciated by many young-but-not-bleeding-edge projects, but acknowledge +that there probably is also a need for higher tempo cross-platform project +dependency mangement for non-library projects (eg, desktop applications and web +apps). + +Ironically (given the difficulty I had installing it), the Elm language's +package manager has a great `diff` tool for checking that any changes in the +API conform to the [documented][8] [semantic versioning][semver] conventions. +For [example](https://gist.github.com/badboy/a302dd0c9020e5759240): + + $ elm-package diff evancz/elm-html 3.0.0 4.0.2 + Comparing evancz/elm-html 3.0.0 to 4.0.2... + This is a MAJOR change. + + ------ Changes to module Html.Attributes - MAJOR ------ + + Removed: + boolProperty : String -> Bool -> Attribute + stringProperty : String -> String -> Attribute + + + ------ Changes to module Html.Events - MINOR ------ + + Added: + type alias Options = + { stopPropagation : Bool, preventDefault : Bool } + defaultOptions : Html.Events.Options + onWithOptions : String -> Html.Events.Options -> Json.Decode.Decoder a -> (a -> Signal.Message) -> Html.Attribute + +This API change information is then used to *[programatically enforce][7]* the +semantic versioning rules for submissions to the Elm language library archive +and prevent a whole class of simple but annoying breakages due to unexpected +API changes. It can't detect *every* breaking change (eg, those which are +internal), but it can detect enough to be worth the effort. + +Another option for detecting fixing breakage is integration testing at the +package ecosystem level. Debian's [reproducible builds][repro] effort has built +out one such system, and continues to catch thousands of "failed to build from +source" (FTBFS) bugs along the way. Dan Luu's ["Everything is Broken"][danluu] +blog post (which to some degree inspired this one) mostly focused on the lack +of quality tests for many contemporary software projects, and proposed the use +of smart fuzzing and heuristically-generated tests to work around the huge +technical debt this represents. As far as I can tell there are hardly ever +automated integration tests for entire package archives; blame for dependency +problems is usually attributed to a bug in one package (instead of being seen +as a systemic failure), and such problems are seen as an unavoidable cost of +rapid and distributed development. + +I think we can do better. For example, some contemporary languages test +compiler changes for regressions against a broad snapshot of public code +written in that language; I know Go and Rust specifically do this to identify +problems before compiler releases, and the Python community occasionally runs +tests against the PyPi archive when considering syntax changes or feature +deprecations. And Debian's unstable and testing archives enforce a waiting +period so that human testers can turn up conflicts before packages can be moved +into stable or backports. I think we can and should automate these processes as +much as possible, and give direct feedback to library developers and package +maintainers when they push updates. Commercial Continuous Integration services +like Travis CI and Circle CI should offer more common target platforms (eg, ARM +architecture, more versions of distributions), and if they don't we should +build and host our own testing infrastructure. CI builds scripts should closely +match the official installation instructions for a given platform, so we catch +problems with those instructions quickly. + +We have more and more developers in the world wasting more and more days +wrangling with dependency hell, but despite that I think we're closer than ever +to taming the beast. + +[5]: https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527 +[6]: https://mail.mozilla.org/pipermail/rust-dev/2014-March/009090.html +[7]: https://github.com/elm-lang/elm-package/blob/master/README.md#version-rules +[8]: https://github.com/elm-lang/elm-package/blob/master/README.md#publishing-updates +[semver]: http://semver.org/ +[appcheck]: http://www.linuxfoundation.org/collaborate/workgroups/lsb/linux-application-checker-getting-started +[repro]: https://tests.reproducible-builds.org/debian/reproducible.html + +
+ +#### Follow up June 18th, 2016 + +A few days after writing this post, a new version of Elm (0.17) was +[released](http://elm-lang.org/blog/farewell-to-frp). The new version made some +big changes, like abandoning the [Functional Reactive Programming][frp] +paradigm. I found that many of the new tutorials weren't working with my 0.16 +install, so I tried to upgrade. After an hour or two of following the +directions above, I gave up and installed npm on a temporary Debian unstable +("sid") virtual machine. Once I added a `node` symlink to `nodejs`, I was able +to install `elm` with npm and develop remotely using `screen` and +`elm-react`. Frustrating! + +[frp]: https://en.wikipedia.org/wiki/Functional_reactive_programming + +
+
+ +
+Also, status.haskell.org seems to be broken again today, though +the hackage website works. +
+ +On the plus side, my USB keyboard problem seems to have been due to the cheap +random USB cable I was using. Since replacing it I haven't had the bad +flakey-ness, though now after my laptop awakes from sleep the pointer and +keyboard work but the middle mouse button ("paste" on UNIX) does not. +Workaround: plug and unplug the whole keyboard. + diff --git a/posts/2016/juliacon.md b/posts/2016/juliacon.md new file mode 100644 index 0000000..52751a9 --- /dev/null +++ b/posts/2016/juliacon.md @@ -0,0 +1,339 @@ +Title: What I Learned At JuliaCon +Author: bnewbold +Date: 2016-07-12 +Tags: tech, recurse, julia + +*Note: It looks like videos of the JuliaCon talks were uploaded [to +Youtube][youtube] the day this post was finally published!* + +[youtube]: https://www.youtube.com/playlist?list=PLP8iPy9hna6SQPwZUDtAM59-wPzCPyD_S + +I was in Cambridge, MA for a few days the other week at [JuliaCon][], a small +conference for the Julia programming language. Julia is a young language +(started around 2014 and currently pre-1.0) oriented towards fast numerical +computation: matrix manipulation, simulation, optimization, signal analysis, +etc. I've done a fair amount of such programming over the years, and it has +never felt as elegant or coherent as it could be. The available tools and +languages are generally either: + +[JuliaCon]: http://juliacon.org + +
+julia logo +
+ +1. stuck in the 1980s in terms of programming language features for safety, + productivity, and collaboration (eg, Fortran and Matlab) +1. expensive proprietary closed-source packages (eg, Matlab and Mathematica) +1. general-purpose languages with numerical features either hacked on or in the + form of libraries (eg, Python) + +There is a lot to be excited about in Julia. It's already pretty fast +(leveraging pre-existing JIT tools, hand-tuned matrix and solver libraries, and +the LLVM compiler suite) and has contemporary high-level language features +(like optional type annotation, polymorphic function dispatch, package +management tools, and general systems tools (eg, JSON and HTTP support)) that +can make the language more faster to develop in, and easier to read and +maintain. I'm personally excited about the progeny of the language: the +birthplace of the language is the CSAIL building at MIT, and the spirit of +[Scheme][sicm] and the work of [Project MAC][] is sprinkled through the +project. One of the [big pitches](graydon2) of Julia is that scientists won't +need to learn both a productive high-level language (eg, Python) and a +low-level performant language (eg, C or Fortran) and interface between the two: +Julia has everything all in one place. + +[graydon2]: http://graydon2.dreamwidth.org/3186.html +[sicm]: https://mitpress.mit.edu/sites/default/files/titles/content/sicm/book.html +[Project MAC]: http://groups.csail.mit.edu/mac/projects/mac/ + +All that being said, while I thought I would be working in Julia a lot during +my time at the Recurse Center, I've ended up being much more drawn to the +[Rust][] language instead. Rust is a general systems language (it's compiled, +has stronger typing, and no garbage collection), and not great for interactive +numerical exploration, but I've found it a joy to program in: for the most part +everything *just works* the way it says it will. My recent experience with +Julia, on the other hand, has been a lot of breakage between library and +interpreter versions, poor developer usability (eg, hard to figure out where +files should live in a package), and very frustrating import/load times. Though +I have to admit that I while I pushed through some frustrations with Rust, I +haven't spent *that* much time with Julia, and may have just been impatient, so +take everything I say here with a grain of salt. + +With these feels going in, what did I learn at JuliaCon and what do I think of +the future of the language now? In the below sections I'll go over the +interesting things I saw, then come back to summary at [the end](#summary). + +### Programming Language Design + +An older research language for numerical computing that I have always been +curious about is Fortress, and the leader of that project (Guy Steele, who also +worked on the design of the Scheme and Java languages) gave one of the opening +keynote speeches at JuliaCon this year. Awesome! I get really excited about +inter-generational learning and dialog. + +Fortress was a very "mathy" language. The number tower was intended to be +"correct" (aka, have the same structure that mathematicians use), physical +units were built-in, and some operator precedence was non-transitive. Operators +on built-in types (like Integers) could be overloaded, unlike in Java, because +Fortress users could apparently be trusted to "preserve algebraic properties". +Steele is a proponent of using whitespace (or lack of whitespace) to clarify +expressions, sort of like extra parentheses, and enforcing this in the +compiler. For example, the following two statements would be equivalent in most +languages, but not in Fortress: + +``` +a + b*c + d // Clear: Ok +a+b * c+d // Misleading: Compiler Error +``` + +This was part of a general effort to allow "whiteboard" style syntax in the +language. Fortress code actually has two representations: a plain text +Scala-style source code, and a LaTeX-y symbolic math format. Steele also used +some font-coloring in his slides to differentiate different types of symbols, +which reminded me of the helpful style my undergraduate physics professors +would use on the blackboard. I think this effort to adapt the "look and feel" +of the language to how the intended audience already writes and communicates is +really cool. I wonder if a third syntax format could have been added in a +one-to-one manner: that of a general purpose language like Scala or Haskell +(both noted as influences to Fortress) to make collaboration with general +purpose programming experts easier. Steele mentioned that some efforts to make +the syntax more math-like resulted in "contortions", so there is probably more +work to be done here. + +In my limited experience, Julia has a pretty clean syntax, and allows some +math-y [unicode characters as operators][unicode_ops] (like ∈, ≠, etc), but +didn't prioritize math-y syntax as much as Fortress. Given the open challenges +with formalizing informal whiteboard syntax this may or may not have been a +missed opportunity. + +[unicode_ops]: http://docs.julialang.org/en/release-0.4/manual/unicode-input/ + +The positive lessons learned from Fortress were summarized as being the type +system, automatic parallelism (via generators and reducers), the math-y syntax, +pretty printing (I assume meaning the LaTeX-y representation), physical units, +and forced syntax clarity (aka, forced use of parentheses and whitespace). One +issue that come up during implementation was that it was hard to bound the +latency and computational complexity of type constraint solving at run-time. + +A few other talks touched on language design decisions and features. There was +a short "Functional HPC" talk by Erik Schnetter, in which it was pointed out +that for some workloads regular old garbage collection can be faster than +reference counting: I've become used to thinking of latency and GC pauses as a +huge performance problem in systems programming, but for number crunching that +isn't as much of an issue, while little reference overheads are (especially if +locks or atomic operations are necessary). + +Keno Fischer gave an overview of the [Gallium][] debugger, which had some cool +features, but is still under development. There are both AST-based and +LLVM-based backends for the debugger, which allows stepping at function calls, +line-by-line, or expression-by-expression, which is something I hadn't seen +before. He demoed stepping through each step of the creation of a matplotlib +graph, with the output shown graphically after each step. Neat stuff! + +One of my personal interests in Julia would be formalizing the syntax into a +machine-readable grammar (eg, [EBNF][] or [ABNF][]). I was lucky enough to run +in to Stefan Karpinski during one of the coffee breaks, and he pointed me to +the Julia plugin for Eclipse, which already has a partial implementation of a +grammar. + +A few talks touched on the issue of Nullable datatypes (also called "Maybe" or +"Option" types in other languages), particularly for data science and +DataFrame-type applications. I only recently encountered [Option][] (and the +related [Result][] type) datatypes, in Rust, and can see why people want these +so badly, but there doesn't seem to be a simple path forward yet. Rust really +leverages these types in function return signatures, a feature which Julia does +not have for now; I think I read rumors about them being added in the future, +but didn't hear any mention of them here or on the 1.0 feature roadmap. + +[Option]: https://doc.rust-lang.org/std/option/index.html +[Result]: https://doc.rust-lang.org/core/result/index.html +[Gallium]: http://juliacon.org/abstracts.html#Gallium +[EBNF]: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_Form +[ABNF]: https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_Form + +### Numeric Abstraction + +One of the big trends I saw was taking advantage of Julia's abstractions around +generic operators and arrays to experiment with novel computation strategies. +Sometimes this means improving precision (with novel data types and +representations), sometimes it means increasing performance (by changing memory +layout or distribution, or targeting special hardware), and sometimes it just +makes code more elegant or semantic. + +For example, Tim Holy gave a talk (titled "To the Curious Incident of the CPU +in the run-time") which covered a bunch of nitty-gritty details for +implementing wrapper classes that re-shape or re-size Arrays, including sparse +arrays. + +Lindsey Kuper gave a nice overview of the [ParallelAccelerator.jl][pajl] +project, which entirely re-compiles Julia into C++ to get some extra performance +from the static full-program compiler. It seems to me that this only makes +sense because the Julia language has clean abstractions that the transpiler can +take advantage of. + +[pajl]: http://juliacon.org/abstracts.html#ParallelAccelerator + +One of my favorite talks from the whole conference was David Sanders' and Luis +Benet's talk on ValidatedNumerics ("Precise and rigorous calculations for +dynamical systems"). Instead of computing on approximate (rounded) scalars, +they compute on intervals of floating point numbers (or in higher dimensions, +boxes): at the end of computation the "correct" solution is known to be within +the final box, which also gives context as to how much numerical error has +accumulated. By defining new *types* to accomplish this (specifically, +DualNumbers), they can re-use any generic code in a relatively performant +manner. They also noted that when there is an analytic form to bound the error +for all following terms, Taylor expansion approximations can be truncated as +soon as the interval error exceeds the error in all following terms. Cool! + +### Other Fun Stuff + +**[Using Julia as a Quick and Dirty Code Generator][10]:** +The speaker (Arch Robison) is clearly having way too much fun! He used Julia to +output assembly code to get fast (real-time) discrete Fourier transform (DFT) +performance for a little video game called "FreqonInvaders". Infectious +enthusiasm! + +**[Autonomous driving for RC cars with ROS and Julia][11]:** +A fun little project doing "Model Predictive Control" on a small model car to +do stunts like drifting and slide parking into a tiny space. They achieved +about a 10Hz closed-loop control latency, which seems to me like barely enough +for this sort of thing, but clearly worked alright. Everything ran on the car +itself (no computation on a remote desktop with wireless control or anything +like that), with an Odroid ARM Linux system and an Arduino-compatible +microcontroller; Julia code using JuMP and other optimization stuff ran on the +ARM system. The code and raw data (for analysis) is available on the [BARC +project website](http://www.barc-project.com). Super cool, having this stuff +being experimented with already means there will be pressure to improve +soft-real-time performance in the language itself. + +**[Astrodynamics.jl: Modern Spaceflight Dynamics in Julia][12]:** +Mostly a bunch of code for doing timebase conversions and interpreting (or +calculating) ephemeris data (which is information about where astro bodies like +the Moon and planets will be at a given time), but some simple demos of orbital +simulation and event detection (eg, perihelion time and position) as well. Would +be cool if the ValidatedNumerics stuff was integrated. + +**[GLVisualize][13]:** +The demos in this talk were really impressive: live editing of mesh vertices, +relatively high performance, real-time feedback, etc. There were a bunch of +good graphics talks: the [GR Framework][14] stuff is really impressive in scope +(though maybe not as big a performance boost over Python as hoped), and +[Vulkan][15] is exciting. + +[10]: http://juliacon.org/abstracts.html#FrequonInvaders +[11]: http://juliacon.org/abstracts.html#RaceCars +[12]: http://juliacon.org/abstracts.html#Astrodynamics +[13]: http://juliacon.org/abstracts.html#GLVisualize +[14]: http://juliacon.org/abstracts.html#GR +[15]: http://juliacon.org/abstracts.html#Vulkan + +### Diversity + +It's sad to say, but the gender diversity at the conference was really poor, +particularly in contrast to the Recurse Center (where I have spent the past +couple months). The women I did meet gave some of the best talks, are crucial +contributors to infrastructure, and are generally amazing: more please! Aside +from the principle of the thing, there is just something about a giant sea of +guys at a tech event that results in a tense group vibe. Everybody I spoke to +one-on-one was friendly and we had great conversations, but as a group there +was a lot of ice to be broken. In my experience even hitting 10-20% women in +attendance can thaw this out, but that's just my anecdotal experience. + +I haven't attended, but I hear that PyCon has done a great job improving +diversity with careful planning and [systemic initiatives][pycon-diversity]. + +Overall, I thought the conference was a great group of people and admirably +well run. I appreciated the efforts to keep costs low, and everything generally +ran on time. Thanks to all the volunteer and MIT staff organizers for their +efforts! + +[pycon-diversity]: https://us.pycon.org/2016/about/diversity/ + +### Julia 1.0 + +Stefan Karpinski gave an overview of features and roadmap for getting to Julia +1.0, which I think was a topic close to most attendee's hearts (including +mine). I ended up with a huge list of written notes, which I'll summarize +below; the punchline was aiming to have a 1.0 release around one year from now. +Apparently the one-year goal has been floated in previous years; I'm not sure +how wise it is in general to float initial release timelines for a project like +this, it seems like it will just "be done when it's done". + +Some of the goals that were interesting to me: + +- Arrays: might refactor Arrays to have a separate backing abstraction of "Buffers" + with arrays on top (apparently Lua and Torch do this). +- Strings: move full Unicode support out of core language (Base) and into a + package. The `@printf` macro will be refactored into a function. To my + surprise, currently Strings are implemented as an Array! This has a + relatively large overhead for each string (72 bytes). +- Modularity and Package infrastructure: currently a mess (I agree), `import`, + `using` and `export` will be refactored. +- Compiler: add non-pthreads multithreading; better static compilation; ability + to define a `main()` function and get a standalone script or binary; ability + to redefine functions and have the changes propagate (cache invalidation + problem); stabilize intermediate representations. Seems like a lot! +- Optimizations: faster garbage collection, more auto-vectorization (eg, for + vector floating point units), improve globals performance. Might pull in part + of ParallelAccelerator? + +I'm a little nervous how many of these goals are big open questions instead of +just implementation tasks. I wish there was a more healthy way to experiment +with new features and refactoring without breaking everything or committing to a +long-term stable API; I think other languages have settled into good patterns +for this kind of development, though maybe they needed to go through a +difficult 1.0 process first. It was mentioned that 0.6 would be the last of the +0.x series of releases and considered 1.0-alpha, and that from 1.x and on +things should generally be backwards compatible. + +Separate from Stefan's talk, there was a short overview of progress on the next +iteration of the Julia package and dependency manger, called Pkg3. The goals +were described as "a mash-up of virtualenv and cargo": virtualenv is a tool for +isolating per-application dependencies and toolchains in Python, and Cargo is +is the Rust dependency manager and build tool (which is also used in a +per-application fashion). Pkg3 sounds like it will have a concept of distinct +"global" (meaning system-wide?) installations and "local" (eg, per-project or +per-directory) installations and name-spacing. The naming could use some work, +as "global" and "local" are pretty overloaded, but I think they are chasing the +right goals. Reproducibility (both for binary generation and data/experiment +reproduction), lock files (which lock in known-good versions of dependencies a +la Cargo), and other concepts that I care about were also thrown around. I +didn't catch all the details (and I'm not sure how much has been worked out and +implemented yet), but after my experiences with [Elm and Rust][elm-broken], and +the current state of packaging for Julia, I'm excited for Pkg3! + +[elm-broken]: /2016/elm-everything-broken/ + + + +### Overall Julia Feels + +There is sort of an explosion of ideas and experiments going on. It feels sort +of like what the Ruby community maybe went through with web frameworks, or the +web community did with languages that compile to Javascript: ambitious ideas, +which may have been on the back-burner for some time, can finally be prototyped +quickly and tested in a mostly-real-world environment, and everybody is excited +to try it out and demo their creations. + +One of the sponsors said: + +> "there is something quite good about not feeling bad about programming" + +and that seemed representative of the current state of Julia. It seems +undeniable that the language is less painful for developing performant +numerical code than the previous generation of languages and library wrappers. + +Perhaps because of this enthusiasm and froth of ideas, I'm a little worried +that the foundations of Julia (the language and the ecosystem) have not yet had +time to fully bake. The more demos and experiments that get implemented, and +the more popular they become, the more delicate it becomes to make hard +decisions about language syntax and features. I think people want stability and +promised features *yesterday*, but these things take time and reflection. My +feelings right now is that it doesn't really matter. The enthusiasm for +*a language like Julia* is proven and growing. Julia itself might end up being +the first try that gets thrown away in a decade or two, but in the end we'll +end up with something which is both exciting and robust. + +[PyX.jl]: https://github.com/bnewbold/PyX.jl +[rust]: https://www.rust-lang.org/ + diff --git a/posts/2016/new-server-2016.md b/posts/2016/new-server-2016.md new file mode 100644 index 0000000..8f2245b --- /dev/null +++ b/posts/2016/new-server-2016.md @@ -0,0 +1,81 @@ +Title: New Web Server (2016) +Author: bnewbold +Date: 2016-05-01 +Tags: tech + +For the past 9 years or so ([February 2007][0] through April 2016), this +website has run reliably despite being a weird machine: a custom prototype web +application for archiving and sharing digital "artifacts" of all sorts. I +remember a bold and sparkling afternoon at a coffee house in Santa Cruz +agonizing over URL structure and refining categories to collect all the +material that would (aspirationally) accumulate here over my adult life. All +things considered I think this effort wasn't wasted: I have indeed collected +wiki notes, photos, images (distinct!), short links, one-off pages, and longer +form writing over the years. My enthusiasm for maintaining and interfacing with +idiosyncratic administrative and upload panels, however, declined rapidly. +Neither the source code web interface nor the git-backed wiki were ever +completed or used. I gave up on spam moderating the comment system pretty +quickly, never actually posted any blog entries, and only ever pecked in a +couple dozen web links and tweet-like microposts. Even the photo gallery system +became too much of a time sink to deal with after traveling. Over the years I +deployed [gitweb][] and [gitit][], and after RSS starting going out of favor I +even started using [tumblr][1] for random content posts. I was still using the +Django admin panel's "flatpages" plugin to update my contact info and project +pages right up through this spring though. + +[gitweb]: https://git-scm.com/docs/gitweb +[gitit]: http://gitit.net/ +[0]: https://git.bnewbold.net/bnewnet/commit/?id=5b31039d4c581048959dc51436f6918f29fbf9ea +[1]: http://journal.bnewbold.net/ + +While it's been amazing how simple and low maintenance running everything +(email, repositories, website, etc) has been, the thrill of being on an old and +unmaintained release of GNU/Linux (Ubuntu 10.04) has worn off, and I'm cleaning +house. This website (`bnewbold.net`) is now a simple statically generated +([pelican][]) site. I've kept `gitit`, but moved to `gitolite` (one of my +favorite pieces of software) and `cgit` for repository hosting. The server runs +Debian stable (`jessie` to start with), and SSL/TLS certificates come gratis +via the Let's Encrypt project, for which I'm very grateful. I've stuck with +[linode][] hosting for this server, though [digital ocean][] is comparable and +cheaper for setups with fewer photos and large files. Many of these components +are deployed in an automated fashion using the `ansible` deployment tool; you +could fork and edit my [infrastructure scripts][infra] to set up those +components in minutes if you like (though it would probably take an afternoon +or longer if you've never done something like this before). Not everything is +settled yet: I haven't moved email, and I'm not sure if I'll stick with +[mediagoblin][] for photo hosting. + +[pelican]: http://blog.getpelican.com/ +[linode]: https://www.linode.com +[digial ocean]: https://digitalocean.com +[infra]: http://git.bnewbold.net/infra/ +[mediagoblin]: http://mediagoblin.org + +
+
+
+ + + +
+An adelie penguin (the tiny black spot) wandering off alone over the Ross Sea +ice shelf +
+
+
+ +The name of my old server is `adelie` (named after a small species of penguin +that [I saw a lot of][4] in Antarctica), and it's days are limited. Though the +[`hier`][5] (filesystem layout) of most of my systems are almost identical, I'm +particularly comfortable on this one. It's seen me through a lot, more than any +other computer system, and I'll probably miss it in a weird way. + +[4]: http://bnewbold.net/photos/detail/527/ +[5]: http://man7.org/linux/man-pages/man7/hier.7.html + +The new server is `adze`; here's hoping it lives even longer! + +
+_**Note:** As initially posted this was just a stub; I re-wrote it in June +2016_ + diff --git a/posts/elm-everything-broken.md b/posts/elm-everything-broken.md deleted file mode 100644 index a9978d1..0000000 --- a/posts/elm-everything-broken.md +++ /dev/null @@ -1,427 +0,0 @@ -Title: Everything Is Broken (Installing Elm Edition) -Author: bnewbold -Date: 2016-05-06 - -*This post is a narrative rant (in the same vein of Dan Luu's ["Everything is -Broken"][danluu] post) about my problems one afternoon getting a Fancy New -Programming Language to work on my laptop. [Skip down](#nowwhat) for my -thoughts on solutions.* - -[danluu]: https://danluu.com/everything-is-broken/ - -
- -
- -photo of E.C. giving an elm talk - -
- -A few folks here at the [Recurse Center][rc] had nice things to say about the -[Elm programming language][rc], and after watching Evan Czaplicki's ["Let's Be -Mainstream" talk][0] I got pretty excited also. "Let's try Elm!" and the -adventure begins. - -[rc]: https://recurse.com/ -[elm]: https://elm-lang.org -[0]: https://www.youtube.com/watch?v=oYk8CKH7OhE - -First I go in the front door with the [install page][install], which redirects -me to a page which instructs me to use npm. - -[install]: http://elm-lang.org/install - -Hrm, I've had painful experiences with npm in the past, and anyways, isn't -the Elm compiler implemented in Haskell? Why are they using a package -management tool for Javascript libraries for installation? No thanks, I'll -[build from source][build], following these directions from the `README.md`: - -[build]: https://github.com/elm-lang/elm-platform - - # If you are on LINUX, you need to install a dependency of elm-repl. - # Uncomment the following line and run it. - # sudo apt-get install libtinfo-dev - - # if you are on windows, or some other place without curl, just download this file manually - curl https://raw.githubusercontent.com/elm-lang/elm-platform/master/installers/BuildFromSource.hs > BuildFromSource.hs - - runhaskell BuildFromSource.hs 0.16 - - - -Weird that these build-from-source directions involve curl-to-file magics. Why -not just clone the repo and use a Makefile, which is roughly the same number of -commands to enter? I clone repo and run the build script from there instead. -I'm on Debian stable ("jessie"), so installing GHC (the most popular way to use -Haskell) is easy. I seem to have a recent-ish version of cabal: - -
-bnewbold@eschaton$ cabal --version
-cabal-install version 1.20.0.3
-using version 1.20.0.2 of the Cabal library 
-
- -... so I think i'm ready to go ahead run the `BuildFromSource.hs` script: - -
-bnewbold@eschaton$ runhaskell ~/src/elm-platform/installers/BuildFromSource.hs 0.16
-
-GHCi runtime linker: fatal error: I found a duplicate definition for symbol
-_hs_bytestring_long_long_uint_hex
-whilst processing object file
-/home/bnewbold/.cabal/lib/x86_64-linux-ghc-7.6.3/bytestring-0.10.6.0/HSbytestring-0.10.6.0.o
-This could be caused by:
-* Loading two different object files which export the same symbol
-* Specifying the same object file twice on the GHCi command line
-* An incorrect `package.conf' entry, causing some object to be
-    loaded twice.
-GHCi cannot safely continue in this situation.  Exiting now.  Sorry.
-
- -Uh-oh! Not sure what that means. A search returns [a FAQ entry][faq-entry] -about "Duplicate Directions" which talks about linker flags, but *I* didn't -compile anything by hand so it doesn't seem like *I* have done anything wrong. -I'll try updating `cabal`'s package index: - -[faq-entry]: https://wiki.haskell.org/GHC/FAQ#Duplicate_Definitions - -
-bnewbold@eschaton$ cabal update
-Downloading the latest package list from hackage.haskell.org
-cabal: Failed to download
-http://hackage.haskell.org/packages/archive/00-index.tar.gz : ErrorMisc "Error
-HTTP code: 502"
-
- -No go! At first I assumed I was doing something wrong, or maybe the WiFi was -flakey, but it turns out that the Hackage package repository [is -down](https://status.haskell.org/): - -> hackage.haskell.org down -> -> Full Service Disruption -> -> [Investigating] The hackage server is in an out of memory condition. We're investigating. - -
-As far as I know, the only Debian packages out of the huge stable archive which -don't receive security updates are web browsers, mediawiki, and all V8/node.js -packages and libraries. -
- -Well, it wasn't even clear if updating my package index would fix the problem, -it was just a wild guess. I'll give up and try npm, which these days has been -[sort of packaged][deb-node] in Debian. I'll just uninstall npm when I'm done -installing Elm. I install with `apt` (which pulls in a huge list of `nodejs` -packages), but then can finally run: - -[deb-node]: https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#libv8 - -
-bnewbold@eschaton$ npm install elm
-|
-> elm@0.16.0 install /home/bnewbold/bin/node_modules/elm
-> node install.js
-
-sh: 1: node: not found
-npm WARN This failure might be due to the use of legacy binary "node"
-npm WARN For further explanations, please read
-/usr/share/doc/nodejs/README.Debian
-
-npm ERR! elm@0.16.0 install: `node install.js`
-npm ERR! Exit status 127
-npm ERR! 
-npm ERR! Failed at the elm@0.16.0 install script.
-npm ERR! This is most likely a problem with the elm package,
-npm ERR! not with npm itself.
-npm ERR! Tell the author that this fails on your system:
-npm ERR!     node install.js
-npm ERR! You can get their info via:
-npm ERR!     npm owner ls elm
-npm ERR! There is likely additional logging output above.
-
-[...]
-
-
- -
-Later I learned that I could have instead installed the nodejs-legacy -package, which just installs a symlink /usr/bin/node which points to -/usr/bin/nodejs. -
- -Looks like Debian wants to refer to node.js as "`nodejs`" while npm/Elm expects -it to be "`node`". Ok, I create this shim named `node` in my `~/bin` directory -(which is on my `$PATH`), and make it executable: - -
-#!/bin/sh
-nodejs $*
-
- -Now I can do things like: - -
-bnewbold@eschaton$ node --version
-v0.10.29
-
- -Great! Now let's install Elm with npm! - -
-bnewbold@eschaton$ npm install elm
-
-> elm@0.16.0 install /home/bnewbold/bin/node_modules/elm
-> node install.js
-
-Downloading Elm Reactor assets from https://dl.bintray.com/elmlang/elm-platform/0.16.0/elm-reactor-assets.tar.gz
-Error communicating with URL https://dl.bintray.com/elmlang/elm-platform/0.16.0/linux-x64.tar.gz Error: CERT_UNTRUSTED
-npm WARN This failure might be due to the use of legacy binary "node"
-npm WARN For further explanations, please read
-/usr/share/doc/nodejs/README.Debian
-
-npm ERR! elm@0.16.0 install: `node install.js`
-npm ERR! Exit status 1
-npm ERR! 
-npm ERR! Failed at the elm@0.16.0 install script.
-npm ERR! This is most likely a problem with the elm package,
-npm ERR! not with npm itself.
-npm ERR! Tell the author that this fails on your system:
-npm ERR!     node install.js
-npm ERR! You can get their info via:
-npm ERR!     npm owner ls elm
-npm ERR! There is likely additional logging output above.
-
-npm ERR! System Linux 3.16.0-4-amd64
-npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "elm"
-npm ERR! cwd /home/bnewbold/bin
-npm ERR! node -v v0.10.29
-npm ERR! npm -v 1.4.21
-npm ERR! code ELIFECYCLE
-npm ERR! 
-npm ERR! Additional logging details can be found in:
-npm ERR!     /home/bnewbold/bin/npm-debug.log
-npm ERR! not ok code 0
-
- -Whoops. There is [a github issue][elm-issue] that has some tips about this: the -recommended solutions online are basically "upgrade node.js and npm" or -"disable SSL security". That doesn't sound great, but i'm feeling very -impatient at this point so I tried disabling SSL checks with `npm config set ca -null`, but even then I get the certificate error. Enough of that approach: - -[elm-issue]: https://github.com/elm-lang/elm-platform/issues/100 - -
-sudo apt remove npm nodejs
-rm ~/bin/node
-
- -Poking around a bit more, I find the promisingly-titled "`fubar.sh`" script in -the `elm-platform` repo cloned earlier, which seems to nuke the local user's -Haskell (`cabal`) stuff (but doesn't muck with system-wide package, eg `apt` -managed GHC libraries). After running `fubar.sh` the `BuildFromSource.hs` -script seems to start working (success!), but Hackage is still down (now, an -hour after I started this process, approaching 5 hours total downtime), so I -can't install dependencies. - -I search for a hackage mirror and find that [FPComplete runs -one](https://www.fpcomplete.com/blog/2015/03/hackage-mirror). In addition to -their configuration instructions I had to set `remote-repo-cache` in my -`~/.cabal/config`, like so: - - remote-repo: hackage.fpcomplete.com:http://hackage.fpcomplete.com/ - remote-repo-cache: /home/bnewbold/.cabal/packages-fpcomplete - -After that I `cabal update` and run the `BuildFromSource.hs` script again. It -fails to build the dependency `websockets-snap`, but I try installing that -regularly (`cabal install websockets-snap`) and that works, and then all the -dependencies with BuildFromSource.hs build. Yay! - -I still get an error with compiling Elm itself though: - - src/Elm/Package.hs:60:25: - Not in scope: `<$>' - Perhaps you meant `' (imported from System.FilePath) - -It [sounds like](https://github.com/elm-lang/elm-platform/issues/30) I am using -an out of date version of GHC (the Haskell compiler): I have 7.6.3 (the version -that comes with Debian), and Elm wants 7.10. Fair enough: unlike a package -manager, I can see how new features in the compiler would be helpful, and when -I go back and check this was outlined in the README. Fortunately there is a new -version of GHC in the `jessie-backports` repo, so: - -
-sudo apt install -t jessie-backports ghc
-
- -And now: - -
-bnewbold@eschaton$ cabal --version
-cabal-install version 1.22.6.0
-using version 1.22.5.0 of the Cabal library 
-bnewbold@eschaton$ ghc --version
-The Glorious Glasgow Haskell Compilation System, version 7.10.3
-
- -After that, I'm miraculously able to compile. Horray! I add -`~/bin/Elm-Platform/0.16/.cabal-sandbox/bin` to my `$PATH` and I can run: - -
-bnewbold@eschaton$ elm-repl 
----- elm repl 0.16.0 -----------------------------------------------------------
-:help for help, :exit to exit, more at 
---------------------------------------------------------------------------------
-> 
-
- -Success! - -To be clear, once I got this far I had few or no problems with Elm. The -language seems reasonably clean and well documented for it's maturity, and it -was only this install procress that felt horribly broken. - -Throughout this debugging process I had problems with both my external [Lenovo -Thinkpad USB keyboard][keyboard] glitching (workaround: unplug and re-plug) and -with WiFi at the Recurse Center disconnecting (workaround: reset wifi card). -Others here using GNU/Linux have the same wifi problems, but we don't have a -fix yet... something to do with wireless regulatory zones. - -It's not all pain though; installing the `elm.vim` syntax highlighting plugin -was very easy, just a single `git clone` into `~/.vim/bundle/`. - -[keyboard]: https://www.amazon.com/Lenovo-ThinkPad-Compact-Keyboard-TrackPoint/dp/B00F3U4TQS - - - -So Now What? ----------------- - -There was a great long-form blog post by Sam Boyer a few months ago ("[So You -Want To Write a Package Manager][5]") which describes the jumble of *system*, -*language*, and *project* dependency manager problems I experienced above. Sam -basically concludes that Rust's project/language dependency manager (Cargo) -both encapsulates best practices for dependable and reproducible builds, while -still allowing rapid evolution of a package "ecosystem". Cargo was designed by -[Yehuda Katz and Carl Lerche][6], and along with the -[crates.io](https://crates.io) archive it does sound very nice. My personal -feeling are usually that system-wide package managers (like Debian's `apt`) are -underappreciated by many young-but-not-bleeding-edge projects, but acknowledge -that there probably is also a need for higher tempo cross-platform project -dependency mangement for non-library projects (eg, desktop applications and web -apps). - -Ironically (given the difficulty I had installing it), the Elm language's -package manager has a great `diff` tool for checking that any changes in the -API conform to the [documented][8] [semantic versioning][semver] conventions. -For [example](https://gist.github.com/badboy/a302dd0c9020e5759240): - - $ elm-package diff evancz/elm-html 3.0.0 4.0.2 - Comparing evancz/elm-html 3.0.0 to 4.0.2... - This is a MAJOR change. - - ------ Changes to module Html.Attributes - MAJOR ------ - - Removed: - boolProperty : String -> Bool -> Attribute - stringProperty : String -> String -> Attribute - - - ------ Changes to module Html.Events - MINOR ------ - - Added: - type alias Options = - { stopPropagation : Bool, preventDefault : Bool } - defaultOptions : Html.Events.Options - onWithOptions : String -> Html.Events.Options -> Json.Decode.Decoder a -> (a -> Signal.Message) -> Html.Attribute - -This API change information is then used to *[programatically enforce][7]* the -semantic versioning rules for submissions to the Elm language library archive -and prevent a whole class of simple but annoying breakages due to unexpected -API changes. It can't detect *every* breaking change (eg, those which are -internal), but it can detect enough to be worth the effort. - -Another option for detecting fixing breakage is integration testing at the -package ecosystem level. Debian's [reproducible builds][repro] effort has built -out one such system, and continues to catch thousands of "failed to build from -source" (FTBFS) bugs along the way. Dan Luu's ["Everything is Broken"][danluu] -blog post (which to some degree inspired this one) mostly focused on the lack -of quality tests for many contemporary software projects, and proposed the use -of smart fuzzing and heuristically-generated tests to work around the huge -technical debt this represents. As far as I can tell there are hardly ever -automated integration tests for entire package archives; blame for dependency -problems is usually attributed to a bug in one package (instead of being seen -as a systemic failure), and such problems are seen as an unavoidable cost of -rapid and distributed development. - -I think we can do better. For example, some contemporary languages test -compiler changes for regressions against a broad snapshot of public code -written in that language; I know Go and Rust specifically do this to identify -problems before compiler releases, and the Python community occasionally runs -tests against the PyPi archive when considering syntax changes or feature -deprecations. And Debian's unstable and testing archives enforce a waiting -period so that human testers can turn up conflicts before packages can be moved -into stable or backports. I think we can and should automate these processes as -much as possible, and give direct feedback to library developers and package -maintainers when they push updates. Commercial Continuous Integration services -like Travis CI and Circle CI should offer more common target platforms (eg, ARM -architecture, more versions of distributions), and if they don't we should -build and host our own testing infrastructure. CI builds scripts should closely -match the official installation instructions for a given platform, so we catch -problems with those instructions quickly. - -We have more and more developers in the world wasting more and more days -wrangling with dependency hell, but despite that I think we're closer than ever -to taming the beast. - -[5]: https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527 -[6]: https://mail.mozilla.org/pipermail/rust-dev/2014-March/009090.html -[7]: https://github.com/elm-lang/elm-package/blob/master/README.md#version-rules -[8]: https://github.com/elm-lang/elm-package/blob/master/README.md#publishing-updates -[semver]: http://semver.org/ -[appcheck]: http://www.linuxfoundation.org/collaborate/workgroups/lsb/linux-application-checker-getting-started -[repro]: https://tests.reproducible-builds.org/debian/reproducible.html - -
- -#### Follow up June 18th, 2016 - -A few days after writing this post, a new version of Elm (0.17) was -[released](http://elm-lang.org/blog/farewell-to-frp). The new version made some -big changes, like abandoning the [Functional Reactive Programming][frp] -paradigm. I found that many of the new tutorials weren't working with my 0.16 -install, so I tried to upgrade. After an hour or two of following the -directions above, I gave up and installed npm on a temporary Debian unstable -("sid") virtual machine. Once I added a `node` symlink to `nodejs`, I was able -to install `elm` with npm and develop remotely using `screen` and -`elm-react`. Frustrating! - -[frp]: https://en.wikipedia.org/wiki/Functional_reactive_programming - -
-
- -
-Also, status.haskell.org seems to be broken again today, though -the hackage website works. -
- -On the plus side, my USB keyboard problem seems to have been due to the cheap -random USB cable I was using. Since replacing it I haven't had the bad -flakey-ness, though now after my laptop awakes from sleep the pointer and -keyboard work but the middle mouse button ("paste" on UNIX) does not. -Workaround: plug and unplug the whole keyboard. - diff --git a/posts/juliacon.md b/posts/juliacon.md deleted file mode 100644 index 52751a9..0000000 --- a/posts/juliacon.md +++ /dev/null @@ -1,339 +0,0 @@ -Title: What I Learned At JuliaCon -Author: bnewbold -Date: 2016-07-12 -Tags: tech, recurse, julia - -*Note: It looks like videos of the JuliaCon talks were uploaded [to -Youtube][youtube] the day this post was finally published!* - -[youtube]: https://www.youtube.com/playlist?list=PLP8iPy9hna6SQPwZUDtAM59-wPzCPyD_S - -I was in Cambridge, MA for a few days the other week at [JuliaCon][], a small -conference for the Julia programming language. Julia is a young language -(started around 2014 and currently pre-1.0) oriented towards fast numerical -computation: matrix manipulation, simulation, optimization, signal analysis, -etc. I've done a fair amount of such programming over the years, and it has -never felt as elegant or coherent as it could be. The available tools and -languages are generally either: - -[JuliaCon]: http://juliacon.org - -
-julia logo -
- -1. stuck in the 1980s in terms of programming language features for safety, - productivity, and collaboration (eg, Fortran and Matlab) -1. expensive proprietary closed-source packages (eg, Matlab and Mathematica) -1. general-purpose languages with numerical features either hacked on or in the - form of libraries (eg, Python) - -There is a lot to be excited about in Julia. It's already pretty fast -(leveraging pre-existing JIT tools, hand-tuned matrix and solver libraries, and -the LLVM compiler suite) and has contemporary high-level language features -(like optional type annotation, polymorphic function dispatch, package -management tools, and general systems tools (eg, JSON and HTTP support)) that -can make the language more faster to develop in, and easier to read and -maintain. I'm personally excited about the progeny of the language: the -birthplace of the language is the CSAIL building at MIT, and the spirit of -[Scheme][sicm] and the work of [Project MAC][] is sprinkled through the -project. One of the [big pitches](graydon2) of Julia is that scientists won't -need to learn both a productive high-level language (eg, Python) and a -low-level performant language (eg, C or Fortran) and interface between the two: -Julia has everything all in one place. - -[graydon2]: http://graydon2.dreamwidth.org/3186.html -[sicm]: https://mitpress.mit.edu/sites/default/files/titles/content/sicm/book.html -[Project MAC]: http://groups.csail.mit.edu/mac/projects/mac/ - -All that being said, while I thought I would be working in Julia a lot during -my time at the Recurse Center, I've ended up being much more drawn to the -[Rust][] language instead. Rust is a general systems language (it's compiled, -has stronger typing, and no garbage collection), and not great for interactive -numerical exploration, but I've found it a joy to program in: for the most part -everything *just works* the way it says it will. My recent experience with -Julia, on the other hand, has been a lot of breakage between library and -interpreter versions, poor developer usability (eg, hard to figure out where -files should live in a package), and very frustrating import/load times. Though -I have to admit that I while I pushed through some frustrations with Rust, I -haven't spent *that* much time with Julia, and may have just been impatient, so -take everything I say here with a grain of salt. - -With these feels going in, what did I learn at JuliaCon and what do I think of -the future of the language now? In the below sections I'll go over the -interesting things I saw, then come back to summary at [the end](#summary). - -### Programming Language Design - -An older research language for numerical computing that I have always been -curious about is Fortress, and the leader of that project (Guy Steele, who also -worked on the design of the Scheme and Java languages) gave one of the opening -keynote speeches at JuliaCon this year. Awesome! I get really excited about -inter-generational learning and dialog. - -Fortress was a very "mathy" language. The number tower was intended to be -"correct" (aka, have the same structure that mathematicians use), physical -units were built-in, and some operator precedence was non-transitive. Operators -on built-in types (like Integers) could be overloaded, unlike in Java, because -Fortress users could apparently be trusted to "preserve algebraic properties". -Steele is a proponent of using whitespace (or lack of whitespace) to clarify -expressions, sort of like extra parentheses, and enforcing this in the -compiler. For example, the following two statements would be equivalent in most -languages, but not in Fortress: - -``` -a + b*c + d // Clear: Ok -a+b * c+d // Misleading: Compiler Error -``` - -This was part of a general effort to allow "whiteboard" style syntax in the -language. Fortress code actually has two representations: a plain text -Scala-style source code, and a LaTeX-y symbolic math format. Steele also used -some font-coloring in his slides to differentiate different types of symbols, -which reminded me of the helpful style my undergraduate physics professors -would use on the blackboard. I think this effort to adapt the "look and feel" -of the language to how the intended audience already writes and communicates is -really cool. I wonder if a third syntax format could have been added in a -one-to-one manner: that of a general purpose language like Scala or Haskell -(both noted as influences to Fortress) to make collaboration with general -purpose programming experts easier. Steele mentioned that some efforts to make -the syntax more math-like resulted in "contortions", so there is probably more -work to be done here. - -In my limited experience, Julia has a pretty clean syntax, and allows some -math-y [unicode characters as operators][unicode_ops] (like ∈, ≠, etc), but -didn't prioritize math-y syntax as much as Fortress. Given the open challenges -with formalizing informal whiteboard syntax this may or may not have been a -missed opportunity. - -[unicode_ops]: http://docs.julialang.org/en/release-0.4/manual/unicode-input/ - -The positive lessons learned from Fortress were summarized as being the type -system, automatic parallelism (via generators and reducers), the math-y syntax, -pretty printing (I assume meaning the LaTeX-y representation), physical units, -and forced syntax clarity (aka, forced use of parentheses and whitespace). One -issue that come up during implementation was that it was hard to bound the -latency and computational complexity of type constraint solving at run-time. - -A few other talks touched on language design decisions and features. There was -a short "Functional HPC" talk by Erik Schnetter, in which it was pointed out -that for some workloads regular old garbage collection can be faster than -reference counting: I've become used to thinking of latency and GC pauses as a -huge performance problem in systems programming, but for number crunching that -isn't as much of an issue, while little reference overheads are (especially if -locks or atomic operations are necessary). - -Keno Fischer gave an overview of the [Gallium][] debugger, which had some cool -features, but is still under development. There are both AST-based and -LLVM-based backends for the debugger, which allows stepping at function calls, -line-by-line, or expression-by-expression, which is something I hadn't seen -before. He demoed stepping through each step of the creation of a matplotlib -graph, with the output shown graphically after each step. Neat stuff! - -One of my personal interests in Julia would be formalizing the syntax into a -machine-readable grammar (eg, [EBNF][] or [ABNF][]). I was lucky enough to run -in to Stefan Karpinski during one of the coffee breaks, and he pointed me to -the Julia plugin for Eclipse, which already has a partial implementation of a -grammar. - -A few talks touched on the issue of Nullable datatypes (also called "Maybe" or -"Option" types in other languages), particularly for data science and -DataFrame-type applications. I only recently encountered [Option][] (and the -related [Result][] type) datatypes, in Rust, and can see why people want these -so badly, but there doesn't seem to be a simple path forward yet. Rust really -leverages these types in function return signatures, a feature which Julia does -not have for now; I think I read rumors about them being added in the future, -but didn't hear any mention of them here or on the 1.0 feature roadmap. - -[Option]: https://doc.rust-lang.org/std/option/index.html -[Result]: https://doc.rust-lang.org/core/result/index.html -[Gallium]: http://juliacon.org/abstracts.html#Gallium -[EBNF]: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_Form -[ABNF]: https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_Form - -### Numeric Abstraction - -One of the big trends I saw was taking advantage of Julia's abstractions around -generic operators and arrays to experiment with novel computation strategies. -Sometimes this means improving precision (with novel data types and -representations), sometimes it means increasing performance (by changing memory -layout or distribution, or targeting special hardware), and sometimes it just -makes code more elegant or semantic. - -For example, Tim Holy gave a talk (titled "To the Curious Incident of the CPU -in the run-time") which covered a bunch of nitty-gritty details for -implementing wrapper classes that re-shape or re-size Arrays, including sparse -arrays. - -Lindsey Kuper gave a nice overview of the [ParallelAccelerator.jl][pajl] -project, which entirely re-compiles Julia into C++ to get some extra performance -from the static full-program compiler. It seems to me that this only makes -sense because the Julia language has clean abstractions that the transpiler can -take advantage of. - -[pajl]: http://juliacon.org/abstracts.html#ParallelAccelerator - -One of my favorite talks from the whole conference was David Sanders' and Luis -Benet's talk on ValidatedNumerics ("Precise and rigorous calculations for -dynamical systems"). Instead of computing on approximate (rounded) scalars, -they compute on intervals of floating point numbers (or in higher dimensions, -boxes): at the end of computation the "correct" solution is known to be within -the final box, which also gives context as to how much numerical error has -accumulated. By defining new *types* to accomplish this (specifically, -DualNumbers), they can re-use any generic code in a relatively performant -manner. They also noted that when there is an analytic form to bound the error -for all following terms, Taylor expansion approximations can be truncated as -soon as the interval error exceeds the error in all following terms. Cool! - -### Other Fun Stuff - -**[Using Julia as a Quick and Dirty Code Generator][10]:** -The speaker (Arch Robison) is clearly having way too much fun! He used Julia to -output assembly code to get fast (real-time) discrete Fourier transform (DFT) -performance for a little video game called "FreqonInvaders". Infectious -enthusiasm! - -**[Autonomous driving for RC cars with ROS and Julia][11]:** -A fun little project doing "Model Predictive Control" on a small model car to -do stunts like drifting and slide parking into a tiny space. They achieved -about a 10Hz closed-loop control latency, which seems to me like barely enough -for this sort of thing, but clearly worked alright. Everything ran on the car -itself (no computation on a remote desktop with wireless control or anything -like that), with an Odroid ARM Linux system and an Arduino-compatible -microcontroller; Julia code using JuMP and other optimization stuff ran on the -ARM system. The code and raw data (for analysis) is available on the [BARC -project website](http://www.barc-project.com). Super cool, having this stuff -being experimented with already means there will be pressure to improve -soft-real-time performance in the language itself. - -**[Astrodynamics.jl: Modern Spaceflight Dynamics in Julia][12]:** -Mostly a bunch of code for doing timebase conversions and interpreting (or -calculating) ephemeris data (which is information about where astro bodies like -the Moon and planets will be at a given time), but some simple demos of orbital -simulation and event detection (eg, perihelion time and position) as well. Would -be cool if the ValidatedNumerics stuff was integrated. - -**[GLVisualize][13]:** -The demos in this talk were really impressive: live editing of mesh vertices, -relatively high performance, real-time feedback, etc. There were a bunch of -good graphics talks: the [GR Framework][14] stuff is really impressive in scope -(though maybe not as big a performance boost over Python as hoped), and -[Vulkan][15] is exciting. - -[10]: http://juliacon.org/abstracts.html#FrequonInvaders -[11]: http://juliacon.org/abstracts.html#RaceCars -[12]: http://juliacon.org/abstracts.html#Astrodynamics -[13]: http://juliacon.org/abstracts.html#GLVisualize -[14]: http://juliacon.org/abstracts.html#GR -[15]: http://juliacon.org/abstracts.html#Vulkan - -### Diversity - -It's sad to say, but the gender diversity at the conference was really poor, -particularly in contrast to the Recurse Center (where I have spent the past -couple months). The women I did meet gave some of the best talks, are crucial -contributors to infrastructure, and are generally amazing: more please! Aside -from the principle of the thing, there is just something about a giant sea of -guys at a tech event that results in a tense group vibe. Everybody I spoke to -one-on-one was friendly and we had great conversations, but as a group there -was a lot of ice to be broken. In my experience even hitting 10-20% women in -attendance can thaw this out, but that's just my anecdotal experience. - -I haven't attended, but I hear that PyCon has done a great job improving -diversity with careful planning and [systemic initiatives][pycon-diversity]. - -Overall, I thought the conference was a great group of people and admirably -well run. I appreciated the efforts to keep costs low, and everything generally -ran on time. Thanks to all the volunteer and MIT staff organizers for their -efforts! - -[pycon-diversity]: https://us.pycon.org/2016/about/diversity/ - -### Julia 1.0 - -Stefan Karpinski gave an overview of features and roadmap for getting to Julia -1.0, which I think was a topic close to most attendee's hearts (including -mine). I ended up with a huge list of written notes, which I'll summarize -below; the punchline was aiming to have a 1.0 release around one year from now. -Apparently the one-year goal has been floated in previous years; I'm not sure -how wise it is in general to float initial release timelines for a project like -this, it seems like it will just "be done when it's done". - -Some of the goals that were interesting to me: - -- Arrays: might refactor Arrays to have a separate backing abstraction of "Buffers" - with arrays on top (apparently Lua and Torch do this). -- Strings: move full Unicode support out of core language (Base) and into a - package. The `@printf` macro will be refactored into a function. To my - surprise, currently Strings are implemented as an Array! This has a - relatively large overhead for each string (72 bytes). -- Modularity and Package infrastructure: currently a mess (I agree), `import`, - `using` and `export` will be refactored. -- Compiler: add non-pthreads multithreading; better static compilation; ability - to define a `main()` function and get a standalone script or binary; ability - to redefine functions and have the changes propagate (cache invalidation - problem); stabilize intermediate representations. Seems like a lot! -- Optimizations: faster garbage collection, more auto-vectorization (eg, for - vector floating point units), improve globals performance. Might pull in part - of ParallelAccelerator? - -I'm a little nervous how many of these goals are big open questions instead of -just implementation tasks. I wish there was a more healthy way to experiment -with new features and refactoring without breaking everything or committing to a -long-term stable API; I think other languages have settled into good patterns -for this kind of development, though maybe they needed to go through a -difficult 1.0 process first. It was mentioned that 0.6 would be the last of the -0.x series of releases and considered 1.0-alpha, and that from 1.x and on -things should generally be backwards compatible. - -Separate from Stefan's talk, there was a short overview of progress on the next -iteration of the Julia package and dependency manger, called Pkg3. The goals -were described as "a mash-up of virtualenv and cargo": virtualenv is a tool for -isolating per-application dependencies and toolchains in Python, and Cargo is -is the Rust dependency manager and build tool (which is also used in a -per-application fashion). Pkg3 sounds like it will have a concept of distinct -"global" (meaning system-wide?) installations and "local" (eg, per-project or -per-directory) installations and name-spacing. The naming could use some work, -as "global" and "local" are pretty overloaded, but I think they are chasing the -right goals. Reproducibility (both for binary generation and data/experiment -reproduction), lock files (which lock in known-good versions of dependencies a -la Cargo), and other concepts that I care about were also thrown around. I -didn't catch all the details (and I'm not sure how much has been worked out and -implemented yet), but after my experiences with [Elm and Rust][elm-broken], and -the current state of packaging for Julia, I'm excited for Pkg3! - -[elm-broken]: /2016/elm-everything-broken/ - - - -### Overall Julia Feels - -There is sort of an explosion of ideas and experiments going on. It feels sort -of like what the Ruby community maybe went through with web frameworks, or the -web community did with languages that compile to Javascript: ambitious ideas, -which may have been on the back-burner for some time, can finally be prototyped -quickly and tested in a mostly-real-world environment, and everybody is excited -to try it out and demo their creations. - -One of the sponsors said: - -> "there is something quite good about not feeling bad about programming" - -and that seemed representative of the current state of Julia. It seems -undeniable that the language is less painful for developing performant -numerical code than the previous generation of languages and library wrappers. - -Perhaps because of this enthusiasm and froth of ideas, I'm a little worried -that the foundations of Julia (the language and the ecosystem) have not yet had -time to fully bake. The more demos and experiments that get implemented, and -the more popular they become, the more delicate it becomes to make hard -decisions about language syntax and features. I think people want stability and -promised features *yesterday*, but these things take time and reflection. My -feelings right now is that it doesn't really matter. The enthusiasm for -*a language like Julia* is proven and growing. Julia itself might end up being -the first try that gets thrown away in a decade or two, but in the end we'll -end up with something which is both exciting and robust. - -[PyX.jl]: https://github.com/bnewbold/PyX.jl -[rust]: https://www.rust-lang.org/ - diff --git a/posts/new-server-2016.md b/posts/new-server-2016.md deleted file mode 100644 index 8f2245b..0000000 --- a/posts/new-server-2016.md +++ /dev/null @@ -1,81 +0,0 @@ -Title: New Web Server (2016) -Author: bnewbold -Date: 2016-05-01 -Tags: tech - -For the past 9 years or so ([February 2007][0] through April 2016), this -website has run reliably despite being a weird machine: a custom prototype web -application for archiving and sharing digital "artifacts" of all sorts. I -remember a bold and sparkling afternoon at a coffee house in Santa Cruz -agonizing over URL structure and refining categories to collect all the -material that would (aspirationally) accumulate here over my adult life. All -things considered I think this effort wasn't wasted: I have indeed collected -wiki notes, photos, images (distinct!), short links, one-off pages, and longer -form writing over the years. My enthusiasm for maintaining and interfacing with -idiosyncratic administrative and upload panels, however, declined rapidly. -Neither the source code web interface nor the git-backed wiki were ever -completed or used. I gave up on spam moderating the comment system pretty -quickly, never actually posted any blog entries, and only ever pecked in a -couple dozen web links and tweet-like microposts. Even the photo gallery system -became too much of a time sink to deal with after traveling. Over the years I -deployed [gitweb][] and [gitit][], and after RSS starting going out of favor I -even started using [tumblr][1] for random content posts. I was still using the -Django admin panel's "flatpages" plugin to update my contact info and project -pages right up through this spring though. - -[gitweb]: https://git-scm.com/docs/gitweb -[gitit]: http://gitit.net/ -[0]: https://git.bnewbold.net/bnewnet/commit/?id=5b31039d4c581048959dc51436f6918f29fbf9ea -[1]: http://journal.bnewbold.net/ - -While it's been amazing how simple and low maintenance running everything -(email, repositories, website, etc) has been, the thrill of being on an old and -unmaintained release of GNU/Linux (Ubuntu 10.04) has worn off, and I'm cleaning -house. This website (`bnewbold.net`) is now a simple statically generated -([pelican][]) site. I've kept `gitit`, but moved to `gitolite` (one of my -favorite pieces of software) and `cgit` for repository hosting. The server runs -Debian stable (`jessie` to start with), and SSL/TLS certificates come gratis -via the Let's Encrypt project, for which I'm very grateful. I've stuck with -[linode][] hosting for this server, though [digital ocean][] is comparable and -cheaper for setups with fewer photos and large files. Many of these components -are deployed in an automated fashion using the `ansible` deployment tool; you -could fork and edit my [infrastructure scripts][infra] to set up those -components in minutes if you like (though it would probably take an afternoon -or longer if you've never done something like this before). Not everything is -settled yet: I haven't moved email, and I'm not sure if I'll stick with -[mediagoblin][] for photo hosting. - -[pelican]: http://blog.getpelican.com/ -[linode]: https://www.linode.com -[digial ocean]: https://digitalocean.com -[infra]: http://git.bnewbold.net/infra/ -[mediagoblin]: http://mediagoblin.org - -
-
-
- - - -
-An adelie penguin (the tiny black spot) wandering off alone over the Ross Sea -ice shelf -
-
-
- -The name of my old server is `adelie` (named after a small species of penguin -that [I saw a lot of][4] in Antarctica), and it's days are limited. Though the -[`hier`][5] (filesystem layout) of most of my systems are almost identical, I'm -particularly comfortable on this one. It's seen me through a lot, more than any -other computer system, and I'll probably miss it in a weird way. - -[4]: http://bnewbold.net/photos/detail/527/ -[5]: http://man7.org/linux/man-pages/man7/hier.7.html - -The new server is `adze`; here's hoping it lives even longer! - -
-_**Note:** As initially posted this was just a stub; I re-wrote it in June -2016_ - -- cgit v1.2.3