Merge pull request #2 from bnewbold/meta-dep

DEP: The Dat Enhancement Proposal Process (for Draft status)

4 files changed, 477 insertions, 1 deletions

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..317f05f
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,9 @@
+.DS_Store
+*~
+*.swp
+*.bak
+*.bkp
+.*
+
+# Don't ignore this file itself
+!.gitignore
diff --git a/0000-template.md b/0000-template.md
new file mode 100644
index 0000000..bea1fae
--- /dev/null
+++ b/0000-template.md
@@ -0,0 +1,72 @@
+
+Title: **DEP-0000: My Proposal**
+
+Short Name: `0000-my-proposal`
+
+Type: (Standard, Process, or Informative)
+
+Status: Undefined (as of YYYY-MM-DD)
+
+Github PR: (add HTTPS link here after PR is opened)
+
+Authors: TBD
+
+
+# Summary
+[summary]: #summary
+
+One paragraph explanation of what this is about.
+
+
+# Motivation
+[motivation]: #motivation
+
+Why are we doing this? What use cases does it support? What is the expected outcome?
+
+
+# Usage Documentation
+[usage-documentation]: #usage-documentation
+
+Document new features or processes as if this proposal has already been accepted and implemented. This section should introduce new concepts and terms, describe use cases or situations, and provide examples. If a user-facing feature is added, this should also include user-accessible documentation.
+
+
+# Reference Documentation
+[reference-documentation]: #reference-documentation
+
+This section is a more formal description of the proposal, written as if it was a sub-section of the standard (for techincal proposals) or a formal process or "fine print" for process proposals.
+
+It should be unambiguous and cover all known corner-cases. Formal language (such as protobuf schemas or binary header diagrams) are appropriate here.
+
+"Standards" proposals should include a section addressing security and privacy implications.
+
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+Why should we *not* do this?
+
+
+# Rationale and alternatives
+[alternatives]: #alternatives
+
+- Why is this design the best in the space of possible designs?
+- What other designs have been considered and what is the rationale for not choosing them?
+- What is the impact of not doing this?
+
+
+# Unresolved questions
+[unresolved]: #unresolved-questions
+
+- What parts of the design do you expect to resolve through the DEP consensus process before this gets merged?
+- What parts of the design do you expect to resolve through implementation and code review, or are left to independent library or application developers?
+- What related issues do you consider out of scope for this DEP that could be addressed in the future independently of the solution that comes out of this DEP?
+
+
+# Changelog
+[changelog]: #changelog
+
+A brief statemnt about current status can go here, follow by a list of dates
+when the status line of this DEP changed (in most-recent-last order).
+
+- YYYY-MM-DD: First complete draft submitted for review
+
diff --git a/README.md b/README.md
index db7fe6d..179e136 100644
--- a/README.md
+++ b/README.md
@@ -1,3 +1,44 @@
+
 # Dat Enhancement Proposals
 
-Dat Enhancement Proposals. Contains all specs for the Dat protocol, including drafts.
+[![#dat IRC channel on freenode](https://img.shields.io/badge/irc%20channel-%23dat%20on%20freenode-blue.svg)](http://webchat.freenode.net/?channels=dat)
+[![datproject/discussions](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/datproject/discussions?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[![docs](https://img.shields.io/badge/Dat%20Project-Docs-green.svg)](http://docs.dat-data.com)
+
+This repository contains a series of "Dat Enhancement Proposal" (DEP)
+documents, part of the dat protocol development and standardization process.
+
+Accepted proposals will be viewable at [datprotocol.com](https://datprotocol.com).  New (draft) proposals and discussion can be viewed on [github][github-deps] under "Pull requests".
+
+These documents might be interesting reading to anybody wanting to learn more
+about protocol nitty gritties or the consensus process, but the documentation
+at [docs.datproject.org][docs] is specifically written with end-users and
+application developers in mind.
+
+[github-deps]: https://github.com/datprotocol/DEPs/
+[docs]: https://docs.datproject.com
+
+## The Process
+
+TL;DR: write up an DEP markdown file; submit a github PR; revise per review;
+community will either accept (merges PR) or not (closes PR). See the ["DEP
+Process"][dep-0001] meta-DEP for details.
+
+For now this process is as simple and informal; as the number of dat developers
+and users grows, it may become more formal. Precedents and inspiration include:
+
+- [Bittorrent Enhancement Proposals (BEP)](http://bittorrent.org/beps/bep_0001.html)
+- [Rust RFC Process](https://github.com/rust-lang/rfcs)
+- [Python Enhancement Proposal](https://www.python.org/dev/peps/pep-0001/)
+
+[dep-0001]: ./proposals/0001-dep-process.md
+
+## Fine Print
+
+The contents of this repository are all under a CC-BY (Creative Commons
+Attribution 4.0) license unless otherwise noted.
+
+Contributors and discussion participants are expected to abide by the Dat
+Project [Code of Conduct][coc].
+
+[coc]: https://github.com/datproject/Code-of-Conduct
diff --git a/proposals/0001-dep-process.md b/proposals/0001-dep-process.md
new file mode 100644
index 0000000..29b557e
--- /dev/null
+++ b/proposals/0001-dep-process.md
@@ -0,0 +1,354 @@
+
+Title: **DEP-0001: The Dat Enhancement Proposal Process**
+
+Short Name: `0001-dep-process`
+
+Type: Process
+
+Status: Draft (as of 2018-XX-YY)
+
+Github PR: [Draft](https://github.com/datprotocol/DEPs/pull/2)
+
+Authors: [Dat Protocol Working Group][wg]:
+[Tara Vancil](https://github.com/taravancil),
+[Paul Frazee](https://github.com/pfrazee),
+[Mathias Buus](https://github.com/mafintosh),
+[Karissa McKelvey](https://github.com/karissa),
+[Joe Hand](https://github.com/joehand),
+[Danielle Robinson](https://github.com/daniellecrobinson),
+[Bryan Newbold](https://github.com/bnewbold)
+
+[wg]: https://github.com/datprotocol/working-group
+
+
+# Summary
+[summary]: #summary
+
+The Dat Enhancement Proposal ("DEP") process is how the Dat open source
+community comes to (distributed) consensus around technical protocol
+enhancements and organizational process.
+
+
+# Motivation
+[motivation]: #motivation
+
+The Dat protocol is still a living standard. A transparent process is needed
+for community members to understand what changes are in the pipeline and how
+new ideas might come to fruition.
+
+The core protocol is being used and extended by several projects with differing
+priorities and use cases. At the same time, lead developer time is very scarce.
+There is a need to parallelize design and implementation work between projects,
+which requires better coordination (process) and communication of technical
+details (standards). There is also an increasing need to be legible to and
+accessible to parties outside the existing Dat ecosystem.
+
+With growing use, the logistics of rolling out protocol changes and
+backwards-incompatible changes becomes more difficult, but at the same time
+more important to coordinate smoothly. Planning requires clear communication of
+change ahead of time.
+
+A public DEP process is expected to improve coordination and planning by
+setting clear expectations for documentation of protocol changes and
+extensions. The technical quality of the protocol itself should be improved by
+increasing the number of people who can view and understand proposals at each
+step of the process. The barrier to entry for independent implementations
+should be lower, allowing new technical and user communities to adopt the
+protocol. The overall development and decision making process should be more
+transparent, accessible, and scalable to a growing group of application
+developers and end users
+
+
+# How To Submit a Proposal
+[submit]: #submit
+
+As a first step, before drafting a DEP or implementing experimental new
+protocol features, it's helpful to informally pitch your idea to see if others
+in the community are already thinking something similar, or have discussed the
+same idea in the past. This discussion could happen over chat, Github issues,
+blog posts, or other channels. If you can recruit collaborators and work out
+some of the details, all the better. This period could be called **pre-DEP**.
+
+Once your idea has been flushed out, the process for proposing and debating a
+new DEP is:
+
+1. Use git to fork the [datprotocol/deps](https://github.com/datprotocol/deps)
+   repository
+1. Copy `0000-template.md` to `proposals/0000-my-proposal.md` (don't chose the
+   "next" number, use zero; `my-proposal` should be a stub identifier for the
+   proposal)
+1. Fill in the DEP template. All proposals should have a Type and Status (see
+   below for details). Feel free to tweak or expand the structure (headers,
+   content) of the document to fit your needs, but your proposal should be
+   "complete" before submission.
+1. You can create an informal WIP (work in progress) PR (pull-request)
+   whenever you like for early feedback and discussion, but there is no
+   expectation that your proposal will be given detailed review until it is
+   complete.
+1. When you are ready, submit your complete proposal for review (this could be
+   opening a PR or removing WIP status from an existing one), with the intent
+   of being accepted with "Draft" status.
+   An editor (somebody who is an owner of the DEPs repository) will look over
+   your proposal for completeness; if acceptable, they will assign one or more
+   reviewers.
+   At this stage, two outcomes are the most likely: your proposal
+   is merged with "Draft" status, or declined.  This first stage of the review
+   is expected to take **3 weeks at most** from when reviewers were assigned.
+   It is appropriate to propose specific community members to review your
+   proposal. The submitter can withdraw a proposal at any time. If accepted, a
+   DEP number will be assigned and the PR merged. If declined, reviewers may
+   give feedback and/or invite proposers to "significantly revise and resubmit".
+1. While in draft status, proposals can be experimented with. Small corrections
+   and clarifications can be submitted by PR expect to be merged quickly if
+   they are reasonable and don't change the broad behavior or semantics of the
+   proposal; larger changes should be re-submitted as Superseding proposals.
+1. When it seems appropriate, a PR can be submitted to upgrade the status of a
+   "Draft" to "Active". At this time a final review will take place, with the
+   outcome being that a proposal stays "Draft" or becomes "Active".
+   This review period is shorter (**2 weeks at most**), as everybody is
+   expected to be more familiar with the proposal at this point.
+   Small changes to the DEP can be included, but it's expected that
+   this is in broad strokes the same proposal that was reviewed earlier (if
+   not, a new "Draft" should be proposed that Supersedes).
+1. Small tweaks (grammar, clarifications) to a merged DEP can take place as
+   regular Github PRs; revisiting or significantly revising should take place
+   as a new DEP. Draft, Process, and Informational DEPs have a lower bar for
+   evolution over time via direct PR.
+
+# Details
+[reference-documentation]: #reference-documentation
+
+## Types and Statuses
+[types]: #types
+
+DEPs should have a type:
+
+* **Standard** for technical changes to the protocol, on-disk formats, or
+  public APIs. These are intended to be *proscriptive*, and to clearly
+  delineate which features and behaviors are mandatory or optional.
+* **Process** for formalizing community processes or other (technical or
+  non-technical) decisions. For example, a security vulnerability reporting
+  policy, a process for handling conflicts of interest, or procedures for
+  mentoring new developers.
+* **Informative** for describing conventions, design patterns, existing norms,
+  special considerations, etc.
+
+The status of a DEP can be:
+
+* **Pre-Merge**: a well-formed DEP has been written and a PR opened. The
+  "Status" line can list Draft when in this state.
+* **Draft**: PR has been merged and a number assigned, but additional time is
+  needed for deeper discussion or more implementation before being "normative"
+  and expected for implementation. It is acceptable to have "competing"
+  proposals in this state at the same time.
+* **Active**: adopted or intended for implementation in mainline libraries and
+  clients as appropriate. Again, DEPs should clarify which aspects of
+  themselves are optional or required for well-behaved clients.
+* **Closed**: either consensus was against, a decision was postponed, or the
+  authors withdrew their proposal. This could apply to any of: a proposal PR
+  that was never merged, a merged Draft (which was never Active), or an Active
+  DEP which there is now consensus against without a specific new DEP to
+  replace it.
+* **Superseded**: a formerly Active DEP has been made obsolete by a new
+  Active DEP; the new DEP should specify specific old DEPs that it would
+  supersede.
+
+## Content and Structure
+[content]: #content
+
+A changelog should be kept in the DEP itself giving the date of any changes of
+status.
+
+A template file is provided, but sections can be added or removed as
+appropriate for a specific DEP.
+
+The DEP text itself should be permissively licensed; the convention is to use
+the Creative Commons Attribution License (CC-BY), with attribution to the major
+contributing authors listed.
+
+For appropriate DEPs (including *all* Standards DEPs), authors should
+explicitly consider and note impacts on:
+
+* Privacy and User Rights. Consider reading IETF [RFC 6973] ("Privacy
+  Considerations for Internet Protocols") and [RFC 8280] ("Research into Human
+  Rights Protocol Considerations")
+* Backwards compatibility of on-disk archives and older network clients. If a
+  backwards-incompatible change is proposed, a migration plan should be
+  sketched out in the proposal
+* Security
+
+[RFC-6973]: https://tools.ietf.org/html/rfc6973
+[RFC-8280]: https://tools.ietf.org/html/rfc8280
+
+
+## Acceptance Criteria
+[criteria]: #criteria
+
+The criteria for a proposal being accepted as a Draft are, at a minimum, that
+the proposal is complete, understandable, unambiguous, and relevant. There is a
+good faith assumption that the submitter believes that the proposal could
+actually be adopted and put to beneficial use. An editor may screen proposals
+before passing on to the group for review.
+
+For Standards and Process DEPs, Draft proposals should be specific enough that
+they can be prototyped and experimented with (eg, in a pilot program or test
+network), but it isn't expected that all details have been worked out. Working
+code is helpful but not required.
+
+For a Draft to migrate to Active, there is an expectation that the proposal has
+been demonstrated (eg, in working code, though not necessarily in "official"
+libraries yet), that the proposal will be the "new normal" and expected
+behavior going forward, and that the chance of unforeseen issues arising during
+complete adoption is low.
+
+
+## Decision Making Process
+[power]: #power
+
+There exists a [Protocol Working Group (WG)][wg] (WG) which makes DEP status
+decisions; see the Github repository for a list of current members and the
+governance process.
+
+By no means should working group members be the only people reviewing or giving
+feedback on proposals.
+
+When deciding on Draft status, at least one WG member must review the entire
+proposal in detail, give feedback, and give informed approval. If no detailed
+review takes place in the fixed time window, the default is to close (reject)
+until a member is willing to commit to review. Any WG member can request
+revisions or clarifications (blocking acceptance until addressed), and any
+member can block. In other words, consensus requires that at least one member
+actively approve, and any member can block, but it isn't required to have every
+member review and actively given an opinion. This is referred to as the "loose
+consensus" model.
+
+For Active status, the expectation is that all working group members will
+review the proposal and actively participate in consensus. In the event that
+not all members can participate, the default is again negative. Any member can
+block. This is referred to as the "complete consensus" model.
+
+For all other status changes, at least one WG member must vouch for or approve
+the change ("loose consensus"). If there is unambiguous consensus (or, eg, a
+DEP is documenting already adopted practice), a DEP can move directly to Active
+status (following the "complete consensus" process).
+
+In all cases, if there is a deadlock (a block can not be overcome after
+further discussion), there is an option to override the block by a vote. The
+details of this process are left to description in the Working Group
+repository, and are expected to be used only in exceptional cases (eg, are not
+invoked by default if a member blocks).
+
+Proposals are expected to be open for at least three days (72 hours) for
+comment (and longer to accommodate special circumstances, like holidays). Vetoes
+(blocks) can happen up to a week after initially being submitted for review,
+which might be retroactive if the proposal was accepted (by other WG members)
+very quickly.
+
+
+# Rationale
+[rationale]: #rationale
+
+In the design space of RFC processes, there are two decision points determining
+the formality and maturity of accepted standards. The first is, does draft
+status mean *could* be implemented, or *has* been implemented? We chose
+"could". Secondly, for active (or "final") status, is the proposal *expected*
+to be dominant in the wild, or *is it already* dominant in the wild? We chose
+"expected". In both cases we are emphasizing clarification and stabilization of
+new ideas, as opposed to enforcing interoperability of competing formulations
+of the same idea. In short, we expect DEPs to lead (rather than tail)
+implementation.
+
+Time limits and default outcomes are used to prevent proposals could get stuck
+in an ambiguous indefinite state anywhere along the process. "Draft" status is
+considered a stable state to linger in.
+
+Setting expectations for "completeness" of proposals, having an editor quickly
+skim proposals before jumping in to a full review, and acknowledging an explicit
+"revise and re-submit" workflow are all attempts to head off the situation of
+partial proposals being submitted and then significantly revised, which places
+extra (time) burden on reviewers.
+
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+There are already multiple sources of technical documentation: the Dat
+[protocol website][proto-website], the Dat [whitepaper][whitepaper], Dat
+website [documentation section][docs], the [discussion repo][discussion-repo]
+issues, and the [datprotocol github group][datproto-group] (containing, eg, the
+`dat.json` repo/spec). Without consensus and consolidation, this would be "yet
+another" place to look.
+
+[proto-website]: https://www.datprotocol.com/
+[whitepaper]: https://github.com/datproject/docs/blob/master/papers/dat-paper.md
+[docs]: https://docs.datproject.org/
+[datproto-group]: https://github.com/datprotocol
+[discussion-repo]: https://github.com/datproject/discussions/issues
+
+
+# Background and References
+[references]: #references
+
+The following standards processes were referenced and considered while
+designing the DEP process:
+
+* **BitTorrent Enhancement Process** as described in [BEP 1][bep-1]. The
+  Bittorrent protocol has a lot of technical similarities to Dat, and as a
+  single protocol family (not a language or full-stack system) is one of the
+  most similar in scope. The de facto BEP model is that Drafts are very stable
+  and widely adopted; only the most universal core components are Final. The
+  DEP process bases it's type categories on the BEP process. There is a single
+  editor and an explicit BDFL in the BEP process.
+* The **[Rust Language RFC Process][rust-rfc]** is relatively new, but has had
+  a huge volume of proposals, rivaling even the IETF. The process is relatively
+  lightweight and happens entirely on Github; it is the most similar to the DEP
+  process proposed here, in terms of Draft/Active distinction. Rust has strong
+  organizational backing with defined leadership roles; proposals are reviewed
+  by specific sub-teams.
+* **[IETF RFC Process][ietf]**: perhaps the oldest and best known RFC process,
+  under the motto of "rough consensus and working code". The process is very
+  bespoke (involving custom file formats and software) and heavy on process
+  (with working groups and in-person meetings).
+* **[XMPP Standards Process][xmpp]**: has the interesting sub-pattern of
+  regularly updated (annual) standards. XMPP is also a protocol, like
+  Bittorrent. The protocol was designed for easy extension, and at various
+  points has seen adoption, extension, and pressure from powerful entities.
+* **Python Enhancement Process** documented in [PEP 1][pep-1]. PEPs are
+  relatively broad in scope (they often speak to process and organizational
+  dynamics), and are widely cited directly by name. Proposals are usually
+  debated in great detail on mailing lists before being proposed. Python has a
+  BDFL (benevolent dictator for life) who has final say over proposals, though
+  he sometimes delegates to deputies.
+* The **W3C** is a paid membership organization which, like the IETF, is made
+  up of entities large and small, for-profit and altruistic, with decent
+  regional diversity. W3C standards are often rather large and verbose
+  documents, and tend to tail (rather than lead) implementation.
+
+
+[bep-1]: http://bittorrent.org/beps/bep_0001.html
+[rust-rfc]: https://github.com/rust-lang/rfcs
+[xmpp]: https://xmpp.org/about/standards-process.html
+[ietf]: https://www.ietf.org/about/process-docs.html
+[pep-1]: https://www.python.org/dev/peps/pep-0001/
+
+
+# Unresolved questions
+[unresolved]: #unresolved-questions
+
+How mutable should "Draft" Standards DEPs be over time? What about
+Informational DEPs? Should there be an additional status ("Living"?) for DEPs
+that are expected to evolve, or is this against the whole philosophy of having
+specific stable documents to reference? This is expected to be clarified while
+this DEP itself is in Draft status.
+
+Does "Active" status mean that implementation is *mandatory*, and that features
+*must* be implemented unless they are explicitly optional? How would this
+expectation be enforced for third-party software? This is expected to be
+clarified when concrete examples arise.
+
+# Changelog
+[changelog]: #changelog
+
+- 2018-01-24: DEP process and governance model is discussed by Working Group.
+  First full draft of DEP-0001 submitted for review.
+