docs/getting_started: revise & expand "Writing your own modules"

docs/getting_started.org

@@ -59,18 +59,21 @@ us know!
     - [[#reloading-your-config][Reloading your config]]
     - [[#binding-keys][Binding keys]]
   - [[#writing-your-own-modules][Writing your own modules]]
-    - [[#load-order][Load order]]
-    - [[#location][Location]]
     - [[#file-structure][File structure]]
       - [[#initel][=init.el=]]
       - [[#configel][=config.el=]]
       - [[#packagesel][=packages.el=]]
       - [[#autoloadel-or-autoloadel][=autoload/*.el= OR =autoload.el=]]
       - [[#doctorel][=doctor.el=]]
+      - [[#cliel][=cli.el=]]
+      - [[#testtest-el][=test/**/test-*.el=]]
       - [[#additional-files][Additional files]]
+    - [[#load-order][Load order]]
     - [[#flags][Flags]]
-    - [[#module-cookies][Module cookies]]
+    - [[#doom-cookies][Doom cookies]]
-    - [[#autodefs][Autodefs]]
+    - [[#if][~;;;###if~]]
+    - [[#package][~;;;###package~]]
+    - [[#autodef][~;;;###autodef~]]
   - [[#common-mistakes-when-configuring-doom-emacs][Common mistakes when configuring Doom Emacs]]
     - [[#packages-are-eagerly-loaded][Packages are eagerly loaded]]
     - [[#manual-package-management][Manual package management]]
@@ -1047,111 +1050,142 @@ also be helpful for debugging.
 + define-key!
 
 ** Writing your own modules
-*** Load order
+To create your own module you need only create a directory for it in
-Module files are loaded in a precise order:
+=~/.doom.d/modules/abc/xyz=, then add =:abc xyz= to your ~doom!~ block in
-
+=~/.doom.d/init.el= to enable it.
-1. =~/.emacs.d/early-init.el= (Emacs 27+ only)
-2. =~/.emacs.d/init.el=
-3. =$DOOMDIR/init.el=
-4. ={~/.emacs.d,$DOOMDIR}/modules/*/*/init.el=
-5. ={~/.emacs.d,$DOOMDIR}/modules/*/*/config.el=
-6. =$DOOMDIR/config.el=
-
-*** Location
-Doom searches for modules in =~/.emacs.d/modules/CATEGORY/MODULE/= and
-=$DOOMDIR/modules/CATEGORY/MODULE/=. If you have a private module with the same
-name as an included Doom module, yours will shadow the included one (as if the
-included one never existed).
 
 #+begin_quote
-Doom refers to modules in one of two formats: ~:category module~ or
+In this example, =:abc= is called the category and =xyz= is the name of the
-~category/module~.
+module. Doom refers to modules in one of two formats: =:abc xyz= and =abc/xyz=.
 #+end_quote
 
+If a private module possesses the same name as a built-in Doom module (say,
+=:lang org=), it replaces the built-in module. Use this fact to rewrite modules
+you don't agree with.
+
+Of course, an empty module isn't terribly useful, but it goes to show that nothing in a module is required. The typical module will have:
+
++ A =packages.el= to declare all the packages it will install,
++ A =config.el= to configure and load those packages,
++ And, sometimes, an =autoload.el= to store that module's functions, to be
+  loaded when they are used.
+
+These are a few exceptional examples of a well-rounded module:
+
++ [[file:/mnt/projects/conf/doom-emacs/modules/completion/company/README.org][:completion company]]
+
+The remainder of this guide will go over the technical details of a Doom module.
+
 *** File structure
-A module consists of several files, all of which are optional. They are:
+Doom recognizes a handful of special file names, none of which are required for
+a module to function. They are:
 
 #+begin_example
-modules/
+category/
-  category/
+  module/
-    module/
+    test/*.el
-      test/*.el
+    autoload/*.el
-      autoload/*.el
+    autoload.el
-      autoload.el
+    init.el
-      init.el
+    cli.el
-      config.el
+    config.el
-      packages.el
+    packages.el
-      doctor.el
+    doctor.el
 #+end_example
 
 **** =init.el=
 This file is loaded early, before anything else, but after Doom core is loaded.
+It is loaded in both interactive and non-interactive sessions (it's the only
+file, besides =cli.el= that is loaded when the =bin/doom= starts up).
 
-Use this file to:
+Do:
-
 + Configure Emacs or perform setup/teardown operations that must be set early;
   before other modules are (or this module is) loaded.
 + Reconfigure packages defined in Doom modules with ~use-package-hook!~ (as a
   last resort, when ~after!~ and hooks aren't enough).
-+ To change the behavior of ~bin/doom~.
++ Configure behavior of =bin/doom= in a way that must also apply in
+  interactive sessions.
 
-Do *not* use this file to:
+Don't:
-
++ Configure packages with ~use-package!~ or ~after!~ from here
-+ Configure packages with ~use-package!~ or ~after!~
 + Preform expensive or error-prone operations; these files are evaluated
-  whenever ~bin/doom~ is used.
+  whenever =bin/doom= is used; a fatal error in this file can make Doom
+  unbootable (but not irreversibly).
++ Define new =bin/doom= commands here. That's what =cli.el= is for.
 
 **** =config.el=
-This file is the heart of every module.
+The heart of every module. Code in this file should expect dependencies (in
+=packages.el=) to be installed and available. Use it to load and configure its
+packages.
 
-Code in this file should expect that dependencies (in =packages.el=) are
+Do:
-installed and available, but shouldn't make assumptions about what /modules/ are
++ Use ~after!~ or ~use-package!~ to configure packages.
-activated (use ~featurep!~ to detect them).
+  #+BEGIN_SRC emacs-lisp
+  ;; from modules/completion/company/config.el
+  (use-package! company  ; `use-package!' is a thin wrapper around `use-package'
+                         ; it is required that you use this in Doom's modules,
+                         ; but not required to be used in your private config.
+    :commands (company-mode global-company-mode company-complete
+               company-complete-common company-manual-begin company-grab-line)
+    :config
+    (setq company-idle-delay nil
+          company-tooltip-limit 10
+          company-dabbrev-downcase nil
+          company-dabbrev-ignore-case nil)
+     [...])
+  #+END_SRC
++ Lazy load packages with ~use-package~'s ~:defer~ property.
++ Use the ~featurep!~ macro to make some configuration conditional based on the
+  state of another module or the presence of a flag.
 
-Packages should be configured using ~after!~ or ~use-package!~:
+Don't:
-
++ Use ~package!~
-#+BEGIN_SRC emacs-lisp
++ Install packages with =package.el= or ~use-package~'s ~:ensure~ property. Doom
-;; from modules/completion/company/config.el
+  has its own package manager. That's what =packages.el= is for.
-(use-package! company
-  :commands (company-mode global-company-mode company-complete
-             company-complete-common company-manual-begin company-grab-line)
-  :config
-  (setq company-idle-delay nil
-        company-tooltip-limit 10
-        company-dabbrev-downcase nil
-        company-dabbrev-ignore-case nil)
-   [...])
-#+END_SRC
-
-#+begin_quote
-For anyone already familiar with ~use-package~, ~use-package!~ is merely a thin
-wrapper around it. It supports all the same keywords and can be used in much the
-same way.
-#+end_quote
 
 **** =packages.el=
 This file is where package declarations belong. It's also a good place to look
 if you want to see what packages a module manages (and where they are installed
 from).
 
-A =packages.el= file shouldn't contain complex logic. Mostly conditional
+Do:
-statements and ~package!~, ~disable-packages!~ or ~depend-on!~ calls. It
++ Declare packages with the ~package!~ macro
-shouldn't produce side effects and should be deterministic. Because this file
++ Disable single packages with ~package!~'s ~:disable~ property or multiple
-gets evaluated in an environment isolated from your interactive session, code
+  packages with the ~disable-packages!~ macro.
-within should make no assumptions about the current session.
++ Use the ~featurep!~ macro to make packages conditional based on the state of
+  another module or the presence of a flag.
 
-See the "[[#package-management][Package Management]]" section for details.
+Don't:
++ Configure packages here (definitely no ~use-package!~ or ~after!~ in here!).
+  This file is read in an isolated environment and will have no lasting effect.
+  The only exception is configuration targeting =straight.el=.
++ Perform expensive calculations. These files are read often and sometimes
+  multiple times.
++ Produce any side-effects, for the same reason.
+
+#+begin_quote
+The "[[#package-management][Package Management]]" section goes over the ~package!~ macro and how to deal
+with packages.
+#+end_quote
 
 **** =autoload/*.el= OR =autoload.el=
-Functions marked with an autoload cookie (~;;;###autoload~) in these files will
+These files are where you'll store functions that shouldn't be loaded until
-be lazy loaded.
+they're needed and logic that should be autoloaded (evaluated very, very early
+at startup).
 
-When you run ~bin/doom autoloads~, Doom scans these files to populate autoload file
+This is all made possible thanks to these autoload cookie: ~;;;###autoload~.
-in =~/.emacs.d/.local/autoloads.el=, which will tell Emacs where to find these
+Placing this on top of a lisp form will do one of two things:
-functions when they are called.
+
+1. Add a ~autoload~ call to Doom's autoload file (found in
+   =~/.emacs.d/.local/autoloads.el=, which is read very early in the startup
+   process).
+2. Or copy that lisp form to Doom's autoload file verbatim (usually the case for
+   anything other then ~def*~ forms, like ~defun~ or ~defmacro~).
+
+Doom's autoload file is generated by scanning these files when you execute ~doom
+sync~.
 
 For example:
-
 #+BEGIN_SRC emacs-lisp
 ;; from modules/lang/org/autoload/org.el
 ;;;###autoload
@@ -1167,14 +1201,19 @@ For example:
 #+END_SRC
 
 **** =doctor.el=
-This file is used by ~make doctor~, and should test for all that module's
+When you execute ~doom doctor~, this file defines a series of tests for the
-dependencies. If it is missing one, it should use the ~warn!~, ~error!~ and
+module. These should perform sanity checks on the environment, such as:
-~explain!~ macros to inform the user why it's a problem and, ideally, a way to
+
-fix it.
++ Check if the module's dependencies are satisfied,
++ Warn if any of the enabled flags are incompatible,
++ Check if the system has any issues that may interfere with the operation of
+  this module.
+
+Use the ~warn!~, ~error!~ and ~explain!~ macros to communicate issues to the
+user and, ideally, explain how to fix them.
 
 For example, the ~:lang cc~ module's doctor checks to see if the irony server is
 installed:
-
 #+BEGIN_SRC emacs-lisp
 ;; from lang/cc/doctor.el
 (require 'irony)
@@ -1182,32 +1221,69 @@ installed:
   (warn! "Irony server isn't installed. Run M-x irony-install-server"))
 #+END_SRC
 
+**** TODO =cli.el=
+This file is read when =bin/doom= starts up. Use it to define your own CLI
+commands or reconfigure existing ones.
+
+**** TODO =test/**/test-*.el=
+Doom's unit tests go here. More information on them to come...
+
 **** Additional files
-Sometimes, it is preferable that a module's config.el file be split up into
+Any files beyond the ones I have already named are not given special treatment.
-multiple files. The convention is to name these additional files with a leading
+They must be loaded manually to be loaded at all. In this way modules can be
-=+=, e.g. =modules/feature/version-control/+git.el=.
+organized in any way you wish. Still, there is one convention that has emerged
+in Doom's community that you may choose to adopt: extra files in the root of the
+module are prefixed with a plus, e.g. =+extra.el=. There is no syntactical or
+functional significance to this convention.
 
-There is no syntactical or functional significance to this convention.
+These can be loaded with the ~load!~ macro, which will load an elisp file
-Directories do not have to follow this convention, nor do files within those
+relative to the file it's used from. e.g.
-directories.
-
-These additional files are *not* loaded automatically. You will need to use the
-~load!~ macro to do so:
 
 #+BEGIN_SRC emacs-lisp
-;; from modules/feature/version-control/config.el
+;; Omitting the file extension allows Emacs to load the byte-compiled version,
-(load! +git)
+;; if it is available:
+(load! "+git")   ; loads ./+git.el
 #+END_SRC
 
-The ~load!~ macro will try to load a =+git.el= relative to the current file.
+This can be useful for splitting up your configuration into multiple files,
+saving you the hassle of creating multiple modules.
+
+*** Load order
+A module's files have a precise load-order, which differs slightly depending on
+what kind of session it is. Doom has three types of sessions:
+
++ Interactive session :: the typical session you open when you intend to use
+  Emacs (e.g. for text editing). This loads the most, because you will likely be
+  using a lot of it.
++ Batch session :: this is a non-interactive session, loaded when you execute
+  Emacs commands on the command line with no UI, e.g. ~emacs --batch --eval
+  '(message "Hello world")'~.
+
+  The expectation for these sessions is that it should quickly spin up, run the
+  command then quit, therefore very little is loaded in this session.
++ CLI session :: this is the same as a batch session /except/ it is what starts
+  up when you run any =bin/doom= command.
+
+With that out of the way, here is the load order of Doom's most important files:
+
+| File                                        | Interactive | Batch | CLI |
+|---------------------------------------------+-------------+-------+-----|
+| ~/.emacs.d/early-init.el (Emacs 27+ only)   | yes         | no    | no  |
+| ~/.emacs.d/init.el                          | yes         | no    | no  |
+| $DOOMDIR/init.el                            | yes         | yes   | yes |
+| {~/.emacs.d,$DOOMDIR}/modules/*/*/init.el   | yes         | yes   | yes |
+| $DOOMDIR/cli.el                             | no          | no    | yes |
+| {~/.emacs.d,$DOOMDIR}/modules/*/*/cli.el    | no          | no    | yes |
+| {~/.emacs.d,$DOOMDIR}/modules/*/*/config.el | yes         | no    | no  |
+| $DOOMDIR/config.el                          | yes         | no    | no  |
 
 *** Flags
-A module flag is an arbitrary symbol. By convention, these symbols are prefixed
+A module's flag is an arbitrary symbol. By convention, these symbols are
-with a ~+~ or a ~-~, to respectively denote the addition or removal of a
+prefixed with a ~+~ or a ~-~ to denote the addition or removal of a feature,
-feature. There is no functional significance to this notation.
+respectively. There is no functional significance to this notation.
 
-A module may choose to interpret flags however it likes. They can be tested for
+A module may choose to interpret flags however it wishes, and can be tested for
-with the ~featurep!~ macro:
+using the ~featurep!~ macro:
 
 #+BEGIN_SRC elisp
 ;; Has the current module been enabled with the +my-feature flag?
@@ -1217,39 +1293,72 @@ with the ~featurep!~ macro:
 (when (featurep! :lang python +lsp) ...)
 #+END_SRC
 
-*** Module cookies
+Use this fact to make aspects of a module conditional. e.g. Prevent company
-A special syntax exists called module cookies. Like autoload cookies
+plugins from loading if the =:completion company= module isn't enabled.
-(~;;;###autoload~), module files may have ~;;;###if FORM~ at or near the top of
-the file. FORM is read determine whether or not to ignore this file when
-scanning it for autoloads (~doom sync~) or byte-compiling it (~doom compile~).
 
-Use this to prevent errors that may occur if that file contains (for example)
+*** Doom cookies
-calls to functions that won't exist if a certain feature isn't available to that
+Autoload cookies were mentioned [[*=autoload/*.el= OR =autoload.el=][earlier]]. A couple more exist that are specific
-module, e.g.
+to Doom Emacs. This section will go over what they do and how to use them.
 
+*** ~;;;###if~
+Any file in a module can have a ~;;;###if FORM~ cookie at or near the top of the
+file (must be within the first 256 bytes of the file). =FORM= is evaluated to
+determine whether or not to include this file for autoloads scanning (on ~doom
+sync~) or byte-compilation (on ~doom compile~).
+
+i.e. if =FORM= returns ~nil~, Doom will neither index its ~;;;###autoload~
+cookies nor byte-compile the file.
+
+Use this to prevent errors that would occur if certain conditions aren't met.
+For example, say =file.el= is using a certain function that won't be available
+if the containing module wasn't enabled with a particular flag. We could safe
+guard against this with:
 #+BEGIN_SRC emacs-lisp
-;;;###if (featurep! +lsp)
+;;;###if (featurep! +particular-flag)
 #+END_SRC
 
+This will prevent errors at compile time or if/when that file is loaded.
+
+Another example, this time contingent on =so-long= *not* being present:
 #+BEGIN_SRC emacs-lisp
 ;;;###if (not (locate-library "so-long"))
 #+END_SRC
 
-Remember that these run in a limited, non-interactive sub-session, so do not
+#+begin_quote
-call anything that wouldn't be available in a Doom session without any modules
+Keep in mind that =FORM= runs in a limited, non-interactive sub-session. I don't
-enabled.
+recommend doing anything expensive or especially complicated in them.
+#+end_quote
 
-*** Autodefs
+*** ~;;;###package~
-An autodef is a special kind of autoloaded function or macro which Doom
+This cookie exists solely to assist the ~doom/help-packages~ command. This
-guarantees will always be defined, whether or not its containing module is
+command shows you documentation about packages in the Emacs ecosystem, including
-enabled (but will no-op without evaluating its arguments when it is disabled).
+the ones that are installed. It also lists a) all the modules that install said
+package and b) all the places it is configured.
+
+It accomplishes A by scanning for at ~package!~ declarations for that package,
+but it accomplishes B by scanning for:
+
++ ~after!~ calls
++ ~use-package!~ or ~use-package~ calls
++ and ~;;;###package X~ cookies, where X is the name of the package
+
+Use it to let ~doom/help-packages~ know where to find config for packages where
+no ~after!~ or ~use-package!~ call is involved.
+
+*** ~;;;###autodef~
+An autodef is a special kind of autoloaded function (or macro) which Doom
+guarantees will /always/ be defined, whether or not its containing module is
+enabled (but will no-op if it is disabled).
+
+#+begin_quote
+If the containing module is disabled the definition is replaced with a macro
+that does not process its arguments, so it is a zero-cost abstraction.
+#+end_quote
 
 You can browse the available autodefs in your current session with ~M-x
 doom/help-autodefs~ (=SPC h d u= or =C-h d u=).
 
-What distinguishes an autodef from a regular autoload is the ~;;;###autodef~
+An autodef cookie is used in exactly the same way as the autoload cookie:
-cookie:
-
 #+BEGIN_SRC elisp
 ;;;###autodef
 (defun set-something! (value)
@@ -1259,11 +1368,13 @@ cookie:
 An example would be the ~set-company-backend!~ function that the =:completion
 company= module exposes. It lets you register company completion backends with
 certain major modes. For instance:
-
 #+BEGIN_SRC emacs-lisp
 (set-company-backend! 'python-mode '(company-anaconda))
 #+END_SRC
 
+And if =:completion company= is disabled, this call and its arguments are left
+unprocessed and ignored.
+
 ** Common mistakes when configuring Doom Emacs
 Having helped many users configure Doom, I've spotted a few recurring oversights
 that I will list here, in the hopes that it will help you avoid the same