BREAKING CHANGE: this changes Doom's CLI framework in subtle ways, which
is listed in greater detail below. If you've never extended Doom's CLI,
then this won't affect you, but otherwise it'd be recommended you read
on below.
This commit focuses on the CLI framework itself and backports some
foundational changes to its DSL and how it resolves command line
arguments to CLIs, validates input, displays documentation, and persists
state across sessions -- and more. This is done in preparation for the
final stretch towarding completing the CLI rewrite (see #4273).
This is also an effort to generalize Doom's CLI (both its framework and
bin/doom), to increase it versatility and make it a viable dev tool for
other Doom projects (on our Github org) and beyond.
However, there is a *lot* to cover so I'll try to be brief:
- Refactor: generalize Doom's CLI framework by moving all bin/doom
specific configuration/commands out of core-cli into bin/doom. This
makes it easier to use bin/doom as a project-agnostic development
tool (or for users to write their own).
- Refactor: change the namespace for CLI variables/functions from
doom-cli-X to doom-X.
- Fix: subcommands being mistaken as arguments. "doom make index" will
resolve to (defcli! (doom make index)) if it exists,
otherwise (defcli! (doom make)) with "index" as an argument. Before
this, it would resolve to the latter no matter what. &rest can
override this; with (defcli! (doom make) (&rest args)), (defcli! (doom
make index)) will never be invoked.
- Refactor!: redesign our output library (was core/autoload/output.el,
is now core/autoload/print.el), and how our CLI framework buffers and
logs output, and now merges logs across (exit! ...) restarts.
- Feat: add support for :before and :after pseudo commands. E.g.
(defcli! (:before doom help) () ...)
(defcli! (:after doom sync) () ...)
Caveat: unlike advice, only one of each can be defined per-command.
- Feat: option arguments now have rudimentary type validation (see
`doom-cli-option-arg-types`). E.g.
(defcli! (doom foo) ((foo ("--foo" num))) ...)
If NUM is not a numeric, it will throw a validation error.
Any type that isn't in `doom-cli-option-arg-types` will be treated as a
wildcard string type. `num` can also be replaced with a specification,
e.g. "HOST[:PORT]", and can be formatted by using symbol quotes:
"`HOST'[:`PORT']".
- Feat: it is no longer required that options *immediately* follow the command
that defines them (but it must be somewhere after it, not before). E.g.
With:
(defcli! (:before doom foo) ((foo ("--foo"))) ...)
(defcli! (doom foo baz) () ...)
Before:
FAIL: doom --foo foo baz
GOOD: doom foo --foo baz
FAIL: doom foo baz --foo
After:
FAIL: doom --foo foo baz
GOOD: doom foo --foo baz
GOOD: doom foo baz --foo
- Refactor: CLI session state is now kept in a doom-cli-context struct (which
can be bound to a CLI-local variable with &context in the arglist):
(defcli! (doom sync) (&context context)
(print! "Command: " (doom-cli-context-command context)))
These contexts are persisted across sessions (when restarted). This is
necessary to support seamless script restarting (i.e. execve
emulation) in post-3.0.
- Feat: Doom's CLI framework now understands "--". Everything after it will be
treated as regular arguments, instead of sub-commands or options.
- Refactor!: the semantics of &rest for CLIs has changed. It used to be "all
extra literal, non-option arguments". It now means *all* unprocessed
arguments, and its use will suppress "unrecognized option" errors, and
tells the framework not to process any further subcommands. Use &args
if you just want "all literal arguments following this command".
- Feat: add new auxiliary keywords for CLI arglists: &context, &multiple,
&flags, &args, &stdin, &whole, and &cli.
- &context SYM: binds the currently running context to SYM (a
`doom-cli-context` struct). Helpful for introspection or passing
along state when calling subcommands by hand (with `call!`).
- &stdin SYM: SYM will be bound to a string containing any input piped
into the running script, or nil if none. Use
`doom-cli-context-pipe-p` to detect whether the script has been
piped into or out of.
- &multiple OPTIONS...: allows all following OPTIONS to be repeated. E.g. "foo
-x a -x b -x c" will pass (list ("-x" . "a") ("-x" . "b") ("-x" .
"c")) as -x's value.
- &flags OPTIONS...: All options after "&flags" get an implicit --no-* switch
and cannot accept arguments. Will be set to :yes or :no depending on which flag is
provided, and nil if the flag isn't provided. Otherwise, a default
value can be specified in that options' arglist. E.g.
(defcli! (doom foo) (&flags (foo ("--foo" :no))) ...)
When called, this command sets FOO to :yes if --foo, :no if --no-foo, and
defaults to :no otherwise.
- &args SYM: this replaces what &rest used to be; it binds to SYM a
list of all unprocessed (non-option) arguments.
- &rest SYM: now binds SYM to a list of all unprocessed arguments, including
options. This also suppresses "unrecognized option" errors, but will render
any sub-commands inaccessible. E.g.
(defcli! (doom make) (&rest rest) ...)
;; These are now inaccessible!
(defcli! (doom make foo) (&rest rest) ...)
(defcli! (doom make bar) (&rest rest) ...)
- &cli SYM: binds SYM to the currently running `doom-cli` struct. Can also be
obtained via `(doom-cli-get (doom-cli-context-command context))`. Possibly
useful for introspection.
- feat: add defobsolete! macro for quickly defining obsolete commands.
- feat: add defalias! macro for quickly defining alias commands.
- feat: add defautoload! macro for defining an autoloaded command (won't
be loaded until it is called for).
- refactor!: rename defcligroup! to defgroup! for consistency.
- fix: CLIs will now recursively inherit plist properties from parent
defcli-group!'s (but will stack :prefix).
- refactor!: remove obsolete 'doom update':
- refactor!: further generalize 'doom ci'
- In an effort to generalize 'doom ci' (so other Doom--or
non-doom--projects can use it), all its subcommands have been
changed to operate on the current working directory's repo instead
of $EMACSDIR.
- Doom-specific CI configuration was moved to .github/ci.el.
- All 'doom ci' commands will now preload one of \$CURRENT_REPO_ROOT/ci.el or
\$DOOMDIR/ci.el before executing.
- refactor!: changed 'doom env'
- 'doom env {-c,--clear}' is now 'doom env {clear,c}'
- -r/--reject and -a/--allow may now be specified multiple times
- refactor!: rewrote CLI help framework and error handling to be more
sophisticated and detailed.
- feat: can now initiate $PAGER on output with (exit! :pager) (or use
:pager? to only invoke pager is output is longer than the terminal is
tall).
- refactor!: changed semantics+conventions for global bin/doom options
- Single-character global options are now uppercased, to distinguish them from
local options:
- -d (for debug mode) is now -D
- -y (to suppress prompts) is now -!
- -l (to load elisp) is now -L
- -h (short for --help) is now -?
- Replace --yes/-y switches with --force/-!
- -L/--load FILE: now silently ignores file errors.
- Add --strict-load FILE: does the same as -L/--load, but throws an error if
FILE does not exist/is unreadable.
- Add -E/--eval FORM: evaluates arbitrary lisp before commands are processed.
- -L/--load, --strict-load, and -E/--eval can now be used multiple times in
one command.
- Add --pager COMMAND to specify an explicit pager. Will also obey
$DOOMPAGER envvar. Does not obey $PAGER.
- Fix#3746: which was likely caused by the generated post-script overwriting
the old mid-execution. By salting the postscript filenames (with both an
overarching session ID and a step counter).
- Docs: document websites, environment variables, and exit codes in
'doom --help'
- Feat: add imenu support for def{cli,alias,obsolete}!
Ref: #4273Fix: #3746Fix: #3844
By deleting straight, the latest pinned version will be reinstalled on
the second part of the 'doom upgrade' process.
This is a clumsy fix for a catch 22 where using straight to update
straight would break due to backwards incompatibilities (see #4950).
This will do while we wait for the CLI rewrite.
The upgrade process leaves a left over ref:
```
$ git branch
..
_upgrade_HEAD
..
```
This commit deletes that.
Signed-off-by: Rudi Grinberg <me@rgrinberg.com>
`doom-cli-upgrade` calls `git fetch` to fetch remote branch and tags and
check output code of this command to complain when it failed.
It fails when remote server isn't available that is good.
But it also fails when it can't to fast-forward a branch that will be
used to upgrade doom-emacs.
The last case created a loop and the only way to broke this loop is
remote a branch that should be used for upgrade.
Unfortunately user has no idea about branch name and can't do anything,
just enjoy crash with `Failed to fetch from upstream` message.
This commit added `--force` flag to `git fetch` that forces it to
overwrite branch that is used to upgrade that prevent such loop.
It also fixed https://github.com/hlissner/doom-emacs/issues/4346
Previously, `doom upgrade` would emit:
```
New revision: fatal: amb (fatal: ambiguous argument '_upgrade/HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
```
It appears it's not possible to remote revisions with
`<remote>/<revision>`. Instead, we fetch the upgrade remote into a named
revision and then use it.
Signed-off-by: Rudi Grinberg <me@rgrinberg.com>
Still a long way to go, but this introduces a few niceties for
debugging CLI failures:
+ The (extended) output of the last bin/doom command is now logged to
~/.emacs.d/.local/doom.log
+ If an error occurs, short backtraces are displayed whether or not you
have debug mode on. The full backtrace is written to
~/.emacs.d/.local/doom.error.log.
+ bin/doom now aborts with a warning if:
- The script itself or its parent directory is a symlink. It's fine if
~/.emacs.d is symlinked though.
- Running bin/doom as root when your DOOMDIR isn't in /root/.
- If you're sporting Emacs 26.1 (now handled in the elisp side rather
than the /bin/sh shebang preamble).
+ If a 'doom sync' was aborted prematurely, you'll be warned that Doom
was left in an inconsistent state and that you must run `doom sync`
again.
May address #3746
elisp lacks an execv implementation (or mature subprocess library), so
we exploit some splenderiffic hackery to get Emacs to execute arbitrary
shell commands after a 'doom ...' command completes. This allows us to
daisy chain doom commands in distinct sessions (wonderful for reloading
doom after a 'doom upgrade', which we do). This minimizes errors when a
'doom upgrade' pulls in breaking changes to Doom's CLI.
We also bring 'doom run' into elisp, since this new functionality
enables us to.
We no longer need two separate autoloads files, so I merged them and
optimized its generation logic.
Other changes
- Doom will refuse to start up (with a helpful error) if it's in an
incomplete state. This should hopefully reduce the number of bug
reports from folks that have done something weird, e.g.
1. You've changed Emacs versions without running 'doom sync -b'.
2. You've updated Doom outside of `doom upgrade` and didn't run `doom
sync -u`.
3. You've forgotten to run 'doom sync' in the first place!
4. If a previous 'doom ...' command was aborted midway without running
'doom sync' afterwards.
- 'doom sync' will emit reminders that you need to reload/restart Emacs
- Autoloads API now uses the `doom-autoloads-` prefix, intead of
'doom-cli-autoloads-', as will be the new convention in the coming
rewrite.
- Errors from within the package autoloads should be easier to invoke
the debugger on.
- `doom-modules` is now stored in your autoloads file. Your module list
will soon be frozen between calls to 'doom sync' to allow for our new,
atomic CLI I'm working on. This will also means the `doom!` block
won't cost anything in interactive sessions.
The rewrite for Doom's CLI is taking a while, so I've backported a few
important changes in order to ease the transition and fix a couple bugs
sooner.
Fixes#2802, #2737, #2386
The big highlights are:
- Fix#2802: We now update recipe repos *before* updating/installing any
new packages. No more "Could not find package X in recipe repositories".
- Fix#2737: An edge case where straight couldn't reach a pinned
commit (particularly with agda).
- Doom is now smarter about what option it recommends when straight
prompts you to make a choice.
- Introduces a new init path for Doom. The old way:
- Launch in "minimal" CLI mode in non-interactive sessions
- Launch a "full" interactive mode otherwise.
The new way
- Launch in "minimal" CLI mode *only* for bin/doom
- Launch is a simple mode for non-interactive sessions that still need
access to your interactive config (like async org export/babel).
- Launch a "full" interactive mode otherwise.
This should fix compatibility issues with plugins that use the
async.el library or spawn child Emacs processes to fake
parallelization (like org's async export and babel functionality).
- Your private init.el is now loaded more reliably when running any
bin/doom command. This gives you an opportunity to configure its
settings.
- Added doom-first-{input,buffer,file}-hook hooks, which we use to queue
deferred activation of a number of packages. Users can remove these
modes from these hooks; altogether preventing them from loading,
rather than waiting for them to load to then disable them,
e.g. (after! smartparens (smartparens-global-mode -1)) -> (remove-hook
'doom-first-buffer #'smartparens-global-mode)
Hooks added to doom-first-*-hook variables will be removed once they
run.
This should also indirectly fix#2386, by preventing interactive modes
from running in non-interactive session.
- Added `doom/bump-*` commands to make bumping modules and packages
easier, and `doom/bumpify-*` commands for converting package!
statements into user/repo@sha1hash format for bump commits.
- straight.el is now commit-pinned, like all other packages. We also
more reliably install straight.el by cloning it ourselves, rather than
relying on its bootstrap.el.
This should prevent infinite "straight has diverged from master"
prompts whenever we change branches (though, you might have to put up
with it one more after this update -- see #2937 for workaround).
All the other minor changes:
- Moved core/autoload/cli.el to core/autoload/process.el
- The package manager will log attempts to check out pinned commits
- If package state is incomplete while rebuilding packages, emit a
simpler error message instead of an obscure one!
- Added -u switch to 'doom sync' to make it run 'doom update' afterwards
- Added -p switch to 'doom sync' to make it run 'doom purge' afterwards
- Replace doom-modules function with doom-modules-list
- The `with-plist!` macro was removed, since `cl-destructuring-bind`
already serves that purpose well enough.
- core/autoload/packages.el was moved into core-packages.el
- bin/doom will no longer die if DOOMDIR or DOOMLOCALDIR don't have a
trailing slash
- Introduces doom-debug-variables; a list of variables to toggle on
doom/toggle-debug-mode.
- The sandbox has been updated to reflect the above changes, also:
1. Child instances will no longer inherit the process environment of
the host instance,
2. It will no longer produce an auto-save-list directory in ~/.emacs.d
refresh isn't going anywhere, but sync will be the canonical command
here on out, because it is more appropriate for kind of work it will be
doing in the future.
Highlights:
- 'doom purge' now purges builds, elpa packages, and repos by default.
Regrafting repos is now opt-in with the -g/--regraft switches.
Negation flags have been added for elpa/repos: -e/--no-elpa and
-r/--no-repos.
- Removed 'doom rebuild' (it is now just 'doom build' or 'doom b').
- Removed 'doom build's -f flag, this is now the default. Added the -r
flag instead, which only builds packages that need rebuilding.
- 'doom update' now updates packages synchronously, but produces more
informative output about the updating process.
- Straight can now prompt in batch mode, which resolves a lot of issues
with 'doom update' (and 'doom upgrade') freezing indefinitely or
throwing repo branch errors.
- 'bin/doom's switches are now positional. Switches aimed at `bin/doom`
must precede any subcommands. e.g.
Do: 'doom -yd upgrade'
Don't do: 'doom upgrade -yd'
- Moved 'doom doctor' from bin/doom-doctor to core/cli/doctor, and
integrated core/doctor.el into it, as to avoid naming conflicts
between it and Emacs doctor.
- The defcli! macro now has a special syntax for declaring flags, their
arguments and descriptions.
Addresses #1981, #1925, #1816, #1721, #1322
Since there is a lot of logic in `bin/doom upgrade` than just running
two commands, I thought we should also add a timeout flag just like
`update` has.
