Make doom-get-outdated-packages asynchronous

CHANGELOG.org

@@ -57,7 +57,6 @@
 + =feature/jump= Automatic etags generation (for dwim go-to-definition and, perhaps, code-completion for some languages; lua maybe?).
 + =lang/php= Automatic and async tags generation using [[https://github.com/xcwen/phpctags][phpctags]].
 + =lang/lua= True, dynamic code-completion? Looks like [[https://github.com/immerrr/lua-mode/pull/119][this PR in lua-mode]] may have the answer. Does it make ~company-lua~ redundant?
-+ Make QUELPA outdated-detection async with ~async-start~ and ~async-get~ (in ~doom/packages-update~).
 + =tools/upload= Add ~+upload/open-remote-file~ command to open current file on the remote (with TRAMP).
 + Add =bin/org-alert= script -- a cron script that scans TODOs in org files and dispatches system alerts.
 + =feature/workspaces= Add a bookmarks feature, but for wconfs, that can revive file buffers. Also needs an interface.
@@ -159,6 +158,7 @@
     + Make ~doom-update-package~ atomic in case of failure.
     + Make ~doom-refresh-packages~ async.
     + Improve the security of package management (via ELPA) by a) forcing Emacs to verify TLS connections and b) use HTTPS sources for MELPA and ELPA.
+    + Make ~doom-get-outdated-packages~ asynchronous, producing a substantial speed-up when updating packages from Quelpa sources.
 + =feature=
   + =feature/evil=
     + Add ~+evil:mc~ command [[https://github.com/gabesoft/evil-mc][evil-mc]].

core/autoload/packages.el

@@ -123,11 +123,41 @@ containing (PACKAGE-SYMBOL OLD-VERSION-LIST NEW-VERSION-LIST).
 If INCLUDE-FROZEN-P is non-nil, check frozen packages as well.
 
 Used by `doom/packages-update'."
-  (cl-loop for pkg in (doom-get-packages)
+  (let (quelpa-pkgs elpa-pkgs)
-           if (or (and (doom-package-prop (car pkg) :freeze)
+    (dolist (pkg (doom-get-packages))
+      (let ((sym (car pkg)))
+        (when (and (or (not (doom-package-prop sym :freeze))
                        include-frozen-p)
-                  (doom-package-outdated-p (car pkg)))
+                   (not (doom-package-prop sym :ignore)))
-           collect it))
+          (push sym
+                (if (eq (doom-package-backend sym) 'quelpa)
+                    quelpa-pkgs
+                  elpa-pkgs)))))
+    (let* ((max-threads 3) ; TODO Do real CPU core/thread count
+           (min-per-part 2)
+           (per-part (max min-per-part (ceiling (/ (length quelpa-pkgs) (float max-threads)))))
+           (leftover (mod (length quelpa-pkgs) per-part))
+           (packages '(t))
+           parts
+           futures)
+      (while quelpa-pkgs
+        (let (part)
+          (dotimes (i (+ per-part leftover))
+            (when-let (p (pop quelpa-pkgs))
+              (push p part)))
+          (setq leftover 0)
+          (push (nreverse part) parts)))
+      (dolist (part (reverse parts))
+        (debug! "New thread for: %s" part)
+        (push (async-start
+               `(lambda ()
+                  (let ((noninteractive t))
+                    (load ,(expand-file-name "core.el" doom-core-dir)))
+                  (delq nil (mapcar #'doom-package-outdated-p ',part))))
+              futures))
+      (apply #'append
+             (delq nil (mapcar #'doom-package-outdated-p elpa-pkgs))
+             (mapcar #'async-get futures)))))
 
 ;;;###autoload
 (defun doom-get-orphaned-packages ()