[[orgstrap][jump to orgstrap block for this file]]

#+title: Orgstrap: Executable Org files #+startup: showall #+options: num:nil \n:nil

#+todo: TODO STARTED | DONE EXPIRED

#+property: header-args :eval no-export #+property: header-args:elisp :lexical yes

#+latex_header: \usepackage[margin=0.8in]{geometry} #+latex_header: \setlength\parindent{0pt}

#+link: yt https://youtu.be/ #+link: gh https://github.com/

[[file:./README.pdf]]

[[file:./README.html]]

#+HTML: #+HTML:

orgstrap

Not quite a unicorn.

If you want to grow up to be a unicorn you're going to have to

pull yourself up by your own bootstraps!

#+name: orgstrap-shebang #+begin_src bash :eval never :results none :exports none set -e "-C" "-e" "-e" { null=/dev/null;} > "{null:=/dev/null}" { args=;file=;MyInvocation=;__p=(mktemp -d);touch {__p}/=;chmod +x {__p}/=;__op=$PATH;PATH=${__p}:PATH;} > "{null}" $file =$MyInvocation.MyCommand.Source { file=$0;PATH=__op;rm {__p}/=;rmdir {__p};} > "{null}" emacs -batch -no-site-file -eval "(let (vc-follow-symlinks) (defun org-restart-font-lock ()) (defun orgstrap--confirm-eval (l _) (not (memq (intern l) '(elisp emacs-lisp)))) (let ((file (pop argv)) enable-local-variables) (find-file-literally file) (end-of-line) (when (eq (char-before) ?^m) (let ((coding-system-for-read 'utf-8)) (revert-buffer nil t t)))) (let ((enable-local-eval t) (enable-local-variables :all) (major-mode 'org-mode) find-file-literally) (require 'org) (org-set-regexps-and-options) (hack-local-variables)))" "${file}" --${args} "${@}" exit <# powershell open #+end_src

=orgstrap= allows Org files to describe their own requirements and define their own functionality, making them self-contained standalone computational artifacts dependent only on Emacs or other implementations of the Org Babel protocol in the future.

This file bootstraps itself to provide the tools to use =orgstrap= in any Org file.

=orgstrap= has a formal [[#specification][specification]] and this file contains 3 implementations that support 3 slightly different use cases along with tooling for common =orgstrap= workflows.

=orgstrap= works with all versions of Emacs since =24.4= and Org since =8.2.10=.

Please see the [[#changelog][changelog]] for the latest updates.

• Contents :noexport: :PROPERTIES: :TOC: :include all :depth 1 :visibility: folded :END: :CONTENTS:

• [[#getting-started][Getting started]]
• [[#hello-orgstrap][Hello orgstrap]]
• [[#inspiration][Inspiration]]
• [[#use-cases][Use cases]]
• [[#details][Details]]
• [[#specification][Specification]]
• [[#local-variables][Local Variables]]
• [[#code][Code]]
• [[#changelog][Changelog]]
• [[#contributing][Contributing]]
• [[#guides][Guides]]
• [[#best-practices][Best practices]]
• [[#bootstrapping-to-emacs-bootstrapping-to-org][Bootstrapping to Emacs, bootstrapping to Org]]
• [[#examples][Examples]]
• [[#background-file-local-variables-and-checksums][Background, file local variables, and checksums]]
• [[#experience-reports][Experience reports]]
• [[#future-work][Future work]] :END:

• Getting started :PROPERTIES: :CUSTOM_ID: getting-started :END: Using =orgstrap= is easy.

If you already have =orgstrap= installed you can enable it for any Org file by running =M-x= =orgstrap-init= which will add the basic =orgstrap= machinery to the current buffer.

=orgstrap= is available on [[https://melpa.org/#/orgstrap][melpa]]. You can install it via =M-x= =package-install= =orgstrap= or (use-package orgstrap).

You can also try out =orgstrap= without installing just by opening this file in Emacs!

1. Obtain the org mode source for this file. (e.g. [[https://raw.githubusercontent.com/tgbugs/orgstrap/master/orgstrap.org][from GitHub]]).
2. Open the source in Emacs[[#bootstrapping-to-emacs-bootstrapping-to-org][*]]. (e.g. =M-x= =url-handler-mode= then =C-x C-f=

   @@latex: \@@

   =https://raw.githubusercontent.com/tgbugs/orgstrap/master/README.org=).
3. Decline the file local variables.
4. Inspect the [[#details][orgstrap block]] and file local variables.
5. Reload the file and accept the file local variables.
6. Congratulations you can now use =orgstrap= with your own files!

If you install =orgstrap= in this way you have to open the file again every time you open a new Emacs, so installing [[file:./orgstrap.el][orgstrap.el]] via package.el or by some other means is recommended.

A minor mode for editing orgstrapped files is included as =orgstrap-edit-mode=. It is activated by =orgstrap-init=. When enabled it automatically updates the =orgstrap-block-checksum= prop line local variable whenever the =orgstrap= block changes.

If you do not use =orgstrap-edit-mode= then the easiest way to add the orgstrap checksum to a file is to invoke =M-x= =orgstrap-add-block-checksum=.

=orgstrap= also includes =orgstrap-mode=, which is a regional minor mode for =org-mode=. When enabled, =orgstrap-mode= detects, checks, and runs orgstrap blocks when visiting Org files, superseding any embedded =eval:= local variables.

The rest of this file is an overview of the use cases for =orgstrap= and the implementation of =orgstrap= along with discussion and commentary.

If you are looking for examples of how to use =orgstrap= this files is a good place to start.

• Hello =orgstrap= :PROPERTIES: :CUSTOM_ID: hello-orgstrap :END: The bare minimum needed to make an =org-mode= file executable (with a bit of safety). #+caption: [[file:./orgstrap-minimal.org]] #+begin_src org :tangle ./orgstrap-minimal.org

-- orgstrap-cypher: sha256; orgstrap-block-checksum: 66ba9b040e22cc1d30b6f1d428b2641758ce1e5f6ff9ac8afd32ce7d2f4a1bae; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; --

[[orgstrap][jump to orgstrap block for this file]]

,#+name: orgstrap ,#+begin_src elisp :results none (message "orgstrap successful!") ; (ref:im-a-coderef-and-thats-ok) ,#+end_src

=orgstrap= a plain-text executable format. Powered by Org mode and Emacs.

Local Variables:

eval: (progn (setq-local orgstrap-min-org-version "8.2.10") (let ((a (org-version)) (n orgstrap-min-org-version)) (or (fboundp #'orgstrap--confirm-eval) (not n) (string< n a) (string= n a) (error "Your Org is too old! %s < %s" a n))) (defun orgstrap-norm-func--dprp-1-0 (body) (let ((p (read (concat "(progn\n" body "\n)"))) (m '(defun defun-local defmacro defvar defvar-local defconst defcustom)) print-quoted print-length print-level) (cl-labels ((f (b) (cl-loop for e in b when (listp e) do (or (and (memq (car e) m) (let ((n (nthcdr 4 e))) (and (stringp (nth 3 e)) (or (cl-subseq m 3) n) (f n) (or (setcdr (cddr e) n) t)))) (f e))) p)) (prin1-to-string (f p))))) (unless (boundp 'orgstrap-norm-func) (defvar-local orgstrap-norm-func orgstrap-norm-func-name)) (defun orgstrap-norm-embd (body) (funcall orgstrap-norm-func body)) (unless (fboundp #'orgstrap-norm) (defalias 'orgstrap-norm #'orgstrap-norm-embd)) (defun orgstrap--confirm-eval-minimal (lang body) (not (and (member lang '("elisp" "emacs-lisp")) (eq orgstrap-block-checksum (intern (secure-hash orgstrap-cypher (orgstrap-norm body))))))) (unless (fboundp #'orgstrap--confirm-eval) (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-minimal)) (let (enable-local-eval) (vc-find-file-hook)) (let ((ocbe org-confirm-babel-evaluate) (obs (org-babel-find-named-block "orgstrap"))) (if obs (unwind-protect (save-excursion (setq-local orgstrap-norm-func orgstrap-norm-func-name) (setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval) (goto-char obs) (org-babel-execute-src-block)) (when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval) (setq-local org-confirm-babel-evaluate ocbe)) (ignore-errors (org-set-visibility-according-to-property))) (warn "No orgstrap block."))))

End:

#+end_src

• Inspiration :PROPERTIES: :CUSTOM_ID: inspiration :END: By default =org-mode= source block headers only take existing elisp functions as arguments.

This means that header arguments can become extremely verbose.

Wouldn't it be great if you could use the magical mystical power of =defun= inside an org file itself to provide simple, reusable functionality rather than +copying and pasting+ +yanking and putting+ killing and yanking raw elisp around the buffer?

With =orgstrap= you can.

=orgstrap= makes sure that the functionality that you need is available when you need it. Whether it is =(defun dir-tramp-sudo (host) (format "/ssh:%s|sudo:%s:" host host))= to simplify a pattern for remote execution when using the =:dir= header argument, or a function to detect and set the right environment variables, =orgstrap= is there for you.

• Use cases :PROPERTIES: :CUSTOM_ID: use-cases :END: =orgstrap= specifies what is essentially a plain-text executable file format. Thus, it can be used for nearly everything[fn::Now, whether it should be....].

While many (including the author) might find this to be totally radically awesome, there are much better, saner, and safer ways to execute arbitrary code than to hash some elisp blocks and use Emacs file local variables to automatically eval a specially named source block only when it matches the hash.

#+caption: Things you can do with arbitrary code execution and checksums. #+name: table-use-cases |----------------------------------------+------------------+------------------------------------| | Use case | Good idea | Alternative | |----------------------------------------+------------------+------------------------------------| | Always run defuns used in file | ✅ Yes | init.el, =C-c C-c= | | Install elisp code directly | ❌ No | Use =package.el=, =straight=, etc. | | Self tangling files | ✅ I do it | =C-c C-v C-t= | | Install packages required by file | Probably | System package manager | | Create an Emacs based botnet | ✅ ✅ Definitely | ??? | | Create Orgware for non-technical users | ✅ Yes | Web server and the unholy trinity. | | Replace hard to follow instructions | ✅ Yes | Hard to follow instructions | | Tangle git hook files for publishing | ✅ Yes | Manually tangle | | System specific behavior without edits | ✅ Yes | #+name: literal blocks via =:= | | Version control for source blocks | ❌ ❌ Please no | git, hg, svn, anything please | | Detect and set environment variables | ✅ Yes | | |----------------------------------------+------------------+------------------------------------|

Actually I'm kind of hyped for though of describing the system used to version

control the code in the file itself. Not so simple to pull off though.

It only sort of works in this case because we have the rest of the file under

version control in another system. Without git, developing this would have been

a complete nightmare.

• Details :PROPERTIES: :CUSTOM_ID: details :END: The first elisp source block named =orgstrap= in an org file is automatically run using an =eval:= file local variable. Users can review and add the file local variables to their known safe list so that the code can be run in the future without the need to bother them again.

When opening a file for the first time, users should decline the local variables, review the =eval:= local variable and the =orgstrap= block directly, and then reload, revisit, or =M-x= =org-mode= and only then accept the local variables. This only needs to be done once for the =eval:= local variables (unless they are updated).

** The orgstrap block :PROPERTIES: :CUSTOM_ID: the-orgstrap-block :END: This is the =orgstrap= block that is used for this file.

FIXME the -r -l is kind of needed here deal with ref blocks

FIXME this is an internal inconsistency in babel

#+caption: The =orgstrap= block that is used for this file. #+name: orgstrap #+begin_src elisp :results none :noweb no-export :lexical yes ;; This is an example that also nowebs in the source for ;; orgstrap-init' and orgstrap-add-block-checksum' along ;; with the rest of the orgstrap machinery so it is easy to ;; use orgstrap to create and update orgstrap blocks

<> <> <> <> <>

;; tangle helpers

(defun ow---strip-empty-lines-and-refs () (save-excursion (goto-char (point-min)) (while (re-search-forward "^ +$\\|[ ;]*(ref:.+)$" nil t) ;; FIXME stripping the refs here can cause a divergence for the checksums ;; FIXME this incorrectly strips refs from orgstrap-minimal.org due to wrong mode (replace-match ""))))

;; XXX reminder that this cannot be a buffer local hook because it ;; doesn't run in this buffer this is likely a bug (add-hook 'org-babel-tangle-body-hook #'ow---strip-empty-lines-and-refs)

;; helper functions to update examples (defun orgstrap--update-examples () "Use with orgstrap-on-change-hook' to automatically keep the contents of the example blocks in sync." ;; XXX WARNING if you update the orgstrap-file-local-variables-common block ;; you MUST reeval-defun' for orgstrap--local-variables--eval-common' and ;; orgstrap--lv-common-with-block-name' otherwise the changes will not take (let ((pairs `(("local-variables-prop-line-example" ,(orgstrap--local-variables-prop-line-string)) ("local-variables-portable-example" ,(orgstrap--file-local-variables-string)) ("local-variables-minimal-example" ,(let ((orgstrap-use-minimal-local-variables t)) (orgstrap--file-local-variables-string)))))) (mapcar (lambda (name-content) (apply #'orgstrap-update-src-block name-content)) pairs)))

(defun orgstrap--local-variables-prop-line-string () "Copy the first logical line of the file since it is easier and faster than trying to sort out which variables were or were not in the prop line." ;; XXX NOTE There are some cases involving bootstrapping to emacs where the first line of ;;an org-mode file is a shebang, but we will deal with those if and when they arrise (buffer-substring-no-properties 1 (save-excursion (goto-char 0) (next-logical-line) (point))))

(defun orgstrap--file-local-variables-string () (let (print-length) (with-temp-buffer (org-mode) (goto-char 0) (insert "#+name: orgstrap\n#+begin_src elisp :lexical yes\n#+end_src\n") (orgstrap--add-file-local-variables orgstrap-use-minimal-local-variables) (goto-char 0) (kill-whole-line 4) (buffer-string))))

;; tangle blocks and update examples on change (add-hook 'orgstrap-on-change-hook #'org-babel-tangle nil t) ;; FIXME should fire on non-semantic changes (add-hook 'orgstrap-on-change-hook #'orgstrap--update-examples nil t) ;; enable orgstrap mode locally for this file when this block runs (orgstrap-edit-mode 1)

(message "orgstrap complete!") #+end_src

The headers for the block above look like this. #+name: orgstrap-example #+begin_example org :eval never :noweb no ,#+name: orgstrap ,#+begin_src elisp :results none :noweb no-export :lexical yes <> ,#+end_src #+end_example

Additional machinery is provided as part of this file to update the local variable value of =orgstrap-block-checksum= so that only known blocks can be run. Note that this DOES NOT PROTECT against someone changing the block and the checksum at the same time and sending you a malicious file! You need an alternate and trusted source against which to verify the checksum of the =orgstrap= block. ** Portability A couple of notes on portability and backward compatibility with older versions of Emacs. I have tried to get =orgstrap= running on emacs-23, however the differences between org =6.33x= and org =8.2.10= are too large to be overcome without significant additional code. First, all uses of =(setq-local var "value")= have to be changed to =(set (make-local-variable 'var) "value")= so that the local variable eval code can run. However once that is done, you discover that all of the org-babel functions are missing, and then you will discover that emacs-23 doesn't support lexical binding. Therefore, we don't support emacs-23 and older versions. ** Version specific behavior There is a major usability issue for =orgstrap= when running Emacs < 27. Specifically, prior to Emacs 27 it is not possible to view the file whose local variables are about to be set because it is impossible to switch out of the file local variables confirmation buffer. Starting in Emacs 27 it is possible to change buffer to view the file that is about to have its file local variables set.

• Specification :PROPERTIES: :CUSTOM_ID: specification :END:

Except for this comment, comments in the spec are not official parts of the spec.

** Terminology The specification for orgstrap makes extensive use of terminology derived from the Emacs manual section on [[info:emacs#Specifying File Variables][Specifying File Variables]] and the Org manual section on the [[info:org#Structure of Code Blocks][Structure of Code Blocks]].

What the Emacs manual calls the first line or prop-line is referred to in this document as the =prop line= and the variables specified in it are referred to as =prop line local variables=. What the Emacs manual explicitly calls the =local variables list= we refer to in the same way[fn::In other sections of the readme that contains this specification the nomenclature is inconsistent, and refers to these variously as end local variables or simply as local variables or file local variables.].

What the Org manual refers to as a =source code block= we refer to in the same way. ** File contents In order for an Org mode file to support the use of =orgstrap= it must contain the following.

The =prop line= of the Org file must include three local variables: =orgstrap-cypher=, =orgstrap-norm-func-name=, and =orgstrap-block-checksum=.

Anywhere in the rest of the file there must be an Org =source code block= that has the == =orgstrap= with whitespace preceding the =o= and only whitespace following the =p= until a newline. Newline and whitespace are as defined by [[https://orgmode.org/worg/dev/org-syntax.html][Org mode syntax]]. This =source code block= is henceforth referred to as the =orgstrap block=. If there is more than one =source code block= with the == =orgstrap= then the =source code block= that starts closest to the beginning of the file is the =orgstrap block=.

The == for the =orgstrap block= must be =elisp= or =emacs-lisp=. [fn:: It is possible that other languages might be supported in the future. However, that is somewhat challenging given that Org and Orb-babel only implicitly specify that a conforming implementation that can execute =source code blocks= must support Emacs lisp =source code blocks= and the use of Emacs lisp in header arguments. There is an infinitesimal possibility that Org-babel will support the use of other languages for inline header arguments since it already supports them via blocks and it is not trivial to allow additional languages to be used inline without some additional way to indicate the language in use for a particular block. On the other hand, there is a small possibility that other languages could be supported in the =orgstrap block= by specifying them as part of the =local variables list=. However it is not clear that this is needed, because it is possible to specify a small orgstrap block that can ensure that the required Org-babel language implementations are installed and then securely run those blocks. This block can probably be stripped down sufficiently to make it possible to implement only the subset of elisp required to run that block.]

Everything else about the =orgstrap block= is delegated to Org mode, including header arguments, and noweb expansion.

TODO With the possible exception being that a header of the form

:var orgstrap-enable-optional=(identity nil) might be added to

make it possible for the user to toggle optional dependencies

obviously authors can do whatever they want with the block and

set as many :vars to t or nil as they want to give users as much

or as little control over what is run as they desire, this should

probably just go in as an example, with note that this is one of the

reasons why we don't hash :vars but also why users need to check those

I'm 99% certain that embedding orgstrap-norm-func in the local variables list

should NOT be required as part of the specification. I do that in the current

implementation, but the 3000 char limit for the local variables list is going

to pose quite the challenge for the portable implementation, and thus I think

all the spec needs to say is that an implementation must be able to reproduce

the orgstrap block hash when the whole file hash is the same.

** Implementation behavior When provided with the same file whose =orgstrap block= was originally hashed (where "the same file" means a file with the same checksum when hashed using the algorithm specified by the =orgstrap-cypher= variable), a conforming implementation must be able to do the following.

A conforming implementation must be able to reproduce the =orgstrap-block-checksum= using only the information contained in the =orgstrap-cypher= and =orgstrap-norm-func-name= =prop line local varaibles=, and information contained in the rest of the file explicitly excluding the contents of the =orgstrap-block-checksum= =prop line local varaible=. The most obvious additional information required being the contents of the =orgstrap block= [fn:: The reference implementation provided in the readme containing this specification uses an Emacs =eval:= local variable (elv) in the =local variables list=. Embedding an elv is not required by this specification. However, such an implementation allows files to depend only on the core Emacs implementation.

In the future an optional extension may be added to this document that specifies the behavior for files using an elv in the =local variables list=.

A minimal implementation that works without elvs is also provided.

Files that contain only the prop line local variables are dependent on an implementation of orgstrap already being present on the system running the file.

There is a fine balance between portability and compactness since a minimal implementation has to make more assumptions about the systems it will run on.

Multi-stage orgstrap or other means of bootstrapping a working runtime for an Org file such as the process implemented in the [[#bootstrapping-to-emacs-bootstrapping-to-org][Bootstrapping to Emacs, bootstrapping to Org]] section of this readme are ongoing areas of exploration.].

#+begin_quote One implementation detail is that conforming implementations must implement noweb expansion and coderef removal prior to passing the contents of the =orgstrap block= to a normalization function. #+end_quote

Normalization functions that produce different output given the same input for at least one input must have different names. One way this can be achieved is by suffixing a name with a version number.

In order for an orgstrap normalization function name to be considered official it must have an implementation bearing that name in the [[#normalization-functions][Normalization functions]] section of the readme that contains this specification. Once a function has been named, no other function shall ever bear the same name unless for all inputs it produces output that is byte-identical to the output of all other previous implementations of the function bearing that name.

#+begin_quote A key point about =orgstrap-norm-func-name= is that the implementation of these functions must be agreed upon by various implementations, if a user inserts a fake hash, implementations should deal with it by running the normalization and hashing process again using a known-conforming implementation on a system that they control. #+end_quote

• Local Variables :PROPERTIES: :CUSTOM_ID: local-variables :END: ** Contents :noexport: :PROPERTIES: :TOC: :include siblings :exclude this :depth 1 :visibility: folded :END: :CONTENTS:

• [[#overview][Overview]]
• [[#org-version-support][Org version support]]
• [[#normalization][Normalization]]
• [[#definitions][Definitions]]
• [[#note-on-noweb-support][Note on noweb support]]
• [[#note-on-coderefs][Note on coderefs]]
• [[#how-local-variables-appear-in-the-file][How local variables appear in the file]] :END: ** Overview :PROPERTIES: :CUSTOM_ID: overview :END: This section contains two implementations of =orgstrap= (minimal and portable) that are small enough to fit in the local variables list at the end of a file. The local variables list must start less than 3000 chars from the end of the file.

We use =setq-local= in =eval:= to set =org-confirm-babel-evaluate= because it is a =safe-local-variable= only when the value is =t= and cannot be set directly as a file local variable. In this context this workaround seems reasonable and not malicious because the use of =eval:= should alert users that some arbitrary stuff is going on and that they should be on high alert to check it.

Below in [[#definitions][Definitions]] there is a more readable version of what the compacted local variables code at the end of the file is doing. Always check that the =eval:= local variables in unknown orgstrapped files match a known set when reviewing and accepting local variables.

=orgstrap= eval local variables, or elvs for short, are little helpers at the end of the file that make everything work in a portable manner when =orgstrap.el= is not present on a system.

While elvs are not required by the specification, they greatly reduce the complexity of implementation. They also simplify the instructions to two steps: 1. install Emacs, 2. open the file.

the orgstrap block can the install orgstrap.el if needed

TODO it is entirely possible to automate that check

but not without already having orgstrap available.

TODO publish the hashes of the eval sexps.

** Org version support :PROPERTIES: :CUSTOM_ID: org-version-support :END: Different versions of the =orgstrap= local variables work with different versions of =org-mode=. We include an explicit version check and fail so that strange partial successes can be avoided and so that newer versions of the local variables can be simplified when backward compatibility is not needed. For example one might imagine a future where no local variables are needed in the file at all, only the cypher and the checksum because we managed to get support for the convention built into =org-mode= directly.

This will also allow us to streamline which block to use based on whether noweb is being used. If it is not then we can decide automatically.

If orgstrap is installed, we use the installed version of orgstrap anyway so don't bother. #+name: orgstrap-check-org-version #+begin_src elisp (let ((a (org-version)) ; actual (n orgstrap-min-org-version)) ; need (or (fboundp #'orgstrap--confirm-eval) ; orgstrap with portable is already present on the system (not n) (string< n a) (string= n a) (error "Your Org is too old! %s < %s" a n))) #+end_src #+caption: Portability note. #+begin_quote =string<= must be used in order to support emacs-24 #+end_quote ** Normalization :PROPERTIES: :CUSTOM_ID: normalization :END: *** Shared normalization machinery Shared normalization code embedded as elvs.

FIXME should orgstrap-norm-func be a local variable ?!?

#+caption: Shared normalization code embedded as elvs. #+name: orgstrap-normalization-common-embed #+begin_src elisp (unless (boundp 'orgstrap-norm-func) (defvar-local orgstrap-norm-func orgstrap-norm-func-name))

(defun orgstrap-norm-embd (body) "Normalize BODY." (funcall orgstrap-norm-func body))

(unless (fboundp #'orgstrap-norm) (defalias 'orgstrap-norm #'orgstrap-norm-embd)) #+end_src

Normalization functions for orgstrap.el. #+caption: Normalization functions for orgstrap.el. #+name: orgstrap-code-normalization-functions #+begin_src elisp :eval never :noweb yes (defun orgstrap-norm (body) "Normalize BODY." (if orgstrap--debug (orgstrap-norm-debug body) (funcall orgstrap-norm-func body)))

(defun orgstrap-norm-debug (body) "Insert BODY normalized with NORM-FUNC into a buffer for easier debug." (let* ((print-quoted nil) (bname (format "body-norm-%s" emacs-major-version)) (buffer (let ((existing (get-buffer bname))) (if existing existing (create-file-buffer bname)))) (body-normalized (funcall orgstrap-norm-func body))) (with-current-buffer buffer (erase-buffer) (insert body-normalized)) body-normalized))

;; orgstrap normalization functions

<>

<>

<>

#+end_src

#+caption: XXX portability note #+begin_quote For emacs < 26 (org < 9) either lowercase =#+caption:= must be placed BEFORE =#+name:=, OR =#+CAPTION:= must be uppercase and can come after =#+name:=, otherwise =#+name:= will not be associated with the block. What a fun bug.

Addendum. Apparently in the older version of Org =:noweb= is always yes. As a result, testing against Emacs 24 or 25 will alert you if you forget to set =:noweb= on a block. #+end_quote *** Normalization functions :PROPERTIES: :CUSTOM_ID: normalization-functions :END: **** prp-1.0 :obsolete: This normalization function is obsolete

#+name: orgstrap-code-normalization--prin1-read-progn-1-0 #+begin_src elisp :eval never (let ((print-quoted nil)) (prin1-to-string (read (concat "(progn\n" body "\n)")))) #+end_src

#+name: block-orgstrap-norm-func--prp-1-0 #+begin_src elisp :noweb yes :eval never (defun orgstrap-norm-func--prp-1-0 (body) "Normalize BODY using prp-1-0." <>) (make-obsolete #'orgstrap-norm-func--prp-1-0 #'orgstrap-norm-func--prp-1-1 "1.2") #+end_src

Normalize BODY by wrapping in =progn=, calling =read=, and then =prin1-to-string=. There are still unresolved issues if tabs are present in the orgstrap block which is why 1.0 is included. =print-quoted= is critical for consistent hashing.

=prin1-to-string= is used to normalize the code in the orgstrap block, removing any comments and formatting irregularities. This is important for two reasons.

First it helps prevent denial of service attacks against human auditors who have low bandwidth for detecting fiddly changes.

Second, normalization that ignores comments makes it possible to improve the documentation of code without changing the checksum. Hopefully this will reduce one of the obstacles to enhancing the documentation of orgstrap code and blocks over time since rehashing will not be required when the meaningful code itself has not changed.

=(print-quoted nil)= is needed for backward compatibility due to a change to the default from =nil= to =t= in emacs-27 (sigh). See [[orgit-rev:~/git/NOFORK/emacs::72ee93d68daea00e2ee69417afd4e31b3145a9fa][emacs commit 72ee93d68daea00e2ee69417afd4e31b3145a9fa]]. **** prp-1.1 #+name: orgstrap-code-normalization--prin1-read-progn-1-1 #+begin_src elisp :eval never (let (print-quoted print-length print-level) (prin1-to-string (read (concat "(progn\n" body "\n)")))) #+end_src

#+name: block-orgstrap-norm-func--prp-1-1 #+begin_src elisp :noweb yes :eval never (defun orgstrap-norm-func--prp-1-1 (body) "Normalize BODY using prp-1-1." <>) #+end_src

I learned that =print-length= and =print-level= exist in the usual way, which is that somehow they got set to something other than =nil= and as a result checksums started failing left and right because the number of expressions in the body of the progn eval was greater than the value of =print-length=, resulting in truncation and replacement with =...=. This can also happens inside =add-file-local-variable= and possibly even inside =format=!? Therefore I'm updating to version to 1.1 of the normalization procedure so that I can defensively bind those variables to =nil=. **** dprp-1.0 A normalization function that is invariant to changes in docstrings.

Walk the tree and setcdr out docstrings.

This normalization function must be portable between versions, which means that the forms that get spliced must be from a static set that does not change between versions.

Since this normalization is mostly a quality of life improvement to allow docstrings to be changed without rehashing, limiting to a specific set of forms is ok. If you use something like cl-defun and change the docstring, then you will have to rehash. The 90% use case is covered here in a compact manner. #+name: orgstrap-code-normalization--dedoc-prin1-read-progn-1-0 #+begin_src elisp :eval never (let ((p (read (concat "(progn\n" body "\n)"))) (m '(defun defun-local defmacro defvar defvar-local defconst defcustom)) print-quoted print-length print-level) (cl-labels ((f (b) (cl-loop for e in b when (listp e) do ; for expression in body when the expression is a list (or (and (memq (car e) m) ; is a form with docstrings (let ((n (nthcdr 4 e))) ; body after docstring (and (stringp (nth 3 e)) ; has a docstring (or (cl-subseq m 3) n) ; var or doc not last (f n) ; recurse for nested ;; splice out the docstring and return t to avoid the other branch (or (setcdr (cddr e) n) t)))) ;; recurse e.g. for (when x (defvar y t)) (f e))) p)) (prin1-to-string (f p)))) #+end_src

#+name: block-orgstrap-norm-func--dprp-1-0 #+begin_src elisp :noweb yes :results none :eval never :eval yes (defun orgstrap-norm-func--dprp-1-0 (body) "Normalize BODY using dprp-1-0." <>) #+end_src

#+begin_src elisp :results code (orgstrap--with-block "orgstrap" (prog1 (read (orgstrap-norm-func--dprp-1-0 body)) nil)) #+end_src

** Definitions :PROPERTIES: :CUSTOM_ID: definitions :END: These blocks are nowebbed into ref:orgstrap-init-helper-defuns and are used directly by =orgstrap-init= to populate file local variables.

The portable confirm eval is extracted to its own block so that we can include it as a backstop for users who have orgstrap installed but are running an older version of =org-mode= than is supported by the file that they are trying to load.

#+caption: Portable confirm eval. #+name: orgstrap-portable-confirm-eval #+begin_src elisp :eval never :noweb yes ;;;###autoload (defun orgstrap--confirm-eval-portable (lang _body) "A backwards compatible, portable implementation for confirm-eval. This should be called by org-confirm-babel-evaluate'. As implemented the only LANG that is supported is emacs-lisp or elisp. The argument _BODY is rederived for portability and thus not used." ;; org-confirm-babel-evaluate' will prompt the user when the value ;; that is returned is non-nil, therefore we negate positive matchs (not (and (member lang '("elisp" "emacs-lisp")) (let* ((body (orgstrap--expand-body (org-babel-get-src-block-info))) (body-normalized (orgstrap-norm body)) (content-checksum (intern (secure-hash orgstrap-cypher body-normalized)))) ;;(message "%s %s" orgstrap-block-checksum content-checksum) ;;(message "%s" body-normalized) (eq orgstrap-block-checksum content-checksum))))) ;; portable eval is used as the default implementation in orgstrap.el ;;;###autoload (unless (fboundp #'orgstrap--confirm-eval) (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-portable)) #+end_src

#+caption: Minimal confirm eval. #+name: orgstrap-minimal-confirm-eval #+begin_src elisp (defun orgstrap--confirm-eval-minimal (lang body) (not (and (member lang '("elisp" "emacs-lisp")) (eq orgstrap-block-checksum (intern (secure-hash orgstrap-cypher (orgstrap-norm body))))))) (unless (fboundp #'orgstrap--confirm-eval) ;; if `orgstrap--confirm-eval' is bound use it since it is ;; is the portable version XXX NOTE the minimal version will ;; not be installed as local variables if it detects that there ;; are unescaped coderefs since those will cause portable and minimal ;; to produce different hashes (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-minimal)) #+end_src

Once =orgstrap--confirm-eval= is defined the rest of the =eval:= local variables are the same.

FIXME vc-find-file-hook aka vc-refresh-state uses find-file NOT

find-file-literally the issue is in vc-link-follow which calls

find-file-noselect blindly we would need to overwrite it or advise

it, I have a workaround which is to set vc-follow-symlinks to nil

for the shebang block, but that causes variant behavior, vc-mode

probably needs a variable to control whether it opens in the mode

that the symlinked file was opened in or the mode of the target to

avoid issues like this

XXX If you modify this block call =M-x orgstrap-run-block=

so that functions that use it internally will be updated.

#+caption: common local variables #+name: orgstrap-file-local-variables-common #+begin_src elisp :eval never (let (enable-local-eval) (vc-find-file-hook)) ; use the obsolete alias since it works in 24 (let ((ocbe org-confirm-babel-evaluate) (obs (org-babel-find-named-block ,orgstrap-orgstrap-block-name))) ; quasiquoted when nowebbed (if obs (unwind-protect (save-excursion (setq-local orgstrap-norm-func orgstrap-norm-func-name) (setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval) (goto-char obs) ; FIXME `org-save-outline-visibility' but that is not portable (org-babel-execute-src-block)) (when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval) ;; XXX allow orgstrap blocks to set ocbe so audit for that (setq-local org-confirm-babel-evaluate ocbe)) (ignore-errors (org-set-visibility-according-to-property))) ;; FIXME warn or error here? (warn "No orgstrap block."))) #+end_src

Since =orgstrap-norm-func= is a dynamic variable it simplifies the potential future case where we don't embed the normalization function, still not sure if we really want to do that though

#+caption: common local variables once lexical is enabled by default #+name: orgstrap-file-local-variables-common-lexical #+begin_src elisp :eval never (let ((orgstrap-norm-func orgstrap-norm-func-name) (org-confirm-babel-evaluate #'orgstrap--confirm-eval) (obs (org-babel-find-named-block ,orgstrap-orgstrap-block-name))) ; quasiquoted when nowebbed (if obs (unwind-protect (save-excursion (goto-char obs) (org-babel-execute-src-block)) (org-set-startup-visibility)) ;; FIXME warn or error here? (warn "No orgstrap block."))) #+end_src ** Note on noweb support :PROPERTIES: :CUSTOM_ID: note-on-noweb-support :END: The minimal set of local variables only works if you don't use noweb or if you are using Org =>== =9.3.8=.

The portable set of local variables described below works with versions of Org as far back as =8.2.10= (the version bundled with =emacs-24.5=). ** Note on coderefs :PROPERTIES: :CUSTOM_ID: note-on-coderefs :END: Older versions of =org-mode= do not know what to do with coderefs. The simplest solution is to hide them in comments as =;(ref:coderef)= if you need them. See [[(clrin)]] and [[(oab)]] for examples in this file. ** How local variables appear in the file :PROPERTIES: :CUSTOM_ID: how-local-variables-appear-in-the-file :END:

DO NOT EDIT THESE BLOCKS THEY ARE UPDATED AUTOMATICALLY

Here is the prop line from the first line of this file that includes the cypher and checksum of the =orgstrap= block. #+name: local-variables-prop-line-example #+begin_src org :eval never

-- org-adapt-indentation: nil; org-edit-src-content-indentation: 0; orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: d33bdc8924478fedbd92ff73836c43e136d90e4c18393ff7c5e0aeda37f892d2; --

#+end_src

BE VERY CAREFUL WITH MANUAL EDITS

If this block is being edited manually the automatic update will not work.

Here are the portable local variables from the end of the file. #+name: local-variables-portable-example #+begin_src org :eval never

Local Variables:

eval: (progn (setq-local orgstrap-min-org-version "8.2.10") (let ((a (org-version)) (n orgstrap-min-org-version)) (or (fboundp #'orgstrap--confirm-eval) (not n) (string< n a) (string= n a) (error "Your Org is too old! %s < %s" a n))) (defun orgstrap-norm-func--dprp-1-0 (body) (let ((p (read (concat "(progn\n" body "\n)"))) (m '(defun defun-local defmacro defvar defvar-local defconst defcustom)) print-quoted print-length print-level) (cl-labels ((f (b) (cl-loop for e in b when (listp e) do (or (and (memq (car e) m) (let ((n (nthcdr 4 e))) (and (stringp (nth 3 e)) (or (cl-subseq m 3) n) (f n) (or (setcdr (cddr e) n) t)))) (f e))) p)) (prin1-to-string (f p))))) (unless (boundp 'orgstrap-norm-func) (defvar-local orgstrap-norm-func orgstrap-norm-func-name)) (defun orgstrap-norm-embd (body) (funcall orgstrap-norm-func body)) (unless (fboundp #'orgstrap-norm) (defalias 'orgstrap-norm #'orgstrap-norm-embd)) (defun orgstrap-org-src-coderef-regexp (fmt &optional label) (let ((fmt org-coderef-label-format)) (format "\([:blank:]\(%s\)[:blank:]\)$" (replace-regexp-in-string "%s" (if label (regexp-quote label) "\([-a-zA-Z0-9][-a-zA-Z0-9_ ]\)") (regexp-quote fmt) nil t)))) (unless (fboundp #'org-src-coderef-regexp) (defalias 'org-src-coderef-regexp #'orgstrap-org-src-coderef-regexp)) (defun orgstrap--expand-body (info) (let ((coderef (nth 6 info)) (expand (if (org-babel-noweb-p (nth 2 info) :eval) (org-babel-expand-noweb-references info) (nth 1 info)))) (if (not coderef) expand (replace-regexp-in-string (org-src-coderef-regexp coderef) "" expand nil nil 1)))) (defun orgstrap--confirm-eval-portable (lang _body) (not (and (member lang '("elisp" "emacs-lisp")) (let ((body (orgstrap--expand-body (org-babel-get-src-block-info))) (body-normalized (orgstrap-norm body)) (content-checksum (intern (secure-hash orgstrap-cypher body-normalized)))) (eq orgstrap-block-checksum content-checksum))))) (unless (fboundp #'orgstrap--confirm-eval) (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-portable)) (let (enable-local-eval) (vc-find-file-hook)) (let ((ocbe org-confirm-babel-evaluate) (obs (org-babel-find-named-block "orgstrap"))) (if obs (unwind-protect (save-excursion (setq-local orgstrap-norm-func orgstrap-norm-func-name) (setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval) (goto-char obs) (org-babel-execute-src-block)) (when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval) (setq-local org-confirm-babel-evaluate ocbe)) (ignore-errors (org-set-visibility-according-to-property))) (warn "No orgstrap block."))))

End:

#+end_src

Here are the minimal local variables from the end of the file. #+name: local-variables-minimal-example #+begin_src org :eval never

Local Variables:

eval: (progn (setq-local orgstrap-min-org-version "8.2.10") (let ((a (org-version)) (n orgstrap-min-org-version)) (or (fboundp #'orgstrap--confirm-eval) (not n) (string< n a) (string= n a) (error "Your Org is too old! %s < %s" a n))) (defun orgstrap-norm-func--dprp-1-0 (body) (let ((p (read (concat "(progn\n" body "\n)"))) (m '(defun defun-local defmacro defvar defvar-local defconst defcustom)) print-quoted print-length print-level) (cl-labels ((f (b) (cl-loop for e in b when (listp e) do (or (and (memq (car e) m) (let ((n (nthcdr 4 e))) (and (stringp (nth 3 e)) (or (cl-subseq m 3) n) (f n) (or (setcdr (cddr e) n) t)))) (f e))) p)) (prin1-to-string (f p))))) (unless (boundp 'orgstrap-norm-func) (defvar-local orgstrap-norm-func orgstrap-norm-func-name)) (defun orgstrap-norm-embd (body) (funcall orgstrap-norm-func body)) (unless (fboundp #'orgstrap-norm) (defalias 'orgstrap-norm #'orgstrap-norm-embd)) (defun orgstrap--confirm-eval-minimal (lang body) (not (and (member lang '("elisp" "emacs-lisp")) (eq orgstrap-block-checksum (intern (secure-hash orgstrap-cypher (orgstrap-norm body))))))) (unless (fboundp #'orgstrap--confirm-eval) (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-minimal)) (let (enable-local-eval) (vc-find-file-hook)) (let ((ocbe org-confirm-babel-evaluate) (obs (org-babel-find-named-block "orgstrap"))) (if obs (unwind-protect (save-excursion (setq-local orgstrap-norm-func orgstrap-norm-func-name) (setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval) (goto-char obs) (org-babel-execute-src-block)) (when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval) (setq-local org-confirm-babel-evaluate ocbe)) (ignore-errors (org-set-visibility-according-to-property))) (warn "No orgstrap block."))))

End:

#+end_src

• Code :PROPERTIES: :CUSTOM_ID: code :END: ** =orgstrap= implementation This section contains the implementation of functions to calculate =orgstrap-block-checksum= and set it as a prop line local variable. It also contains functions to embed the bootstrapping code as an =eval:= local variable in the local variables list, along with other quality of life functionality for the user such as =orgstrap-mode=, =orgstrap-edit-mode=, and =orgstrap-init=.

[[info:elisp#File Local Variables][info:elisp#File Local Variables]] is a useful reference

*** Expand Testing =org-src-coderef-regexp= with =fboundp= in ref:orgstrap-expand-body is needed due to changes in the behavior of =org-babel-get-src-block-info= roughly around the =9.0= release.

The changes in behavior for =org-babel-get-src-block-info= are commits orgit-rev:/git/NOFORK/org-mode::88659208793dca18b7672428175e9a712af7b5ad and orgit-rev:/git/NOFORK/org-mode::9738da473277712804e0d004899388ad71c6b791. They both occur before the introduction of =org-src-coderef-regexp= in orgit-rev:/git/NOFORK/org-mode::9f47b37231b3c45afcd604a191e346200bd76e98. All of this happend before orgit-rev:/git/NOFORK/org-mode::release_9.0. By testing =org-src-coderef-regexp= with =fboundp= there are only a tiny number of versions where there might be some inconsistent behavior, e.g. orgit-rev:~/git/NOFORK/org-mode::release_8.3.6, but I suspect that the probability that anyone anywhere is running one of those versions is approximately zero.

#+name: orgstrap-expand-body #+begin_src elisp :eval never (defun orgstrap-org-src-coderef-regexp (_fmt &optional label) "Backport org-src-coderef-regexp' for 24 and 25. See the upstream docstring for info on LABEL. _FMT has the wrong meaning in 24 and 25." (let ((fmt org-coderef-label-format)) (format "\\([:blank:]*\\(%s\\)[:blank:]*\\)$" (replace-regexp-in-string "%s" (if label (regexp-quote label) "\\([-a-zA-Z0-9_][-a-zA-Z0-9_ ]*\\)") (regexp-quote fmt) nil t)))) (unless (fboundp #'org-src-coderef-regexp) (defalias 'org-src-coderef-regexp #'orgstrap-org-src-coderef-regexp)) (defun orgstrap--expand-body (info) "Expand noweb references in INFO body and remove any coderefs." ;; this is a backport of org-babel--expand-body' (let ((coderef (nth 6 info)) (expand (if (org-babel-noweb-p (nth 2 info) :eval) (org-babel-expand-noweb-references info) (nth 1 info)))) (if (not coderef) expand (replace-regexp-in-string (org-src-coderef-regexp coderef) "" expand nil nil 1)))) #+end_src *** Run In order for orgstrap to be maximally portable and not depend on already being installed, the implementation needs to work with the local variables list eval variable without complicating the situation when orgstrap is installed as a package.

While ideally this would be done using only the standard hooks around =hack-local-variables= such an approach does not work because the variables are filtered before those hooks can run. Therefore, we have to advise =hack-local-variables-confirm= in order to capture and remove any orgstrap elvs that we find. For maximum safety this minimally requires mutation of the =all-vars= list passed to =hack-local-variables-confirm=.

This is a fairly deep tampering with the way that hack-local-variables works, so special attention should be given when reviewing the security implications of any changes.

#+caption: run helpers #+name: orgstrap-run-helper-defuns #+begin_src elisp :noweb yes (require 'cl-lib)

;;;###autoload (defvar orgstrap-mode nil "Variable to track whether `orgstrap-mode' is enabled.")

(cl-eval-when (eval compile load) ;; prevent warnings since this is used as a variable in a macro (defvar orgstrap-orgstrap-block-name "orgstrap" "Set the default blockname to orgstrap by convention. This makes it easier to search for orgstrap if someone encounters an orgstrapped file and wants to know what is going on."))

(defvar orgstrap-default-cypher 'sha256 "The default cypher passed to `secure-hash' when hashing blocks.")

(defvar-local orgstrap-cypher orgstrap-default-cypher "Local variable for the cypher for the current buffer. If you change orgstrap-default-cypher' you should update this as well using setq-default' since it will not change automatically.") (put 'orgstrap-cypher 'safe-local-variable (lambda (v) (ignore v) t))

(defvar-local orgstrap-block-checksum nil "Local variable for the expected checksum for the current orgstrap block.") ;; `orgstrap-block-checksum' is not a safe local variable, if it is set ;; as safe then there will be no check and code will execute without a check ;; it is also not risky, so we leave it unmarked

(defconst orgstrap--internal-norm-funcs '(orgstrap-norm-func--prp-1-0 orgstrap-norm-func--prp-1-1 orgstrap-norm-func--dprp-1-0) "List internally implemented normalization functions. Used to determine which norm func names are safe local variables.")

(defvar-local orgstrap-norm-func-name nil "Local variable for the name of the current orgstrap-norm-func.") (put 'orgstrap-norm-func-name 'safe-local-variable (lambda (value) (and orgstrap-mode (memq value orgstrap--internal-norm-funcs)))) ;; Unless `orgstrap-mode' is enabled and the name is in the list of ;; functions that are implemented internally this is not safe

(defvar-local orgstrap-norm-func #'orgstrap-norm-func--dprp-1-0 "Dynamic variable to simplify calling normalizaiton functions. Defaults to `orgstrap-norm-func--dprp-1-0'.")

(defvar orgstrap--debug nil "If non-nil run `orgstrap-norm' in debug mode.")

(defgroup orgstrap nil "Tools for bootstraping Org mode files using Org Babel." :tag "orgstrap" :group 'org :link '(url-link :tag "README on GitHub" "https://github.com/tgbugs/orgstrap/blob/master/README.org"))

(defcustom orgstrap-always-edit nil "If non-nil command orgstrap-mode' activates command orgstrap-edit-mode'." :type 'boolean :group 'orgstrap)

(defcustom orgstrap-always-eval nil "Always try to run orgstrap blocks even when populating `org-agenda'." :type 'boolean :group 'orgstrap)

(defcustom orgstrap-always-eval-whitelist nil "List of files that should always try to run orgstrap blocks." :type 'list :group 'orgstrap)

(defcustom orgstrap-file-blacklist nil "List of files that should never run orgstrap blocks.

For files on the blacklist orgstrap-block-checksum' is removed from the local variables list so that the checksum will not be added to the safe-local-variable-values' list. If it were added it would then be impossible to prevent execution of the source block when `orgstrap-mode' is disabled.

This is useful when developing a block that modifies Emacs' configuration. NOTE this variable only works if `orgstrap-mode' is enabled." :type 'list :group 'orgstrap)

;; orgstrap blacklist

(defun orgstrap-blacklist-current-file (&optional universal-argument) "Add the current file to orgstrap-file-blacklist'. If UNIVERSAL-ARGUMENT is provided do not run orgstrap-revoke-current-buffer'." ;; It is usually better to revoke a checksum when its file is blacklisted since ;; it is easier for the user to add the checksum again when needed than it is ;; for them to revoke manually. The prefix argument allows users who know that ;; they only want to blacklist the file and not revoke to do so though such ;; cases are expected to be fairly rare.

;; FIXME blacklisting a bad file that has already been approved is painful ;; right now, you have to manually set enable-local-eval' to nil, load the ;; file, run this function, and then reset enable-local-eval'. (interactive "P") (unless universal-argument (orgstrap-revoke-current-buffer)) (add-to-list 'orgstrap-file-blacklist (buffer-file-name)) (customize-save-variable 'orgstrap-file-blacklist orgstrap-file-blacklist))

(defun orgstrap-unblacklist-current-file () "Remove the current file from `orgstrap-file-blacklist'." (interactive) (setq orgstrap-file-blacklist (delete (buffer-file-name) orgstrap-file-blacklist)) (customize-save-variable 'orgstrap-file-blacklist orgstrap-file-blacklist))

;; orgstrap revoke

(defun orgstrap-revoke-checksums (&rest checksums) "Delete CHECKSUMS or all checksums if nil from `safe-local-variables-values'." (interactive) (cl-delete-if (lambda (pair) (cl-destructuring-bind (key . value) pair (and (eq key 'orgstrap-block-checksum) (or (null checksums) (memq value checksums))))) safe-local-variable-values) (customize-save-variable 'safe-local-variable-values safe-local-variable-values))

(defun orgstrap-revoke-current-buffer () "Delete checksum(s) for current buffer from safe-local-variable-values'. Deletes embedded and current values of orgstrap-block-checksum'." (interactive) (let* ((elv (orgstrap--read-current-local-variables)) (cpair (assoc 'orgstrap-block-checksum elv)) (checksum-existing (and cpair (cdr cpair)))) (orgstrap-revoke-checksums orgstrap-block-checksum checksum-existing)))

(defun orgstrap-revoke-elvs () "Delete all approved orgstrap elvs from `safe-local-variable-values'." (interactive) (cl-delete-if #'orgstrap--match-elvs safe-local-variable-values) (customize-save-variable 'safe-local-variable-values safe-local-variable-values))

(define-obsolete-function-alias 'orgstrap-revoke-eval-local-variables #'orgstrap-revoke-elvs "1.2.4" "Replaced by the more compact `orgstrap-revoke-elvs'.")

;; orgstrap run helpers

<>

;; orgstrap-mode implementation

(defun orgstrap--org-buffer () "Only run when in org-mode' and command orgstrap-mode' is enabled. Sets further hooks." (when enable-local-eval ;; if enable-local-eval' is nil we honor it and will not run ;; orgstrap blocks natively, this matches the behavior of the ;; embedded elvs and simplifies logic for cases ;; where orgstrap should not run (e.g. when populating org-agenda') (advice-add #'hack-local-variables-confirm :around #'orgstrap--hack-lv-confirm) (unless (member (buffer-file-name) orgstrap-file-blacklist) (add-hook 'before-hack-local-variables-hook #'orgstrap--before-hack-lv nil t))))

(defun orgstrap--hack-lv-confirm (command &rest args) "Advise hack-local-variables-confirm' to remove orgstrap eval variables. COMMAND should be hack-local-variables-confirm' with ARGS (all-vars unsafe-vars risky-vars dir-name)." (advice-remove #'hack-local-variables-confirm #'orgstrap--hack-lv-confirm) (cl-destructuring-bind (all-vars unsafe-vars risky-vars dir-name) (cl-loop for arg in (if (member (buffer-file-name) orgstrap-file-blacklist) (cl-loop ; zap checksums for blacklisted for arg in args collect (if (listp arg) (cl-delete-if (lambda (pair) (eq (car pair) 'orgstrap-block-checksum)) arg) arg)) args) collect ; use `cl-delete-if' to mutate the lists in calling scope (if (listp arg) (cl-delete-if #'orgstrap--match-elvs arg) arg)) ;; After removal we have to recheck to see if unsafe-vars and ;; risky-vars are empty so we can skip the confirm dialogue. If we ;; do not, then the dialogue breaks the flow. (or (and (null unsafe-vars) (null risky-vars)) (funcall command all-vars unsafe-vars risky-vars dir-name))))

(defun orgstrap--before-hack-lv () "If orgstrap' is in the current buffer, add hook to run the orgstrap block." ;; This approach is safer than trying to introspect some of the implementation ;; internals. This hook will only run if there are actually local variables to ;; hack, so there is little to no chance of lingering hooks if an error occures (remove-hook 'before-hack-local-variables-hook #'orgstrap--before-hack-lv t) ;; XXX we have to remove elvs here since hack-local-variables-confirm' is not called ;; if all variables are marked as safe, e.g. via `orgstrap-whitelist-file' ;; FIXME other interactions between blacklist and whitelist may need to be handled here (setq file-local-variables-alist (cl-delete-if #'orgstrap--match-elvs file-local-variables-alist)) (add-hook 'hack-local-variables-hook #'orgstrap--hack-lv nil t))

(defun orgstrap--used-in-current-buffer-p () "Return t if all the required orgstrap prop line local variables are present." (and (boundp 'orgstrap-cypher) orgstrap-cypher (boundp 'orgstrap-block-checksum) orgstrap-block-checksum (boundp 'orgstrap-norm-func-name) orgstrap-norm-func-name))

(defmacro orgstrap--lv-common-with-block-name () "Helper macro to allow use of same code between core and lv impls." `(progn <>))

(defun orgstrap--hack-lv () "If orgstrap is present, run the orgstrap block for the current buffer." ;; we remove this hook here and we do not have to worry because ;; it is always added by `orgstrap--before-hack-lv' (remove-hook 'hack-local-variables-hook #'orgstrap--hack-lv t) (when (orgstrap--used-in-current-buffer-p) (orgstrap--lv-common-with-block-name) (when orgstrap-always-edit (orgstrap-edit-mode 1))))

(defun orgstrap--match-elvs (pair) "Return nil if PAIR matchs any elv used by orgstrap. Avoid false positives if possible if at all possible." (and (eq (car pair) 'eval) ;;(message "%s" (cdr pair)) ;; keep the detection simple for now, any eval lv that ;; so much as mentions orgstrap is nuked, and in the future ;; if orgstrap-nb is used we may need to nuke that too (string-match "orgstrap" (prin1-to-string (cdr pair)))))

;;;###autoload (defun orgstrap-mode (&optional arg) "A regional minor mode for org-mode' that automatically runs orgstrap blocks. When visiting an Org file or activating org-mode', if orgstrap prop line local variables are detect then use the installed orgstrap implementation to run the orgstrap block. If orgstrap embedded local variables are present, they will not be executed. orgstrap-mode' is not a normal minor mode since it does not run any hooks and when enabled only adds a function to org-mode-hook'. ARG is the universal prefix argument." (interactive "P") (ignore arg) (let ((turn-on (not orgstrap-mode))) (cond (turn-on (add-hook 'org-mode-hook #'orgstrap--org-buffer) (setq orgstrap-mode t) (message "orgstrap-mode enabled")) (arg) ; orgstrap-mode already enabled so don't disable it (t (remove-hook 'org-mode-hook #'orgstrap--org-buffer) (setq orgstrap-mode nil) (message "orgstrap-mode disabled")))))

;; orgstrap do not run aka `org-agenda' eval protection

(defun orgstrap--advise-no-eval-lv (command &rest args) "Advise COMMAND to disable elvs for files loaded inside it. ARGS vary by COMMAND.

If the elvs are disabled then orgstrap-block-checksum' is added to the ignored-local-variables' list for files loaded inside COMMAND. This makes it possible to open orgstrapped files where the elvs will not run without having to accept the irrelevant variable for `orgstrap-block-checksum'." ;; continually prompting users to accept a local variable when they ;; cannot inspect the file and when accidentally accepting could ;; allow unchecked execution at some point in the future is bad ;; better to simply pretend that the elvs and the block checksum ;; do not even exist unless the file is explicitly on a whitelist

;; orgstrapped files are just plain old org files in this context ;; since agenda doesn't use any babel functionality ... of course ;; I can totally imagine using orgstrap to automatically populate ;; an org file or update an org file using orgstrap to keep the ;; agenda in sync with some external source ... so need a variable ;; to control this (if orgstrap-always-eval (apply command args) (let* ((enable-local-eval (and args orgstrap-always-eval-whitelist (member (car args) orgstrap-always-eval-whitelist) enable-local-eval)) (ignored-local-variables (if enable-local-eval ignored-local-variables (cons 'orgstrap-block-checksum ignored-local-variables)))) (apply command args))))

(advice-add #'org-get-agenda-file-buffer :around #'orgstrap--advise-no-eval-lv) #+end_src *** Edit #+caption: edit helpers #+name: orgstrap-edit-helper-defuns #+begin_src emacs-lisp :results none :lexical yes :noweb yes ;;; edit helpers (defvar orgstrap--clone-stamp-source-buffer-block nil "Source code buffer and block for `orgstrap-stamp'.")

(defcustom orgstrap-on-change-hook nil "Hook run via before-save-hook' when command orgstrap-edit-mode' is enabled. Only runs when the contents of the orgstrap block have changed." :type 'hook :group 'orgstrap)

(defcustom orgstrap-use-minimal-local-variables nil "Set whether minimal, smaller but less portable variables are used. If nil then backward compatible local variables are used instead. If the value is customized to be non-nil then compact local variables are used and `orgstrap-min-org-version' is set accordingly. If the current version of org mode does not support the features required to use the minimal variables then the portable variables are used instead." :type 'boolean :group 'orgstrap)

;; edit utility functions (defun orgstrap--current-buffer-cypher () "Return the cypher used for the current buffer. The value is orgstrap-cypher' if it is bound otherwise orgstrap-default-cypher' is returned." (if (boundp 'orgstrap-cypher) orgstrap-cypher orgstrap-default-cypher))

<>

<>

(defun orgstrap--goto-named-src-block (blockname) "Goto org block named BLOCKNAME. Like `org-babel-goto-named-src-block' but non-interactive, does not use the mark ring, and errors if the block is not found." (let ((obs (org-babel-find-named-block blockname))) (if obs (goto-char obs) (error "No block named %s" blockname))))

(defmacro orgstrap--with-block (blockname &rest macro-body) "Go to the source block named BLOCKNAME and execute MACRO-BODY. The macro provides local bindings for four names: info', params', body-unexpanded', and body'." (declare (indent defun)) `(save-excursion (let* ((info (org-save-outline-visibility 'use-markers (orgstrap--goto-named-src-block ,blockname) (org-babel-get-src-block-info))) (params (nth 2 info)) (body-unexpanded (nth 1 info)) (body (orgstrap--expand-body info))) ,@macro-body)))

(defun orgstrap--update-on-change () "Run via the before-save-hook' local variable. Test if the checksum of the orgstrap block has changed, if so update the orgstrap-block-checksum' local variable and then run `orgstrap-on-change-hook'." (let* ((elv (orgstrap--read-current-local-variables)) (cpair (assoc 'orgstrap-block-checksum elv)) (checksum-existing (and cpair (cdr cpair))) (checksum (orgstrap-get-block-checksum))) (unless (eq checksum-existing (intern checksum)) (remove-hook 'before-save-hook #'orgstrap--update-on-change t) ;; for some reason tangling from a buffer counts as saving from that buffer ;; so have to remove the hook to avoid infinite loop (unwind-protect (save-excursion (undo) (undo-boundary) ; insert an undo boundary so that the ;; changes to the checksum are transparent to the user (undo) ; undo the undo above (orgstrap-add-block-checksum nil checksum) (run-hooks 'orgstrap-on-change-hook)) (add-hook 'before-save-hook #'orgstrap--update-on-change nil t)))))

(defun orgstrap--get-actual-params (params) "Filter defaults, nulls, and junk from src block PARAMS." (let ((defaults (append org-babel-default-header-args org-babel-default-header-args:emacs-lisp))) (cl-remove-if (lambda (pair) (or (member pair defaults) (memq (car pair) '(:result-params :result-type)) (null (cdr pair)))) params)))

(defun orgstrap-header-source-element (header-name &optional block-name &rest more-names) "Given HEADER-NAME find the element that provides its value. If BLOCK-NAME is non-nil then search for headers for that block, otherwise search for headers associated with the current block. If MORE-NAMES are provided return the value for each (or nil)." ;; get the current headers, see if the value is set anywhere ;; or if it is default, search for default anyway just to be sure ;; return nil if not found ;; when searching for any header go to the end of the src line ;; re-search-backward' from that point for :header-arg but not ;; going beyond the affiliated keywords for the current element ;; (if you can get affiliated keywords for the current element ;; that might simplify the search as well? check the impl for how ;; the actual values are obtained during execution etc) ;; when found use org-element-at-point' to obtain the element

;; in another function the operates on the element ;; the element will give start, end, value, etc. ;; find bounds of value from element or sub element ;; delete the value, replace with new value (ignore header-name block-name more-names) (error "Not implemented TODO"))

(defun orgstrap-update-src-block-header (name new-params &optional update) "Add header arguments to block NAME from NEW-PARAMS from some other block. Existing header arguments will NOT be removed if they are not included in NEW-PARAMS. If UPDATE is non-nil existing header arguments are updated." (let ((new-act-params (orgstrap--get-actual-params new-params))) (orgstrap--with-block name (ignore body body-unexpanded) (let ((existing-act-params (orgstrap--get-actual-params params))) (dolist (pair new-act-params) (cl-destructuring-bind (key . value) pair (let ((header-arg (substring (symbol-name key) 1))) (if (assq key existing-act-params) (if update (unless (member pair existing-act-params) ;; TODO remove existing ;; `org-babel-insert-header-arg' does not remove ;; and it is not trivial to find the actual location ;; of an existing header argument there are 4 places ;; that we will have to look and then in some cases ;; we will have to append even if we do find them (org-babel-insert-header-arg header-arg value) ;; This message works around the fact that we don't ;; have replace here, only append TODO consider ;; changing the way update works to be nil, replace, ;; or append once an in-place replace is implemented (message "%s superseded for block %s." key name)) (warn "%s already defined for block %s!" key name)) (org-babel-insert-header-arg header-arg value)))))))))

(unless (fboundp #'flatten-tree) ;; backwards compatibility for Emacs < 27 (defun flatten-tree (tree) (let (elems) (while (consp tree) (let ((elem (pop tree))) (while (consp elem) (push (cdr elem) tree) (setq elem (car elem))) (if elem (push elem elems)))) (if tree (push tree elems)) (nreverse elems))))

(defun orgstrap--check-portable-subset (body) "Ensure that BODY uses only symbols that are portable for prin1-to-string'." ;; XXX Note that [.] may diverge again because .asdf' can be read without ;; escaping the leading . whereas \?asdf' cannot. This is an artifact of ;; the c implementation of prin1' being reused to handle both chars. (let* ((l (flatten-tree (read (concat "(progn\n" body "\n)")))) (symbols (cl-remove-duplicates (cl-remove-if-not #'symbolp l))) (bads (cl-remove-if-not (lambda (s) (string-match "[.?]" (cl-subseq (symbol-name s) 1))) symbols))) (when bads (error "checksum failed: non-portable symbols detected: %S" bads))))

;; edit user facing functions (defun orgstrap-get-block-checksum (&optional cypher) "Calculate the `orgstrap-block-checksum' for the current buffer using CYPHER." (interactive) (orgstrap--with-block orgstrap-orgstrap-block-name (ignore params body-unexpanded) (orgstrap--check-portable-subset body) (let ((cypher (or cypher (orgstrap--current-buffer-cypher))) (body-normalized (orgstrap-norm body))) (secure-hash cypher body-normalized))))

(defun orgstrap-add-block-checksum (&optional cypher checksum) "Add orgstrap-block-checksum' to file local variables of current-buffer'.

The optional CYPHER argument should almost never be used, instead change the value of `orgstrap-default-cypher' or manually change the file property line variable. CHECKSUM can be passed directly if it has been calculated before and only needs to be set.

If orgstrap-save-developer-checksums' is non-nil then add the checksum to orsgrap-developer-checksums'." (interactive) (let* ((cypher (or cypher (orgstrap--current-buffer-cypher))) (orgstrap-block-checksum (or checksum (orgstrap-get-block-checksum cypher)))) (when orgstrap-block-checksum (save-excursion (add-file-local-variable-prop-line 'orgstrap-cypher cypher) (add-file-local-variable-prop-line 'orgstrap-norm-func-name orgstrap-norm-func) (add-file-local-variable-prop-line 'orgstrap-block-checksum (intern orgstrap-block-checksum))) (when orgstrap-save-developer-checksums (add-to-list 'orgstrap-developer-checksums (intern orgstrap-block-checksum)))) orgstrap-block-checksum))

(defun orgstrap-run-block () "Evaluate the orgstrap block for the current buffer." ;; bind to :orb or something like that (interactive) (save-excursion (orgstrap--goto-named-src-block orgstrap-orgstrap-block-name) (org-babel-execute-src-block)))

(defun orgstrap-clone (&optional universal-argument) "Set current block or orgstrap block as the source for `orgstrap-stamp'. If a UNIVERSAL-ARGUMENT is supplied then the orgstrap block is always used." ;; TODO consider whether to avoid the inversion of behavior around C-u ;; namely that nil -> always from orgstrap block, C-u -> current block ;; this would avoid confusion where unprefixed could produce both ;; behaviors and only switch when already on a src block (interactive "P") (let ((current-element (org-element-at-point)) (current-buffer (current-buffer))) (if (and (eq (org-element-type current-element) 'src-block) (not universal-argument)) (let ((block-name (org-element-property :name current-element))) (if block-name (setq orgstrap--clone-stamp-source-buffer-block (cons current-buffer block-name)) (warn "The current block has no name, it cannot be a clone source!"))) (if (orgstrap--used-in-current-buffer-p) (setq orgstrap--clone-stamp-source-buffer-block (cons current-buffer orgstrap-orgstrap-block-name)) (warn "orgstrap is not used in the current buffer!")))))

(defun orgstrap-stamp (&optional universal-argument overwrite) "Stamp orgstrap block via orgstrap-clone' to current buffer. If UNIVERSAL-ARGUMENT is \\='(16) aka (C-u C-u) this will OVERWRITE any existing block. If you are not calling this interactively all as (orgstrap-stamp nil t) for calirty. You cannot stamp an orgstrap block into its own buffer." (interactive "P") (unless (eq major-mode 'org-mode) (user-error "orgstrap-stamp' only works in org-mode buffers")) (unless orgstrap--clone-stamp-source-buffer-block (user-error "No value to clone! Use `orgstrap-clone' first")) (let ((overwrite (or overwrite (equal universal-argument '(16)))) (source-buffer (car orgstrap--clone-stamp-source-buffer-block)) (source-block-name (cdr orgstrap--clone-stamp-source-buffer-block)) (target-buffer (current-buffer))) (when (eq source-buffer target-buffer) (error "Source and target are the same buffer. Not stamping!")) (cl-destructuring-bind (source-body source-params org-adapt-indentation org-edit-src-content-indentation) (save-window-excursion (with-current-buffer source-buffer (orgstrap--with-block source-block-name (ignore body-unexpanded) (list body params org-adapt-indentation org-edit-src-content-indentation)))) (if (and (not overwrite) (member orgstrap-orgstrap-block-name (org-babel-src-block-names))) (warn "orgstrap block already exists not stamping!") (orgstrap--add-orgstrap-block source-body) ; FIXME somehow the hash is different !?!??! (orgstrap-update-src-block-header orgstrap-orgstrap-block-name source-params t) (orgstrap-add-block-checksum) ; I think it is correct to add the checksum here (message "Stamped orgsrap block from %s" (buffer-file-name source-buffer))))))

;;;###autoload (define-minor-mode orgstrap-edit-mode "Minor mode for editing with orgstrapped files." :init-value nil :lighter "" :keymap nil (unless (eq major-mode 'org-mode) (setq orgstrap-edit-mode 0) (user-error "`orgstrap-edit-mode' only works with org-mode buffers"))

(cond (orgstrap-edit-mode (add-hook 'before-save-hook #'orgstrap--update-on-change nil t)) (t (remove-hook 'before-save-hook #'orgstrap--update-on-change t)))) #+end_src

orgstrap-embed-normalization-code

is a potential future variable but for sanity

I am leaving it out for now because it is easier

to have a rule that says "always use orgstrap-embedded-norm-func"

and then we don't have to wonder about it, the size tradeoff can

be made by the user based on their use case

*** Dev #+caption: dev helpers #+name: orgstrap-dev-helper-defuns #+begin_src elisp ;;; dev helpers

(defcustom orgstrap-developer-checksums-file (concat user-emacs-directory "orgstrap-developer-checksums.el") "Path to developer checksums file." :type 'path :group 'orgstrap)

(defcustom orgstrap-save-developer-checksums nil ; FIXME naming "Whether or not to save checksums of orgstrap blocks under development." :type 'boolean :group 'orgstrap :set (lambda (variable value) (set-default variable value) (if value (add-hook 'orgstrap-on-change-hook #'orgstrap-save-developer-checksums) (remove-hook 'orgstrap-on-change-hook #'orgstrap-save-developer-checksums))))

(defvar orgstrap-developer-checksums nil ; not custom because it is saved elsewhere "List of checksums for orgstrap blocks created or modified by the user.")

(defun orgstrap--pp-to-string (value) "Ensure that we actually print the whole VALUE not just the summarized subset." (let (print-level print-length) (pp-to-string value)))

(defun orgstrap-revoke-developer-checksums (&optional universal-argument) "Remove all saved developer checksums. UNIVERSAL-ARGUMENT is a placeholder." (interactive "P") (ignore universal-argument) (setq orgstrap-developer-checksums nil) (orgstrap-save-developer-checksums t))

(defun orgstrap-save-developer-checksums (&optional overwrite) "Function to update orgstrap-developer-checksums-file'. If OVERWRITE is non-nil then overwrite the existing checksums." (interactive "P") (if orgstrap-save-developer-checksums (let* ((checksums orgstrap-developer-checksums) (buffer (find-file-noselect orgstrap-developer-checksums-file))) (with-current-buffer buffer (unwind-protect (progn (lock-buffer) (let* ((saved (and (not (= (buffer-size) 0)) (cadr (nth 2 (read (buffer-string)))))) ;; XXX NOTE saved is not used to updated orgstrap-developer-checksums' here ;; FIXME massively inefficient (combined (or (and (not overwrite) (cl-remove-duplicates (append checksums saved))) checksums))) ;; TODO do we need to check whether combined and saved are different? ;; (message "checksums: %s\nsaved: %s\ncombined: %s" checksums saved combined) (erase-buffer) (insert ";;; -- mode: emacs-lisp; lexical-binding: t --\n") (insert ";;; DO NOT EDIT THIS FILE IT IS AUTOGENERATED AND WILL BE OVERWRITTEN!\n\n") (insert (string-replace " " "\n" (orgstrap--pp-to-string (setq orgstrap-developer-checksums ',combined)))) (insert "\n;;; set developer checksums as safe local variables\n\n") (insert (orgstrap--pp-to-string '(mapcar (lambda (checksum-value) (add-to-list 'safe-local-variable-values (cons 'orgstrap-block-checksum checksum-value))) orgstrap-developer-checksums))) (pp-buffer) (indent-region (point-min) (point-max)) (save-buffer))) (unlock-buffer) (kill-buffer)))) (warn "No checksums were saved because orgstrap-save-developer-checksums' is not set.")))

#+end_src *** Init A note on filter aka =cl-remove-if-not= in =orgstrap--add-file-local-variables= at [[(clrin)]]. | emacs version | require | |---------------+---------| | < 24 | 'cl | | < 25 | 'cl-lib | | < 27 | 'seq | The most portable thing to do for now is =(require 'cl-lib)= since we don't currently support anything below 23. Then use =cl-remove-if-not=.

There is a similar issue with =pcase=, which is that in =emacs-24= the syntax was closer to =cl-case= when dealing with symbols. Since =cl-lib= is already in use, =cl-case= is the logical solution for portability.

Not all functionality works in older versions of Org. For example see [[(obubb-issue)][update block issue]] which is caused by the fact that org-babel-update-block-body is broken prior to revision orgit-rev:~/git/NOFORK/org-mode::7d6b8f51ec1993a66a385b98b2df42d0853fe289 which is not present in the versions of Org released with Emacs < 26.

We have to cache this result to avoid ocbe issues when tangling

XXX this also has to be manually converted to the : style because

old versions of org don't support :cache yes and we wind up with

circular dependencies

#+name: orgstrap-shebang-body-command #+begin_src elisp :results code :exports none :cache yes (orgstrap--with-block "orgstrap-shebang" body) #+end_src

XXX must keep this in sync manually to support old versions of org

#+name: orgstrap-shebang-body : "set -e "-C" "-e" "-e"\n{ null=/dev/null;} > "{null:=/dev/null}\"\n{ args=;file=;MyInvocation=;__p=(mktemp -d);touch {__p}/=;chmod +x {__p}/=;__op=$PATH;PATH=${__p}:PATH;} > \"{null}"\n$file =$MyInvocation.MyCommand.Source\n{ file=$0;PATH=__op;rm {__p}/=;rmdir {__p};} > \"{null}"\nemacs -batch -no-site-file -eval "(let (vc-follow-symlinks) (defun org-restart-font-lock ()) (defun orgstrap--confirm-eval (l _) (not (memq (intern l) '(elisp emacs-lisp)))) (let ((file (pop argv)) enable-local-variables) (find-file-literally file) (end-of-line) (when (eq (char-before) ?\^m) (let ((coding-system-for-read 'utf-8)) (revert-buffer nil t t)))) (let ((enable-local-eval t) (enable-local-variables :all) (major-mode 'org-mode) find-file-literally) (require 'org) (org-set-regexps-and-options) (hack-local-variables)))" "{file}\" -- {args} "${@}"\nexit\n<# powershell open"

#+caption: init helpers #+name: orgstrap-init-helper-defuns #+begin_src emacs-lisp :results none :lexical yes :noweb yes ;;; init helpers (defvar orgstrap-link-message "jump to the orgstrap block for this file" "Default message for file internal links.")

(defvar-local orgstrap--local-variables nil "Variable to capture local variables from `hack-local-variables'.")

;; local variable generation functions

(defun orgstrap--get-min-org-version (info minimal) "Get minimum org mode version needed by the orgstrap block for this file. INFO is the source block info. MINIMAL sets whether to use minimal local vars." (if minimal (let ((coderef (or (nth 6 info) org-coderef-label-format)) (noweb (org-babel-noweb-p (nth 2 info) :eval))) (if noweb "9.3.8" (let* ((body (or (nth 1 info) "")) (crrx (org-src-coderef-regexp coderef)) (pos (string-match crrx body)) (commented (and pos (string-match (concat (rx ";" (zero-or-more whitespace)) crrx) body)))) ;; FIXME the right way to do this is similar to what is done in ;; `org-export-resolve-coderef' but for now we know we are in elisp (if (or (not pos) commented) "8.2.10" "9.3.8")))) "8.2.10"))

(defun orgstrap--have-min-org-version (info minimal) "See if current version of org meets minimum requirements for orgstrap block. INFO is the source block info. MINIMAL is passed to `orgstrap--get-min-org-version'." (let ((actual (org-version)) (need (orgstrap--get-min-org-version info minimal))) (or (not need) (string< need actual) (string= need actual))))

(defun orgstrap--dedoc (sexp) "Remove docstrings from SEXP. WARNING mutates sexp!" (let ((m '(defun defun-local defmacro defvar defvar-local defconst defcustom))) (cl-loop for e in sexp when (listp e) do ; for expression in sexp when the expression is a list (or (and (memq (car e) m) ; is a form with docstrings (let ((n (nthcdr 4 e))) ; body after docstring (and (stringp (nth 3 e)) ; has a docstring (or (cl-subseq m 3) n) ; var or doc not last (orgstrap--dedoc n) ; recurse for nested ;; splice out the docstring and return t to avoid the other branch (or (setcdr (cddr e) n) t)))) ;; recurse e.g. for (when x (defvar y t)) (orgstrap--dedoc e)))) sexp)

(defun orgstrap--local-variables--check-version (info &optional minimal) "Return the version check local variables given INFO and MINIMAL." `( (setq-local orgstrap-min-org-version ,(orgstrap--get-min-org-version info minimal)) <>))

(defun orgstrap--local-variables--norm (&optional norm-func-name) "Return the normalization function for local variables given NORM-FUNC-NAME." (let ((norm-func-name (or norm-func-name (default-value 'orgstrap-norm-func)))) (cl-case norm-func-name (orgstrap-norm-func--dprp-1-0 '( <>)) (orgstrap-norm-func--prp-1-1 '( <>)) (orgstrap-norm-func--prp-1-0 (error "orgstrap-norm-func--prp-1-0' is deprecated. Please update orgstrap-norm-func-name' to `orgstrap-norm-func--prp-1-1'")) (otherwise (error "Don't know that normalization function %s" norm-func-name)))))

(defun orgstrap--local-variables--norm-common () "Return the common normalization functions for local variables." '( <>))

(defun orgstrap--local-variables--eval (info &optional minimal) "Return the portable or MINIMAL elvs given INFO." (let* ((minimal (or minimal orgstrap-use-minimal-local-variables)) (minimal (and minimal (orgstrap--have-min-org-version info minimal)))) (if minimal '( <>) '( ;(ref:elv-noweb-issue) ;; if you automatically reindent it will break these two <>

<>))))

(defun orgstrap--local-variables--eval-common () "Return the common eval check functions for local variables." ( ; quasiquote to fill in orgstrap-orgstrap-block-name' <>))

;; init utility functions

(defun orgstrap--new-heading-elisp-block (heading block-name &optional header-args noexport) "Create a new elisp block named BLOCK-NAME in a new heading titled HEADING. The heading is inserted at the top of the current file. HEADER-ARGS is an alist of symbols that are converted to strings. If NOEXPORT is non-nil then the :noexport: tag is added to the heading." (declare (indent 1)) (save-excursion (goto-char (point-min)) (outline-next-heading) ;; alternately outline-next-heading (org-meta-return) (insert (format "%s%s\n" heading (if noexport " :noexport:" ""))) ;;(org-edit-headline heading) ;;(when noexport (org-set-tags "noexport")) (move-end-of-line 1) (insert "\n#+name: " block-name "\n") (insert "#+begin_src elisp") (mapc (lambda (header-arg-value) (insert " :" (symbol-name (car header-arg-value)) " " (symbol-name (cdr header-arg-value)))) header-args) (insert "\n#+end_src\n")))

(defun orgstrap--trap-hack-locals (command &rest args) "Advice for hack-local-variables-filter' to do nothing except the following. Set orgstrap--local-variables' to the reversed list of read variables which are the first argument in the lambda list ARGS. COMMAND is unused since we don't actually want to hack the local variables, just get their current values." (ignore command) (setq-local orgstrap--local-variables (reverse (car args))) nil)

(defun orgstrap--read-current-local-variables () "Return the local variables for the current file without applying them." (interactive) ;; orgstrap--local-variables is a temporary local variable that is used to ;; capture the input to `hack-local-variables-filter' it is unset at the end ;; of this function so that it cannot accidentally be used when it might be stale (setq-local orgstrap--local-variables nil) (let ((enable-local-variables t)) (advice-add #'hack-local-variables-filter :around #'orgstrap--trap-hack-locals) (unwind-protect (hack-local-variables nil) (advice-remove #'hack-local-variables-filter #'orgstrap--trap-hack-locals)) (let ((local-variables orgstrap--local-variables)) (makunbound 'orgstrap--local-variables) local-variables)))

(defun orgstrap--add-link-to-orgstrap-block (&optional link-message) "Add an org-mode' link pointing to the orgstrap block for the current file. The link is placed in comment on the second line of the file. LINK-MESSAGE can be used to override the default value set via orgstrap-link-message'" (interactive) ; TODO prompt for message with C-u ? (goto-char (point-min)) (next-logical-line) ; required to get correct behavior? (let ((link-message (or link-message orgstrap-link-message))) (unless (save-excursion (re-search-forward (format "^# \[\[%s\]\[.+\]\]$" orgstrap-orgstrap-block-name) nil t)) ; XXX for some reason save-excursion fails so we have to reset (goto-char (point-min)) (next-logical-line) ; use logical-line to avoid issues with visual line mode (insert (format "# [[%s][%s]]\n" orgstrap-orgstrap-block-name (or link-message orgstrap-link-message))))))

(defun orgstrap--add-orgstrap-block (&optional block-contents) "Add a new elisp source block with #+name: orgstrap to the current buffer. If a block with that name already exists raise an error. Insert BLOCK-CONTENTS if they are supplied." (interactive) (let ((all-block-names (org-babel-src-block-names))) (if (member orgstrap-orgstrap-block-name all-block-names) (warn "orgstrap block already exists not adding!") (goto-char (point-max)) (insert "\n") (orgstrap--new-heading-elisp-block "Bootstrap" orgstrap-orgstrap-block-name '((results . none) (exports . none) (lexical . yes)) 'noexport) (goto-char (point-max)) (insert "\n** Local Variables :ARCHIVE:\n") (orgstrap--with-block orgstrap-orgstrap-block-name (ignore params body-unexpanded body) (when block-contents ;; FIXME `org-babel-update-block-body' is broken in < 26 (ref:obubb-issue) ;; for now warn and fail if the version is known bad NOTE trying to backport ;; is not simple because there are changes to the function signatures (if (string< org-version "8.3.4") (warn "Your version of Org is too old to use this feature! %s < 8.3.4" org-version) (org-babel-update-block-body block-contents))) nil))))

(defun orgstrap--lv-command (info &optional minimal norm-func-name) "Create the elvs for an orgstrapped file. INFO is the output of `org-babel-get-src-block-info' for the orgstrap block. MINIMAL determines whether a non-portable block has been requested. NORM-FUNC-NAME names the function used to normalize orgstrap blocks." (let ((lv-cver (orgstrap--local-variables--check-version info minimal)) (lv-norm (orgstrap--local-variables--norm norm-func-name)) (lv-ncom (orgstrap--local-variables--norm-common)) (lv-eval (orgstrap--local-variables--eval info minimal)) (lv-ecom (orgstrap--local-variables--eval-common))) (cons 'progn (orgstrap--dedoc (append lv-cver lv-norm lv-ncom lv-eval lv-ecom)))))

(defun orgstrap--add-file-local-variables (&optional minimal norm-func-name) "Add the file local variables needed to make orgstrap work. MINIMAL is used to control whether the portable or minimal block is used. If MINIMAL is set but the orgstrap block uses features like noweb and uncommented coderefs and function org-version' is too old, then the portable block will be used. NORM-FUNC-NAME is an optional argument that can be provided to determine which normalization function is used independent of the current buffer or global setting for orgstrap-norm-func'.

When run, this function replaces any existing orgstrap elv with the latest implementation available according to the preferences for the current buffer and configuration. Other elvs are retained if they are present, and the orgstrap elv is always added first." ;; switching comments probably wont work ? we can try ;; Use a prefix argument (i.e. C-u) to add file local variables comments instead of in a :noexport: (interactive) (let ((info (save-excursion (orgstrap--goto-named-src-block orgstrap-orgstrap-block-name) (org-babel-get-src-block-info))) (elv (orgstrap--read-current-local-variables))) (let ((lv-command (orgstrap--lv-command info minimal norm-func-name)) (commands-existing (mapcar #'cdr (cl-remove-if-not (lambda (l) (eq (car l) 'eval)) elv)))) ;(ref:clrin) (let* ((stripped (cl-remove-if (lambda (cmd) (orgstrap--match-elvs (cons 'eval cmd))) commands-existing)) (eval-commands (cons lv-command stripped))) (when commands-existing (delete-file-local-variable 'eval)) (let ((print-escape-newlines t) ; needed to preserve the escaped newlines ;; if print-length' or print-level' is accidentally set ;; add-file-local-variable' will truncate the sexp with and elispsis ;; this is clearly a bug in add-file-local-variable' and possibly in ;; something deeper, `print-length' is the only one that has actually ;; caused issues, but better safe than sorry print-length print-level) (mapcar (lambda (sexp) (add-file-local-variable 'eval sexp)) eval-commands))))))

(defun orgstrap--before-first-dull () "Goto the first non-empty line not starting with a sharp sign." (goto-char (point-min)) (re-search-forward "\n[^#\n \t]") (beginning-of-line))

(defun orgstrap--goto-elvs () "Goto the start of the elvs for the current buffer. If no elvs are found goto `point-max' instead." (widen) (goto-char (point-max)) (search-backward "\n^L" (max (- (point-max) 3000) (point-min)) 'move) (when (let ((case-fold-search t)) (search-forward "Local Variables:" nil t)) (beginning-of-line)))

(defconst orgstrap--shebang-body <<orgstrap-shebang-body()>> "Shebang block body content.")

(defun orgstrap--add-shebang-block (&optional update) "Add a shebang block to the current buffer." ;; goto correct location ;; create empty bash block ;; fill block ;; go to start of elvs ;; add powershell closing line (let ((block-name "orgstrap-shebang") (header-args '((eval . never) (results . none) (exports . none)))) (if (org-babel-find-named-block block-name) (if update (orgstrap-update-src-block "orgstrap-shebang" orgstrap--shebang-body) (warn "A shebang block already exists. Not adding.")) (if update (warn "A shebang block does not exist. Not updating.") (save-excursion (orgstrap--before-first-dull) (insert "\n#+name: " block-name "\n") (insert "#+begin_src bash") (mapc (lambda (header-arg-value) (insert " :" (symbol-name (car header-arg-value)) " " (symbol-name (cdr header-arg-value)))) header-args) (insert "\n#+end_src\n") (orgstrap-update-src-block "orgstrap-shebang" orgstrap--shebang-body)

      (orgstrap--goto-elvs)
      (insert (format "# close powershell comment %s>\n" "#")))))))

(defun orgstrap-update-shebang-block (&optional universal-argument) "Update an existing shebang block. UNIVERSAL-ARGUMENT is ignored." (interactive "P") (ignore universal-argument) (orgstrap--add-shebang-block 'update))

;; init user facing functions

;;;###autoload (defun orgstrap-init (&optional prefix-argument shebang) "Initialize orgstrap in a buffer and enable command `orgstrap-edit-mode'. If PREFIX-ARGUMENT is non-nil and has a value of 4 or 64 init will attempt to use the minimal local variables if possible.

If SHEBANG is non-nil or PREFIX-ARGUMENT is greater than or equal to 16 then a shebang block will also be added to the file.

Example usage. M-x orgstrap-init -> portable elvs C-u M-x orgstrap-init -> minimal elvs C-u C-u M-x orgstrap-init -> portable elvs + shebang C-u C-u C-u M-x orgstrap-init -> minimal elvs + shebang" (interactive "P") (unless (eq major-mode 'org-mode) (error "Cannot orgstrap, buffer not in `org-mode' it is in %s!" major-mode)) ;; TODO option for no link? ;; TODO option for local variables in comments vs noexport (let (onf) (let ((shebang (or shebang (and prefix-argument (>= (car prefix-argument) 16)))) (orgstrap-norm-func (or (cdr (assoc 'orgstrap-norm-func-name (orgstrap--read-current-local-variables))) (default-value 'orgstrap-norm-func)))) (save-excursion (orgstrap--add-orgstrap-block) (orgstrap-add-block-checksum) (orgstrap--add-link-to-orgstrap-block) ;; FIXME sometimes local variables don't populate due to an out of range error (orgstrap--add-file-local-variables (or (and prefix-argument (memq (car prefix-argument) '(4 64))) orgstrap-use-minimal-local-variables)) (when shebang (orgstrap--add-shebang-block)) (orgstrap-edit-mode 1) (setq onf orgstrap-norm-func))) ;; reset to ensure that a stale value is not inserted on next save (setq-local orgstrap-norm-func onf))) #+end_src

Note that multi-line strings cause issues with indentation if they are

nowebbed with leading whitespace. We avoid this by left aligning the

[[(elv-noweb-issue)][noweb references]] so that no leading whitespace

is inserted. This is something to watch out for in general when trying

to ensure consistent hashing.

I suspect that the underlying issue may be an org-mode bug related

to incorrect handling of leading whitespace when inserting contents

into a new block

dedoc testing

(orgstrap--dedoc '(defvar lol 'hello))

(orgstrap--dedoc '(defvar lol 'hello "there"))

(orgstrap--dedoc '(defun lol () "there" 1))

(orgstrap--dedoc '(defun lol (arg) "there" arg))

(orgstrap--dedoc '(defun lol (arg) "there"))

(orgstrap--dedoc '(defmacro lol (arg) "there"))

(orgstrap--dedoc '(defmacro lol (arg) "there" arg))

(orgstrap--dedoc '((defmacro lol (arg) "there" arg) (defvar lol 'hello "there")))

*** Extras #+caption: extra helpers #+name: orgstrap-extra-helper-defuns #+begin_src elisp :noweb yes ;;; extra helpers

(defun orgstrap-update-src-block (name content) "Set the content of source block named NAME to string CONTENT. XXX NOTE THAT THIS CANNOT BE USED WITH #+BEGIN_EXAMPLE BLOCKS." ;; FIXME this seems to fail if the existing block is empty? ;; or at least adding file local variables fails? (let ((block (org-babel-find-named-block name))) (if block (save-excursion (orgstrap--goto-named-src-block name) (org-babel-update-block-body content)) (error "No block with name %s" name))))

(defun orgstrap-get-src-block-checksum (&optional cypher) "Calculate of the checksum of the current source block using CYPHER." (interactive) (let* ((info (org-babel-get-src-block-info)) (params (nth 2 info)) (body-unexpanded (nth 1 info)) (body (orgstrap--expand-body info)) (body-normalized (orgstrap-norm body)) (cypher (or cypher (orgstrap--current-buffer-cypher)))) (ignore params body-unexpanded) (secure-hash cypher body-normalized)))

(defun orgstrap-get-named-src-block-checksum (name &optional cypher) "Calculate the checksum of the first sourc block named NAME using CYPHER." (interactive) (orgstrap--with-block name (ignore params body-unexpanded) (let ((cypher (or cypher (orgstrap--current-buffer-cypher))) (body-normalized (orgstrap-norm body))) (secure-hash cypher body-normalized))))

(defun orgstrap-run-additional-blocks (&rest name-checksum) ;(ref:oab) "Securely run additional blocks in languages other than elisp. Do this by providing the name of the block and the checksum to be embedded in the orgstrap block as NAME-CHECKSUM pairs." (ignore name-checksum) (error "TODO"))

(defun orgstrap--get-elvs (&optional from-flv-alist) "Return the elvs as they are written in the current buffer. If FROM-FLV-ALIST is not null display the elvs that are in `file-local-variables-alist'." (cl-loop for var in (if from-flv-alist file-local-variables-alist (orgstrap--read-current-local-variables)) when (orgstrap--match-elvs var) return (cdr var)))

(defun orgstrap-inspect-elvs (&optional from-flv-alist) "Display the elvs for the current buffer. If FROM-FLV-ALIST is not null display the elvs that are in `file-local-variables-alist'." (interactive "P") (let ((buffer (get-buffer-create (format "%s orgstrap elvs" (buffer-file-name)))) (elvs (orgstrap--get-elvs from-flv-alist)) (cypher orgstrap-cypher) print-length print-level) (with-current-buffer buffer (emacs-lisp-mode) (read-only-mode) (let ((inhibit-read-only t)) (erase-buffer) ;; TODO insert checksum in comment (insert ";; -- elvs-checksum: " (secure-hash cypher (orgstrap-norm (pp-to-string elvs))) "; --\n") (insert (pp-to-string elvs)) (goto-char (point-min)) (while (re-search-forward "(\(let\|defun\|when\|unless\|if\|read\)" nil t) (join-line 1)) (indent-region (point-min) (point-max))) (local-set-key (kbd "q") #'quit-window) (goto-char (point-min))) (display-buffer buffer)))

(defun orgstrap--whitelist-current-buffer () "Mark local variable values in the current buffer as safe." (let ((lvs (orgstrap--read-current-local-variables))) (customize-push-and-save 'safe-local-variable-values lvs)))

;; extra user facing functions

;;;###autoload (defun orgstrap-whitelist-file (path) "Add local variables in PATH as safe custom variable values. This is useful when distributing orgstrapped files.

Use with a command similar to the following. Since -batch implies -q, `user-init-file' must be passed explicitly.

emacs -batch -eval \ "(let ((user-init-file (pop argv)) (file (pop argv))) (package-initialize) (orgstrap-whitelist-file file))" \ ~/.emacs.d/init.el /path/to/whitelist.org" (let (enable-local-variables) ; < 28 don't run orgstrap block ;; `find-file-literally' is broken on 27 so regularize behavior (with-current-buffer (find-file-literally path) (orgstrap--whitelist-current-buffer) (kill-buffer)))) #+end_src

Ideally we want to call [[(oab)][orgstrap-run-additional-blocks]] as =(orgstrap-run-additional-blocks "additional-block-name" "checksum-value-hash-thing" "ab2" "cs2")= It probably makes sense to house this in its own orgstrap-aux block or something. I want to keep the file local variables as minimal as possible, so having another aux block that could be automatically updated with the names and hashes of additional blocks would be nice ... probably via something like =orgstrap-add-additional-block= but it will not go in the local variables because we want there to be some hope of orgstrap being portable to other platforms outside of Emacs at some point in the very distant future, so keeping the machinery outside of the org file itself as minimal as possible is critical. ** orgstrap.el :noexport:

XXX TODO it would be a super cool feature if xref could resolve to elisp source

blocks in org-mode files, because then half the need for the .el file would go away

#+caption: Retangle this if something changes. #+name: orgstrap.el #+header: :exports none #+begin_src elisp -r -l "([[:space:]]|;)(ref:%s)$" :noweb yes :eval never :tangle ./orgstrap.el ;;; orgstrap.el --- Bootstrap an Org file using file local variables -- lexical-binding: t -*-

;; Author: Tom Gillespie ;; URL: https://github.com/tgbugs/orgstrap ;; Keywords: lisp org org-mode bootstrap ;; Version: 1.5.6 (ref:orgstrap.el-version) ;; Package-Requires: ((emacs "24.4"))

;;;; License and Commentary

;; License: ;; SPDX-License-Identifier: GPL-3.0-or-later

;;; Commentary:

;; orgstrap is a specification and tooling for bootstrapping Org files.

;; It allows Org files to describe their own requirements, and ;; define their own functionality, making them self-contained, ;; standalone computational artifacts, dependent only on Emacs, ;; or other implementations of the Org-babel protocol in the future.

;; orgstrap.el is an elisp implementation of the orgstrap conventions. ;; It defines a regional minor mode for org-mode' that runs orgstrap ;; blocks. It also provides orgstrap-init' and `orgstrap-edit-mode' ;; to simplify authoring of orgstrapped files. For more details see ;; README.org which is also the literate source for this orgstrap.el ;; file in the git repo at ;; https://github.com/tgbugs/orgstrap/blob/master/README.org ;; or wherever you can find git:c1b28526ef9931654b72dff559da2205feb87f75

;; Code in an orgstrap block is usually meant to be executed directly by its ;; containing Org file. However, if the code is something that will be reused ;; over time outside the defining Org file, then it may be better to tangle and ;; load the file so that it is easier to debug/xref functions. The code in ;; this orgstrap.el file in particular is tangled for inclusion in one of the ;; *elpas so as to protect the orgstrap namespace and to make it eaiser to ;; use orgstrap in Emacs.

;; The license for the orgstrap.el code reflects the fact that the ;; code for expanding and hashing blocks reuses code from ob-core.el, ;; which at the time of writing is licensed as part of Emacs.

;;; Code:

(require 'org)

(require 'org-element)

<>

<>

<>

<>

<>

(provide 'orgstrap)

;;; orgstrap.el ends here

#+end_src

have to have an empty line at the end so that a newline shows up

when tangled ... surely this is a bug?

** Testing :noexport: *** Simple #+name: test-portable #+begin_src bash :var THIS_FILE=(buffer-file-name) :results none emacs-25 -Q $THIS_FILE emacs-26 -Q$THIS_FILE emacs-27 -Q $THIS_FILE emacs-28 -Q$THIS_FILE emacs-29 -Q $THIS_FILE emacs-30 -Q$THIS_FILE #+end_src

#+name: test-minimal #+begin_src bash :var THIS_FILE=(buffer-file-name) :results none emacs-25 -Q orgstrap-minimal.org emacs-26 -Q orgstrap-minimal.org emacs-27 -Q orgstrap-minimal.org emacs-28 -Q orgstrap-minimal.org emacs-29 -Q orgstrap-minimal.org emacs-30 -Q orgstrap-minimal.org #+end_src *** Matrix Before running the tests below you need to generate [[file:./orgstrap-autoloads.el]]. Newer version of =autoload-generate-file-autoloads= add functions that may not be supported by older versions of Emacs. Thus you should run this on the oldest version of Emacs you will be testing against.

XXX This must be run with emacs-26 to avoid bytecode compatibility issues

#+name: generate-autoloads-for-test #+begin_src elisp :results none (require 'autoload) (with-current-buffer (find-file-noselect "orgstrap-autoloads.el") (erase-buffer) (let* ((cb (current-buffer)) (fn (buffer-file-name cb)) (generated-autoload-file fn)) (autoload-generate-file-autoloads "orgstrap.el" cb fn)) (save-buffer) (kill-buffer)) #+end_src

XXX note that you cannot use append t with add-to-list for local variables

it puts any additional values in the wrong place, need to check other cases

reminder that you can't use bare bangs ! anywhere in bash scripts :/

XXX also reminder that unbound variables will break at export time

XXX remove elc files before running this due to bytecode mismatches

XXX regenerate autoloads, it seems that the version in 26 works for everyone

#+name: test-matrix-run #+begin_src bash :results drawer output versions=( 25 26 27 28 29 30 ) test_files=( test-no-lv-list.org test-lv-list-portable test-lv-list-minimal ) for v in ${versions[@]}; do [ -d test/emacs-$v ] || mkdir -p test/emacs-$v for f in${test_files[@]};do

uncomment and reorder to debug tests

#f=test-lv-list-minimal #emacs -Q
emacs-$v -Q -batch \ -eval "(setq user-init-file (concat default-directory \"test/emacs-${v}/init.el"))"
-l orgstrap-autoloads.el
-eval "(message "\n%s"(emacs-version))"
-f toggle-debug-on-error
-eval "(add-to-list 'load-path "(pwd)/\")" \ -eval "(orgstrap-mode)" \ -eval "(defun orgstrap-test () (error \"test failed.\"))" \ -eval "(orgstrap-whitelist-file \"{f}")"
-visit $f
-eval "(orgstrap-test)" 2>&1 done done #+end_src **** TODO Full matrix #+begin_src elisp ;; create buffer ;; fill buffer ;; set code ;; hash ;; save buffer ;; open in all the other impls having set the checksum as accepted ;; with orgstrap-mode enabled ;; without orgstrap-mode enabled ;; with minimal ;; with portable ;; with noweb ;; without noweb ;; iterate over norm funcs ;; orgstrap-norm-func-name mismatch ;; orgstrap-norm-func-name not in internal list ;; .org extension and mode: org local variable (defconst orgstrap--test-matrix `(((orgstrap-mode (nil t)) (lv-type (nil minimal portable)) (noweb (nil t)) ;; (comments (nil link noweb)) ; comments aren't actually relevant here I think? (norm-func ,orgstrap--internal-norm-funcs) (path-suffix-lv ((".org" . (mode . org)) (".org" nil) ("" . (mode . org)) ("" . (mode . nil)))))) "The dimensions of the test files that need to be generated." ) #+end_src **** TODO One time tests

1. Blacklist and open, then unblacklist and open. This requires an actual file since we need buffer file name.
2. Need a way to test revoke as well. **** Test files

TODO automatically generate all of these test files

maybe even generate them as buffers that can be run

by loading this files with orgstrap--test t?

#+header: :comments link # broken atm

#+name: test-no-lv-list #+begin_src org :tangle ./test-no-lv-list.org

-- orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: 8d941e14e89664b834f5b28c070f9f7b0ec55b092b55cc23dd903c010fdaeda5; --

[[orgstrap][jump to the orgstrap block for this file]]

,* Bootstrap :noexport:

,#+name: orgstrap ,#+begin_src elisp :results none :lexical yes (defun orgstrap-test () (if (cl-remove-if-not #'orgstrap--match-elvs file-local-variables-alist) (error "elv is still present!") (message "No local variables here!"))) (message "orgstrap successful") ,#+end_src #+end_src

#+name: test-lv-list-portable #+begin_src org :tangle ./test-lv-list-portable :noweb yes

-- mode: org; orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: 14e85d1213ef7a6739ca6ca7361a227b0a55346d4c7c6457bdd5f7ba91ff5dff; --

[[orgstrap][jump to the orgstrap block for this file]]

,* Bootstrap :noexport:

,#+name: orgstrap ,#+begin_src elisp :results none :lexical yes (defun orgstrap-test () (if (cl-remove-if-not #'orgstrap--match-elvs file-local-variables-alist) (error "elv is still present!") (message "Portable local variables here!"))) (message "orgstrap successful") ,#+end_src

,** Local Variables :ARCHIVE:

<> #+end_src

#+name: test-lv-list-minimal #+begin_src org :tangle ./test-lv-list-minimal :noweb yes

-- mode: org; orgstrap-cypher: sha256; orgstrap-norm-func-name: orgstrap-norm-func--dprp-1-0; orgstrap-block-checksum: 3008580fd616cdfca904c7508ae023f782585229a87adab33fb8c2d391f89561; --

[[orgstrap][jump to the orgstrap block for this file]]

,* Bootstrap :noexport:

,#+name: orgstrap ,#+begin_src elisp :results none :lexical yes (defun orgstrap-test () (if (cl-remove-if-not #'orgstrap--match-elvs file-local-variables-alist) (error "elv is still present!") (message "Minimal local variables here!"))) (message "orgstrap successful") ,#+end_src

,** Local Variables :ARCHIVE:

<> #+end_src ** Release :noexport: *** Flycheck Use flycheck-mode on [[file:./orgstrap.el]] to checkdoc for melpa. Don't forget to run flycheck-package-setup to get better reports. *** Byte compile Before a release run the following block and fix any byte compile errors and warnings. Using Emacs 26 ensures that bytecode is forward and backward compatible. #+begin_src elisp :noweb yes :results drawer (ow-run-command "emacs" "-batch" "-f" "batch-byte-compile" "orgstrap.el") (ow-run-command "emacs-27" "-batch" "-f" "batch-byte-compile" "orgstrap.el") (delete-file "orgstrap.elc") #+end_src *** Run test matrix Run ref:test-matrix-run. #+call: test-matrix-run() *** Run init tests These are not automated at the moment. Run ref:test-portable and do the following for each version of Emacs.

1. Accept lvs.
2. For Emacs >= 26 orgstrap-clone.
3. switch to scratch
4. enable org-mode
5. orgstrap-init
6. optional edit block
7. optional save to file and test reload the saved file
8. For Emacs >= 26 in scratch buffer undo
9. For Emacs >= 26 orgstrap-stamp
10. For Emacs >= 26 check that the checksum matches the checksum for this file
11. quit

#+call: test-portable() *** Final steps Things that need to be done for a release.

• Bump the version number in the [[(orgstrap.el-version)][orgstrap.el header comment]]. You will need to manually retangle after this step.
• Update the [[#changelog][changelog]].
• Convert the changelog entry to markdown for the GitHub release. =C-c C-e C-s m M=.

• Changelog :PROPERTIES: :CUSTOM_ID: changelog :END: ** 1.5.6

• Update orgstrap--shebang-body for Emacs 30.1 and Org 9.7.11.

  Workaround for issue where org-restart-font-lock is not defined when defcustom tries to run org-link--set-link-display.

** 1.5.5

• Fix missing paren.

  Missed due to a duplicate orgstrap block for debug in the same file.

** 1.5.4

• Update orgstrap--shebang-body to set find-file-literally to nil.

  If you use shebang blocks you should update them so that a call to find-file on buffer-file-name of the orgstrapped file will continue without prompting the user.

** 1.5.3

• Ignore errors from org-set-visibility-according-to-property.

  Recent changes to org-set-visibility-according-to-property result in errors if it is called when a file is not in org-mode and/or when a buffer is in org-mode but Emacs is noninteractive.

  This requires and update to the elvs which wraps the call in ignore-errors which is more space efficient than e.g. testing whether we are non-interactive and also mostly +fool+ future proof.

  If you use orgstrap shebang blocks you should updated your elvs.

** 1.5.2

• Update orgstrap--shebang-body to simplify and improve safety.

  If you use shebang blocks you should update them so that shebang blocks are not affected by leaking environment variables. See [[./shebang.org]] for a full explication of the operation of each line of shell/powershell code.

• Add ability to update shebang blocks to latest version.

  New interactive function orgstrap-update-shebang-block updates an existing shebang block to the latest orgstrap--shebang-body. Internally orgstrap--add-shebang-block was updated to accept an optional update argument which is used to control whether to add a new or update an existing block.

** 1.5.1

• Update orgstrap--shebang-body so that shebang blocks pass args correctly.

  If you use shebang blocks you should update them so that they pass the correct args to emacs. See [[./shebang.org]] for more details on the changes.

** 1.5

• Regularize behavior of orgstrap-whitelist-file.

  An internal call to find-file-literally has different behavior in 27 vs 28 (see notes about that in [[./shebang.org]]).

• Rename all normalization functions to remove use of =.=.

  See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=55645 for details.

  You will need to update any orgstrapped files to use the new names. The simplest way to do this is to update to to 1.5 and then delete the prop line local variables and then run orgstrap-init.

• Add a check to ensure that only portable symbol names are used.

  Symbols with =[.?]= appearing anywhere other than at the start of the symbol are banned since their prin1 representation diverges across the 28/29 boundary.

  If you have orgstrap blocks that contain such symbols you will need to change the symbol names. An error will be raised when calling orgstrap-add-block-checksum with a list of symbols that need to be changed. ** 1.4

• Update orgstrap-init so that it can insert a shebang block.

  Use 2 or more prefix arguments to add a shebang block when using orgsgrap-init. For example =C-u C-u C-u M-x orgstrap-init=. ** 1.3

• Remove all orgstrap-do variables.

  The core of orgstrap is not the right place to maintain these. They will reappear under ow-do in the future.

• Add orgstrap-norm-func--dprp-1.0 and make it the default norm func.

  The default normalization function for orgstrap is now invariant to changes in the docstring for defun, defun-local, defmacro, defvar, defvar-local, defconst, and defcustom. This allows improved documentation without requiring the user to re-audit.

  Note that orgstrap-norm-func--prp-1.1 has NOT been deprecated, but is no longer the default. It is still useful if for whatever reason you want to minimize the elvs.

• Add support for batch execution.

  The preferred method is to use an org shebang block (see [[./shebang.org]]) It is also possible to maintain an automatically updating list of developer checksums. This approach was deemed to be silly given shebang blocks, however the functionality is retained.

• Add orgstrap-whitelist-file to make it easier to mark known safe files in batch.

  See the docstring for example usage.

• Add orgstrap-inspect-elvs to inspect the elvs for the current buffer.

  The command also calculates the elvs checksum for comparison.

  Known elv checksums for this release are below. The order is minimal, minimal-noweb, and portable (aka minimal-noweb-eval).

  For prp-1.1 \ =446d0c80d72bb89dd149181e6a24eafa011d12d6dc99fad958a03ddebd9a95ad= \ =b294539a74f2a1932d39790d6377a4229bd3d5e84df64d968baa8ff3f85349cc= \ =543e3400c80e2cc7b9bf94b1799d1460b240776c2c7415a2f6c0c7a9507978ef= \

  For dprp-1.0 \ =aa080a6469c22dfe960c43fa3bff3b92a6bc3da9383ec8fcd7d0a019192e7aa0= \ =72c52a3483905aff6b83c6cd2c36899a2a8d1cbc603b6e5ce8c9e98f0dd7b099= \ =9e33bc67b8850147962edcece4ea1193bb6cf3711264a26bde91b1f838912ffc= \

• Fix org-edit-mode so that it now activates correctly.

• Fix orgstrap-init so that it no longer misplaces the link to the orgstrap block if there is already content in the buffer.

• Fix orgstrap-init so that invocation in files with existing elvs updates only the existing orgstrap elv and preserves other elvs.

• Fix orgstrap-init to read and pass the current value of orgstrap-norm-func-name when creating local variables.

  If orgstrap-norm-func-name is missing, the default value of orgstrap-norm-func is used.

  This prevents klobbering while also providing an easy way to update the normalization function --- just update the variable value and run orgstrap-init.

• Fix orgstrap-norm-func by always declaring it with defvar-local.

• Update the elvs to handle issues with symlinks and vc mode. The core functionality remains compatible.

• Update the elvs so that org-confirm-babel-evaluate can be set by an orgstrap block without having to modify the elvs.

• Update the elvs so that they only restore visibility set via property drawers.

  This makes it possible to use the orgstrap block to control initial visibility and narrowing to simplify the presentation of orgstrapped files (and avoid distracting users with the orgstrap machinery).

• Update orgstrap-init to put the Bootstrap section at the end of the file and to put the elvs in an archived heading inside that.

  This pattern has been found to be quite effective for a number of different use cases.

** 1.2.7

• Fix bad defaults on orgstrap-do-* custom variables.

  If these are not set to t by default then it is impossible individual blocks to know whether a nil value was intentional on the part of the user or not. Users must set values to nil in their config if they do not want certain sections to run. ** 1.2.6

• Ignore orgstrap-block-checksum when loading org-agenda.

  If enable-local-eval is t (following the behavior described in the [[#122][1.2.2]] changelog), then orgstrap-block-checksum is not ignored and if the local variable value has not been added to the safe list then the user will be prompted.

• Add orgstrap-do-* variables.

  Boolean control variables that can be used enable/disable standard functionality/steps needed by org files. See [[file:./do.org]] for more details. ** 1.2.5

• Improve behavior of orgstrap-blacklist-current-file.

  Now revokes the current buffer checksum by default, this can be overridden by providing a universal argument. The blacklist is immediately saved via customize-save-variable.

• Fix orgstrap-mode to use the universal argument.

  Behavior is now correct when (orgstrap-mode t) is called. ** 1.2.4

• Add orgstrap-clone and orgstrap-stamp commands.

  orgstrap-clone stores the current buffer and current or orgstrap block orgstrap-stamp copies the expanded contents and headers of that block to a new orgstrap block in a new file. Useful in cases where users want to duplicate functionality in a new file.

  NOTE orgstrap-stamp only works for org version >= 8.3.4 which means that it does not work for versions of Emacs < 26.

• orgstrap--add-orgstrap-block add block-contents argument.

  This simplifies the implementation of stamp, and makes it possible to set the initial contents of the orgstrap block programmatically. There are some lingering issues with indentation that may need to be resolved for this to work seamlessly.

• Fix byte compile bug from destructuring-bind not being aliased.

• Fix for issues with noweb blocks containing multi-line docstrings.

  We need this to test orgstrap-clone with this readme file. Without it the checksum will not match in the stamped file due to differences in the leading whitespace in docstrings.

• Make orgstrap-revoke-eval-local-variables obsolete.

  Replaced by the more compact orgstrap-revoke-elvs. ** 1.2.3

• Add orgstrap-file-blacklist to block eval of specific files.

  The functionality only works if orgstrap-mode is enabled. ** 1.2.2 :PROPERTIES: :CUSTOM_ID: 122 :END:

• Do not run eval local variables when loading org-agenda.

  orgstrap-always-eval can be set to t by users who want to evaluate orgstrap blocks in all situations. More granular control is provided by adding the full path of files that should always try to run their orgstrap block to orgstrap-always-eval-whitelist.

  In all cases, if the global setting for enable-local-eval is more restrictive then it is honored (i.e., nil will block any execution and the default 'maybe will continue to prompt).

• Add ability to revoke previously approved orgstrap-block-checksums.

  Rapid revocation of permissions is an important part of any security system. Therefore we now provide a way to revoke all previously approved values for orgstrap-block-checksum in a single command orgstrap-revoke-checksums. This command can also be provided with a specific list of checksums to revoke. Another convenience function orgstrap-revoke-current-buffer is provided that revokes the checksum of the orgstrap block for the current buffer.

• Add ability to revoke previously approved eval local variables.

  As with revocations for orgstrap-block-checksum values, we also need a way to revoke eval local variables. There is no granular control. Use orgstrap-revoke-eval-local-variables to nuke all orgstrap eval local variables from orbit. ** 1.2.1

• Fix bad startup visibility when using orgstrap.

  You should run =M-:= =(orgstrap--add-file-local-variables)= to update embedded eval local variables.

  Running =org-babel-execute-src-block= changes the visibility of the tree holding the orgstrap block. As a result the startup visibility of any org file using orgstrap was incorrect. Adding a call to =org-set-startup-visibility= in the unwind forms ensures that startup visibility is correct. ** 1.2

• Add =orgstrap-norm-func--prp-1.1= and make it the default norm func.

  This change does not effect the default behavior of =orgstrap=. The reason for the change is to defensively shadow =print-length= and =print-level= to =nil= so that if they are somehow non-nil, Emacs will not truncate the contents of the src block prior to hashing.

• Mark =orgstrap-norm-func--prp-1.0= as obsolete.

  You should update any files using prp-1.0 to use prp-1.1.

  Whenever there is a case where a change in the environment can cause a change in the output of a normalization function there is a risk that it could be exploited. ** 1.1.1

• Fix =orgstrap--hack-lv= to remove itself from the local =hack-local-variables-hook=. ** 1.1 :PROPERTIES: :CUSTOM_ID: 11 :END:

• Renamed existing =orgstrap-mode= to =orgstrap-edit-mode=.

  This is a BREAKING CHANGE. Please update your workflows.

• Added the new =orgstrap-mode= implementation.

  This is a regional minor mode for =org-mode= which makes it possible to use orgstrap without the embedded local variables. This allows for greater security at the expense of portability, depending on the exact use case. By a stroke of good fortune it is possible to use the =hack-local-variables= hooks to trap and remove the embedded local variables if they are present so that the orgstrap block is not evaluated twice.

• Added =orgstrap-always-edit= as a custom variable.

  If non-nil then =orgstrap-edit-mode= will be automatically activated by =orgstrap-mode=.

• =orgstrap--add-file-local-variables= update existing =eval:= vars.

  This change makes it vastly easier to switch between portable and minimal implementations, and should make it easier to switch the normalization function once we get that implemented.

  If an existing orgstrap eval file local variable is detected it is removed and the latest version is added. Other =eval:= variables are not modified. Note however that the orgstrap eval variable will always be placed first.

• Contributing :PROPERTIES: :CUSTOM_ID: contributing :END: The primary =orgstrap= repository lives at https://github.com/tgbugs/orgstrap.

There are a number of ways to contribute to =orgstrap=.

1. Have an Org file that use =orgstrap=? Create an issue or a pull request to add it to the list of [[#examples-from-around-the-web][examples from around the web]].

2. Encounter a bug? Please [[https://github.com/tgbugs/orgstrap/issues/new/choose][submit an issue]]!

3. Feel like writing some elisp? Check out [[#future-work][future work]] for a list of potential projects (TODO status not visible on GitHub).

• Guides ** User guide This guide is for users of Org files that have =orgstrap= blocks.

This includes Emacs users who have their own configs as well as users who might be encountering Emacs for the first time.

** Developer guide This guide is for developers who want to use =orgstrap= in their own org files.

It covers workflows for development, distribution, and maintenance of orgstrap files.

It also covers best practices and effective strategies for making Org files accessible to users. *** There can be only one. Dealing with elisp's absent namespaces. A key issue that orgstrap must contend with is the fact that there is only one global namespace for all elisp functions. Variables are not an issue for orgstrap because buffer local variables provide sufficient separation.

To this end there are two conventions that orgstrap and orgware follow. The ersatz namespaces orgstrap--- and ow--- may be used in any orgstrap block. Developers may assume that any such definitions will remain unchanged at least until another orgstrap block runs. Clearly it is impossible to know for sure that no one else will use a function with the same name ow---you-re-standing-on-my-toe-! that has different behavior. We can make an attempt to keep a record of all known ow--- function names, but ultimately it is up to the user to run a check.

TODO need to implement a local ow--- collision checker that searchers

all orgs files for ow--- usage and reports any collisions

TODO implement a way to report a name collision to upstream

** Contributor guide This guide is for developers who want to contribute to the core =orgstrap= implementation or documentation. *** Setup *** Testing *** Making changes *** Submitting a pull request

• Best practices :PROPERTIES: :CUSTOM_ID: best-practices :END: ** Accepting local variables In short. If you use orgstrap.el don't accept the elvs.

Suggestions for users. If you run an orgstrapped file via -Q or similar, then you have to accept the file. The elvs probably aren't going to include package initialize and require orgstrap, but maybe they could. Essentially, in -Q mode the user should know that they have to initialize it all themselves (orgware could do it for them).

When you aren't running in -Q mode and you DO have orgstrap.el installed on your system then I strongly suggest that you should NOT accept, or even actively remove, any and all approved elvs and use orgstrap-mode since it has a number of features that can enhance the security of execution such as blacklists etc.

** Use the system package manager. There is a big difference between using a script to install a program directly from the internet and using a script to ask the host system to install a program.

Even if you audit a random script from the internet it is unlikely that you will be able to do due diligence. On the other hand, if you ask your system package manager to install something for you, there is a much better chance that it has at least been somewhat audited, and there is usually an existing process for getting a package into the system which helps to mitigate certain types of attacks.

To give a military example it is the difference between inspecting and accepting a package from a random person because they say you asked for it yesterday (maybe you did!) versus only every allowing packages to come through procurement. You are much less likely to get a bomb or a packaged rigged to exfil data if you go through procurement because there is an established process for how to do things and that process enshrines generations experience about how to not get blown up by the pizza guy.

So, if you are writing instructions that require a certain tool, it is better to tell whoever is following them to ask procurement to get the tool for them than to tell them to going out to the hardware store and get it themselves, or worse, give them the address of a random tool delivery man who happens to be a good buddy of yours. Even if everyone involved is trustworthy those kinds of relationships are much easier for some third party to compromise and use for their own purposes.

The obvious corollary when you are the user rather than the author, is that if you encounter instructions that ask you to directly install software from a random place you should be suspicious, even, perhaps especially, if that random place is housed within a larger reputable site. If you're not in a hurry, ask for the software to be packaged, or package it yourself so that it can go through the process. ** Opening orgstrapped files as an Emacs user One problem that orgstrap has is that an orgstrap block can modify the Emacs configuration. Modifying the config is often critical to get the desired behavior for the particular target user population. This is not an issue if the users do not use Emacs for anything else. However, if the user opening the file is an Emacs user then blocks that modify the configuration are a serious problem.

There are three ways around this issue: one a standard convention for naming a variable to control evaluation of config related code or two always use emacs -q and of course three a combination of both.

In the first case a set of standard conventions for variable names can be used to control whether some, or all configuration variables are set. However, this can only attain the status of a best practice because orgstrap blocks run arbitrary code and there is no way to enforce the convention (thus why it is in this section).

One convention that I have been testing is to ...

TODO enable-*

orgstrap-enable-config ? not a good name ...

In the second case we suggest that users always open orgstrapped files with emacs -q. This is as close as we can get to having a sandboxed execution environment. If the user also wants to load their config then they would have to use -l as well. This is a pain, but without converting all of the variables into buffer local variables this is unlikely to work.

Another major drawback of using emacs -q is that there are many more advanced features enabled by orgstrap.el that cannot be used without running emacs -q -l orgstrap.el or some equivalent. This adds significant complexity to the command line invocation. There is no easy way to work around this since the features of interest are ones that must be available before the orgstrap block is run. Another approach is to use emacs -q -f package-initialize.

There is also the issue of how to handle potential name collisions.

TODO orgstrap--- prefix is too long, but is one solution

In summary, Emacs and orgstrap are very sharp tools. I have tried to provide a bit of protection via the checksum mechanism, however if you are an Emacs user, you should probably always check the orgstrap block to make sure that it won't completely klobber your config.

If you are authoring a file that uses orgstrap, it is friendly to put any config related code in its own block so that it can be controlled globally via the-orgstrap-variable-name-to-be-determined.

• Bootstrapping to Emacs, bootstrapping to Org :PROPERTIES: :CUSTOM_ID: bootstrapping-to-emacs-bootstrapping-to-org :END: See [[file:./get-emacs.org]].
• Examples :PROPERTIES: :CUSTOM_ID: examples :END: ** Examples from around the web :PROPERTIES: :CUSTOM_ID: examples-from-around-the-web :END:

• [[https://raw.githubusercontent.com/SciCrunch/sparc-curation/master/docs/apinatomy.org][apinatomy.org]] \ An executable Org file for running pipelines that build computational models of anatomy.
• [[https://raw.githubusercontent.com/SciCrunch/sparc-curation/master/docs/queries.org][queries.org]] \ An org file set up as an interface to query knowledge bases that is configured to provide a familiar (cua+) interface for users who may be unfamiliar with Emacs. Has examples for loading packages, and of one way to hide configuration to avoid information overload. A simpler version of the file (which leverages the same orgstrap block because the files are distributed together) is at [[https://raw.githubusercontent.com/SciCrunch/sparc-curation/master/docs/sckan/scratch.org][scratch.org]].
• [[https://raw.githubusercontent.com/SciCrunch/sparc-curation/master/docs/sckan/welcome.org][welcome.org]] \ An org file that acts as a static welcome page for a docker image, with links to other interactive org files. Has an example of how to use narrowing to hide the orgstrap machinery from the user.

TODO release.org

TODO git-share/README.org

** Tooling for orgstrap

• [[./reval.org][reval]]
• [[./orgware.org][orgware]]
• [[./shebang.org][shebang]] *** Services Configurations for services that can be reused across orgstrap files.
• [[./services/blazegraph.org][Blazegraph]] ** Useful orgstrap blocks Include these in part or whole to simplify common orgstrap workflows.
• [[file:reval.org::#minimal][reval-minimal]]

- [[file:orgware.org::#run-command][run-process-as-command]]

- [[file:orgware.org::#securl][securl]]

• Background, file local variables, and checksums :PROPERTIES: :CUSTOM_ID: background-file-local-variables-and-checksums :END: As mentioned above, the primary use case for =orgstrap= was that I was sick of having to work around the limitation that I had to do one of four things. I either one, had to remember to eval the source block containing defuns used later before I could eval other source blocks that used those functions in headers, or two, had to put those functions in =init.el=, destroying the ability to use org files as standalone self describing portable and reusable computational artifacts, three, had to copy and paste verbose elisp bits around to achieve what I wanted, or four, had to double tangle a file so that the results of the first tangle could be loaded before calling the second tangle so that the functionality would be available (this also produces the situation described in three). Furthermore, it is hard for humans to follow all the steps needed to get everything working -- even when 'everything' is just invoking =C-c C-c= on a single source block I still forget. This can lead to bad things if some of those source blocks were interdependent, or proceeded with a nil, etc.

File local variables to the rescue! I'm slightly embarrassed to say how long it took me to arrive at the current solution. I had known for quite a while that file local variables are a pathway to +abilities that+ the evils of arbitrary code execution, but it didn't click that all I was looking for was the ability to just run some arbitrary elisp code every time a particular file was loaded, which of course is exactly what file local variables are for.

The only question then was how to avoid the very real dangers of enabling arbitrary code execution of plain text. Actually it was more along the lines of "How can I keep org-babel happy without also pwning myself?" Fortunately =org-confirm-babel-evaluate= can be customized to be a function that accepts the body of the code to be evaluated. Therefore we can do the following.

When creating a file.

1. Hash the block to be run before distributing the file. Make sure to test if there are any changes to the header. For example I have a bad habit of accidentally setting =:noweb no-export= incorrectly without the dash and that will prevent the checksum from updating if a nowebbed block changes.
2. Embed the checksum in the file local variable property line. The property line is highly visible as the first line of the file. This makes it easy for users to verify that the embedded checksum matches a known independent checksum (running step 2). Thus if the embedded checksum does not match a known checksum the user will notice, and if the code to be executed does not match the embedded checksum then the user will at least be prompted by org-mode to run the block even in the case where they accepted the file local variables. Emacs also prompts for verification of the property line value which is another opportunity for the user to check.
3. Publish the checksum independent of the file itself. It is trivial for someone to change the contents of the orgstrap block and rerun =M-x= =orgstrap-add-block-checksum=. Therefore known checksums need to be published independent of the files themselves.

When running a file.

1. Audit, accept, and store permanently the eval file local variables. Storing audited variables permanently is critical for improving signal to noise so that unexpected mismatches retain their salience and can elicit the correct response (i.e., suspicion).

   XXX there may be an issue here if the property line tags along with the rest

   because we want to be able to mark the exact variables used in this file

   as safe and if they are couple to a random hash that is bad

2. Audit the orgstrap block I assume most people are not going to do this. However, one of the advantages of the current approach is that the same orgstrap blocks can be reused across multiple files which reduces the audit load such that one only needs to review unique orgstrap blocks, not all files. [fn::NOTE there are certain patterns inside blocks that are NOT safe to accept because they introduce a level of indirection that orgstrap cannot verify. Examples of these kinds of dangerous blocks are ones that make any reference to other blocks in the file via some means other than noweb. This isn't really surprising, and for use cases where =org-babel-execute-src-block= is called multiple times on different blocks, the default execution protection will work. In addition, any blocks which want to run automatically without prompting should use the =orgstrap--confirm-eval= function (see [[file:::#future-work][Future work]]).]
3. Verify that the embedded checksum matches the independent checksum. A known embedded checksum matching the content checksum only means that the content matches the content observed by the provider of the independent checksum (assuming no hash collisions).
4. Observe whether org-mode complains that the orgstrap block has changed.

• Experience reports :PROPERTIES: :CUSTOM_ID: experience-reports :END: ** First trial The first use case for orgstrap beyond personal use was to create a stand alone application that would allow a user to run, modify, and create SPARQL queries running from a local server[fn::The file itself is in a private git repo at the moment, but the functionality will be extracted and made public, and the repo itself will be as well.]

The primary users had no prior Emacs experience. Under supervision via a video call they were able to follow the instructions to download Emacs, download a zip of the file plus data and additional software, unzip, and click on the file (which opened in Emacs by default), and accept the local variables.

There were a couple of hiccups. In one case Java for macos was missing. In the other case the built-in version of Org mode was not correctly replaced so Emacs had to be restarted. In the second case there was also an issue with multiple windows being opened and the confirmation window for the local variables thus being hard to find.

Even with the slowdown they were able to get up and running within about 20 minutes. This is an enormous improvement over previous attempts which involved many hard to follow instructions that could take nearly 3 hours to complete and debug.

• Future work :PROPERTIES: :CUSTOM_ID: future-work :visibility: children :END: ** TODO separate user-emacs-directory :PROPERTIES: :CREATED: [2023-02-01 Wed 21:22] :END: Emacs 29 introduces the long desired command line option --init-directory=DIR. This solves a number of issues related to sandboxing the impact on existing Emacs configurations of using orgstrap for various things. That is somewhat secondary however.

More importantly, after a significant amount of experience working with this setup, it seems fairly clear to me that the default behavior when using orgstrap shebang blocks is that orgstrap configuration files and more importantly packages should be kept separate from the default user-emacs-directory to avoid a wide variety of issues.

Probably add this to ow-enable-use-packages with options to use something like ~~/.config/orgstrap/~ or ~~/.orgstrap.d/~ and additionally to sandbox packages for the whole file. I'm thinking that probably won't be necessary for most use cases. ** TODO detect whether dprp-1.0 is needed, otherwise use prp-1.1 :PROPERTIES: :CREATED: [2021-09-26 Sun 14:26] :END: The hashes that these generate are the same so long as there aren't any docstrings. For orgstrap blocks that don't use comments, we can save quite a bit of space in this way. ** DONE bug secondary runs of the orgstrap block klobber modified obce :PROPERTIES: :CREATED: [2021-08-19 Thu 21:47] :END: [[file:~/git/sparc-curation/docs/queries.org::orgstrap][orgstrap]] the quick fix is just to manually remove the step where we restore ocbe ** DONE somehow a stale orgstrap norm func managed to sneak in I have no idea how :PROPERTIES: :CREATED: [2021-08-11 Wed 12:32] :END: I'm 99% sure that this was coming from release.org and orgstrap-norm-func was not being reset and sticking around and messing stuff up.

This was part of the issue, though it wasn't that it was not being reset, it was that orgstrap-init did not source the default value because orgstrap-norm-func was incorrectly marked as a global dynamic variable instead of as defvar-local.

The other part of the issue was the we were not using the current orgstrap-norm-func-name for the buffer during orgstrap-init.

Even all that wasn't quite right. orgstrap-norm-func has to be overwritten before the checksum is added for existing files, otherwise the stale value will persist for files where local variables were actually accepted instead of ignored. ** TODO data section base64 (or whatever) encode a compressed data blob, stick it in an archived heading and then add =--pack= and =--unpack= and =--repack= or something equivalent. This is probably the most reasonable way to manage distributing org files that have dynamic data associated with them, such as an sqlite database or something.

TODO consider adding --check sha256 or one of the others

TODO -7 -8 and -9 when the input file is > 8 16 and 32 mb

#+begin_src bash :eval never xz -zk file base64 -w 127 file.xz > file.xz.base64 # 127 is a nicely sized prime xz -d file.xz xz -dc file.lz > /dev/null #+end_src

#+begin_src elisp (math-prime-test 109 9999) (math-prime-test 113 9999) (math-prime-test 127 9999) ; this one #+end_src

These are likely to be of interest since they can also be used to tar a whole directory, at which point it is possible to deal with the dissociation issue. =dired-compress-files-alist= =dired-do-compress= =dired-compress-file=

=base64-encode-region= =base64-decode-region=

Well, that was productive for figuring out what was going wrong. Turns out that =:ARCHIVE:= sections are still seen by flyspell and by auto-complete. =narrow-to-region= seems to have the behavior we want but it doesn't do multiple. For some reason archived text is still being searched by =auto-complete-mode= which causes massive slowdowns.

I have looked into modifying =org-hide-archived-subtrees= so that isearch does not open and search inside, however it does not seem to make any difference.

apparently flyspell doesn't honor read-only so we have to use something else and it also seems to ignore the first regexp I list here, so no good solutions thus far, I still think that narrowing to before and after are the best solution ... #+begin_src elisp (add-to-list 'ispell-skip-region-alist '("^\* data :ARCHIVE:$" . "=$")) (add-to-list 'ispell-skip-region-alist '("^#+begin_data" . "^#+end_data")) (auto-complete-mode 0) (rainbow-delimiters-org-mode 0) (flyspell-mode 0) #+end_src

#+begin_src elisp (use-package zones) ; not part of the core so hard to make use of #+end_src

#+begin_src org ,#+startup: showall ,* data :ARCHIVE: :PROPERTIES: :visibility: folded :END: put the big stuff here ,* Bootstrap :noexport:

,#+begin_src elisp (let ((inhibit-read-only t)) (add-text-properties 21 44543976 '(read-only t))) ,#+end_src #+end_src ** DONE orgstrap-inspect-elvs ** DONE symlinks and vc :PROPERTIES: :CREATED: [2021-08-02 Mon 01:23] :END: vc is called via find-file-hook which explicitly runs after hack-local-variables which explains how we are getting in so early with the orgstrap blocks, a solution has been found ** Safe local variable values need backlinks. orgstrap-block-checksum-sources alist as a custom variable so that it is easier for people to know what came from where in a summary even though we have the revoke functionality. ** DONE orgstrap-edit-mode fails to active when orgstrap-mode is enabled :PROPERTIES: :CREATED: [2020-11-29 Sun 00:10] :END: annoying this is because orgstrap-mode is idiotically broken and can't be activated globally despite being a global minor mode >_< ** Tutorial videos for various workflows Record a series of short screen casts to illustrate common orgstrap authoring and consumption workflows. ** TODO Display contents of orgstrap block in other window during confirm :PROPERTIES: :CREATED: [2020-10-08 Thu 23:27] :END: One major usability feature would be to figure out how to display the full body of the orgstrap block in the other window when the confirm local variables dialogue was presented. It seems easy enough when orgstrap.el is installed, however it seems like it might be hard to implement a minimal version, but maybe not, it is basically just save excursion and rearrange so that the local variables confirm buffer and the orgstrap block are the only two windows visible.

Another issue is whether it is possible to do this in Emacs < 27, since it is not possible to switch out of the confirm dialogue.

Probably use =org-babel-expand-src-block= via =call-interactively=. This will eat into the elvs budget. This also addresses some of the security concerns. ** EXPIRED Async blocks :PROPERTIES: :CREATED: [2020-10-02 Fri 17:03] :END: One issue that needs to be resolved is how to ensure that slow running blocks don't freeze Emacs. Ideally this could be done via ob-async, however using ob-async essentially means that we have to either tangle the block or we have to sneakily noweb it in if it is an elisp block, or something, to ensure that the inferior Emacs process has the definitions. Tangle and inject a call to load the file in the prologue or something?

The answer is that orgstrap isn't the place to handle these issues beyond providing documentation on best practices.

The best practice for this is to use the orgstrap block in a sane manner. For interactive use orgstrap blocks should include variable settings and defuns at most. Little to no actual computation should be done at that stage.

Longer running processes, such as tangling or building etc, should be masked in =(when noninteractive body ...)=. That allows orgstrap blocks to make the functionality defined in the file available via command line arguments (including editing via ./orgstrapped.org --edit).

In this context the evolution of orgstrap do will be to provide a library to make command line interaction with orgstrapped files discoverable. We are most of the way there because there is already an implementation of docopt for elisp. ** Spec extension to support arbitrary orgstrap block names. Since all the conventions for how this is done are defined locally by each file, you could in principle rename the special block as you see fit, perhaps from =orgstrap= to =main= if you need to pretend that the file is actually c source code with some special syntax. However, this is not advisable if you care about portability since it depends on an implementation detail of orgstrap.el which is not required by the specification. Namely that =orgstrap-orgstrap-block-name= is not required as one of the prop line local variables. Given the desire for the orgstrap machinery to be as unobtrusive as possible, it is unlikely that support for an arbitrary name for the block will be added to the spec.

That said, it is worth considering how and whether to update the spec so that a conforming implementation could do this if it wanted to. All the change does is move a convention to an optional variable. Maybe a compact variable such as orgstrap-bn could be specified as an optional prop line variable, and if absent the orgstrap block name defaults to orgstrap, otherwise the block name searched is the value of the variable.

Still not 100% sure about this. It would increase the complexity of the implementation for sure. It will require updating how and when we populate the link to orgstrap block, and makes auditing more difficult. It also opens up a way to trick the user, namely by having a link to an innocent looking orgstrap block, and no convention set in the prop line, or maybe even having it in the prop line (people are habitual and assume things are not present when they are), and then using setq-local, or some other means to change the block name to a malicious block elsewhere in the file. Implementations would have to know to check for this and fail if it was detected. Basically we would have to specify that if a block named orgstrap is present in a file when orgstrap-bn is present and not set to orgstrap, then it is a fatal error and orgstrap will not continue. Sticking this in the 900 or so chars we have left for further features seems like it would be a stretch, but might be possible. ** Security considerations =orgstrap= currently does not check all the headers or vars properties that materialized onto a source block we probably need to do this. For the time being users need to check for any hidden header properties that might be attached if the source block is buried within a tree somewhere. See org-babel-one-header-arg-safe-p for one way that this might be implemented in the elvs. ** DONE Batch mode This is more effectively implemented in [[./shebang.org]] because it bypasses the orgstrap checksum entirely. It is still secure because the user has to intentionally run the org file as a script. The overall complexity for the user is lower as well since they do not have to maintain or worry about the batch helper file, and the churn in the batch helper file is also eliminated. This section is retained for the record.

There are a number of use cases for being able to process orgstrapped files in batch mode. For example being able to load a file and have it automatically tangle itself vastly simplifies a number of different workflows. =emacs -q --batch -l orgstrap-known-safe.el my-file.org= seems like a reasonable approach. Essentially =orgstrap-known-safe.el= needs to contain the safe eval blocks and the audited hashes so that local variable prompts will not be triggered since they always return no when in batch mode. One additional feature is be to able to pass the checksum on the command line. The eval variables would still have to be loaded in some way, but avoiding the need to open and edit =orgstrap-known-safe.el= for each new file, and possibly edit it again to remove the approval in the future.

#+begin_src elisp :results none (let ((buffer (find-file-noselect "orgstrap-batch-helper.el.example"))) (with-current-buffer buffer (erase-buffer) (let (print-length print-level (print-escape-newlines t)) (insert ";;; -- mode: emacs-lisp; lexical-binding: t --\n\n") (insert ";;; add audited checksums here\n\n") (insert "(setq-local\n orgstrap-audited-checksums\n '(\n\n ))\n\n") ;; TODO insert test file block checksums (insert ";;; set audiated checksums as safe local variables\n\n") (insert (pp-to-string '(mapcar (lambda (checksum-value) (add-to-list 'safe-local-variable-values (cons 'orgstrap-block-checksum checksum-value))) orgstrap-audited-checksums)))) (insert "\n;;; helper local variables\n\n") (cl-loop for local-variable in '((orgstrap-cypher . sha256) (orgstrap-norm-func-name . orgstrap-norm-func--prp-1-1) (orgstrap-norm-func-name . orgstrap-norm-func--dprp-1-0)) do (let (print-length print-level (print-escape-newlines t)) (insert (prin1-to-string (add-to-list 'safe-local-variable-values ',local-variable))) (insert "\n"))) (insert "\n;;; known eval local variables\n\n") ) (cl-loop for (eval-local-variable elv-checksum) in (cl-remove-duplicates (cl-loop for block-name in '("example-noweb-no" "example-noweb-yes" "example-noweb-eval") append (cl-loop for minimal in '(nil t) append (cl-loop for norm-func-name in '(;; orgstrap-norm-func--prp-1-0 ; deprecated do not include orgstrap-norm-func--prp-1-1 orgstrap-norm-func--dprp-1-0) collect (let ((info (save-excursion (orgstrap--goto-named-src-block block-name) (org-babel-get-src-block-info)))) (setf (nth 6 info) "hrm") (list (orgstrap--lv-command info minimal norm-func-name) (secure-hash orgstrap-cypher (orgstrap-norm (let (print-quoted print-length print-level) (prin1-to-string (orgstrap--lv-command info minimal norm-func-name))))) ))))) :test #'equal) do (with-current-buffer buffer (let (print-length print-level (print-escape-newlines t)) (insert (prin1-to-string (add-to-list 'orgstrap-known-elvs ,elv-checksum))) (insert "\n") (insert (prin1-to-string `(add-to-list 'safe-local-eval-forms ',eval-local-variable))) (insert "\n")))) (with-current-buffer buffer (save-buffer))) #+end_src

#+name: example-noweb-no #+begin_src elisp :noweb no #+end_src

#+name: example-noweb-yes #+begin_src elisp :noweb yes #+end_src

#+name: example-noweb-eval #+begin_src elisp :noweb eval #+end_src

#+name: test-batch-helper #+begin_src bash emacs -q --batch
-l orgstrap-batch-helper.el
--eval "(message "\n%s"(emacs-version))"
--eval "(defun orgstrap-test () (error "Test failed."))"
-f toggle-debug-on-error
--visit test-lv-list-minimal
--eval "(message "done")" 2>&1

emacs -q --batch
-l orgstrap-batch-helper.el
--eval "(message "\n%s"(emacs-version))"
--eval "(defun orgstrap-test () (error "Test failed."))"
-f toggle-debug-on-error
--visit test-lv-list-portable
--eval "(message "done")" 2>&1 #+end_src

** Run once In principle the simplest way to do this is to use the =:cache yes= header on a block. However, unless the state is persisted into a users =init.el= file or equivalent, then the file would need a way to know that it had not been run when opened again in a new Emacs session. Similar issue with opening the same file in multiple Emacs sessions at the same time. The block simply will not run again if the cached result is present.

Therefore, since =:cache yes= by itself is a dead end for ensuring that functionality is always available any time a file is loaded there are a couple of options.

1. Persist to =init.el=. This is evil.
2. Request to tangle and install as package. A variant of this is simply to use package.el to install the desired functionality in a persistent way in combination with accept klobbering.
3. Figure out how to transparently wrap an elisp block in =unless=.
4. Advise =defun= (say what!?)? @@comment: TERROR@@
5. Figure out how to un-cache a block when Emacs exits. This will fail in nasty, unpredictable, and hard to debug ways.
6. Set =:cache (if (boundp 'orgstrap-already-run) "yes" "no")=. This ALMOST works. If =:cache no= embedded the sha1 sum then we would be golden. This seems like the best bet.
7. Accept klobbering.
8. Advise org-babel-eval to run with org-babel-sha1-sum even when cache is not set to yes

Another possibility would be

1. put checksums of orgstrap blocks that have been run in a list
2. use a special header arg :run-once yes to mark blocks that should not run if their checksum is already on the list.

If used with multiple blocks/multiple revals this would make it possible to run only a subset ** Tangle once When bootstrapping a new system there are many times when want to create a file only if it does not already exist. The =:tangle= header does not support this use case, but we can implement it anyway using the example below. #+name: tangle-once-example #+begin_src org ,#+name: orgstrap ,#+begin_src elisp (defun tangle-once (path) (if (file-exists-p path) "no" path)) ,#+end_src

,#+begin_src bash :tangle (tangle-once "./path-to-tangle") echo lol ,#+end_src

I think I've seen this before but you apparently can't have ,#+end_src on the line before #+end_src ... fun bug

#+end_src ** Multiple blocks There must be only a single one of those blocks so that the rest of the blocks can safely use the functions defined in the orgstrap block.

A single elisp block is sufficient to enable nearly all use cases involving tangling source blocks to file without having to fight the prompts. However, it is very much not sufficient for any use cases that involve other languages. This is particularly an issue for org files that want to bootstrap whole systems.

The simplest solution to me seems to be to add a second prompt variable which is an alist of source block checksums and names[fn::the names are not technically required but are for human readability]. As soon as the =orgstrap= block is run =orgstrap--confirm-eval= is no longer needed and can be replace with a function that validates the other blocks from the prompt variable.

This seems like a tractable approach, but also over complicated because it is surely easier in a case like this where blocks are very unlikely to be reused across org files to simply =(setq-local org-confirm-babel-evaluate nil)= and tell people to audit the whole file. The alternative in that case might be to hash all the source blocks and validate all of them at once at the start of the orgstrap block. This might need some additional machinery, not entirely sure, maybe just have =orgstrap-all-blocks-checksum= that can be used in cases like that. The advantage here is that the core of the process can be verified once and then the documentation around it can change and grow as needed. ** STARTED Support for orgstrap blocks beyond elisp See https://github.com/tgbugs/laundry for the start of work implementing org mode in Racket that would make it possible to have a practical discussion about how to approach Org babel beyond Emacs.

The spec may need to be extended to allow multiple orgstrap blocks with the same name. We might need to make a provision for the implementation specific language blocks to allow multiple names with the block for the main implementation language getting priority, however that may add significantly more complexity than e.g. just adding =orgstrap-elisp=, =orgstrap-racket=, =orgstrap-{lang}= as block names that are also searched.

This is unlikely to be useful any time soon given that there aren't full implementations of org mode outside of Emacs. If some setup is written in another language the best approach is to call the other block using the orgstrap block.

If at some point it becomes possible to use another language in the top level of org, then I imagine that such functionality will go in as a property or a file local variable or something like that. Early candidates for other languages that might be supportable are other lisp dialects. With such mechanisms in place, it would be relatively straight forward to lift the restriction on the language type, or rather, to include the normalized language name as part of the hash, or possibly to include it as a prop line local variable. More exploration would be required, but until there is some other implementation that has an extension language that is not elisp, this is a moot point. ** DONE Remove defun docstrings from hashing One additional source of noise in addition to comments are defun and defmacro docstrings. These should be dropped from the tree if they are present. This is now partially implemented via =orgstrap--dedoc=.

The issues in [[*1.2.4][1.2.4]] with inconsistent noweb block indentation. Is yet another reason for this. There is some lingering inconsistency in exactly how many leading spaces are included when nowebbing in elisp forms that include a multi-line string. This is one of those extremely annoying but hard to fix issues related to noweb and leading whitespace. ** Deterministic semantics preserving reordering Reorder the expressions used in the orgstrap block alphabetically (or something like that) according to a deterministic rule, but not in a way that changes program semantics. For example a function definition cannot be moved after a top level invocation of that function.

1. defuns with different names can be reordered
2. defuns with the same name can be reordered as a block but cannot internally be reordered because the order of shadowing matters
3. While it might be nice to completely erase the names of functions as well as internal variable names, this would make it trivial to shadow existing function names in ways that are malicious. The exact names matter, so we have to preserve them. Also the cost of not being able to tell that =(lambda (a) (+ a a))= and =(lambda (b) (+ b b))= are the same seems fairly small.
4. One potential approach is to lift all defuns to the top, and then function calls or whatever the more generic procedure invocation means. The simple local rule is that all definitions must occur before usage except in the case where there is a shadowing event that happens after a first invocation. This is annoying, but if a call to a function happens before that function is defined we have to assume that the call is calling some other function and those statements cannot be reordered. So the ordering is calls to functions with names matching any later defuns or any later assignment. Then defuns and assignments, finally procedure invocations which might also include assignments. I get the sense that this is covered under some part of compiler theory but can't quite put my finger on it. ** Figure out how to demo loading the packages used in this file #+begin_src elisp (use-package org-ref) ; for ref: (use-package ox-gfm) ; simplify export of changelog for release (use-package flycheck-package) ; linting for melpa #+end_src ** Orgware run One potential way to simplify command line execution of orgstrapped files is to write a wrapper that is aware of the internal variables to control various aspects of orgstrap behavior, such as config, testing, tangling, setup, build/make, dependencies, exporting, publishing, etc. Essentially we wind up with a set of functionalities that are commonly needed and set conventions for how to run only the desired subset.

Dealing with dependencies between functionalities is probably out of scope. The assumption is that Emacs and system packages are required for all further functionality, but beyond that it is not clear.

This is also related to the self describing nature of the orgstrap

approach which means that the diversity explosion steps, such as

dealing with the huge variety of package managers (discussed above),

will have to accumulate over time in the files themselves, or in a

portability layer that outside the files (e.g. via reval)

** Testing and continuous integration Ideally would like to move toward using ert for testing in this file. Generally it would be good to have an established workflow for testing orgstrap files beyond just nowebbing into when orgstrap-do-test.

As those workflows become established it would be nice to have a set of minimal wrappers for the various CI systems that will be sufficient to kick off the testing. See https://github.com/purcell/nix-emacs-ci for one way we might approach this. ** STARTED Lexical variant This saves about 300 chars over the current minimal version. #+begin_src elisp :results none (cl-defun orgstrap--length-of-sexp-at-point (&aux print-level print-length (print-escape-newlines t)) (interactive) (message "%s" (length (prin1-to-string (read ;; (org-babel-get-src-block-info) (thing-at-point 'sexp))))))

(defalias 'length-of-sexp-at-point #'orgstrap--length-of-sexp-at-point) #+end_src

#+begin_src elisp (let ((n "8.2.10") ; need (a (org-version)) ; actual (org-confirm-babel-evaluate #'orgstrap--confirm-eval) (obs (org-babel-find-named-block "orgstrap"))) (or (fboundp #'orgstrap--confirm-eval) (not n) (string< n a) (string= n a) (error "Org too old! %s < %s" a n)) (defun orgstrap-norm-func--prp-1-1 (body) (let (print-quoted print-length print-level) (prin1-to-string (read (concat "(progn\n" body "\n)"))))) (unless (fboundp #'orgstrap-norm) (defun orgstrap-norm (body) (funcall orgstrap-norm-func-name body))) (unless (fboundp 'orgstrap--confirm-eval) (defun orgstrap--confirm-eval (lang body) (not (and (member lang '("elisp" "emacs-lisp")) (eq orgstrap-block-checksum (intern (secure-hash orgstrap-cypher (orgstrap-norm body)))))))) (unwind-protect (save-excursion (goto-char obs) (org-babel-execute-src-block)) (org-set-startup-visibility))) #+end_src ** DONE Buffer local functions See [[file:./defl.org]] for an awful hack. ** DONE fix orgstrap--update-on-change hook so that it does not modify buffer fold state CLOSED: [2021-08-29 Sun 16:36] :PROPERTIES: :CREATED: [2020-10-21 Wed 01:28] :END: Not sure exactly which function is causing this but getting the checksum of the block seems like it might be the one, unfolding and not restoring the state.

The issue was in =orgstrap--with-block= because =goto-char= will cause outlines to open and we have to use =org-save-outline-visibility= with =use-markers= set to ensure that folding is restored.

=save-restriction= was not what we needed. https://stackoverflow.com/a/44158824 put me on the right track. ** DONE fix orgstrap-init formatting issue where comment goes in headline :PROPERTIES: :CREATED: [2020-10-19 Mon 18:01] :END: ** DONE blacklist and whitelist :PROPERTIES: :CREATED: [2020-10-15 Thu 22:32] :END: #+begin_example elisp (defcustom orgstrap-blacklist-files nil "List of files that should not be orgstrapped." ;; TODO but that should still have orgstrap-edit-mode enabled? :type 'list :group 'orgstrap) #+end_example ** TODO BUG :comments link breaks prop line :noexport: or rather not breaks, but tries to co-exist with it which duplicates the line which seems to break the ability to detangle among other things ** TODO behavior for activating orgstrap-mode in a buffer What to do if we are in a buffer that has an orgstrap block? I think the answer is to check for the local variables, and if they are present, do nothing, if they are absent run the block? ** TODO resolve the issue with tabs in < 26 :noexport: prin1-to-string has inconsistent behavior when the string in question contains an escaped tab character \t. This can make checksums inconsistent. ** DONE command to checksum the file local variables :noexport: This is now done as part of orgstrap-inspect-elvs. ** TODO ruby org so that github can render footnotes correctly :noexport: [[file:~/git/NOFORK/org-ruby]] ** DONE orgstrap-mode should not skip filter :PROPERTIES: :CREATED: [2020-10-01 Thu 21:04] :END: A potential issue in when running in =orgstrap-mode=. When the block checksum has not been confirmed =orgstrap--hack-lv-confirm= might filter/skip the checksum. I don't think this is actually what I observed (see below). *** I think this was never an issue :PROPERTIES: :CREATED: [2020-10-08 Thu 22:59] :END: I can't seem to reproduce this issue using the following. #+begin_src bash emacs -q -l orgstrap-autoloads.el
-f toggle-debug-on-error
--eval "(add-to-list 'load-path "$(pwd)/")"
--eval "(orgstrap-mode)" orgstrap-minimal.org #+end_src

I'm fairly certain that the issue that I was actually observing was the fact that when there is a checksum mismatch, then nothing is displayed and the block asks you to eval it -- you should decline and inspect the block.

The issue it seems is actually that the block that we are being asked to evaluate is not visible in the other window. So I have added a todo item for that. ** DONE Smart update/upgrade using the version specifier This is much easier now that I have put all the normalization and hashing machinery in a single elv, and it no longer needs to rely on the version of the block, just the version of the normalization function, which is currently embedded along with everything else. ** DONE orgstrap-mode If orgstrap is installed, then having a orgstrap-mode which could add itself to org-mode-hook when enabled would supersede the local variables list implementation, or possibly just do it when the local variables version was absent. Disabling the local variables when orgstrap-mode is detect to be on might be possible, but without it the user would have to explicitly decline the variables or risk having the orgstrap block run twice. Going to have to get this resolved before doing any announcements since it could leave users with quite a bit of pain if we don't get that check in before people start using it. ** DONE Auto update block checksum on save Before save hook and/or before commit hook to automatically update the block checksum. ** DONE use orgstrap to automatically keep example blocks in sync :noexport: ** DONE melpa :noexport: Woo! https://melpa.org/#/orgstrap gh:melpa/melpa/pull/7111 ** DONE Determine whether to use minimal or portable based on the :noweb header This is essentially what is implemented via =orgstrap--have-min-org-version=. The choice to prefer the minimal local variables is left up to the user and is controlled via the =orgstrap-use-minimal-local-variables= custom variable. The only time when minimal is not used is if the version of org that is drafting the file is too old (i.e. < =9.3.8=).

• Local Variables Footer :noexport: :PROPERTIES: :visibility: folded :END:

close powershell comment #>

Local Variables:

eval: (progn (setq-local orgstrap-min-org-version "8.2.10") (let ((a (org-version)) (n orgstrap-min-org-version)) (or (fboundp #'orgstrap--confirm-eval) (not n) (string< n a) (string= n a) (error "Your Org is too old! %s < %s" a n))) (defun orgstrap-norm-func--dprp-1-0 (body) (let ((p (read (concat "(progn\n" body "\n)"))) (m '(defun defun-local defmacro defvar defvar-local defconst defcustom)) print-quoted print-length print-level) (cl-labels ((f (b) (cl-loop for e in b when (listp e) do (or (and (memq (car e) m) (let ((n (nthcdr 4 e))) (and (stringp (nth 3 e)) (or (cl-subseq m 3) n) (f n) (or (setcdr (cddr e) n) t)))) (f e))) p)) (prin1-to-string (f p))))) (unless (boundp 'orgstrap-norm-func) (defvar-local orgstrap-norm-func orgstrap-norm-func-name)) (defun orgstrap-norm-embd (body) (funcall orgstrap-norm-func body)) (unless (fboundp #'orgstrap-norm) (defalias 'orgstrap-norm #'orgstrap-norm-embd)) (defun orgstrap-org-src-coderef-regexp (fmt &optional label) (let ((fmt org-coderef-label-format)) (format "\([:blank:]\(%s\)[:blank:]\)$" (replace-regexp-in-string "%s" (if label (regexp-quote label) "\([-a-zA-Z0-9][-a-zA-Z0-9_ ]\)") (regexp-quote fmt) nil t)))) (unless (fboundp #'org-src-coderef-regexp) (defalias 'org-src-coderef-regexp #'orgstrap-org-src-coderef-regexp)) (defun orgstrap--expand-body (info) (let ((coderef (nth 6 info)) (expand (if (org-babel-noweb-p (nth 2 info) :eval) (org-babel-expand-noweb-references info) (nth 1 info)))) (if (not coderef) expand (replace-regexp-in-string (org-src-coderef-regexp coderef) "" expand nil nil 1)))) (defun orgstrap--confirm-eval-portable (lang _body) (not (and (member lang '("elisp" "emacs-lisp")) (let ((body (orgstrap--expand-body (org-babel-get-src-block-info))) (body-normalized (orgstrap-norm body)) (content-checksum (intern (secure-hash orgstrap-cypher body-normalized)))) (eq orgstrap-block-checksum content-checksum))))) (unless (fboundp #'orgstrap--confirm-eval) (defalias 'orgstrap--confirm-eval #'orgstrap--confirm-eval-portable)) (let (enable-local-eval) (vc-find-file-hook)) (let ((ocbe org-confirm-babel-evaluate) (obs (org-babel-find-named-block "orgstrap"))) (if obs (unwind-protect (save-excursion (setq-local orgstrap-norm-func orgstrap-norm-func-name) (setq-local org-confirm-babel-evaluate #'orgstrap--confirm-eval) (goto-char obs) (org-babel-execute-src-block)) (when (eq org-confirm-babel-evaluate #'orgstrap--confirm-eval) (setq-local org-confirm-babel-evaluate ocbe)) (ignore-errors (org-set-visibility-according-to-property))) (warn "No orgstrap block."))))

End: