Skip to content

Enable Safari Remote Automation on SIP-enabled macOS 14/15 workers#1152

Open
rcurranmoz wants to merge 18 commits into
masterfrom
sip-compatible-safari-automation
Open

Enable Safari Remote Automation on SIP-enabled macOS 14/15 workers#1152
rcurranmoz wants to merge 18 commits into
masterfrom
sip-compatible-safari-automation

Conversation

@rcurranmoz
Copy link
Copy Markdown
Contributor

  • Replace bash-wrapped osascript with a direct LaunchAgent that runs osascript on an applescript file, avoiding TCC attribution to bash
  • Add safari-enable-remote-automation.applescript and com.mozilla.safari.enableautomation.plist as new deployment artifacts
  • Update macos_safaridriver::init to bootstrap the LaunchAgent into gui/ on Darwin 23/24 and poll for the semaphore file
  • Remove system TCC DB sqlite3 writes from tcc_perms.sh and add_tcc_perms.sh on macOS 14/15 (system DB is SIP-protected/read-only)
  • Add org.mozilla.ci-tcc-pppc.mobileconfig for upload to SimpleMDM as a Custom Configuration Profile to supply system-level TCC grants
aerickson
aerickson previously approved these changes Mar 26, 2026
Copy link
Copy Markdown
Member

@aerickson aerickson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

@rcurranmoz rcurranmoz force-pushed the sip-compatible-safari-automation branch 8 times, most recently from 8d3fdea to c009881 Compare April 3, 2026 16:56
@rcurranmoz rcurranmoz marked this pull request as ready for review April 3, 2026 17:19
@rcurranmoz rcurranmoz requested a review from a team April 3, 2026 17:20
aerickson
aerickson previously approved these changes Apr 7, 2026
Copy link
Copy Markdown
Member

@aerickson aerickson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one note about inventory.

Comment thread inventory.d/macmini-m4.yaml Outdated
@rcurranmoz
Copy link
Copy Markdown
Contributor Author

target host: macmini-m4-118.test.releng.mdc1.mozilla.com

rcurranmoz and others added 11 commits May 15, 2026 12:09
- Replace bash-wrapped osascript with a direct LaunchAgent that runs
  osascript on an applescript file, avoiding TCC attribution to bash
- Add safari-enable-remote-automation.applescript and
  com.mozilla.safari.enableautomation.plist as new deployment artifacts
- Update macos_safaridriver::init to bootstrap the LaunchAgent into
  gui/<uid> on Darwin 23/24 and poll for the semaphore file
- Remove system TCC DB sqlite3 writes from tcc_perms.sh and
  add_tcc_perms.sh on macOS 14/15 (system DB is SIP-protected/read-only)
- Add org.mozilla.ci-tcc-pppc.mobileconfig for upload to SimpleMDM as
  a Custom Configuration Profile to supply system-level TCC grants

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
macOS 12+ requires the string key Authorization: Allow rather than
the boolean Allowed: true in PPPC payload entries. The old format
caused ErrorCode 22 (invalid value) when pushed via SimpleMDM.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
StaticCode is not a valid PPPC payload key and was causing ErrorCode 22
on install. Also removed SystemPolicyDesktopFolder to reduce surface area
while iterating on getting the core services to install cleanly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ScreenCapture is not a valid PPPC service key on macOS 15 — every
entry format tried (Authorization:Allow, Allowed:true) results in
ErrorCode 22. Confirmed via binary search on macmini-m4-184 (15.3).

Working services: SystemPolicyAllFiles, Accessibility, AppleEvents.
ScreenCapture grants for bash/Terminal must be handled separately
(user DB fallback for start-worker already in tcc_perms.sh).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously the plist was deployed to ~/Library/LaunchAgents/ which caused
launchd to auto-load the LaunchAgent when cltbld logged in after the first
reboot — before puppet had written the user TCC DB entries. This resulted
in osascript hitting a TCC permission prompt and blocking indefinitely.

Fix: store the plist at /usr/local/lib/ so launchd never auto-loads it.
Only puppet bootstraps it, and only after the perms script succeeds.
Also remove any previously deployed plist from ~/Library/LaunchAgents/.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The applescript creates an empty semaphore file at startup as a lock,
then writes '1' on success. The puppet unless condition only checked
file existence, so a failed run left an empty semaphore that prevented
puppet from ever retrying.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The perms script (add_tcc_perms.sh) was refreshonly, meaning it only ran
when the file changed. On first run the TCC DB didn't exist so it failed;
on subsequent runs the file was unchanged so it never re-ran, leaving
the osascript TCC entries unwritten.

Replace with an unless condition that checks if the osascript AppleEvents
entry is present in cltbld's TCC DB, so it re-runs whenever entries are
missing regardless of whether the file changed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
BlackHole pkg install fails on fresh macOS 15 machines; comment out
from both m4 and m4-staging roles until the install issue is fixed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
BlackHole pkg install fails on fresh macOS 15 machines; comment out
from both m4 and m4-staging roles until the install issue is fixed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The exec command polled with `test -f semaphore` which would return
success immediately if the semaphore file existed but was empty (written
by the applescript at startup before GUI interaction completes). This
caused false success when a prior failed run left an empty semaphore.

Also increase the poll timeout from 60s to 120s to give the applescript
sufficient margin — it has ~50s of programmed delays alone.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LaunchDaemon-spawned puppet hits "authorization denied" writing
/Users/cltbld/Library/Application Support/com.apple.TCC/TCC.db.
Existing PPPC FDA grants cover sshd/sshd-keygen-wrapper only, so the
chain works for SSH-driven puppet (inherits sshd's TCC context) but
not for launchd-driven puppet (which is what a SimpleMDM script-job
bootstrap needs). Adding bash and sqlite3 lets the user-DB sqlite3
writes in add_tcc_perms.sh and tcc_perms.sh succeed regardless of
the invoking process tree.

Validated on macmini-m4-118 2026-05-15: SSH-driven puppet succeeded
end-to-end, but LaunchDaemon-driven puppet from the SimpleMDM
bootstrap script failed at the user-DB write step before this change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rcurranmoz rcurranmoz force-pushed the sip-compatible-safari-automation branch from 64d8ba5 to 4c41b33 Compare May 15, 2026 16:10
rcurranmoz added a commit that referenced this pull request May 15, 2026
…emaphore write

The old applescript would write the success semaphore even if the GUI
checkbox toggle silently failed (osascript-can't-read-value-back, or
some race in Safari's Settings tab population). A bootstrapped host
could land with safaridriver effectively disabled while looking "green"
to every automated check — and we only catch it when somebody opens
Safari in VNC and eyeballs the menu. Which is exactly how we lost a
chunk of today.

After clicking each of the two toggles ("Show features for web developers"
and "Allow remote automation"), the script now reads the checkbox value
back and `error`s out if it isn't 1. The script's only path to writing
the success semaphore is a clean run through both clicks, so a silent
skip now surfaces as a puppet exec failure and the SimpleMDM driver's
outer retry loop keeps going instead of declaring victory.

---

CLAUDE HANDOFF — PR #1152 + SimpleMDM bootstrap, pick up next week

Hello! If you're reading this, you have inherited the SIP-compatible
Safari Remote Automation saga. Here's where today (2026-05-15) left
things and what to chase next. Buckle up.

THE BIG PICTURE
PR #1152 ("sip-compatible-safari-automation") replaces the bash-wrapped
osascript Safari-enable dance with a LaunchAgent that runs osascript
directly, plus a SimpleMDM-delivered PPPC mobileconfig for system-level
TCC grants. End goal: zero-touch DEP enrollment → puppet → worker live
in TC, on SIP-enabled macOS 15.3 M4 Mac minis, without the human ever
SSHing in.

WHAT GOT VALIDATED TODAY ON macmini-m4-118 (gecko_t_osx_1500_m4_staging)
- In-place pivot from existing puppet state to PR branch: green
- Fresh post-EACS + admin VNC + SSH-driven bootstrap: green
- Fresh post-EACS + admin VNC + SimpleMDM-driven bootstrap (via the
  ~/Desktop/m4-simplemdm-bootstrap-sip-safari.sh one-shot script):
  bootstrap completed, BUT the applescript silently skipped the
  "Allow remote automation" toggle on the first MDM-driven run.
  Hence this commit. We then manually re-ran the applescript and it
  landed visually correctly. The bug that hid this from us all day
  was the semaphore-on-success-without-actual-success behavior.
- Bootstrap Token custody stays on admin (cltbld DISABLED): confirmed
  via sysadminctl -secureTokenStatus + APFS crypto users listing.
  The bootstrap script's admin-BST gate (sysadminctl poll, 30 min cap)
  works as intended.
- PPPC FDA grants for /bin/bash + /usr/bin/sqlite3 land on the host
  (verify via /Library/Managed Preferences/com.apple.TCC.configuration-
  profile-policy.plist).

THE FOOTGUNS I LEFT IN THE SAND FOR YOU
1. PPPC grants for sshd/sshd-keygen-wrapper alone are NOT enough for a
   LaunchDaemon-spawned puppet to write cltbld's user TCC DB. We hit
   "authorization denied" on user-DB sqlite3 writes when the driver
   ran puppet directly. Workaround: the bootstrap driver invokes
   run-puppet.sh via `ssh root@localhost` so puppet inherits sshd's
   user-session TCC domain. One-shot ed25519 key at
   /var/root/.ssh/bootstrap_id_ed25519 during bootstrap, removed on
   sentinel write. This lives in the Desktop bootstrap script — when
   we evolve this, that workaround should probably move into a real
   puppet/PR change.
2. `launchctl asuser 0 <cmd>` does NOT actually escape the system
   domain when invoked from a LaunchDaemon. It only transitions when
   the caller is already in a user-session domain (sshd-spawned, GUI
   login). I went down this path for an embarrassing amount of time
   today before pivoting to ssh-to-localhost. Save yourself.
3. LaunchDaemon RunAtLoad fires before DNS is up at boot. First puppet
   tick on a freshly-rebooted host died on `git fetch`'s DNS lookup and
   our one-shot LaunchDaemon never retried. Driver now waits on
   dscacheutil resolving github.com before doing anything, with an
   outer while-loop retrying run-puppet.sh until both semaphores land.

ARTIFACTS WORTH BOOKMARKING
- ~/Desktop/m4-simplemdm-bootstrap-sip-safari.sh — the SimpleMDM script.
  Upload to SimpleMDM as a script-job, run-on-enrollment, scoped to the
  M4 staging group. Idempotent, self-cleans on success.
- ~/Desktop/org.mozilla.ci-tcc-pppc.mobileconfig — extracted from PR
  branch; upload to SimpleMDM as a Custom Configuration Profile,
  replacing the previous "Mozilla CI TCC Permissions". Has the new
  bash + sqlite3 FDA grants this branch added.
- modules/macos_mobileconfig_profiles/files/org.mozilla.ci-tcc-pppc.mobileconfig
  — source of truth in-tree.
- modules/macos_safaridriver/files/safari-enable-remote-automation.applescript
  — this commit's change lives here.
- ~/Desktop/m4-provisioning-handoff.md — earlier handoff from the M4
  rollout, the procedural shape still applies.

REMAINING WORK FOR ACTUAL ZERO-TOUCH
- vault.yaml delivery. Still a manual scp/drop step. Bootstrap script
  polls /var/root/vault.yaml up to 10 min. Open question: secure-fetch
  via an internal endpoint? SimpleMDM Custom Attribute with the file
  body? Conversation worth having with whoever runs SimpleMDM admin.
- admin VNC login is still required to plant the Bootstrap Token before
  puppet flips autologin to cltbld. No known mechanism to script a
  GUI/console login during DEP / Setup Assistant. If you find one,
  you become a legend.
- start-worker / generic-worker-multiuser need Developer ID signing
  to be coverable by the PPPC profile (ScreenCapture specifically).
  start-worker is currently a user-DB fallback entry in tcc_perms.sh.

VERIFICATION METHOD (per Ryan, after a long debug detour today)
Forget `defaults read com.apple.Safari AllowRemoteAutomation`. The pref
doesn't live there in any way our tooling can read. The gold standard
is opening Safari in cltbld's VNC session and visually confirming:
  - Develop menu is in the menu bar
  - Safari > Settings > Developer > "Allow remote automation" is checked
This is the dance the RelOps team has been doing since macOS 10.15.

RECOMMENDED REORIENTATION PLAYBOOK
  1. Read: this commit + the mobileconfig + the Desktop bootstrap script
  2. EACS macmini-m4-118 (or any spare staging M4)
  3. Admin VNC login → drop vault.yaml → trigger the SimpleMDM script
  4. Once bootstrap completes (sentinel /var/log/m4-bootstrap-complete),
     eyeball Safari > Settings > Developer in cltbld's VNC session
  5. If anything is weird: /var/log/m4-bootstrap.log (one-shot setup)
     and /var/log/m4-bootstrap-driver.log (boot-by-boot puppet) are
     your friends. ssh keys and LaunchDaemon plists self-clean on
     success; if they're still on disk, the driver hasn't declared
     victory yet.

Memory notes for ~/.claude/projects/-Users-rcurran/memory/ are already
updated for: [[apple-silicon-bootstrap-token]] (the BST custody fix),
[[sip-safari-automation]] (the PR's overall shape and validated state).

Good luck. May your AppleScript clicks actually persist.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rcurranmoz added a commit that referenced this pull request May 15, 2026
…emaphore write

The old applescript would write the success semaphore even if the GUI
checkbox toggle silently failed (osascript-can't-read-value-back, or
some race in Safari's Settings tab population). A bootstrapped host
could land with safaridriver effectively disabled while looking "green"
to every automated check — and we only catch it when somebody opens
Safari in VNC and eyeballs the menu. Which is exactly how we lost a
chunk of today.

After clicking each of the two toggles ("Show features for web developers"
and "Allow remote automation"), the script now reads the checkbox value
back and `error`s out if it isn't 1. The script's only path to writing
the success semaphore is a clean run through both clicks, so a silent
skip now surfaces as a puppet exec failure and the SimpleMDM driver's
outer retry loop keeps going instead of declaring victory.

---

🎁 CLAUDE HANDOFF — PR #1152 + SimpleMDM bootstrap, pick up next week 🎁

Hello! 👋 If you're reading this, you have inherited the SIP-compatible
Safari Remote Automation saga. Here's where today (2026-05-15) left
things and what to chase next. Buckle up. 🎢

🗺️ THE BIG PICTURE
PR #1152 ("sip-compatible-safari-automation") replaces the bash-wrapped
osascript Safari-enable dance with a LaunchAgent that runs osascript
directly, plus a SimpleMDM-delivered PPPC mobileconfig for system-level
TCC grants. End goal: zero-touch DEP enrollment → puppet → worker live
in TC, on SIP-enabled macOS 15.3 M4 Mac minis, without the human ever
SSHing in. ✨

✅ WHAT GOT VALIDATED ON macmini-m4-118 (gecko_t_osx_1500_m4_staging)
- 🟢 In-place pivot from existing puppet state to PR branch
- 🟢 Fresh post-EACS + admin VNC + SSH-driven bootstrap
- 🟡 Fresh post-EACS + admin VNC + SimpleMDM-driven bootstrap (via the
  ~/Desktop/m4-simplemdm-bootstrap-sip-safari.sh one-shot script):
  bootstrap completed, BUT the applescript silently skipped the
  "Allow remote automation" toggle on the first MDM-driven run.
  Hence this commit. We then manually re-ran the applescript and it
  landed visually correctly. The bug that hid this from us all day
  was the semaphore-on-success-without-actual-success behavior. 🫠
- 🔐 Bootstrap Token custody stays on admin (cltbld DISABLED):
  confirmed via sysadminctl -secureTokenStatus + APFS crypto users.
  The bootstrap script's admin-BST gate (sysadminctl poll, 30 min cap)
  works as intended.
- 🪪 PPPC FDA grants for /bin/bash + /usr/bin/sqlite3 land on the host
  (verify via /Library/Managed Preferences/com.apple.TCC.configuration-
  profile-policy.plist).

🪤 THE FOOTGUNS I LEFT IN THE SAND FOR YOU
1. 🔒 PPPC grants for sshd/sshd-keygen-wrapper alone are NOT enough for a
   LaunchDaemon-spawned puppet to write cltbld's user TCC DB. We hit
   "authorization denied" on user-DB sqlite3 writes when the driver
   ran puppet directly. Workaround: the bootstrap driver invokes
   run-puppet.sh via \`ssh root@localhost\` so puppet inherits sshd's
   user-session TCC domain. One-shot ed25519 key at
   /var/root/.ssh/bootstrap_id_ed25519 during bootstrap, removed on
   sentinel write. This lives in the Desktop bootstrap script — when
   we evolve this, that workaround should probably move into a real
   puppet/PR change.
2. 🐚 \`launchctl asuser 0 <cmd>\` does NOT actually escape the system
   domain when invoked from a LaunchDaemon. It only transitions when
   the caller is already in a user-session domain (sshd-spawned, GUI
   login). I went down this path for an embarrassing amount of time
   today before pivoting to ssh-to-localhost. Save yourself. 😅
3. 🌐 LaunchDaemon RunAtLoad fires before DNS is up at boot. First puppet
   tick on a freshly-rebooted host died on \`git fetch\`'s DNS lookup and
   our one-shot LaunchDaemon never retried. Driver now waits on
   dscacheutil resolving github.com before doing anything, with an
   outer while-loop retrying run-puppet.sh until both semaphores land.

📦 ARTIFACTS WORTH BOOKMARKING
- ~/Desktop/m4-simplemdm-bootstrap-sip-safari.sh — the SimpleMDM script.
  Upload to SimpleMDM as a script-job, run-on-enrollment, scoped to the
  M4 staging group. Idempotent, self-cleans on success.
- ~/Desktop/org.mozilla.ci-tcc-pppc.mobileconfig — extracted from PR
  branch; upload to SimpleMDM as a Custom Configuration Profile,
  replacing the previous "Mozilla CI TCC Permissions". Has the new
  bash + sqlite3 FDA grants this branch added.
- modules/macos_mobileconfig_profiles/files/org.mozilla.ci-tcc-pppc.mobileconfig
  — source of truth in-tree.
- modules/macos_safaridriver/files/safari-enable-remote-automation.applescript
  — this commit's change lives here.
- ~/Desktop/m4-provisioning-handoff.md — earlier handoff from the M4
  rollout, the procedural shape still applies.

🚧 REMAINING WORK FOR ACTUAL ZERO-TOUCH
- 🔑 vault.yaml delivery. Still a manual scp/drop step. Bootstrap script
  polls /var/root/vault.yaml up to 10 min. Open question: secure-fetch
  via an internal endpoint? SimpleMDM Custom Attribute with the file
  body? Conversation worth having with whoever runs SimpleMDM admin.
- 👤 admin VNC login is still required to plant the Bootstrap Token
  before puppet flips autologin to cltbld. No known mechanism to script
  a GUI/console login during DEP / Setup Assistant. If you find one,
  you become a legend. 🏆
- 🖋️ start-worker / generic-worker-multiuser need Developer ID signing
  to be coverable by the PPPC profile (ScreenCapture specifically).
  start-worker is currently a user-DB fallback entry in tcc_perms.sh.

🔬 VERIFICATION METHOD (per Ryan, after a long debug detour today)
Forget \`defaults read com.apple.Safari AllowRemoteAutomation\`. The pref
doesn't live there in any way our tooling can read. The gold standard
is opening Safari in cltbld's VNC session and visually confirming:
  - Develop menu is in the menu bar
  - Safari > Settings > Developer > "Allow remote automation" is checked
This is the dance the RelOps team has been doing since macOS 10.15. 🕺

🚀 RECOMMENDED REORIENTATION PLAYBOOK
  1. Read: this commit + the mobileconfig + the Desktop bootstrap script
  2. EACS macmini-m4-118 (or any spare staging M4)
  3. Admin VNC login → drop vault.yaml → trigger the SimpleMDM script
  4. Once bootstrap completes (sentinel /var/log/m4-bootstrap-complete),
     eyeball Safari > Settings > Developer in cltbld's VNC session
  5. If anything is weird: /var/log/m4-bootstrap.log (one-shot setup)
     and /var/log/m4-bootstrap-driver.log (boot-by-boot puppet) are
     your friends 🪵. ssh keys and LaunchDaemon plists self-clean on
     success; if they're still on disk, the driver hasn't declared
     victory yet.

🧠 Memory notes for ~/.claude/projects/-Users-rcurran/memory/ are already
updated for: [[apple-silicon-bootstrap-token]] (the BST custody fix),
[[sip-safari-automation]] (the PR's overall shape and validated state).

Good luck. May your AppleScript clicks actually persist. 🍀

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rcurranmoz rcurranmoz force-pushed the sip-compatible-safari-automation branch from 4bf1b8b to 3678adb Compare May 15, 2026 18:50
…emaphore write

The old applescript would write the success semaphore even if the GUI
checkbox toggle silently failed (osascript-can't-read-value-back, or
some race in Safari's Settings tab population). A bootstrapped host
could land with safaridriver effectively disabled while looking "green"
to every automated check — and we only catch it when somebody opens
Safari in VNC and eyeballs the menu. Which is exactly how we lost a
chunk of today.

After clicking each of the two toggles ("Show features for web developers"
and "Allow remote automation"), the script now reads the checkbox value
back and `error`s out if it isn't 1. The script's only path to writing
the success semaphore is a clean run through both clicks, so a silent
skip now surfaces as a puppet exec failure and the SimpleMDM driver's
outer retry loop keeps going instead of declaring victory.

---

🎁 CLAUDE HANDOFF — PR #1152 + SimpleMDM bootstrap, pick up next week 🎁

Hello! 👋 If you're reading this, you have inherited the SIP-compatible
Safari Remote Automation saga. Here's where today (2026-05-15) left
things and what to chase next. Buckle up. 🎢

🗺️ THE BIG PICTURE
PR #1152 ("sip-compatible-safari-automation") replaces the bash-wrapped
osascript Safari-enable dance with a LaunchAgent that runs osascript
directly, plus a SimpleMDM-delivered PPPC mobileconfig for system-level
TCC grants. End goal: zero-touch DEP enrollment → puppet → worker live
in TC, on SIP-enabled macOS 15.3 M4 Mac minis, without the human ever
SSHing in. ✨

✅ WHAT GOT VALIDATED ON macmini-m4-118 (gecko_t_osx_1500_m4_staging)
- 🟢 In-place pivot from existing puppet state to PR branch
- 🟢 Fresh post-EACS + admin VNC + SSH-driven bootstrap
- 🟡 Fresh post-EACS + admin VNC + SimpleMDM-driven bootstrap (via the
  ~/Desktop/m4-simplemdm-bootstrap-sip-safari.sh one-shot script):
  bootstrap completed, BUT the applescript silently skipped the
  "Allow remote automation" toggle on the first MDM-driven run.
  Hence this commit. We then manually re-ran the applescript and it
  landed visually correctly. The bug that hid this from us all day
  was the semaphore-on-success-without-actual-success behavior. 🫠
- 🔐 Bootstrap Token custody stays on admin (cltbld DISABLED):
  confirmed via sysadminctl -secureTokenStatus + APFS crypto users.
  The bootstrap script's admin-BST gate (sysadminctl poll, 30 min cap)
  works as intended.
- 🪪 PPPC FDA grants for /bin/bash + /usr/bin/sqlite3 land on the host
  (verify via /Library/Managed Preferences/com.apple.TCC.configuration-
  profile-policy.plist).

🪤 THE FOOTGUNS I LEFT IN THE SAND FOR YOU
1. 🔒 PPPC grants for sshd/sshd-keygen-wrapper alone are NOT enough for a
   LaunchDaemon-spawned puppet to write cltbld's user TCC DB. We hit
   "authorization denied" on user-DB sqlite3 writes when the driver
   ran puppet directly. Workaround: the bootstrap driver invokes
   run-puppet.sh via \`ssh root@localhost\` so puppet inherits sshd's
   user-session TCC domain. One-shot ed25519 key at
   /var/root/.ssh/bootstrap_id_ed25519 during bootstrap, removed on
   sentinel write. This lives in the Desktop bootstrap script — when
   we evolve this, that workaround should probably move into a real
   puppet/PR change.
2. 🐚 \`launchctl asuser 0 <cmd>\` does NOT actually escape the system
   domain when invoked from a LaunchDaemon. It only transitions when
   the caller is already in a user-session domain (sshd-spawned, GUI
   login). I went down this path for an embarrassing amount of time
   today before pivoting to ssh-to-localhost. Save yourself. 😅
3. 🌐 LaunchDaemon RunAtLoad fires before DNS is up at boot. First puppet
   tick on a freshly-rebooted host died on \`git fetch\`'s DNS lookup and
   our one-shot LaunchDaemon never retried. Driver now waits on
   dscacheutil resolving github.com before doing anything, with an
   outer while-loop retrying run-puppet.sh until both semaphores land.

📦 ARTIFACTS WORTH BOOKMARKING
- ~/Desktop/m4-simplemdm-bootstrap-sip-safari.sh — the SimpleMDM script.
  Upload to SimpleMDM as a script-job, run-on-enrollment, scoped to the
  M4 staging group. Idempotent, self-cleans on success.
- ~/Desktop/org.mozilla.ci-tcc-pppc.mobileconfig — extracted from PR
  branch; upload to SimpleMDM as a Custom Configuration Profile,
  replacing the previous "Mozilla CI TCC Permissions". Has the new
  bash + sqlite3 FDA grants this branch added.
- modules/macos_mobileconfig_profiles/files/org.mozilla.ci-tcc-pppc.mobileconfig
  — source of truth in-tree.
- modules/macos_safaridriver/files/safari-enable-remote-automation.applescript
  — this commit's change lives here.
- ~/Desktop/m4-provisioning-handoff.md — earlier handoff from the M4
  rollout, the procedural shape still applies.

🚧 REMAINING WORK FOR ACTUAL ZERO-TOUCH
- 🔑 vault.yaml delivery. Still a manual scp/drop step. Bootstrap script
  polls /var/root/vault.yaml up to 10 min. Open question: secure-fetch
  via an internal endpoint? SimpleMDM Custom Attribute with the file
  body? Conversation worth having with whoever runs SimpleMDM admin.
- 👤 admin VNC login is still required to plant the Bootstrap Token
  before puppet flips autologin to cltbld. No known mechanism to script
  a GUI/console login during DEP / Setup Assistant. If you find one,
  you become a legend. 🏆
- 🖋️ start-worker / generic-worker-multiuser need Developer ID signing
  to be coverable by the PPPC profile (ScreenCapture specifically).
  start-worker is currently a user-DB fallback entry in tcc_perms.sh.

🔬 VERIFICATION METHOD (per Ryan, after a long debug detour today)
Forget \`defaults read com.apple.Safari AllowRemoteAutomation\`. The pref
doesn't live there in any way our tooling can read. The gold standard
is opening Safari in cltbld's VNC session and visually confirming:
  - Develop menu is in the menu bar
  - Safari > Settings > Developer > "Allow remote automation" is checked
This is the dance the RelOps team has been doing since macOS 10.15. 🕺

🚀 RECOMMENDED REORIENTATION PLAYBOOK
  1. Read: this commit + the mobileconfig + the Desktop bootstrap script
  2. EACS macmini-m4-118 (or any spare staging M4)
  3. Admin VNC login → drop vault.yaml → trigger the SimpleMDM script
  4. Once bootstrap completes (sentinel /var/log/m4-bootstrap-complete),
     eyeball Safari > Settings > Developer in cltbld's VNC session
  5. If anything is weird: /var/log/m4-bootstrap.log (one-shot setup)
     and /var/log/m4-bootstrap-driver.log (boot-by-boot puppet) are
     your friends 🪵. ssh keys and LaunchDaemon plists self-clean on
     success; if they're still on disk, the driver hasn't declared
     victory yet.

🧠 Memory notes for ~/.claude/projects/-Users-rcurran/memory/ are already
updated for: [[apple-silicon-bootstrap-token]] (the BST custody fix),
[[sip-safari-automation]] (the PR's overall shape and validated state).

Good luck. May your AppleScript clicks actually persist. 🍀

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rcurranmoz rcurranmoz force-pushed the sip-compatible-safari-automation branch from 3678adb to 85876cc Compare May 15, 2026 18:54
rcurranmoz and others added 6 commits May 15, 2026 15:13
Post-EACS-#2 the SimpleMDM-driven bootstrap landed clean — Develop menu
visible and "Allow remote automation" checked, confirmed visually in
cltbld's VNC session. That validates the verify-after-click commit
(4bf1b8b/85876cc5) end-to-end through the LaunchDaemon → ssh-to-localhost
→ puppet → LaunchAgent → osascript chain.

Also flagging the one human-touch needed on this round (manual
/var/tmp/semaphore/run-buildbot), which is already fixed in the Desktop
bootstrap script — just hadn't been re-uploaded to SimpleMDM between
iterations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Link long-standing taskcluster/taskcluster#7413 for the start-worker /
  generic-worker-multiuser Developer ID signing gap.
- Trim the "Forget defaults read AllowRemoteAutomation" framing on the
  verification section — that was overstated; visual VNC check is still
  the gold standard but no need to bury the defaults command.
- Tighten the vault.yaml delivery note (Ryan is the SimpleMDM admin —
  the decision-maker is the same human who picks this up).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The handoff was referencing ~/Desktop/ paths on my laptop, which is
useless to anyone else picking this up. Move the SimpleMDM bootstrap
script into the repo at
modules/macos_safaridriver/simplemdm-bootstrap-sip-safari.sh
so it lives next to the puppet module it bootstraps onto, and rewrite
the HANDOFF artifact table to point at in-tree relative paths only.

Also drop the "Related memory notes" section — those pointers were to
my local Claude memory dir, equally useless to someone else.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The footnote claimed SimpleMDM had a pre-touch version of the bootstrap
script during EACS#2; per Ryan, the latest script was in fact uploaded.
Dropping the inaccurate claim. The in-tree script
(`modules/macos_safaridriver/simplemdm-bootstrap-sip-safari.sh`) has
the `touch /var/tmp/semaphore/run-buildbot` baked into the driver
cleanup block (line 206), so the next EACS should bring the worker
online on its own.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eview

Safari Technology Preview has its own separate "Allow Remote Automation"
toggle that the stable Safari enable flow does NOT flip. Without this,
test-macosx1500-aarch64-shippable/opt-browsertime-benchmark-safari-tp-*
tasks fail with "raptor-browsertime Critical: No data to collect" — the
browsertime webdriver session can't drive Safari TP.

Mirrors the stable Safari LaunchAgent + applescript pattern:

  - files/safari-tp-enable-remote-automation.applescript
      Drives "Safari Technology Preview" through the same Settings ->
      Advanced -> Show features for web developers -> Developer ->
      Allow remote automation dance, with the same verify-after-click
      defensive gate before writing its semaphore.
  - files/com.mozilla.safari-tp.enableautomation.plist
      LaunchAgent that runs the TP applescript via osascript in cltbld's
      gui/<uid> session.
  - manifests/init.pp
      New `execute enable remote automation script (Safari TP)` exec
      bootstrapped alongside the existing stable Safari exec on macOS
      14/15. Gated on `/Applications/Safari Technology Preview.app`
      existing via `onlyif` so a host without Safari TP installed yet
      (e.g. mid-EACS, before packages::safari_preview lands) becomes a
      no-op rather than a hard puppet failure — the next puppet apply
      picks it up once the .app dir is in place.

Semaphore: /Users/cltbld/Library/Preferences/semaphore/safari-tech-preview-enable-remote-automation-has-run
(name matches the legacy pre-PR convention so any older tooling that
looked for it keeps working.)

The system-wide `safaridriver --enable` already covers both Safari and
Safari TP — no additional exec needed for that side.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Safari TP applescript + LaunchAgent landed in b36069a, and the
follow-up try push (revision 0502d2099c8a) confirms 6 of 7 safari-*
benchmark tasks pass on m4-118 — including both safari-tp-speedometer3
and safari-tp-jetstream3, which were failing for "No data to collect"
on prior pushes when the TP applescript didn't exist yet. Update the
HANDOFF to:

- Move the SimpleMDM-driven bootstrap row to unambiguously green and
  drop the "(2nd EACS round)" qualifier — it's the steady state now.
- Add a try-push validation row + a dedicated Safari TP row pointing
  at the proof-of-life push.
- Add the two new safari-tp artifacts (applescript + plist) to the
  artifacts table, and call out the matching plist for stable Safari.
- Generalize the verification method to call out BOTH Safari and
  Safari Technology Preview, and link the verified try push as a
  backup signal that's faster than a VNC eyeball.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

2 participants