DEPLOYING

Deploying tmbo, and getting back out again.

This is a playbook, not a record of what happened.  If a past outage taught us
something, the lesson belongs in the relevant section below as an instruction; the
story does not belong here at all.

If you add to this file, prefer naming files and functions over line numbers.  Line
numbers are the first thing to rot, and a stale line number makes a true statement
look false.


###############################################################################
BEFORE YOU TOUCH ANYTHING
###############################################################################

Write down the commit you are on.  This is the single most useful thing you can do
and it takes two seconds:

    git -C ~/sites/tmbo rev-parse HEAD

Check whether the site is already in a non-default state.  The off switches below
are uncommitted edits to tracked files, so the working tree is the record:

    git -C ~/sites/tmbo status --short
    git -C ~/sites/tmbo diff

Back up any table you are about to change, and any table you are about to delete
rows from.  Restoring from a dump is easy; reconstructing destroyed rows from a
stack trace is not:

    mysqldump -u tmbo -p tmbo users  > ~/backup-users-$(date +%F-%H%M).sql
    mysqldump -u tmbo -p tmbo tokens > ~/backup-tokens-$(date +%F-%H%M).sql

No dump is taken on a schedule by anything in this repository, and there is no other
copy.  Confirm the server's own crontab before you rely on that -- but if you did not
just take a dump, assume one does not exist.

To put a dump back:

    mysql -u tmbo -p tmbo < ~/backup-tokens-<stamp>.sql

A mysqldump of a single table begins with DROP TABLE IF EXISTS, so restoring
replaces the table wholesale.  Anything the site wrote to it since the dump is
gone.  That matters most for tokens, which the code recreates rows in on its own:
restore it and you will also revive rows the app had already replaced.


###############################################################################
DEPLOY ORDER
###############################################################################

Config first.  Code and migrations depend on which direction the migration goes.

Secrets in admin/.config must exist before the code that reads them.  tmbo_secret()
fails hard rather than falling back to a default, so a code deploy that lands first
fails closed on whatever paths use the missing key.

For an ADDITIVE migration -- a new nullable column, a new table -- run the migration
FIRST.  Old code ignores columns it does not select, so the migration is invisible
to the running site, and new code that selects the new column would otherwise fatal
until the migration lands.

For a DESTRUCTIVE migration -- dropping or renaming a column, changing a type --
deploy the code FIRST, so nothing is still reading what you are about to remove.

The failure mode when you get this backwards is not subtle but it is confusing:
tmbo_query() escalates any failed query to E_USER_ERROR, so a missing column takes
out every page that runs that query, while pages that do not are fine.


###############################################################################
THE THREE OFF SWITCHES
###############################################################################

None of these are in a config file.  Two are hand-edited on the server and do not
survive a git checkout of another commit -- see BEFORE YOU TOUCH ANYTHING.

$fixing in offensive/assets/header.inc.  Set it to true and non-admins get the
fixing page instead of the front end; admins pass through and see a notice.

  Know what it does NOT cover.  It is tested in index.php and nowhere else, so every
  other entry point that includes header.inc keeps serving and keeps writing:
  api.php, logn.php, registr.php, pwreset.php, setPref.php, subscribe.php, the
  *_rss.php feeds.  $fixing keeps people out of the web UI.  It is not a maintenance
  mode for the site.  If you need writes to actually stop, that is readonly.

$downtime in offensive/index.php, near the top, currently commented out.  Set it to
a unix timestamp and header.inc flips $upgrading once that time passes, which swaps
the front end for the upgrade page; before then index.php shows a countdown, and
$downtime_link alongside it points at an explanatory post.  Use it for planned work
you want announced.  Use $fixing for right now.  Note it is edited in index.php, not
in header.inc, which is only where it is read.

TMBO::readonly in offensive/classes/tmbo.inc.  Change the initialiser to true and
tmbo_query() discards every query that is not a SELECT or SHOW, and any query
containing a semicolon.  Silently: it returns without touching the database and
without raising anything.  Three consequences worth internalising:

  * Any write path lacking an explicit TMBO::readonly() guard appears to succeed and
    does not.  Most of classes/ has the guard.  Assume nothing.
  * core_createtoken() returns false, which is what makes the tokens table dangerous
    while readonly is set.  See the next section.
  * A discarded query returns null rather than a result, so callers that pass it
    straight to mysql_num_rows() will warn.

This one is also set at runtime, not only by hand: a request authenticated with the
" rss" token turns it on for that request.  So RSS requests are always in readonly
regardless of what the initialiser says.

The combination table surgery needs is writes ON and users OFF -- readonly false and
$fixing true.  Remember that $fixing does not stop the API or the feeds, so for
anything genuinely destructive, take the site out at the web server instead.
Reaching for only one of these is how a maintenance window becomes an outage.


###############################################################################
SANDBOX AND PROD SHARE A DATABASE
###############################################################################

sandbox is not an isolated copy of the site.  It runs against the same MySQL
database as production; the difference between the two is the UI, not the data.

The word "sandbox" implies somewhere safe to experiment.  Here it isn't.  Every
UPDATE, DELETE and ALTER you run from the sandbox host lands on production data.
There is no undo and no separate copy to fall back on.  Running destructive queries
"just on sandbox to test it" will break the live site.  In general, nobody should be
interacting with the database directly without a buddy.

Practical consequences:

* Migrations are run ONCE, not once per environment.  If a column exists on
  sandbox it is because it exists in production, and vice versa.
* Backups protect both or neither.  See the section above; take one.
* Test destructive changes against a local Vagrant VM, which does have its own
  database, not against sandbox.
* Sharing auth secrets between the two hosts is fine, and separating them buys
  very little.  The trust boundary is already shared: anyone who can deploy to
  sandbox can read and write production data directly.
* Cookies prod sets do not reach sandbox: setcookie() passes no domain, so they
  are host-only.  This does NOT hold in reverse.  Any host under thismight.be can
  set a cookie scoped to the parent domain, so sandbox can put a cookie into
  prod's scope.  Treat a sandbox compromise as reaching production.


###############################################################################
WHAT IS PER-ENVIRONMENT
###############################################################################

The code checkout, admin/.config, and the system configuration under /etc.

Neither of the last two is version controlled, so two hosts on identical commits
can still behave differently, and a config problem cannot be diagnosed from the
repository alone.

admin/configroot/ looks like it covers /etc and does not.  Those are the Vagrant
dev VM's templates.  vm_setup.sh installs them on the dev VM and nowhere else, so
do not read them as a description of any real server.

admin/.config is gitignored and holds the only copy of the database credentials and
the auth secrets.  Losing it takes the whole site down, not just authentication:
mysqlConnectionInfo.inc raises E_USER_ERROR when parse_ini_file() fails.  Back it
up somewhere outside this repository.


###############################################################################
THE TOKENS TABLE IS NOT WHAT IT LOOKS LIKE
###############################################################################

`DELETE FROM tokens` can take the entire site down.  It does not just log out API
clients.

The table mixes user API credentials with internal per-user tokens that the code
creates lazily and then assumes exist forever.  At the time of writing:

    " tmbo"      leading space.  constructor default, used by User::token()
    " rss"       leading space.  Link::rss(), embedded in RSS feed URLs
    "realtime"   NO leading space.  socket.io

Do not trust that list.  Enumerate:

    grep -rn "new Token(" --include="*.php" --include="*.inc" .

Token::getTokenRow() treats any row count other than one -- zero OR duplicates --
as "create a replacement", calls core_createtoken(), and dereferences the result
without checking it.

Whether that is fatal depends on the readonly switch, and this is the part that is
easy to get wrong:

  * readonly FALSE: core_createtoken() succeeds and the row is recreated on demand.
    A wholesale delete of the internal tokens mostly heals itself.
  * readonly TRUE: core_createtoken() returns false at its first line, and
    getTokenRow() calls a method on it.  On PHP 5 that reads

        Call to a member function tokenid() on a non-object in classes/token.inc

    on every page that renders, which is nearly all of them.

So the danger is not the DELETE by itself, it is a DELETE while writes are off.  If
you are purging this table, turn readonly off and $fixing on first.

To purge user API credentials while leaving the internal tokens alone:

    DELETE FROM tokens WHERE issued_to NOT IN (' tmbo', ' rss', 'realtime');

Know what that misses.  issued_to is whatever the client sent as its User-Agent
(see api_login in offensive/api.php), so a user token can be named "realtime" or
" rss" and will survive this DELETE.  The comparison is also case-insensitive under
the table's utf8 collation.  If the point of the purge is that credentials are
compromised, verify afterwards rather than assuming the NOT IN caught everything.

To rebuild internal tokens that have gone missing, do NOT write them by hand.  With
readonly off and $fixing on, browse the site as an admin: the pages that need each
token will recreate it through core_createtoken(), which checks for collisions and
draws from a CSPRNG.  Minting tokenids in SQL means minting bearer credentials out
of RAND() or UUID(), neither of which is one.

Deleting tokens is user-visible beyond the site itself: Link::rss() embeds a live
token in every feed URL, so subscribers' saved URLs stop working permanently and
API clients lose their credentials.  Say so somewhere before you do it.


###############################################################################
INVALIDATION LEVERS
###############################################################################

What you can revoke, and what it costs.  All of these are one-way.

Rotate remember_pepper in admin/.config.  Invalidates every remember cookie on the
site at once.  It writes nothing to the database, which makes it the only cookie
invalidation that works while readonly is set.  Not reversible unless you kept the
old value.

Change a user's password.  The remember cookie is an HMAC keyed on the stored
password hash, so changing it invalidates that user's cookies and nothing else.
This is the only per-user lever.  Note that neither the change-password page nor
the reset flow issues a replacement cookie or tells the user, so they silently stop
being remembered on every device.

Rotate activation_salt or pwreset_salt in admin/.config.  Permanently invalidates
every activation and password-reset email already sent.  There is no undo, only
asking those users to request a new one.

Bump TMBOSESS<n>.  PHP finds a session by the cookie named in session.name, so
changing the name orphans every session file on disk in one step.  Short of root on
the box this is the only session invalidation there is: expiring them properly needs
the garbage collector, and session.gc_probability is set in the server's php.ini,
which is not in this repository -- check it rather than assuming.  Bumping the name
is also the only way to change session cookie flags for users who already have a
session, since PHP sends Set-Cookie only when it creates one.

Note what a password change does NOT cover: it invalidates that user's remember
cookies, not their sessions.  Nothing rebinds a session to the password and nothing
regenerates the id, so a stolen session survives it.  For a compromised session the
only lever is the site-wide one above.

The name appears in three places -- header.inc, logn.inc and logout.php -- and
logout.php includes nothing, so it cannot share a constant.  Only the header.inc and
logout.php copies are reachable today; header.inc starts the session before logn.inc
is even loaded, so logn.inc's copy never runs.  Change all three anyway: a divergence
there is invisible until it isn't.


###############################################################################
AFTERWARDS: DID IT WORK, AND WHERE DID IT BREAK
###############################################################################

Confirm the deploy took effect rather than assuming it did, and pick a check that
distinguishes the version you just shipped from the one before it -- "the cookie
looks right" usually does not, because cookie shape rarely changes.  Something that
does: verify the schema matches what the code expects,

    SHOW COLUMNS FROM users;

and exercise one behaviour the change was for, end to end, as a real user.

Reading a failure.  There are three outcomes, and they look nothing alike:

  * A handled fatal, with output buffering on.  trigger_error(E_USER_ERROR) and any
    failed query go through tmbo_error_handler, which logs, then renders
    offensive/index.outoforder.php for non-admins.  That page sends HTTP 500.
  * A handled fatal, with output buffering off.  The kaboom branch requires
    ob_get_level() > 0, and nothing in the code calls ob_start() -- the one in
    header.inc is commented out -- so the buffer only exists if php.ini
    output_buffering is on.  Without it, the handler logs and exits with no page and
    no status set.  You get a blank response.
  * A real fatal.  Parse errors, E_ERROR, calling a method on a non-object: PHP does
    not route these to a user error handler, and nothing registers a shutdown
    function, so they never reach the kaboom page at all.  With display_errors on
    they print into the response body at whatever status was already set, which can
    be a 200 with an error message in the middle of an otherwise normal page.

Admins never get the kaboom page.  They get a var_dump of the backtrace inline
instead, which includes call arguments and so can contain plaintext passwords,
password hashes and config secrets.  Do not paste it into a chat or a ticket.

So a 500 is the good case: the handler ran.  A blank page or a 200 with an error in
it are the ones that need the log.  Check output_buffering and display_errors on the
host you are debugging before you interpret any of this; they are set in the server's
php.ini, which is not in this repository -- admin/configroot/ is the dev VM only.

Logs.  tmbo_error_handler calls error_log(), which goes wherever the error_log
directive points for the SAPI serving the request.  That is php-fpm, so check the
pool and fpm config rather than the CLI:

    php-fpm -i | grep error_log        # not `php -i`, which reads the CLI ini

The nginx error_log under ~/logs/ is a different file and will not necessarily have
PHP's output.  log_errors_max_len also truncates, so a backtrace in the log may be
cut off.


###############################################################################
ROLLING BACK
###############################################################################

Fix forward by default.  Roll back only if the new code is itself causing an outage,
and only for as long as the fix takes.

    git -C ~/sites/tmbo checkout <the commit you wrote down>

Before you do, check what you are reverting past.  Three things make an older commit
the more dangerous option rather than the safer one:

* A security fix.  Reverting one republishes the hole, and because this repository
  is public, the vulnerable code stays readable forever -- so anyone can tell what
  an old commit is vulnerable to.  Read the log for the range you are crossing:

      git -C ~/sites/tmbo log --oneline <old>..<current>

* A destructive migration.  Older code is fine with columns it does not select, so
  crossing an additive migration backwards costs nothing.  Crossing a DROP or a
  rename backwards means the old code selects something that is gone, and
  tmbo_query() escalates that to a fatal on every page that runs the query.  You
  would have to restore the schema too, so usually: don't go that far back.

* A session cookie rename.  Reverting the name makes every orphaned session
  reachable again.  If the name was bumped to invalidate sessions, rolling back
  undoes exactly that.

The off switches are uncommitted edits to tracked files, so a checkout discards
them.  If the site was in $fixing or readonly, put it back -- and put it back again
after you roll forward, for the same reason.

Do not leave production checked out on a detached commit or a feature branch.  Merge
first, then move the server back to develop, then delete the branch.  A branch
deleted out from under a live checkout is its own outage.