Setup & Usage

Contents

Setup & Usage#

Overview#

The VR4Mice pipeline is a DataJoint-based workflow for ingesting raw rig data, computing analysis tables, running DLC and interpolation, and generating summary outputs. It is designed to run from the client container and can be scheduled via cron.

End-to-end workflow (data acquisition → emails)#

  1. Acquisition (rig): .pickle files (and optional .npy metadata via gui_transfer/).

  2. Ingest: run.py populate → raw tables in vr4mice/base.

  3. Analysis: run.py analysis → base_analysis tables (DataFrame, BoxDataFrame, GitCommit).

  4. DLC + interpolation: run.py dlc then run.py interp (includes session_metrics tables).

  5. Latency: run.py latency → photodiode alignment tables.

  6. Environment-specific (manual or via cron_scenario.py):

    • Local server: run.py inputs_videos, then run.py fetch → refreshes /shared/gui_menu.npy for the rig GUI.

    • AWS / remote (--aws): run.py decision on /data/processed.

  7. Summaries: run.py summary → SummaryPlots for sessions with both DataFrame and BoxDataFrame; cron also runs summary_emails.send_pending_summary_emails to send or retry plot notification emails (see Emails).

After upgrading to DataJoint 2.x (once per rig, before relying on cron):

# Existing database only: add :<blob>: markers to column comments (dry-run first)
python scripts/migrate_to_dj2.py --dry-run
python scripts/migrate_to_dj2.py

# Always after a DJ 2.x upgrade (and after migrate_to_dj2 on existing DBs)
python run.py maintenance

  • migrate_to_dj2.py — metadata-only; fixes blob column comments on legacy databases. Skip on a fresh DJ 2.x install (it will report nothing to migrate).

  • run.py maintenance — rebuilds DataJoint lineage tables for all pipeline schemas (including summary_emails). Does not repopulate computed tables.

Normal cron runs do not replace either step.

Repository layout (dj_pipeline)#

  • run.py: manual CLI entrypoint for running pipeline modes (user mode).

  • cron_scenario.py: scheduled runner for full pipeline execution (crontab call).

  • vr4mice/: core schema, analysis, and action code.

  • gui_transfer/: GUI configuration and data transfer tools.

  • base/: base schemas and actions (full and minimal modes: mice, exp tables, connections and email functions).

  • backup/: backup helpers (local/remote sync scripts).

  • cron_*.sh: cron wrappers for scheduled runs (standard, reboot, AWS).

  • Dockerfile + requirements-docker.txt: client image build and pinned runtime dependencies.

  • docker/entrypoint.sh: creates runtime user (PUID/PGID) and drops privileges with gosu.

  • docker/cron_common.sh: shared helpers for cron scripts and compose exec.

  • generators/: small utilities to create helper tables/files (e.g., add mice).

  • parse_mice/: parsing helpers for mouse metadata files.

  • mysql_access/: SQL templates for user setup.

  • notebooks/: analysis and figure notebooks (research use).

  • Logs are written on the server in a local logs/ folder and are not part of the repo.

vr4mice module guide#

See vr4mice/README.md. Key components:

  • schema/: DataJoint table definitions.

  • analysis/: analysis helpers and plotting.

  • actions/: orchestration helpers (ingest, sync, fetch).

  • utils/: logging, schema config, bootstrap, populate_helpers, maintenance.

gui_transfer#

Rig GUI and transfer utilities:

  • gui.py + config/ for metadata capture.

  • utils/ and modules/ for GUI logic.

  • Dropdown menu file (gui_menu.npy) generated on the server by fetch_data.py and copied to the rig at GUI startup (see GUI dropdown menu and rig setup).

  • Rig session filenames ({mouse}_{date}_{attempt}, typed suffixes) — see Rig filename contract and gui_transfer/README.md.

base (schemas)#

  • base_schemas: full exp and mice schema definitions.

  • base_min_schemas: minimal exp/mice for GUI + collab when full schemas are not available.

Getting Started — For Everyday Users#

This quick guide helps you connect to the database and run the pipeline without deployment experience.

Requirements#

  • Bash

  • Docker + Docker Compose

  • GNU Make

  • Database credentials (host/port, user, password)

Quick start — client mode (connect to an existing DB)#

Use when the database already runs elsewhere.

bash quick_start.sh

Choose:

  • Mode: client

  • DJ host: ip:port or hostname (e.g., 127.0.0.1:3309)

  • DJ user / DJ password

This starts only the client container.

Quick start — deployment mode (DB + client on server)#

Use when deploying both database and client containers on a server.

bash quick_start.sh

Choose:

  • Mode: deployment

  • Provide mount paths for database, data, shared, and screen recordings

  • Set DB bind IP/port and MySQL root password

Defaults shown by the script:

  • /mnt/database/vr4mice/vr4mice_database/database → /var/lib/mysql

  • /mnt/database/shared → /shared

  • /mnt/database/vr4mice/vr4mice_database/data → /data

  • /mnt/neuropixel_data/vr4mice/raw_screen_recordings → /vr4mice_screen_recordings

  • repo root (./) → /app

  • ./base/base_min_schemas → /base_schemas

  • ./base/base_actions → /base_actions

Quick start — import DB dumps#

Use this when you have restricted_dump_*.sql files or a .zip/.tar.gz archive.

bash quick_start.sh

When prompted:

  • Import DB dumps now? → yes

  • Dump directory or .zip/.tar.gz archive → path to the dump folder or archive

The script creates missing databases and imports each restricted_dump_*.sql. Imports stay in the foreground and show progress (via pv or dd status=progress when available).

Step 1 — Clone the repo#

git clone git@github.com:MMathisLab/FreelyMovingVR4Mice.git
cd dj_pipeline

Step 2 — Configure credentials#

Update your credentials in .env (or env.py for local conda runs).

vim .env

Step 3 — Start the client container#

make client_build
make client_up

Step 4 — Connect via IPython or Jupyter#

make ipython
%run run.py connect

Optional Jupyter:

make notebook

Remote Jupyter via SSH tunnel:

ssh -NL 8887:localhost:8887 <user>@<server>

Append & if you want the tunnel in the background. You can add an SSH config alias (e.g., wm) and then connect with:

ssh -NL 8887:localhost:8887 wm

Step 5 — Verify connection#

from vr4mice.schema import vr4mice
vr4mice.Dataset()

Stop the client#

make client_down

Host Jupyter (legacy / outside Docker)#

  1. Install matching packages and base schemas:

    pip install notebook "datajoint>=2.0.1"
pip install base/base_actions/
pip install base/base_min_schemas/

  2. Install graphviz if you want schema diagrams: sudo apt install -y graphviz

  3. Update env.py (server IP, credentials from your administrator)

  4. Start jupyter notebook, open a Python 3 kernel, then:

    %run env.py
%run run.py connect
from vr4mice.schema import vr4mice
vr4mice.Dataset()

Deployment (server)#

Server deployment (database + client containers)#

The server runs two containers:

  • Database (MySQL/DataJoint)

  • Client (pipeline runner)

Note: store MySQL admin credentials in ~/.my.cnf on the server.

make build_all
make up_all
make mysql
make ipython
%run run.py connect

Docker client: base packages and user mapping#

The client image is built from DeepLabCut’s Docker image. Build-time Python dependencies are pinned in requirements-docker.txt and installed by Dockerfile (pip install -r requirements-docker.txt). Pipeline commands use plain python -m pip (no conda activate). At build time and container start, docker/ensure_python_shims.sh symlinks the base image’s Python into /usr/local/bin so make, cron, and docker compose exec all find python on the compose PATH.

base_schemas / base_actions are bind-mounted from the repo and installed at runtime with pip:

python -m pip install --user --no-deps /base_schemas/
python -m pip install --user --no-deps /base_actions/

make client_up, make base_install, and the cron scripts run pip as the caller’s UID/GID; packages and cache live under /app/.local and /app/.cache/pip on the bind-mounted repo (not under /home/<username>), so any user with access to the repo can run the client.

Running as your host user (file permissions on /app, /data, etc.) is handled by Docker, not conda:

Layer

Mechanism

Container start

docker/entrypoint.sh — useradd with PUID/PGID/USERNAME, then gosu

Compose

passes UID, GID, USER_NAME from the host (Makefile exports these)

make / cron exec

docker compose exec --user "$(UID):$(GID)" client …

Host conda (env.py, vr4mice_env on the rig, DLCliveGUI) is a separate install path for non-Docker workflows. It is not used inside the client container.

Mounts and storage#

In docker-compose.yml, map host storage for database and data volumes:

  • /data and /data/summary_plots must exist and be writable.

  • /shared is used for GUI menu exports.

  • Use persistent paths (e.g., /mnt/database/...) on the server. Network mode:

  • Default is host. Set CLIENT_NETWORK_MODE=bridge in .env.compose if you need bridge networking.

Deployment defaults (quick_start.sh)#

When using bash quick_start.sh in deployment mode, the default host paths map to:

  • /mnt/database/vr4mice/vr4mice_database/database → /var/lib/mysql

  • /mnt/database/shared → /shared

  • /mnt/database/vr4mice/vr4mice_database/data → /data

  • /mnt/neuropixel_data/vr4mice/raw_screen_recordings → /vr4mice_screen_recordings

  • repo root (./) → /app

  • ./base/base_min_schemas → /base_schemas

  • ./base/base_actions → /base_actions

Database deployment notes (server)#

  • Add user to Docker group:

    sudo usermod -aG docker <username>

  • Create host storage paths (example):

    mkdir -p /mnt/database/vr4mice/vr4mice_database/database
mkdir -p /mnt/database/vr4mice/vr4mice_database/data
mkdir -p /mnt/database/vr4mice/vr4mice_database/data/data/data
mkdir -p /mnt/database/vr4mice/vr4mice_database/data/data/dlc_video
mkdir -p /mnt/database/vr4mice/vr4mice_database/data/summary_plots
mkdir -p /shared

  • Add MySQL credentials in ~/.my.cnf for make mysql:

    [client-vr4mice]
host=127.0.0.1
user=root
password=simple
port=3309

Environment variables#

Pipeline (.env — copy from .env.example; loaded into the client container):

  • DJ_HOST (include port, e.g. 127.0.0.1:3309), DJ_USER, DJ_PWD

  • DJ_LAB, GUI, EMAIL, IMG_SRC, VR4MICE_EMAIL_RECIPIENTS, VR4MICE_EMAIL_SINCE

When using the local Docker database, DJ_HOST port must match DB_PORT in .env.compose.

Docker Compose (.env.compose — copy from .env.compose.example; used by make, cron, and docker compose only):

  • COMPOSE_PROJECT (default vr4mice) — must match between make, cron scripts, and manual docker compose -p … calls

  • DB_BIND_IP, DB_PORT, MYSQL_ROOT_PASSWORD

  • DB_DATA_PATH, SHARED_PATH, DATA_PATH, SCREEN_RECORDINGS_PATH

  • DB_IMAGE — Docker image for the local database service (see below)

  • CLIENT_IMAGE, CLIENT_CONTAINER_NAME, DB_CONTAINER_NAME, CLIENT_NETWORK_MODE, JUPYTER_PORT

MySQL version (5.7 and 8.0)#

The database service in docker-compose.yml uses ${DB_IMAGE:-mysql:8.0}.

Situation

What to use

New install (empty DB_DATA_PATH)

mysql:8.0 (default)

Existing on-disk datadir from MySQL 5.7

mysql:5.7

If DB_IMAGE is not set in .env.compose, docker/compose_env.sh (used by make and cron) auto-detects the image from DB_DATA_PATH via docker/detect_mysql_image.sh. quick_start.sh writes the detected value into .env.compose.

Manual override in .env.compose:

DB_IMAGE=mysql:5.7   # legacy datadir
# DB_IMAGE=mysql:8.0 # new installs

Check what Compose will use:

bash docker/compose_env.sh get DB_IMAGE

Upgrading 5.7 → 8.0 (keep data): run MySQL 5.7 against the datadir, shut down cleanly, then switch to 8.0:

docker compose exec db mysql -uroot -p"${MYSQL_ROOT_PASSWORD}" \
  -e "SET GLOBAL innodb_fast_shutdown=0; SHUTDOWN;"
# set DB_IMAGE=mysql:8.0 in .env.compose, then:
make up_all

Fresh start (discard local datadir): stop the stack, move or delete DB_DATA_PATH, then start with DB_IMAGE=mysql:8.0.

If the db container exits immediately with code 1 and logs mention a failed upgrade from MySQL 5.7, the datadir version likely does not match DB_IMAGE — set DB_IMAGE=mysql:5.7 or wipe the datadir for a new 8.0 install.

Remote/AWS DB credentials for scheduled AWS runs live in .env-aws (copy from .env-aws.example); this file is not committed.

If you still have Docker settings in .env from an older setup, move them to .env.compose (Makefile falls back to .env for COMPOSE_PROJECT only during migration).

Notes:

  • Default summary email recipients come from VR4MICE_EMAIL_RECIPIENTS in .env (comma-separated experimenter names, e.g. mathislab). The session experimenter from exp.Session is added when present. See Emails.

  • VR4MICE_EMAIL_RECIPIENTS is required if base schemas (exp/mice) are not in use, because experimenter names are otherwise missing from GUI metadata.

  • VR4MICE_EMAIL_SINCE (format YYYY-MM-DD) limits automatic summary emails to new sessions on or after that date. If unset, cron does not send summary emails (avoids emailing the full backlog). Set this on the rig when enabling EMAIL=true.

Schemas (what they do)#

  • vr4mice: core tables for datasets, raw signals, metadata, and derived features.

  • base: links datasets to exp and mice schema information.

  • base_analysis: analysis outputs and summary plots.

  • summary_emails: tracks summary plot notification emails (SummaryPlotEmail).

  • dlc, interpolated_trajectories, latency_tests, decision, inputs_videos: downstream analysis modules.

Schema modes#

Two base schema modes are supported:

  • Full base schemas: base_schemas (exp/mice full definitions).

  • Minimal schemas: base_min_schemas (minimal exp/mice for collab + basic metadata).

Both modes work; choose minimal when only GUI dropdowns and basic metadata are needed.

Data import/export (restricted dumps)#

See Data import/export for export/import workflows, or:

  • Exporting restricted dumps (export_restricted_dump.sh)

  • Importing dumps with quick_start.sh

  • Manual mysql import

See Deploy the DataJoint database locally for a full local deploy workflow that clones the data archive, runs quickstart, and connects Jupyter.

GUI vs non-GUI mode#

  • GUI mode (GUI=true): expects .pickle + .npy inputs.

  • Non-GUI (GUI=false): uses .pickle only; metadata is reconstructed as needed. Use GUI mode when experiment metadata is collected via the rig GUI.

GUI dropdown menu and rig setup#

The rig GUI (gui_transfer/) uses a DataJoint-exported .npy file for dropdown menus (mice, experimenters, rigs, tasks, etc.). The file is generated on the pipeline server and copied to the rig when the GUI starts.

Server: generate the menu file#

Item

Default path / value

Menu output (in container)

/shared/gui_menu.npy

Host mount

SHARED_PATH in .env.compose → /shared in the client container

Generator script

vr4mice/actions/fetch_data.py

CLI entrypoint

run.py fetch

Generate or refresh the menu manually (inside the client container or via Jupyter):

python run.py fetch

This queries exp / mice tables and writes /shared/gui_menu.npy. On local (non-AWS) cron runs, cron_scenario.py executes this as the final step after populate, analysis, DLC, and inputs_videos tables. AWS cron runs do not export the GUI menu.

Ensure the shared directory exists on the host (see Deployment defaults):

mkdir -p /mnt/database/shared   # or your SHARED_PATH

Rig: configure paths and fetch the menu#

Linux / macOS rig#

  1. Copy gui_transfer/ to the rig computer.

  2. Install Python 3 + PyQt5 (make env from gui_transfer/).

  3. Create the GUI config from the template (do not commit the result):

    cp gui_transfer/config/local_config.json.example gui_transfer/config/config.json

  4. Edit gui_transfer/config/config.json. Key fields for the dropdown menu:

    Key

    Purpose

    Example

    ip

    Pipeline server IP (localhost for local testing)

    192.168.1.10

    host

    SSH user on the server (empty if not needed for localhost)

    vr4mice

    remote_dropdown_menu

    Path to menu file on the server

    /shared/gui_menu.npy

    host_dropdown_menu

    Local copy path on the rig

    ./gui_menu.npy

    On startup, config.get_menu_path (a property on the Config instance) copies the menu file to the rig:

    • localhost: local file copy from remote_dropdown_menu to host_dropdown_menu (both paths must be readable on the same machine)

    • remote server: scp host@ip:remote_dropdown_menu → host_dropdown_menu (requires SSH access and the menu file on the server)

    Also set transfer paths (remote_dst, gui_output_folder, teensy_path, processed_path, etc.) for your rig layout. See the inline example in gui_transfer/config/config.py.

  5. Start the GUI:

    cd gui_transfer
make run_gui

Windows rig#

Most VR rigs run Windows 10/11. You do not need Docker, WSL, or GNU Make on the rig — only Python 3, the gui_transfer/ folder, and OpenSSH (scp/ssh) to reach the pipeline server.

1. Install Python 3.9+

  • Download from python.org.

  • During setup, enable “Add python.exe to PATH”.

  • Verify in Command Prompt: python --version (or py -3 --version).

2. Install OpenSSH Client (for scp)

  • Settings → Apps → Optional features → Add a feature → OpenSSH Client.

  • Verify: scp and ssh run without “not recognized”.

3. Copy the GUI folder

Copy FreelyMovingVR4Mice/dj_pipeline/gui_transfer/ to the rig, e.g. C:\vr4mice\gui_transfer\.

4. Install Python dependencies

cd C:\vr4mice\gui_transfer
python -m pip install --upgrade pip
python -m pip install PyQt5 numpy "moviepy>=1.0.3"

5. Create and edit config

cd C:\vr4mice\gui_transfer\config
copy windows_config.json.example config.json

Edit config.json with your server IP, SSH user, and Windows paths (use forward slashes, e.g. C:/vr4mice/raw). See gui_transfer/README.md for a full key table.

Required for a production rig:

Key

Example

ip

192.168.1.10

host

vr4mice

remote_dropdown_menu

/shared/gui_menu.npy (path on the Linux server)

host_dropdown_menu

C:/vr4mice/gui_transfer/gui_menu.npy

teensy_path, dlc_path, …

Your local data folders

remote_dst

/data/data (destination on the server)

6. Test SSH / menu download

ssh vr4mice@192.168.1.10
scp vr4mice@192.168.1.10:/shared/gui_menu.npy C:\vr4mice\gui_transfer\gui_menu.npy

Set up an SSH key if you do not want a password prompt every session (see gui_transfer/README.md).

7. Preflight check (recommended)

cd C:\vr4mice\gui_transfer
check_rig_setup.bat
check_rig_setup.bat --test-menu

8. Start the GUI

cd C:\vr4mice\gui_transfer
run_gui.bat

Or double-click run_gui.bat. If the window closes immediately, run the same commands from Command Prompt to read the error.

Local dry-run (no SSH, Linux/macOS): from gui_transfer/test/ run make menu config build_tree && make run_gui. Creates fake menu and session files under /tmp/vr4mice_test_gui and runs the GUI on localhost (no scp). Full walkthrough: gui_transfer/README.md → Local test.

If the menu file is missing or scp fails, the GUI logs a warning and exits — fix paths/credentials and ensure run.py fetch has been run on the server before restarting the GUI.

Canonical GUI code: deploy FreelyMovingVR4Mice/dj_pipeline/gui_transfer/ on rigs. Do not use the legacy flat layout from auxPipelines-DataJoint_Mathis/vr4mice/gui_transfer/.

Rig filename contract#

The rig GUI and populate_rig on the server assume the same session filenames. Auto-discovery (select one file → find related pickle / TS / DLC / PROC / VIDEO), metadata auto-fill, and pipeline ingest all rely on this.

Dataset stem: {mouse_name}_{YYYY-MM-DD}_{attempt} (e.g. Testmouse_2023-02-22_2).

Typical session files (camera prefix = IMG_SRC, default Imagingsource):

File

Example

Teensy

Testmouse_2023-02-22_2.pickle

Timestamps

Imagingsource_Testmouse_2023-02-22_2_TS.npy

DLC

Imagingsource_Testmouse_2023-02-22_2_DLC.hdf5

Processed

Imagingsource_Testmouse_2023-02-22_2_PROC

Video (rig only)

Imagingsource_Testmouse_2023-02-22_2_VIDEO.avi

Classification uses keyword tags (TS, DLC, VIDEO, PROC) and glob patterns in gui_transfer/modules/transfer.py; parsing lives in gui_transfer/utils/session_files.py. The server mirror is vr4mice/actions/populate_rig.py → get_files_paths().

If naming changes, update GUI + populate + tests in one change set — patterns are not configurable in config.json. Full checklist and limitations: dj_pipeline/gui_transfer/README.md → Rig filename contract.

Further GUI module details: dj_pipeline/gui_transfer/README.md.

The GUI transfers experiment metadata and rig files only (videos stay on the rig).

Manual runs (run.py)#

Use --verbose for DEBUG logging and DataJoint lineage details (cron stays quiet by default).

Typical sequence:

%run run.py populate
%run run.py analysis
%run run.py summary
%run run.py fetch    # refresh GUI dropdown menu on /shared
...

One-off after a DataJoint upgrade or when adding a new schema:

%run run.py maintenance

See Maintenance for what this step does and when it is required.

Maintenance#

python run.py maintenance rebuilds DataJoint lineage metadata. It is a one-off setup step — not part of cron or normal populate / analysis runs.

What it maintains#

Each pipeline schema (mice, exp, vr4mice, base, base_analysis, dlc, …) has a hidden table called ~lineage. DataJoint 2.x stores, per table attribute, where that attribute came from in the foreign-key dependency graph (semantic lineage). Joins and populate() use this to match attributes that share the same origin, not just the same column name.

run.py maintenance calls schema.rebuild_lineage() on every pipeline schema (in dependency order) and repopulates those ~lineage rows from current FK definitions.

What it does not touch#

  • Raw rig files (.pickle, .npy, DLC outputs)

  • Computed table contents (DataFrame, SummaryPlots, DLC, etc.)

  • Blob column comments (scripts/migrate_to_dj2.py handles that separately)

When to run#

Situation

migrate_to_dj2.py

run.py maintenance

Legacy DB upgraded to DJ 2.x

Yes (first)

Yes (after)

Fresh DJ 2.x database

Skip (no-op)

Yes (once)

New schema added to pipeline

No

Yes (once)

Lineage / join errors

Maybe

Yes

After a DJ 2.x upgrade on an existing database:

python scripts/migrate_to_dj2.py --dry-run
python scripts/migrate_to_dj2.py
python run.py maintenance

See also docs/migration/minimum_migration_guide.md.

Testing / quick sanity#

  • Populate test:

    %run run.py populate
from vr4mice.schema import vr4mice
vr4mice.Dataset()

  • Analysis test:

    %run run.py analysis

Cron and Docker operations#

Scheduled pipeline runs use wrapper shell scripts that call docker compose with project name vr4mice (override via COMPOSE_PROJECT in .env.compose or the environment). Shared logic lives in docker/cron_common.sh.

Architecture#

Component

Role

cron_scenario.py

Runs populate → analysis → DLC → … with per-step logging; exits non-zero if any step fails

docker/cron_common.sh

Shared compose/exec helpers (project name, UID/GID, pip install)

cron_script.sh

Local nightly run (/data/data, video tables, shared export)

cron_script_aws.sh

AWS/remote run (/data/processed, decision tables); requires .env-aws

cron_script_reboot.sh

After host reboot: wait 120s, start db + client, install base packages

Makefile

Same compose project and exec pattern as cron scripts

Docker client vs host conda: the client container uses system Python + pip (DeepLabCut base image). Conda on the rig or laptop (env.py, vr4mice_env, DLCliveGUI) is a separate install path and is not used inside the Docker client. Base package install and user mapping are described under Docker client: base packages and user mapping.

Makefile targets (common)#

Target

Purpose

make client_up

Start client, install base_schemas / base_actions

make up_all

Start db + client, install base packages

make base_install

Re-install base packages in running client

make ipython / make notebook

Interactive session in client

make add-cron

Install/update crontab entries (see below)

make cron-local

Manual local cron run

make cron-aws

Manual AWS cron run

make cron-reboot

Manual reboot startup script

All compose commands use docker compose -p vr4mice by default (COMPOSE_PROJECT in Makefile / .env.compose).

Crontab setup#

From dj_pipeline/ on the server:

make add-cron

This merges (does not wipe) your crontab and registers:

Schedule

Script

Log

@reboot

cron_script_reboot.sh

~/vr4mice_logs/cron_reboot.log

0 2 * * *

cron_script.sh && cron_script_aws.sh

~/vr4mice_logs/cron.log then ~/vr4mice_logs/cron_aws.log

The nightly job runs local first; AWS runs only if local exits successfully (&& chain). cron_scenario.py must exit 0 (no failed steps) for AWS to start.

Prerequisites on the server#

  1. .env — pipeline / DataJoint credentials (see .env.example)

  2. .env.compose — Docker Compose settings (see .env.compose.example)

  3. .env-aws — remote DB credentials for AWS runs (see .env-aws.example)

  4. Docker daemon running; cron user in the docker group

  5. Run make add-cron from dj_pipeline/ (paths are resolved from that directory)

Manual test (before enabling cron)#

cd dj_pipeline
make cron-local && make cron-aws
# or:
bash cron_script.sh && bash cron_script_aws.sh

Migrating compose project name#

Older setups may have containers under project mysqltest. After updating:

COMPOSE_PROJECT=mysqltest make down_all   # stop old stack

Then start with your project name in .env.compose (or legacy COMPOSE_PROJECT=mysqltest make up_all if you have not migrated yet).

Multiple deployments on one server#

Other vr4mice_* compose projects on the same host are fine for the primary stack (COMPOSE_PROJECT=vr4mice) as long as container names and DB_PORT do not clash.

make check-compose-project blocks only real collisions:

  • container names (vr4mice_db, vr4mice_${USER}) already used by another compose project

  • host port (DB_PORT, default 3309) already in use by another stack

For a second deployment (sandbox / personal DB), use unique values in .env.compose:

COMPOSE_PROJECT=vr4mice_dev
DB_CONTAINER_NAME=vr4mice_db_dev
CLIENT_CONTAINER_NAME=vr4mice_${USER}_dev
DB_PORT=3310
DB_DATA_PATH=/path/to/dev/database/
DATA_PATH=/path/to/dev/data/

Use separate data paths so MySQL volumes do not overwrite each other.

To skip the check (not recommended): VR4MICE_COMPOSE_FORCE=1 make up_all.

Shared repo checkout (another user, same server)#

When the pipeline lives in a shared folder (e.g. /mnt/database/shared/FreelyMovingVR4Mice/dj_pipeline) and was first set up under someone else’s account:

Do not run git config --local user.* in that tree — it is stored inside .git/ and affects everyone using the same checkout. Prefer per-session env vars or a personal clone for heavy git work.

Git — use your own credentials#

There is no make target that changes GitHub SSH keys (those live in ~/.ssh/). Use:

Command

Purpose

make git-whoami

Show remote, local/global git name, env overrides

eval "$$(make git-user-env NAME='…' EMAIL='…')"

Set identity for this shell only (shared checkout)

make git-user-local NAME='…' EMAIL='…'

Set --local config (personal clone only; affects all users on a shared tree)

SSH remotes use ~/.ssh/ of the logged-in user, not the repo owner. As the new user:

cd /path/to/shared/FreelyMovingVR4Mice/dj_pipeline

make git-whoami
ssh -T git@github.com

# Shared checkout (recommended):
eval "$$(make git-user-env NAME='Your Name' EMAIL='you@example.com')"
git pull

Personal clone only:

make git-user-local NAME='Your Name' EMAIL='you@example.com'

If git pull / git push fails with permissions on .git/, ask the admin to fix group write access on the repo, or clone your own copy elsewhere:

git clone git@github.com:MMathisLab/FreelyMovingVR4Mice.git ~/FreelyMovingVR4Mice
cd ~/FreelyMovingVR4Mice/dj_pipeline

Docker — client under your username#

Each user should have their own client container and (if needed) compose project. Production DB is often shared; client containers are per user.

Client only (connect to an existing DB — typical on a shared server):

cd /path/to/shared/FreelyMovingVR4Mice/dj_pipeline

# Per-user compose project (avoids clashing with others' containers) — in .env.compose or export:
export COMPOSE_PROJECT="vr4mice_${USER}"

# Pipeline credentials stay in .env (or export DJ_* if the shared .env is not yours)

make check-compose-project
make client_up          # container name defaults to vr4mice_${USER}
make ipython

Full stack (your own DB + client — dev / sandbox):

cd /path/to/shared/FreelyMovingVR4Mice/dj_pipeline

export COMPOSE_PROJECT="vr4mice_${USER}"
export DB_PORT=3310     # pick a free port; check with: ss -ltn | grep 3309

# Point to your own data dirs in .env.compose (or quick_start.sh), then:
make check-compose-project
make up_all

To persist settings, add to .env.compose (file is gitignored):

COMPOSE_PROJECT=vr4mice_${USER}
CLIENT_CONTAINER_NAME=vr4mice_${USER}
# DB_CONTAINER_NAME / DB_PORT / DB_DATA_PATH only if you run your own database

Stop your client without touching others:

COMPOSE_PROJECT=vr4mice_${USER} make client_down

AWS mode (pipeline behaviour)#

--aws mode uses /data/processed and disables moving raw files.

  • run.py --aws populate

  • cron_scenario.py --aws

Backup and AWS sync#

Backup helpers live in backup/ and cron scripts:

  • backup/backup.sh

  • cron_script_aws.sh

These scripts are intended for append-only sync to AWS or remote storage. If older versions should be replaced, manual cleanup may be required before re-populating. Update paths and credentials to match your storage backend.

Error handling and FailedSession#

Failures are recorded in vr4mice.FailedSession keyed by (dataset, failed_table_name):

  • Used as a per-table skip list during populate_pending (a photodiode skip does not block SummaryPlots).

  • Some entries are intentional permanent skips (e.g. no photodiode signal, no trials after excluding initialization trial 1).

  • Remove entries to retry after fixing data or deploying a fix.

Example removal (one table):

from vr4mice.schema import vr4mice
(vr4mice.FailedSession() & 'dataset="Grizzly_2026-01-29_1"' & 'failed_table_name="DataFrame"').delete()

Bulk cleanup after duplicate-key fixes:

(vr4mice.FailedSession() & 'failed_table_name="SignalsPhotodiode"' & 'error_message LIKE "%Duplicate entry%"').delete()

GitCommit table#

base_analysis.GitCommit stores commit hash and modified files for provenance.

Emails#

Summary emails are sent when EMAIL=true and VR4MICE_EMAIL_SINCE is set.

Who receives emails#

Recipients are resolved in two layers:

  1. Lab default list — VR4MICE_EMAIL_RECIPIENTS in .env (comma-separated experimenter names looked up in exp.Experimenter), e.g. VR4MICE_EMAIL_RECIPIENTS=mathislab or mathislab,experimenter_a,experimenter_b.

  2. Session experimenter — if exp.Session has an experimenter for that dataset, their mail is added when not already on the list.

Names must match experimenter_name in exp.Experimenter (lowercase). Addresses always come from exp.Experimenter.mail, not from the env var directly.

How to add or change recipients#

  1. Add the person to exp.Experimenter with a valid mail address (required for lookup). From Python or a notebook:

    from base_schemas.schemas import exp
exp.Experimenter.insert1(
    dict(
        experimenter_name="experimenter_a",
        full_name="Example User",
        mail="user@example.com",
    ),
    skip_duplicates=True,
)

  2. Include them in the default list — add their experimenter_name to VR4MICE_EMAIL_RECIPIENTS in .env.

  3. Session-only recipient — assign that experimenter on exp.Session for the dataset; they are added automatically for that session’s summary email.

Enable on the rig:

EMAIL=true
VR4MICE_EMAIL_RECIPIENTS=mathislab
VR4MICE_EMAIL_SINCE=2026-07-01   # only sessions on/after this date

Tracking and retry: the summary_emails schema stores one row per send attempt in SummaryPlotEmail (timestamp, recipients, error if any). After SummaryPlots populate, cron calls send_pending_summary_emails to send or retry emails for eligible sessions (those with a plot row, on/after VR4MICE_EMAIL_SINCE, and no successful send yet).

Inspect pending or sent emails:

from vr4mice.schema import summary_emails
summary_emails.pending_summary_email_keys()
(summary_emails.SummaryPlotEmail() & "send_error IS NULL").fetch()

Logs#

Runtime logs are written to the local logs/ folder on the server/container. Cron logs per-step timing ([cron] start/done …) at INFO; routine skip and duplicate messages are DEBUG unless you run with --verbose.

Contents