Version: v1

Migrations

Migrating content from a legacy site means Drupal has to read that site's database. Vortex can run that database for you as an optional second database. Turning the migration database on adds 3 parts:

A database2 Docker service next to the primary database
A settings.migration.php file that registers the $databases['migrate'] connection, which Drupal's Migrate API uses by default
A provision script that imports the legacy dump and runs your migrations in every environment except production

This page is for developers bringing content from a legacy site into a project built from Vortex. It shows how to turn the migration database on, fetch the legacy dump and run your migrations during provisioning. It also lists every variable and command involved. For writing the migrations themselves, see the Migrate API overview in the Drupal documentation.

Enable the migration database

With the installer

When you run the Vortex installer, answer Yes at the Use a second database for migrations? prompt. Then choose where the migration dump comes from at the Migration database source prompt. The source defaults to your hosting provider, acquia or lagoon, and to url without a hosting provider.

Manual setup

To add migration support to an existing Vortex project, copy the following pieces from a fresh Vortex install with migration enabled:

In docker-compose.yml, add the database2 service, the DATABASE2_* environment variables and the database2 entries in the wait-for-dependencies service.
In .env, add VORTEX_FETCH_DB2_FILE, VORTEX_FETCH_DB2_SOURCE, VORTEX_FETCH_DB2_ENVIRONMENT and the variables for your fetch source, such as VORTEX_FETCH_DB2_URL.
Create web/sites/default/settings.migration.php (see Drupal settings).
In web/sites/default/settings.php, add the block that loads settings.migration.php, between the per-module overrides and the local overrides.
Add the migrate connection to the expected values in tests/phpunit/Drupal/DatabaseSettingsTest.php and tests/phpunit/Drupal/EnvironmentSettingsTest.php.
Add the DATABASE2_ prefix to tests/phpunit/Drupal/SettingsTestCase.php. Without steps 5 and 6, the drupal_settings tests fail on the extra connection.
Add scripts/provision-20-migration.sh and make it executable.
Add the fetch-db2, reload-db2 and db2 commands to .ahoy.yml.
Require drupal/migrate_plus and drupal/migrate_tools.
Add a migration module. The provision script installs ys_migrate.
Add the "Fetch migration DB" step to the database job and the dump copy to the build job of your CI configuration. On Lagoon, also add the "Fetch migration database" task to .lagoon.yml.

Confirm that ahoy fetch-db2 saves the dump as .data/db2.sql, and that the next ahoy provision prints Verified migration source database.

Example workflow

The following steps use the url fetch source on a project with the migration database enabled:

Set VORTEX_FETCH_DB2_URL in .env (or .env.local):

VORTEX_FETCH_DB2_URL=https://example.com/legacy-database.sql

Fetch the migration database:
```
ahoy fetch-db2
```
Build and provision:
```
ahoy build
```
Add your migrations to scripts/provision-20-migration.sh:
```
run_migration my_migration
```
Re-provision to run migrations:
```
ahoy provision
```

Confirm that the provisioning output shows Migrated: my_migration. and that the drush migrate:status table at the end lists my_migration.

Environment variables

The following table lists the variables that configure the migration database:

Variable	Default	Description
`VORTEX_FETCH_DB2_FILE`	`db2.sql`	Migration database dump filename
`VORTEX_FETCH_DB2_SOURCE`	`url`	Fetch source (`url`, `ftp`, `acquia`, `lagoon`, `container_registry`, `s3`)
`VORTEX_FETCH_DB2_URL`	None	URL to fetch the migration database dump
`VORTEX_FETCH_DB2_ENVIRONMENT`	`prod`	Environment to fetch from on Acquia and Lagoon (the installer sets `main` for Lagoon)
`VORTEX_DB2_IMAGE`	None	Migration database container image, for the `database2` service and the `container_registry` source
`DATABASE2_HOST`	`database2`	Migration database host
`DATABASE2_NAME`	`drupal`	Migration database name
`DATABASE2_USERNAME`	`drupal`	Migration database user
`DATABASE2_PASSWORD`	`drupal`	Migration database password
`DATABASE2_PORT`	`3306`	Migration database port

The DATABASE2_* defaults are the values docker-compose.yml sets for the local stack. When a variable isn't set at all, settings.migration.php falls back to localhost for the host and an empty port. It also accepts DATABASE2_DATABASE in place of DATABASE2_NAME.

Each fetch source reads its own settings under the same VORTEX_FETCH_DB2_ prefix, such as VORTEX_FETCH_DB2_FTP_HOST or VORTEX_FETCH_DB2_S3_BUCKET. The installer adds the settings for the source you pick to .env.

Docker service

By default, the database2 service runs the same uselagoon/mysql-8.4 image as the primary database. It uses the image directly, without the build step that adds the primary database's my.cnf. To run a different image, set VORTEX_DB2_IMAGE, for example to a database-in-image seed, which is an image with the migration database already in it:

database2:
  image: "${VORTEX_DB2_IMAGE:-uselagoon/mysql-8.4:26.10.0}"
  environment:
    <<: *default-environment
    MYSQL_DATABASE: drupal
    MYSQL_USER: drupal
    MYSQL_PASSWORD: drupal
  <<: *default-user
  ports:
    - '3306'
  labels:
    lagoon.type: mariadb

The image tag tracks the primary database image. See the shipped docker-compose.yml for the tag in use.

Drupal settings

Drupal reads the migration database connection from web/sites/default/settings.migration.php. settings.php loads it between the per-module overrides and the local overrides.

`settings.migration.php`

The file registers the migrate connection from the DATABASE2_* variables:

<?php

/**
 * @file
 * Migration settings.
 *
 * @todo Migration. Remove when migration configuration is no longer required.
 */

declare(strict_types=1);

// Migration DB settings.
$databases['migrate']['default'] = [
  'database' => (getenv('DATABASE2_NAME') ?: getenv('DATABASE2_DATABASE')) ?: 'drupal',
  'username' => getenv('DATABASE2_USERNAME') ?: 'drupal',
  'password' => getenv('DATABASE2_PASSWORD') ?: 'drupal',
  'host' => getenv('DATABASE2_HOST') ?: 'localhost',
  'port' => getenv('DATABASE2_PORT') ?: '',
  'prefix' => '',
  'driver' => 'mysql',
];

Source plugins built on the Migrate API's SqlBase class use key: migrate by default, which resolves to this connection.

Ahoy commands

The migration feature adds the following Ahoy commands:

Command	Description
`ahoy fetch-db2`	Fetches the migration database dump
`ahoy fetch-db2 --fresh`	Forces a fresh fetch
`ahoy reload-db2`	Recreates the `database2` container from its image, discarding its data
`ahoy db2`	Opens the migration database in Sequel Ace (macOS)

With the default image, ahoy reload-db2 brings the database back empty. The next ahoy provision finds the probe table missing and imports the dump again (see Corruption detection).

ahoy fetch-db2 runs the same vendor/bin/vortex-fetch-db script as ahoy fetch-db, with VORTEX_DB_INDEX=2. With the index set, the fetch script and its source scripts read indexed variable names, such as VORTEX_FETCH_DB2_SOURCE in place of VORTEX_FETCH_DB_SOURCE and VORTEX_DB2_IMAGE in place of VORTEX_DB_IMAGE.

Every fetch source works for the migration database too: URL, FTP, Acquia, Lagoon, container registry and S3. The shared import, export and provision scripts ignore the index, so scripts/provision-20-migration.sh imports the dump itself.

Provision script

scripts/provision-20-migration.sh runs during every provisioning run that reaches the custom scripts, including ahoy provision. Custom provision scripts run in alphabetical order, so it comes after the demo and development modules scripts and before search indexing. In the prod environment, the script skips the migrations.

The script does the following, in order:

Probes the source database for a known table, unless the import is already due (see Corruption detection)
Imports the dump into the database2 container, when the import is due or the probe failed
Verifies the source database with the same probe
Installs the ys_migrate module
Disables the Solr search server, on projects with the Solr service
Rolls back all migrations, when you turn the rollback on
Runs the migrations with drush migrate:import
Enables the Solr search server again
Prints drush migrate:status

The script needs the dump whenever it imports the source database. Each environment gets the dump in its own way:

Locally, ahoy fetch-db2 downloads it.
In CI, the database job fetches it. See CI integration.
On Lagoon, .lagoon.yml fetches it before the provision step.
In any other environment, provide the dump yourself or set DRUPAL_MIGRATION_SKIP=1.

On projects with the Solr service, the script disables the search server before the migrations and enables it again afterward, so Search API indexes nothing while the entities arrive. The scripts/provision-30-search-index.sh script rebuilds the index right after.

To leave the search server running and index entities as they're migrated, set DRUPAL_MIGRATION_SEARCH_DISABLE=0.

Configuration variables

The script reads the following variables. Set them in .env, in .env.local on your machine or in your hosting provider's environment variables: vortex-provision loads .env and .env.local and passes the values on to the custom scripts.

Variable	Default	Description
`DRUPAL_MIGRATION_SKIP`	`0`	Skip all migration operations
`DRUPAL_MIGRATION_ROLLBACK_SKIP`	`1`	Skip rollback before import
`DRUPAL_MIGRATION_IMPORT_LIMIT`	`50`	Limit entities per migration (use `all` for unlimited)
`DRUPAL_MIGRATION_UPDATE`	`0`	Update already imported entities
`DRUPAL_MIGRATION_FEEDBACK`	`50`	Progress feedback frequency
`DRUPAL_MIGRATION_SOURCE_DB_IMPORT`	`$VORTEX_PROVISION_OVERRIDE_DB`	Import source database (`1` to import, `0` to skip)
`DRUPAL_MIGRATION_SOURCE_DB_PROBE_TABLE`	`categories`	Table name to probe for corruption detection
`DRUPAL_MIGRATION_SEARCH_DISABLE`	`1`	Disable the Solr search server while migrations run (`0` to keep indexing)

vortex-provision sets VORTEX_PROVISION_OVERRIDE_DB=1 whenever the primary database is fresh, so by default the migration script imports the source database too. The script reads the dump from VORTEX_DB_DIR (default ./.data) and VORTEX_FETCH_DB2_FILE (default db2.sql), where ahoy fetch-db2 saves the dump.

Add migrations

Each migration is a YAML definition in a module's migrations directory. The demo definition is web/modules/custom/ys_migrate/migrations/ys_migrate_categories.yml. Drupal finds the definitions of enabled modules only. The script's Enabling migration modules step installs ys_migrate alone. Put your definitions in that module, or add a drush pm:install line for your own module to that step.

To run a migration, call the run_migration helper in the script's Add your migrations below. section, next to the demo run_migration ys_migrate_categories line:

run_migration my_migration

The helper passes DRUPAL_MIGRATION_IMPORT_LIMIT as --limit and DRUPAL_MIGRATION_FEEDBACK as --feedback. DRUPAL_MIGRATION_UPDATE=1 adds --update.

The helper resets the migration's status first, so a migration left stuck by an interrupted run starts cleanly. It passes any extra arguments directly to drush migrate:import. When an import fails, it prints the migration's messages and stops the provisioning.

Corruption detection

If DRUPAL_MIGRATION_SOURCE_DB_IMPORT isn't 1, the script probes the source database for the table specified in DRUPAL_MIGRATION_SOURCE_DB_PROBE_TABLE. If the table is missing (corrupted or empty database), the script automatically re-imports the dump file.

Before any migration runs, the script probes again, with or without an import. If the table is still missing, the script lists the tables it found and fails with the Migration source database is corrupted. message. A missing dump file also stops the script, with a reminder to run ahoy fetch-db2.

The default probe table, categories, belongs to the demo source database. For a real source, set DRUPAL_MIGRATION_SOURCE_DB_PROBE_TABLE to a table in that database, or the script fails at this check and stops the provisioning.

Demo migration module

Vortex ships with a demo migration module ys_migrate in web/modules/custom/ys_migrate/. It shows the full migration workflow by migrating categories from the source database into Drupal's tags taxonomy vocabulary.

Migration: `ys_migrate_categories`

The following table describes the demo migration:

Property	Value
Source plugin	`table` from Migrate Plus (reads the `categories` table, keyed by `id`)
Source key	`migrate` (the `$databases['migrate']` connection)
Destination	`entity:taxonomy_term`
Vocabulary	`tags`
Migration tag	`ys_migrate`

The migration maps the following source fields:

name → term name
description → term description (plain text format)

Dependencies

The module depends on core's Migrate module and on drupal/migrate_plus and drupal/migrate_tools, which the installer keeps in composer.json when you enable the migration feature. Migrate Plus provides the table source plugin. Migrate Tools provides the drush migrate:* commands the provision script runs.

CI integration

GitHub Actions and CircleCI both include a migration database fetch step in the database job. The 2 dump files share the .data directory and its cache, so the migration dump follows the same daily cache as the primary dump.

Before provisioning, the build job copies the migration database file into the CLI container, next to the primary database.

➡️ See Provision for how the custom provision scripts run, and Contributed modules for the shipped migration modules.

Enable the migration database​

With the installer​

Manual setup​

Example workflow​

Environment variables​

Docker service​

Drupal settings​

settings.migration.php​

Ahoy commands​

Provision script​

Configuration variables​

Add migrations​

Corruption detection​

Demo migration module​

Migration: ys_migrate_categories​

Dependencies​

CI integration​