Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update documentation for Getting Started with data migration #9030

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Update documentation for Getting Started with data migration #9030

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ef2018f
Update documentation for Getting Started with data migration
jugal-chauhan Jan 7, 2025
6bcc966
Merge branch 'main' into update-getting-started-page
Naarcha-AWS Jan 14, 2025
6a33ece
Update _migration-assistant/deploying-migration-assistant/getting-sta…
Naarcha-AWS Jan 14, 2025
5640ecc
Update _migration-assistant/deploying-migration-assistant/getting-sta…
Naarcha-AWS Jan 14, 2025
9e750b4
Update _migration-assistant/deploying-migration-assistant/getting-sta…
Naarcha-AWS Jan 14, 2025
751c270
Move list of deployed stacks to beginning of Step 5.
Naarcha-AWS Jan 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 46 additions & 43 deletions ...ation-assistant/deploying-migration-assistant/getting-started-data-migration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
* Verify that your migration path [is supported]({{site.url}}{{site.baseurl}}/migration-assistant/is-migration-assistant-right-for-you/#migration-paths). Note that we test with the exact versions specified, but you should be able to migrate data on alternative minor versions as long as the major version is supported.
* The source cluster must be deployed Amazon Simple Storage Service (Amazon S3) plugin.
* The target cluster must be deployed.
* Verify that the `CDKToolkit` stack exists and is set to `CREATE_COMPLETE`. For more information about on how to bootstrap your AWS account in the required AWS Region, see [the CDKTooklit documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html).

Check failure on line 23 in _migration-assistant/deploying-migration-assistant/getting-started-data-migration.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: CDKTooklit. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: CDKTooklit. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_migration-assistant/deploying-migration-assistant/getting-started-data-migration.md", "range": {"start": {"line": 23, "column": 179}}}, "severity": "ERROR"}

The steps in this guide assume the following:

Expand Down Expand Up @@ -70,6 +71,7 @@
{% include copy.html %}

3. Name the policy, for example, `SSM-OSMigrationBootstrapAccess`, and then create the policy by selecting **Create policy**.
4. Attach the newly created policy to your EC2 instance's IAM Role.

---

Expand Down Expand Up @@ -107,14 +109,15 @@

## Step 4: Configure and deploy RFS (~20 minutes)

Use the following steps to configure and deploy RFS:
Use the following steps to configure and deploy RFS, deploy Migration Assistant and verify installation of required stacks:

1. Add the target cluster password to AWS Secrets Manager as an unstructured string. Be sure to copy the secret Amazon Resource Name (ARN) for use during deployment.
1. Add the source and target cluster password as separate **Secrets** in the [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) as an unstructured string. Be sure to copy the secret Amazon Resource Name (ARN) for use during deployment.
2. From the same shell as the Bootstrap instance, modify the `cdk.context.json` file located in the `/opensearch-migrations/deployment/cdk/opensearch-service-migration` directory:

```json
{
"migration-assistant": {
"stage": "dev",
"vpcId": "<TARGET CLUSTER VPC ID>",
"targetCluster": {
"endpoint": "<TARGET CLUSTER ENDPOINT>",
Expand All @@ -126,14 +129,14 @@
},
"sourceCluster": {
"endpoint": "<SOURCE CLUSTER ENDPOINT>",
"version": "<SOURCE ENGINE VERSION>",
"auth": {
"type": "basic",
"username": "<TARGET CLUSTER USERNAME>",
"passwordFromSecretArn": "<TARGET CLUSTER PASSWORD SECRET>"
}
},
"reindexFromSnapshotExtraArgs": "<RFS PARAMETERS (see below)>",
"stage": "dev",
"otelCollectorEnabled": true,
"migrationConsoleServiceEnabled": true,
"reindexFromSnapshotServiceEnabled": true,
Expand All @@ -152,54 +155,52 @@
```
{% include copy.html %}

4. Deploy the stacks:
4. Deploy Migration Assistant and any required stacks when `cdk.context.json` is fully configured:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Clarity] This phrasing is confusing, if the cdk.context.json needs to be configured, that should be assumed to be part of the previous step. Is there something else we are trying to convey?


```bash
cdk deploy "*" --c contextId=migration-assistant --require-approval never --concurrency 5
```
{% include copy.html %}

5. Verify that all CloudFormation stacks were installed successfully.

### RFS parameters

If you're creating a snapshot using migration tooling, these parameters are automatically configured. If you're using an existing snapshot, modify the `reindexFromSnapshotExtraArgs` setting with the following values:
These commands deploy the following stacks:
* Migration Assistant network stack
* `Reindex-from-snapshot` stack
* Migration console stack
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Writing Style] All the stacks should be either inside tic marks or not.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll fix this one.

5. Verify from the same Bootstrap instance shell that all CloudFormation stacks were installed successfully:

```bash
--s3-repo-uri s3://<bucket-name>/<repo> --s3-region <region> --snapshot-name <name>
aws cloudformation list-stacks --query "StackSummaries[?StackStatus!='DELETE_COMPLETE'].[StackName,StackStatus]" --output table
```
{% include copy.html %}

You should receive a similar output for your region:

You will also need to give the `migrationconsole` and `reindexFromSnapshot` TaskRoles permissions to the S3 bucket.

---

## Step 5: Deploy Migration Assistant

To deploy Migration Assistant, use the following steps:
```bash
------------------------------------------------------------------------
| ListStacks |
+--------------------------------------------------+-------------------+
| OSMigrations-dev-us-east-1-MigrationConsole | CREATE_COMPLETE |
| OSMigrations-dev-us-east-1-ReindexFromSnapshot | CREATE_COMPLETE |
| OSMigrations-dev-us-east-1-MigrationInfra | CREATE_COMPLETE |
| OSMigrations-dev-us-east-1-default-NetworkInfra | CREATE_COMPLETE |
| MigrationBootstrap | CREATE_COMPLETE |
| CDKToolkit | CREATE_COMPLETE |
+--------------------------------------------------+-------------------+
```

1. Bootstrap the account:

```bash
cdk bootstrap --c contextId=migration-assistant --require-approval never --concurrency 5
```
{% include copy.html %}
### RFS parameters

2. Deploy the stacks when `cdk.context.json` is fully configured:

```bash
cdk deploy "*" --c contextId=migration-assistant --require-approval never --concurrency 3
```
{% include copy.html %}
If you're creating a snapshot using migration tooling, these parameters are automatically configured. If you're using an existing snapshot, modify the `reindexFromSnapshotExtraArgs` setting with the following values:

These commands deploy the following stacks:
```bash
"reindexFromSnapshotExtraArgs": "--s3-repo-uri s3://<bucket-name>/<repo> --s3-region <region> --snapshot-name <name>"
```

* Migration Assistant network stack
* `Reindex-from-snapshot` stack
* Migration console stack
You will also need to give the `migrationconsole` and `reindexFromSnapshot` TaskRoles permissions to the S3 bucket.

---

## Step 6: Access the migration console
## Step 5: Access the migration console

Run the following command to access the migration console:

Expand All @@ -214,7 +215,7 @@

---

## Step 7: Verify the connection to the source and target clusters
## Step 6: Verify the connection to the source and target clusters

To verify the connection to the clusters, run the following command:

Expand All @@ -226,15 +227,17 @@
You should receive the following output:

```bash
* **Source Cluster:** Successfully connected!
* **Target Cluster:** Successfully connected!
SOURCE CLUSTER
ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='')
TARGET CLUSTER
ConnectionResult(connection_message='Successfully connected!', connection_established=True, cluster_version='')
```

To learn more about migration console commands, see [Migration commands].

---

## Step 8: Create a snapshot
## Step 7: Create a snapshot

Run the following command to initiate snapshot creation from the source cluster:

Expand Down Expand Up @@ -263,7 +266,7 @@

---

## Step 9: Migrate metadata
## Step 8: Migrate metadata

Run the following command to migrate metadata:

Expand All @@ -276,7 +279,7 @@

---

## Step 10: Migrate documents with RFS
## Step 9: Migrate documents with RFS

You can now use RFS to migrate documents from your original cluster:

Expand Down Expand Up @@ -312,7 +315,7 @@

---

## Step 11: Backfill monitoring
## Step 10: Backfill monitoring

Use the following command for detailed monitoring of the backfill process:

Expand All @@ -339,7 +342,7 @@

---

## Step 12: Verify that all documents were migrated
## Step 11: Verify that all documents were migrated

Use the following query in CloudWatch Logs Insights to identify failed documents:

Expand Down
Loading