[Feature]: Storage State Encryption

[Grant-Archibald-MS]

Is your feature request related to a problem? Please describe.

How can the state.json which can contain browser cookies, be encrypted, to ensure that:

• Information cannot be disclosed
• Encrypted files be shared easily across authorized users or application users involved in the CI/CD process
• Support key rotation
• Support auditing

Describe the solution you'd like

To address this problem, we could consider a combination of AES (Advanced Encryption Standard) for encrypting the state.json content and RSA (Rivest-Shamir-Adleman) for encrypting the AES key. This goal to ensure robust security and allows for secure sharing of encrypted files across authorized users or application users.

Provide the ability to use Windows Data Protection (DPAPI) for Windows only local machine protection

Certificate Public/Private key encryption combined with AES

Summary

• Enhance current storage state user authentication provider is extended to Save and load the Playwright browser context as Encrypted values
• Make use of the Microsoft.AspNetCore.DataProtection package to offer Windows Data Protection (DAPI) or Certificate public / private encryption
• Use the current logged in Azure CLI session to obtain an access token to the Dataverse instance where Test Engine Key values and encrypted key data is stored
• Use a C# class that provide implementation to query and create Data Protection state by implementing IXmlRepository
• Store XML state of data protection in Dataverse Table. Encryption of XML State managed by Data Protection API and selected protection providers
• Store encrypted base 64 string version of state in Dataverse encrypted by Data Protection API
• Dataverse Security model, sharing and auditing features are enabled to control access and record access to key and key data
• Data Protection API is used to decrypt values and apply the state json to other test sessions.

Primer on AES and RSA Encryption

AES (Advanced Encryption Standard):

Type: Symmetric encryption algorithm.
Key Concept: Uses the same key for both encryption and decryption.
Strengths: Fast and efficient, suitable for encrypting large amounts of data.
Usage: Commonly used for securing data at rest and in transit.
How It Protects: Encrypts data into an unreadable format that can only be decrypted with the same key, ensuring data confidentiality.

RSA (Rivest-Shamir-Adleman):

Type: Asymmetric encryption algorithm.
Key Concept: Uses a pair of keys – a public key for encryption and a private key for decryption.
Strengths: Secure key exchange, suitable for encrypting small amounts of data like encryption keys.
Usage: Often used for securing data transmission and digital signatures.
How It Protects: Encrypts data with a public key that can only be decrypted with the corresponding private key, ensuring secure key exchange and data integrity.

Possible Approach

1. Encrypting state.json Content with Windows Data protection API (DPAPI)

2. Encrypt data with Microsoft ASPNET Data Protection Packages that use private key and AES encryption

   • AES Encryption: Use AES to encrypt the state.json file. AES is a symmetric encryption algorithm that is fast and efficient for encrypting large amounts of data.
   • Key Management: Generate a unique AES key for encryption.

3. Storing and Managing Keys in Dataverse:

   • Dataverse Tables: Create tables in Dataverse to store the encrypted AES keys.
   • Store AES Encrypted key value in dataverse with the public key help outside dataverse
   • Support Windows Data Protaction API (DAPI) or Certificate based encryption
   • Access Control: Use Dataverse's role-based security to control access to the keys. Only authorized users or application users should have access to the RSA private key.

4. Key Rotation and Audit Logging:

   • Key Rotation: Implement policies to periodically update encryption keys and re-encrypt the state.json content.
   • Audit Logging: Track key access and operations in an AuditLogs table to monitor and detect any unauthorized attempts.

Sharing Encrypted Files

Authorized Users: Ensure that users involved in the development and CI/CD process have the necessary roles to access the encrypted AES keys and RSA private key.
Application Users: Configure application users with appropriate permissions to access and decrypt the state.json content during automated processes.

Technical Approach

Possible technical approach with Audit Enabled tables using direct Encryption libraries or ASP.NET Core Data Protection described below

Dataverse Table Schema

1. Test Engine Keys Table:
   Columns:

   • KeyId (Primary Key)
   • Name (String)
   • Xml (IXmlRepository)

2. Test Engine Key Data Table:
   Columns:

   • DataId (Primary Key)
   • KeyName (String)
   • ValueName (String)
   • Data (String, Base64 encoded encrypted)

Dataverse Record-Level Security and Column-Level Security

Record-Level Security
Dataverse allows you to control access to individual records based on user roles and permissions. This ensures that only authorized users can access specific records.

Key Concepts:
- Security Roles: Define a set of privileges that determine what actions users can perform on different types of records.
- Business Units: Segment data and users into business units, providing additional layers of security.
- Record Ownership: Records can be owned by users or teams, and the owner has full control over the record.
- Access Levels: Privileges can be set at different levels (User, Business Unit, Parent: Child Business Unit, Organization).

Describe alternatives you've considered

Alternative Solution: Windows Data Protection API (DAPI)

As Microsoft Learn How to: Use Data Protection states

.NET provides access to the data protection API (DPAPI), which allows you to encrypt data using information from the current user account or computer. When you use the DPAPI, you alleviate the difficult problem of explicitly generating and storing a cryptographic key.

This implementation which would be Microsoft Windows centric could provide default level of protection to state.json files. Given the encryption is tied to the user account and machine the state.json file would only be able to be decrypted on the machine.

Alternative Solution: ASP.NET Core Data Protection

ASP.NET Core security topics gives and overview of multi machine method to encrypt sensitive information.

How It Works

• Data Protection API: The API provides methods to encrypt (protect) and decrypt (unprotect) data.
• Key Management: Keys are automatically managed, including generation, storage, and rotation.
• Configuration: The system can be configured to store keys in Dataverse

This could be combined with Dataverse based solution of a console application to read and save values for a set of secure keys. Key names would need to be unique to specific set of tests for a user context.

using Microsoft.AspNetCore.DataProtection;
using Microsoft.AspNetCore.DataProtection.Repositories;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.PowerPlatform.Dataverse.Client;
using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Query;
using System.Security.Cryptography.X509Certificates;
using System.Xml.Linq;

public class Program { 
    public static void Main(string[] args)
    {
        ServiceProvider services = null;

        var serviceCollection = new ServiceCollection();
        serviceCollection.AddLogging(configure => configure.AddConsole());

        var api = new Uri("https://contoso.crm.dynamics.com/");

        string keyName = "Sample";

        // Configure Dataverse connection
        var serviceClient = new ServiceClient(api, (url) => Task.FromResult(AzureCliHelper.GetAccessToken(api)));

        serviceCollection.AddSingleton<IOrganizationService>(serviceClient);

        serviceCollection.AddDataProtection()
            .ProtectKeysWithCertificate(GetCertificateFromStore("localhost"))
            //.ProtectKeysWithDpapi()
            .AddKeyManagementOptions(options =>
            {
                options.XmlRepository = new DataverseKeyStore(services?.GetRequiredService<ILogger<Program>>(), serviceClient, keyName);
            });
        services = serviceCollection.BuildServiceProvider();
        var protector = services.GetDataProtector("ASP Data Protection");


        string valueName = string.Empty;
        while ( string.IsNullOrEmpty(valueName))
        {
            Console.WriteLine("Variable Name");
            valueName = Console.ReadLine();
        }
        
        var matches = FindMatch(serviceClient, keyName, valueName);

        if (matches.Count() == 0)
        {
            string newValue = string.Empty;
            while ( string.IsNullOrEmpty(newValue))
            {
                Console.WriteLine("Value does not exist. What would you like the value to be?");
                newValue = Console.ReadLine();
            }
            
            StoreValue(serviceClient, keyName, valueName, protector.Protect(newValue));

            Console.WriteLine($"Saved value for {valueName}");
        } 
        else
        {
            string data = protector.Unprotect(matches.First().Data);
            Console.WriteLine($"Value {valueName}: {data}");
        }

        Console.ReadLine();
    }
}

Possible code to query and store encrypted values

    public static IReadOnlyCollection<ProtectedKeyValue> FindMatch(IOrganizationService service, string keyName, string? valueName)
    {
        // Retrieve keys from Dataverse
        FilterExpression filter = new FilterExpression(LogicalOperator.And);
        filter.Conditions.Add(new ConditionExpression("te_keyname", ConditionOperator.Equal, keyName));
        filter.Conditions.Add(new ConditionExpression("te_valuename", ConditionOperator.Equal, valueName));

        var query = new QueryExpression("te_keydata")
        {
            ColumnSet = new ColumnSet("te_keyname", "te_valuename", "te_data"),
            Criteria = filter
        };

        var keys = service.RetrieveMultiple(query)
        .Entities
        .Select(e => new ProtectedKeyValue {
            KeyId = e.Id.ToString(),
            KeyName = e["te_keyname"]?.ToString(),
            ValueName = e["te_valuename"]?.ToString(),
            Data = e["te_data"]?.ToString(),
        })
        .ToList();

        return keys.AsReadOnly();
    }

    public static void StoreValue(IOrganizationService service, string keyName, string valueName, string data)
    {
        var keyEntity = new Entity("te_keydata")
        {
            ["te_keyname"] = keyName,
            ["te_valuename"] =  valueName,
            ["te_data"] = data,
        };

        service.Create(keyEntity);
    }

Optionally query certificate from Windows Store

    static X509Certificate2 GetCertificateFromStore(string friendlyName)
    {
        using (var store = new X509Store(StoreLocation.CurrentUser))
        {
            store.Open(OpenFlags.ReadOnly);
            var certs = store.Certificates.Find(X509FindType.FindBySubjectName, friendlyName, false);
            if (certs.Count == 0)
            {
                throw new Exception("Certificate not found");
            }
            return certs.First();
        }
    }

Possible code to query and store encryption key values encrypted via DAPI or public key of certificate

public class DataverseKeyStore : IXmlRepository
{
    private readonly ILogger<Program>? _logger;
    private readonly IOrganizationService _service;
    private string _friendlyName;

    public DataverseKeyStore(ILogger<Program>? logger, IOrganizationService organizationService, string friendlyName)
    {
        _logger = logger;
        _service = organizationService;
        _friendlyName = friendlyName;
    }

    public IReadOnlyCollection<XElement> GetAllElements()
    {
        // Retrieve keys from Dataverse
        var query = new QueryExpression("te_key")
        {
            ColumnSet = new ColumnSet("te_xml"),
            Criteria = new FilterExpression
            {
                Conditions =
                {
                    new ConditionExpression("te_name", ConditionOperator.Equal, _friendlyName)
                }
            }
        };

        var keys = _service.RetrieveMultiple(query)
        .Entities
        .Select(e => XElement.Parse(e.GetAttributeValue<string>("te_xml")))
        .ToList();

        return keys.AsReadOnly();
    }

    public void StoreElement(XElement element, string friendlyName)
    {
        var keyEntity = new Entity("te_key")
        {
            ["te_name"] = _friendlyName,
            ["te_xml"] = element.ToString(SaveOptions.DisableFormatting)
        };

        _service.Create(keyEntity);
    }
}

Class to query encrypted value from Dataverse

public class ProtectedKeyValue
{
    public string KeyId { get; set; }
    public string KeyName { get; set; }
    public string ValueName { get; set; }
    public string Data { get; set; }
}

Alternative Solution: Managed Information Protection (MIP)

Overview: Managed Information Protection (MIP) is a comprehensive data protection solution integrated with Microsoft 365. It provides advanced capabilities for data classification, labeling, and protection across various Microsoft 365 services.

Key Features

• Data Classification and Labeling: MIP allows you to classify and label data based on its sensitivity. Labels can be applied manually by users or automatically based on predefined policies.
• Encryption: MIP uses both symmetric (AES) and asymmetric (RSA) encryption to protect data. This ensures that sensitive information is encrypted and can only be accessed by authorized users.
• Policy Enforcement: MIP enables the creation and enforcement of data protection policies, ensuring consistent application of security controls across the organization.
• Integration with Microsoft 365: MIP is deeply integrated with Microsoft 365 services such as SharePoint, Teams, and Exchange, providing seamless protection for data across different platforms.

Pros

Comprehensive Protection: MIP offers a unified solution for data protection, covering classification, labeling, and encryption.
User-Friendly: The labeling and classification features are intuitive, making it easy for users to apply protection to documents and emails.
Policy Management: Centralized management of data protection policies ensures consistent enforcement across the organization.

Notes

• Microsoft 365 Dependency: MIP requires a Microsoft 365 subscription with the appropriate SKU that includes MIP capabilities. This may not be available to all Power Platform users.
• Additional licensing costs may be involved, especially for advanced features.

Additional context?

No response

[Grant-Archibald-MS]

Key elements:

1. No secrets are stored on the local file system
2. Encrypted values are stored inside Dataverse
3. Dataverse security rules applied to Dataverse tables
4. Public Key values are stored in Dataverse and can be shared across users or Application user accounts
5. CI/CD account using Managed Service Identity can be registered as Application Users. Key record can be shared with Application user accounts
6. Private key values are managed via DAPI for the user and machine or private key of certificate

[Grant-Archibald-MS]

Refined choices for local machine encryption with Windows Data Protection API (DPAPI) and ASPNET Data Protection.

Key differences:

1. DPAPI is limited to windows only. The protected file can only be accessed on local machine using user account
2. Machine use of ASPNet Data Protection API and Microsoft Dataverse. Using this approach encrypted storage state can be access across machines.
   • Private Key using an X.509 certificate can be used to encrypt the symmetric AES Key
   • Makes use of out of the box Key rotation, tampering checks that built into ASPNET Data Protection
   • Takes advantage of Microsoft Dataverse Security Model for Access Token to the Microsoft Dataverse API with row level and role based security. Data tables with audit tracking

[Grant-Archibald-MS]

Added Pull request #500 to for local encryption using Current User.

Will create second PR for dataverse based encryption that will allow saved state to be used across machines