Bump Pydantic to version 2

.github/workflows/pr_checks.yml

@@ -93,9 +93,9 @@ jobs:
   Pytest-Matrix:
     strategy:
       matrix:
-        python_version: ["3.8", "3.9", "3.10", "3.11"]
-        pydantic_version: ["1.8.2", "1.9.2", "1.10.9"]
-        pyyaml_version: ["5.4.1", "6.0"]
+        python_version: ["3.9", "3.10", "3.11"]
+        pydantic_version: ["2.2.1", "2.3.0", "2.4.2", "2.5.0"]
+        pyyaml_version: ["5.4.1", "6.0.1"]
     runs-on: ubuntu-latest
     continue-on-error: true
     steps:

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.8-slim-bullseye as base
+FROM python:3.9-slim-bullseye as base
 
 # Update pip in the base image since we'll use it everywhere
 RUN pip install -U pip

README.md

@@ -48,19 +48,6 @@ Examples of a Data Subject are:
 
 Learn more about [Data Subject Categories in the taxonomy reference now](https://ethyca.github.io/fideslang/taxonomy/data_subjects/).
 
-### 4. Data Identification Qualifiers
-
-Data Identification Qualifiers describe the degree of identification of the given data.
-Think of this as a spectrum: on one end is completely anonymous data, i.e. it is impossible to identify an individual from it; on the other end is data that specifically identifies an individual.
-
-Along this spectrum are labels that describe the degree of identification that a given data might provide, such as:
-
-- `identified_data`
-- `anonymized_data`
-- `aggregated_data`
-
-Learn more about [Data Identification Qualifiers in the taxonomy reference now](https://ethyca.github.io/fideslang/taxonomy/data_qualifiers/).
-
 ### Extensibility & Interoperability
 
 The taxonomy is designed to support common privacy compliance regulations and standards out of the box, these include GDPR, CCPA, LGPD and ISO 19944.

demo_resources/demo_dataset.yml

@@ -5,8 +5,6 @@ dataset:
   description: Data collected about users for our analytics system.
   meta: null
   data_categories: []
-  data_qualifiers:
-  - aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
   retention: "30 days after account deletion"
   third_country_transfers:
     - GBR
@@ -15,37 +13,29 @@ dataset:
   - name: users
     description: User information
     data_categories: []
-    data_qualifiers:
-    - aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
     fields:
     - name: created_at
       description: User's creation timestamp
       data_categories:
       - system.operations
-      data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
     - name: email
       description: User's Email
       data_categories:
       - user.contact.email
-      data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
       retention: Account termination
     - name: first_name
       description: User's first name
       data_categories:
       - user.name
-      data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
       retention: Account termination
     - name: food_preference
       description: User's favorite food
       data_categories: []
-      data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
     - name: state
       description: User's State
       data_categories:
       - user.contact.state
-      data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
     - name: uuid
       description: User's unique ID
       data_categories:
       - user.unique_id
-      data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified

demo_resources/demo_policy.yml

@@ -17,4 +17,3 @@ policy:
           matches: ANY
           values:
             - customer
-        data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified

demo_resources/demo_system.yml

@@ -23,7 +23,6 @@ system:
         data_use: improve.system
         data_subjects:
           - customer
-        data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
         ingress:
           - demo_users_dataset
 
@@ -40,4 +39,3 @@ system:
         data_use: advertising
         data_subjects:
           - customer
-        data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified

dev-requirements.txt

@@ -2,7 +2,7 @@ black==23.3.0
 mypy==1.4.0
 nox>=2023
 packaging>=22.0
-pre-commit==2.9.3
+pre-commit==3.5.0
 pylint==2.10.0
 pytest==7.3.1
 pytest-cov==2.11.1

mkdocs/docs/explorer.md

@@ -9,7 +9,6 @@ The taxonomy explorer is a useful way to visualize and review the taxonomy for t
         <button class="btn is-selected" data-chart-data="categories">Data Categories</button>
         <button class="btn" data-chart-data="uses">Data Uses</button>
         <button class="btn" data-chart-data="subjects">Data Subjects</button>
-        <button class="btn" data-chart-data="qualifiers">Data Qualifiers</button>
       </div>
     </div>
     <div id="chart-type-control" class="control-group">
@@ -35,6 +34,3 @@ The taxonomy explorer is a useful way to visualize and review the taxonomy for t
 </div>
 <script src="https://d3js.org/d3.v7.min.js"></script>
 <script src="../js/vis2.js"></script>
-
-
-

mkdocs/docs/index.md

@@ -17,7 +17,6 @@ The Fides taxonomy, or categorization, is made up of four main classification gr
         <button class="btn is-selected" data-chart-data="categories">Data Categories</button>
         <button class="btn" data-chart-data="uses">Data Uses</button>
         <button class="btn" data-chart-data="subjects">Data Subjects</button>
-        <button class="btn" data-chart-data="qualifiers">Data Qualifiers</button>
       </div>
     </div>
     <div id="chart-type-control" class="control-group">
@@ -59,6 +58,7 @@ Data Uses are also hierarchical with natural inheritance, meaning you can easily
 Learn more about [Data Uses in the taxonomy reference now](taxonomy/data_uses.md).
 
 ### 3. Data Subjects
+
 Data Subjects is a label commonly used in the regulatory world to describe the users of a system who's data is being processed. In many systems a generic user label may be sufficient, however the taxonomy is intended to provide greater control through specificity where needed.
 
 Examples of this are:
@@ -71,23 +71,10 @@ Examples of this are:
 
 Learn more about [Data Subjects in the taxonomy reference now](taxonomy/data_subjects.md).
 
-### 4. Data Qualifiers
-Data Qualifiers describe the degree of identification of the given data. Think of this as a spectrum: on one end is completely anonymous data, i.e. it is impossible to identify an individual from it, and on the other end is data that specifically identifies an individual. 
-
-Along this spectrum are labels that describe the degree of identification that a given data might provide, such as:
-
-- `identified`
-- `anonymized`
-- `aggregated`
-
-Learn more about [Data Qualifiers in the taxonomy reference now](taxonomy/data_qualifiers.md).
-
 ### Extensibility and Interoperability
+
 The taxonomy is designed to support common privacy compliance regulations `and standards out of the box, these include GDPR, CCPA, LGPD and ISO 19944. 
 
 You can extend the taxonomy to support your system needs. If you do this, we recommend extending from the existing class structures to ensure interoperability inside and outside your organization.
 
 If you have suggestions for missing classifications or concepts, please submit them for addition.
-
-
-

mkdocs/docs/js/vis.js

@@ -733,8 +733,7 @@ Promise.all([
   d3.csv("csv/data_categories.csv"),
   d3.csv("csv/data_uses.csv"),
   d3.csv("csv/data_subjects.csv"),
-  d3.csv("csv/data_qualifiers.csv"),
-]).then(([categoriesCSV, usesCSV, subjectsCSV, qualifiersCSV]) => {
+]).then(([categoriesCSV, usesCSV, subjectsCSV]) => {
   const tooltip = new VisTooltip();
 
   const colors = {
@@ -820,24 +819,6 @@ Promise.all([
         "#f73ffc",
         "#fb409e",
       ]),
-    qualifiers: d3
-      .scaleOrdinal()
-      .domain([
-        "data_qualifier",
-        "aggregated",
-        "aggregated.anonymized",
-        "aggregated.anonymized.unlinked_pseudonymized",
-        "aggregated.anonymized.unlinked_pseudonymized.pseudonymized",
-        "aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified",
-      ])
-      .range([
-        "#2a3045",
-        "#0861ce",
-        "#8459cc",
-        "#c14cbb",
-        "#ed43a0",
-        "#ff4a7f",
-      ]),
   };
 
   const accessor = {
@@ -859,13 +840,11 @@ Promise.all([
   const categoriesRoot = stratify(categoriesCSV);
   const usesRoot = stratify(usesCSV);
   const subjectsRoot = stratify(subjectsCSV);
-  const qualifiersRoot = stratify(qualifiersCSV);
 
   const chartData = {
     categories: categoriesRoot,
     uses: usesRoot,
     subjects: subjectsRoot,
-    qualifiers: qualifiersRoot,
   };
   const chartDataButtons = d3
     .select("#data-control")

mkdocs/docs/js/vis2-absolute.js

@@ -733,8 +733,7 @@ Promise.all([
   d3.csv("https://ethyca.github.io/fideslang/csv/data_categories.csv"),
   d3.csv("https://ethyca.github.io/fideslang/csv/data_uses.csv"),
   d3.csv("https://ethyca.github.io/fideslang/csv/data_subjects.csv"),
-  d3.csv("https://ethyca.github.io/fideslang/csv/data_qualifiers.csv"),
-]).then(([categoriesCSV, usesCSV, subjectsCSV, qualifiersCSV]) => {
+]).then(([categoriesCSV, usesCSV, subjectsCSV]) => {
   const tooltip = new VisTooltip();
 
   const colors = {
@@ -820,24 +819,6 @@ Promise.all([
         "#f73ffc",
         "#fb409e",
       ]),
-    qualifiers: d3
-      .scaleOrdinal()
-      .domain([
-        "data_qualifier",
-        "aggregated",
-        "aggregated.anonymized",
-        "aggregated.anonymized.unlinked_pseudonymized",
-        "aggregated.anonymized.unlinked_pseudonymized.pseudonymized",
-        "aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified",
-      ])
-      .range([
-        "#2a3045",
-        "#0861ce",
-        "#8459cc",
-        "#c14cbb",
-        "#ed43a0",
-        "#ff4a7f",
-      ]),
   };
 
   const accessor = {
@@ -859,13 +840,11 @@ Promise.all([
   const categoriesRoot = stratify(categoriesCSV);
   const usesRoot = stratify(usesCSV);
   const subjectsRoot = stratify(subjectsCSV);
-  const qualifiersRoot = stratify(qualifiersCSV);
 
   const chartData = {
     categories: categoriesRoot,
     uses: usesRoot,
     subjects: subjectsRoot,
-    qualifiers: qualifiersRoot,
   };
   const chartDataButtons = d3
     .select("#data-control")

mkdocs/docs/js/vis2.js

@@ -733,8 +733,7 @@ Promise.all([
   d3.csv("../csv/data_categories.csv"),
   d3.csv("../csv/data_uses.csv"),
   d3.csv("../csv/data_subjects.csv"),
-  d3.csv("../csv/data_qualifiers.csv"),
-]).then(([categoriesCSV, usesCSV, subjectsCSV, qualifiersCSV]) => {
+]).then(([categoriesCSV, usesCSV, subjectsCSV]) => {
   const tooltip = new VisTooltip();
 
   const colors = {
@@ -820,24 +819,6 @@ Promise.all([
         "#f73ffc",
         "#fb409e",
       ]),
-    qualifiers: d3
-      .scaleOrdinal()
-      .domain([
-        "data_qualifier",
-        "aggregated",
-        "aggregated.anonymized",
-        "aggregated.anonymized.unlinked_pseudonymized",
-        "aggregated.anonymized.unlinked_pseudonymized.pseudonymized",
-        "aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified",
-      ])
-      .range([
-        "#2a3045",
-        "#0861ce",
-        "#8459cc",
-        "#c14cbb",
-        "#ed43a0",
-        "#ff4a7f",
-      ]),
   };
 
   const accessor = {
@@ -859,13 +840,11 @@ Promise.all([
   const categoriesRoot = stratify(categoriesCSV);
   const usesRoot = stratify(usesCSV);
   const subjectsRoot = stratify(subjectsCSV);
-  const qualifiersRoot = stratify(qualifiersCSV);
 
   const chartData = {
     categories: categoriesRoot,
     uses: usesRoot,
     subjects: subjectsRoot,
-    qualifiers: qualifiersRoot,
   };
   const chartDataButtons = d3
     .select("#data-control")

mkdocs/docs/resources/dataset.md

@@ -56,7 +56,6 @@ An optional array of contact information if a Joint Controller exists. This info
 An optional string to describe the retention policy for a dataset. This field can also be applied more granularly at either the Collection or field level of a Dataset
 
 **data_categories**&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[_string_]<br/>
-**data_qualifiers**&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[_string_]<br/>
 
 Arrays of Data Category and Data Qualifier resources, identified by `fides_key`, that apply to all collections in the Dataset.
 
@@ -73,7 +72,6 @@ A UI-friendly label for the collection.
 A human-readable description of the collection.
 
 **collections.data_categories**&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[_string_]<br/>
-**collections.data_qualifiers**&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[_string_]<br/>
 
 Arrays of Data Category and Data Qualifier resources, identified by `fides_key`, that apply to all fields in the collection.
 
@@ -97,10 +95,6 @@ A human-readable description of the field.
 
 Arrays of Data Categories, identified by `fides_key`, that applies to this field.
 
-**collections.fields.data_qualifier**&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;_string_<br/>
-
-A Data Qualifier that applies to this field. Note that this field holds a single value, therefore, the property name is singular.
-
 **collections.fields.retention**<span class="required"/>&nbsp;&nbsp;_string_
 
 An optional string to describe the retention policy for a field within a Dataset collection.

mkdocs/docs/resources/policy.md

@@ -1,6 +1,6 @@
 # Policy
 
-A Policy is your privacy policy as code, it lists a set of acceptable and non-acceptable rules and uses all 4 privacy attributes  (`data_category`, `data_use`, `data_subject`, and `data_qualifier`). The purpose of the policy is to state what types of data are allowed for certain usages.
+A Policy is your privacy policy as code, it lists a set of acceptable and non-acceptable rules and uses all 3 privacy attributes  (`data_category`, `data_use`, `data_subject`). The purpose of the policy is to state what types of data are allowed for certain usages.
 
   ```
   organization
@@ -34,10 +34,6 @@ The [Data Use](../../taxonomy/data_uses/) privacy attribute describes the variou
 
 The [Data Subjects](../../taxonomy/data_subjects/) privacy attribute describes the individual persons whose data your rule pertains to.
 
-**data_qualifier**     _string_     
-
-The [Data Qualifier](../../taxonomy/data_qualifiers/) privacy attribute describes the acceptable or non-acceptable level of deidentification for this data.
-
 **matches**     _enum_     
 
 * `ANY`
@@ -76,7 +72,6 @@ policy:
           matches: ANY
           values:
             - customer
-        data_qualifier: aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified
 ```
 
 **Demo manifest file:** `/fides/demo_resources/demo_policy.yml`
@@ -113,7 +108,6 @@ policy:
           "customer"
         ]
       },
-      "data_qualifier": "aggregated.anonymized.unlinked_pseudonymized.pseudonymized.identified"
     }
   ]
 }