Merge pull request #27 from Constellation-Labs/merging-mainnet

Merging mainnet
Constellation-Labs · Nov 30, 2023 · a39e94d · a39e94d
2 parents b2c5ed0 + aa7fed0
commit a39e94d
Show file tree

Hide file tree

Showing 14 changed files with 385 additions and 123 deletions.
diff --git a/README.md b/README.md
@@ -0,0 +1,27 @@
+DOR Metagraph
+===============
+
+Welcome to the official repository of DOR Metagraph.
+
+The DOR Metagraph operates on Constellation's global Layer 0 network, known as the Hypergraph. It serves as a decentralized ledger-based platform for the distribution, validation, and incentivization of data generated by the Dor Traffic Miner and upcoming product offerings.
+
+This technology is built using Constellation Network's Hypergraph Transfer Protocol (HGTP) and features the DOR utility token. The Dor Metagraph and its associated token have various applications, including:
+
+-   Establishing a global community of individual "Datapreneurs" dedicated to mining valuable datasets.
+-   Providing support for hardware and software tools for real-world data collection.
+-   Programmatically managing all fee requirements for Hypergraph network usage.
+
+The key users of this metagraph are Datapreneurs, responsible for collecting valuable data for the network, and node operators, who contribute resources such as bandwidth and processing power to the network.
+
+Project Structure
+-----------------
+
+This project consists of two main subdirectories:
+
+### Metagraph
+
+The Metagraph folder contains all the content of the metagraph and is constructed on top of the Data API provided by Constellation Network. For more detailed information about this metagraph, please refer to the README.md file within this folder.
+
+### API
+
+The API subdirectory houses a "mock API" designed to simulate returns from the official DOR API. This mock API serves the purpose of testing returns from the API before integrating them into the metagraph. It was developed using NodeJS with Typescript and Express.
diff --git a/api/README.md b/api/README.md
@@ -0,0 +1,20 @@
+API
+===
+
+This component is crafted to replicate responses from the official DOR API. The purpose of this mock API is to conduct thorough testing of responses from the API before their seamless integration into the metagraph.
+
+It was developed using NodeJS, leveraging Typescript and Express.
+
+Dependencies
+------------
+
+-   NodeJS 16
+-   Yarn
+
+How to Run
+----------
+
+Navigate to the API directory:
+
+-   Execute the command `yarn` to download the necessary dependencies.
+-   Launch the project with `yarn dev`.
diff --git a/metagraph/README.md b/metagraph/README.md
@@ -0,0 +1,125 @@
+Metagraph
+=========
+
+This directory encompasses all aspects of the metagraph and is built on top of the Data API provided by Constellation Network.
+
+Here, you will find the entire project structure, including validations, state building, testing, and custom endpoints.
+
+### Dependencies
+
+-   Java 11
+-   Scala 2
+
+State and CalculatedState
+===========================
+
+At Metagraphs, we distinguish between two types of states: `State` and `CalculatedState`.
+
+In the DOR Metagraph, the `State` holds all information pertaining to check-ins in the current snapshot, including:
+
+-   `publicId`
+-   `signature`
+-   `dts`
+-   `dtmCheckInHash`
+-   `maybeDorAPIResponse`
+
+On the other hand, the `CalculatedState` contains metadata information about the devices. In contrast to only storing check-ins from the current snapshot, the CalculatedState maintains a map that establishes the relationship between all devices and their metadata. This metadata includes:
+
+-   `devices: Map[Address, DeviceInfo]`
+    -   where `Address` represents the device address, and
+    -   `DeviceInfo` includes:
+        -   `lastCheckIn`
+        -   `dorAPIResponse`
+        -   `nextEpochProgressToReward`
+
+Code
+--------
+
+The main code for this example is located in the following directories:
+
+-   `modules/l0/src/main/scala/com/my/currency/l0`
+-   `modules/l1/src/main/scala/com/my/currency/l1`
+-   `modules/data_l1/src/main/scala/com/my/currency/data_l1`
+-   `modules/shared_data/src/main/scala/com/my/currency/shared_data`
+
+### Application Lifecycle
+
+The methods of the DataApplication are invoked in the following sequence:
+
+-   `validateUpdate`
+-   `validateData`
+-   `combine`
+-   `dataEncoder`
+-   `dataDecoder`
+-   `calculatedStateEncoder`
+-   `signedDataEntityDecoder`
+-   `serializeBlock`
+-   `deserializeBlock`
+-   `serializeState`
+-   `deserializeState`
+-   `serializeUpdate`
+-   `deserializeUpdate`
+-   `setCalculatedState`
+-   `getCalculatedState`
+-   `hashCalculatedState`
+-   `routes`
+
+For a more detailed understanding, please refer to the [complete documentation](https://docs.constellationnetwork.io/sdk/frameworks/currency/data-api) on the Data API.
+
+### Lifecycle Functions
+
+#### -> `validateUpdate`
+
+* This method initiates the initial validation of updates on the L1 layer. Due to a lack of contextual information (state), its validation capabilities are constrained. Any errors arising from this method result in a 500 response from the `/data` POST endpoint.
+
+#### -> `validateData`
+
+* This method validates data on the L0 layer, with access to contextual information, including the current state. In this example, we ensure that the provided address matches the one that signed the message. Additionally, we verify the most recent update timestamp to prevent the acceptance of outdated or duplicated data.
+
+#### -> `combine`
+
+* This method takes validated data and the prior state, combining them to produce the new state. In this instance, we update device information in the state based on the validated update.
+
+#### -> `dataEncoder` and `dataDecoder`
+
+* These are the encoder/decoder components used for incoming updates.
+
+#### -> `calculatedStateEncoder`
+
+* This encoder is employed for the calculatedState.
+
+#### -> `signedDataEntityDecoder`
+
+* This function handles the parsing of request body formats (JSON, string, xml) into a `Signed[Update]` class. In this case, we receive a string and parse it into `Signed[CheckInUpdate]`.
+
+#### -> `serializeBlock` and `deserializeBlock`
+
+* The serialize function accepts the block object and converts it into a byte array for storage within the snapshot. The deserialize function is responsible for deserializing into Blocks.
+
+#### -> `serializeState` and `deserializeState`
+
+* The serialize function accepts the state object and converts it into a byte array for storage within the snapshot. The deserialize function is responsible for deserializing into State.
+
+#### -> `serializeUpdate` and `deserializeUpdate`
+
+* The serialize function accepts the update object and converts it into a byte array for storage within the snapshot. The deserialize function is responsible for deserializing into Updates.
+
+#### -> `setCalculatedState`
+
+* This function sets the calculatedState. You can store this as a variable in memory or use external services such as databases. In this example, we use in-memory storage.
+
+#### -> `getCalculatedState`
+
+* This function retrieves the calculated state.
+
+#### -> `hashCalculatedState`
+
+* This function creates a hash of the calculatedState to be validated when rebuilding this state, in case of restarting the metagraph.
+
+#### -> `routes`
+
+Customizes routes for our application.
+
+In this example, the following endpoints are implemented:
+
+-   GET `/data-application/calculated-state/latest`: Returns the latest calculatedState.
diff --git a/metagraph/modules/data_l1/src/main/scala/com/my/dor_metagraph/data_l1/Main.scala b/metagraph/modules/data_l1/src/main/scala/com/my/dor_metagraph/data_l1/Main.scala
@@ -1,10 +1,10 @@
 package com.my.dor_metagraph.data_l1
 
 import cats.data.NonEmptyList
-import cats.effect.IO
-import cats.implicits.catsSyntaxValidatedIdBinCompat0
+import cats.effect.{IO, Resource}
+import cats.implicits.{catsSyntaxOptionId, catsSyntaxValidatedIdBinCompat0}
 import com.my.dor_metagraph.shared_data.Data
-import com.my.dor_metagraph.shared_data.calculated_state.CalculatedState
+import com.my.dor_metagraph.shared_data.calculated_state.CalculatedStateService
 import com.my.dor_metagraph.shared_data.decoders.Decoders
 import com.my.dor_metagraph.shared_data.deserializers.Deserializers
 import com.my.dor_metagraph.shared_data.serializers.Serializers
@@ -18,6 +18,7 @@ import org.tessellation.currency.l1.CurrencyL1App
 import org.tessellation.schema.cluster.ClusterId
 import org.tessellation.security.signature.Signed
 import org.http4s.EntityDecoder
+import org.tessellation.ext.cats.effect.ResourceIO
 import org.tessellation.schema.SnapshotOrdinal
 import org.tessellation.security.hash.Hash
 
@@ -30,42 +31,60 @@ object Main
     ClusterId(UUID.fromString("517c3a05-9219-471b-a54c-21b7d72f4ae5")),
     version = BuildInfo.version
   ) {
-  override def dataApplication: Option[BaseDataApplicationL1Service[IO]] = Option(BaseDataApplicationL1Service(new DataApplicationL1Service[IO, CheckInUpdate, CheckInStateOnChain, CheckInDataCalculatedState] {
+  private def makeBaseDataApplicationL1Service(
+    calculatedStateService: CalculatedStateService[IO]
+  ): BaseDataApplicationL1Service[IO] =
+    BaseDataApplicationL1Service(new DataApplicationL1Service[IO, CheckInUpdate, CheckInStateOnChain, CheckInDataCalculatedState] {
 
-    override def validateData(state: DataState[CheckInStateOnChain, CheckInDataCalculatedState], updates: NonEmptyList[Signed[CheckInUpdate]])(implicit context: L1NodeContext[IO]): IO[DataApplicationValidationErrorOr[Unit]] = IO.pure(().validNec)
+      override def validateData(state: DataState[CheckInStateOnChain, CheckInDataCalculatedState], updates: NonEmptyList[Signed[CheckInUpdate]])(implicit context: L1NodeContext[IO]): IO[DataApplicationValidationErrorOr[Unit]] = IO.pure(().validNec)
 
-    override def validateUpdate(update: CheckInUpdate)(implicit context: L1NodeContext[IO]): IO[DataApplicationValidationErrorOr[Unit]] = Data.validateUpdate(update)
+      override def validateUpdate(update: CheckInUpdate)(implicit context: L1NodeContext[IO]): IO[DataApplicationValidationErrorOr[Unit]] = Data.validateUpdate(update)
 
-    override def combine(state: DataState[CheckInStateOnChain, CheckInDataCalculatedState], updates: List[Signed[CheckInUpdate]])(implicit context: L1NodeContext[IO]): IO[DataState[CheckInStateOnChain, CheckInDataCalculatedState]] = IO.pure(state)
+      override def combine(state: DataState[CheckInStateOnChain, CheckInDataCalculatedState], updates: List[Signed[CheckInUpdate]])(implicit context: L1NodeContext[IO]): IO[DataState[CheckInStateOnChain, CheckInDataCalculatedState]] = IO.pure(state)
 
-    override def routes(implicit context: L1NodeContext[IO]): HttpRoutes[IO] = HttpRoutes.empty
+      override def routes(implicit context: L1NodeContext[IO]): HttpRoutes[IO] = HttpRoutes.empty
 
-    override def dataEncoder: Encoder[CheckInUpdate] = implicitly[Encoder[CheckInUpdate]]
+      override def dataEncoder: Encoder[CheckInUpdate] = implicitly[Encoder[CheckInUpdate]]
 
-    override def dataDecoder: Decoder[CheckInUpdate] = implicitly[Decoder[CheckInUpdate]]
+      override def dataDecoder: Decoder[CheckInUpdate] = implicitly[Decoder[CheckInUpdate]]
 
-    override def calculatedStateEncoder: Encoder[CheckInDataCalculatedState] = implicitly[Encoder[CheckInDataCalculatedState]]
+      override def calculatedStateEncoder: Encoder[CheckInDataCalculatedState] = implicitly[Encoder[CheckInDataCalculatedState]]
 
-    override def calculatedStateDecoder: Decoder[CheckInDataCalculatedState] = implicitly[Decoder[CheckInDataCalculatedState]]
+      override def calculatedStateDecoder: Decoder[CheckInDataCalculatedState] = implicitly[Decoder[CheckInDataCalculatedState]]
 
-    override def signedDataEntityDecoder: EntityDecoder[IO, Signed[CheckInUpdate]] = Decoders.signedDataEntityDecoder
+      override def signedDataEntityDecoder: EntityDecoder[IO, Signed[CheckInUpdate]] = Decoders.signedDataEntityDecoder
 
-    override def serializeBlock(block: Signed[DataApplicationBlock]): IO[Array[Byte]] = IO(Serializers.serializeBlock(block)(dataEncoder.asInstanceOf[Encoder[DataUpdate]]))
+      override def serializeBlock(block: Signed[DataApplicationBlock]): IO[Array[Byte]] = IO(Serializers.serializeBlock(block)(dataEncoder.asInstanceOf[Encoder[DataUpdate]]))
 
-    override def deserializeBlock(bytes: Array[Byte]): IO[Either[Throwable, Signed[DataApplicationBlock]]] = IO(Deserializers.deserializeBlock(bytes)(dataDecoder.asInstanceOf[Decoder[DataUpdate]]))
+      override def deserializeBlock(bytes: Array[Byte]): IO[Either[Throwable, Signed[DataApplicationBlock]]] = IO(Deserializers.deserializeBlock(bytes)(dataDecoder.asInstanceOf[Decoder[DataUpdate]]))
 
-    override def serializeState(state: CheckInStateOnChain): IO[Array[Byte]] = IO(Serializers.serializeState(state))
+      override def serializeState(state: CheckInStateOnChain): IO[Array[Byte]] = IO(Serializers.serializeState(state))
 
-    override def deserializeState(bytes: Array[Byte]): IO[Either[Throwable, CheckInStateOnChain]] = IO(Deserializers.deserializeState(bytes))
+      override def deserializeState(bytes: Array[Byte]): IO[Either[Throwable, CheckInStateOnChain]] = IO(Deserializers.deserializeState(bytes))
 
-    override def serializeUpdate(update: CheckInUpdate): IO[Array[Byte]] = IO(Serializers.serializeUpdate(update))
+      override def serializeUpdate(update: CheckInUpdate): IO[Array[Byte]] = IO(Serializers.serializeUpdate(update))
 
-    override def deserializeUpdate(bytes: Array[Byte]): IO[Either[Throwable, CheckInUpdate]] = IO(Deserializers.deserializeUpdate(bytes))
+      override def deserializeUpdate(bytes: Array[Byte]): IO[Either[Throwable, CheckInUpdate]] = IO(Deserializers.deserializeUpdate(bytes))
 
-    override def getCalculatedState(implicit context: L1NodeContext[IO]): IO[(SnapshotOrdinal, CheckInDataCalculatedState)] = CalculatedState.getCalculatedState
+      override def getCalculatedState(implicit context: L1NodeContext[IO]): IO[(SnapshotOrdinal, CheckInDataCalculatedState)] = calculatedStateService.getCalculatedState.map(calculatedState => (calculatedState.ordinal, calculatedState.state))
 
-    override def setCalculatedState(ordinal: SnapshotOrdinal, state: CheckInDataCalculatedState)(implicit context: L1NodeContext[IO]): IO[Boolean] = CalculatedState.setCalculatedState(ordinal, state)
+      override def setCalculatedState(ordinal: SnapshotOrdinal, state: CheckInDataCalculatedState)(implicit context: L1NodeContext[IO]): IO[Boolean] = calculatedStateService.setCalculatedState(ordinal, state)
+
+      override def hashCalculatedState(state: CheckInDataCalculatedState)(implicit context: L1NodeContext[IO]): IO[Hash] = calculatedStateService.hashCalculatedState(state)
+
+      override def serializeCalculatedState(state: CheckInDataCalculatedState): IO[Array[Byte]] = IO(Serializers.serializeCalculatedState(state))
+
+      override def deserializeCalculatedState(bytes: Array[Byte]): IO[Either[Throwable, CheckInDataCalculatedState]] = IO(Deserializers.deserializeCalculatedState(bytes))
+    })
+
+  private def makeL1Service: IO[BaseDataApplicationL1Service[IO]] = {
+    for {
+      calculatedStateService <- CalculatedStateService.make[IO]
+      dataApplicationL1Service = makeBaseDataApplicationL1Service(calculatedStateService)
+    } yield dataApplicationL1Service
+  }
+
+  override def dataApplication: Option[Resource[IO, BaseDataApplicationL1Service[IO]]] =
+    makeL1Service.asResource.some
 
-    override def hashCalculatedState(state: CheckInDataCalculatedState)(implicit context: L1NodeContext[IO]): IO[Hash] = CalculatedState.hashCalculatedState(state)
-  }))
 }
diff --git a/metagraph/modules/l0/src/main/scala/com/my/dor_metagraph/l0/Main.scala b/metagraph/modules/l0/src/main/scala/com/my/dor_metagraph/l0/Main.scala
@@ -1,12 +1,12 @@
 package com.my.dor_metagraph.l0
 
 import cats.data.NonEmptyList
-import cats.effect.IO
-import cats.implicits.catsSyntaxValidatedIdBinCompat0
+import cats.effect.{IO, Resource}
+import cats.implicits.{catsSyntaxOptionId, catsSyntaxValidatedIdBinCompat0}
 import com.my.dor_metagraph.l0.custom_routes.CustomRoutes.getLatestCalculatedState
 import com.my.dor_metagraph.l0.rewards.MainRewards
 import com.my.dor_metagraph.shared_data.Data
-import com.my.dor_metagraph.shared_data.calculated_state.CalculatedState
+import com.my.dor_metagraph.shared_data.calculated_state.CalculatedStateService
 import com.my.dor_metagraph.shared_data.decoders.Decoders
 import com.my.dor_metagraph.shared_data.deserializers.Deserializers
 import com.my.dor_metagraph.shared_data.serializers.Serializers
@@ -25,8 +25,8 @@ import org.tessellation.sdk.domain.rewards.Rewards
 import org.tessellation.security.SecurityProvider
 import org.tessellation.security.hash.Hash
 import org.tessellation.security.signature.Signed
-
 import org.http4s.dsl.io._
+import org.tessellation.ext.cats.effect.ResourceIO
 
 import java.util.UUID
 
@@ -37,8 +37,10 @@ object Main
     ClusterId(UUID.fromString("517c3a05-9219-471b-a54c-21b7d72f4ae5")),
     version = BuildInfo.version
   ) {
-  def dataApplication: Option[BaseDataApplicationL0Service[IO]] =
-    Option(BaseDataApplicationL0Service(new DataApplicationL0Service[IO, CheckInUpdate, CheckInStateOnChain, CheckInDataCalculatedState] {
+  private def makeBaseDataApplicationL0Service(
+    calculatedStateService: CalculatedStateService[IO]
+  ): BaseDataApplicationL0Service[IO] =
+    BaseDataApplicationL0Service(new DataApplicationL0Service[IO, CheckInUpdate, CheckInStateOnChain, CheckInDataCalculatedState] {
       override def genesis: DataState[CheckInStateOnChain, CheckInDataCalculatedState] = DataState(CheckInStateOnChain(List.empty), CheckInDataCalculatedState(Map.empty))
 
       override def validateUpdate(update: CheckInUpdate)(implicit context: L0NodeContext[IO]): IO[DataApplicationValidationErrorOr[Unit]] = IO.pure(().validNec)
@@ -69,18 +71,31 @@ object Main
 
       override def deserializeUpdate(bytes: Array[Byte]): IO[Either[Throwable, CheckInUpdate]] = IO(Deserializers.deserializeUpdate(bytes))
 
-      override def getCalculatedState(implicit context: L0NodeContext[IO]): IO[(SnapshotOrdinal, CheckInDataCalculatedState)] = CalculatedState.getCalculatedState
+      override def getCalculatedState(implicit context: L0NodeContext[IO]): IO[(SnapshotOrdinal, CheckInDataCalculatedState)] = calculatedStateService.getCalculatedState.map(calculatedState => (calculatedState.ordinal, calculatedState.state))
 
-      override def setCalculatedState(ordinal: SnapshotOrdinal, state: CheckInDataCalculatedState)(implicit context: L0NodeContext[IO]): IO[Boolean] = CalculatedState.setCalculatedState(ordinal, state)
+      override def setCalculatedState(ordinal: SnapshotOrdinal, state: CheckInDataCalculatedState)(implicit context: L0NodeContext[IO]): IO[Boolean] = calculatedStateService.setCalculatedState(ordinal, state)
 
-      override def hashCalculatedState(state: CheckInDataCalculatedState)(implicit context: L0NodeContext[IO]): IO[Hash] = CalculatedState.hashCalculatedState(state)
+      override def hashCalculatedState(state: CheckInDataCalculatedState)(implicit context: L0NodeContext[IO]): IO[Hash] = calculatedStateService.hashCalculatedState(state)
 
       override def routes(implicit context: L0NodeContext[IO]): HttpRoutes[IO] = HttpRoutes.of {
-        case GET -> Root / "calculated-state" / "latest" => getLatestCalculatedState
+        case GET -> Root / "calculated-state" / "latest" => getLatestCalculatedState(calculatedStateService)
       }
-    }))
 
-  def rewards(implicit sp: SecurityProvider[IO]): Option[Rewards[IO, CurrencySnapshotStateProof, CurrencyIncrementalSnapshot, CurrencySnapshotEvent]] = Some(
-    MainRewards.make[IO]
-  )
+      override def serializeCalculatedState(state: CheckInDataCalculatedState): IO[Array[Byte]] = IO(Serializers.serializeCalculatedState(state))
+
+      override def deserializeCalculatedState(bytes: Array[Byte]): IO[Either[Throwable, CheckInDataCalculatedState]] = IO(Deserializers.deserializeCalculatedState(bytes))
+    })
+
+  private def makeL0Service: IO[BaseDataApplicationL0Service[IO]] = {
+    for {
+      calculatedStateService <- CalculatedStateService.make[IO]
+      dataApplicationL0Service = makeBaseDataApplicationL0Service(calculatedStateService)
+    } yield dataApplicationL0Service
+  }
+
+  override def dataApplication: Option[Resource[IO, BaseDataApplicationL0Service[IO]]] =
+    makeL0Service.asResource.some
+
+  override def rewards(implicit sp: SecurityProvider[IO]): Option[Rewards[IO, CurrencySnapshotStateProof, CurrencyIncrementalSnapshot, CurrencySnapshotEvent]] =
+    MainRewards.make[IO].some
 }