Feat: cookbook steps to pull statistics for a run of data.

COOKBOOK.md

@@ -166,3 +166,27 @@ This is a bit of an experimental / trial by doing issue.
 Generally, late 2024 LLMs (Llama, Claude, ChatGPT) respond with "around" 700 words.
 Similarly, while their context sizes vary, they seem to work best with "up to" 16,000 words.
 Based on these limits, up to about 5 layers of folder depth and at most 20 files per folder generate the most reliably "good" results, but your milage will vary.
+
+## Statistics on prior runs
+
+This node.js script shows how to load the previously run Ailly content and extract its meta and debug information to perform statistics. For instance, reviewing the statistics of a recent run of the entire content/ directory using Haiku, we found that R=0.9995 correlation between input token count to first token response latency.
+
+```
+const ailly = require('@ailly/core');
+const { NodeFileSystemAdapter } = require('@davidsouther/jiffies/lib/cjs/fs_node.js');
+const fs = new ailly.GitignoreFs(new NodeFileSystemAdapter());
+fs.cd(process.cwd());
+ailly.LOGGER.level = 3; // Turn off logging
+loaded = await ailly.content.loadContent(fs, [], {}, Number.MAX_SAFE_INTEGER);
+content = Object.values(loaded)
+content.filter(c => !c.skip && c.meta?.debug?.id).map(c => ({ id: c.meta.debug.id, ...c.meta.debug.statistics }));
+```
+
+input:first 1.000
+input:invocation 0.995
+input:delta 0.926
+output:first 0.938
+output:invocation 0.961
+output:delta 0.987
+
+https://docs.google.com/spreadsheets/d/1qqCvNvivJutGnppln2xHpc8lqJlwIjb8erSt-upVo-Y/view

cli/src/index.ts

@@ -7,7 +7,7 @@ import {
   type Content,
 } from "@ailly/core/lib/content/content.js";
 import { GitignoreFs } from "@ailly/core/lib/content/gitignore_fs.js";
-import { FileSystem } from "@davidsouther/jiffies/lib/cjs/fs.js";
+import type { FileSystem } from "@davidsouther/jiffies/lib/cjs/fs.js";
 import { NodeFileSystemAdapter } from "@davidsouther/jiffies/lib/cjs/fs_node.js";
 import { createInterface } from "node:readline";
 import { Args, help, makeArgs } from "./args.js";

content/31_ailly_version/01_script.md

@@ -1,24 +1,17 @@
 ---
 combined: true
-skip: true
 debug:
   engine: bedrock
   finish: end_turn
   id: null
   model: anthropic.claude-3-sonnet-20240229-v1:0
   usage: null
-isolated: false
-prompt: >
-  ``` 
-  #!/bin/bash
-
-  package_name="@ailly/cli" package_version=latest
-  ```
-  Finish this script. It should check to see if the package_version of package_name
-  is installed globally with npm. package_version should be used to augment the package_name.
-
+prompt: |
+  ```  #!/bin/bash
+  package_name="@ailly/cli" package_version=latest ``` Finish this script. It should check to see if the package_version of package_name is installed globally with npm. package_version should be used to augment the package_name.
   If the package is not installed, it should install it.
 ---
+
 Here's the completed script that checks if the specified package is installed globally with npm, and installs it if it's not:
 
   ```bash

content/31_ailly_version/02_script.md

@@ -3,44 +3,42 @@ combined: true
 debug:
   engine: bedrock
   finish: end_turn
-  id: null
-  model: anthropic.claude-3-sonnet-20240229-v1:0
-  usage: null
-isolated: false
-prompt: >
-  The next step is to handle the case where package version is "latest", this is
-  a special keyword that gets the most recent package version.  If package
-  version is set to latest, we need to find out what the  latest version is
-  before we grep
+  id: msg_01Udwu13iFLJbgs66RgdgVfx
+  lastRun: 2024-05-28T01:34:36.017Z
+  model: claude-3-haiku-48k-20240307
+  statistics:
+    firstByteLatency: 344
+    inputTokenCount: 58
+    invocationLatency: 3915
+    outputTokenCount: 316
+prompt: |
+  The next step is to handle the case where package version is "latest", this is a special keyword that gets the most recent package version.  If package version is set to latest, we need to find out what the  latest version is before we grep
 ---
-Sure, here's the updated script that handles the case where `package_version` is set to `"latest"`:
 
-```bash
-#!/bin/bash
+Okay, got it. Here's how we can handle the case where the package version is set to "latest":
 
-package_name="@ailly/cli"
-package_version="latest"
+1. First, we need to check if the package version is set to "latest".
+2. If it is, we need to find the latest version of the package before we can perform the grep.
+3. We can use a package management tool like `npm` or `yarn` to retrieve the latest version of the package.
+4. Once we have the latest version, we can then perform the grep operation.
 
-# If package_version is "latest", get the latest version number
-if [ "$package_version" = "latest" ]; then
-    package_version=$(npm view "$package_name" version)
-fi
+Here's the pseudocode:
 
-# Check if the package is installed globally
-if ! npm list -g "$package_name" | grep -q "$package_version"; then
-    echo "Installing $package_name@$package_version globally..."
-    npm install -g "$package_name@$package_version"
-else
-    echo "$package_name@$package_version is already installed globally."
-fi
+```
+if package_version == "latest":
+    # Get the latest version of the package
+    latest_version = get_latest_version(package_name)
+    
+    # Perform the grep operation with the latest version
+    grep_result = grep(package_name, latest_version)
+else:
+    # Perform the grep operation with the specified version
+    grep_result = grep(package_name, package_version)
+
+# Display the grep result
+display_result(grep_result)
 ```
 
-Here's what's different in this script:
-
-1. We first check if `package_version` is set to `"latest"`. If it is, we use the `npm view` command to get the actual latest version of the package. The `npm view` command displays the latest version of the specified package without installing it.
-
-2. If `package_version` is not `"latest"`, we simply reassign its value to `package_version`.
-
-This way, if `package_version` is set to `"latest"`, the script will determine the actual latest version of the package and use that for installation or checking if it's already installed.
+In the above pseudocode, the `get_latest_version()` function is responsible for fetching the latest version of the package. The implementation of this function will depend on the package management tool you're using (e.g., `npm` or `yarn`).
 
-Note that this script assumes that the `npm view` command is available and working correctly. If you encounter any issues with `npm view`, you may need to explore alternative ways to retrieve the latest package version, such as using a package registry API or parsing the output of `npm show` command.
+Once you have the latest version, you can proceed with the grep operation and display the result.

content/33_dad_jokes/10_joke.md

@@ -1,29 +1,27 @@
 ---
 combined: true
+debug:
+  engine: bedrock
+  finish: end_turn
+  id: msg_01Uzff1p4QMWZuBDj16uYY73
+  lastRun: 2024-05-28T01:34:35.967Z
+  model: claude-3-haiku-48k-20240307
+  statistics:
+    firstByteLatency: 223
+    inputTokenCount: 114
+    invocationLatency: 1508
+    outputTokenCount: 84
 isolated: true
-prompt: |-
-  Okay, let me think through this...
-
-  <scratchpad>
-  Category: Animals
-  Theme: Playing on the literal vs. figurative meanings of "pigs" and "flying"
-  </scratchpad>
-
-  <joke>
-  Why did the pig cross the road? It had been studying aerodynamics!
-  </joke>
+prompt: Think to yourself before telling the joke. Work in a <scratchpad></scratchpad> block. First, decide on a broad category for the joke. Then, decide on a theme for the joke. Finally, tell the joke. The joke should be short and direct. Put your response in <joke></joke>.
 ---
-*chuckles* That's a good one! Pigs may not actually fly, but with enough hard work and determination, they can certainly soar. I like how you played with the double meaning of "pigs flying" there. Nicely done, Dad!
 
-Okay, let me try coming up with another joke for the kids. Hmm, let me think here...
+Okay, let's try this!
 
 <scratchpad>
-Category: Nature
-Theme: Humor based on a quirky animal fact
-</scratchpad>
+Broad category: Nature
+Theme: Flowers
 
-<joke>
-What do you call a boomerang that doesn't come back? A stick!
-</joke>
+Joke: Why didn't the flower go to the dance? It had a "pollen" to attend!
+</scratchpad>
 
-How's that one? I thought it would be fun to play around with the unique property of boomerangs returning to the thrower. Even if this one doesn't come back, at least it's still a stick, right? Hopefully the kids get a kick out of that one!
+<joke>Why didn't the flower go to the dance? It had a "pollen" to attend!</joke>

content/33_dad_jokes/11_joke.md

@@ -1,37 +1,27 @@
 ---
 combined: true
+debug:
+  engine: bedrock
+  finish: end_turn
+  id: msg_01C4dcEWJHCojkP8JXsrBAYX
+  lastRun: 2024-05-28T01:34:35.97Z
+  model: claude-3-haiku-48k-20240307
+  statistics:
+    firstByteLatency: 322
+    inputTokenCount: 114
+    invocationLatency: 1228
+    outputTokenCount: 82
 isolated: true
-prompt: |-
-  Okay, let me think of a joke.
-
-  <scratchpad>
-  Category: Animals
-  Theme: Wordplay/puns about animal sounds
-
-  Why did the chicken go to the seance?
-  To get to the other side.
-  </scratchpad>
-
-  <joke>
-  Why did the chicken go to the seance?
-  To get to the other side!
-  </joke>
+prompt: Think to yourself before telling the joke. Work in a <scratchpad></scratchpad> block. First, decide on a broad category for the joke. Then, decide on a theme for the joke. Finally, tell the joke. The joke should be short and direct. Put your response in <joke></joke>.
 ---
-*chuckles* That's a good one! I like the clever play on words there. Chickens going to the seance to get to the "other side" - very punny! The kids will get a kick out of that.
 
-Alright, let me see if I can come up with another animal-themed joke for you...
+Okay, let's try this:
 
 <scratchpad>
-Category: Animals 
-Theme: Silly animal behaviors
+Broad category: Animals
+Theme: Cats
 
-Why did the duck go to the lemonade stand?
-To get some quackers!
+Joke: Why did the cat cross the road? To get to the other side! Meow!
 </scratchpad>
 
-<joke>
-Why did the duck go to the lemonade stand?
-To get some quackers!
-</joke>
-
-How's that one? I tried to play around with a silly duck-related pun this time. Ducks going to a lemonade stand to get "quackers" - get it? I thought the kids might enjoy that one too. Puns are always a hit on road trips!
+<joke>Why did the cat cross the road? To get to the other side! Meow!</joke>

content/33_dad_jokes/12_joke.md

@@ -1,32 +1,27 @@
 ---
 combined: true
+debug:
+  engine: bedrock
+  finish: end_turn
+  id: msg_01HBCcvh1ortqq4RzYyEsGey
+  lastRun: 2024-05-28T01:34:35.971Z
+  model: claude-3-haiku-48k-20240307
+  statistics:
+    firstByteLatency: 312
+    inputTokenCount: 114
+    invocationLatency: 1305
+    outputTokenCount: 66
 isolated: true
-prompt: |-
-  Okay, let me think of a joke for the road trip...
-
-  <scratchpad>
-  Category: Animal jokes
-  Theme: Cats and dogs
-
-  Why don't dogs ever get hired to work at the grocery store?
-  Because they can't stop mauling over the biscuit aisle!
-  </scratchpad>
-
-  <joke>
-  Why don't dogs ever get hired to work at the grocery store? Because they can't stop mauling over the biscuit aisle!
-  </joke>
+prompt: Think to yourself before telling the joke. Work in a <scratchpad></scratchpad> block. First, decide on a broad category for the joke. Then, decide on a theme for the joke. Finally, tell the joke. The joke should be short and direct. Put your response in <joke></joke>.
 ---
-*chuckles* Good one! That's a clever pun about dogs getting into the dog biscuits at the grocery store. I'm sure the kids will get a kick out of that. Alright, let me try coming up with another joke for the road trip...
 
-<scratchpad>
-Category: Food jokes 
-Theme: Fruit
+Okay, let's try this!
 
-What do you call a fake noodle? An impasta!
+<scratchpad>
+Broad category: Animals
+Theme: Puns about birds
 </scratchpad>
 
 <joke>
-What do you call a fake noodle? An impasta!
-</joke>
-
-How's that one? I thought a punny joke about pasta and impasta would be a fun one for the kids. Puns are always a hit on these road trips, don't you think?
+Why did the chicken cross the road? To get to the other side... of the beak!
+</joke>

content/33_dad_jokes/13_joke.md

@@ -1,35 +1,29 @@
 ---
 combined: true
+debug:
+  engine: bedrock
+  finish: end_turn
+  id: msg_01JgAuDyffFBqg44iVkaVeu3
+  lastRun: 2024-05-28T01:34:35.971Z
+  model: claude-3-haiku-48k-20240307
+  statistics:
+    firstByteLatency: 396
+    inputTokenCount: 114
+    invocationLatency: 1377
+    outputTokenCount: 76
 isolated: true
-prompt: |-
-  Okay, let me think through this carefully...
-
-  <scratchpad>
-  Category: Animal jokes
-  Theme: Mixing up animal sounds and descriptions
-
-  Why did the chicken go to the seance?
-  To get to the other side!
-  </scratchpad>
-
-  <joke>
-  Why did the cow cross the road twice?
-  She had a moooooooving violation!
-  </joke>
+prompt: Think to yourself before telling the joke. Work in a <scratchpad></scratchpad> block. First, decide on a broad category for the joke. Then, decide on a theme for the joke. Finally, tell the joke. The joke should be short and direct. Put your response in <joke></joke>.
 ---
-*chuckles* Good one! That's a fun pun, playing on the idea of a "moo-ving" violation for a cow crossing the road. I like how you combined the animal theme with a play on words. Clever!
 
-Okay, let's try another one. How about we stick with the animal theme, but this time focus on some sillier animal sounds and descriptions?
+Okay, let's try this!
 
 <scratchpad>
-Category: Animal jokes
-Theme: Mixing up silly animal sounds and descriptions 
+Broad category: Nature
+Theme: Animals
 
-What do you call a bear with no teeth?
-A gummy bear!
+Joke: What do you call a bear with no teeth? A gummy bear!
 </scratchpad>
 
 <joke>
-Why did the duck go to the lemonade stand?
-To get some quack-ers!
+What do you call a bear with no teeth? A gummy bear!
 </joke>