Why not delegate publishing completely to quarto publishing framework?

[Rahuketu86]

Version 2.3.0

Request for enhancement

Why not delegate publishing completely to quarto publishing framework?

With the introduction of _proc in version 2.3.0 , nbdev resolves a few issues around publishing. But it still doesn't completely capitalize _quarto publishing functionality.

Let me try to explain
When I run the nbdev_preview command a _docs folder is generated in the _proc directory. Looking at the source code I believe same/similar _docs folder is generated by the nbdev_docs command at the root directory.

My thought process is following

1. Generate a _publish.yml in _proc folder using the quarto publish command.
2. We can still use the nbdev_docs command but this _proc/_docs folder is updated with everything expected in terms of rendering ( without prior cache etc...)
3. Modify deploy action at https://github.com/fastai/workflows/blob/master/quarto-ghp/action.yml to something more general which (a) runs nbdev_docs (b) uses something similar to quarto publish action to deploy the docs based on _publish setting.
   (In fact, default deployment to GitHub pages can be already configured for _publish.yml while generating _proc folder using nbdev_prepare but can be modified by a user on demand based on a case-by-case basis.)

The below setup for deploy has not worked ( might be an issue with quarto-action) - with various combinations of path[ _proc, ./_proc] or putting _publish.yml in multiple places ( _proc dir, root dir, both, etc..)
Note : - I can run quarto publish netlify locally (it's just that quarto action is failing for some reason)

name: CI
on:
  push:
    paths-ignore:
      - 'book/**'
      - 'nbs_pluto/**'
      - 'nbs_pluto_html/**'
      - 'devops/**'
  pull_request:
  workflow_dispatch:

jobs:
  test-deploy:
    runs-on: ubuntu-latest
    
    steps:
    - name: Install Dependencies
      run: |
        sudo apt install -y graphviz

    - name: Fastai CI
      uses: fastai/workflows/nbdev-ci@master

    - name: Run nbdev_docs
      shell: bash
      run: |
        nbdev_docs

    - name: Render and Publish 
      uses: quarto-dev/quarto-actions/publish@v2
      with:
        target: netlify
        path: _proc
        NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}

Failing with following message :-

No _publish.yml file available (_publish.yml specifying a destination required for non-interactive publish)

I am using the following to achieve netlify deployment which solves what I want to achieve for now. However, it would be much better if we can link to publishing framework directly from quarto. This way it would be easier to adapt actions once they move to a profile system (See my issue in quarto-actions quarto-dev/quarto-actions#44 (comment))

name: CI
on:
  push:
    paths-ignore:
      - 'book/**'
      - 'nbs_pluto/**'
      - 'nbs_pluto_html/**'
      - 'devops/**'
  pull_request:
  workflow_dispatch:


jobs:
  test-deploy:
    runs-on: ubuntu-latest
    
    steps:
    - name: Install Dependencies
      run: |
        sudo apt install -y graphviz

    - name: Fastai CI
      uses: fastai/workflows/nbdev-ci@master

    - name: Run nbdev_docs
      shell: bash
      run: |
        nbdev_docs

    - name: Push to netlify site
      uses: nwtgck/[email protected]
      with:
        publish-dir: './_docs'
        production-branch: master
        github-token: ${{ secrets.GITHUB_TOKEN }}
        deploy-message: "Deploy from GitHub Actions"
        enable-pull-request-comment: false
        enable-commit-comment: true
        overwrites-pull-request-comment: true
      env:
        NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
        NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
      timeout-minutes: 1

Originally posted by @Rahuketu86 in #998 (comment)

[hamelsmu]

Can you try putting _publish.yml in your notebook directory (whatever is nbs_path is in your settings.ini)? That way, your _publish.yml file will be copied over to the right places successfully

[hamelsmu]

Also read these docs https://nbdev.fast.ai/explanations/docs.html#deploying-your-docs-on-other-platforms can you make this work locally? If so, you should be able to do the same in GitHub Actions

[Rahuketu86]

My current workflow is following :

1. I create _publish.yml manually in _proc dir [ I also move generated _publish.yml to project as quarto documentation recommends]
2. I am able to run quarto publish netlify locally in both cases. However, integrating with quarto action fails for some reason with error message No _publish.yml file available (_publish.yml specifying a destination required for non-interactive publish) in both cases.

Nevertheless, my reason for the request is to avoid an additional render done in the nbdev_docs command and then again in the quarto-actions when following this workflow. Also a possible use case with multiple user setups / "Knowledge repo" is to have a different deployment target/ URL for users with different roles/ permissions which I am unable to achieve cleanly. I raised a ticket on quarto forums quarto-dev/quarto-actions#44 (comment) and they mentioned something like profiles that could be in work for quarto publish.

[hamelsmu]

Did you try this?

Can you try putting _publish.yml in your notebook directory (whatever is nbs_path is in your settings.ini)? That way, your _publish.yml file will be copied over to the right places successfully

[Rahuketu86]

No, not yet. Let me try this today and report back