Bring back Python backend based PyTorch backend #117

kthui · 2023-11-14T03:44:54Z

Related PRs:

Add Python backend based PyTorch runtime. Add script to build Conda-Pack environment.

rmccorm4 · 2023-11-27T23:03:11Z

src/model.py

+
+
+def _get_model_path(config):
+    filenames = ["model.py", "model.pt"]


If I am understanding the whole flow correctly:

The runtime flag for python based implementation in Triton core will look for model.py (configurable by user, but would typically be called model.py) in model/version repo, and then backend repo (in that order)

The python runtime for pytorch backend supports loading pytorch NN module style models as a model.py file

Assume the following setup:

models/ -- resnet50 -- config.pbtxt # runtime: "model.py" -- 1/ -- model.py # resnet50 pytorch NN implementation backends/ -- pytorch/ -- model.py # python-based backend implementation

Since the core looks in model repo first, if you tried to load a pytorch models/resnet50/1/model.py (nn module file) that uses the Triton python runtime pytorch implementation (/opt/tritonserver/backends/pytorch/model.py) -- Triton core would probably try to load this models/resnet50/1/model.py as the backend implementation first and fail, right?

I guess this might be the most confusing part. If a model wants to load on top of Python backend, the backend_libdir and backend_libpath will initially point to the C++ runtime of the Python backend. After that, the backend_libdir will be "updated" to point to the Python runtime (i.e. /opt/tritonserver/backends/pytorch/model.py), without modifying backend_libpath.

The reason why the "update" logic is always correct is the Python runtime path is always assembled from backend_dir and model.py, where backend_dir always points to /opt/tritonserver/backends/pytorch, so the eventual backend_libdir can only be /opt/tritonserver/backends/pytorch/model.py for example.

However, you are right if the user does not specify the runtime (or filled in by autocomplete), the latter runtime resolution step can find the wrong file as outlined, but this should not be an issue for our backends (vLLM and PyTorch) because the autocomplete will always fill the correct runtime. This will be an issue for custom Python based backends.

I think we should rename the runtime from model.py to backend.py to avoid any ambiguity and allow for better search logic, but model.py is used since vLLM, so it will be a behavioral change if we do so.

Otherwise, we will have to limit the search for Python based runtime to only within backend_dir (i.e. /opt/tritonserver/backends/pytorch)

Limit Python based backend search to backend directory

Issue is addressed, with commit up to triton-inference-server/core@f502bfc point

Try to avoid force pushing if possible, as the commit referenced previously is no longer valid.

Can you clarify the current behavior now?

For python-based backends, it will only look for model.py in the backend directory.

For C++ backends, will it still look in (version_dir, model_dir, backend_dir)?

how does this interact with escape logic?

can you verify running a custom backend with shared library located in the model folder still works?

For python-based backends, it will only look for model.py in the backend directory.
For C++ backends, will it still look in (version_dir, model_dir, backend_dir)?

Yes.

how does this interact with escape logic?

The backend_libdir will be one of the (version_dir, model_dir, backend_dir) path. The runtime is essentially the backend_libname, which <backend_libdir>/<runtime> is the backend_libpath. At the end after everything are determined, there is a one time check to ensure the backend_libpath is within the backend_libdir. If the runtime tries to escape from the backend_libdir (i.e. runtime = "../my_backend_lib.so" -> backend_libpath = "backend_dir/../my_backend_lib.so"), this will be caught.

can you verify running a custom backend with shared library located in the model folder still works?

Yes. I think the L0_lifecycle covers such case, and it is part of the CI run. It copies the libtriton_identity.so into the model folder, see cp libtriton_identity.so models/identity_zero_1_int32/1/. line.

I also ran L0_passive_instance on the test container from CI and it passed, which uses a custom C++ backend: https://github.com/triton-inference-server/server/blob/main/qa/L0_passive_instance/models/distributed_int32_int32_int32/config.pbtxt#L28

oandreeva-nv · 2023-11-29T20:25:38Z

tools/gen_pb_exec_env.sh

+conda install -c conda-forge libstdcxx-ng=12 -y
+
+# install PyTorch
+conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia -y


if we don't specify version 12.1 for pytorch-cuda, will it still install the latest stable version?

Without specifying version 12.1 for pytorch-cuda worked locally. I am ok with removing the specified version. My only concern is why PyTorch suggests setting pytorch-cuda=12.1 at the first place:

conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

Would it break in the future? Is it only a mean to lock CUDA to 12.1?
https://pytorch.org/get-started/locally/ (Stable (2.1.1) -> Linux -> Conda -> Python -> CUDA 12.1 -> get the command)

Do not specify pytorch cuda version

oandreeva-nv · 2023-11-29T20:36:24Z

src/model.py

+
+
+def _get_model_path(config):
+    filenames = ["model.py", "model.pt"]


do we want to support <XXX>.pt2 ? i.e result of torch.export : https://pytorch.org/docs/stable/export.html#serialization

How about we add .pt2 support as a follow-up ticket? It is because I think it depends on torch.export which is not part of the previous "platform handler", so we do not introduce additional risks for this "bring back platform handler" ticket and get some functionality in sooner than later. (unless .pt2 requires behavioral changes if introduced separately?)

Works for me. Can we add this note as a FIXME? relevant ticket is: https://jirasw.nvidia.com/browse/DLIS-5694

Add note for adding .pt2 model support

Tabrizian · 2024-01-02T16:15:32Z

CMakeLists.txt

@@ -504,6 +505,21 @@ install(
    ${INSTALL_CONFIGDIR}
 )

+if (${TRITON_PYTORCH_ENABLE_PYTHON_RUNTIME})


Does this mean that we'll include the runtime by default? I believe the environment size could be significant since it contains CUDA, PyTorch, etc.

Yes, but the behavior can be easily changed. The current size of the conda pack is 3.00 GB.

Tabrizian · 2024-01-02T16:15:54Z

src/model.py

+# triton_python_backend_utils is available in every Triton Python model. You
+# need to use this module to create inference requests and responses. It also
+# contains some utility functions for extracting information from model_config
+# and converting Triton input/output types to numpy types.


Let's remove the comment.

Removed: Remove legacy comment

…kend into jacky-python-based-pytorch

Tabrizian · 2024-01-10T15:56:54Z

README.md

+model_repository/
+`-- model_directory
+    |-- 1
+    |   |-- model.py


I think this is probably a little bit misleading since we don't always require both model.py and model.pt (e.g., if torchscript is provided only model.pt is required as you mentioned).

Good spotting! I created different sections for PyTorch 2.0 and TorchScript model layout. Clarify model layout between PyTorch and TorchScript

rmccorm4

This PR generally looks good to me - just looking for some clarifications to close this thread: https://github.com/triton-inference-server/pytorch_backend/pull/117/files#r1447954567

This was referenced Nov 14, 2023

Incorporate runtime into model configuration triton-inference-server/core#285

Merged

Add runtime to model configuration triton-inference-server/common#103

Merged

Bring back Python backend based PyTorch backend triton-inference-server/server#6518

Merged

kthui marked this pull request as draft November 14, 2023 17:09

Add Python backend based PyTorch runtime

d126b08

kthui force-pushed the jacky-python-based-pytorch branch from 22e5150 to 990aeb8 Compare November 14, 2023 17:12

Add exec env build

8d50071

kthui force-pushed the jacky-python-based-pytorch branch from 990aeb8 to 8d50071 Compare November 14, 2023 18:25

kthui requested review from Tabrizian, tanmayv25 and nnshah1 November 14, 2023 18:28

kthui marked this pull request as ready for review November 14, 2023 18:40

rmccorm4 reviewed Nov 27, 2023

View reviewed changes

oandreeva-nv reviewed Nov 29, 2023

View reviewed changes

kthui added 3 commits December 8, 2023 12:06

Add note for adding .pt2 model support

4381340

Do not specify pytorch cuda version

f71dd17

Do not install Python runtime on non x86

b459f73

Tabrizian reviewed Jan 2, 2024

View reviewed changes

kthui added 5 commits January 2, 2024 17:19

Merge branch 'main' of github.com:triton-inference-server/pytorch_bac…

8ff6a3a

…kend into jacky-python-based-pytorch

Remove legacy comment

78c47fe

Merge branch 'main' of github.com:triton-inference-server/pytorch_bac…

e7cf4d2

…kend into jacky-python-based-pytorch

User to build PyTorch env

8b856f6

Add docs

7d8a3a7

kthui mentioned this pull request Jan 8, 2024

Python backend based PyTorch backend documentations triton-inference-server/backend#94

Merged

Update copyright

9aa6b41

kthui force-pushed the jacky-python-based-pytorch branch from 3d6d797 to 9aa6b41 Compare January 8, 2024 23:40

Tabrizian reviewed Jan 10, 2024

View reviewed changes

kthui added 2 commits January 10, 2024 10:11

Clarify model layout between PyTorch and TorchScript

b8abcaa

Fix header size

63251ba

kthui requested review from oandreeva-nv, rmccorm4 and Tabrizian January 10, 2024 18:16

rmccorm4 reviewed Jan 10, 2024

View reviewed changes

kthui requested a review from rmccorm4 January 11, 2024 00:28

rmccorm4 approved these changes Jan 11, 2024

View reviewed changes

kthui merged commit 7468381 into main Jan 11, 2024
1 check passed

kthui deleted the jacky-python-based-pytorch branch January 11, 2024 17:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bring back Python backend based PyTorch backend #117

Bring back Python backend based PyTorch backend #117

kthui commented Nov 14, 2023 •

edited

Loading

rmccorm4 Nov 27, 2023 •

edited

Loading

kthui Nov 28, 2023 •

edited

Loading

kthui Nov 28, 2023 •

edited

Loading

kthui Nov 28, 2023

kthui Dec 8, 2023

rmccorm4 Jan 10, 2024

rmccorm4 Jan 10, 2024 •

edited

Loading

kthui Jan 11, 2024

kthui Jan 11, 2024

oandreeva-nv Nov 29, 2023

kthui Dec 8, 2023 •

edited

Loading

kthui Dec 8, 2023

oandreeva-nv Nov 29, 2023

kthui Dec 5, 2023

oandreeva-nv Dec 5, 2023

kthui Dec 8, 2023

Tabrizian Jan 2, 2024

kthui Jan 3, 2024

Tabrizian Jan 2, 2024

kthui Jan 3, 2024

Tabrizian Jan 10, 2024

kthui Jan 10, 2024

rmccorm4 left a comment



		def _get_model_path(config):
		filenames = ["model.py", "model.pt"]

Bring back Python backend based PyTorch backend #117

Bring back Python backend based PyTorch backend #117

Conversation

kthui commented Nov 14, 2023 • edited Loading

rmccorm4 Nov 27, 2023 • edited Loading

Choose a reason for hiding this comment

kthui Nov 28, 2023 • edited Loading

Choose a reason for hiding this comment

kthui Nov 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmccorm4 Jan 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kthui Dec 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmccorm4 left a comment

Choose a reason for hiding this comment

kthui commented Nov 14, 2023 •

edited

Loading

rmccorm4 Nov 27, 2023 •

edited

Loading

kthui Nov 28, 2023 •

edited

Loading

kthui Nov 28, 2023 •

edited

Loading

rmccorm4 Jan 10, 2024 •

edited

Loading

kthui Dec 8, 2023 •

edited

Loading