Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experiment SDK Development #451

Draft
wants to merge 41 commits into
base: agent-framewok-refactoring
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
50ce119
modify python sdk to improve compatibility with experiment sdk
yasithdev Nov 25, 2024
aa6c5ea
make connector return path
yasithdev Nov 25, 2024
2d0c914
add experiment sdk code
yasithdev Dec 3, 2024
e211311
fix relative dependency
yasithdev Dec 3, 2024
ccbcde7
log errors
yasithdev Dec 4, 2024
7442bec
fix bugs, improve poc notebook
yasithdev Dec 5, 2024
af9f337
update poc notebook, add stop functionality, prevent keyerror exception
yasithdev Dec 5, 2024
b61a6b8
merge experiment sdk with python sdk
yasithdev Dec 5, 2024
801532f
remove line breaks in f strings (for python < 3.12), fix bugs
yasithdev Dec 5, 2024
e99edd7
update version to 0.0.2
yasithdev Dec 5, 2024
f6033c3
add cat cmd
yasithdev Dec 5, 2024
d9c1ac4
add refresh token flow, improve auth flow
yasithdev Dec 8, 2024
34aa4f3
update api to download and upload files, reduce settings.ini dependen…
yasithdev Dec 9, 2024
6d3b39e
add plan CRUD apis
yasithdev Dec 9, 2024
b9c90bd
fix issues, add python code invocation
yasithdev Dec 9, 2024
1ef875f
tabular display using pandas
yasithdev Dec 12, 2024
1830cfb
update notebooks and bump package version
yasithdev Dec 12, 2024
e4318d9
add venv script to agent
yasithdev Dec 13, 2024
75dcc5a
fix string escape in code
yasithdev Dec 13, 2024
99322f7
change parentExperimentId value to "/data" temporarily.
yasithdev Dec 13, 2024
14dd81d
fix bugs
yasithdev Dec 13, 2024
b8c2e23
fix bug
yasithdev Dec 13, 2024
7a0bba6
remove runtime configs from settings.ini, ability to pick runtimes by…
yasithdev Dec 16, 2024
4c1872f
update string handling, add agent scripts
yasithdev Dec 16, 2024
df1625e
update notebooks and sdk outputs
yasithdev Dec 16, 2024
d3a15c5
ux changes
yasithdev Dec 17, 2024
e1af552
move poc into MD folder, update sdk for new file upload/download API
yasithdev Dec 17, 2024
4c3e269
text-mode upload and download
yasithdev Dec 17, 2024
b20a66b
cleanup runtime.py and update airavata.py with fallback apis
yasithdev Dec 17, 2024
43d2296
take CONNECTION_SVC_URL and FILEMGR_SVC_URL from settings.ini
yasithdev Dec 17, 2024
29d5469
implement pre-submission validation, change storage directory
yasithdev Dec 17, 2024
ad897aa
fix bug in agent_id
yasithdev Dec 17, 2024
93d896c
update file listing and python execution cmds
yasithdev Dec 18, 2024
5a3447f
update notebooks
yasithdev Dec 18, 2024
93ab5fb
lock agent to python 3.12 and update pyproject.toml
yasithdev Dec 18, 2024
e5b88f8
separate python env creation and code execution steps. always return …
yasithdev Dec 18, 2024
a7a3d70
remove verbose log from remote code execution
yasithdev Dec 18, 2024
aa21e99
update notebooks
yasithdev Dec 18, 2024
e9163a0
reset changes to sftp_file_handling_client
yasithdev Dec 18, 2024
47b4875
add support to cold-start analysis agents
yasithdev Dec 18, 2024
7f7f298
fix bug in code execution, remove jupyter kernel start from agent
yasithdev Dec 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,12 @@ airavata_python_sdk.egg-info
.tox
dist
build
__pycache__/
.DS_Store
.ipynb_checkpoints
*.egg-info/
data/
results*/
plan.json
settings*.ini
auth.state
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

from __future__ import annotations

from . import base, plan
from .auth import login, logout
from .runtime import list_runtimes, Runtime

__all__ = ["login", "logout", "list_runtimes", "base", "plan"]

def display_runtimes(runtimes: list[Runtime]):
"""
Display runtimes in a tabular format
"""
import pandas as pd

records = []
for runtime in runtimes:
record = dict(id=runtime.id, **runtime.args)
records.append(record)

return pd.DataFrame(records)

def display_experiments(experiments: list[base.Experiment]):
"""
Display experiments in a tabular format
"""
import pandas as pd

records = []
for experiment in experiments:
record = dict(name=experiment.name, application=experiment.application.app_id, num_tasks=len(experiment.tasks))
for k, v in experiment.inputs.items():
record[k] = ", ".join(v) if isinstance(v, list) else str(v)
records.append(record)

return pd.DataFrame(records)

def display_plans(plans: list[plan.Plan]):
"""
Display plans in a tabular format
"""
import pandas as pd

records = []
for plan in plans:
for task in plan.tasks:
record = dict(plan_id=str(plan.id))
for k, v in task.model_dump().items():
record[k] = ", ".join(v) if isinstance(v, list) else str(v)
records.append(record)

return pd.DataFrame(records)

def display(arg):

if isinstance(arg, list):
if all(isinstance(x, Runtime) for x in arg):
return display_runtimes(arg)
if all(isinstance(x, base.Experiment) for x in arg):
return display_experiments(arg)
if all(isinstance(x, plan.Plan) for x in arg):
return display_plans(arg)
else:
if isinstance(arg, Runtime):
return display_runtimes([arg])
if isinstance(arg, base.Experiment):
return display_experiments([arg])
if isinstance(arg, plan.Plan):
return display_plans([arg])

raise NotImplementedError(f"Cannot display object of type {type(arg)}")
Loading