AgentKit
AgentKit
AgentKit
文档指南
请输入
AgentKit
产品动态
产品计费
快速入门
用户指南
智能体运行时
沙箱模板
沙箱模板管理
在 Agent 中集成沙箱模板
Skills Sandbox 沙箱模板
技能
A2A 中心
会话与记忆管理
知识库
网关
MCP 接入
身份与权限
快速入门
观测
开发者指南
API 参考
Runtime
Tool
Skill
A2A
SessionStore
Memory
Knowledge
MCP
ChatKit 参考
DeliveryAI
产品动态
产品计费
开始使用
步骤一：企业管理员-开通并配置服务
- 文档首页
AgentKitAgentKit用户指南智能体运行时调用Agent多模态调用示例（prompt和files）
复制全文
下载 pdf多模态调用示例（prompt和files）
文档反馈
问问助手本文介绍Agent多模态调用方法，实现将图片、视频、PDF等多模态文件直接作为AgentKit模型输入，让模型直接解析多模态内容，无需借助外部工具分析（例如，输入URL到Prompt中依赖文件解析工具分析）。
环境准备
- Python环境初始化。
- 使用uv构建独立的Python虚拟环境（推荐Python 3.10+）。
- # 初始化 uv 配置uv init# 创建 uv 虚拟环境uv venv
- 安装依赖包。
- 安装最新版本的VeADK。
- uv pip install "git+https://github.com/volcengine/veadk-python.git"
- 安装AgentKit SDK。
- uv pip install agentkit-sdk-python
- 开启多模态文件上传能力。
- 在agent代码中添加enable_responses=True，enable_responses参数开启后会兼容Response API从而打开图片视频和PDF的文件上传能力。
- agent = Agent(name=agent_name,description=description,instruction=system_prompt,tools=tools,enable_responses=True # 开启多模态文件输入能力)
修改Agent的入口函数
AgentKit调用/invoke接口时会触发Agent入口函数@app.entrypoint，需修改该函数以读取payload中的files参数（多模态文件URL列表），并传递给VeADK模型。
原始代码（仅支持文本） | 修改后代码（支持多模态文件） |
原始代码（仅支持文本） | 修改后代码（支持多模态文件） |
@app.entrypoint async def run(payload: dict, headers: dict): prompt = payload["prompt"] user_id = headers["user_id"] session_id = headers["session_id"] logger.info( f"Running agent with prompt: {prompt}, user_id: {user_id}, session_id: {session_id}" ) session_service = runner.short_term_memory.session_service # type: ignore # 防止会话重复创建 session = await session_service.get_session( app_name=app_name, user_id=user_id, session_id=session_id ) if not session: await session_service.create_session( app_name=app_name, user_id=user_id, session_id=session_id ) new_message = Content(role="user", parts=[Part(text=prompt)]) try: async for event in runner.run_async( user_id=user_id, session_id=session_id, new_message=new_message, run_config=RunConfig(streaming_mode=StreamingMode.SSE), ): # 格式化SSE数据 sse_event = event.model_dump_json(exclude_none=True, by_alias=True) logger.debug("Generated event in agent run streaming: %s", sse_event) yield sse_event except Exception as e: logger.exception("Error in event_generator: %s", e) # 返回错误信息 error_data = json.dumps({"error": str(e)}) yield error_data | @app.entrypoint async def run(payload: dict, headers: dict): prompt = payload["prompt"] user_id = headers["user_id"] session_id = headers["session_id"] logger.info( f"Running agent with prompt: {prompt}, user_id: {user_id}, session_id: {session_id}" ) session_service = runner.short_term_memory.session_service # type: ignore # 防止会话重复创建 session = await session_service.get_session( app_name=app_name, user_id=user_id, session_id=session_id ) if not session: await session_service.create_session( app_name=app_name, user_id=user_id, session_id=session_id ) new_message = Content(role="user", parts=[Part(text=prompt)]) # 读取payload中的files列表，将每个文件URL封装为Part.from_uri并添加到消息体中 files = payload.get("files", []) for file_url in files: new_message.parts.append(Part.from_uri(file_uri=file_url)) try: async for event in runner.run_async( user_id=user_id, session_id=session_id, new_message=new_message, run_config=RunConfig(streaming_mode=StreamingMode.SSE), ): # 格式化SSE数据 sse_event = event.model_dump_json(exclude_none=True, by_alias=True) logger.debug("Generated event in agent run streaming: %s", sse_event) yield sse_event except Exception as e: logger.exception("Error in event_generator: %s", e) # 返回错误信息 error_data = json.dumps({"error": str(e)}) yield error_data |
多模态文件调用测试
修改完成后，可通过curl命令调用/invoke接口，传入prompt（文本指令）和files（多模态文件URL列表），以下是不同文件类型的调用示例：
图片文件调用
curl --location 'http://127.0.0.1:8000/invoke' \
--header 'Content-Type: application/json' \
--header 'user_id: 123' \
--header 'session_id: 1234243' \
--data '{
"prompt": "describe this image , without using any tool",
"files": ["your image url"]
}'
响应示例
Start with the chicken feet: they have a bright, glossy golden-yellow color from the marinade, with the characteristic textured, wrinkled skin of cooked chicken feet, showing the raised scales/ridges clearly. The marinade is a thin, slightly reddish-orange oily sauce coating every surface, with small bits of chopped red chili peppers scattered amongst them, adding pops of bright red color. They are stacked in a heaping pile, some overlapping each other, looking plump and tender.
The plate is a simple rectangular white porcelain dish with rounded corners, sitting on a warm brown wooden surface that has visible wood grain. In the background, there's a blurred dark woven rattan or wicker texture, giving a rustic, homey food photography setting. A tiny hint of a whole red chili is visible on the far left edge of the frame. The lighting is warm and bright, highlighting the glossy sheen of the sauce on the chicken feet, making them look appetizing and savory, likely a spicy or marinated snack (like salted lemon chicken feet or spicy pickled chicken feet).
视频调用
curl --location 'http://127.0.0.1:8000/invoke' \
--header 'Authorization: Bearer U2asJlgLEaUtTMwVmp2iOhKDe_mkhfJu****' \
--header 'Content-Type: application/json' \
--header 'user_id: 123' \
--header 'session_id: 122442' \
--data '{
"prompt": "describe this video, without using any tool",
"files": [" your video url"]
}'
响应示例
Opening (0-2s): A slow-motion close-up of glossy golden marinated chicken feet, with thick amber marinade dripping slowly off the ridged skin onto the white plate below, warm backlighting making the sauce glisten brightly.
3-4s: Cut to a hand slicing a plump bright red chili pepper on a wooden cutting board, releasing its juice and seeds onto the board, highlighting the spicy ingredient in the dish.
5-6s: A chef in a white coat places marinated chicken feet from a ladle onto the heaping plate, pouring the remaining thick yellow marinade over the pile, coating every piece.
7-8s: A close-up shot of the finished piled chicken feet, showing the wrinkled skin coated in sauce with red chili pieces scattered throughout.
9-11s: A person takes a bite of one of the chicken feet, looking satisfied as they chew the tender snack.
12-13s: The plate of chicken feet is shown paired with a frosty glass of beer and a small dish of dried red chilies, framing it as a perfect drinking appetizer.
14-15s: The plate is shown next to a takeout paper bag with a Chinese "tea" character logo, positioning it as a takeaway snack that pairs well with tea.
16-17s: Final close-up with text overlay (with minor typos) describing the snack as crunchy, spicy, and irresistible, with a 2-second countdown graphic.
PDF文件
curl --location 'http://127.0.0.1:8000/invoke' \
--header 'Authorization: Bearer U2asJlgLEaUtTMwVmp2iOhKDe_mkhfJu****' \
--header 'Content-Type: application/json' \
--header 'user_id: 123' \
--header 'session_id: 122442' \
--data '{
"prompt": "describe this PDF main idea, without using any tool",
"files": ["Your PDF FILE URL"]
}'
响应示例
This PDF is a full overview of Creative Agent, an AI-powered end-to-end platform for creating professional quality videos from a single initial concept:
Core Value: It bridges the gap between a creative idea and a finished polished video, offering two workflows: a hands-on storyboard canvas for full manual control (for bespoke campaigns) and an automated UGC one-click workflow for fast, scalable content production.
Step-by-Step Workflow:
Start with a creative brief (prompt, duration, industry templates, reference images) to generate a structured storyboard.
Use an infinite node-based storyline canvas to refine shots: adjust the timeline, work with script nodes, source images, and use a context-aware AI chat panel to refine visuals conversationally, creating variation nodes to iterate on initial drafts.
Convert static image nodes into animated video clips by defining camera motion/effects, then assemble the final cut on the timeline, adjusting pacing and shot order before exporting the finished video.
Underlying AI Framework: The platform uses a proprietary context engineering system that expands simple user prompts into detailed, production-ready video instructions, following a three-stage pipeline:
Structure: Define the narrative blueprint
Dynamics: Add motion and environmental effects
Validation: Harmonize style and finalize optimized prompts
It uses two specialized pipelines to build the story blueprint and engineer cinematic motion, with quality checks at each stage.
Technical Architecture: Built on a modern enterprise stack: Next.js 13 frontend with React/Tailwind, Prisma + MySQL backend, cloud-native containerized deployment, integrating with best-in-class AI services (BytePlus for image/video generation, OpenAI for LLM orchestration, HiAgent for workflow tasks) using an adapter architecture for flexibility.
Final Pitch: Creative Agent acts as an intelligent co-pilot, combining user control with AI consistency to turn creative concepts into finished videos, framing it as augmented creativity rather than just automation.
最近更新时间：2026.01.12 19:27:24
这个页面对您有帮助吗？
有用
有用
无用
无用