Skip to content

Agent

examples/agent.ipynb builds two AI agents on the PhysiCar AI Services — chat (text) and realtime (voice) — sharing one brain: the instructions and the eight robot tools defined in examples/assets/agent/web/prompt.js. Ask them to drive, look around, read the LiDAR, or play music, and you're watching LLM tool calling — the loop behind every AI agent — run against your own robot. The logic lives in the browser; the notebook's Python only serves the pages.

Inside the notebook

Section What happens
0. The brain the INSTRUCTIONS and the 8 tools — drive, look, sleep, camera, lidar, states, music_search, music_player — explained. This is the part you edit
1. Chat the POST /chat protocol — the prompt sent with every request, streamed answers, the tool-call loop — then a small messenger UI in the MYAPP tab
2. Realtime the realtime voice protocol — your mic streams up, voice and tool calls stream down — then the voice UI

Run it

  1. Open examples/agent.ipynb and pick the Python 3 (PhysiCar AI) kernel.
  2. Run section 0, then the run cell of the agent you want — chat (1.2) or realtime (2.2).
  3. Open app.physicar and sign in on the App page — the agent runs on your account. Switch to the MYAPP tab; for realtime, allow the microphone when the browser asks.
  4. Talk (or type): "Drive forward", "What do you see?", "Anything around me?", "Play some music".

Every robot-facing tool calls the robot's local Web API, so the same agents run unchanged in the simulator and on a real kit (music_search is the exception — it queries the iTunes search API on the internet).

Make it yours

INSTRUCTIONS is the agent's persona — it's why it drives gently at 0.5 m/s and stops the moment you say stop. Edit it in the notebook, rerun the cell, reload the MYAPP tab.

Each entry in TOOLS is a description, parameters, and a run() function. The model reads the descriptions to decide when to call a tool — writing them well is the programming. Copy an entry's shape to add a ninth tool; the model discovers it automatically. Things to try:

  • A new personality — a pirate, a racing coach, another language.
  • See how sleep turns single commands into timed sequences: "drive 2 seconds then stop" becomes drive → sleep → drive.
  • The camera tool returns a photo into the conversation — that's how "what do you see?" works.

The PHYSICAR AI chat panel in VS Code speaks exactly this chat API — after this notebook, you know what happens inside it.

Watch out

  • The agents bill your credits — chat per turn, realtime per session. Don't leave a voice session open chatting to itself.
  • Open the app through the MYAPP tab (/myapp/), not port 5000 directly — only there do sign-in injection and robot API routing exist.
  • The mic is live from page load; the button is a mute toggle. If you hear no voice, click the page once (browser autoplay policy).
  • Tools take degrees, the Web API wants radians — run() converts. Keep the units straight in your own tools.

🛠 Mission: give the agent a new personality in INSTRUCTIONS, then add a ninth tool — a dance move built from drive and sleep is a good first one.

Learn more

How agents call tools → Agent & tool calling. The models behind it → AI & LLMs. The chat protocol in full → Chat API spec.

AI