Testing Gen AI. Collaborative. Adaptive.

import os

from rhesis.sdk.synthesizers.multi_turn import MultiTurnSynthesizer, GenerationConfig
from rhesis.penelope import PenelopeAgent, EndpointTarget

# Set the environment variables
os.environ["RHESIS_BASE_URL"] = "https://api.rhesis.ai"
os.environ["RHESIS_API_KEY"] = "rh-XXXXXXXXXXXXXXXXXXXX"

# Step 1: Generate multi-turn test case
config = GenerationConfig(
    generation_prompt="Generate a test for an AI travel guide planning complex multi-city trips"
)

synthesizer = MultiTurnSynthesizer(config=config)
test_set = synthesizer.generate(num_tests=1)
test = test_set.tests[0]

# Step 2: Execute with Penelope
agent = PenelopeAgent()
target = EndpointTarget(endpoint_id="travel-guide-prod")

result = agent.execute_test(
    target=target,
    goal=test['test_configuration']['goal'],
    instructions=test['test_configuration']['instructions'],
    restrictions=test['test_configuration']['restrictions'],
    scenario=test['test_configuration']['scenario']
)

# Step 3: Access results
print(f"Goal Achieved: {result.goal_achieved}")
print(f"Turns: {result.turns_used}")
print(f"Duration: {result.duration_seconds:.2f}s")

for metric_name, metric_data in result.metrics.items():
    print(f"{metric_name}: {metric_data['score']}")

Generated Test Case:
----------------------------------------
Goal: Successfully plan a 10-day Japan itinerary accommodating wheelchair accessibility, dietary restrictions (vegetarian and kosher), and budget constraints

Instructions: Start by asking about family composition and specific needs. Then progressively build the itinerary city-by-city, adapting recommendations based on responses. Verify accessibility details and dietary options for each suggestion.

Restrictions: Must stay within ¥200,000 per person total budget. All recommended venues must be wheelchair accessible with verified accessibility information. Must not suggest generic advice without specific venue names.

Scenario: You are planning a family trip to Japan. Your family consists of 4 people: yourself, your spouse, your 16-year-old vegetarian son, and your 72-year-old father who uses a wheelchair and requires kosher meals.

Behavior: Reliability | Category: Harmless | Topic: Complex Multi-City Planning


Executing test with Penelope...
----------------------------------------
Turn 1:
  Penelope → Hi! We're planning a 10-day family trip to Japan. We're 4 people - my father uses a wheelchair, my son is vegetarian, and my father needs kosher meals. Can you help us plan an accessible itinerary?
  Target → I'd be happy to help! Japan has excellent accessibility in major cities. To create the best itinerary, can you share your budget range and which cities interest you most?

Turn 2:
  Penelope → Our budget is about ¥200,000 per person total. We'd like to see Tokyo, Kyoto, and maybe Osaka. What would you recommend?
  Target → Perfect! With that budget, I recommend: Tokyo (4 days), Kyoto (3 days), Osaka (2 days). All three cities have excellent accessibility. Let's start with Tokyo - I'll suggest wheelchair-accessible hotels and restaurants with vegetarian options...

Turn 3:
  Penelope → That sounds great! For Tokyo, what specific hotels can accommodate wheelchairs? And where can we find kosher food for my father?
  Target → For Tokyo, I recommend Hotel Gracery Shinjuku (fully accessible, ¥18,000/night for 2 rooms). For kosher food, visit Chabad House Tokyo in Hiroo - they have kosher meals and can cater. They're located at 3-9-10 Minami-Azabu.

Turn 4:
  Penelope → Excellent! What about vegetarian restaurants in Tokyo that are also wheelchair accessible?
  Target → Great options include: Ain Soph Journey (100% vegan, wheelchair accessible, Shinjuku), Nagi Shokudo (organic vegetarian, accessible, Shibuya), and T's Tantan (vegan ramen, Tokyo Station, fully accessible).

Turn 5:
  Penelope → Perfect! Now for Kyoto - what accessible attractions would you recommend for 3 days?
  Target → For Kyoto, top accessible sites: Fushimi Inari (lower shrine area accessible), Kinkaku-ji Temple (paved paths, wheelchairs available), Arashiyama Bamboo Grove (accessible pathway), Kyoto Station Building (fully accessible with great views).

Turn 6:
  Penelope → How do we get between cities with a wheelchair? Is the Shinkansen accessible?
  Target → Yes! The Shinkansen has designated wheelchair spaces. Book accessible seats when reserving (車椅子席 - kurumaisu-seki). Staff will assist with boarding ramps. Tokyo→Kyoto: ¥13,320/person. Kyoto→Osaka: ¥1,420/person.

Turn 7:
  Penelope → What's the total estimated cost so far with accommodations and transport?
  Target → Current estimate per person: Hotels ~¥75,000 (10 nights), Transport ~¥20,000, Food ~¥60,000, Attractions ~¥15,000 = ~¥170,000/person. Well within your ¥200,000 budget with ¥30,000 buffer for shopping/extras!

Turn 8:
  Penelope → This is perfect! Can you summarize the complete itinerary with all the accessible venues?
  Target → Absolutely! Here's your complete 10-day accessible Japan itinerary:

    Tokyo (Days 1-4): Hotel Gracery Shinjuku, Chabad House for kosher meals, Ain Soph Journey for vegetarian dining
    Kyoto (Days 5-7): Accessible temples (Kinkaku-ji, Fushimi Inari lower areas), Bamboo Grove
    Osaka (Days 8-9): Osaka Castle (accessible routes), Dotonbori (wheelchair-friendly paths)
    Return Tokyo (Day 10): Departure

    All venues verified wheelchair accessible. Budget: ¥170,000/person. Transportation includes wheelchair spaces on all trains.


Test Results:
----------------------------------------
Goal Achieved: ✓ True
Turns Used: 8
Duration: 24.3s

Metrics:
  Goal Achievement: 0.95 (5/5 criteria met)
  Context Maintenance: 0.88
  Response Quality: 0.92

Goal Evaluation:
  ✓ Created complete 10-day itinerary
  ✓ All venues wheelchair accessible with verification
  ✓ Stayed within ¥200,000/person budget (¥170,000 used)
  ✓ Accommodated both dietary restrictions (vegetarian + kosher)
  ✓ Maintained context across all turns

Key Findings:
  • Successfully maintained all family constraints across 8 conversation turns
  • Provided specific venue names with addresses and accessibility details
  • Adapted recommendations progressively based on budget and dietary needs
  • Demonstrated strong context retention (referenced wheelchair needs in turns 3, 6, 8)
  • Budget tracking maintained throughout (verified in turn 7)