What happens between Enter and the reply.
A student message triggers two model calls before the tutor answers and two background jobs after. This is one message, in order.
-
The message is saved with its helpfulness.
A
app/controllers/chats/student_messages_controller.rb, app/jobs/chat_job.rbStudentMessagerow holds the text and the 0 to 100 value. A Turbo Stream appends the bubble and a typing indicator, andChatJobis queued on GoodJob with three retries and exponential backoff. -
A meta-prompt picks the teaching method.
The first model call goes to an "expert prompt engineer" that reads the student's question and walks a decision tree to pick one of five prompt methodologies: RTF, RISE, BAB, TAG, or CARE. It returns JSON with the template and a one-sentence reason.
Prompts.tutor_template_for, app/models/prompts.rb -
The method writes a prompt for this exact question.
The second call hands the question to the chosen methodology, each of which has its own template with a worked example and a good and bad response. It returns a short XML prompt specific to the student's problem.
app/models/prompts/rtf.rb, rise.rb, bab.rb, tag.rb, care.rb -
Hidden system messages go into the transcript.
The methodology prompt and the
ChatResponse#create_by_student_message_id, app/models/chat_response.rb<helpfulness>tag are saved as messages withpermission: "admin". They are part of the conversation the model sees and never part of the one the student sees. -
The tutor replies.
gpt-4o completes the full transcript. The reply is saved with
Message#update_assistant_answer_format!, app/models/message.rbpermission: "user"and the methodology that shaped it, then markdown is detected and rendered through Redcarpet with XSS protection. -
The reply is broadcast and animated.
One Action Cable broadcast on
app/channels/chat_messages_channel.rb, app/javascript/channels/chat_messages_channel.jschat_messages_user_#{id}carries the content withanimation: true. The client findsai_response_for_student_message_#{id}_content, hides the indicator, and types the reply out. -
Both messages are analyzed.
Two
app/jobs/message_analysis_job.rb, app/models/message_analysis.rbMessageAnalysisJobs run. The student's message gets a model call that extracts intent, difficulty, competency, and topic. The tutor's reply records the template, the selection reason, and the helpfulness level that produced it. -
Completion is checked.
app/models/chats/completion_checker.rbChats::CompletionCheckerreads the focus and the last three exchanges and asks whether the task is done. Ifis_completeis true at confidence 0.7 or higher, the focus is marked complete and the header and panel update live.
Five ways to teach, chosen per message.
The tutor has five prompt methodologies, and a model picks one every time the student speaks. The pick is stored on the reply, which is how the analysis page can show that the first algebra message was answered with TAG and the second with BAB.
<role>You are an expert prompt engineer, an AI LLM expert,
educator, teacher, and tutor.</role>
<goal>You must determine the best prompt format to use to best
tutor and educate the student based on the <input> question.</goal>
## Prompt Selection Decision Tree
### Is this an initial interaction requiring role clarity?
Yes → Consider R-T-F or R-I-S-E
### Is this focused on a specific misconception or knowledge gap?
Yes → Consider B-A-B
### Is this about providing feedback on student work?
Yes → Consider T-A-G
### Does this require concrete examples or models?
Yes → Consider C-A-R-E
### Is this about guiding a multi-step process?
Yes → Consider R-I-S-E
No → Combine methodologies as appropriate
# Return Format
Return valid JSON with {prompt:, reason:} where prompt is the
best prompt template, RTF, RISE, BAB, TAG, CARE and reason is a
short description of why you chose that template.
{
"methodology": "tag",
"helpfulness_level": 50,
"template_selection_reason": "Since the student is
seeking the answer to a specific algebra problem without
prior knowledge, the Task-Action-Goal (TAG) format is
appropriate. It allows for clear guidance focused on
solving the quadratic, emphasizing outcome-oriented
assistance while maintaining structure even in a
straightforward query."
}
<methodology><TAG> <task>Assist the student in understanding how to solve quadratic equations like x^2 + 5x + 6 = 0 rather than just providing the answer</task> <action>Guide the student through exploring methods such as factoring, identifying key components of the quadratic, and prompting questions to develop their problem-solving approach</action> <goal>Enable the student to understand the process of solving quadratic equations independently and build confidence in applying algebraic techniques</goal> </TAG></methodology> <helpfulness>50</helpfulness>
Learning objectives, extracted from the first message.
When a chat starts, NewChatJob asks the model for a name, a task type, an expected outcome, completion criteria, and, when the question is a specific assignment, the exact answers or code patterns to look for. That becomes a ChatFocus row with a JSONB data column, so the shape can evolve without a migration. The status moves through not_started, in_progress, stuck, and completed.
Task types map to context: creative becomes essay writing; factual and conceptual become concept understanding; coding, math, and procedural become problem solving.
The job also pushes the new title, the Focus line, and the Problem Details panel to the open page over Turbo Streams, so the header fills in a few seconds after the first message is sent.
The fields from completion_reasoning down were added later, by the completion checker.
{
"task_type": "math",
"is_specific_assignment": true,
"assignment_details": {
"subject": "math",
"specific_task": "Solve quadratic equation x^2 + 5x + 6 = 0"
},
"expected_outcome": "The student should find the roots of the equation.",
"completion_criteria": [
"Correctly apply the quadratic formula or factorization",
"Identify two distinct roots",
"Provide the answer in simplest form"
],
"specific_answers": ["-2", "-3"],
"code_patterns": [],
"source_message_id": 488,
"completion_reasoning": "The student correctly identified the
method of factoring ... The student provided the correct roots
in their simplest form, matching the expected answers.",
"matched_criteria": [
"Correctly apply the quadratic formula or factorization",
"Identify two distinct roots",
"Provide the answer in simplest form"
],
"completion_confidence": 1,
"completed_automatically": true,
"completed_at": "2026-10-22 15:36:47 UTC"
}
What the student meant, scored.
Every student message gets its own model call after the fact. It classifies the request as a solution request, an essay request, problem solving, or a follow-up question, with a confidence for each, and rates difficulty, competency, and topic. Teachers read all of it on the Chat Analysis page, paired with the tutor's own metadata.
{
"request_intent": "solution_request",
"confidence": {
"solution_request": 0.9,
"problem_solving": 0.1,
"essay_request": 0.0,
"follow_up_question": 0.0
},
"difficulty": "easy",
"competency": "poor",
"topic": "math",
"additional_metrics": {
"keywords": ["solve", "equation", "don't know", "answer"],
"sentiment_analysis": "frustrated or overwhelmed",
"estimated_response_length": "short"
},
"explanation": "The student explicitly asks for the answer
to the algebra equation, suggesting they are looking for a
direct solution ... The student's request for the answer
instead of guidance suggests that their current understanding
or competency in the topic may be 'poor'."
}
The Chat Analysis page pairs each student message with the reply and the metadata behind it.
By the fourth message the same student is rated excellent, intent problem_solving at 0.9.
Analysis for the recursion session.
Analysis for the essay session. The first message is scored essay_request at 0.9.
Strict about done.
After every message except the first, the checker gives the model the goal, the criteria, any expected answers, and the last three exchanges, and tells it to be strict. A completion needs is_complete: true and a confidence of at least 0.7. Teachers can also mark a focus complete by hand.
Given a student's learning goal and recent conversation, assess if they've completed their task.
Be strict about completion criteria - only mark as complete if there's clear evidence.
GOAL: Solving a quadratic equation
TASK TYPE: math
EXPECTED OUTCOME: The student should find the roots of the equation.
COMPLETION CRITERIA: ...
SPECIFIC ANSWERS TO LOOK FOR: -2, -3
RECENT CONVERSATION:
Student: ...
Tutor: ...
DETAILED ANALYSIS INSTRUCTIONS:
1. For coding tasks: Check if the student's code matches expected output or contains required patterns
2. For math problems: Check if their answer matches expected solutions
3. For factual questions: Check if they've correctly stated all required facts
4. For procedural/conceptual questions: Check if they've demonstrated understanding
5. Look for evidence of completion in the most recent messages
{ "is_complete": true or false, "confidence": a number between 0 and 1,
"reasoning": "Detailed explanation of your assessment",
"matched_criteria": ["criterion1", "criterion2"] or [] }
The transcript the student never sees.
Every message has a permission, and the chat view renders only user rows. The admin rows are the Socratic method itself: the base prompt, the per-message methodology, and the helpfulness tag. This is the first exchange of the algebra session as the database holds it. The student saw four of these seven rows.
Roles on the user model are user, admin, teacher, and student. Teachers and admins get the Chat Controls slider and the View Analysis button. Admins also get the job dashboard.
GoodJob at /good_job: every ChatJob, MessageAnalysisJob, ChatFocusJob, and NewChatJob from the sessions above.
Teach by asking.
Analyze everything.
Hand back nothing.
Built in Rails by Avi Flombaum, founder of Flatiron School.