Local AI Camera Detection Documentation#

codex resume 01a01163-4a3e-7cd2-b780-30eef5e8f62d Continuation: codex resume 01a01b8f-a60c-7f01-9173-c4c189929d3b

This document describes the working local visual-classification implementation and the information required to recreate it. It is documentation, not a task list.

1. Purpose and Design Principles#

1.1 Purpose#

The system adds a second-stage visual classifier to the existing UniFi Protect and Home Assistant installation.

UniFi Protect remains responsible for recording video and detecting broad event categories such as animal, person, or vehicle. Home Assistant receives the Protect event, obtains an image, sends it to a local vision-capable Ollama model, evaluates the structured result, records the decision, and sends a notification only when the objective-specific policy accepts the result.

The classification objective is deliberately independent from the Protect event category. The same architecture can classify:

• Chickens versus other animals.
• Potentially dangerous animals around the hens.
• Clothing or other visible properties of a person.
• Vehicle type or color.
• A future target such as a coyote.

1.2 Processing Flow#

UniFi Protect broad Smart Detection event
        -> Home Assistant Protect event entity
        -> event and camera validation
        -> event-time camera snapshot
        -> Home Assistant native AI Task
        -> local Ollama vision model
        -> structured classification result
        -> Home Assistant policy and confidence gate
        -> Logbook/system-log decision record
        -> notification only when the policy accepts the result

1.3 Operating Constraints#

The implementation is local and uses the existing infrastructure:

• UniFi UNVR and UniFi Protect cameras remain the NVR and first-stage detector.
• Home Assistant remains the automation, orchestration, policy, logging, and notification layer.
• Ollama remains local at 10.1.2.4:11434.
• No cloud AI API is used.
• No Frigate, replacement NVR, MQTT, Redis, database, Kubernetes, custom object-detection framework, model training, networking change, or hardware purchase is required.
• Protect camera configuration and unrelated Home Assistant automations remain unchanged.

2. Current Environment#

2.1 Home Assistant#

The working installation is Home Assistant OS 2026.8.2.

The Home Assistant UniFi Protect integration is already configured and exposes the Protect cameras and Smart Detection event entities.

The native AI Task entities used by the camera automations are:

ai_task.ollama_qwen3_8_ai_task
ai_task.openai_ai_task

The entity is an orchestration interface. The local model and Ollama endpoint are configured by the Ollama integration rather than as attributes on the AI Task entity. The OpenAI entity is provided by the OpenAI integration and uses the API key configured there.

2.2 Ollama#

The Home Assistant Ollama integration points to:

http://10.1.2.4:11434

The single model used by all current camera AI automations is:

qwen3.8:latest

The model runs on the Mac Studio Ollama endpoint and is kept loaded in memory with keep_alive: -1 for rapid notifications. The model occupies approximately 18 GB of unified memory when loaded.

Home Assistant uses the dedicated AI Task entity ai_task.ollama_qwen3_8_ai_task for all seven camera AI automations. The former Qwen3-VL task is no longer referenced by any automation and has been unloaded from Ollama memory, but its integration configuration is retained for possible future comparison.

The host also contains other vision-capable candidates, including qwen3.8:latest, gemma4:12b, gemma4:26b, gemma4:31b, gemma4:e4b, qwen2.5vl:3b, llama3.2-vision:latest, and minicpm-v:latest.

The current model is known to process the attached camera images successfully. It should not be assumed to be the universally optimal model without an image-for-image comparison using real Protect events. Model choice affects accuracy and latency, but image timing, object size, blur, lighting, and camera framing also have a major effect.

2.3 Cameras and Protect Event Entities#

The current animal and person classifiers listen to these seven Protect Smart Detection event entities:

event.cameras_backyard_camera_g5_bullet_smart_detection
event.cameras_basement_door_camera_g5_bullet_smart_detection
event.cameras_driveway_camera_g5_bullet_smart_detection
event.cameras_front_camera_g5_bullet_smart_detection
event.cameras_garage_camera_g5_bullet_smart_detection
event.cameras_shed_hq_camera_g5_turret_smart_detection
event.cameras_side_camera_g5_bullet_smart_detection

These are event entities whose state changes contain Protect attributes such as event_type, event_id, and the event friendly name. The automations trigger on state changes and then filter the event type in a template condition.

2.4 Protect-to-Camera Mapping#

Protect event entities and Home Assistant camera entities use different entity IDs. The automation must map the triggering event entity to the camera entity that supplies the image.

The mapping must be revalidated in Home Assistant before adding another camera. Do not infer a camera entity ID from its display name.

3. Image Acquisition#

3.1 Event Thumbnail and Camera Snapshot#

UniFi Protect exposes an event-associated thumbnail through Home Assistant. The thumbnail is the preferred image for a notification because it corresponds to the actual Protect event:

/api/unifiprotect/thumbnail/{{ config_entry_id(trigger.entity_id) }}/{{ trigger.to_state.attributes.event_id }}

The native AI Task attachment mechanism does not directly consume this authenticated Protect API path in the current implementation. The AI input therefore uses a camera snapshot captured immediately after the event trigger. Most existing notifications use the Protect event thumbnail; the two animal-target comparison automations now attach the exact snapshot file that their respective model analysed.

This creates two deliberately different image paths:

3.2 Controlled Snapshot#

The snapshot action is:

- action: camera.snapshot
  target:
    entity_id: "{{ camera_entity }}"
  data:
    filename: /media/ai_objective_event.jpg

The filename is overwritten on every execution. It is not an image archive and does not grow with every event.

The AI Task attaches the saved file through the Home Assistant media source:

attachments:
  media_content_id: media-source://media_source/local/ai_objective_event.jpg
  media_content_type: image/jpeg

When two independent classifiers can run for the same event, each classifier uses a different filename. The current animal-related classifiers use:

/media/ai_chicken_event.jpg
/media/ai_non_chicken_ollama_event.jpg
/media/ai_non_chicken_openai_event.jpg

The local Ollama animal classifier attaches and notifies with /media/ai_non_chicken_ollama_event.jpg. The OpenAI comparison classifier attaches and notifies with /media/ai_non_chicken_openai_event.jpg. This prevents the two independent classifiers from overwriting each other’s model input or notification image while processing the same Protect event.

These are still fixed per-automation handoff files, not a permanent archive. Each file is overwritten by the next run of that automation. The current notification therefore matches the model input for that run, but later runs replace the file.

This correction addresses image handoff and concurrent-file races. It does not yet prevent a stale Protect event entity from re-triggering after an unavailable/recovery cycle; event freshness validation remains a separate reliability change.

3.3 Exact Protect Thumbnail as AI Input#

Using the exact Protect thumbnail as model input would require a helper that downloads the authenticated thumbnail, base64-encodes it, and sends it directly to the Ollama API. That is not part of the current native implementation because the controlled camera snapshot is simpler and already works.

The direct Ollama API pattern, if required later, is conceptually:

{
  "model": "qwen3.8:latest",
  "stream": false,
  "messages": [
    {
      "role": "user",
      "content": "Classify the attached camera image.",
      "images": ["BASE64_ENCODED_JPEG"]
    }
  ],
  "format": "JSON_SCHEMA"
}

The native AI Task is preferred while it satisfies the image and structured-output requirements.

4. Protect Event Trigger Contract#

4.1 Trigger#

The automation state trigger contains the applicable Protect event entities:

triggers:
  - trigger: state
    entity_id:
      - event.cameras_backyard_camera_g5_bullet_smart_detection
      - event.cameras_basement_door_camera_g5_bullet_smart_detection
      - event.cameras_driveway_camera_g5_bullet_smart_detection
      - event.cameras_front_camera_g5_bullet_smart_detection
      - event.cameras_garage_camera_g5_bullet_smart_detection
      - event.cameras_shed_hq_camera_g5_turret_smart_detection
      - event.cameras_side_camera_g5_bullet_smart_detection

The state trigger is broad. The condition determines whether the event belongs to the classifier.

4.2 Event-Type Condition#

The minimum validation condition is:

conditions:
  - condition: template
    value_template: >-
      {{ trigger.to_state is not none
         and trigger.to_state.attributes.event_type == 'animal'
         and trigger.to_state.attributes.event_id is defined
         and camera_entity is not none }}

Replace animal with person, vehicle, or another Protect category for a different classifier. Do not trigger the AI classifier on every motion event unless a later design adds an explicit rate limit, because ordinary motion events are too numerous and produce unnecessary inference work.

4.3 Dynamic Camera Variables#

The automation defines a camera map and derives the camera entity and display name from the triggering event:

variables:
  camera_map:
    event.cameras_backyard_camera_g5_bullet_smart_detection: camera.g5_bullet_high_resolution_channel_2
    event.cameras_basement_door_camera_g5_bullet_smart_detection: camera.basement_door_camera_g5_bullet_high_resolution_channel
    event.cameras_driveway_camera_g5_bullet_smart_detection: camera.driveway_camera_g5_bullet_high
    event.cameras_front_camera_g5_bullet_smart_detection: camera.front_camera_g5_bullet_high_resolution_channel
    event.cameras_garage_camera_g5_bullet_smart_detection: camera.g5_bullet_high_resolution_channel
    event.cameras_shed_hq_camera_g5_turret_smart_detection: camera.g5_turret_ultra_high_resolution_channel
    event.cameras_side_camera_g5_bullet_smart_detection: camera.side_camera_g5_bullet_high_resolution_channel
  camera_entity: "{{ camera_map.get(trigger.entity_id) }}"
  camera_name: "{{ trigger.to_state.attributes.friendly_name | replace(' Smart detection', '') }}"

5. Native AI Task and Ollama Interface#

5.1 Native Service#

The model call uses the native Home Assistant action:

- action: ai_task.generate_data
  data:
    entity_id: ai_task.ollama_ai_task
    task_name: objective_detection_all_cameras
    instructions: >-
      Examine the attached event-time camera snapshot. Determine whether the
      requested target is clearly visible. Return only the requested structured
      fields. Do not guess from ambiguous shapes. Confidence must be a numeric
      value from 0.0 to 1.0.
    structure: OBJECTIVE_STRUCTURE
    attachments:
      media_content_id: media-source://media_source/local/ai_objective_event.jpg
      media_content_type: image/jpeg
  response_variable: ai_result

The AI result is read from:

ai_result.get('data', {})

The Ollama entity ID must be supplied in the service data. Putting it in an incorrectly shaped action target was the cause of the first real automation failure.

5.2 Structured Output#

The structured fields are defined in the AI Task request. Free-form prose is not parsed. A generic objective structure is:

structure:
  object_detected:
    description: Whether the broad Protect-detected object is visible.
    required: true
    selector:
      boolean: {}
  target_detected:
    description: Whether the requested target is clearly visible.
    required: true
    selector:
      boolean: {}
  image_clear:
    description: Whether the image is sufficiently clear for the requested classification.
    required: true
    selector:
      boolean: {}
  confidence:
    description: Confidence in the requested classification from 0.0 to 1.0.
    required: true
    selector:
      number:
        min: 0
        max: 1

An optional class field can be added for multi-class objectives:

  species:
    description: Most likely animal species, or unknown when unclear.
    required: true
    selector:
      text: {}

A valid result resembles:

{
  "object_detected": true,
  "target_detected": true,
  "image_clear": true,
  "confidence": 0.88
}

The automation rejects or fails closed when required fields are missing, boolean fields are not boolean, or confidence is outside 0.0–1.0. Values such as 90 or 95 are not converted to percentages.

5.3 Model Selection Considerations#

The current model is qwen3.8:latest on the Mac Studio. The working result proves that the configured model can process images, but not that it is optimal for all surveillance classifications.

The installed comparison candidates are:

General multimodal benchmarks do not directly measure chicken, fox, raccoon, clothing, or surveillance-frame classification. A model comparison should use the same preserved camera images and measure false positives, false negatives, structured-output validity, and response time. The model’s reported confidence is not a calibrated probability.

5.4 Retired Qwen3-VL Structured-Output Requirement#

The former Mac Studio Qwen3-VL task used qwen3-vl:8b. For that model, the Home Assistant Ollama AI Task subentry required:

think: true

When think: false is used, Qwen3-VL can return the JSON classification in Ollama’s thinking field while leaving the normal content field empty. The Ollama API still returns HTTP 200, but Home Assistant cannot parse that response as an AI Task structured result and reports Error with Ollama structured response. With think: true, the model returns the structured JSON in content, where Home Assistant can validate it.

This setting may add reasoning latency, but it is required for reliable structured-output handling with the current Qwen3-VL configuration. A future model or Home Assistant integration update should be retested for this behavior before changing the setting.

The active qwen3.8:latest AI Task is configured with think: false, num_ctx: 8192, max_history: 0, and keep_alive: -1. Every automation must select ai_task.ollama_qwen3_8_ai_task under data.entity_id in ai_task.generate_data.

6. Current Classifier Implementations#

6.1 Chicken Classifier#

The current chicken classifier is:

Entity: automation.ai_camera_chicken_detection_all_cameras
Unique ID: 1787002234196
Name: AI Camera - Chicken Detection All Cameras
Mode: single
Snapshot: /media/ai_chicken_event.jpg
AI task name: chicken_detection_all_cameras
Trigger category: animal

Its structured fields are:

animal_detected: boolean
chicken_detected: boolean
confidence: number from 0.0 to 1.0
image_clear: boolean

Its prompt requires the model to return false when no chicken is visible, when another species is present, or when the image is unclear. It must not guess from ambiguous shapes.

The notification policy is fail-closed:

{{ ai_result is defined
   and ai_result.get('data', {}).get('chicken_detected', false) == true
   and ai_result.get('data', {}).get('image_clear', false) == true
   and (ai_result.get('data', {}).get('confidence', 0) | float(0)) >= 0.75
   and (ai_result.get('data', {}).get('confidence', 0) | float(0)) <= 1.0 }}

The notification title includes the source camera:

title: "Chicken detected — {{ camera_name }}"

The notification image is the matching Protect thumbnail:

data:
  image: >-
    /api/unifiprotect/thumbnail/{{ config_entry_id(trigger.entity_id) }}/{{ trigger.to_state.attributes.event_id }}

6.2 Positive Animal-Target Classifier#

The current animal-target classifier is:

Entity: automation.ai_camera_non_chicken_animal_alert_all_cameras
Unique ID: 1787003552986
Name: AI Camera - Non-Chicken Animal Alert All Cameras
Mode: single
Snapshot: /media/ai_non_chicken_ollama_event.jpg
AI task name: animal_target_detection_all_cameras
Trigger category: animal

Its structured fields are:

species: text
target_animal: boolean
visual_analysis: text
confidence: number from 0.0 to 1.0

The local and OpenAI animal-target classifiers use the same prompt:

Examine the attached snapshot from a UniFi Protect camera event.

Step 1: Analyze the primary animal:
- Determine whether it is bird-like or non-bird.
- Use visible features such as beak, wings, feathers, legs, tail, body shape, and posture.

Step 2: Classify the animal using ONLY this target list:
fox, coyote, raccoon, skunk, mink, weasel, fisher, dog, cat, black bear, crow, deer.

Rules:
- Normalize variants (e.g., red fox -> fox, ermine -> weasel).
- Never classify a bird or feathered silhouette as a mammal.
- If the animal is a chicken, non-target bird, or clearly not on the target list, target_animal is false.
- If the animal is a crow, target_animal is true.
- If the animal is on the target list, target_animal is true.
- If the exact species cannot be determined, return species as "unknown".
- If the exact species is unknown but visible features strongly indicate it belongs to one or more species on the target list, target_animal is true.
- If the image is too ambiguous to determine whether it belongs to the target list, target_animal is false.
- Confidence represents certainty in the target vs. non-target determination and must be between 0.0 and 1.0.

Return ONLY JSON:
{
  "visual_analysis": "Brief description of visible physical features and posture",
  "species": "string",
  "target_animal": boolean,
  "confidence": float
}

The target classifier is independent from the chicken call. It captures its own snapshot and invokes Ollama separately. It does not reuse the chicken classifier’s result.

The classifier uses a positive target list rather than asking the model to identify every animal except a chicken. The current positive targets are:

fox
coyote
raccoon
skunk
mink
weasel
fisher
dog
cat
black bear
crow
deer

Species variants are normalized where practical: red or gray fox becomes fox, and long-tailed weasel or ermine becomes weasel. Deer is included intentionally as a known, frequently visible test animal even though it is not normally dangerous to hens.

The notification policy is positive and recall-oriented. It notifies only when the structured result reports a visible animal, a positive target match, and confidence from 0.50 through 1.0:

ai_result.data.target_animal == true
ai_result.data.confidence >= 0.50
ai_result.data.confidence <= 1.0

Chicken, unknown, excluded, malformed, unavailable, or non-target results do not produce a notification. This is a deliberate change from the former negative non-chicken policy. The positive list is easier for the vision model to evaluate consistently and prevents every unrecognized animal from automatically becoming an alert.

The action uses continue_on_error: true for the AI call. The notification includes the normalized animal type, camera name, confidence, model name, and the same Ollama snapshot file that was supplied to the model. The automation uses single mode and a two-minute end-of-run delay, so a second trigger is ignored while the current run is active.

6.3 Person Black-T-Shirt Validation Classifier#

The person test classifier remains enabled for validation:

Entity: automation.ai_camera_driveway_protect_thumbnail_test
Unique ID: 1786999780223
Name: AI Camera - Person Black T-Shirt Test All Cameras
Mode: queued
Maximum queued runs: 10
Snapshot: /media/ai_black_tshirt_event.jpg
AI task name: black_tshirt_detection_all_cameras
Trigger category: person

Its structured fields are:

person_detected: boolean
black_t_shirt: boolean
confidence: number from 0.0 to 1.0

During the current recall-focused testing phase, notification requires black_t_shirt == true and confidence from 0.50 through 1.0. This is a test classifier and is not the long-term animal-detection objective.

6.4 Vehicle Type and Color Classifier#

The vehicle classifier listens only for Protect vehicle events. It does not analyse ordinary motion events. It captures a full-frame camera snapshot when the event is processed, sends it to the dedicated Qwen3.8 AI Task, and requests only a broad vehicle type and dominant exterior color.

Its structured fields are:

vehicle_detected: boolean
vehicle_type: text
vehicle_color: text
image_clear: boolean
confidence: number from 0.0 to 1.0

Vehicle types are normalized to values such as car, suv, pickup_truck, van, truck, semi_truck, tractor, bus, motorcycle, trailer, other, or unknown. Colors are normalized to common values such as black, white, gray, silver, red, orange, yellow, green, blue, brown, beige, multi_color, or unknown.

The notification includes the normalized type, color, camera name, confidence, model name, and matching Protect thumbnail. The model must return unknown rather than guess when the vehicle is too small, obscured, poorly lit, or otherwise unclear.

The local and OpenAI vehicle prompts retain the full-frame input but include a camera-specific rule for the Garage Camera (G5 Bullet): ignore the known stationary black sedan in the lower-right foreground and prioritize vehicles on the street or farther driveway. If that sedan is the only visible vehicle, the model must return vehicle_detected: false. This is a prompt-level heuristic, not an object crop or Protect bounding box; it may still miss a genuine vehicle event occurring in the parked sedan’s position.

6.5 Per-Automation Cooldown#

Each active camera classifier uses mode: single and ends its action sequence with:

- delay: "00:02:00"

The first qualifying Protect event is processed immediately. While that automation is processing the image or waiting through its two-minute delay, additional qualifying triggers for the same automation are ignored. This prevents repeated Ollama calls and repeated notifications during a burst of Protect detections.

The cooldown is per automation, not global. For example, the chicken and positive non-chicken animal classifiers are independent and may each analyse the same Protect animal event. This is intentional because they answer different classification questions; a shared global cooldown would require a separate helper or coordinator and could suppress one of those analyses.

6.6 OpenAI Comparison Classifier#

For model comparison, a second copy of the positive animal-target classifier uses ai_task.openai_ai_task instead of the local Ollama AI Task. It listens to the same Protect animal events and uses the same target list, schema, confidence policy, camera mapping, and two-minute per-automation cooldown. It uses its own snapshot file:

/media/ai_non_chicken_openai_event.jpg

The two automations are intentionally independent. Both may analyse the same Protect event and write separate decision records. Each notification attaches the exact snapshot used by its own model. The OpenAI notification is labelled with OpenAI GPT-5.6 Luna; the local notification is labelled with qwen3.8:latest. This makes side-by-side testing possible without changing or disabling the local classifier.

The same side-by-side arrangement is also used for the black T-shirt and vehicle type/color classifiers. Their OpenAI copies use separate snapshot files so the local and OpenAI automations cannot overwrite each other’s image while processing the same Protect event:

/media/ai_black_tshirt_openai_event.jpg
/media/ai_vehicle_type_color_openai_event.jpg

These OpenAI test notifications identify GPT-5.6 Luna. The corresponding local notifications continue to identify qwen3.8:latest.

7. Notification and Logging#

7.1 Notification Service#

The current notification action is:

- action: notify.notify
  data:
    title: "OBJECTIVE detected — {{ camera_name }}"
    message: >-
      The local vision classifier detected OBJECTIVE on {{ camera_name }}.
    data:
      image: >-
        /api/unifiprotect/thumbnail/{{ config_entry_id(trigger.entity_id) }}/{{ trigger.to_state.attributes.event_id }}

Protect notifications and AI notifications are independent. Receiving a Protect notification proves only that Protect detected an event; it does not prove that the AI automation ran or that its policy accepted the result.

7.2 System Log and Logbook#

Every AI result should be recorded after the model call and before the notification policy:

- action: system_log.write
  data:
    level: info
    logger: ai_camera.objective_detection
    message: >-
      Classification from {{ camera_name }} for Protect event
      {{ trigger.to_state.attributes.event_id }}:
      {{ ai_result.get('data', {}) | tojson }}

- action: logbook.log
  data:
    name: AI Camera classification
    message: >-
      Classification from {{ camera_name }}:
      {{ ai_result.get('data', {}) | tojson }}
    entity_id: AI_AUTOMATION_ENTITY

The Logbook record is the normal place to compare classifications and investigate false positives or false negatives. Automation traces are useful for immediate debugging but are not a permanent result archive.

7.3 Retention#

The fixed image files in /media are overwritten and therefore do not accumulate one file per event. They do not require deletion for normal operation. The one-day Logbook purge does not delete these files. During a notification, the animal-target automations attach the matching per-model snapshot; after a later run, that file contains the later run’s image.

The AI classification Logbook history is intentionally limited to approximately one day by the automation:

Entity: automation.ai_camera_purge_classification_logs
Unique ID: 1787002827019
Schedule: hourly at minute 0
Service: recorder.purge_entities
Retention: 1 day

The purge scope is limited to the AI classifier automation entities and does not change global Recorder retention. The purge list must be kept synchronized with the active classifier automations.

7.4 Maintaining the Classification Purge List#

Deleting an AI classifier automation does not automatically remove its entity ID from recorder.purge_entities. The stale entity ID is harmless to the purge service, but it should be removed to keep the configuration accurate. The purge automation also needs the entity ID of every active classifier whose Logbook history should be retained for one day.

The active list currently consists of:

automation.ai_camera_driveway_protect_thumbnail_test
automation.ai_camera_chicken_detection_all_cameras
automation.ai_camera_non_chicken_animal_alert_all_cameras
automation.ai_camera_tractor_vehicle_detection_all_cameras
automation.ai_camera_vehicle_type_and_color_all_cameras
automation.ai_camera_non_chicken_animal_alert_all_cameras_openai
automation.ai_camera_person_black_t_shirt_test_all_cameras_openai
automation.ai_camera_vehicle_type_and_color_all_cameras_openai

The deleted White Vehicle and Orange Vehicle automations must not remain in the list:

automation.ai_camera_white_vehicle_detection_all_cameras
automation.ai_camera_orange_vehicle_detection_all_cameras

To correct the list through the Home Assistant UI:

1. Open Settings → Automations & Scenes.
2. Edit AI Camera - Purge Classification Logs.
3. Open its recorder.purge_entities action.
4. Remove the entity IDs of deleted automations.
5. Add the entity IDs of any active classifier automations that should receive one-day Logbook retention.
6. Save the automation. A restart is not required.

This procedure changes future purge scope only. It does not immediately delete old Logbook entries, and it does not delete the fixed snapshot files in /media.

8. Recreating the Implementation#

8.1 Required Existing Components#

Recreation requires:

1. A working UniFi Protect integration in Home Assistant.
2. At least one Protect Smart Detection event entity.
3. A matching Home Assistant camera entity.
4. A reachable local Ollama server with a vision-capable model.
5. The Home Assistant native Ollama integration and AI Task entity.
6. A Home Assistant notification service.

8.2 Configuration Sequence#

The implementation is recreated in this order:

1. Inspect the actual Protect Smart Detection event entities and confirm their event attributes.
2. Inspect the actual camera entity corresponding to the selected Protect camera.
3. Confirm that a Protect event contains an event_type and event_id.
4. Configure or verify the local Ollama integration using the existing private endpoint.
5. Confirm that the AI Task entity is available and accepts image attachments.
6. Create a single-camera automation using the event trigger, event-type condition, camera mapping, snapshot, AI Task call, structured fields, logging, and policy gate.
7. Test the classifier with positive, negative, unclear, and unrelated-object images.
8. Expand the trigger list and camera map only after the single-camera implementation behaves correctly.
9. Add the Protect event thumbnail to the accepted notification.
10. Add or verify the one-day Logbook purge after classification logging is confirmed.

8.3 Generic Automation Skeleton#

The following is a reconstruction reference. In this Home Assistant installation, automations are managed through Home Assistant’s automation configuration rather than by manually editing .storage files.

alias: AI Camera - OBJECTIVE All Cameras
description: Classify Protect OBJECT_TYPE events with the local Ollama vision model.
triggers:
  - trigger: state
    entity_id:
      - PROTECT_EVENT_ENTITY_1
      - PROTECT_EVENT_ENTITY_2
conditions:
  - condition: template
    value_template: >-
      {{ trigger.to_state is not none
         and trigger.to_state.attributes.event_type == 'OBJECT_TYPE'
         and trigger.to_state.attributes.event_id is defined
         and camera_entity is not none }}
actions:
  - action: camera.snapshot
    target:
      entity_id: "{{ camera_entity }}"
    data:
      filename: /media/ai_objective_event.jpg

  - action: ai_task.generate_data
    data:
      entity_id: ai_task.ollama_ai_task
      task_name: objective_detection_all_cameras
      instructions: >-
        Examine the attached event-time camera snapshot. Determine whether
        OBJECTIVE is clearly visible. Return only the requested structured
        fields. Do not guess from ambiguous shapes. Confidence must be a
        numeric value from 0.0 to 1.0.
      structure:
        object_detected:
          description: Whether the broad object is visible.
          required: true
          selector:
            boolean: {}
        target_detected:
          description: Whether OBJECTIVE is clearly visible.
          required: true
          selector:
            boolean: {}
        image_clear:
          description: Whether the image is sufficiently clear to classify.
          required: true
          selector:
            boolean: {}
        confidence:
          description: Confidence from 0.0 to 1.0.
          required: true
          selector:
            number:
              min: 0
              max: 1
      attachments:
        media_content_id: media-source://media_source/local/ai_objective_event.jpg
        media_content_type: image/jpeg
    response_variable: ai_result

  - action: logbook.log
    data:
      name: AI Camera classification
      message: >-
        {{ camera_name }} result:
        {{ ai_result.get('data', {}) | tojson }}
      entity_id: automation.ai_camera_objective_all_cameras

  - choose:
      - conditions:
          - condition: template
            value_template: >-
              {{ ai_result is defined
                 and ai_result.get('data', {}).get('target_detected', false) == true
                 and ai_result.get('data', {}).get('image_clear', false) == true
                 and (ai_result.get('data', {}).get('confidence', 0) | float(0)) >= 0.75
                 and (ai_result.get('data', {}).get('confidence', 0) | float(0)) <= 1.0 }}
        sequence:
          - action: notify.notify
            data:
              title: "OBJECTIVE detected — {{ camera_name }}"
              message: >-
                The local vision classifier detected OBJECTIVE on {{ camera_name }}.
              data:
                image: >-
                  /api/unifiprotect/thumbnail/{{ config_entry_id(trigger.entity_id) }}/{{ trigger.to_state.attributes.event_id }}
mode: single
variables:
  camera_map:
    PROTECT_EVENT_ENTITY_1: CAMERA_ENTITY_1
    PROTECT_EVENT_ENTITY_2: CAMERA_ENTITY_2
  camera_entity: "{{ camera_map.get(trigger.entity_id) }}"
  camera_name: "{{ trigger.to_state.attributes.friendly_name | replace(' Smart detection', '') }}"

8.4 Reusable Objective Changes#

Adding a new classifier changes the following objective-specific values:

• Protect event category, such as animal, person, or vehicle.
• Snapshot filename.
• AI Task name.
• Instructions.
• Structured fields.
• Acceptance policy and confidence threshold.
• Notification title and message.

The following infrastructure remains shared:

• Protect event trigger mechanism.
• Protect-to-camera mapping.
• Event-time snapshot approach.
• Local Ollama endpoint.
• Native AI Task action.
• Structured result validation.
• Logbook/system-log recording.
• Protect thumbnail notification attachment.
• One-day retention policy.

9. Testing and Troubleshooting#

9.1 Test Cases#

A new classifier should be tested with:

• A clear positive example.
• A clear negative example.
• A visually similar negative example.
• An unrelated object.
• A partially obscured or distant target.
• A frame where the Protect event occurred but the target has moved.
• Multiple cameras if the classifier is expanded beyond one camera.

For animal safety monitoring, testing should include chickens, deer, dogs, foxes, raccoons, coyotes if available, people, shadows, and ambiguous dark objects. Verify that listed positive targets notify, while chickens and excluded or unknown animals do not. The 0.50 target threshold intentionally favors recall.

9.2 Where to Investigate a Missed Notification#

Use the following order:

1. Confirm Protect generated the expected Smart Detection event.
2. Confirm the event entity changed state in Home Assistant.
3. Inspect the automation trace for the event timestamp.
4. Confirm the event-type condition passed.
5. Confirm the camera map selected the correct camera.
6. Confirm camera.snapshot completed.
7. Inspect the structured AI result.
8. Evaluate the Home Assistant policy condition.
9. Confirm notify.notify was called.
10. If the service was called but no alert appeared, investigate the notification target or mobile delivery rather than Ollama.

9.3 Important Timing Observation#

An apparent multi-minute notification delay may be the gap between two separate Protect detections rather than AI inference time. The trace must be checked for the Protect event ID and timestamps. In a validated backyard chicken event, the Protect trigger occurred at approximately 18:20:21, the AI result was chicken_detected: true, image_clear: true, confidence 0.95, and Home Assistant called the notification service at approximately 18:20:34.

9.4 Common Failure Modes#

9.5 Logging and Privacy#

The system stores the current classifier snapshots at fixed filenames and retains classification decision history for approximately one day. Protect’s own event/video retention is separate and is controlled by UniFi Protect. The AI automations do not create a permanent image database.

10. Current Configuration Reference#

10.1 Core Automation Entities#

10.2 Current Media Files#

/media/ai_chicken_event.jpg
/media/ai_non_chicken_event.jpg
/media/ai_black_tshirt_event.jpg
/media/ai_black_tshirt_openai_event.jpg
/media/ai_vehicle_type_color_event.jpg
/media/ai_vehicle_type_color_openai_event.jpg

These files are transient working files. Each is overwritten by its corresponding automation.

10.3 Current Policy Summary#

11. Architecture Limitations#

The system is a second-stage classifier, not a dedicated object detector. It does not create bounding boxes, track objects across frames, or recover an object that is absent from the captured snapshot.

Protect may detect an animal in a video sequence while the immediate Home Assistant camera snapshot shows a later or earlier frame. This is the main reason event-thumbnail input could eventually be useful, but the current controlled snapshot is simpler and has been validated.

Protect may also produce only a motion event when its own classifier does not identify an animal. The current design intentionally listens only for Protect animal events because analyzing all motion events would create excessive inference volume and false alarms.

The model confidence is an application field constrained by the schema, not a calibrated probability. Thresholds should be selected from observed test results.

12. Positive-Target Animal Detection#

The current animal-target implementation reuses the existing animal-event architecture. The classifier prompt uses an explicit species list and a positive-match field:

{
  "animal_detected": true,
  "species": "raccoon",
  "target_animal": true,
  "image_clear": true,
  "confidence": 0.91
}

Positive target classification is preferred over a negative class such as “any animal that is not a chicken.” New species can be added to the explicit target list after testing. Unknown and unclear results are logged but do not currently notify; this keeps the automation testable and avoids turning every unrecognized animal into an alert.

The Protect trigger remains event_type == animal; only the AI classification contract and policy change.

13. Operational Review To-Do List#

These items are review points for reliability testing and future refinement. They do not change the current architecture by themselves.

13.1 Event Image Reliability#

• Compare the image analysed by Ollama with the Protect event thumbnail for the same event_id.
• Record cases where Protect detects an animal but the Home Assistant snapshot shows an empty, late, or different frame.
• Evaluate whether the Protect event thumbnail should become the primary Ollama input if snapshot drift causes missed classifications.
• Give the local and OpenAI animal classifiers separate snapshot files and attach each classifier’s own file to its notification.
• If permanent forensic replay is required, replace the fixed handoff files with event-specific filenames plus an explicit cleanup policy.

13.2 Camera Mapping Maintenance#

• Verify the camera_map whenever a camera is added, replaced, renamed, or its stream entity changes.
• Confirm that each Protect event entity resolves to the intended camera entity.
• Test one event from each camera after any camera or Protect integration change.

13.3 Model Confidence Calibration#

• Treat model confidence as an operational score, not a calibrated probability.
• Evaluate thresholds separately for chickens, deer, dogs, foxes, raccoons, and other target animals.
• Preserve representative positive, negative, unclear, and false-positive examples before changing thresholds.

13.4 AI Task Entity Syntax#

• Preserve the Ollama AI Task entity under data.entity_id in ai_task.generate_data.
• Do not move ai_task.ollama_ai_task into a top-level target block; that action shape is invalid.
• If the AI call fails after a future edit, inspect this action structure before investigating the model.
• Verify that notification conditions use the exact field names declared in the AI Task schema. For example, orange_vehicle and orange are different fields; a schema/condition mismatch can silently prevent notifications even when Ollama returns a positive result.

13.5 Snapshot File Safety#

• Give the local and OpenAI animal classifiers separate snapshot files and attach each classifier’s own file to its notification.
• Verify from the saved automation configuration that each model attachment and notification attachment use the same per-automation file.
• If permanent forensic replay is required, replace the fixed handoff files with event-specific filenames plus an explicit cleanup policy.