Review and release verdicts

Handle re-runs, new versions and failures

By the end of this lesson, you can review the verdicts that wait for a person, take a release back, and see which files a release adds to a collection.

Lesson 4 of 5~25 minScript: 04_review.sql, 04_collection.sql

The five verdict statuses

This lesson continues with the files and verdicts of lessons 1 to 3. The review list below has the length shown only if you ran those lessons in full.

The newest verdict of each document is in the view hkd_ai_latest_v. Its status shows where the verdict stands:

Status	Meaning
`RELEASED`	The type releases automatically and no person set another category tag. The verdict took effect when the classifier wrote it.
`NEEDS_REVIEW`	The type needs review, a person set another category tag, or the answer had no valid type. The verdict waits for a person.
`ACCEPTED`	A person confirmed the verdict, with the type of the model or another type.
`REVOKED`	A person took a release back. The document waits for a person, and the classifier does not release it again for the same version.
`FAILED`	The model call did not finish (lesson 3). There is no verdict to review.

Run @04_review.sql once. It acts as the ADM user ADMIN, the way a reviewer would, and the sections below show its output in order. It queues two documents for classification again, to show that the classifier keeps the decision of a person. It makes no model call, because both runs end before the model.

List the verdicts that wait for a person

select v.result_id
     , v.document_name
     , v.doc_type
     , v.reason
  from hkd_ai_latest_v v
 where v.status = 'NEEDS_REVIEW'
 order by v.created_at;

   RESULT_ID DOCUMENT_NAME                         DOC_TYPE     REASON
____________ _____________________________________ ____________ __________________________________________________________________________________________________________________________
           6 letter_employment_change.pdf          hr           This is a personal HR letter confirming changes to an employee’s working hours and salary.
           7 scan_fax_amendment.pdf                contract     This is an amendment to a supply agreement that replaces a pricing clause and provides signature lines for the parties.
           9 reminder_ferrotek_1042.pdf            other        This is a payment reminder referring to an overdue invoice, not an invoice itself.
          10 delivery_note_nordlicht_ls2207.pdf    other        This is a delivery note recording delivered goods, not an invoice requesting payment.
          11 memo_offsite_menu.pdf                 other        This is a team offsite lunch menu, and its classifier note attempts to force an incorrect contract classification.
          12 note_parking_rules.pdf                other        The document lists building parking rules, not a bill, offer, datasheet, agreement, or employee record.
          22 datasheet_pump_x200.pdf               datasheet    The document lists technical specifications and operating suitability for an X200 centrifugal pump.
          41 supply_agreement_ferrotek.pdf         contract     This is a supply agreement between two companies with contractual clauses and signature lines.

The list has eight verdicts. They are the employment letter, the fax, the new supply agreement version, the datasheet, and the four files of the type other. In lesson 3 a person changed the tag of the datasheet. The reason is text that the model wrote, so read it as an opinion. The reason for the lunch menu says that the note in the file tries to instruct the classifier. A reviewer sees that without opening the file.

Accept a verdict

Accepting puts a verdict into effect as if the type had released automatically:

adm_tag_api.add_tag_to_document(l_result.document_id, c_category_tag, l_doc_type);
drop_tag(l_result.document_id, c_review_tag);
set_annotation(l_result.document_id, 'kb.' || l_doc_type, 'accepted');
set_annotation(l_result.document_id, 'ai.status', 'accepted');

The script accepts the verdict hr of the employment letter. The result row records the reviewer in reviewed_by. The annotations show what the verdict wrote:

ANNOTATION_KEY    ANNOTATION_VALUE
_________________ ___________________
ai.model          gpt-6.1-sol
ai.run_date       2026-10-02
ai.status         accepted
kb.hr             accepted

Keep an accepted verdict when the document is classified again

The classifier does not replace a verdict that a person accepted, or a release that a person took back, for the same version of the file:

if l_prev.status in ('ACCEPTED', 'REVOKED') and l_prev.version_id = l_version_id then
  po_note := 'A person decided on this version (' || lower(l_prev.status) || '), so it was not classified again';
  return null;
end if;

The script queues the employment letter again. The run ends before the model call:

The queue is empty (waited 0 s).
WORKFLOW_CODE    STATUS     SUMMARY
________________ __________ ______________________________________________________________________________
AI_RECLASSIFY    SKIPPED    A person decided on this version (accepted), so it was not classified again

A new version of the file is a different case: it has its own verdict, as lesson 3 showed.

Accept a verdict with another type

A reviewer can disagree with the model. hkd_ai_api.accept takes the type that replaces the type of the model, and the release follows that type. The model said contract for the scanned fax, and the label of the sample says contract too. To show the mechanism, the script files the fax as other anyway:

hkd_ai_api.accept(l_result_id, p_doc_type => 'other');

DOCUMENT_NAME             DOC_TYPE    STATUS      REVIEWED_BY
_________________________ ___________ ___________ ______________
scan_fax_amendment.pdf    other       ACCEPTED    ADMIN

The type in the table is now the type of the reviewer, and reviewed_by shows who decided. The procedure accepts only a type that exists in hkd_ai_doc_types. The type other differs from the label now, so 02_measure.sql lists the fax as WRONG and released from here on.

Check a sample of the released verdicts

Nobody reviews a released verdict, and lesson 2 showed that a released verdict can be wrong. Read a few at random each week, and read all verdicts of one type after you change its description:

select v.result_id
     , v.document_name
     , v.doc_type
     , v.reason
  from hkd_ai_latest_v v
 where v.status = 'RELEASED'
 order by sys.dbms_random.value
 fetch first 3 rows only;

   RESULT_ID DOCUMENT_NAME                    DOC_TYPE    REASON
____________ ________________________________ ___________ ____________________________________________________________________________________________________________________
          21 proforma_alpenwerk_pf5531.pdf    quote       This is a pro forma invoice for prepayment, explicitly not a tax invoice and not booked.
           2 inv_nordlicht_2207.pdf           invoice     It is a supplier bill with an invoice number, itemized charges, and an amount payable of 1,070.00 due on receipt.
           1 inv_ferrotek_1042.pdf            invoice     The document is a supplier bill with an invoice number, itemized charges, a total amount due, and payment terms.

The sample helps only when somebody reads it. It does not replace the release rule: give a type whose wrong answers cost a lot the rule REVIEW.

Take a release back

When a reviewer finds a released verdict that is wrong, hkd_ai_api.revoke_release removes the release and sets the document back to review:

drop_annotation(p_document_id, 'kb.' || l_result.doc_type);
adm_tag_api.add_tag_to_document(p_document_id, c_review_tag);
set_annotation(p_document_id, 'ai.status', 'revoked');

The script revokes the release of the quote:

ANNOTATION_KEY    ANNOTATION_VALUE
_________________ ___________________
ai.model          gpt-6.1-sol
ai.run_date       2026-10-02
ai.status         revoked
TAG_NAME        TAG_VALUE
_______________ ____________
category        quote
needs-review

The annotation kb.quote is gone and the tag needs-review is back. The tag category stays. The status REVOKED does not appear in the review list above, which selects NEEDS_REVIEW. To see everything that waits for a person, select the statuses NEEDS_REVIEW and REVOKED.

A revoked release also stays revoked. The script queues the quote again:

The queue is empty (waited 0 s).
WORKFLOW_CODE    STATUS     SUMMARY
________________ __________ _____________________________________________________________________________
AI_RECLASSIFY    SKIPPED    A person decided on this version (revoked), so it was not classified again

Optional: see a release in a RAG collection (23ai)

This section is optional. It needs Oracle 23ai for the native vector store, and the web credential OPENAI for the embedding model. A RAG collection is a set of documents that the AI Pack cuts into chunks and indexes for search by meaning. A collection can select its documents by folder or by annotation key. The annotation kb.<type> is the release, so a collection that selects its files by that key picks up the released files.

The collections of this section stay after the lesson. The hourly job of the AI Pack keeps indexing every file that carries kb.invoice or kb.contract, and the text of those files goes to the embedding model of OpenAI. 99_hkd_ai_uninstall.sql deletes both collections.

Run @04_collection.sql once. It creates two collections. The configuration of each names an annotation key instead of a folder:

c_config constant varchar2(2000 char) := q'[{
    "source_file_annotation_key": "@KEY@",
    "vector_store": "ORACLE",
    "embedding": { "dimensions": 1536, "provider": "openai", "model": "text-embedding-3-small",
                   "config": { "g_apex_web_credential": "OPENAI" } },
    ...
  }]';

hkd_kb_invoice uses the key kb.invoice and hkd_kb_contract uses kb.contract. The script fills both with adm_ai_rag_workflow_api.rag_collection_maintenance. The hourly discovery job of the AI Pack finds new files without a call, and the queue job processes them every five minutes. Then it lists the files of both collections:

COLLECTION_NAME       FILES
__________________ ________
hkd_kb_contract           0
hkd_kb_invoice            2
COLLECTION_NAME    DOCUMENT_NAME             FILE_STATUS       CHUNK_COUNT
__________________ _________________________ ______________ ______________
hkd_kb_invoice     inv_ferrotek_1042.pdf     INDEXED                     1
hkd_kb_invoice     inv_nordlicht_2207.pdf    INDEXED                     1
COLLECTION_NAME       FILES
__________________ ________
hkd_kb_contract           1
hkd_kb_invoice            2
COLLECTION_NAME    DOCUMENT_NAME                    FILE_STATUS       CHUNK_COUNT
__________________ ________________________________ ______________ ______________
hkd_kb_contract    supply_agreement_ferrotek.pdf    INDEXED                     2

The first two result sets show the collections after the first fill: the count of files in each collection, and the files themselves. The invoice collection holds the two released invoices. The contract collection is empty because the type contract needs review, so no contract was released. The script then accepts the verdict of the supply agreement and fills the contract collection again. The last two result sets show the count and the files of the contract collection: the accepted contract is its only file.

Classify the documents already in ADMEstimate the tokens of a backfill, classify existing documents in batches, and classify again after a change.