The problem
Quarterly and closeout reports from more than 40 Strategic Implementing Partners contained thousands of locally defined themes, indicators, lessons and evidence points. The reports were useful individually, but difficult to compare as one programme evidence base. Similar indicators could have different names, the same indicator name could sometimes point to different measurement logic, and activity, output and outcome measures were often mixed together. The programme needed a way to compare evidence across organisations and funding rounds without stripping away the local context behind each report.
Context
The Social Employment Fund was launched as part of the Presidential Employment Stimulus to support civil society organisations and social enterprises creating part-time work that serves the common good. Strategic Implementing Partners deliver community-based work across areas such as food security, health and care, education, environmental work, community safety, digital inclusion, placemaking, and arts and culture. This case study focuses on the evidence workflow built around historical partner reporting, not on programme management or final programme evaluation.
Before and after
- Before: Four years of quarterly and closeout reporting sat across different report structures, partner-specific themes, partner-specific indicators, quantitative counts, narrative evidence, lessons learned, financial and compliance information, and cross-cutting findings. The reports were not poor quality. They were created for different reporting needs and had not yet been arranged as one cross-programme evidence system.
- After: The material can move through one structured route: messy reports to controlled source register, extraction schemas, repeated extraction and audit, universal themes and indicators, source-linked evidence, structured exports, an AI knowledge layer, and a portal-ready data layer.
Constraints
The work had to standardise evidence without pretending that all partners reported in the same way. Organisation-specific language needed to be preserved, uncertain matches needed review, source counts by theme could not be treated as additive totals, and derived records needed enough locator detail for reviewers to inspect the source instead of accepting a consolidated value without context.
What the team needed
- A shared way to compare partner reports across organisations, themes and funding rounds
- A clear distinction between source-level indicator wording and consolidated universal indicators
- A source-linked evidence register that retained report, page, heading and passage context where available
- A review process for uncertain mappings, duplicate meanings and conflicting indicator logic
- A data layer that could feed databases, dashboards, an AI knowledge base and future reporting outputs
- A cautious route into the theory-of-change phase without claiming final impact before sign-off
If your reports are hard to compare or check, test your source-traceability risk before the next reporting round creates more loose evidence.
What I built
A local evidence-processing application and connected workflow that takes controlled source documents through extraction, review, standardisation and publication into structured databases, an approved-source AI knowledge base, and a linked evidence portal architecture.
Named systems and workflow pieces
- A controlled source register for quarterly and closeout reports in scope
- Defined extraction schemas for themes, subthemes, indicators and supporting evidence
- Repeated AI-assisted extraction passes with comparison across runs
- Audit views for uncertain, conflicting or duplicated records
- Human review points for records that could not be standardised safely
- Universal theme, subtheme and indicator libraries linked to source-level records
- An evidence register with source ID, report name, SIP, funding round, report type, page, heading, passage, extraction run and review status fields where available
- Structured CSV and database exports with data dictionaries
- An approved-source AI knowledge base for retrieval, comparison and first-pass reporting support
- A linked evidence portal and dashboard architecture for SIP, theme and indicator profiles
Where this connects to the services
This case study sits mainly under Traceable Evidence Workflow Support because the source material already existed and needed to become structured, source-linked evidence with theme and indicator mappings. It also connects to Data Use, Reporting & Communication Systems because the same reviewed data layer feeds structured exports, dashboards, an AI knowledge base, portal architecture and future reporting outputs.
One route from partner reports to reusable evidence outputs
The workflow keeps source traceability visible while moving from inconsistent reporting into universal themes, indicators and linked output layers.
Each quarterly or closeout report entered a controlled source register with programme and document metadata.
Theme, indicator and evidence fields were extracted through repeated AI-assisted passes and compared across runs.
Source-level wording was mapped to universal themes and indicators without removing local context.
Derived records retained locators back to report, page, heading and supporting passage where available.
The reviewed data layer can feed exports, dashboards, an AI knowledge base and evidence portal views.
Report to source record to extracted evidence to universal library to structured data, AI knowledge base and portal layer
How it worked
The workflow moved from raw material to usable output through a short sequence of controlled steps.
Process
- 01
Registered the reports in scope so each source document had a controlled identifier and basic metadata.
- 02
Defined extraction schemas for themes, subthemes, source-level indicators and supporting evidence.
- 03
Ran repeated AI-assisted extraction passes, then compared outputs to improve coverage and identify weak records.
- 04
Separated source-level wording from universal themes and indicators so local context was not overwritten by consolidated categories.
- 05
Mapped more than 8000 theme-coded records into 13 universal programme themes, with source counts treated as document coverage rather than additive programme totals.
- 06
Mapped more than 4300 source-level indicator records into about 119 consolidated universal indicators, with uncertain cases held for review.
- 07
Built source locator fields so derived records could point back to the report, page, heading and supporting passage where available.
- 08
Prepared structured exports, knowledge-base inputs and portal-ready data tables from the same reviewed evidence layer.
- 09
Prepared the next phase to develop and test a programme-level theory of change, then map universal indicators to the results chain.
Outputs
These were the named assets, dated deliverables, and working materials left behind by the project.
Working outputs
- Source register for approved reports in scope
- Theme database and 13 universal programme themes
- Subtheme library and source-level theme mappings
- Source-level indicator database
- Consolidated universal indicator library of about 119 indicators
- Evidence register with detailed source locator fields where available
- CSV and database exports with supporting data dictionaries
- Approved-source AI knowledge base for authorised querying and retrieval
- Evidence portal and dashboard architecture for linked SIP, theme and indicator views
- Theory-of-change preparation material for the next phase
Current progress
The first phase has produced a shared theme structure, a first consolidated indicator library and a traceable evidence base covering four reporting rounds. Extraction is being rerun to improve indicator and subtheme coverage before the theory-of-change and publishing phases are completed.
What has been completed so far
- Created one structured evidence base instead of repeated manual searches across four years of reports
- Established consistent cross-SIP categories while retaining source-level wording and context
- Improved retrieval of supporting material for review, reporting and future communication outputs
- Made indicator coverage and reporting gaps easier to inspect without claiming final programme impact
- Created one data layer that can feed structured databases, dashboards, an AI knowledge base and evidence portal views
- Built a foundation for the next phase, where evidence from four programme rounds can support a programme-level theory of change and future reporting design
Large programme reporting backlogs become more useful when standardisation and source traceability are designed together. The point is not to flatten every partner into one generic template, but to compare common concepts while keeping the evidence trail reviewable.
What this proves
- Indicator standardisation across multiple organisations and reporting rounds
- Longitudinal evidence handling across several years of programme reporting
- Source traceability at programme scale
- Reusable data infrastructure rather than a single report
- Linked data feeding structured exports, dashboards, knowledge tools and portal views
- AI-assisted processing inside a controlled review workflow with human checks
- A route from historical reporting into future theory-of-change and reporting design
Best fit
These are the situations where this kind of evidence workflow tends to be the strongest fit.
Who this is best for
- Multi-partner programmes with several years of decentralised reports
- Donor-funded or public-interest programmes where evidence needs to be compared across organisations
- NPO, NGO and programme teams with indicators that cannot be compared cleanly
- Monitoring and evaluation teams that need source-level evidence behind consolidated categories
- Teams building an AI knowledge base around approved reports and structured evidence
- Projects where historical reporting needs to inform future theory-of-change and indicator design
Service stack connected to this case study
This case study sits inside the same delivery work, service logic, and practical outcomes shown across the site.
Turn interviews, submissions, case studies, survey comments, documents, and field notes into coded evidence, quote banks, synthesis tables, findings, recommendations, and report-ready outputs.
Use structured data in reports, dashboards, internal tools, public microsites, applications, presentations, annual reports, and decision-support workflows.
Traceable Evidence Workflow Support
This service packages the same kind of source register, extraction schema, evidence database, review logic and reporting workflow for teams working with large document backlogs.
A route for partner-reporting evidence work
Use this route when programme reports, partner submissions, monitoring records or qualitative evidence need to become a structured, source-linked evidence base that can support review and reporting.
View Traceable Evidence Workflow Support