OParl Bundle#

Indexes the data of a council information system into the search index via its OParl API: meetings, papers, organizations, persons, agenda items and files.

The bundle provides one indexer per OParl type on top of the index bundle. The data is fetched with sitepark/oparl-client and written to the Solr index of the search bundle.

Sources#

The sources can be accessed via the GitHub project https://github.com/sitepark/atoolo-oparl-bundle.

Installation#

First add the Sitepark Flex Repository before installing the bundle.

See: Sitepark Flex Repository

composer require atoolo/oparl-bundle

The bundle and the index bundle have to be registered in config/bundles.php:

return [
    // ...
    Atoolo\Index\AtooloIndexBundle::class => ['all' => true],
    Atoolo\Oparl\AtooloOparlBundle::class => ['all' => true],
];

Indexers#

Each indexer fetches the list of its type from every configured body of the OParl system. Every indexer has its own status, schedule and cleanup, so a project only switches on the types it needs.

CMS side configuration#

An indexer is only active if the CMS provides its configuration configs/indexer/<source>.php - see CMS side configuration. Without the file a scheduled run is skipped. This is usually done via a configuration article in the CMS: an object type of the customer module in the group Indexer of the Module Administration, which writes the file.

<?php
// configs/indexer/oparl-meeting.php
return [
    'name' => 'Ratsinformationssystem - Sitzungen',
    'data' => [
        'systemUrl' => 'https://ris.example.org/oparl/v1.1',
        'bodyUrls' => ['https://ris.example.org/oparl/v1.1/body/1'],
        'cleanupThreshold' => 10,
        'chunkSize' => 500,
        'fullSyncInterval' => 24,
        'modifiedSinceOverlap' => 300,
        'omitInternal' => false,
        'group' => 1234,
        'groupPath' => [1, 12, 1234],
        'category' => [5678],
        'categoryPath' => [56, 5678],
    ],
];

Further keys are passed on to the enricher and filter of a project as $parameter->data.

Scheduling#

The indexers are scheduled through the scheduler of the index bundle:

parameters:
  atoolo_index.indexer.schedules:
    oparl-meeting: "15 * * * *"
    oparl-paper: "30 * * * *"

The schedule is processed by the worker. An indexer can also be started manually, see Indexing:

bin/console index:indexer --source oparl-meeting

Full and delta runs#

The lists of an OParl API are paginated and can have hundreds of pages, a complete run can take several minutes. A run is therefore either

• a full run, fetching every object. Afterwards it deletes every document of the source that it did not write, if it wrote at least cleanupThreshold documents - a server that returns an empty list by mistake must not empty the index.
• a delta run, fetching only the objects modified since the last successful run (modified_since). Objects the server reports as deleted are removed from the index, everything else stays untouched.

A full run is done if the last one is longer ago than fullSyncInterval hours. Only a run that completed and whose updates all succeeded is remembered. After an error, a failed update or an abortion the next run starts over from the same point, and a full run deletes nothing then.

Mapping#

The documents are filled by enricher tagged atoolo_oparl.indexer.document_enricher.schema2x. The common fields are set with priority 100, the fields of each type with priority 50.

A paper additionally sets its paper type as sp_meta_string_kicker.

The bundle sets neither sp_id nor sp_objecttype: both refer to resources of the CMS. description is not set either. It is used as teaser text, but OParl offers no such text for most objects - a project that wants one sets it in an enricher of its own.

Custom Document Enricher#

A project adds or overwrites fields with an enricher of its own and a lower priority. The options of the CMS side configuration are available in $parameter->data.

declare(strict_types=1);

namespace App\Indexer;

use Atoolo\Index\Service\Indexer\IndexDocument;
use Atoolo\Oparl\Dto\Indexer\OparlIndexerParameter;
use Atoolo\Oparl\Service\Indexer\OparlDocumentEnricher;
use Atoolo\Search\Service\Indexer\IndexSchema2xDocument;
use SP\OparlClient\V1\Objects\OparlObjectV1;
use SP\OparlClient\V1\Objects\OparlPaper;

/**
 * @implements OparlDocumentEnricher<IndexSchema2xDocument>
 */
class PaperKickerEnricher implements OparlDocumentEnricher
{
    public function enrichDocument(
        OparlObjectV1 $object,
        OparlIndexerParameter $parameter,
        IndexDocument $doc,
        string $processId,
    ): IndexDocument {
        if (
            !$object instanceof OparlPaper
            || !$doc instanceof IndexSchema2xDocument
        ) {
            return $doc;
        }

        $doc->setMetaString('kicker', 'Drucksache ' . $object->getReference());

        return $doc;
    }
}

services:
  App\Indexer\PaperKickerEnricher:
    tags:
      - { name: "atoolo_oparl.indexer.document_enricher.schema2x", priority: 10 }

The enricher of the bundle only use what the object itself contains and resolve no references - the names of the organizations of a meeting would cost a request each. An enricher that needs referenced objects resolves them with the OparlReferenceResolver (service atoolo_oparl.reference_resolver), which caches them.

Filter#

Which objects are indexed is decided by the service atoolo_oparl.indexer.object_filter, an OparlObjectFilter. By default every object is accepted. A project replaces the service to leave objects out. An object that is not accepted is removed from the index, should it have been indexed before. The filter is shared by all OParl indexers, $parameter->source tells which one calls it.

declare(strict_types=1);

namespace App\Indexer;

use Atoolo\Oparl\Dto\Indexer\OparlIndexerParameter;
use Atoolo\Oparl\Service\Indexer\OparlObjectFilter;
use SP\OparlClient\V1\Objects\OparlFile;
use SP\OparlClient\V1\Objects\OparlObjectV1;

/**
 * Only objects with a page to link to.
 */
class WebPageFilter implements OparlObjectFilter
{
    public function accept(
        OparlObjectV1 $object,
        OparlIndexerParameter $parameter,
    ): bool {
        if ($object instanceof OparlFile) {
            return $object->getAccessUrl() !== null;
        }
        return $object->getWeb() !== null;
    }
}

services:
  atoolo_oparl.indexer.object_filter:
    class: App\Indexer\WebPageFilter

Cache#

The state of the delta runs and the objects resolved with the OparlReferenceResolver are kept in the cache pool atoolo_oparl.cache. By default this is a filesystem cache in var/oparl, outside of the kernel cache directory, so that cache:clear keeps it. Losing it is harmless, the next run is a full run.

bin/console cache:pool:clear atoolo_oparl.cache

The pool is configured like any other cache pool, e.g. with Redis to share it between several servers:

config/packages/cache.yaml

parameters:
  redis_host: "%env(REDIS_HOST)%"
  env(REDIS_HOST): localhost

framework:
  cache:
    default_redis_provider: "redis://%redis_host%"
    pools:
      atoolo_oparl.cache:
        adapter: cache.adapter.redis

Parameters#

The parameters apply to all OParl indexers. Without a proxy option, the proxy is taken from the environment (HTTPS_PROXY, NO_PROXY).