# At production scale

> What a production reinsurance analytics system handles that the 20-trial demo does not: trial and row counts, loss-set multiplicity, currency, reproducibility.

Source: https://reinsurance.dev/applications/production-scale/

Each story in this chapter ran its pipeline on the [20-trial demo tier](https://reinsurance.dev/toolkit/trials/#demo-scale): the [post-model report](https://reinsurance.dev/applications/risk-profiling/) on two loss sets, the [program price](https://reinsurance.dev/applications/standalone-pricing/) on three layers, the [portfolio roll-up](https://reinsurance.dev/applications/portfolio-analytics/) on six contracts. The pipelines are the real workflow in miniature. What changes at production scale is the same for all three, and is the subject here. Each story's page covers the one consideration specific to it: the [row arithmetic](https://reinsurance.dev/applications/risk-profiling/#at-production-scale) of a loss set, the [evaluation order](https://reinsurance.dev/applications/standalone-pricing/#at-production-scale) a program imposes, the [trial alignment](https://reinsurance.dev/applications/portfolio-analytics/#at-production-scale) a roll-up depends on.

## Trials and rows

Production loss sets run [10,000 to 1,000,000+ trials](https://reinsurance.dev/toolkit/trials/#demo-scale), and the row count of a year event loss table ([YELT](https://reinsurance.dev/glossary/#yelt-year-event-loss-table)) is the trial count times the occurrences per trial times the rows per occurrence, one for each geography and line of business the event touches. A single cedent's loss set can reach 10⁹ rows; a portfolio holds tens of thousands of loss sets.

| Axis | Demo tier | Production | What it changes |
| --- | ---: | ---: | --- |
| Trials per loss set | 20 | 10,000 to 1,000,000+ | Resolution: which return periods a tail quote can be based on |
| Rows per loss set | 1,657 (SunCoast) | 10⁵ to 10⁹ | Whether the loss set fits in memory, and where the `groupby` runs |
| Loss sets in play | 2 | tens of thousands | Bookkeeping: which loss sets entered which result |
| Contracts in the portfolio | 6 | tens of thousands | Evaluation order, and the cost of the roll-up |
| Currencies | 1 (\$M throughout) | several | A conversion to the reporting currency before losses in different currencies are added |

The arithmetic does not change shape. Every metric on the site is a `groupby` plus one sort, $O(\text{rows})$ to reduce a YELT to trial totals with a hash grouping ([pandas sorts its keys by default](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.groupby.html)) and $O(N \log N)$ to rank them, and every financial term is a filter, a clip, or a running sum in claims order over the same rows, the last of which is the one term that sorts. At 10⁷ rows on the demo schema, [pandas](https://pandas.pydata.org/) runs the whole pipeline in memory in [seconds](https://reinsurance.dev/applications/risk-profiling/#at-production-scale). At 10⁹ rows on one loss set, in-memory pandas is no longer the right tool: columnar storage, chunked or out-of-core aggregation, and pushing the `groupby` into a query engine are what change. The extra trials add resolution, not speed: the 1-in-200 and 1-in-500 quotes that capital decisions need and 20 trials cannot resolve. The canonical statement of that gap, and the 1,000-trial tier that makes it runnable, is the [Demo scale](https://reinsurance.dev/toolkit/trials/#demo-scale) section of the trial worldview.

Every demo YELT is in one currency, stated in millions of US dollars, so losses add directly: within a loss set, across a program, and across the portfolio. A production portfolio written across the US, Europe, and Japan carries losses in several currencies, and each contract's YELT is converted to the reporting currency before losses in different currencies are added;[^currency] at which rates, and with what allowance for their movement, is a treasury and accounting question this site treats as outside the analytics, as the [conventions index](https://reinsurance.dev/conventions/) records.

A table of 10⁹ rows is tractable because [the work partitions by trial](https://reinsurance.dev/applications/production-scale/#one-computational-shape).

## One computational shape

All three pipelines have the same shape at scale, because [contract terms apply to each trial independently](https://reinsurance.dev/financial-modelling/dissection/#one-trial-at-a-time): nothing a contract does reaches across trials. Partition the YELT by trial, evaluate every term on every partition, and reduce the per-trial results into the distribution the metrics are computed from.

```mermaid
graph TB
    Y[YELT · N trials]:::Losses --> P1[Trials 1 … n]:::Losses
    Y --> P2[Trials n+1 … 2n]:::Losses
    Y --> P3[…]:::Losses
    P1 --> E1(Evaluate terms<br/>filter · period · excess):::Process
    P2 --> E2(Evaluate terms):::Process
    P3 --> E3(Evaluate terms):::Process
    E1 --> R(Reduce<br/>sum per trial · sort · rank):::Process
    E2 --> R
    E3 --> R
    R --> M[EP curve · EL · VaR · TVaR]:::Metric

    classDef Losses fill:#F5CEBC,stroke:#B16C4B,stroke-width:1.5px,color:#2B2620
    classDef Process fill:#F7CBCB,stroke:#B46669,stroke-width:1.5px,color:#2B2620
    classDef Metric fill:#BDE2CB,stroke:#459269,stroke-width:1.5px,color:#2B2620
```

The evaluate stage is the expensive one, and it is embarrassingly parallel: partitions share no state, so they run on as many workers as there are partitions. The reduce stage is one sort over $N$ trial totals. A [program with an inuring edge](https://reinsurance.dev/applications/standalone-pricing/#at-production-scale) imposes an order between contracts (C3 cannot be evaluated until C1 and C2 have been) but never between trials, so the partition holds inside each stage. A [roll-up](https://reinsurance.dev/applications/portfolio-analytics/#at-production-scale) sums within a trial across contracts, so it matches partitions by `trial_id`, and the partition key becomes a join key. That join is valid only when every contract's loss set comes from one simulation and one trial numbering, where equal trial ids name the same simulated future; equal ids from two simulations name unrelated futures, which is why a portfolio view is built on one simulation basis. Each partition reads named loss sets, and [a reinsurer tracks tens of thousands of them, across several such bases](https://reinsurance.dev/applications/production-scale/#loss-set-multiplicity).

## Loss-set multiplicity

```mermaid
graph LR
    S[One submission]:::Losses --> V1["Vendor A · v2025<br/>baseline climate"]:::Process
    S --> V2["Vendor A · v2026<br/>baseline climate"]:::Process
    S --> V3["Vendor B · v2025<br/>warmed climate"]:::Process
    V1 --> L1[HU loss set]:::Losses
    V1 --> L2[EQ loss set]:::Losses
    V2 --> L3[HU loss set]:::Losses
    V3 --> L4[HU loss set]:::Losses
    L1 --> P1["Portfolio view on basis A · v2025<br/>which loss sets entered it?"]:::Metric
    L2 --> P1
    L3 --> P2["Portfolio view on basis A · v2026"]:::Metric
    L4 --> P3["Portfolio view on basis B · warmed"]:::Metric

    classDef Losses fill:#F5CEBC,stroke:#B16C4B,stroke-width:1.5px,color:#2B2620
    classDef Process fill:#F7CBCB,stroke:#B46669,stroke-width:1.5px,color:#2B2620
    classDef Metric fill:#BDE2CB,stroke:#459269,stroke-width:1.5px,color:#2B2620
```

Two loss sets entered the post-model report, for one cedent's submission; a reinsurer's portfolio has far more. Take a reinsurer writing several thousand programs across its cedents, each program covering several combinations of peril and line of business and modelled across [multiple vendors and model versions](https://reinsurance.dev/foundations/the-market/#model-proliferation) and climate assumptions. That is tens of thousands of loss sets to track across the portfolio, before any are combined. Reconciliation stops being a sanity check and becomes the central step of the workflow. The step leaves every loss set validated the way [risk profiling validates one at ingest](https://reinsurance.dev/applications/risk-profiling/#step-1-validate-and-reconcile-the-loss-sets), every combination accounted for, and the "which loss sets entered this result?" question answered precisely. That makes loss sets named, versioned inputs, not files on a share. And it is combining them into a portfolio view, not profiling any one, where performance becomes the constraint. Named, versioned inputs are also what lets [a result be regenerated](https://reinsurance.dev/applications/production-scale/#reproducibility) when any one of them changes.

The vendors, versions and climate assumptions are alternative views of the same risk, and in the diagram each is its own simulation, with its own trials and event catalog. Alternative views are compared side by side or blended; they are never added, because adding two views of one risk counts that risk twice. A reinsurer that keeps three views keeps three portfolio views. The trial-identity precondition matters in a different case: different risks modelled on different bases, such as one vendor's hurricane loss sets and another's earthquake loss sets, or a basis that pairs a v2026 hurricane model with a v2025 earthquake model, which [portfolio analytics](https://reinsurance.dev/applications/portfolio-analytics/#at-production-scale) covers.

## Reproducibility

The chapter's [standing principle](https://reinsurance.dev/applications/#design-principle-reproducibility) applies to every story without modification: each result is a pure function of its inputs, and the argument list is short enough to write down.

| Result | Pure function of | Regenerates when |
| --- | --- | --- |
| Post-model report | (loss sets, parameters) | A vendor ships a model update and the loss sets regenerate |
| Technical premium | (subject loss, program structure, pricing parameters) | The cost of capital or the structure changes: the whole portfolio reprices |
| Portfolio distribution and capital | (the contracts, their participations, the YELT) | A participation changes: the whole portfolio reprices |

Same inputs, same result, to the cent.[^float] When the vendor ships a model update and the loss sets regenerate, every result regenerates with them automatically, not by an analyst re-running notebook cells in the right order. The moment a result is hand-assembled, two versions of it can disagree, and the reader is back to guessing which one to trust.

[^float]: "To the cent" takes care: with monetary values as floating point, the order of
    summation affects the result, so a reproducible pipeline fixes its reduction order, or uses
    fixed-point or decimal arithmetic. David Goldberg,
    ["What Every Computer Scientist Should Know About Floating-Point Arithmetic"](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html),
    *ACM Computing Surveys* 23(1), 1991: "Due to roundoff errors, the associative laws of algebra
    do not necessarily hold for floating-point numbers", and $(x+y)+z$ differs from $x+(y+z)$ at
    $x = 10^{30}$, $y = -10^{30}$, $z = 1$.

:::note[Consider]
Think of the engine behind these stories as a single pure function with three arguments: losses, structure, policy. Standalone pricing is that function evaluated once, for one program. The portfolio roll-up is the same function over a base portfolio; marginal pricing is the same function evaluated twice, with and without the new contract, and differenced. Building the function once, cleanly, is what makes the rest of the chapter a matter of calling it differently rather than rewriting it.
:::

The application that this shape is built for is **marginal pricing with portfolio view** *(planned)*: a portfolio whose roll-up is kept current as a live object, so that the second evaluation, with one more contract, is as cheap as the first. The portfolio it would price against is the one the [portfolio analytics story](https://reinsurance.dev/applications/portfolio-analytics/) rolls up.

[^currency]: Moody's RMS, ["Platform Jobs"](https://developer.rms.com/platform/docs/platform-jobs), Intelligent Risk Platform documentation, which schedules currency conversion as its own job over each object a roll-up consumes: `CONVERT_ACCOUNT_CURRENCY`, `CONVERT_PORTFOLIO_CURRENCY`, `CONVERT_AGGREGATE_PORTFOLIO_CURRENCY`, `CONVERT_RESULT_CURRENCY`, and `CONVERT_TREATY_CURRENCY`, with rates read from a scheme that maps each currency to an anchor. The platform makes conversion a separate step; that it precedes the sum follows from the sum itself.