Nobody writes blog posts about field mappings. We do.

Field notes on the layer nobody demos: getting records out of SAP, MES and PLM in a form a retrieval system can actually use. This hub collects the extraction paths, the masking of personal data inside the pipeline, and the permission model that has to survive retrieval for an answer to pass a security review.

Posts on the data foundation

12 posts, newest first

  1. source systems

    CRM sales questions an agent may ask the pipeline

    CRM sales questions with a documented read: open deals, the owner, the next date. Stage changes, sends and deletes are not questions. They are edits.

    5 min read · Phillip Pham

  2. source systems

    DATEV API: what an agent may read, and may not post

    The DATEV API splits in two: DATEVconnect on the workstation for account postings, and the cloud gateway for accounting sequences. An agent may read, not post.

    6 min read · Phillip Pham

  3. source systems

    Utility billing integration: what an agent may read

    Utility billing integration for a Stadtwerk: the agent reads a contract on Schleupen.CS under the caller's login. Market messages stay in the billing system.

    9 min read · Phillip Pham

  4. source systems

    Energy data management: what an agent may not send

    Energy data management at a Stadtwerk sits in BelVis, Robotron or SOPTIM. An agent may read a schedule status. Sending that schedule stays inside the system.

    8 min read · Phillip Pham

  5. Permission-Aware Retrieval: Test the File Share Next

    The demo passed on curated files. Permission-aware retrieval fails when the file share has no permission metadata on each chunk. Test that before the next pilot.

    12 min read · Noel Dinger · updated 29 Sept 2026

What this area covers

The layer everyone skips and then rebuilds.

This area is about making enterprise data reachable: reading SAP, MES, PLM, SharePoint and thirty-year-old SQL schemas, working out which field is actually authoritative when four of them disagree, masking personal data before anything is indexed, and carrying the real permission model through to retrieval.

It is written for the people who will do that work — data engineers, integration architects, and the one person in the company who still understands the custom Z-tables.

The argument running through this area is that AI projects rarely fail on the model. They fail because a demo was built on a curated folder of two hundred clean documents, and the production corpus turned out to be four million files with three naming conventions, no revision discipline and an access-control list that nobody has audited since 2019. Fixing that is not preparation for the project. It is most of the project.

Common questions

Why does the data layer decide whether an AI project ships?

Because the model is the interchangeable part. Retrieval quality, field-level authority and entitlements determine whether an answer can be trusted, and none of those improve when you swap in a better model. A pilot that skips this layer works on a curated folder and stops working on the real corpus.

Do we need a data warehouse before any of this works?

Usually not. A warehouse is optimised for aggregates, and most AI use cases need the specific record with its context and its permissions attached. We generally read from the source system or a replica and build a retrieval layer beside the warehouse rather than waiting for one.

How to start

Not a sales call. An architecture call.

Thirty minutes with the architect who would actually run the engagement.