Knowledge management is in vogue again

Knowledge management is in vogue again. People are talking about company “brains” thanks to AI agents needing them.

The more things change, the more they stay the same.

The portal era

When I worked at Microsoft in the late 1990s, a sister team to the one I was on was building SharePoint Portal Server. It was meant to be a company-wide information portal. Curating essential information was one of the core features, and that part of the product was built around the Dublin Core Metadata standard. The product didn’t really catch on, but some of its features moved into the SharePoint team product that’s widely used today.

Entities and attributes

When I worked at Google, one of my first projects was DAFFIE, which stood for Database of All Fact, Fiction, Information and Exaggeration. The vision was enormous, the team was great, and though we didn’t realize all of it, we did ship the first question answering feature in Google search and used the technology behind it in other products.

The key to DAFFIE was the observation that lots of information can be organized around entities. Since it was the early 2000s, our favorite examples of entities included Britney Spears, Tupac, China and various models of digital camera. The way DAFFIE used entities was to attach attribute-value pairs to them and interconnect them. China has a population (an attribute) of 1.4 billion people (the value). Britney Spears has a height (attribute) of 5’4” (the value). The album Oops!… I Did It Again was recorded by (attribute) Britney Spears (value). And in this case the value is another entity, so there ought to be a connection between those two entities via the recorded by attribute.

Turns out a lot of questions are about attributes for entities. “How many people live in China?” can be answered by looking up the entity China and retrieving its population attribute. “How tall is Britney Spears?” is even more obvious.

Systems of record and their schemas

We populated our database by crawling the web and doing information extraction, but most companies have databases filled with information organized around entities that they maintain by hand. Systems of record like CRMs (HubSpot, Salesforce, Attio), HR information systems (Workday) and inventory management systems (like the massive bespoke one Amazon has built) organize most of their information around entities: customers, employees and products.

Systems of record typically have schemas which specify what attributes an entity can have. They’re often extensible, but the key ones are standard across companies using each type of system: a CRM has a company’s name, primary contact, phone number and addresses; an HRIS has an employee’s name, taxpayer ID and address; an inventory system has a product’s name, price and a unique identifier, such as an ISBN for a book.

The information that doesn’t fit

But not all information fits easily into this mold. If a salesperson talks to a customer and learns that they’re planning to open 200 new stores next year because they’re expanding into a new country, where does that go in the CRM? There’s no attribute for “number of new stores probably being opened next year,” nor should there be, since it’s not relevant to most entries in the CRM and since there are nearly an infinite number of rare or even one-off attributes like this: date the CEO intends to retire, name of the company they’re likely to acquire this quarter, name of a company they may pursue a joint venture with.

As a result, companies need somewhere else to put less structured information. We think entities are still key for this but that the attribute-value pair model breaks down. That’s fine; people and AI agents are pretty good at dealing with text. The reason the attribute-value approach reigned for so long is that AIs as powerful as today’s didn’t exist and software needed structure to operate smoothly.

Schemas get simpler

We believe that the value of schemas will decline in an AI-powered world. Or, put another way, that schemas may become much simpler. We’re working with a design partner to build a system to update their company’s operating “brain.” The example of the salesperson comes in handy to explain this. If the salesperson learns that their contact at the company has changed, they (or their AI agent) needs to update the CRM, since that’s a standard CRM field. But if they hear that the company is opening 200 stores next year or pursuing a joint venture with a partner or moving into an adjacent market segment, there’s nowhere to put that in the CRM except in a free-form notes field. And there’s little advantage to putting it there. Instead, their approach is to have a page about each customer that a human can read and write to and that an agent can also read and update. And, since the list of customers is known, they have an index page listing customers.

Our approach to having agents update the page about each company is based on writing a log of what the agent learned and then updating the page. That way there’s a historical record for debugging purposes and maybe human reference.

Three kinds of pages

Putting this all together, there are three sorts of pages about customers: 1) a list of customers, which links to the page about each customer and the corresponding log, 2) the customer pages and 3) the log pages. We think this model generalizes pretty well. We’re going to use it for partners, competitors, products, employees and more. And we offer it up to anyone else trying to tackle knowledge management in the AI era.

Building in public

To close this out and in the spirit of building in public, we’re working closely with a company to flesh this out. They’re relatively small but very advanced in their thinking about AI. We’re confident we’ll be able to help them build workflows that use all of this company knowledge once we get their company brain to maintain itself. And we believe we can translate our learning into a set of features that make this+that self-serve for small and medium-sized companies.

Nonetheless we wonder whether the platform we’ve built can also serve larger companies with bigger knowledge management and workflow needs. If so, we may need to partner with a consulting firm since an “engagement” may be required to sort out their knowledge management strategy and then identify and build workflows on top of it.

We believe companies are going to grow more effective and that people’s jobs will get more interesting along the way since the mundane tasks, like updating the CRM and sharing notes about customers, will be delegated to AI agents.