DAE Search

When the analyst searches for "revenue" they are given 40 results; the actual work then begins, which involves twenty minutes spent deciding which table is the current one, which one belongs to the team that is responsible, and which had been deprecated three quarters ago and was never cleaned up.

That isn't really searching; it's just opening up tabs. You click on a table, identify the owner, check when it last refreshed, look for anything else that's still reading from it, then return and carry out the same process for the next item, and if you're being careful you'll do this forty times, even though no one actually does. The analysts had a lot of results. The only thing they lacked was one that they could claim as their own.

The kind of data shown in these screenshots relates to the life sciences: specifically, therapeutic areas, registries, trial sets and claims sets. If you replace 'revenue' with 'diabetes' then nothing about the problem is affected, which is the reason why I am at ease in using the finance example.

The reason the list contains forty items is due to keyword matching; when someone types "revenue" they get back all the entries that have the word in the column name, in the description or in a stale comment and these are ranked according to their string similarity to a word which no one has named their tables after.

The search therefore proceeds semantically through the metadata, and when you type "revenue" the table named `arr_monthly` is returned since the model knows what annual recurring revenue is and a simple string match never could have.

It reduced the result set from 40 items to about 4 for selection, but it didn't deal with the underlying question since four of the candidates still required a decision and the decision was still being made in the other tabs.

Therefore I transferred them to the row. The origin of the table and the time at which it last refreshed, likewise, are all visible in the list rather than requiring you to click through to find this information.

It seems to be just a minor change to the layout and yet involves the entire product. The real question that an analyst is asking is "which of these can I show to a VP on Thursday", and it is freshness and lineage that provide the answer, whereas a relevance score cannot.

It also alters the cost of having a bad result, since under the previous system you only discovered that a table was out of date after you had already started to use it, sometimes even days later, whereas now you know about it before you open any row.

They remain locked and can be given access with just one click, though they still stay visible. I initially tried to hide them since that seems to be the straightforward option.

It had the opposite effect; people decided that the data did not exist and set about rebuilding it themselves, which is more dangerous from a security point of view than simply showing them a padlock, since there are now two copies of the sensitive data and only one of them is protected.

I developed an autocomplete feature that is aware of permissions. If you type in three letters, it will provide suggestions based on everything in the catalogue, whether or not that thing is restricted, on the basis that it's harmless to know that a name exists.

It is by no means harmless; the suggestions revealed the existence of restricted datasets to anyone who typed in a sufficient portion of the name so that it could be matched by pattern, which is essentially a slower way of just looking at the list of tables. The security team picked up on it during their review and I didn't contest this.

Here's a clean ranked list with as little chrome as possible and get out of the way—targeting the wrong audience.

What the people want is density; they prefer having as many signals as possible in each row since it's faster to scan ten dense rows than it is to open three clean ones, and each bit of white space that I had added cost them a click.

They also never form complete questions; instead they enter fragments and use internal abbreviations since they already know what they want and are not having a conversation with the search box. "Diabtes", with the spelling error, is in fact a real query from the screenshots.

It came with its own style guide, and therefore the swatches shown here are taken directly from it rather than being reconstructed. There are nine named colours divided into three groups, each group serving a different function: one primary group which handles all the actions, a second group which handles state, and the neutrals which are responsible for separating the result rows.

What's interesting is the use of the neutrals; it is thanks to mint, sage and khaki that a dense row can appear fresh even without a legend or an extra column, that being the only reason why the row can show lineage at all.

The nature of semantic search, with lineage and permissions included, has changed. It is no longer just a search box but has become the place you go to find out whether data is worth using, which within a company has always been the question people have been asking, albeit in other tabs and in a slow manner.

It was shipped with its own style guide, and the swatches were pulled from that guide rather than reconstructed for this page. The autocomplete underwent a security review during which the design was altered before it was released, a kind of review that only takes place when a thing is actually being launched.

What I can't tell you is what happened after. I don't have adoption or time-saved numbers for this one, and I'd rather say so than reach for a number from somewhere else and let you assume it came from here.