What we’ve learnt about AI in 4 years of building ForestreetPosted November 10, 2021
Recently, a tweet from a former Capgemini colleague, Steve Jones, got me thinking.
Firstly, I whole-heartedly support his statement that “AI…will be based on multiple approaches”. That is entirely accurate in my experience, both pre-Forestreet and implementing AI (and ML) solutions for clients for the 10+ years prior.
Let me explain.
What we’ve built at Forestreet
In the last year, we’ve built a solution to find any company in any market in real time. We’re not building pre-trained lists or forcing a market to be our definition (or, worse, a general industry defintion) of a market. So if a client wants to answer the question “which SaaS onboarding tools are best for me?” their definitions of “onboarding tools” and “best” will be unique to them.
This poses a problem for AI market intelligence.
It’s also one of the reasons that Google isn’t the answer for professionals: Not only does it take a huge amount of time, but most of the results are SEO-optimised blogs, thought pieces, lists of lists or ideas about how to find a company.
Neither are lists like Capterra that still require a lot of weeding through, and forces the Gartner/Capterra view of market definitions.
Instead, our clients in procurement, M&A, PE, solution architecture just want to find every company and quickly, in less than an hour, review every one of them and be able to present a strong opinion on the market and what to do next.
Why we require an ensemble approach
To deliver on that use case, our AI requires a huge amount of flexibility to create not a single overarching model that pre-trains one model that says, “these are what Onboarding Tool companies look like” because that introduces the huge bias of that training set. Instead, we are essentially creating a new set of models on the fly for each search request and returning the results in sub 30 seconds.
Of course, this flexibility — essentially building a better professional search engine — comes with constraints and compromises. That’s why when Steve wrote his tweet that it resonated so strongly.
In addition to the “multiple approach” approach, I’ve outlined a few other key learnings from the Forestreet journey.
Key learnings from our 4 years of production-grade AI
NLP Ensemble approach
Even the “best” NLP models in the world (I’m looking at you GPT 2/3) still only have limited application for approaches like ours. We have to understand full website texts, not bits of text like a tweet or chat-bot response.
An ensemble approach is absolutely fundamental. We cannot do what we do without a number of different models working together. I’ve yet to see anything more than a lovely PoC that is built on a single model (AI or ML). The real “AI companies” who are building world-class products would (and should) all admit that they use a host of models working together to get to a product. We’re not building models; we’re building value.
The value of R&D with automation
Our IP is based on what we tried and know what doesn’t work more than what we have in our stack. I estimate that 4 out of 5 models and approaches didn’t yield the necessary results; that probably means we have 80 approaches on the cutting-room floor and 20 in production
Machine/human/machine automation is key. Anyone who is looking for silver bullets where you take training data and get great results is bonkers. Building fast automation tools into the AI/ML workflows to remove minutes, seconds from normal workflows is perhaps even more important than building the perfect AI model. (I knew this before we started with my work in data science for 5+ years prior to Forestreet and it’s influenced everything we’ve done).
The long journey
AI platforms have come a LONG way in 3 years. The ability to get a model from training into production has made massive strides forward. In particular, the offerings from AWS and Azure for their platforms now make it many days easier to get from prototype into prod. In 2018, we were somewhat avantgarde in our selection of KNIME as our data science (and engineering) platform, but it was the best in the market for what it does at the time, although the big guys have gotten closer and will almost certainly pass KNIME in 2022, if they haven’t already around certain use cases.
Rabbit holes are everywhere. There’s always a new and better model to train and you can lose weeks exploring the next silver bullet. I’ve learnt the lessons, the hard way, that there are no silver bullets and that once you have a good working set of models, to move onto the next problem on the roadmap once it’s GEFN (good enough for now). Building a big commercial product requires a suite of GEFNs working together and, unlike probability where .75 x .75 = .56 it’s more like 75% GEFN x 75% GEFN = 85% or something. Keep adding these to your process, and you’re to that AI magical serendipity and an amazing product.
AI vs ML vs RPA vs Automation
The bucket of things called “AI” is a wide range of AI, ML, business rules, automation, human training and other company’s for-sale APIs that are likely based on similar collection of “AI”. There’s been a lot made of “AI companies” that don’t have much AI. I wonder if there’s going to be some sort of threshold saying that “to call yourself an AI company, you need 50% of all models to be neural-net based”. But that would allow for a company that has a single AI model and not much else to call themselves AI, despite it not really working. The companies who are really creating commercial products don’t hold themselves to a standard that says, “everything we do must be AI”. Instead, all of us say: let’s find the right tool for the job.
This is genuinely fun stuff. Since I’ve been demoing our customer-facing ‘Pro’ product over the past month I’ve gotten to see how well this ensemble approach works. If you want to know more, get in touch for a demo. The proof is when you see it.