Mar
25
In the history of software, as I have suffered it in the past 45 years, the most time wasting difficulty has been the false dawn syndrome. My first CTO, Norman Nunn Price was a grizzled Welshman with an unquenchable enthusiasm for the ability of software to solve all problems. As a young man, he had worked on radar in submarines in the Second World War. When, as his more youthful CEO, I sometimes questioned his predictions, the reply often included “look, we won the bloody war using this stuff didn’t we? “. But as the years passed by in our development of a start-up in legal information retrieval, we began to notice that when Norman and his team announced that the job was done, or the fix was in place, or the application was ready and the assignment was completed, we were actually at the beginning of another work phase, and not at a point of implementation. Once, in frustration, I pointed out forcibly to Norman that, despite his optimistic announcement that he had once more brought us successfully to a moon landing, it appeared that I was still 50 feet above the surface with no available mechanism to get me down there. It became a company saying.
I find myself using it regularly as I listen to the way in which data and analytics companies are learning to live with AI. I cannot fault the ambition. it is clear that many service providers are framing solutions that are going to provide really dramatic advances in value to the widest possible range of societal requirements. But once the service design and the value added has been determined, we come to that familiar place which the software engineers describe in terms of ETL – the whole business of extracting, transforming and loading data. it is here that we discover that our data is not like other data. It is too structured, or not structured at all. It has been marked up in a way that makes it difficult to transform, or it hasnot been marked up at all, which makes it difficult to transform. It either lacks metadata to guide the process, or has too much metadata, or nobody can understand and use the metadata. So we must pause and create a solution.
This is a well trodden track. Others have gone before us. The problems about integrating data into cloud data services like Databricks and Snowflake have slowed progress and added to costs for the past five years. It is interesting to see that the small industry has grown up to ease a problem, with companies like prophecy.com, emerging with effective solutions. One might imagine that the same will happen with AI. Data transformation will cease in time to be an issue, since a raft of services will have emerged to deal with common problems, and the data creators will have reacted and adapted to the issues that arise when data is ingested into AI environment of all sorts.
But of course, this will not stop the press releases, which will continue to claim that something has happened some time before it might possibly happen. Yet, it should moderate our expectations a little bit. Many feel that we have not yet hit the problems of getting first generation AI services fully operational, even if we are talking as if we were rolling out, second generation services, tried and tested by legions of users. 50 feet above the moon can be a good place to be if it provides an opportunity to pause for thought, and realign our thinking before we make the slow eventual descent to the lunar surface.