AI-Pipelines-2020

What happened since 2019?

By admin, August 1, 2022

So a few months after landing a new job in Toronto in 2019 Q3, I got new administrative and analytical tasks in a global position. Working for a worldwide gold-mining company was rewarding albeit really, really fast-paced. When Things started to settle, Covid-19 came on Q1 2020.

Meanwhile, I had not only close contact with ERP users but also got contact with other financial systems and new integration projects. It was rewarding to learn an SAP contender, the Hitach/ABB Ellipse, and integrate it into other industrial and financial systems such as PI-Historian, SAP Concur and Oracle Finance products.

In parallel, the cloud infrastructure providers kept moving at a breaking-neck speed. I collected a couple of images from posts, normally posted on LinkedIn or Meetup groups.

Find the Cloud AI Pipelines post from that moment in time. The below example was provided by Satish Chandra Gupta – @scgupta.

Note that this schema does not mention the OCI – Oracle Cloud Infrastructure, which I got more contact with later.

On 2019-2021, I faced some challenges in using the philosophy and equivalent solutions from AWS to Azure. However, Azure Function and Cloud Function, keys for serverless application development were (and still are) at different maturity points.

At the end of 2021, I got in contact with Google Cloud Platform (GCP) and it is safe to say that the services offered are more or less adequate for usage with AI/BigData when compared to AWS. The price advantage is not really clear (August 2022), since some prices were reviewed and even doubled for cloud storage. Naturally, we are talking about the standard pricing. Some corporates and startups are generally able to cut good deals with the main cloud providers and get the best from each one.

Still, in 2021, the opportunity to watch (remotely) a couple of meetups made clear that knowing how software architecture is defined by the big players, does not mean much for on how would you advise the company you are working for to implement a given pipeline. Below is an extract of how the microservices are implemented in Netflix.

Why would one care about how the architecture implementation on Netflix, beyond curiosity?

The architecture above works well for Netflix. Why would it make sense for a gold-mining company or for a company that provides AI solutions to the mining Supply-Chain sector? It doesn’t.

What to expect from 2022-2024

In 2022, because of the price policy changes on GCP, avoiding vendor-locking solutions is also part of the concern.

I expect to share more on the latest and discuss more what Is being done recently.