Our recent participation in the Intel Ignite Program in Munich marked a pivotal moment in our company's trajectory. What began as an opportunity to connect with fellow European Deeptech companies evolved into a transformative experience that reshaped our approach to business in unexpected ways.

From the outset, including the extensive selection process, the program radiated with high-energy, attracting us in due to the level of innovation we could see which was present - This was something I would not have previously associated Intel with - 

High Energy Environments, Startups & Innovation

. Although I can't comment about Intel at large, Intel Ignite is a place that has these in abundance. 

One of the most remarkable outcomes of our Intel Ignite journey was the evolution of our day-to-day operations. Initially we had a set of goals and outcomes we wanted out of the program, but it was actually the structure of this goal & objective setting system that became the biggest value creation for us. When it comes to mastering the art of Objectives and Key Results (OKRs), who better to learn from than the pioneers themselves? Intel. Maybe this should not have come as a surprise but with Intel's insights into the power of OKRs in driving organizational alignment and performance, along with daily/weekly/monthly alignment checks, we really changed how we operate. 

Implementing OKRs and adopting the Level 10 Meeting framework from the 

 were transformative steps that brought clarity, alignment, and accountability to our team. Personally, I know leaving our weekly Level 10 Meeting, exactly what I should and should not be worried about / focused on. 

The combination of OKRs and Level 10 Meetings worked seamlessly for us, allowing us to set ambitious objectives while ensuring that everyone in the organization was aligned and accountable for their contributions. The Level 10 Meetings, with their focus on problem-solving and accountability, provided the perfect platform for us to track our progress towards our OKRs and address any obstacles in real-time. We have even implemented a second Level 10 meeting associated with our Engineering team. The key for me is that the level 10 meeting happens every week, with the same people, the same agenda, and at the same time. Consistency is what makes it work. 

However, beyond the strategic frameworks and business insights gained, the other true value of our Intel Ignite experience lay in the connections forged within the cohort. The camaraderie and support shared among our peers was awesome. Through discussions, problem-solving, and encouragement, we discovered the power of community in propelling us towards our goals. It was super insightful to see the common struggles amongst the teams as they tried to build their solutions. It was within this environment that we figured out how far we had truly come, and to take on the mindset of, 

"of course there will be issues to solve each day, we are working on a monumental problem!

". This mindset was the motivation to push the boundaries of what we thought possible.

Special shout out to all the companies involved: 

 – Pioneering the construction of stellarator fusion plants

 – Advanced testing and QA framework tailored for AI systems

 – Quantum sensing revolutionizing semiconductor technology

 – Industry-leading data processing and compression at massive scales

 – Innovative impulse printing of 3D interconnects for vertical chips

 – Cutting-edge high-density edge AI inference chip technology

 – High-speed, multispectral analysis technology for comprehensive detection

 – Integrating AI seamlessly into database operations

 - The conflict free sync layer for local apps. 

Finally, the mentors in the program were fantastic, and I know they will continue to guide Zerve in what we build from here on out, to name the core team: Markus Bohl, Martha Ivanovas, Alois Eder & Kate Hach. Rockstar support structure that we are truly grateful for. If any Deeptech founder is considering the program, feel free to dm me for my thoughts. 

Amongst Great Compan(ies)

When we started building Zerve, we had a couple of perspectives on the space that we learned both through our own experience and through hundreds of interviews that we performed over the course of about a year. First, the tools available to data scientists force them to choose between stability and flexibility; i.e., exploring data in VS Code is a nightmare, and trying to deploy a notebook is practically delusional. Second, the only people generating value from data today are doing it with code. In other words, 100% of AI projects require code in one way or another.

That’s why we built Zerve, and that’s why we made it free for our users.

We launched on January 30 of this year and in the first week had over 1000s of people sign up to use Zerve. We got tons of great feedback directly and on social media, and that let us know that we are on the right track. Here are some things that we’ve learned along the way.

Everyone that signed up for Zerve got a direct email from me because we really want to understand their experience with Zerve and their needs in tooling. Going into the launch we thought we knew our users very well. We had run a small scale beta test earlier in 2023, and learned a lot through that experience, so this time we were looking for confirmation of what we’ve built.

We’ve gotten lots of great feedback. Things like, “Zerve AI is like Miro for developers, except cooler. 😎” and “It’s like the Google Colab upgrade we’ve all been waiting for.” These comments are particularly awesome as they would have been part of pitches we have given about our vision, but now we were seeing people we had never spoken to, see the problem and Zerve just like we did.

We have long since called the state of tooling in the data science space the “Data Science Stockholm Syndrome.” It definitely seems like people are waking up to the limitations of the tools. Since the seminal talk, “

,” posted in October of 2018 highlighting some of these issues, people have begun to realize that better tools are needed to do serious data science. Our launch further validated this.


Self-hosting is an absolutely critical part of data science

Scores of data science tools have been built over the last decade in the cloud, and it always surprised me when they would stutter and stumble when asked about self-hosting. At the end of the day, nobody wants their data to have to leave their cloud environment and go to a third party to be analyzed or processed. Even more, vendors shouldn’t want this either. It creates all sorts of problems when it comes to data security and working with larger and more sophisticated organizations.

That’s why we built Zerve to be self-hosted from the beginning. We’re on the AWS marketplace, and a full self installation takes less than 10 minutes.

The feedback that we got from the launch (on our cloud environment) proved to us that people are very hesitant to send their data out to a third party, so we’ve had lots of conversations about getting people up and running in their own environments so that they can use Zerve for real projects on real data.

We also often hear that offering Zerve locally is important to the community. Zerve is, however, unlikely to be offering this. It’s not to diminish that this is important to some folks, for us it’s not how we see we’ll be able to elevate the impact of data science. For lots of reasons, but most crucially true collaboration, whether within the team or with stakeholders like Data Engineers - the benefits we have built are centred around streamlining the process of going from Prototype to Production using foundational cloud technologies to make handover, or direct deployment infinitely more straightforward.

Of course there have been issues, but the community has a real sense of wanting to like what we are doing, and the patience shown to us to get things resolved has been really awesome. Additionally the community has quickly pointed out gaps in what we are doing. Some we knew about, and some new ones. It’s been insanely valuable to meet all of these people, understand their ‘why’ and look to add the desired functionality when it aligns with our vision.

Early on, I thought that generative AI was largely hype from a business perspective, but I was completely wrong about that. Nearly everyone that we speak to is working on some sort of LLM or other generative AI project. Discovering and tapping into the capabilities of these powerful new models is a fundamental leap forward.

The bad news is that the tooling for fine-tuning and rolling out these models is very challenging. From the dev/ops issues related to utilizing GPUs and the costs associated with compute, many organizations have stumbled out the gate on these projects. Because Zerve provides easy, targeted, serveless GPU support, we resolve both of these issues pretty seamlessly.

That’s why next month we’ll be launching a host of new features including integrations with 

, GPU support, and more. These features make Zerve incredibly well-positioned to be THE IDE for Gen AI.

Stay tuned for details of that launch on our website and our 

Lots done, a lot more to do. 1 month after launching, our analysis on how that went and what we learned.

A month after launching, here's what we've learned

The inception of interactive computational notebooks can be traced back to the 1980s with the development of Maple, Mathematica, and SageMath. These early versions were predominantly academic tools, designed to facilitate computational teaching and learning through a mix of code, visualizations, and narrative text. Mathematica's notebook interface inspired a new way of interactive computing, setting the stage for future developments. Fernando Pérez, deeply influenced by his experiences with Mathematica, initiated the development of IPython in 2001, laying the groundwork for what would become a significant shift in data science tools.

By 2007, the IPython team embarked on creating a notebook system, integrating text, calculations, and visualizations. This effort culminated in the release of the first version in 2011. The transformation of IPython into Project Jupyter in 2014 marked a pivotal moment, splitting the notebook functionality into a separate project. Jupyter's single kernel system design allowed it to serve a broad spectrum of scientific and research applications. Early data scientists, transitioning from academia, adopted Jupyter for its flexibility and interactive capabilities, despite its limitations for production environments.

The evolution of Jupyter notebooks was significantly influenced by the open-source movement, with pivotal contributions like SciPy and Matplotlib laying the foundation for Python's scientific computing capabilities. These developments, coupled with Jupyter's ability to share and collaborate on python projects easily, highlighted its utility in academic and professional settings. However, the growing demand within the corporate world for projects to go beyond the exploratory limitations of a notebook meant that, in order to attempt to get their work into production, data scientists needed to move into some sort of IDE once they had completed their exploratory data analysis inside of a notebook. This often involved handing their work over to another data practitioner, such as a data engineer or MLops engineer, who would likely need to re-code the work in an IDE. The origins of this problem can be seen in the 

 and data scientists coming from academia into the corporate world brought this reproducibility problem with them by bringing in the tooling they already knew.

Following this evolution within the corporate world, a new era of notebooks began to take shape. Cloud-based platforms such as Google Colabemerged, building upon Jupyter's foundational, single kernel architecture. These platforms attempt to modernise the data science development journey, with enhanced collaborative and interactive features. However, they are still in essence forks of the original Jupyter notebook and therefore pose the same problems when it comes to deployment.

What has become clear is that as the field of data science has matured, the limitations of notebooks have become apparent. They are excellent for exploration and visualization but less suited for more extensive software development or production-level code. This led to a gap between the exploratory work done in notebooks and the development work done in traditional Integrated Development Environments (IDEs). Why this gap is more important today than ever before is because the world is demanding that AI and models make it into production. This gap is where Zerve, a unified developer space for data science & AI, was born. Zerve blends the exploratory flexibility of notebooks with the robustness and scalability of IDEs, enabled by its novel underlying architecture.

The Subtle Integration of Advanced Features

With this novel architecture comes new benefits like language interoperability, parallelization capabilities, and cloud compute optimization. Such features represent the natural progression of data science development, addressing the growing demands for efficiency and collaboration in the field.

Looking Ahead: The Future of Data Science Environments

The progression of data science environments will be shaped by the burgeoning need for organizations to harness AI, including LLMs and generative AI. This necessitates an efficient development process, where data science and AI teams can minimize their time to value and optimize the path from ideation to deployment. The evolution toward environments that integrate these advanced capabilities more deeply, support seamless collaboration, and enhance both development and deployment stages is crucial in meeting the escalating demands for leveraging AI technologies effectively.

An analysis of the history of notebooks, how they entered into the corporate world and a look to the future 

The Evolution of Data Science Environments: From Jupyter to the Future

Today we’re excited to be unveiling something totally new in the world of data science and AI development. We’ve been building in stealth for the last two years, and we're finally opening the door to let the world in. 

My two co-founders Greg & Jason, two PhD level statisticians, are two of the smartest people I know. They have tales of not being able to get their organization's AI initiatives over the line, or crazy inefficiencies in the process while doing so. They have lived the problems. Teams separately working on local notebooks and meeting twice a day to share findings, and emailing / zipping the latest data to other team members. Inefficient, and not fit for purpose in a world that will demand impact from Data Science teams like never before.

We’ve spent 100s of hours interviewing ‘Code-First Data Users’ who not only confirmed the efficiency losses we knew, but shedded light on even more. We heard all too often that the Data Scientist's impact was set up to be distinctly 

, a screenshot of a Matplotlib chart on a PPT, an uninspiring Data App, or a handover that resulted in a complete re-coding of the original work.

We believe to truly unlock the opportunities in AI, LLMs and Data Science over the next decade, expert Data Scientists will be the difference makers, and we set out to build a platform that puts Data Scientists at the heart of the innovation.

The trend over the last decade has been moving towards low-code/no-code solutions for developing data science. Software vendors have developed a dizzying variety of tools that supposedly allow anyone to train sophisticated models and to interact with their data without needing to write code. It has become obvious to anyone who has actually ever deployed a Data Science project though, that these tools are wildly inflexible and therefore entirely unsuitable for doing anything serious. That’s because 100% of data science projects require code, and that’s why Zerve is built for code first data teams. The introduction of Large Language Models presents the opportunity to help build data solutions even further, however our thesis remains the same, these tools need to be used in addition to the knowledge of a code-first data user who is building the solution. We believe these tools will lead to even more code-first data teams, not less.

The coding toolkit for data, though, is problematic to say the least. Data engineers might be working in snowflake or DBT or some other tool where they write SQL. Passing the data along for analysis, the data scientists typically start in a notebook environment, like Jupyter. The problem with notebooks is that they are brittle. Jupyter was developed by academics to be used as a classroom scratch pad. Jupyter is wonderful for exploring data, but it’s undeployable - meaning that if you’re working on something serious, you’ll eventually need to move to an IDE, like PyCharm or VSCode at some point.

Because of the architecture of notebooks, it’s very easy to get them into a bad state. In the example below, by running the cells of the notebook too many times, it’s easy to get into a state where 1+1=7. That’s why every modern notebook has a “restart kernel” button. Because they all operate with a global variable space that makes production work with a notebook infeasible and dangerous.

Since notebooks can’t handle production workloads, developers turn to scripting environments or IDE’s. These tools are terrific for writing stable code, but they were built for software developers and so they’re unsuitable for data exploration. Maybe that’s why the Jupyter plug in for VSCode has been downloaded over 

Given the importance of where AI is at a board level, it's utterly surprising that this toolkit is involved in every serious data science project, at some point or another.

That’s where Zerve comes in. Zerve is the first Data Science Development Environment of it's kind, built upon a novel architecture, giving you all the benefits of interactive analysis with notebooks without sacrificing the stability of a scripting environment. With Zerve you can explore, build, and deploy all from the same environment. To truly do this, the underlying architecture had to ensure stability. Let's explore that.

The foundation of Zerve’s revolutionary architecture is separating storage from compute. In Zerve, code is organized as a DAG, and code executes from left to right. Each code block inherits the state of the preceding blocks. When you run your code in Zerve, Zerve launches serverless compute to execute your code. The state of the code block after code execution is then cached, serialized, and stored on disk so that it’s ready to be inherited by the next code block. This architecture changes the development dynamic in data science, and produces incredibly important benefits.

There is no restart kernel button in Zerve. You read that correctly. That's because there is no way of getting your Zerve canvas into a bad state, so there is no need to ever restart it. Jupyter is a fantastic exploration and academic tool, but the fact that you can explore data is where the similarity stops with Zerve. No matter the bells and whistles that have been added to new cloud notebooks, they all share the same architectural flaw that makes them unsuitable for serious work.

The main innovation made by these tools is that they are cloud based, which makes collaboration easier. With these sorts of tools, it’s straightforward for an entire team to login to the same notebook and potentially run code and collaborate on a project together.

Unfortunately, this actually makes these environments even more unstable. Because of the global variable space of these tools, having multiple people all able to run code at the same time means that notebooks are constantly being put into a bad state, with kernels constantly needing to be restarted. In other words, innovation in the notebook space is moving in the wrong direction.

Zerve, on the other hand, has no restart kernel button because it’s impossible to get a Zerve canvas into a bad state. Zerve is guaranteed to produce a consistent result no matter how the code is executed. Whether in a single or multiplayer mode, this changes everything.

Supposedly collaborative notebook environments have been around for years, and really all the “innovation” happening in the notebook space involves collaboration in one way or another. Anyone who has used a collaborative notebook knows that the experience is pretty terrible. Because notebooks are brittle, having multiple users accessing the same notebook at the same time presents a number of problems that are extraordinarily difficult to solve given the architectural flaws with Jupyter.

In Zerve, though, the collaboration is real-time and doesn’t present these same challenges. Users can run as many blocks of code as they want at the same time, write in multiple languages simultaneously, leave comments and tag one another, and more. Because of Zerve’s architecture, none of these activities poses any problem to stability, and with our upcoming GitHub integration, version control and change management is easy.

Bouncing between tools is second nature to most data scientists and data engineers. SQL, R, and Python are written in different environments. It is our view however, that the need to do this, should be alleviated. The architectural flaws that exist in Data Science Development tooling here means the best that can be offered is to translate with packages like reticulate. This tends to be an offputting experience, and as a result an R coder just codes in Python, even when their skillset or the problem they are looking to solve, may be more suitable to R.

Because of the way that Zerve serializes the data from an executed block, different languages are suddenly able to utilize the same basic data types at the same time in the same project. For example, data frames are serialized as parquet files, so SQL, R, and python are all able to interact with these objects without having to do any complex and fragile code conversions. For v0.1, Zerve has built the architecture for mixing & matching languages, but certain aspects are still in our alpha process, if you’d like early access, please email info@zerve.ai

In Zerve, data/ML engineers, R users, and python users can develop in the same environment at the same time. We think this is very cool. 

Parallelizing code execution can be a complicated process that most coding tools are not designed to handle without specific, detailed instructions. If you’re training 5 different models, most people will just allow them to train in sequence rather than go through the trouble of parallelizing the compute. Given the availability of cloud compute, this inefficiency is pretty alarming.

Zerve’s serverless technology allows users to run as many code blocks at the same time as needed. That means that cloud compute costs only accrue when code is actually running and there’s no need to worry about remembering to shut down servers. Zerve handles the orchestration flawlessly, saving both compute costs and clock time.

Moving a model from the design and development phase to production is a challenging task. Many new companies have sprouted up in the last few years around MLOps and hosting and maintaining models in production, and many companies have built their own procedures and pipelines for deploying their AI solutions. Things like managing dependencies and hosting models are outside the scope of knowledge of many data scientists.

Zerve’s automatically serializes all of the artifacts of every analysis by default. That means that the models and data sets created in a project are accessible from elsewhere in Zerve and from external applications as well.

Zerve is free to use and our free plan includes significant storage and compute resources. It’s perfect for code-first data users. Whether you’re querying a database with SQL, visualizing data with R, or deploying models in python, Zerve can enhance your workflow, increase your productivity, and enable you to collaborate with friends and colleagues on all of your data projects.

Release of Zerve v0.1 Data Science Development Environment

Announcing Zerve - The Next Generation Data Science Development Environment

Our mission at Zerve AI is crystal clear: to 

 This mission was born out of the frustrations our team had in delivering data science projects at scale, but also shaped by the community of “code first data users”, we have spoken to along the way (friendships were fostered!). The journey began with a commitment to understanding the daily challenges faced by those on the front lines of data—Data Scientists, Data Engineers, and ML Engineers. Zoom after zoom, we invested 100s of hours interviewing and getting to know every person. We knew our mission affected us, but having spent time with the communtiy, we knew it was something larger. 

Our conversations created a sobering reality—the state of data science tools is hindering the full potential of truly exceptional people. The impact they should be making is stifled by the barriers erected by current solutions. Despite awesome inputs, too often limited outcomes were achieved by the individual or the team. 

However, it's a bell curve, and of course there are representations not depicted in this blog, but it is the view we saw expressed most, otherwise we would not have dedicated ourselves to this mission. 

Although we have a team of statisticians, we didn't just want statistics; we wanted stories. The anecdotes and quotes collected during our interviews brought life to the pain points experienced by our interviewees. It was a moment of shared frustration and laughter when we presented a slide encapsulating the daily struggles we had previously identified from interviews, as there tended to be a realization that their work-life was mirrored heavily by other folks in similar shoes around the world. I wanted to show you the exact slide I shared, (despite my web team telling me it's "ugly"!). Here it is, and like I mentioned, it often sparked laughter due to the resonation with the viewer (and maybe the design!). 

One recurring theme echoed through the interviews was the substantial time spent getting set up in various systems at the beginning of an employment or at the start of a new project. Ensuring the right packages are used across a project was a constant pain point when it came to collaborating at exploration or handover stage. We heard of folks months into a job "ssh porting into a server to run a notebook" and describing it as "misery". 

Aligning with colleagues about the intricacies of a notebook or the nuances of the data they're working with becomes a bottleneck, hindering collaboration and progress. We heard stories of people who have added daily meetings to ensure Notebooks are run in the right order, teams waiting to make progress on a project while a colleague was on leave due to local data or code, or a quite remarkable story where slack emojis were used to manage the statuses of an online notebook. Better processes exist of course, and these are not a reflection of teams we met, but the circumstances they found themselves in organisationally, when BAU often made it difficult to rectify the origin of the situation. 

Deploying models is not a solo endeavor; it requires aligning with software engineers, often getting on their roadmap, and explaining the intricate details. What we felt from the interviews here, was an often awkward handover process. Depending on the type of organisation, the pain here could be felt on different levels, with frustrations occurring on both the engineering and data science side. Machine Learning, AI or Analytics Engineers are brilliant, but finite resources for working between the two disciplines.  

Configuring cloud and compute resources shouldn't be a roadblock. However, for many, it is. The inability to easily configure these resources themselves—or sometimes not at all—adds layers of complexity, involving additional teams and further delays. They end up as a ticket for another team, and yet again not in complete control of the level of impact they can make. 

The struggle to demonstrate their findings is real. Whether it's the manual process of copying and pasting visualisations into PowerPoint or wrestling with a BI tool, the inefficiency in showcasing their work remains a significant pain point. The key problem here is that the lack of impact is felt across the board on this one. We had a CDO say “I thought I hired coders, but all I have 'gotten' is screenshots of charts in a PPT”. Of course the content of that presentation was all built with code, however the organisation had not given the team an effective way to share, via data apps, or integrations with core systems. It's just not where any organisation needs to be today. 

In the next section, we'll explore how these pain points extend beyond individual experiences and affect the broader landscape of data science teams and their perceived value within organisations.

The pain points identified in our interviews weren't isolated incidents. Leaders across organizations echoed a common sentiment: they weren't receiving the expected value from their data science teams. This sentiment reverberated through discussions with executives and managers who, time and time again, expressed dissatisfaction with the outcomes of their data science initiatives.


These leaders emphasized the gap between the immense talent within their data science teams and the tangible impact on business objectives. The disconnect was stark, with data scientists sometimes struggling to translate their high-level statistical work into solutions that could be efficiently integrated into existing workflows. This hit home really hard for the team at Zerve. The frustrations we felt coming back from the executive team was they had been misguided with their original hire, the data scientist. They were made to believe they originally hired something different to what they got, when in fact, the fault was organisational and there was not a road for delivering impactful results from the data scientist,  and that’s why no impact was seen by the executive team.

Check out this post where we launch our solution

 in response to the information we received in the interviews above. 

Zerve's mission to increase the impact of Data Scientists

Blog

Latest Updates

Amongst Great Compan(ies)

A month after launching, here's what we've learned

The Evolution of Data Science Environments: From Jupyter to the Future

Amongst Great Compan(ies)

A month after launching, here's what we've learned

The Evolution of Data Science Environments: From Jupyter to the Future

Announcing Zerve - The Next Generation Data Science Development Environment

The Journey to Unearthing the Pain Points in Data Science Development

Ready?