Tag Archives: Data

Lean In Data Science

How do we apply the principles of lean to data science and data engineering? We discuss this broadly into using AI and machine learning more generally.

This is a topic that we had discussed over the summer and wanted to come back to six months later because so much has changed and transformed in the industry. What does agile lean process control look like in an infrastructure automation platform? How can we make these very difficult and challenging components of data and data management, more agile, more lean?

I think you will get a lot out of this conversation considering our current hypercharged AI ml and LM environment.

Transcript: otter.ai/u/1ZuALgSXcPw-bIf2GO…?utm_source=copy_url
DALL-E Prompt: please create a picture of a very large truck stuck under a low bridge. please label the truck as ai and the bridge as lean

Identity vs Privacy? Trade-offs required?

How can digital identity be used to build better trust and systems in our daily transactions? There are really significant challenges and consequences to having a national guaranteed identity – a single identity provider.

Knowing who you’re interacting with, in every form, in every situation is not as simple as you might think. There’s a lot of analogues to physical identity that are worth considering.

What would it mean for us to not have privacy? Does identity mean we don’t have privacy in our interactions? Who can we trust and what authority do they have?

Transcript: otter.ai/u/o_43fyGjxu24Ur5rpz…?utm_source=copy_url
Image by Dall-e prompte: a cartoon like image of a humanoid robot looking into a mirror and seeing a masked pirate version of itself

Data Darkages – do LLMs drive paywalls?

A coming Data Darkage is on its way, where we’re watching Reddit, Twitter and other companies take what used to be publicly available information and put it behind a paywall or gate.

Because of the way large language models are using this data and the value of the data, we are expecting to see that trend accelerate. This will have profound implications for how we think of, share, and use data in the coming years.

Transcript: otter.ai/u/e1XCyhSa9V81bgMpbo…?utm_source=copy_url
Photo by Pollianna Bonnett: www.pexels.com/photo/young-brune…e-chair-17687131/

Data + Operations = DataOps

We talk about DataOps, but if you’re expecting this to be DevOps for data – you are mistaken. Today we talk about engineering data through the idea of data stewardship or how you manage and control the data.

Beyond permissions and access into the costs and how things are stewarded, how logs are handled, who controls how much access and how quickly people have access, where you’re putting the data to improve its effectiveness, and more. There is so much going on above the infrastructure, but below that actual analytics, this conversation will open your mind to a whole layer of challenges related to governing and managing data.

Transcript: otter.ai/u/SDaRomv41JtQYM7a6C…?utm_source=copy_url
Photo by Taryn Elliott: www.pexels.com/photo/person-hold…g-on-top-4099125/

Domains And Access For Metadata

What actually is used to describe the provenance and information that comes with our data? Today we discuss metadata and the governance, security, hint, domains, date that accompany data in ways that in some senses are more important than data.

How can we move, change and transform data? We had a really robust conversation about how access to data is so critical in actually understanding how data is used.

This is a topic we struggle with: figuring out the naming, how things work, and the context. More than anything else, the context makes things challenging. As you listen to this, think through how challenging it is to define data, data governance, and using data effectively.

Transcript: otter.ai/u/VE2u_jTwG_h4z9ewfI…?utm_source=copy_url
Image: www.pexels.com/photo/cheerful-bl…hoolkids-5905918/

Strengthening Security’s Weakest Link

How do you deal with the weakest link in security?

Today we talk through how we can secure systems, all the way from what technical processes put in place to the people involved to legal enforcement, and who pays the price when data is compromised? There’s a lot to digest here that comes back to thoughtful ways in which we can deal with the weakest link in the systems.

How do we create robust security models?

Transcript: otter.ai/u/mkup2hKSzyP0Pkpxkk…?utm_source=copy_url
Image: www.pexels.com/photo/brown-thread-2072872/

Data Cartels Book Discussion

The book Data Cartels by Sarah Lambda serves as a starting point for our discussion today. www.amazon.com/Data-Cartels-Comp…ion/dp/1503633713

A dense and thoughtful book, it is straight up the alley of the type of conversations of the2030.cloud has. Our analysis of the book and the challenges it provides – the data compliance governance, the legality, the threat, and broader implications of what Dr. Lambda lays out – are all really important.

Today’s podcast is understandable and interesting without having had to go through and read the book, but I still recommend that you do.

Transcript: otter.ai/u/T5CJzO8pMrpGnLVGo4…?utm_source=copy_url
Image: www.pexels.com/photo/lady-justic…-a-gavel-6077123/

Metadata Architecture

Every time we look at data analytics and data systems, the idea of having a way to manage control and explain the data is actually as important as the data itself. This episode is all about metadata, specifically, metadata related to data analytics analysis, Big Data computation, sort of the data lake metadata problem.

Today we discuss the challenges of data management, but also the potential of understanding so much more about how data is used. If you are an IT professional or a data professional, you will find this conversation about how we’re going to draw inferences, manage and control all of that data that were collected fascinating.

Transcript: otter.ai/u/dFF6CfBYOMR3QFydae…?utm_source=copy_url
Image: www.pexels.com/photo/woman-stand…-mirrors-6208385/

Meta Data, Dark Data and Intent

https://soundcloud.com/user-410091210/meta-data-dark-data-and-intent

Data comes from many different places, sources, and ways. Some data we call dark data, which is data not accessible to you, and all of it relevant. Today we talk about metadata as part of the governance control management exposure of data.

An important layer beyond the data itself is the governance intent, how people access it, and how you combine data. We discuss exactly what that is, but still only touch the surface.

References:
yago-knowledge.org
en.wikipedia.org/wiki/Ontology

Transcript: otter.ai/u/rP_4gi-iDont0r2CaZ…?utm_source=copy_url
Image: www.pexels.com/photo/apples-and-…n-books-14098062/

Rob’s Hot Take:

In the Cloud 2030 Podcast episode on metadata and building a data control plane, Rob Hirschfeld emphasizes the challenge of controlling data egress due to its diverse sources and destinations. He contends that rather than attempting to create a locked box for all data, the focus should be on embedding information, particularly metadata, to control data consumption. Hirschfeld envisions a distributed system for a data control plane, involving multiple parties managing and providing consistent rules for data use, acknowledging the complexity of data storage across various locations. He encourages listeners to explore the insightful February 2nd episode and engage in ongoing conversations at the2030.cloud.

Metadata For A Data Control Plane

https://soundcloud.com/user-410091210/metadata-for-a-data-control-plane

Metadata is information that travels with the raw data that provides context, provenance, security, authorship, controls, and indexing.

The number of ways that you can expand the use of data is controlled by adding metadata. It creates a change in how we look at and manage data. Instead of creating control systems that contain the data, it’s actually packaged the control infrastructure, or the data control plane, as part of the data so that all of the systems can participate in it.

We also talk about data mesh a lot in that context.

Transcript: otter.ai/u/bho2SHu0N1YfECix6X8qNvKTlB4
Image: www.pexels.com/photo/pensive-eth…up-phone-3760607/

Rob’s Hot Take:

In the Cloud 2030 Podcast episode from January 19th, Rob Hirschfeld discusses the challenges and opportunities associated with metadata, particularly in the context of managing and sharing data effectively. The conversation explores the concept of packaging raw data in a way that makes it available on request without being included, accompanied by metadata detailing its provenance, access permissions, context, and even expiration dates. Hirschfeld emphasizes the importance of building a data control plane to navigate the complexities of data consumption and production, envisioning a future where metadata plays a crucial role in weaving together diverse data sources. Listeners are encouraged to explore the full 40-minute discussion for comprehensive insights, and to join ongoing conversations at the2030.cloud.