AJ: 00:00

So are we gonna talk about Adam speedrunning my hobbies?

Dax: 00:26

How's it going?

AJ: 00:28

It's great. Thanks for having me on Dax. I've been wanting to do this for a long time. I'm glad that you finally gone through all of the better people and gotten all the way down the list to me. That's that's awesome.

Dax: 00:38

Well, you finally did some stuff that was interesting enough, know? It it was more you than me. You have a new camera set up? You you look really good.

AJ: 00:45

Thank you. It's the same it's just a a webcam, but the the microphone is better and I actually did I went out and bought a key light, on on recommendation of you and others. Yeah. So I fixed a couple of those things.

Dax: 00:56

Yeah. The lighting makes a huge difference. It, you know, it can like make your camera seem like it's more expensive than it's not. Because I for a second, I thought you had maybe like a non webcam.

AJ: 01:04

No. Just a crappy webcam. You know, what it'll what it'll never do is make my beard look as good as yours and for that, I'll be jaded forever.

Dax: 01:12

That's funny. Yeah. The mic sounds great too. I there's a new purchase. You got the m v seven.

Dax: 01:17

Right?

AJ: 01:17

M v seven. Yeah. It's a I was convinced. Theo convinced me Mhmm. On Twitter to do that.

AJ: 01:22

So so far, I I've liked it a lot. I've gotten some comments on it. I've been told I have to go and just smoke a pack of Newport's a day to get that voice level down where we want it.

Dax: 01:32

Yeah. Even I feel that way. I have a pretty naturally deep voice, but Liz's dad, my wife's dad, she, he, like, smokes a crazy amount of cigars. And he's got that, like, graveliness to his voice, which I'm just like, I don't think I'm ever gonna have that, but it sounds amazing.

AJ: 01:49

I mean, you don't want it. Right? Because it's not a it comes with all sorts of downsides.

Dax: 01:54

It's kinda worth it. I don't know.

AJ: 01:58

Podcast is doing that well. You

Dax: 02:01

It's not that. It's just like whenever you're in, of course, it's for recording, it sounds great. But whenever you're in any type of scenario, any meeting, any like anything, you just sound way more authoritative and good and people wanna hear you talk. And I think that has just subtle effects on on everything you try to do. So it does make a difference.

AJ: 02:18

I think so too. So are we gonna talk about Adam speedrunning my hobbies?

Dax: 02:22

Oh, really? Okay. Let's let's let's start there. Let's talk about your hobbies. The thing I wanted to start off with that I mentioned yesterday is, so again, so for those of you that don't know AJ, I'm gonna start off with the weirdest part about him, which is he literally jumps off of bridges.

Dax: 02:37

And, there's plenty of videos on Twitter of him jumping off things, back flipping off things. So tell us more about why do you do that? How did that happen? Just give us a rundown.

AJ: 02:47

I mean, that's a long story. It starts I grew up in a town called Neenah, Wisconsin, which is about two miles north of Oshkosh, which hosts the single busiest fly in on the planet. For a week in July every year, it's the busiest airport in the world. It's called the Experimental Aviation Association, EAA fly in. It's this giant people who build planes, and you can do this.

AJ: 03:10

You're within the rights of of you as American, I think, just a just a resident. You can build an aircraft in your garage and put a experimental slap an experimental sticker up on the door and just take take it up once you have your pilot license. So I grew up around aviation, and I wasn't clear

Dax: 03:27

Oh, nice.

AJ: 03:28

It did it wasn't it was normal to me to have friends that just had planes they built or, like, family friends. So I like, oh, could go over to uncle Bob's house. He's got a hangar in his backyard where he built this kit plane and just, like, takes it off from his his grass field. That wasn't typical. It wasn't like every single person, but it wasn't abnormal either.

AJ: 03:44

I'd been in a lot of aircraft before I was 18. Mhmm. And I I always wanted to do it, it's super expensive. And the cheapest single cheapest way to get into aviation is to skydive. Mhmm.

AJ: 03:55

So I got into that around 02/2010. I got my license, and then I did a bunch of jumps for, like, two summers, and then had the opportunity to to learn to bass jump. There's some people that did it, and there's this bridge you can go to in Idaho where it's legal year round. It's free. You just show up, jump jump over the railing, and and jump.

AJ: 04:13

And, yeah, it's a it's a parachute kit that you just you could you can get, and you you can just go and and learn and do this thing. So I started doing that in November 2012. And had the pleasure of doing that ever since.

Dax: 04:25

Nice. How like dangerous of a thing is it?

AJ: 04:28

Right. So so the safety of base jumping, I think is is a little skewed towards more dangerous because not every jump is reported

Dax: 04:35

Mhmm.

AJ: 04:36

But every accident becomes news.

Dax: 04:38

Mhmm.

AJ: 04:38

So there are there are you you can jump anything tall enough. There are people that like, have a friend that owns a cell tower and runs a rural internet company in Minnesota, and I'm probably have a dozen two dozen jumps off of that tower. Wow. Because it's know, you just climb it and jump it, and it is. And there's no there's no organization.

AJ: 04:55

There is no governing body. There's no licensing program. So there's not a reporting structure nationally or internationally that reports this. There are very popular places to jump in the world. In the Lauterbrunn Valley of Switzerland is extremely popular.

AJ: 05:09

Mhmm. And there are several times. Northern Italy has a lot of great spots, and they have some organization, and they have some reporting, but that is just like these small bits. There is just no, you know, body that collects information on how many jumps are done. So it's all speculative.

AJ: 05:25

But, yeah, I've had the pleasure of having 870 some bass jumps. I've never been hurt. I splinted a couple legs and helped carry people out, but otherwise have been have been pretty safe. And there's a lot I mean, it's risk management. I think anytime you look at the stunts or these, you know, base jumps or YouTube, it seems like it's completely wild and out of control.

AJ: 05:47

But you're not seeing all the preparation that goes into it, and you're not necessarily understanding how the systems are engineered to keep you safe.

Dax: 05:55

Nice.

AJ: 05:55

Which I think is like that's one of the coolest parts about this board is just how simple the system is engineered to be. Right. And as a software engineer, I think you're you're like primed to appreciate it.

Dax: 06:04

Yeah.

AJ: 06:04

Because there is not a part of that kit that isn't necessary.

Dax: 06:08

Mhmm.

AJ: 06:09

And there is only like, every single part is designed for a specific purpose. It's designed to be bomb proof. It's designed to never fail or if it does fail, fail obviously. So you check it, you know, you catch it in a gear check. Things are color coded.

AJ: 06:23

Things are are ordered. Thing there's a check process. Manufacturers have collaborated on ways to ensure that the safety is sort of built into the systems. And it's it's just like a really interesting interesting from a technical standpoint achievement for these systems to be just so light, so small, and so simple. And so they're like extremely linear.

AJ: 06:42

Yeah. That's the way I would explain it.

Dax: 06:44

Yeah. That's very cool. What did you think of the the new stunt in the Mission Impossible movie from this year? I'm sure you've seen it.

AJ: 06:52

Yeah. I have. Tom Cruise is sort of legendary in both acting, of course, but also in the sport. Mhmm. He has a skydiving license.

AJ: 07:01

He has his private pilot license. He has a helicopter, a rotary aircraft certificate. He can he kind of does everything. He's an avid jumper too. Like, he takes it seriously and does a lot of jumps.

AJ: 07:12

Mostly, I think, in the large Arizona drop zone. But for the for the most part, you know, he did it. Yeah. Sure. Like and and you could tell they were, like, insurance company is probably sweating and he's sort of sitting down there.

AJ: 07:23

He's like, yeah, know, I could I could go again. I I could go again.

Dax: 07:26

Yeah. Did it a few times. Right?

AJ: 07:28

Yeah. Probably and I'm sure you, like there's, a behind the scenes interview where the the director's, yeah, we got a time we got it. He's, no. No. No.

AJ: 07:34

I wanna like, we're gonna go again. Yeah. And that's exactly what any base jumper would do. They're like, I'm gonna just press the fun button. There's like a little dopamine button.

AJ: 07:41

Yeah. And he Tom Cruise is walking over to the wall, and he's just pressing that dopamine button as his entire staff and crew and insurance company, I'm sure, are just sweating and holding their heads and tweeted out every time. He's like, no no no. Like, let's go again. Really impressive.

Dax: 07:56

So for those of you that haven't seen the that haven't seen the movie, he basically rides a dirt bike off of a ramp of a cliff, then he, like, separates from the dirt bike and then pulls a chute. And, yeah, he's he did it a few times, which is which is crazy. And I think for that was a crazy stunt. I think for me, the craziest one will still be when he's strapped to the side of the plane when it's taking off.

AJ: 08:18

Oh, yeah. That's wild.

Dax: 08:19

Yeah. Like he's strapped in so he's not really too much in control of anything but I cannot think of anything has anyone ever done that before? It's just like how many people on earth have had that experience of like knowing what it's like to be on the side of the plane as it's off? I mean

AJ: 08:32

more than you'd think.

Dax: 08:33

Oh, yeah. Because I'm just thinking. Okay. Yeah.

AJ: 08:35

So, like so when you're skydiving, you have to get out of the aircraft and

Dax: 08:37

That's true.

AJ: 08:38

The the large planes have jump doors. Mhmm. But I grew up jumping like a Cessna one seventy two or I think it was two zero six, two zero eight. The small Cessna where you have to open the door and climb out, and then you're holding onto the strut. So you're in the prop blast.

AJ: 08:50

You're behind the prop, and you're just getting you know, you're holding on to it, and and you have to get clear of the aircraft, and then also know where you're above the ground where you where you drop. Because you you're not you're not making a lot of horizontal progress even in a wingsuit Mhmm. Compared to where you can be in an aircraft. So you have to kind of know all this and and pick your spot and then climb out. So holding on is is a little easier than you think, but you're just gripped with fear Yeah.

AJ: 09:13

And that if it's true, you've got that like, you know, strength where you're you're like, I'm never letting go until the jump master is looking at me like, you go now. And you're like, okay.

Dax: 09:22

Yeah. It's, like in that scene, I was like, oh, well, the nice part is he's he's never thinking about acting because I'm sure his face is just reflecting exactly what he needs to be saying. Yeah.

AJ: 09:31

Yeah. I have no mean, must be terrifying to be outside of the aircraft on takeoff. I've never done that. That's like totally unsafe. The the parachutes don't work below a certain height, especially skydiving parachute.

AJ: 09:42

So I you know, that would be freaky from, like, an equipment standpoint.

Dax: 09:46

Yeah. Do do

AJ: 09:47

you think he had

Dax: 09:47

a parachute on when he was doing that scene?

AJ: 09:49

No. They had a cable up the arm. But Yeah. I can tell you from personal experience, I am a terrified person by, like, cliff edges without a parachute.

Dax: 09:58

Yeah. Me too.

AJ: 09:59

And then I put this stupid little backpack on, and then I'm happy to just walk over and, like, kick a rock over and just stand and be like, oh, yeah. Looks good. And and it it it shouldn't it's not right. It's like it's to to do this hobby, to do this as an activity, you have to, like, convince yourself that it's totally safe with a parachute even though you know it's not. Like, statistically speaking, it's just not a safe activity.

AJ: 10:20

But that same mentality that gets you to jump off the edge makes you think you're invincible with the with the backpack on. Mhmm. And I do think that's interesting because if I did slip and fall, I would get very badly hurt or killed. You know, you need to you need to have a good exit. You need to push hard.

AJ: 10:36

You need to present your body such that the parachute opens away from the cliff or the object you're jumping off of. There are all these things you have to do Mhmm. To do this safely. But that mentality of, like, okay, I can just walk up to the edge and just, like, peer down off a cliff. I think I think it like does you a disservice actually, but it's required.

Dax: 10:53

Yeah.

AJ: 10:53

Right?

Dax: 10:54

Yeah. Very cool. Yeah. I I I've always like heard about this. I never really known anyone that was so deep in it.

AJ: 10:59

So Oh, there's a bunch of great drop zones in Florida too. Should give it a shot. The one thing you're gonna miss out on, yeah, Skydive DeLand.

Dax: 11:05

Wait. Dade Land? Like D A D E Land?

AJ: 11:08

Or No. D E L A N D Oh, Dland.

Dax: 11:10

Oh, got it.

AJ: 11:10

Got Has has a really big drop zone.

Dax: 11:12

Nice. Well, I'm assuming these are all well, hang on. You're talking about skydiving or base jumping?

AJ: 11:16

Skydiving. Okay. Yeah. Don't think there's a lot of I

Dax: 11:18

was gonna say the highest point in Florida is like 300 feet

AJ: 11:21

in all of Yeah. It's just flat flat and marshy.

Dax: 11:24

Yeah. So it would have to be like office hell tower or something. So you were saying that Adam is speedrunning through your hobbies. What did you what did you mean by that?

AJ: 11:30

In high school, I started jujitsu and mixed martial arts.

Dax: 11:33

Nice.

AJ: 11:34

I like jujitsu a lot more because I didn't like getting hit in the face. I did one amateur MMA fight. I won. Nice. And then I was like, I don't know if this is for me.

AJ: 11:46

So I went back to just the jujitsu thing. I got a got a blue belt and then I was in school so I had to kinda put it aside for a while and then that next summer I picked up skydiving. So that's like, I'm waiting for Adam to be like, I'm done the jujitsu thing, but the skydiving thing Yeah. I'm gonna check that out.

Dax: 12:02

Well, his brother's a pilot. So Yep. That's he's set

AJ: 12:06

up And there is nothing stopping you. Well, there's a couple of rules, but nothing really stopping you from leaving that aircraft. I mean, so so I had a friend that built a powered parachute, and the way these systems work, they are the scariest piece of amateur aviation you've ever imagined. It is nothing more than a go go kart and a fan and then a a skydiving tandem parachute

Dax: 12:24

Wow.

AJ: 12:24

Over it. And there's a video on YouTube of it. And one of the interesting things about about bass drumming parachutes is they can generally be configured two ways. They can be configured for jumping off of really low stuff, and then the parachute is packed to open as fast as absolutely possible. They can be configured for jumping all at terminal velocity out of, like, a high aircraft or off of a cliff or with a wingsuit.

AJ: 12:45

And in that case, you have to have a you have to pack it so that it deploys slower, otherwise, you could really get rocked and damage the equipment yourself. And I'm coming up on this little powered tricycle. The thing is shaking. He can't get it trimmed right, so it's barely climbing. And we're like coming up kind of to power lines.

AJ: 13:01

We're maybe, you know, eighty, ninety feet in the air trying to gain altitude.

Dax: 13:05

Yeah.

AJ: 13:05

And all I'm thinking is why did I pack this for a high jump? Why did I why did I configure this? Because now I can't leave. You know, once once we're 200 feet up, I'm I'm you know, if I if I was packed for for we call it slider down and slider up. If I was packed slider down or or with a reefing device that slows the canopy off, I can jump very low.

AJ: 13:24

And that is the biggest comfort when you're in some somebody's hobby project they built on their farm Yeah. In their spare time with a BAC of point o eight, you know, the whole time they were building it. And and that that every time I kick myself when I'm like, why did I convince myself to to take the slider up and convince myself I would get altitude and I'm watching this thing climb so slowly, like, oh god. So, yeah, I'm waiting I'm waiting for Adam to speed run the rest of these hobbies. He's gonna get a motorcycle next, and then he's gonna sell that quickly and start skydiving.

Dax: 13:53

Yeah. The the amateur aviation thing is crazy. I don't know if you saw that news where these I think it was two people from Cuba. They escaped and made it to America with some kind of, like, powered glider thing that they built. And it looked it looked really cool.

Dax: 14:11

It like looked really legit, but it was just made out of like motorcycle parts and just like random stuff. They made it all the way here and I was like, know what? If someone does that, you should just we should be waiting here with like a citizenship and, like, a job offer.

AJ: 14:22

Yeah. Mean, it's crazy. Yeah.

Dax: 14:24

Like Yeah.

AJ: 14:25

The immigration in this country is nuts. So I'm I'm in Boston now. I was in Minneapolis for the last ten years, which has a lot more general aviation airspace.

Dax: 14:33

Mhmm.

AJ: 14:33

But my wife has a PhD and we we've been together we've been together as long as I've been base jumping, so almost eleven years. And we got married a few years ago. And we're like, she's a PhD, she's a master's degree, she's working on gene editing at a biotech now, but previously was doing cancer research. And we still had to fight the government for like four years to get a green card. Yeah.

AJ: 14:52

It's ridiculous.

Dax: 14:53

Yeah. It is really crazy. Like this one it's like I don't get super involved in political things but this is one area where I'm just like, I just get so frustrated because there's so many amazing people that wanna be here and pay taxes and give like us more money and it's just like we make it so hard for no reason when it's so obvious that we should be taking advantage of that and kind of starving our our competitors with those people but

AJ: 15:16

Yeah. And Canada is doing a I know. Great job. Right? Their their h one b process, like, you you you were in the lottery but didn't get it, you're automatically

Dax: 15:23

approved here or you're

AJ: 15:25

you're fast

Dax: 15:25

tracked here.

AJ: 15:26

That's like Yeah. Yeah. A no brainer. Right? For for a comp like, and we're a country that needs people.

AJ: 15:31

I don't know. Like, I think there's this misconception that we don't need more people. We desperately need more people.

Dax: 15:36

Yeah.

AJ: 15:37

And that that is a a important thing to do to make that happen. But I do like Boston. You know, I like it a lot. It's it's nice to be out here. Like I said, less less general aviation, but I, you know, pick up new hobbies and I'm riding my bike a lot and Yeah.

AJ: 15:50

In an office most days. So I'm at home today, but for the most part, I work at Datadog. I'm an engineer. I work on the serverless stuff. And I get to go into an office most days which is nice.

AJ: 16:01

I know I used to work for a remote company like UDAX and I like that too. But sometimes it's nice to just like talk to people in person.

Dax: 16:08

No. I definitely I definitely miss it. I posted the other day about how if I could literally have anything in the world, it wouldn't be a 100% remote. Like, I would love to just have a place where I can handle my coworkers whenever I wanted to.

AJ: 16:21

Yeah. Let's unpack that. Like, what is it what, you know, what did what did Jay and Frank think about too?

Dax: 16:25

No. They're all they're all into that as well. I think, I mean, the two of them are both in Canada and Toronto. So they they

AJ: 16:31

don't actually,

Dax: 16:32

so Jay works in a WeWork every day and Frank comes in, not every single day, but occasionally. Yeah. At the end of the day, it's we're such a small team and we all like each other, so we're just down to hang out. So it would be, it would be nice. Of course, it's not like an extreme where, like, we're in there eight hours a day every single day, but just some of that mixed back in.

Dax: 16:54

Yeah. I mean, I've always worked at small companies, and I know people think this is bad, but a big part of my social life did come from the people I worked with. Because at small companies, you just get really close to people. You tend to be you tend to pick people you could be friends with in terms of who joins the team. You hang out, you like get pretty close.

Dax: 17:14

And all the people that I work with and will continue to work with are people that I've just gotten made that relationship with. So

AJ: 17:22

Totally.

Dax: 17:22

I do like that aspect of it. And it's not for everyone. Like, I tend to mix every part of my life together. I smash work and out of work and all that stuff together. People like more clear boundaries.

Dax: 17:32

But, yeah, for me, I I I definitely miss that.

AJ: 17:35

Yeah. At 100%. 100%. I, I really do like having the option to be in an office. For me, I I think Boston is famous for for just horrific traffic, but I get to ride my bike or take the train, so it's pretty easy.

AJ: 17:51

And I think you'd said in your tweet, you're like, if I could just be walking distance, like, I'm I'm a mile and a half or two miles from an office, and I can just go in. And, yeah, it is such a powerful part of like my social life. I moved here with no friends. I didn't know anybody on the East Coast. And we both, my wife and I both sort of made it a point to try and like be super outgoing and super proactive about it.

AJ: 18:10

And then yeah, a lot of my friends here are coworkers and I really do really do enjoy that. I am excited for you to post the demand letter from Frank that says you must move to Toronto. You know, other you have thirty days or whatever it is to to leave Miami.

Dax: 18:24

Yeah. That's funny.

AJ: 18:26

But yeah I do think so so I do I suspect and again, I have no evidence of this. I suspect the next, you know, billion dollar startup or the next whatever, you know, big tech company is gonna be one that was started like in person.

Dax: 18:39

Yeah.

AJ: 18:40

Sort of have that feeling. And and again, I I'm a big proponent of remote work. That was a huge it unlocked a huge part of my career. And maybe we can kinda unpack that a little bit. So so Dax and I used to be competitors.

Dax: 18:51

I don't know if people know this. Yeah. So let let's so we talked about your hobbies which seemed like they would fill up all your time, but you also have this side thing where you're a software engineer, so everyone's aware. Yeah. So, yeah, just tell me a little bit about, like, your history with that.

AJ: 19:06

Yeah. I was not a good student, but I did manage to claw my way out of school. And I left mechanical engineering. I think I had, like, a c average. And I was like, I'm gonna just do software.

AJ: 19:16

And I finally was able to, like, get things together and graduate. Started working at a medical device company and actually liked I loved the team. I loved what the work that they were doing, but I couldn't I couldn't love the velocity. I like to build things as quick as I can and ship them, and and that is just not what you want in an engineer who works on medical So so I quickly moved on and joined a a startup that some friends were at in Minneapolis. That was awesome.

AJ: 19:42

That was a formative part of my career. I started working on Ruby on Rails Tech, and that company experienced really rapid growth, which which I think I credit a lot to my development, both as just like a developer, but also just a person. Mhmm. When you go from having whatever it was like a 100 people to 500 people very quickly, it's it's a big life change. And at the same time that company grew, we were approached by it was a sports tech company.

AJ: 20:07

We did youth and amateur athletic registration, payment processing, background checks. If anyone has a kid or volunteers on a sports team, you've probably used this this this service before. And we were then acquired by NBC Sports, and they did a very interesting thing. They said, we're gonna give you a blank check-in three years. And at the end of the three years, we are going to subtract whatever you spend from your valuation and pay everyone out.

AJ: 20:33

So we didn't have to worry about fundraising. Didn't have to worry about, you know, making payroll. And again, I'm just an engineer here, but wasn't necessarily involved in that. But from an engineering standpoint, it allowed the company to go and acquire all of these sports vertical, like small companies that were doing, you know, three, five person companies that were doing. Like we own gymnastics, we own swimming, we own golf or whatever it was, and sort of bring them into the platform.

AJ: 20:57

And at the time, the the result of that where we were a PCI compliant shop, which meant we were able to process credit cards directly with merchants. And there's a lot of rules and regulation about that. We also had to to bear in mind the legal world of the the, like, child online privacy act or COPPA, which, you know, it's like a little bit like GDPR for kids in The US. I mean, that's a horrible summation, but you get the point. We're we're we're adhering to all these rules, and then we're acquiring these companies that just didn't do any of it.

AJ: 21:26

And we had this incredible operations team that made so much of our jobs easier, and then they were just gone. And my VP came to me and said, you have a blank check, but I cannot give you an operations team. Figure out how to build your product. Mhmm. And that's how I found serverless.

AJ: 21:41

Yep. And started using serverless early on. This is probably twenty seventeen ish

Dax: 21:46

Mhmm.

AJ: 21:46

Before when we rolled it out to production. And I loved it. Found it really, really productive. Made it made a bunch of headway and was able to that's when I learned operations and I learned, you know, how to how to build on the cloud. And that picked that up, thought it was great sort of working on some of the open source aspects of it.

AJ: 22:02

And a couple of years later, reached out like the company, and they had raised a series a and they wanted to build a platform. So so I jumped a couple years after the acquisition with NBC closed, and that's how I started competing against Stacks.

Dax: 22:14

I was

AJ: 22:15

working on the serverless framework. And, you know, I worked on their competitor to s s t to seed run, which was like a CICD product

Dax: 22:23

for

AJ: 22:23

serverless. And I know you talk about this a lot, but I do wanna bring up the point. We had, I think, some of the same problems that you didn't have because you you were bootstrapped and we were VC.

Dax: 22:34

Mhmm.

AJ: 22:34

So we had growth, we had revenue, we had users, but it just isn't at a velocity that would necessitate venture capital multiples on exit. Yeah. Which made it hard to pursue. But I think for you and Jay and Frank, you can use it to extend your own runway with three people and build out what you really wanna work on while you have this other product that sort of creates some cash. And I think that is key to anyone who's maybe out there thinking about starting their own thing on the side or working on, you know, their own, like, kind of quote lifestyle company sort of startup.

AJ: 23:06

That is a really big opportunity and a really, really important thing to look at is can you do something you don't wanna do long term, but just long enough to pay the bills so you can work on the thing you actually wanna do long term?

Dax: 23:17

Yeah. Exactly. Getting to a point sustainability. Before we talk about that, I wanna go back for a second. So it sounds like what so do I understand this right?

Dax: 23:24

Like some of the compliance burden is what pulled you into serverless? Because that's exactly how how I got in there as well.

AJ: 23:29

It was two parts, right? First, the compliance burden where it's like, we have to adhere to these rules in this AWS account. So I can't just stand up a VM and call it a day. That is a big part of it. The other part was just my operations team was so busy bringing these other companies into compliance that I didn't have time for them to at the time we were on Chef and OpsWorks, and I didn't have time for them to like, they didn't have time to stand up new services for me or to run the operations for it.

AJ: 23:53

So that's when we we went on call for our own services. We own our own production deployments, own pipelines kind of the end to end process. I think for for some folks it's hard to imagine a time before DevOps, but there truly was like a wall between DevOps. It was like different rooms, different different people. You would just build the code and you would, you know, even with GitHub we would put like a label on it that would say this is ready to be deployed.

AJ: 24:13

And then sometime later someone would deploy it and then you would just get, you know, if it broke something you would get paged or then you would get brought in and they would be like, we reverted your code, it wasn't very good. Yeah. You know? So so the DevOps thing was really taking off about then. And that's now we we're in a world where, you know, any at virtually any large tech company, you're expected to run your own services for the most part outside of, you know, some core things.

Dax: 24:38

Mhmm.

AJ: 24:39

And that that I do think has been good for the industry. It's like a a point of mature maturation and it also helps helps people really understand what they're building and the impact it has versus like, oh, yeah. I'll hit you know, it's ready. I'll just hit the merge button. We'll see what happens.

AJ: 24:53

You know?

Dax: 24:54

Yeah. It's funny because even, even at a lot of, like, modern ish companies, they like, I've been I was at a company where I was kinda surprised to find that wall was still pretty strong. They still called it like DevOps, but and it wasn't to the extreme of like, literally, like, you had no idea how was deployed, but it was close to that. Like, people were very good and this was, this company was using Elixir for everything. They were very very good at using Elixir and the Phoenix framework and all of that.

Dax: 25:23

But they had no idea what happened once it left and went into production. And I found that really odd because for my entire career, it's like how it runs in production impacts, like, how I build the application and like things I can take advantage of. Yeah, there there still is that one and it's weird. I think that's why maybe a lot of people, don't understand or, like, don't appreciate some of the value of serverless. Like, so similar to me, similar to you, I was in the health care space for a while and we were running Kubernetes and our own containers and and all of that.

Dax: 25:58

And every time we signed a new client, I had the fun task of filling out a 30 page security questionnaire asking all kinds of crazy stuff like, what kind of antivirus do you run on your on your VMs? And like and it is just a huge pain, time sucks, such a waste. And like my least favorite part of the job by far. Then I realized, oh, you know, if I literally don't have access to the underlying infrastructure, nobody can ask me those questions. And that was what, like, pulled me into serverless where I was like, okay, if I build my things in this way, I the right answer for every single question is not my problem, not my problem, not my problem.

AJ: 26:35

Yep. Manage manage by provider, I think is the key. Right. It's like m you get to write MSP, like managed service provider and then dash and then AWS. Yeah.

AJ: 26:43

And that's just such a flex. You're just like, oh, I'm done with that.

Dax: 26:45

Yeah, exactly. So was like, if I'm gonna keep building stuff in a compliant in compliant spaces, which is actually not even that uncommon. Like, everyone kind of eventually ends up there because even if you're not directly in healthcare, you now have a customer that's in health care and they like wanna make sure certain things

AJ: 27:02

And now you are.

Dax: 27:02

Yep. Exactly. So you eventually

AJ: 27:04

Yeah. Or SOC two compliance. Right? Like, oh, you gotta be SOC two. You gotta be, you know, FedRAMP or there's a a billion of those.

AJ: 27:11

Yeah. Contrary to popular belief in the Forbes 30 under 30 list, ignoring the law does not make you immune from it.

Dax: 27:19

Yeah. Well, it does up until a certain point. But until people start looking into, like, wait, what's actually going on here? And at that point, it's like, yeah. So that's that's what pulled me into it and I found all these all these other benefits as well.

Dax: 27:30

And yeah, it's been crazy how long you've been doing it. So I really I got pulled into it maybe in like 2019 while I was at this Elixir company just because I was bored of Elixir and this compliance thing I I was thinking about. And, yeah, it's been awesome. So I met AJ in person at AWS re:Invent and it was the first time we met in per maybe the only time we met in person. And AJ has just been doing this stuff for so long.

Dax: 27:53

And whenever I have, like, a question about some weird obscure thing about how Lambda works or how does any AWS thing works under the hood, Ada is a person I I ask, and it's been amazing to have him as a resource. And you even, like, respond to questions in our in our Discord, which we definitely really appreciate.

AJ: 28:07

Oh, yeah. I mean, love SST. It like it's a great tool and I think everyone who's considering or or wants to experiment building on AWS should should try it out. Yeah. So I'm happy to answer questions there and I, you know, answer questions on my own blog.

AJ: 28:19

I'm exploring and experimenting with the idea of of streaming while I work on some other things. So I'm happy to announce on this podcast that my RFP for a talk at re:Invent was accepted. Nice. Thank you. I'm gonna be talking about Lambda and cold certs and initialization, and I'm running kind of a suite of benchmarks and working closely with with other other people at Datadog and also a couple of people at AWS to answer some of those questions that I have, and then we're gonna be putting together this talk.

AJ: 28:49

So I'll be there. I'm pretty excited about it. Nice. Yeah. Yeah.

AJ: 28:53

But the serverless space, I mean, yeah, we you know, I think we'd interacted a lot online before we met in person. And I do think serverless is at an interesting jumping off point where it's still possible to do that, but it is sort of eating the world.

Dax: 29:08

Mhmm.

AJ: 29:09

I mean, I think in our Datadog state of serverless report last year, we shared that like 50% of organizations have adopted serverless at some point in their stack, which makes perfect sense because it's it fills so many gaps in ways that you don't have to, like, be you know, own the entire operationalization of the of the the tool. You can just sort of run it and it just executes. So I I do think that there's just a ton of value there. I also think that there are a whole folk number of folks getting into to software development and learning serverless first. That is really great, and I'm I'm really happy to see that.

AJ: 29:43

I do think people struggle with some of the distributed systems rules that govern the world we work in. And I think that's funny because I think both of both you and I have kind of a distributed systems back end background before we got into serverless. And I do think that has helped me a lot in my, like, journey and over the over the years even though I have been working in the serverless space for so long. But it operates on the same fundamental principles. Right?

AJ: 30:07

You have to build in idempotency. You have to handle race conditions. You have to handle retries. You have to handle things like either load shedding or just graceful failures, like what happens if the service is down. And that part, I think, is really like, that's that's just, an important part to learn if you're gonna pick up serverless to understand that you are in a world of multiple systems running in concert, which is like a really hard thing to learn if you're just coming straight from, okay, I'm gonna build this app on my computer and then just deploy

Dax: 30:36

it. Yeah.

AJ: 30:37

But once you get past that, yeah, it's such a force multiplier, isn't it? Yeah. Like, it's so much leverage you have.

Dax: 30:42

Yeah. It's it's fine. The the thing I point out all the time is, so prior I was doing Elixir, and Elixir has really incredible concurrency and distribution, primitives that you can use. So taking a task and running it in parallel, is very natural in Elixir. You can run across multiple machines, etcetera.

Dax: 31:02

And I thought that was great. But whenever you did that, whenever you did like, I'm gonna run this parallel, you set a limit of some mental you had some mental sense of like, this has to have a limit because I've only provisioned four servers and I know that I can't just be like, okay, run this at like a million x in currency. So like you like kinda had that, but there was still a cap of like, I wanna go up, let's have to go spin up another server, join to the cluster, etcetera. And once I moved into the serverless world, that suddenly became an option where I'm like, this needs to go faster. It's can I run it in parallel, I'm just gonna do it a 100% in parallel?

AJ: 31:38

A 100%.

Dax: 31:39

And it's like, it's crazy how much how much is just solved by doing that. Like, so many problems are highly parallel and you don't have to worry about optimizing the individual units runtime when you can just run it a 100% in parallel.

AJ: 31:51

And you learn that lesson one time and it becomes your superpower. Yeah. Like if you look for the aspect of a problem you're faced with and you you figure out how to in in the in industry parlance, we'd say scale horizontally. But you know, how to how to do this separately and isolated from from other transactions you're processing or other jobs you're doing. You can do them independently.

AJ: 32:11

It's so easy to just rapidly scale them out, especially with something like Lambda. And I would say the big benefit for Lambda is that they're they're isolated guaranteed.

Dax: 32:21

Yep.

AJ: 32:21

You have this guaranteed isolation where you're going to get this amount of CPU and this amount of of memory. And beyond that, you don't have to worry about the noisy neighbor problem affecting

Dax: 32:31

Mhmm.

AJ: 32:31

Your individual request. So if you have a 100 requests and the the server starts to slow down in a normal load balanced container application, one request is gonna affect 99 others. However, in Lambda, that is just slowing down its own its own request. And this is a principle. This is just queuing theory.

AJ: 32:49

Right? So I talk a lot about the Trader Joe's model of queuing theory. And I wanted to I wanted to bounce this off you before I build this in my talk. But are you have have you shopped at a Trader Joe's or do have those in in Miami?

Dax: 32:58

Yeah. We do. I haven't been to it much, but I have been to a Trader Joe's before.

AJ: 33:01

I I recently was told that this is not true for all of them so it does the metaphor breaks down a little bit. But at the Trader Joe's in my experience, the way they're set up is they have one line of people waiting to check out

Dax: 33:12

Yep.

AJ: 33:12

And then they have multiple cashiers.

Dax: 33:13

Yep.

AJ: 33:14

And the benefit there is you get limited variability in the p 99 or the ninety ninth percentile. Right? Those outliers. And almost everything we do as performance engineers and software is outliers. Like, what what is that point o 1% or point 1% of requests?

AJ: 33:31

Why is it taking so long? Yep. And the benefit of of the Trader Joe's system of checkout compared to a typical grocery store where you just pick a line and wait is that one person in front of you, you know, writing a check or

Dax: 33:42

you would you would use gift cards or something like that.

AJ: 33:44

Counting pennies. Yeah. Doesn't impact you as much. Right? Because it all it does is is it's like a thread pool.

AJ: 33:49

It reduces one worker in the pool, but every other cashier is there and also at the same time they can quote spin up, know, new cashiers can come forward and help burn down the line.

Dax: 33:58

Yeah.

AJ: 33:58

And that is very similar to the way I think about Lambda and distributed systems is having this ability to consume work. And if you're a Go programmer, if you're a back end developer looking at Go, you already probably do this because you already are familiar with Go routines

Dax: 34:12

and Within a single process. Doing workers.

AJ: 34:15

Yeah. Absolutely. Absolutely. And the same same with Rust if you're using, like, Tokyo or even Node. Right?

AJ: 34:19

A lot of Node developers don't realize, like, how the like, new Node developers don't realize how the event loop works, but it's very similar. You know, you just the the event loop is processing whatever events are being admitted. As you get more, you push more things into the event loop, and then it it continues to cycle through and and execute any awaiting transaction in the in that loop.

Dax: 34:35

Yeah. I mean, people do get hung up on that because when I asked I asked a question the other day about doing something concurrently in Node, and I had a few people respond being like, that's not possible. You can't do that. And they didn't realize that, well, for IO work, you can because it like Totally. It jumps to the to the like pauses that and jumps to the next CPU work it needs to do.

Dax: 34:55

Yeah. It's a the I mean, the thing you bring up, the trader the the trader that queuing thing is great. It's that's another problem. I I totally forgot that these were problems I used to deal with, but, again, very typical. Right?

Dax: 35:06

Like, build some service where you can upload a CSV and get it processed. Most people have CSVs that have a thousand lines. Someone shows up with a 2,000,000 line CSV, and you prior process it as aggressively as you process everything else, and suddenly everyone's requests are slow. So being able to, like, preempt that and, like, and deal with those situations is is again, I totally like, this used to be a constant problem for me. It's like, okay, how do I deal with this like one one big user?

Dax: 35:32

But I just totally forgot that it's it is not an issue anymore.

AJ: 35:36

Yeah. And I think, you know, you we're getting to point in our careers where you can look back on things you solved before, which is that's experience. Right?

Dax: 35:42

Like

AJ: 35:43

that's that's what people talk about when you're like, what does experience mean? It's like, oh, I've solved this.

Dax: 35:46

Yeah.

AJ: 35:47

And the CSV thing is something I think everyone has to go through. And it's yeah. It's it's exactly what you're describing. We we you used have to do these running a big Ruby on Rails app with background processor called delayed job and later sidekick. It would enqueue these tasks in Redis.

AJ: 36:00

But to make the task horizontally scalable in Redis, would split the file either by line or by chunk and then enqueue those as different items. And that still doesn't quite solve the the noise and neighbor problem, right, where one person is doing 2,000,000 and the other person is doing six. Why are they way waiting behind that person? But with s three and lambda and event triggers, all of a sudden you get that isolation. Right?

AJ: 36:19

And I can say, okay, here is a URL, put your CSV file here, and then that that URL, once that user writes the binary data to s three, will trigger a Lambda function. That Lambda function can then you can do so many things right away. You can even use something like step functions dynamic map to just entirely have it split and processed, and then you don't have to worry about any of the isolation at all. Right? It's done for you.

AJ: 36:42

Yeah. That's pretty cool. You also get so like another thing you're gonna do is is just by by the nature of CSVs, people are gonna have duplicates. I don't know about you, but everybody's data is crappy. So they upload these giant CSV files, and you can you know, Levenshtein distance is an algorithm to detect how close words are together.

AJ: 36:59

It's used for autocorrect. You may not know the algorithm, but you've just you've been a user. You've you've definitely used it before. And it's such a common thing to, like, run leveraging distance against the rest of the file and and flag any dupes and have people correct those. And I I think I built that twice at, like, production scale.

Dax: 37:14

Mhmm.

AJ: 37:14

And the first time, I got paged a lot. Yeah. I didn't handle I didn't handle the p 99 well.

Dax: 37:20

Right.

AJ: 37:20

But the second time, using, like, step functions and using serverless was just so effortless that all I had to do is watch the metrics. Yeah. And just say, oh, wow. We just we just processed a 200,000,000 line file that this giant Yeah. Like, national governing body uploaded every single person in their contact list ever.

Dax: 37:35

Yeah.

AJ: 37:35

And it just churned through it.

Dax: 37:37

Yeah. It's, it's really cool. And even yeah. You can definitely see it in your own I mean, you you mentioned the metrics and that that's an interesting thing I think about all the time. If I look at the metrics that I used to obsessively look at and the ones I look at now, they're completely different.

Dax: 37:53

I could never go to production before without having memory monitoring in place, CPU monitoring in place, alerts, alarms, all of that stuff just in case something happened and I need to go intervene and scale something up. I don't know the last time I really looked at any of those metrics. Now I just look at, woah, look how much we did. That's cool. That's about it.

Dax: 38:13

That's the only time I I I look at stuff. Yeah. So it's like a completely different world.

AJ: 38:19

Totally. And Lambda solves that? Like Lambda Lambda solves that, but let's talk about s three.

Dax: 38:23

Yeah. S three.

AJ: 38:24

Which might be the single most impactful data system that any of us have ever ever used ever. S three does one trick, and that trick is called erasure coding. And if you've ever read about RADARASE or built a gaming computer, you maybe know a little bit about this, but the just the idea is break breaking up a file or a binary blob. It's arbitrary binary data into chunks, and then writing a multiple of those chunks to different services. So you take the bits, you run them through an algorithm, you scramble the bits, and then you take different sets of those bits and write them to five places.

AJ: 38:59

So maybe or maybe you have you have you have one record that needs to be divided into three chunks to store at scale. Mhmm. Then they're gonna divide them into five. And the idea is you can you can lose two of those five disks or two of those five data sets at any point and your data will still be fine. But it's not just about data storage, it's about retrieval.

Dax: 39:19

Mhmm. So

AJ: 39:19

when you read a file from s three, it's going to do a scatter gather operation. You talk to a control plane or a service that tells you where this data lives, and that control plane will say, okay, here are the, you know, 15 nodes you need to talk to. But once you get 10 reads back or or, you know, 10 responses back, you now have enough to reconstitute that file, you could just drop the other five reads you were doing. And that's so important because in any distributed data store, the there's always gonna be one node that is doing garbage collection or hot spotting, where there's a 100 other requests that happen to hit that node at that point or or any any number of things that caused it to drop that request. And this is why you get this very consistent response out of out of s three.

AJ: 40:00

And now every cloud has their own version of of s three. So you have like a binary compatible blob store. And that that is just such a powerful thing to be able to do. It's not necessarily super super fast. You don't get single millisecond responses like you do out of Dynamo.

AJ: 40:14

But you you sure as hell aren't gonna have, you know, one minute long response times like that long tail of data systems and data services that we would typically see in distributed systems. You don't have to deal with that. Yeah. And that becomes the basis for so much of what we do. Yeah.

AJ: 40:29

We wrote a Datadog wrote this great blog post in the service called Husky we use that stores our event data. It's you you can read it. It's on our website. It's great. And that's all backed by Blobstore.

AJ: 40:39

Alright? Like, that's that's how that that service works. Yeah. So many other systems built on AWS are just built on s three as well.

Dax: 40:48

Yeah. It's funny because that's one of their oldest services. I think for a lot of people, they might have even got introduced to AWS through s three. It's also like it fits the mindset of serverless. It's been that way since the beginning as maybe the first serverless, offering that people use.

Dax: 41:02

Like, you just pay for your usage and you don't think about provisioning resources at all. Yeah. And it's, you know, great. So much is built on top of it. I actually, like, got into, like, managing servers through distributed file systems.

Dax: 41:15

My

AJ: 41:16

Oh, really? Tell me about that.

Dax: 41:17

My dad bought my dad's also he's also a software engineer. He was doing, like, a Hadoop prototype for his job he was working at the time. So he bought, like, this rack of eight servers that he just installed in our basement and he was using that to, like, set up Hadoop and prototype. And and Hadoop is like it's similar to what AJ was saying. Like, it's it splits up data and you have like some metadata server that tells you where everything is, you can kind of retrieve it efficiently in in in the same way.

Dax: 41:44

Basically, it's kinda like build your own s three, which people were doing back then.

AJ: 41:50

Yeah, he had to.

Dax: 41:51

He, yeah, exactly. He prototyped with it for not that long. I think it took him like one month. And after that month he was like, okay, these are yours now. And I was like, oh.

Dax: 42:00

And then that's how I learned how to do Linux and servers. And I would like I was like putting up Minecraft servers and doing all this stuff. But, yeah, like that that was technically the only reason those machines made it into my house was was for this this reason. So I was doing the distributed faucets and stuff, but that's why I got there. Yeah.

AJ: 42:20

Yeah. That's why they're there. That that's awesome. My my my my dad worked part time as a janitor at the school I went to to sort of make ends meet when things were rough. And he they were gonna they were gonna recycle these, like, Pentium two machines.

AJ: 42:35

Mhmm. And he took home two of them for my little sister and I, and that's kind of how I got my start. And that's actually how I got into, like, networking, because I wanted to play, like, SimCity and Streets of SimCity and all these games with my with my sister. And two machines was it solved such a huge problem in our life, which was, you know, one controller on the Super Nintendo or the Sega Genesis or whatever. Right?

AJ: 42:54

There's only one. Yeah. And how are you gonna share with your sibling? That the unplugged controller trick doesn't work past, like, age four. You know?

AJ: 43:01

They they they catch on. So so that was such a big part of of me growing up too is was digging around on on used machines. And Yeah. I do think it definitely, like, lit the spark.

Dax: 43:12

Yeah.

AJ: 43:12

The other thing that did it for me was Legos. Yes. Yeah. I wrote about this a few years ago. It's probably I'm most embarrassed by it because it's just not a good piece of writing, but you have to you have to be bad at something before you can be okay at something.

AJ: 43:26

And I wrote about how LEGO League impacted me as an engineer. And LEGO League was this, like, ultra nerdy hobby I picked up where you have this this LEGO kit, which is based on a microcontroller, and then motors and sensors and then just Lego parts. Yeah. And you build a robot and you use that robot to compete on a on a map to score objectives. And every year, the objectives were different.

AJ: 43:48

You might have to rescue a villager from a volcano or it's like a Mars mission, drop a supply crate off at some launch site or something like that. Right? Just like little like move parts around on a board. And that it it it was written with this visual Wait.

Dax: 44:02

Are you talking about like Lego Mindstorm? Was that related?

AJ: 44:04

Yes. It's a Lego Mindstorms.

Dax: 44:05

Yeah. I had that too. The visual the visual programming thing. Yeah.

AJ: 44:09

Yep. Yeah. Yeah. And that was such a cool thing for me because like I said, like we talked about, I'm not a good student. I need to see things and and be like, this is important to me because it does something I want to do and not because I'm a better person for learning it.

Dax: 44:22

Yeah.

AJ: 44:23

And that clicked for me. Like the Lego thing clicked for me, where I could drive this robot around my living room or or whatever. And I do think that definitely keyed my interest. Because the people that coached all that were all engineers. They were all hardware engineers or electrical engineers or software engineers that parents

Dax: 44:38

that were

AJ: 44:38

at the school that And that definitely piqued my interest. And I do still see some of that, like in the engineering world today, I see people kind of bringing aspects of those old little hobbies and toys to their day to day. And I think that is neat. It's a good way to teach, like to see if kids have that spark and are interested in in this type of thing. If they have the patience or frustration to like see it through till something works.

AJ: 45:01

And I think software is kind of one big frustration exercise after another. Yeah.

Dax: 45:06

Yeah. That's a big part of it. Just nothing worked when I guess when we were growing up, probably some of age. Right?

AJ: 45:11

So nothing It still doesn't work now. Worked.

Dax: 45:12

But but to a crazy degree, like, every single time I wanted to print something, had go debug the printer. Like, what's wrong? The driver, the thing. Then it start printing do you remember when, like, it would start printing out, like, all this nonsense, like, weird

AJ: 45:23

Test page and yeah.

Dax: 45:24

Not not even that, just, weird care my printer had this thing where it would, like, not decode something correct, so would just start printing out, like, all these, like, weird characters and stuff. Was actually kinda scary now in hindsight.

AJ: 45:35

I think that's your dad trolling you. Like, 99% sure your dad was, this kid keeps messing with my printer. He's already got my my rack. Like, what more does he want? I'm just gonna print some scary stuff that says go to bed and he'll be done.

AJ: 45:46

Yeah. But,

Dax: 45:48

yeah, nothing worked and I had to constantly debug it and you just kinda learn that debugging skill, which you learn from like literally anything. It doesn't have to be something

AJ: 45:55

Oh, totally.

Dax: 45:56

Peer related. Anything that's not working, have to figure out what the issue is. And he's developed a lot of patience for My brother is very different. He got I think he grew up in an era where stuff would just just worked a lot better.

AJ: 46:06

Yeah.

Dax: 46:06

And he just you know, for him, it's just like there's this classic story my dad always tells where my brother was complaining the TV wasn't working. The TV's not working. And my dad goes to check and it just like wasn't plugged in. Classic. It's just like, it just didn't register for my brother to like dig in and figure out what was wrong.

Dax: 46:25

So I'm I definitely appreciate the era that kinda we grew up in. Think a lot of a lot of stuff kinda pulled us in that direction.

AJ: 46:32

I think there's still a lot of that too. And ultimately for me it boils down to the the theory of fun. Have you heard about, like, the levels of fun? Are you okay. No.

AJ: 46:40

Tell me. So I think this comes from the military. I've heard it from other other, like, jumpers for comes from the the jumping community I'm part of. And the idea is there's three levels of fun. The first level is a roller coaster or something that you do.

AJ: 46:51

It's enjoyable. You say that was fun. You know, you talk to your friend or your spouse. You're like, oh, that was fun. But the next day you're not like, oh, man.

AJ: 46:57

I gotta I gotta get back on that roller coaster, man. Know? Ephemeral. Level two yeah. Level two fun is not fun when you're doing it.

AJ: 47:04

Level two fun can be like hiking a mountain or building a big system or, you know, whatever, doing something that or running or, know, pick pick it, whatever you wanna do. Right? Like learning I think you're learning to kite surf. I think there's a lot of times where that is not fun when you're doing it. And it's very true for base jumping.

AJ: 47:19

You you know, you're on this, like, six hour hike up a mountain. We I jumped the Eiger with a wingsuit a few years ago, and it is not fun. It is not it's not a fun hike. It's not a it's a very pretty hike, but it's on this like crusty shale rock that keeps breaking under your feet.

Dax: 47:33

Oh my

AJ: 47:33

So it feels like two steps forward is only one. You're just kind of crushing. You're just you're just willing your way up this thing. And it's not technical to do. It's not technical to achieve.

AJ: 47:44

There's not it's like, okay, walked up the eiger. We can see it from here.

Dax: 47:47

It's just annoying. It's an annoying thing.

AJ: 47:48

Yeah. It's an annoying thing and you're you're and you're at altitude and I'm already it wasn't in great shape, still not in amazing shape. The whole thing is a grind. And and you get up there and then you jump and you land and then you're like, okay, that was fun. Yeah.

AJ: 48:02

So type two fun

Dax: 48:04

Interesting.

AJ: 48:04

Is not fun

Dax: 48:05

I like this. When you're

AJ: 48:05

doing it but it's fun in retrospect. And that's the thing that gets people to do a marathon again or do a day or whatever

Dax: 48:11

it is. Is a thing.

AJ: 48:12

Yeah. Yeah. That feeling that feeling of accomplishment is the fun part, not doing the task. Right. Level three fun is never fun.

AJ: 48:20

You're never happy about that. You never that that is never a good know, if you've ever been on a camping trip or I went on this backpacking trip in Wyoming and it was in July And we got, like, snowed in, and we didn't no idea why there was snow in the mountains at this point, but there was. And, of course, we didn't know. We're bunch of Minnesotans. We don't know anything about the mountains.

AJ: 48:36

Get back to the car, and, course, the car's battery is dead. We hadn't seen anybody in five days. And we're freaking out and we're like, okay. How much food do we have? Whatever.

AJ: 48:43

And finally, like, someone comes by and jumps the car and whatever. But it but, like, that's never been fun. I've never looked back on that and said, oh, man. Would love to have

Dax: 48:50

that experience.

AJ: 48:50

It's just, like, not camping in the snow when you're expecting, like, a summer camp and then thinking you're gonna have to live out in the wilderness until it thaws and people come and rescue you because you have no cell service. Like, that's not a fun thing. Yeah. That's like level three fun.

Dax: 49:03

Yeah. And I

AJ: 49:03

do look at software projects like that too. Like, there's like a suffer fest

Dax: 49:06

Yeah.

AJ: 49:06

In software in sometimes. And it's a key part of your development as an engineer is is do like, figuring out what has to be done and what, you know, what you can sort of really have fun with and figuring out how to make yourself do the things you don't necessarily wanna do. But there's this whole class of problems that I never wanna solve in software that are like the the Kobayashi Meru

Dax: 49:25

Right.

AJ: 49:25

To to borrow the Star Trek reference, where if you do everything 100% right, you nail it, you have the entire software is flawless, the operations are flawless, the deployment is great, maybe you're, like, migrating a system, and you do it perfectly, nobody notices.

Dax: 49:40

Right.

AJ: 49:40

Right? And anything less than that is abject failure, and every stakeholder is like, how did the system go down? What were you doing? What were you thinking? What you know.

AJ: 49:49

Those types of problems I think are key to understand as a software developer. And and how to do them is unlocks a lot of value. And especially how to do them in a way that you can motivate the team and be fun, right, and enjoy it and do the work even though you know it's a suffer fest. You know this is like level three fun. This isn't gonna be no good comes out of this, no promotions come out of this, no blog post comes out of this, it is just work.

AJ: 50:11

But you do have to do it, know, those things come up.

Dax: 50:13

Yeah. Yeah. Yeah. Exactly. And it's it's it's nice.

Dax: 50:16

I feel like as my career has progressed, I've been able to do less and less of things in that category just because like as an industry things are maturing and like there's solutions for things like that. It is like a weird thing because in the moment, it's always fun to like learn something new and like you're proud of the skills that you acquired and and you like that you know how to do certain things. Simultaneously, I am happy to never use certain skills again. I'm happy that I'm never gonna have to, like manually run a database again and like make sure it's running and make sure it doesn't go down, make sure backups are up. I know how to do that.

Dax: 50:49

I've done it before, but I'm glad to not have to do that again. I can appreciate what other people are doing for me. So yeah, I think sometimes people look at, especially people like in the serverless world that are like really using managed services. Think I sometimes externally people look at that and they're like, oh, that's you're only doing that because you don't know how to do that yourself. But it's actually the opposite.

Dax: 51:11

We know how to do it ourselves and we just never wanna do it again because we know it's just not Right. The best use of

AJ: 51:16

our time. You know it so intimately. Yeah. You know it so intimately and so well you never want to do it again. I I did I I wrote about a a user service migration Yeah.

AJ: 51:24

Where we we shut down. We had, like, acquired this company. We were migrating the users and our users. And that is, one of those things I just never wanna do again. It it wasn't it it went well and everything worked, but it is definitely a a process where if like, data could be lost if it wasn't done right.

AJ: 51:39

And and so much of your career, your development as an engineer and your developer, you know, were contributors to a business. And understanding the value of that business is key to knowing your your place and how to improve your prospects, but also the company and your team and all those things. And identifying those areas that you've done before and steering away from the quagmires is a mark of, you know, senior engineer or the person that's got that experience. And and also, like, the the thing that you're doing now at your company, you're avoiding those pitfalls so you never have to solve that problem. And doing that with serverless, that is the whole point.

AJ: 52:13

Right? You're like, no no no. This I'm gonna solve this problem once and I'm never gonna have to open this code or or Yep. Touch the system again. It will just work.

AJ: 52:22

Yeah. You know, Alex DeBry is a good friend of mine. He's a mentor. I actually I had the pleasure of he managed me at Serverless Ink for a while when we were both there.

Dax: 52:29

Oh, wow.

AJ: 52:30

So if you don't know Alex, Alex wrote literally wrote the DynamoDB book. And last week, I was in Seattle at the AWS Hero Summit, hanging out with him, and he kept getting, like, product managers at AWS were, like, just pulling him aside out of the summit to talk with him. Right? Like, he he is very respected by that group and by that company. And I think, you know, one of the things he taught me about Dynamo and just systems in general is, like, the guardrails prevent you from writing a bad query.

AJ: 52:56

And you have to do all of that work upfront to to model your system and and that is hard. And I think a lot of people look at that and say, wow, could just, you know, throw it in SQL. No no disrespect to Aaron Francis here. I love your content. But, you know and and that that didn't exist also at the time we this conversation was happening.

Dax: 53:13

Yeah.

AJ: 53:13

So you you get this world of of, you know, this guaranteed world of very low latency services just by doing a little upfront work, you never have to worry about it again. It won't let you write a bad query. And I do think those, like I look for those levers

Dax: 53:29

Yeah.

AJ: 53:29

In both serverless, but also other, know, I do a lot of containers work, love App Runner, love Fargate. Like there's a lot of these tools that exist out there for you, where you just never have to think about it again if you sort of go down the right path. And that is experience, Like

Dax: 53:42

Yeah.

AJ: 53:42

I've I've made those mistakes.

Dax: 53:44

Yeah. Yeah. Exactly. I've made those mistakes. Opting into the upfront pain, think, is once you start maturing, you start to, like, you start to say, okay, I'm just gonna this is gonna suck for the first week, but then I get this long term benefit and, like, making those choices.

Dax: 53:57

It's hard, but I think that's what as you get more senior, you start to make more of those. The Diamond one's a great example because I think people often respond saying, well, I don't have high scale. Like, I'm not dealing with massive scale, so why should I care? And the thing I always point out is it's not the fact that it can scale to a crazy place, it's what that experience looks like. When you work with, like, a database that doesn't like you said, DynamoDB doesn't let you build anything that doesn't scale automatically.

Dax: 54:27

When you're in a situation where that's not true, like if you're using a standard SQL database, which, you know, our our current products are all on MySQL, so I I still use it. But the reality is is one day, you you have a week planned. You're like, we're gonna build all these really cool things this week. We're excited. We have momentum.

Dax: 54:43

We just shipped a bunch of we just shipped a bunch of things the last two weeks. We're on a high that we're gonna keep going with it. You wake up one day and it's like, oh, this table is overloaded. Right? Like Yep.

Dax: 54:53

People are using this specific feature more than we expected and now we have to go change all of our plans that we have for this week to go do this firefighting of like, you know, figuring out, okay, what indexes do we need? Like, what optimizations do we need to make? How do we rework this? And you're gonna do it. You're gonna do it.

Dax: 55:08

It's not gonna be a big deal. You're gonna achieve it in a couple days and you're gonna forget about it gonna move on. But that moment just killed all this momentum that you had in your plans. Let's get shifted to the next week and then it's a disruption and people kind of lose lose that momentum. So, yeah, a lot of this for me is really understanding how people work and how companies work and how teams become efficient and start shipping a lot.

Dax: 55:30

And those disruptions really, really make it's it's very easy to underestimate them, but that's the difference between teams that are shipping a 100 x more than other ones. Like, they know how to build momentum and maintain it. And those, like, unexpected disruptions, those surprises, they're, like, they're killer for that type of thing.

AJ: 55:48

Absolutely. The thing I always say is that, preserve momentum. If you're, if you're if you're an engineer, even a leader or an IC or a new engineer, like, everything we do is to preserve the momentum of the team going forward. So if there is a way that I can trade okay work or good work today for, you know, the best work tomorrow, I'm gonna take the good work today. Mhmm.

AJ: 56:10

And I know that I can make things move tomorrow. And that the other thing is, you know, always be ready to ship at any point. I need to be able to press the merge button or press the deploy button. This is true. Now I'm mostly on a library team.

AJ: 56:21

I do a lot of open source work, but I also do manage a back end service that distributed system think like Go and Kafka and a few other components. And that, like always be shippable, or always be ready to ship is such a key mantra. We we we do not want to exist in a world where for a coalescing of reasons, we cannot do a deployment or make a change or roll something forward if we need to. We do that through things like feature flagging. We do that through things like small pull requests.

AJ: 56:49

Coding small is a big mantra of mine. Right? How many I don't care that you can, you know, re refactor this whole service in a 20,000 line PR. I wanna see how you can start today with three lines and how that's safe and how I can roll that back if we need to and you've thought about the impacts that could have on the underlying data store if this goes wrong. And that mentality I think carries you so far.

AJ: 57:09

And I do think, you know, there is this I I I think that that so many, like, especially early career and and hungry engineers want to kind of move mountains, not realizing that you move mountains a little bit Yeah. At a time. And should talk about this because you and I are both in our thirties. We've been doing this for for some time. So like Dax, how do you, you know, like stay motivated to keep working?

AJ: 57:32

You're you're I think we're both ICs. I'm not sure how your company's organized. But you know, everyone's contributing and you're pushing forward. Like what keeps you going through this?

Dax: 57:40

The I mean, to be honest, people are always always laugh at me for saying this, it's true. I just love money, if I'm gonna be completely honest. Like I just love finding ways to say, I put in five minutes of work, here's how much value I created and increasing that that lever over time. I'm just addicted to it. So the the it's a joke because I'm saying money and it is money but at end of the day it's like, that satisfaction I get remembering that a year ago, this thing, this value I created would have taken me two hours.

Dax: 58:10

Today, it took me ten minutes. It's like I always liken it to like a like an RTS game where in the beginning of the game, you have like nothing, just like a few villagers and you're like just figuring out food and basic things. By the end of the game, you've got like more than the you have so many systems set up for like the basics, you've moved on to like much higher level things. Building a business, building anything in tech to me feels that way and it feels addicting in in in that same way.

AJ: 58:35

Totally. I think since since we're confessing, I'm like a deeply insecure person. That definitely drove me to want to bass jump because I wanted I was a cringey kid in high school, in college. I was not, you know, it was not a good experience for me. I did not have fun.

AJ: 58:49

And I just wanted to sort of be accepted, and as a result, I had this, like, the classic millennial, you know, crave attention, crave validation

Dax: 58:55

Right.

AJ: 58:57

Life, you know. And and and that definitely that's like a big part of why I'm active on social media in in technology, and I like to give talks, is definitely, like, it comes from that part within that, like, I need to I need to be validated. Yeah. But, you know, I I that's not really, I think, what keeps me going now. I think so much of what keeps me going now is unlocking the the ability of others in making really big things move.

Dax: 59:20

Yeah.

AJ: 59:20

So I'm I'm a staff engineer at Datadog for my day job. I do have a couple of reports, but I'm not predominantly a people manager. My role, and I think sort of the canonical role of staff engineer, is to solve problems that exist horizontally at the company. And that means taking people that maybe, you know, this person's working on front and this person's working on back end or teams, right, like at our scale. It's like teams of people are doing this part and that part.

AJ: 59:45

And then there's the back end for for Datadog is not a postgres database. I just wanna go right out here and say, yeah. It's it's a little bit more than that. And so so our our front engineers are full stack and they do back and they build their own API endpoints. They build their own API contracts.

AJ: 01:00:03

And then we have a whole group of distributed systems engineers that build the actual streaming and the real time data services that power it all. And then there's people like me that are more closer to to the, like, front front end or the client libraries where this is open source libraries that you run-in your infrastructure and send data back to us. And a big part of my role is is finding the right people in all these different groups and getting them excited about a vision or an idea and helping them move forward on it together so that any one part of that doesn't seem like it was a big change, but when you step back, we've we've moved a mountain. We've done something big here. And I think as somebody in my thirties now, that has been a really big it's like unlocked that fun button all over again.

Dax: 01:00:40

Yeah. Yeah. Exactly.

AJ: 01:00:41

All over again. So so instead of, you know, oh, I shipped this this API and it's really good and I like using it, which I still do find value in. So much so much of it now is is, like, this group of six people sort of skunk work style, which is the code name for the group that built the s r 71 Blackbird kind of under the covers, right, at Lockheed. Like, that that kind of of of building and and moving pieces inside of a company is is huge. And I think that's what, like, senior and super senior staff ICs kind of do is they they have this, higher level vision of what has to be done for us to be successful in five or ten years.

AJ: 01:01:16

And they're able to work within all those groups and identify the siloed people or the people that are in different teams to to move those projects forward. And that like, I work with some incredibly motivated I I I have a couple of engineers on my team now that are young and 10 x what I ever ever was at that that age and ever will be, you know, in my career. And I love being able to sort of direct them and help them solve a problem and then watch them just go out and crush it. And that, you know, it's it's not management. It's not full time management.

AJ: 01:01:45

It's still very technical work. Like, I'm shipping code every day. But that is that's the role. You're you're you're leading, you're mentoring, you're helping people have an impact, helping people build it build a product. And I do because I am so deep in need of validation, I do like working on product teams so that I can like go off and do that.

Dax: 01:02:02

Yeah. You know, that that's awesome. Mean, that's really well put. I think it's I can relate a lot to that. And it kind of in a different angle though because we're just a team of three.

AJ: 01:02:10

Totally.

Dax: 01:02:10

But Yeah. The thing you're talking about is you see what's going on in like a very large broad way, and you see certain opportunities and you connect the dots and make sure that like certain things come together and ends up having a huge impact. And at the end the day, all we're talking about. We're having trying to, as you go further and further in your career, like the same amount of time you spend on something just starts to have a much bigger impact. We're able to now do this kind of in the open source space, right?

Dax: 01:02:35

Like, because we're so plugged into all the stuff going on, all the different tools coming out, being able to identify again, just like you said, like, there's like I mean, a great example is is a team behind Drizzle, which is, know, TypeScript the typeset way to talk to your database that is awesome and we love. But we we we saw that team early on. We were like, they built something way better than any of like, I could ever build. Like, this is just like like, you said, like, people are way better than me at certain things and it's super impressive, but very underrated. Right?

Dax: 01:03:07

Like, it's not fitting into the right place. It's being overlooked x y z things. And being able to, like, connect the dots with I know that a lot of people would benefit from this. I know they're just it's like arbitrary that they're not more popular. And finding a way to get them in front of people and and and get adoption and and see where that grows and kind of maneuver things in that way.

Dax: 01:03:28

It's, like I said, it's a fun button. It's like a new thing that I realized, oh, I can do this now. I can like see different things and have influence and and impact this impact how all this stuff comes together. Yeah. It's like the same satisfaction, like I said, the same satisfaction I get from like shipping like, a literal thing that I built with code.

Dax: 01:03:43

It's all the same. It's like the job looks very different and you're doing very different things, but your, your intelligence around it, like how you approach it, the type of thinking it takes, it's the exact same thing you've always been doing, just the pieces you're using are are quite different.

AJ: 01:03:57

Yeah, you're moving logical groups of people and their efforts and you're not necessarily doing the technical work or you know, you mean you're doing the technical work but not every single component of it. Need to compose those pieces and you also have to direct you know direct the traffic.

Dax: 01:04:12

Yeah. It

AJ: 01:04:12

is yeah. It it is type two fun. But but I think as you as you grow and you you build more software build more do whatever it is that you know you you sort of incentivize to do what you like to do. You go through the hard parts with a kind of a wry smile on your face, knowing that in three weeks or six weeks or when the release is done or when customers start to adopt it, it's gonna be good. Yeah.

AJ: 01:04:35

And you know, you're gonna be really happy you did that. And that is, you look back on that, you're like, that was a good project.

Dax: 01:04:41

Yeah.

AJ: 01:04:41

You know, and that lasts forever. I still have a team that I worked with at that sports tech company that we sort of built around the ashes of a project that was just horrible. And it burned out a lot of people. Like most of the people on that project quit. And they, like a year later, they tapped me, they're like, do you wanna get a ragtag group of people that wanna try and attempt this again?

AJ: 01:05:01

You can sort of decide how it's gonna go and how we're gonna work on it, but we do have contracts signed. Know? And it's like, oh great, thanks. No problem. Like let's just run through the ringer again.

AJ: 01:05:10

But we were able to build up a team that got so close and and we're so successful that we still we still talk to each other all the time. Like, have a Slack instance and we just get to to hang out. Now everyone's at different places all over the place. But we still, you know, connect because those bonds are sort of forged in that process. And we I think everyone looks back kinda fondly at it.

AJ: 01:05:29

At the time, it sucked. There it was a suffer fest. There was not a lot of fun that was happening day to day. There was just enough to keep people from burning out. But then we did, you know, at the end we did this really big thing.

AJ: 01:05:39

We looked back on it and we're like, wow, that was that was cool. And it is, you know, think to your point, right, like it makes money. Like that, it's not gonna be successful if it didn't make money. It was a business, we're working in a business.

Dax: 01:05:50

Yeah.

AJ: 01:05:51

So, you know, if that motivates you, that's great. But I don't think, you know, not really. Right? I think I think it motivates you to a point. Yeah.

AJ: 01:05:57

But I think really, like, when people use something I work on, like, oh, this is good. Like, oh, I like this. That feels so good. That that I

Dax: 01:06:05

definitely Making money is a side effect of that thing. Exactly. Exactly.

AJ: 01:06:08

Yeah. Building building something good. If you if you haven't read this, I think everyone everyone should read Zen and the Art of Motorcycle Maintenance by Piercing. It's a classic. I'm sure your dad was like, here son, here's your server, here's your here's your motorcycle maintenance book, here's your Piercing book.

AJ: 01:06:22

But it's just a book about quality and like what is quality? It's it's it's it's an abject measurement of something, but what what actually is it? How do you know? You know, you pick something up, you know it's quality.

Dax: 01:06:33

Yeah.

AJ: 01:06:33

But describe the quality parts of this, the qualitative parts that make this thing good. Yeah. And I I do love that about software. And and you talked a lot about linear, you know, and you you know, I I listened to your your episode. I forget the

Dax: 01:06:46

name Tomas, yeah.

AJ: 01:06:47

But Tomas, yeah, thank you. And I could tell that this is a person that also really values the work they do for the purpose of knowing that it's good. Yeah. That people will objectively assess this as good. They may not be able to tell you what's good, but when they use it, when they they touch it, when they feel it or interact with it, it feels good.

AJ: 01:07:07

And that that is a really powerful motivating factor for me too. Like it's a big big part.

Dax: 01:07:14

Likewise. Like, I I wanna make sure what I put out in the world, I can point to it and say like, that is that represents me, that represents the best of my abilities. You can judge me based off of off of that piece of thing. And it's it's it's kinda going back to what you're saying. It's just that, yeah, we can be honest about this.

Dax: 01:07:28

There's just a need for validation. Like I am the same way I want it, I need it. Like we don't have to pretend like we're all like emotionless like

AJ: 01:07:35

No, it's a part of your hierarchy of needs. Yeah, absolutely. Like respect for your work or what you do or what you spend a huge amount your waking hours do is is important. I do think it's important to walk a line and have be able to unwind because burnout is real. And I'm someone someone who also intertwines, deeply intertwines my day job with my hobbies, with my, you know, being an AWS hero and working, you know, towards towards kind of that type of of blogging, content creation sort of role.

AJ: 01:08:05

Like, that's all me, that's all one part. And I sort of bring my same part, my same self to each of those. It's not like I like compartmentalize Right. My blog when I go and work on a bug. Like so many of my blog posts, if you read them, are about bugs.

AJ: 01:08:16

Yeah. Like they started as a bug. This proactive initialization thing was a bug. You know, it started as a bug ticket. So so that is I think a superpower, but it's like a nuclear core.

AJ: 01:08:27

You have to be really careful to temper how much of that you you do or don't use. Because first off, you have to go long. We have a long career.

Dax: 01:08:35

Mhmm.

AJ: 01:08:35

Everyone's living longer than ever before. You're gonna be doing this thirty, forty years from now probably. Mhmm. Right, well hopefully not forty, but you know, longer than maybe our parents or their parents, definitely longer than their parents. Yeah.

AJ: 01:08:46

So figuring out how to do that over time is the real secret. I think anyone can have fun and anyone can sort of burn the midnight oil and hack away on something. Or whatever, right? If maybe you're not a developer, you wanna be a streamer and play games or entertain people, like that is hard work.

Dax: 01:09:02

Yeah.

AJ: 01:09:03

That is not an easy thing. I don't, I think there's this like concept of easy jobs. There are no easy jobs. Yeah. And once you have that in mind, you can, you know, you have to find a way to temper that over the long term.

AJ: 01:09:15

And I think that's what leads companies eventually to be successful, because, and you said this too, right? The most important thing you can do to be successful in five years is to be there in five years.

Dax: 01:09:25

Yep.

AJ: 01:09:25

You have to be there. You have to be there. That is that is the people giving up is the number one killer of startups, the number one killer of dreams,

Dax: 01:09:33

the number

AJ: 01:09:33

one killer of ideas. You you p zero, you have to be there.

Dax: 01:09:38

Yeah. Yeah. It's a I think someone said I don't remember where I read this, but someone was like, almost every struggle and challenge in life is just your internal debate between the short term and the long term. And almost anything in life you struggle with can probably be summarized in that way.

AJ: 01:09:55

But yeah. Yeah. I mean, short short of horrible, like, sickness and disease Right. And and all those other things that do come up, you know, and governments. But yeah, your your ability to Yeah.

AJ: 01:10:05

Your ability to persist through is is key.

Dax: 01:10:07

Yeah. Okay, awesome. Well, okay, we're at an hour now. But this was great. Okay.

Dax: 01:10:11

Thank you so much for joining. Lot of wisdom there, appreciate it.

AJ: 01:10:15

Oh, it's been fun. Yeah, I I look forward to, jumping back on when Adam buys his motorcycle and then later when he gets his skydiving license. That'll be great.

Dax: 01:10:24

Yeah. No, I think we'd definitely love to have you on again. Maybe Adam can do can do the next one and you guys can talk about it. You can influence it maybe on on some of these other hobbies. I feel like Adam is like in the his hobby is hobbies right now.

Dax: 01:10:37

That's what I feel like. I feel like he's like trying Oh, get into also such a superpower though.

AJ: 01:10:40

Like just be like, so I think so many people are scared to look dumb.

Dax: 01:10:44

Yeah.

AJ: 01:10:44

I have always looked dumb and I've never been athletic and I've never been coordinated. So it's always it's just like if you're just, okay, I'm gonna just do this thing. I'm gonna suck at it for a while. If you can embrace the suck, embrace the suffer fest, you it's just such a you have more fun

Dax: 01:11:00

Yeah. In life.

AJ: 01:11:01

You get to try more things.

Dax: 01:11:02

It's so true. And I I can give you, like, a little example of this. So I live in Miami. My wife is Cuban. Her whole family speaks Spanish.

Dax: 01:11:09

I have made basically no progress on learning how to speak Spanish because of exactly this issue. I am kind of afraid to look and sound dumb. Like, my whole identity of myself is that I, you know, I'm pretty articulate, I can express myself well. And then suddenly turning that off, like, I'm like, I don't know who I am, like, can I even? So that's like I'm letting that get in the way.

Dax: 01:11:30

But you know, that's obviously a short term thing like I'm gonna sound and look dumb for a little bit and I'm gonna figure it out and I'm gonna have a new ability to communicate which I seem like kinda get over. But

AJ: 01:11:42

Totally. Yeah. It's that's life, man. It's like a wild little adventure and no one gets out alive anyway. So you may as well may as well try a bunch of stuff.

Dax: 01:11:50

Okay. Awesome.