Stephen and I hop on the podcast to chat about some of our recent tooling, local development, and DevOps work. A little while back, we cleaned up our entire monorepo’s circular dependency problems using Madge and elbow grease. That kind of thing usually isn’t the biggest of deals and the kind of thing a super mature bundler like webpack deals with, but other bundlers might choke on. Later, we learned that we had more dependency issues like inter-package circular dependencies (nothing like production deployments to keep you honest) and used more tooling (shout out npx depcheck) to clean more of it up. Workspaces in a monorepo can also paper over missing dependencies — blech.
Another change was moving off using a .dev domain for local development, which oddly actually caused some strange and hard-to-diagnose DNS issues sometimes. We’re on .test now, which should never be a public TLD.
Time Jumps
Sponsor: Notion
Notion is an amazing collaborative tool that not only helps organize your company’s information but helps with project management as well. We know that all too well here at CodePen, as we use Notion for countless business tasks. Learn more and get started for free at notion.com. Take your first step toward an organized, happier team, today.
Transcript
[Radio channel adjustment]
Announcer: Today, on CodePen Radio.
Chris Coyier: Hey, everybody. CodePen Radio 398. This one is titled by Stephen because who else would it be titled by. DevOops.
Stephen Shaw: Oops.
Chris: [Laughter] Hey, Stephen. How are ya?
Stephen: Doing well, Chris.
Chris: There were a couple of... Actually, as we speak, there's just so much little dev ops stuff going on. It just is the season for it. It's almost like a really premature spring cleaning, in a way.
Our senior developer CTO extraordinaire Alex has been working on, like, some deployment stuff and dep ops stuff, and ran into some issues, and kind of brought it all to the team. [Laughter]
And it's like, okay, we've got some stuff to fix. And then other problems that you've surfaced, too. Let's start with one of those small ones. One of them was like we use, at CodePen, a local development environment (like every other app in the world).
And believe it or not, we use codepen. -- wait for it -- dev. We used it for a long, long time because it's like, "Yeah, development, developer, whatever. Let's use that." Long after we have chosen that, I think it became what they call a TLD.
Stephen: It wasn't. It wasn't a registerable TLD back when the dev environment was set up. Then I think it was like 2019 or so, Chrome or Google came around.
Chris: Mm-hmm.
Stephen: And it was like, "Yeah, let's make this a real thing."
Chris: Which is fun. A bunch of people registered names for it and stuff. Now you see it all over the place, I find. But yeah, I think it's Google-owned, right? If you want to buy one, you've got to buy it through them.
Along with that means that for something about TLDs -- I wish I could articulate this -- have certain special rules and stuff to them in how they behave in Chrome. What that is exactly and how that manifests is just an absolute mystery to me. But once in a while--
For me, what happened - I don't know - I'm using my cellular Internet for some reason because I'm traveling or something. I go to Chrome. I got to codepen.dev, and it just like [fart sound], "No."
It would be really quick, too. It wouldn't hang. It would just be this, like, "That's unresolvable," or something. You'd be like, "Oh, no! What's the problem?" And it would take just, you know, "Restart your computer. Restart your dev environment. Restart Chrome. Flush the DNS cache in Chrome."
You'd launch on this hourlong personal investigation. Then I'd be like, "Oh, yeah. I'm using my cellular Internet. Maybe that's the problem," or something.
Then sometimes it would resolve and sometimes it wouldn't. You had that happen to you recently. Although, I don't think it was related to your Internet, right?
Stephen: Well, no, it was. I needed to upgrade my router. I was just having trouble with Zoom dropping out and everything. My wi-fi coverage just wasn't great. So, I swapped out my router, and then this started happening all the time.
I'd encountered it before, like using a guest network or whatever at a coffee shop or whatever. It would come up, but I'd go back home and it'd be fine, so I wouldn't dig too much into it.
But there's some kind of internal host resolver within Chrome itself that's separate from the operating system and bypasses even the operating systems like redirects and ETC host file and all that.
Chris: Wow.
Stephen: And so, .dev in particular is a lot more strict about that because it's a Google TLD, but I don't know what the reasoning is.
Chris: Yeah. There was... Oh, now I feel grossly unprepared. Sorry, listeners.
There's this list of sites that are like HTTPS only that is baked into browsers. You apply to be on it. It skips this resolution step. It's just a little bit of extra security that can happen. I think we put ourselves on that. That may be part of the issue, too. But I don't know.
The point was using .dev is just fraught. Because it's a real TLD I think is the root of the problem. You would never use codepen.com. We can't use codepen.io because that's the real website, so you can't have a browser that hits that URL. We need to see production, too.
Theoretically, we could use .com, but it's just too weird when you're working locally. A lot of websites just use literally the word "localhost" or they use an IP address or something. It's a little bit more common to have a real domain because the point of the real domain is then you can make a local certificate for it, so HTTPS works locally, and that's a huge deal in browsers because there are all these APIs and stuff that only work in HTTPS. Can't do that with localhost, as far as I know.
You have to have a fricken' local URL that resolves to your dev environment. So, after all this time -- and Alex did this for us, I believe -- switched us to .test.
Stephen: CodePen.test.
Chris: Test -- I just learned this when it was brought up. Everybody apparently knew this but me -- is not a TLD nor will it ever be. It's one of these, like... It won't just happen to us that .test becomes a TLD next year or something. It's a protected one.
Stephen: In theory, it's reserved for specifically this kind of use case.
Chris: Yeah.
Stephen: Local testing domain. It can't be public or anything like that. It can't be registered.
Chris: We switched to it, and it was absolutely painless. Now everything is fine. So, I would recommend that to everyone. Use .test.
Stephen: Painless on our part.
Chris: Yeah.
Stephen: Probably a big pain for Alex converting.
Chris: Yeah. Right. I should say that as we're doing this, all this dev op stuff is really a lot of Alex's domain. I'm sure, as we're speaking, he's working on dev op stuff literally right now. [Laughter]
Stephen: Yeah.
Chris: He's just harder to get for the podcast, let's say. But he's the DevOops master.
Okay, so test is a good idea. You know you don't even have to... Like localhost, it's not localhost.com. It's just localhost. You don't need a dot in a domain thing in your ETC file - or whatever - or your host file. You can just put a word. It could just be CDPN with nothing, and that would resolve in a browser. You don't need the dot, weirdly enough. [Laughter]
But I think that's... Go with the flow, people. Don't do anything too weird because you're going to regret it.
Stephen: Yeah. Browsers tend to have opinions about what is valid and what you can have a certificate for and all that.
Chris: Mm-hmm. That's all a little mysterious.
[Guitar music starts]
Chris: This episode of CodePen Radio is brought to you in part by Notion. It's a new year, folks. I'm sure a lot of you are doing planning and all that kind of stuff for the new year. Certainly, we are at CodePen and using Notion to do it.
We've used it for a long time. That really helps keep the team on the same page. Just absolutely love it. Thanks so much for the support.
Whether you're starting a new gym routine, organizing a trip with friends, or even planning your company goals, Notion is a flexible, collaborative workspace that helps you make meaningful progress in every part of your life. Get started in seconds by choosing from thousands of templates for every task. Make it your own from to-do lists, OKRs trackers, and so much more.
Notion helps you build the exact system you want so you can work the way you work best. Start your year off right and get organized now with a free Notion account at -- all lowercase letters -- notion.com/codepen. Notion.com/codepen to learn more and get started for free right now. So awesome. Again, thanks so much for the support. Love Notion here at CodePen. The best.
[Guitar music ends]
Chris: I guess even more related to actual dev ops... How did this come up the very first time? Cut to the chase.
Before this latest round, which we'll talk about more recently, we had this problem with something called circular dependencies, meaning that something imports... X imports from Y and Y imports from X. That's weird. And we're talking about JavaScript here.
Somehow, tools like Webpack... and this is probably... you know everybody likes to make fun of Webpack, but it's been around so long that these are the kind of problems that it smooths over for you that you may not have the luxury of getting smoothed over for you as you move away from something like that. It kind of just handled.
Then as we used that less and used different things, it became a problem. Is that how it came up? It came up... It was just a thing we had to deal with a couple of months ago, and it was just a beast. [Laughter]
Stephen: [Laughter] Yeah. We have so many different compilation steps and different kind of environments. Like we're running Next, and we're running Rails. And we've got basically everything in between. Esbuild for some things, and there are too many piecemeal things. That was starting to bring up some issues, especially in deployment where we're running all of these in yet another environment to actually get things deployed.
Yeah, one of those, I think it was esbuild, was starting to break down about a file importing another file that imported the first file. And so, we had to run through every instance of that and fix those by moving things around, separating out concerns, and importing things more directly rather than having these larger index files that just made imports easier. Yeah, that was a big mess.
Chris: Yes. It was sad, right? Those index files were useful. They kept our imports a little cleaner looking and stuff.
But to solve, to not have a circular dependency problem (according to computer science) it ended up being kind of worth it for us - a little cleaner in a way. And we used this tool called Madge, M-a-d-g-e, to look for them and make a report. Then the job was to clean them all up. And we did that. [Laughter]
Pretty soulless work, in a way, but contributes to a cleaner codebase in the end. And then we put it in CI, right? Maybe we put it as a pre-commit hook? One of those or both?
Stephen: Yeah. Right. It's now... It was now in place as like a validator to make sure that we didn't get any of these circular dependencies committed so that we could still deploy the site if we wanted to.
Chris: Yeah. What's the point of doing all this work if you don't protect yourself from it happening again? I mean that's been big on our minds lately is like, if you're going to do this spring cleaning project and fix code, it's really not done until you put something in place that prevents it from happening again.
Stephen: Yeah. You can have all the internal documentation you want about how things should be done.
Chris: Right.
Stephen: But if there's not an actual pre-commit hook or anything, then that's going to break down at some point.
Chris: Yeah. That lasted for a while. It did pretty good. But then during a recent round of dev ops work, we found some--
Now here's another piece of information you should know, though. We have a mono repo. We've talked. I think it was me and you that talked about it on the podcast the last time, and it's really been working out good. At least I personally really like the mono repo. Do you want to be caught up with what's going on at CodePen? You pull main and you've got what's going on. That's great. Everything is in there.
But it does mean that it's a little unusual. If you have a really specific little project that you're working on that has its own little specific set of node dependencies that it needs, that can be pretty inefficient and complicated and weird in a setup like that. We used to use garden workspaces and Lerna. We kind of determined that Lerna wasn't actually that useful for us.
We're just on Yarn workspaces now. So, at the top level package.json, it says these are these additional workspaces. And it does some efficiency stuff, right? If there are packages that are shared between two things, it moves those packages up to the top-level node modules folder and references from there.
I don't know. Is efficiency the only reason to use it? I think it may be. [Laughter]
Stephen: It basically sets everything up as like a symlink so that, as you're developing, you can be working across all these different packages.
Chris: Mm-hmm.
Stephen: And because it's symlinked, those changes are instant and all the other packages that are depending on it.
Chris: Nice.
Stephen: That's at least my understanding of that. But that definitely speeds up our development so we're not having to work in one package and compile everything and publish it in some kind of way.
Chris: Right.
Stephen: We're able to work on everything. Yeah.
Chris: Yeah. Pretty cool. We have our own little internal packages set up, which is nice. But we started running into some more problems.
Madge can check for circular dependencies across packages but didn't check for internal to one package.
Stephen: Inter-package circular dependencies.
Chris: Right.
Stephen: I think it's what we were calling it.
Chris: And it didn't really look for unused ones, which is an interesting one, too - stuff that's in your package.json that isn't used anywhere else. That should be cleaned up. Why not do that? Kind of stuff.
There's some NPM tool called depcheck that we look for.
I think it turns out, in the end... I mean this is all evolving fast as we speak that we got rid of Madge because it had some issues with it too. I think Alex is basically writing his own dependency checker at this point.
Stephen: Yeah. Stay tuned for that in a podcast episode.
Chris: Yeah. [Laughter] Right, but it's working pretty good. It's like spring cleaning time around here. We're cleaning up our package.json files, making sure our dependencies are super clean, yadda-yadda-yadda. A lot of work going on in that way. Right? [Laughter]
Stephen: Yeah.
Chris: It's been a big journey because you've got to make sure everything works. You can't just go around messing with your dependencies and cross your fingers. Every time you touch anything, you've really got to rerun everything to make sure it's okay - up to and including totally wiping out your node modules before testing it again.
Stephen: Yeah.
Chris: Because, for whatever reason, you just cannot trust node modules that have been yarned over ten times.
Stephen: Have you been doing Yarn cache clean as well?
Chris: No!
Stephen: That's a fun one. That completely erases your local cache of these packages.
Chris: Oh...
Stephen: That Yarn globally has so that the next time you Yarn install, it has to redownload everything.
Chris: Extra slow.
Stephen: Oh, boy.
Chris: Great.
Stephen: Yeah.
Chris: [Laughter] It's already slow enough. Just wiping node modules on our big ass project takes a minute just there. Then if you've got to do the dev environment too, it's slow. I think that's actually an important distinction here is that I think all of us are real capable of doing this really clean, detailed work, but sometimes it kind of gets skipped when you're busy working on a feature or a bug or there's some kind of, like, "We really need to get this done," energy that comes that's not particularly concerned about one little line of a package.json file that might be wrong.
Stephen: Yeah.
Chris: Even if you think about it for a minute, you're like, "Ugh... In order to actually test this, I really need to stop, wipe my node modules, run this tool, cache clean, do all this extra work that's really kind of slow."
I don't know. It's just not... We just don't do it enough. So, there's that.
TypeScript is involved with some of this stuff, too. How, in a way? I guess we have these individual builds for all of our packages that we have to make decisions about. Do they each have their individual version of TypeScript, or can we depend on the top-level workspace one? Are the builds consistent?
Stephen: Yeah, we had to drop Madge for that reason. Something in Madge's dependencies was referencing a really old version of TypeScript, and that was forcing that version of TypeScript for all the other packages that we were doing.
Chris: Oh, really?!
Stephen: And so, yeah.
Chris: That's horrible.
Stephen: Alex and Robert were going back and forth on the right solution for this, and we have some advanced stuff that's pulling the TS configs and building all of those from the top level and all this kind of stuff. Ultimately, Madge just needed to go.
Chris: [Laughter] Farewell. You were useful once.
Stephen: Yeah. I think we're searching for a replacement if Alex doesn't just write one first.
Chris: [Laughter] Yeah.
Stephen: Yeah. Madge is more about the import circular dependencies whereas depcheck is just checking the package.json--
Chris: Right.
Stephen: --to see the packages that are referenced and then making sure that those are used and that those aren't circular.
Chris: It's very interesting that there's not one package. They're such related concepts.
Stephen: Yeah.
Chris: You know? It's weird that they don't do both. But you know, so be it.
While we're doing this, we're syncing up on versions of dependencies and stuff like that. And then, in some cases, upgrading it, like, "Ooh, this package is using Jest 29 whereas this one is using Jest 27," or something. Then our spirit is, "Just get it upgraded." You know? Just do it. You know?
But the problem is there tends to be thought behind that that's a little hard to document because JSON sucks. [Laughter] I shouldn't say that.
I'm a big fan of JSON. I'm beginning to like it more and more as time goes on. But just the one fact you can't put comments in those dang files really can grate on a guy.
Stephen: Yeah.
Chris: Because there's knowledge that you would love to share in those files sometimes. You're like, "This is on this version for this reason." You know?
Stephen: This is a loadbearing package. Do not remote.
Chris: [Laughter] I know. So, it was just this tiny one that came up yesterday that was like, "Oh, let's upgrade to Jest 29. Oh, Jest 29 has this requirement that a loader returns an object instead of a string," blah-blah-blah.
And so, this loader that clearly hasn't been touched in six years is now broken, so we're like, "Are we monkey patching? Are we writing our own?"
While you're doing this dev oops work, [laughter] it opens up little rabbit holes all over the place.
Stephen: That snowball.
Chris:. Yeah. Yeah. You know today is like, okay, we've done all of this work and cleaned up all this stuff. Let's clean up the package.json files themselves because they're in different orders, and some of them have extra keys in there that are like, "Why does this one have keywords in it? Is something using that? Does this one really need a homepage link?" You know? [Laughter]
There's just weird stuff, and it's not that important of work, but if you get it all fixed up once, that's good. It's part of the spring cleaning.
Stephen: The more consistent they are between the different files, that's just going to help us as we maintain things going forward. We're not having to parse why we're using files in one and exports in another.
Chris: Yeah.
Stephen: If we can find those consistencies, that'll help us.
Chris: Yes. Indeed. There's also, we chose this architecture. We chose the fact that we're going to build individual packages, and we're going to import the packages from other packages that we built, all to serve the kind of highest-level things that we're building. This was all human-invented stuff here.
One of the things that Madge nor depcheck nor anything else can really help you with is our kind of handshake agreements on what the hierarchy of these packages are. For example, we have an icons package. In a perfect world, that icons package does not import anything else that we wrote.
Stephen: Self-contained.
Chris: Self-contained. In fact, our entire design system library components are that way too. I mean they depend on React and stuff, but they don't depend on anything else that we wrote in there, which is just kind of this line in the sand for cleanliness because when you're dealing with circular dependencies, which we've been talking about as this weird kind of deployment problem, you don't want a hooks package that depends on your library and a library package that depends on your hooks. Then that's a circular dependency and that's a problem.
Your tools will catch that. But you won't even get into that situation if you are adhering by your handshake rules of the packages that you build.
Anyway, that's dev ops. It's been a journey. I think it's been cool to involve the whole team in it. It tends to be something that if you have a dev ops person that they can be frustrated by this and think it's entirely their job. But the package.json JavaScript universe is kind of - like it or not - a front-end developer concern that the load can be spread to everybody. And thus, the knowledge is shared too. It ends up for being better work.
I'm very glad that Alex isn't just doing this himself because, anyway, I just feel a lot more knowledgeable about what's going on now that I wouldn't if he chose not to share the burden here.
Stephen: It's helpful to get your hands dirty with that every now and then.
Chris: And it's all making us all the hungrier, like, "Can we do something else now?" [Laughter]
I like the one where we make the border-radius. That's fun.
Stephen: [Laughter]
Chris: That's a lot more fun.
Stephen: That's my favorite.
Chris: [Laughter] All right. Any final thoughts?
Stephen: That's it. That's it.
Chris: Dev ops, baby. Thanks. Special thanks to Alex who didn't appear on this show but does most of this work.
All right. Take care. Bye.
Stephen: Bye.
[Radio channel adjustment]